Hunting for Insights

The plight of the analyst

I dove right into the basics of the web, learning php, html, css, and javascript as quickly as I could. For some reason programming really clicked for me. Topics like object oriented programming and scope seemed to intuitively make sense, even as a complete beginner. Within a month I was mocking up and building real websites. The instant reward of seeing the fruits of my labor on my browser was invaluable. Without that gratification, I might have given up the computer science thing altogether.

After a year of CS at Columbia and an internship as a data scientist at Facebook, my interests began to shift. Instead of using computers to build things, I became more interested in using them to make sense of data. I slowly transitioned from using sublime text (dating myself here) to jupyter notebooks. I began to favor a more “fast and loose” coding style over the production quality style required by an engineer working on a large codebase. I began using computers to understand, not necessarily to create.

7 years after my first foray into the world of bits, there’s nothing more interesting to me than using machines to understand human behavior. The major events of our time aren’t usually singular actions taken by an individual, but an amalgamation of decisions and interactions that foment into world-changing dynamics. We can usually identify these dynamics when we see them at a macro level, but piecing the story together from a hundred million little data points is a skill in its own right.

I’ve found that my relationship with data analysis is different from that of building in how I’m rewarded. When I’m building something like a website, 9 times out of 10 I’ll be successful if I just don’t give up. I might struggle. I might have to go take a walk. I might have to go to bed because I’m so frustrated. The end result might suck. But if I work out the kinks there’s no reason why the program shouldn’t work. It’s very rare that something I begin building ends up being completely infeasible technically (this doesn’t include ML projects 😅).

With data analysis, there’s no such guarantee. When you’re playing with data you’re looking for an effect that may or not be there. That sigh of relief that the people give at the end of the video above might never come.

As an example, you may suspect that there’s something fishy going with your company’s finances, so you shovel out a heap of transaction records from the database and investigate. It’s entirely possible that you won’t find anything to justify the claim. This might be because there’s actually nothing going on. It might be because you failed to find it. It might be because you didn’t include the one feature you’d need to find the effect. You can work for days on end and have nothing to show for it but a directory full of messy notebooks and a backlog of unopened text messages and emails.

The experience of diving deep into an analysis feels much closer to hunting or fishing than it does to engineering. The code you write is simply a tool to tease out some meaning from the data. You’re playing sleuth, making sense of disparate pieces of information hoping that they’ll lead you to your target. Like hunting though, the target might be concealed in brush or burrowed away in a den. You could be chasing a target that doesn’t exist at all. But you press on, hoping to stumble upon the 14-point buck.

Related posts:

The door is ajar and I push through slowly, unsure for a second if I should have knocked. He looks up from his desk. He has been expecting me of course, sending me a text saying he could see me…

Machine learning pipeline building explained: a pipeline, why you need it, pipeline’s key elements, and tools to use.

Perusahaan Konstruksi Spesialis & Profesional di Bidang Pembuatan Green House Sekala Kecil, Menengah Dan Besar Dengan dukungan team ahli kita akan mendesain Green House sesuai dengan kebutuhan dan…