04 3 / 2013

As DuckDuckGo continues to grow, I’ve been thinking about how to hire the right people.  A strong filter we employ is only considering inbound requests.  I’ve read all sorts of stuff on the topic ranging from the crazy “unicorn” theories to Orson Scott Card’s Developer Bees article.

A crowd favorite seems to be the 10x developer.  The tech scene is pretty obsessed with the whole order of magnitude thing in general right now (moon shots).  As usual though, there is no real measurement involved.  

"Now, in software, and it used to be the case in hardware, the difference between the average software developer and the best is 50:1; Maybe even 100:1.” - Steve Jobs

These multipliers are often unjustly converted to include a less valuable concept: productivity.  This is a trap.  As engineers, we are very sensitive to productivity and efficiency, but it’s not really what separates a good engineer from a great one.  All smart people learn fast.

"I learned a huge amount in a short period of time […] basically doubling my programming skills in the space of six months, I realized how relative it all was." - John Carmack

Great engineers don’t solve problems - they redefine them.  In other words, great engineers deftly navigate false premises.  This is an invaluable asset and is far more rare than a productive engineer.  It changes the playing field entirely.  If you’re not constrained by the problem, the solution can be far more valuable.

At DuckDuckGo we had a problem where we wanted to allow users to store their settings for the site on our servers (easy), but we wanted to do it without deanonymizing them (ouch).  Maximizing for anonymity and transparency while explicitly retaining state for a particular user are not things that usually coincide.

Russell Holt, a senior engineer here, redefined the problem and just asked for a pass phrase - that way the users could remain anonymous.  He even made up a clever way to check for collisions.

Solving complicated problems with simple solutions requires the ability to recognize false premises and jettison them (in this case the need for a UUID).  In this way, the redefinition of a problem helps free you from being bound to conventional solutions that often don’t map well to the original problem.

25 2 / 2013

Art for somewhere in DDG.

Art for somewhere in DDG.

24 2 / 2013

I have been thinking about how people think and how that affects communication.  A thoughtfully communicated good idea is more pervasive than a carelessly communicated great one.  The way we reason about an idea often dictates how we communicate it.  I’ve noticed two prominent methods of reasoning amongst my peers.

Analogy

Analogical reasoning is a shortcut for reasoning from first principles.  It helps us encapsulate and apply a group of perceived like concepts from one thing to another instead of having to start from scratch every time.  

image

It’s great for assimilating knowledge of a concept or system, but it can also be a trap when we build new things because new things aren’t analogous to old things.  In biological evolution, everything is being built by analogy - that’s part of the reason why it takes so long.  All of this interpolation and extrapolation gets complicated fast.

When you reason via analogy your premises are inferences from prior knowledge instead of well established axioms.  This makes communication much more difficult because it leads to untenable whims like instincts and “gut” feelings.

First Principles

Neatly packed first principles define the foundation of our understanding the universe.  Reasoning from first principles can be a much more arduous process.  Solving problems by starting directly with established axioms can lead to extraordinary and unexpected results.  Smaller postulates leave more room for greater outcomes.

image

Under the auspices of communication, first principles help unify the team by demanding discipline in the dialogue.  If you’re able to understand and agree with the premises that lead up to the conclusion, you can rely on the conclusion.

Agreeing on first principles helps to mitigate disputes as well because of this reason.  Gabe and I come from different backgrounds that have a lot of overlap and when we disagree we’ve employed this method to move on quickly.

I suggest creatives try to do this as well.  I’ve often found myself reasoning about a problem via analogy and then doing the implementation from first principles.  It’s often the inverse that I really want.  I strive to reason about how to solve the problem from first principles, find as much code as possible that gets me there and then add whatever else I need on top.

Next time you are trapped by an analogy, try backing up and starting with what you know to be true.

16 2 / 2013

Can someone please tell me what "clean" code is?  Because I find that term to be ambiguous.  Striving to write clean code is like making a new years resolution to go on a diet without any measurable goals - it just won’t happen and can be unhealthy.

As DuckDuckGo has grown, I have found myself thinking about refactoring and when / where it is appropriate to spend time revisiting something already done versus doing something new.

The three main camps I’ve seen comprise: always be refactoring, refactor on some condition, and just build new stuff.

Always be refactoring:

I cannot see an argument, at least in the startup space, to always be refactoring; but this happens all the time.  It’s a result of an engineer’s internal conflict to always be honing a craft.  This is not necessarily a bad thing, but it is for your startup because you’re not really moving.  Worse yet, since it’s a continuum, it’s hard to really measure any gains since they are always approaching 0.

Always be building:

Fine.  So I’ll just build new stuff.  This is often the most fun (at least in my experience :) and can become intoxicating because it’s the shortest path to tangible gratification.

image

Since it’s hard to anticipate future requirements, and since you’re building things that rely on one another, this invariably leads to a critical mass of technical overhead.  Each new thing built yields more resistance than the last until it becomes intractable.

Conditional refactoring:

Refactoring when a condition is met is a nice middle ground.  Defining a condition is difficult.  Software is complex, so conditions are complex.  It’s a slippery slope with the “always be refactoring” state at the bottom.  A trick I’ve found to define that condition is to continue building new stuff until something becomes a nuisance (readability, performance, interface, etc.) but is not an arrant pain.  It’s interesting to watch the conditions surface.  An added benefit to refactoring this way is the distribution of how things work since refactoring happens along the way instead of being a focal point.

DuckDuckGo and other startups are filled with keyboard cowboys building a product.  Staying out of the way of each other is important.  If you hit a refactor condition,”clean” the code up :).

25 1 / 2013

I am often asked if DuckDuckGo plans to launch a query suggestion service into the search box, so I figured I would provide some insight:

Probably not in it’s industry standard form, and here is why.


Part of using DuckDuckGo is about getting unbiased and unfiltered results.  From a technology point of view, we take your query and do our best to derive your intent.  We then either answer it directly or dispatch requests to places we think can answer it better and then agglomerate those results in some relatively clever and easy to consume way.

Instant search doesn’t allow for this process to happen in an unbiased way because it becomes a numbers game.  The user is often lead to an intent / final destination.  This is kind of a cop out because it means:

1) Certain queries and partial queries proliferate (like autocomplete).
2) The user can be distracted from the original intent, by tangential and more popular material (as a result of 1).

Google Instant is impressively fast; but it can be very distracting.  I’m unfamiliar with how they do it, but you can sort of see the pieces that would be required.  A longtail autocomplete service based on previous queries and the intense caching of some top percentage of what’s in that tree.

As a user using an instant service, you have two things at your disposal to get your answer faster.  The autocomplete query suggestions, and then the whole page of results. The suggeted queries can be nice because you might have a fuzzy question in your mind and being led to a result can be nice.  The subsequent pages being loaded on the fly are just information overload.

We have our own take on all this, but it is only a part of our upcoming mobile apps.  It is a permutation of autocomplete and instant, because it offers you instant answers on the fly.  This currently extends to much more than instant answers you’ll see in the industry like 2+2, but doesn’t contain our entire zero click corpus.  We are launching an internal effort to remedy this and also include all of the wonderful work that has come from DuckDuckHack.

11 10 / 2011

So it’s official, I work at DuckDuckGo!  I’ve gotten a lot of congratulations across a wide spectrum of mediums, but I’ve also gotten a lot of “how do you like it” and “how did you get it” questions.  This post aims to address working at DuckDuckGo, if you’d like to know why in Saint Peter’s name Gabe picked me, see: ye.gg/inbound.
 
 
a hat trick 
First, I’d like to say that I’ve never done anything like this previously; working at DuckDuckGo is vastly different than (in my limited experience) any other place I’ve worked before.  People talk about wearing many hats at startups, and I too have experienced this before; but working at DuckDuckGo, you wake up and realize your whole wardrobe is made of hats.  At DuckDuckGo, hats are all you wear.
 
Today I woke up and realized we didn’t get a discount we should have so I put on my financial-hat underwear.  I wanted to give some feedback to an API we use and put on my partner relations-hat on my left foot.  Gabe pinged me with a bug I introduced in our Punchfork integration on my dev machine late last night (wearing my insomnia-hat), so the debug-hat went on the right foot.  Then I found a bottle-neck in some core code from a while ago … and having no other place to put the systems engineer-hat (not cold enough for gloves yet) it went on my head.  Working at DuckDuckGo is controlled chaos; and I love it.  
 
Story time! When I was in elementary school, it was clear to everyone that I had pretty fierce ADHD, and still have it now.  Then and now I combat it with a simple kitchen timer.  In grade school it helped me because I would start with a homework subject, put 30 minutes on the timer and start the pipeline.  When the 30 was up, I’d context switch to the next subject.  If I was having trouble focusing for 30, I’d mitigate it with a time decrease and recurse.  I still use this method to this day breaking it up into design and implementation intervals.  This method enables me to allocate my chaotic energy (entropy) into some useful work.
 
At DuckDuckGo, chaos and entropy are synonomous.  At least for long as I’ve been associated with the company, seeing people come and go, I’ve come to realize that how we harness this energy is based on mindset, where static tools arrive from an evolutionary cycle.
 
vision
This is super broad and gimmicky I know, but this company is an outlier.  So hear me out.  When I first started working on the core of DuckDuckGo (before I was hired) it was like entering Narnia.  It is a complex system.  One minute it was sunny and the next it was snowing and some bi-pedal goat was pissing me off.  Gabriel is a hacker, plain and simple; he get’s the job done.  It’s not always the prettiest, but it works and often works fast.  And yes, it has been a sink or swim kind of arragement.  But the best thing is, once your swimming or even just avoiding death, Gabe is all about learning.  To survive at DuckDuckGo you have to always want to be learning.  If something sucks or you can do it better, it gets replaced - period (hence evolutionary cycle).  Learning, working smart (more on this later), consistently trying to use things that suck less, and a zero-tolerance b.s. policy have served the company well.
 
trust
I know people at DuckDuckGo have my back and will do the right thing even if it’s not in their best interest.  Enough said.
 
asynchrony and agility
Gabe and I are at different stages in our lives.  I stay up on 36 hour benders, but he has a whole family and a ton of responsibilities.  I’d be lying to you if I didn’t say it frustrates me sometimes, but I totally understand.  To be honest, him being busy with other things has done more for me since it has accelerated my ability to be agile throughout the entire system.  We’ve grown to be a very asynchronous team with no explicit tasks.  If one person is sick the other person can pick up the slack, if one person is pissed off with too many emails the other can adjust.  It’s really nice to know I can just stop everything else and ship something I’ve wanted to for a while and everything will be ok.  And if not, I’ll know when I need to.  Having multiple mediums to broadcast share vs. direct share helps a lot too.  Skype versus Yammer etc.
 
get on with it… 
 
I’m really excited to be where I am at right now.  I have a lot of things to do and I really enjoy working everyday.  While the actual technical work is very rewarding (read challenging :) it’s been fun getting to know Gabe and the rest of the team.  I expect to make a life-long friends from this job that I respect and can rely on.  In my roughly 1 1/2 years of contributions to DuckDuckGo I’ve learned a ton and expect to learn a lot more.  There is more to computer science than just understanding math, computer architecture and how to write a simple compiler or a driver.  I’ve come to understand that this company is going to give me one of the most important components to my career: shipping a product that I’ve helped build from the ground up.  It’s one thing to work at a company and be told to spot-weld weaknesses in an I-beam all day, it’s a totally different thing when you’re boss asks you if you approve of the cement grade chosen for the foundation.