clouds and compilers and startups

re#define the problem

As DuckDuckGo continues to grow, I’ve been thinking about how to hire the right people.  A strong filter we employ is only considering inbound requests.  I’ve read all sorts of stuff on the topic ranging from the crazy “unicorn” theories to Orson Scott Card’s Developer Bees article.

A crowd favorite seems to be the 10x developer.  The tech scene is pretty obsessed with the whole order of magnitude thing in general right now (moon shots).  As usual though, there is no real measurement involved.  

“Now, in software, and it used to be the case in hardware, the difference between the average software developer and the best is 50:1; Maybe even 100:1.” - Steve Jobs

These multipliers are often unjustly converted to include a less valuable concept: productivity.  This is a trap.  As engineers, we are very sensitive to productivity and efficiency, but it’s not really what separates a good engineer from a great one.  All smart people learn fast.

“I learned a huge amount in a short period of time […] basically doubling my programming skills in the space of six months, I realized how relative it all was.” - John Carmack

Great engineers don’t solve problems - they redefine them.  In other words, great engineers deftly navigate false premises.  This is an invaluable asset and is far more rare than a productive engineer.  It changes the playing field entirely.  If you’re not constrained by the problem, the solution can be far more valuable.

At DuckDuckGo we had a problem where we wanted to allow users to store their settings for the site on our servers (easy), but we wanted to do it without deanonymizing them (ouch).  Maximizing for anonymity and transparency while explicitly retaining state for a particular user are not things that usually coincide.

Russell Holt, a senior engineer here, redefined the problem and just asked for a pass phrase - that way the users could remain anonymous.  He even made up a clever way to check for collisions.

Solving complicated problems with simple solutions requires the ability to recognize false premises and jettison them (in this case the need for a UUID).  In this way, the redefinition of a problem helps free you from being bound to conventional solutions that often don’t map well to the original problem.

types of thinking

I have been thinking about how people think and how that affects communication.  A thoughtfully communicated good idea is more pervasive than a carelessly communicated great one.  The way we reason about an idea often dictates how we communicate it.  I’ve noticed two prominent methods of reasoning amongst my peers.

Analogy

Analogical reasoning is a shortcut for reasoning from first principles.  It helps us encapsulate and apply a group of perceived like concepts from one thing to another instead of having to start from scratch every time.  

It’s great for assimilating knowledge of a concept or system, but it can also be a trap when we build new things because new things aren’t analogous to old things.  In biological evolution, everything is being built by analogy - that’s part of the reason why it takes so long.  All of this interpolation and extrapolation gets complicated fast.

When you reason via analogy your premises are inferences from prior knowledge instead of well established axioms.  This makes communication much more difficult because it leads to untenable whims like instincts and “gut” feelings.

First Principles

Neatly packed first principles define the foundation of our understanding the universe.  Reasoning from first principles can be a much more arduous process.  Solving problems by starting directly with established axioms can lead to extraordinary and unexpected results.  Smaller postulates leave more room for greater outcomes.

Under the auspices of communication, first principles help unify the team by demanding discipline in the dialogue.  If you’re able to understand and agree with the premises that lead up to the conclusion, you can rely on the conclusion.

Agreeing on first principles helps to mitigate disputes as well because of this reason.  Gabe and I come from different backgrounds that have a lot of overlap and when we disagree we’ve employed this method to move on quickly.

I suggest creatives try to do this as well.  I’ve often found myself reasoning about a problem via analogy and then doing the implementation from first principles.  It’s often the inverse that I really want.  I strive to reason about how to solve the problem from first principles, find as much code as possible that gets me there and then add whatever else I need on top.

Next time you are trapped by an analogy, try backing up and starting with what you know to be true.

The Recurse of the Refactor

Can someone please tell me what “clean” code is?  Because I find that term to be ambiguous.  Striving to write clean code is like making a new years resolution to go on a diet without any measurable goals - it just won’t happen and can be unhealthy.

As DuckDuckGo has grown, I have found myself thinking about refactoring and when / where it is appropriate to spend time revisiting something already done versus doing something new.

The three main camps I’ve seen comprise: always be refactoring, refactor on some condition, and just build new stuff.

Always be refactoring:

I cannot see an argument, at least in the startup space, to always be refactoring; but this happens all the time.  It’s a result of an engineer’s internal conflict to always be honing a craft.  This is not necessarily a bad thing, but it is for your startup because you’re not really moving.  Worse yet, since it’s a continuum, it’s hard to really measure any gains since they are always approaching 0.

Always be building:

Fine.  So I’ll just build new stuff.  This is often the most fun (at least in my experience :) and can become intoxicating because it’s the shortest path to tangible gratification.

Since it’s hard to anticipate future requirements, and since you’re building things that rely on one another, this invariably leads to a critical mass of technical overhead.  Each new thing built yields more resistance than the last until it becomes intractable.

Conditional refactoring:

Refactoring when a condition is met is a nice middle ground.  Defining a condition is difficult.  Software is complex, so conditions are complex.  It’s a slippery slope with the “always be refactoring” state at the bottom.  A trick I’ve found to define that condition is to continue building new stuff until something becomes a nuisance (readability, performance, interface, etc.) but is not an arrant pain.  It’s interesting to watch the conditions surface.  An added benefit to refactoring this way is the distribution of how things work since refactoring happens along the way instead of being a focal point.

DuckDuckGo and other startups are filled with keyboard cowboys building a product.  Staying out of the way of each other is important.  If you hit a refactor condition,”clean” the code up :).

Autocomplete and DuckDuckGo

I am often asked if DuckDuckGo plans to launch a query suggestion service into the search box, so I figured I would provide some insight:

Probably not in it’s industry standard form, and here is why.

Part of using DuckDuckGo is about getting unbiased and unfiltered results.  From a technology point of view, we take your query and do our best to derive your intent.  We then either answer it directly or dispatch requests to places we think can answer it better and then agglomerate those results in some relatively clever and easy to consume way.

Instant search doesn’t allow for this process to happen in an unbiased way because it becomes a numbers game.  The user is often lead to an intent / final destination.  This is kind of a cop out because it means:

1) Certain queries and partial queries proliferate (like autocomplete).
2) The user can be distracted from the original intent, by tangential and more popular material (as a result of 1).

Google Instant is impressively fast; but it can be very distracting.  I’m unfamiliar with how they do it, but you can sort of see the pieces that would be required.  A longtail autocomplete service based on previous queries and the intense caching of some top percentage of what’s in that tree.

As a user using an instant service, you have two things at your disposal to get your answer faster.  The autocomplete query suggestions, and then the whole page of results. The suggeted queries can be nice because you might have a fuzzy question in your mind and being led to a result can be nice.  The subsequent pages being loaded on the fly are just information overload.

We have our own take on all this, but it is only a part of our upcoming mobile apps.  It is a permutation of autocomplete and instant, because it offers you instant answers on the fly.  This currently extends to much more than instant answers you’ll see in the industry like 2+2, but doesn’t contain our entire zero click corpus.  We are launching an internal effort to remedy this and also include all of the wonderful work that has come from DuckDuckHack.

Working at DuckDuckGo