Engineering complexification happens because of how we teach engineering.

Last month Elon Musk announced his plan to make the source code to the Twitter recommendation algorithm open. His motivation was apparently that the “algorithm is overly complex and not fully understood internally”. It is tempting to blame this on the mass layoffs that took place recently. But that might be an easy excuse for something much more problematic in engineering.

For many years I have carried around a suitcase talk entitled “Engineering Complexity”. The gist of it is that the systems we create have a tendency to become more complex over time for a variety of good reasons. One of the main reasons is that most engineers are motivated by design as a creative process but taught engineering as a problem solving process. We become engineers more because we want to make new things than because we want to make existing things better.

The effect of this mentality on engineering is profound. Imagine a freshly graduated engineer in training (EIT) who is instructed to solve a problem with a system that is widely used. Chances are the solution will be an addition to the system that has gone through a rigorous requirements analysis, design development, production, verification, and validation process before being evaluated by a select group of users and ultimately released to the general public. This is a standard procedure because (1) that’s how it’s taught in engineering school, (2) it’s always been done that way, and (3) EITs cannot imagine saying “the problem with the system X is you need to remove component Y”.

The result of this is that nearly all engineered systems become more complex over time. We add new components to existing systems to meet new needs not anticipated when the original systems were created. As more components are added, the opportunities for unexpected and undesirable behaviors grow and systems become increasingly unwieldy and unpredictable. Eventually systems start to exhibit emergent behavior that runs counter to the original design, they become brittle, and suddenly a seemingly unrelated innocuous event causes them to fail spectacularly. Then we retire them and start over, usually at great expense.

This hardly seems like an optimal approach. But it is the inevitable outcome from an engineering philosophy and methodology that cannot back up, undo previous design decisions, and remove components from existing systems without causing them to fail. Engineering is a one-way process.

You can see this all play out in the approach that Twitter is taking to fixing their recommendation algorithm. They’d like to fix the existing algorithm. But it was complexified to the point that it’s no longer clear what it actually does or how it works. Adding more elements to it isn’t going to make it better and nobody knows how to identify what should be removed or replaced. So Twitter’s only option is to start over.

If only they offered a course in engineering simplification.