The Four Horsemen of the Appocalypse

After being around software for a decade, I have realized just how difficult things have gotten. While the accessibility of software development has gotten better with languages like Node.js, the difficulty in building a production ready application has more moving parts than ever. These moving parts also have a lot of complexity in production environments. I want to talk about what I believe the four horsement of the appocalypse are and how they are increasing the entropy of our software.

Infrastructure

The first horseman of complexity is the infrastructure, but not just how difficult it is to create it. When creating infrastructure, you have to take into account how long you intend the service to be available and how much there is to maintain. The more moving parts you place in your network, the more opportunities for mis-configured servers and other similar digital incompetencies.

Architecture

The second horseman is the architecture of your application. The pieces of the puzzle we call the web haven't changed much over the past couple of decades, but where it has changed has not been exactly for the best. We have web servers, caching servers, database servers, firewalls, and virtual networks. This is done with great amounts of effort on very smart people to keep the infrastructure stable and consistent over the years while squeaking out more performance.

Where we have seeen complication slip into our lives is how the service software we write communicate with each other. We went from monolithic backend services to dozens and in some cases thousands (looking at you Netflix) of microservices that need to communicate directly with each other and in some cases communicate with all other microservices in the architecture of your application. These microservices are driven by message brokers (a.k.a. pubsubs) that have messages stored in channels. The microservices pull or consume these messages and then perform a task. This task might be updating your user account's address or something like that.

The problem comes in when a pubsub goes down and loses a bunch of messages. The microservice no longer has the ability to consume these messages and so they don't get processed. While high-availability clusters can mitigate this issue, you end up spending double to triple the amount of money on infrastructure to achieve this without those machines contributing towards the processing of data. Effectively complicating our infrastructure without gaining any efficiency, only resillience.

Software

The next horseman is the complexity of the software itself. There are a few things in this category that can bloat complexity. The first one being the number of dependencies you have to manage. Adding more dependencies to your application is a massive risk in the maintainence and operation of your software. Additional dependencies mean more migrations to newer versions of your dependencies (which usually means more developer resources needed to maintain what you already had) and more dependencies to audit for security concerns. While an update to CSS has never forced anybody to change how their website code, a change to TailwindCSS will have you refactoring code that you might not have ever had to change. So how can we combat this in our applications? The answer is very simple. Don't add dependencies to your application.

Another way our software can explode in complexity is the number of external services that we depend on. While another form of dependency, this one is a bit different. By depending on external services we are giving someone else influence over how our software works. If they change their service you will once again be refactoring code that worked perfectly fine before the API changes. These changes to upstream services can also affect your businesses bottom line. We learned this during the Reddit debacle recently, where independently developed mobile reddit clients had their API costs skyrocket. While it may not be easier to develop our own full-stack software, it will reduce the number of headaches and refactors that eat up our time and productivity.

Operations

The final and potentially most egregious horseman is operations. This one is by far the worst because it affects everybody. The more build steps there are in your pipeline, the more steps you are adding between pushing code and seeing it live on your website. The whole point of DevOps was to create the shortest path that provides the most confidence that this software is ready to release. And if it is not ready, we will check tomorrow to see if it is ready. What we actually got is a rigid set of rules that sometimes works, but often times fails for reasons that are not even related to our code.

Takeaways

I think that we should be solving the problems we create. We cannot continue to stack abstractions on top of abstractions and expect it to operate efficiently. At some point we need a consolidation of how software is deployed to a server. I recommend reducing the need to build as much as possible. I don't think it is too far-fetched to need to build your application, but when you need a build system for your application, your libraries, and your styles on top of unit tests, end-to-end tests, and integration tests I feel like we may have taken it too far. This is the window we of time we not only want to simplify but need to simplify to release quickly. This simplicity will compound vertically. By having simpler builds we will also have simpler deployments to higher environments (test and staging) which also makes testing simpler.