Something I hear occasionally from some software people¹ is something along the lines of: "Well, the hard part is figured out, and the rest is just implementation details." This typically means they've created an algorithm to do something, and the rest of it is all the supporting activities to build an application or production system around this algorithm. I hear variations on this also from software engineers who dismiss some web apps as "just CRUD²" and thus trivial.

These statements don't usually come from malice³, but they do still diminish the work of many software engineers. There is so much complexity, difficulty, and beauty in the art of "just getting it to production" or "just CRUD" apps. If these parts were trivial, we wouldn't need highly skilled software engineers to lead execution of precisely these areas at startups⁴.

So, what is that complexity that underlies moving things toward production? What's hard about something that's "just CRUD"? And why do people not notice this?

The hard things about going to production

When people say the hard part is shown and done, they're often referring to the part that's interesting to them, academically, and where we're not necessarily sure if it's even possible. Beyond the fact that it's not necessarily harder to show something's possible than to do it⁵, there's still quite a bit that's hard and necessary remaining. These parts are unlikely to succeed if given to an inexperienced engineer. Some of these things are deeply interesting and sometimes we're not even sure if they're possible, either, in the real world.

Here is a quick survey of some of the hard-and-maybe-impossible parts of getting things into production that I've run into in my own work.

Getting started

The first hard thing you run into is just getting started. It seems almost trivial, but it takes way more time than people expect, even with past experience doing it. This time spent is very important. You could move quickly and cut corners, but the way you set things up at the beginning form the foundation of the project and have a ripple effect on everything you do afterwards.

Getting started well requires that you can make some good predictions about what your software will need. Which foundational technologies should we use? How should we structure the project? What tooling will work well for us? Answering these questions takes a lot of experience and a little magic. You can kick the can down the road on some decisions, but that will cost you because a deferred decision often slows down development. It's helpful to predict well as early as you can.

Creating a maintainable design

Writing software itself is also a hard problem. In a research context, the maintainability of code is less critical: the code isn't being used long-term (usually), it's more self-contained, and it's worked on by a narrower set of maintainers. For a production system, you want to make sure that it's designed soundly in a way that you can evolve and maintain for the life of the product. And this code is long-term: it will be around for many years longer than you expect.

That's a big challenge, not least because we usually don't know what the future holds. While we can try to predict it (as we do when getting started), some things are out of our control. We have to make our code flexible enough to be able to add new features, but not so flexible that it starts to impede our ability to work on the product itself.

This is a huge topic, and it's one that is a really big part of getting things into production.

Making it robust (and observable)

We also have to make the system robust. In a research context, things can fail or they can be unpredictable, and it's easier to deal with. In a production setting, that results in bug reports and getting woken up at 2am each night for two weeks. If it's not robust, things will go wrong. I mean, they will anyway—but more often.

So when things do go wrong, you have to have observability in place to be able to figure out what went wrong and why. This is something people can dedicate whole careers to. Figuring out what information is going to be helpful, how to record it, and then later how to use that information is a big field.

User experience and user interface design

Of course, there's also the whole question of how are people even going to use this? A proof-of-concept or an algorithm can show you that something is possible if people do the right things. And how are we going to make it so that that's a reasonable experience for them? If the proof-of-concept requires a lot of data entry, maybe people won't do that! Or maybe there are clever ways to approach it where it is a better experience, and more appetizing.

I've tried my hand at frontend and user interface design enough times to really deeply respect that this is a very wide and deep field. It's certainly far from trivial, and things that seem like they're sure to work will run into the pesky problem of "people." Until the prototype exists in a real-world thing that people can touch and use, including a UI, it's probably not really a sealed deal as working.

This particular area is incredibly interesting to me⁶, because any issues that are discovered require collaboration between researchers, designers, product managers, and software engineers. It's a multi-disciplinary festival!

Acceptable performance

We don't even have to aim for good performance to hit a snag. Even getting to something acceptable is often pretty hard. Prototypes are often slow or assume conditions that don't exist in the real world.

Maybe your prototype finishes its computations in a minute, but users will bounce off the page in a few seconds if they don't see something. (We come back to UI/UX concerns!) Or maybe it works if you have really powerful hardware, but it doesn't work on the devices your users will have. Or it just falls down on production data sizes.

Whatever the case may be, this is a project in itself. You have to understand what performance is required for production use, and how the prototype performs, and then do a lot of work to bridge that gap. If you can.

The hard parts of CRUD

In addition to all the normal concerns of going to production, "just CRUD" apps have some particular concerns that are sometimes missed. An app that's really just CRUD is also extremely rare today, because they're typically dealing with complex associations of data or they need some trickier user interactions.

Designing the database

CRUD apps are heralded as being simple because they expose the database design, so you have fairly standard patterns for the views in your app. That assumes, though, that you have a database schema that's going to work to show users. If the DB design and views are 1:1, then the DB design is user interface design, and how you design it has big UX implications⁷. If they're not 1:1, then your "just CRUD" app now requires creating views that wrap around the database schema with a lot of business logic, and you bring in a lot of the non-CRUD difficulties again. Oh, and when you change your DB design? Bye bye CRUD benefits!

Production support and observability

As mentioned above, making things robust enough to withstand a production workload is hard. You have the entire fields of SRE and DevOps because there is so much to consider here. Reliability, observability, logging, alerting, deployment, change management, security. Not to mention supporting users!

You don't escape performance here

Being CRUD doesn't make it so performance is trivial. Your data might grow to be large, or your schema might be hard to scale up. Who knows! From all the other hidden complexity, it's easy to run into performance problems.

Background jobs

Many CRUD apps require background work to be done. This might be pre-computing things, sending reminder emails, or processing asynchronous tasks. There are some standard ways to do these jobs.

And when you set them up, you now get to manage extra servers, a message broker, and a distributed system. Throw in observability and monitoring for the lot, and you've really piled on quite a bit.

User login and permissions

When people want to use the system, you have to check permissions. You also have to validate their credentials at the door to make sure they can actually log in. Both of these are very nuanced, even if you're using a service provider, and have a lot of depth that you have to grok.

There are standard patterns for user logins. User permissions are more commonly bespoke per application, with some shared patterns but a lot is highly domain specific. Even so, there is a lot of complexity to wind up mired in here, especially once you start getting into SAML and SSO or other more intricate login mechanisms.

Even communicating about these is hard, because there are lots of different words for the concepts. And the standard choices are bad!

It all adds up

The thing about putting something into production is that each individual piece looks pretty easy when you talk about it in isolation. We know how to make user logins. We know how to design schemas. We know how to profile for performance. We know how to do background jobs.

But the pile of all of these together? Each one of these can interact with the other pieces of the system. They impact the design and implementation of other pieces. And we expect all of them in the application!

It's a lot of things to know, and each of them is a field in itself. A lot of the complexity is the breadth, and knowing what you need to know. You can't solve it by hiring an expert in each individual thing, either. You have to have people who can bridge the domains, or you'll end up with a mishmash of pieces from completely different jigsaw puzzles, none of which are the one you were trying to put together.

Why do we do this?

I don't think people set out to miss the complexity in other fields. We don't wake up in the morning and say "today I'm going to call someone's work trivial!" (If you do wake up in the morning and say that, please stop.)

It's more that a lot of complexity is hidden, especially when people do their jobs well. You get to see all the complexity in your own job, because it's what you wade through every day. It's harder to see it in work you're less familiar with, because you don't have that same closeness. Instead you just see people breeze through it, and you don't see the rough edges.

We do this a lot though. Backend engineers have a history of belittling frontend engineers (I personally find frontend much harder, and also very different). Systems programmers have a history of belittling web developers. And we as a field tend to label other fields as lesser for being "non-technical."

It's unfortunate, because there's such beauty out there! Almost every job has complexity, and all the different roles I've seen in tech are interesting and challenging. In each of those jobs, there is the beauty of wrangling complexity into something useful, and making it look easy.

Instead, we should approach things we don't know about with curiosity. Each time you think "huh, that doesn't seem like it should take so long" is an opportunity to figure out what complexity you're not seeing and gain a deeper appreciation. Or, maybe you'll find out you can build Twitter in a weekend. Who knows?

Thank you to Adam Anthony and Dan Reich for providing feedback to me on a draft of this post.