Nobody budgets for the deploy pipeline

Nobody budgets for the deploy pipeline

DigitalOcean's cheapest droplet is four dollars a month, and gets you 512MB of memory, one vCPU and 10GB of disk. Six dollars gets you a gigabyte and 25GB of disk, which is what most people actually pick, because 512MB runs out faster than you would think once Postgres is sitting on the same box.

It may not seem like a big deal to most engineers, but to someone shipping their first product it's the entire company. That single box runs the API, the database and the frontend, and you can SSH into all of it.

And it works, which is the part worth sitting with before anything else. A single VPS will happily run a real product with real users for a short time - as long as your user base is tiny, and you can ship a change to it in about ninety seconds.

Four years later the same product is running across twenty services and three environments, with Terraform, a secrets manager, a Kubernetes cluster and a deploy process that maybe one person fully understands and everyone depends on.

Nobody made a bad decision in between, which is the part I find interesting. So I want to walk through how it actually happens, stage by stage, because the shape of it turns out to be more predictable than most people expect.

The four jobs

Let's start with what stays the same, since it's the only thing that does.

Whatever you are building, an application has to do four things: get code from a laptop onto a machine that runs it, keep data somewhere it won't vanish, store passwords and API keys somewhere that possibly isn't your repo, and answer requests on a domain over HTTPS.

That list doesn't get any longer at three people, or thirty, or three hundred. What changes is how many copies of it you are running.

Customers are what drives that. More of them means splitting responsibilities out of the single service that used to do everything. One service becomes four, each with its own datastore, its own secrets and its own place in staging as well as production. Nobody decides to run twenty of anything. It arrives one sensible decision at a time.

One person, one box

The solo stack is boring and cheap, and that's rather the point of it.

You might have one or two VPSes from DigitalOcean or Linode. You might have nothing at all, if you go serverless, because AWS gives you a million Lambda requests and 400,000 GB-seconds a month, free, permanently. The database is Postgres on Supabase's free tier, or Firebase, or just Postgres installed on the same droplet as the app. GitHub or GitLab handles the deploy, and Vercel or Cloudflare Pages puts a frontend on a domain. All four jobs are covered, mostly for free, and mostly configured by somebody who isn't you.

If you are on the VPS path you SSH in, install the runtime, install Postgres beside it, write an nginx config, run certbot, and add a systemd command so the server and database comes back when it falls over. Deploys are git pull && systemctl restart, or a GitHub Action doing the same thing over SSH.

The cost of infrastructure here is small, but there are other invisible costs which show up late.

You built that box by hand over a few evenings, and somewhere in the nginx config there's a line you added at 1am to fix something, which you would have to go and look at today before you could tell me what it does, and then you'd have to think about it for a minute. Nothing anywhere describes how to rebuild that machine. If the droplet died tonight you'd rebuild it over a weekend, probably, and the new one would end up subtly different from the old one in ways you wouldn't discover for a month.

So here's the thing I actually want to name, because the rest of this article runs on it. Something is already deciding what runs, at what version, with which secrets, and what happens when any of that changes. In networking that layer has a name: it's called a control plane, and it's the part that decides rather than the part that does the work. At one person, your control plane is your memory. It's undocumented, it can't be inspected, it can't be handed to anyone, and it's still the fastest one you will ever have.

That isn't a problem yet. It's just worth knowing the layer exists, because from here it only gets more crowded.

Three to ten people, and the week staging appears

The team grows, you hire two engineers, and for a while nothing dramatic happens. Then somebody runs a test against production, a customer sees something they shouldn't have, and by the end of that week you have a staging environment.

That's the week the shape of the problem changes, and almost nobody notices it at the time.

Because now there's two of everything. There are two databases, and the staging one holds a copy of production from some Tuesday in March that nobody has refreshed since. Four environment variables have become twenty, in two sets, with different values, and one of them exists only in production because somebody added it there one evening and never wrote it down. There are two domains and two certificates, and the frontend has to know which API to talk to, which means the API URL is now itself a piece of config that differs per environment.

Three people pushing to main also breaks "push to main and it deploys," so there's a workflow file now, and branch protection, and a staging deploy that does slightly different things from the production deploy, and nobody wrote down what's different or why, because at the time it was completely obvious.

Secrets move into Doppler or SSM around here, not because anything broke, but because three people editing production environment variables by hand is a reliable way to lose an afternoon. (The twelve-factor rule has been saying keep config out of code since 2011, and plenty of teams still end up with it in three places that quietly disagree.)

And that first VPS is still running, and now exactly one person on the team is willing to touch it.

The four jobs haven't changed at all. There's simply two of everything now, and one person holding the map.

It's worth trying to put a figure on that. There's the effort of doing the same setup by hand every time something new gets added. There's the repetition of wiring staging up to match production, then doing it again for the next service. And there's the forgotten piece, a variable or a renewal date, that costs somebody an evening and leaves everyone slightly less sure of the setup afterwards. Those three together come to more than you saved by picking the cheaper server, which is the awkward bit. The cost you optimised was the small one.

You are probably leaking a bit of real money as well, since it's all tracked by hand. A staging database nobody switched off. An orphaned volume from a service that got deleted in March. Never quite big enough to stand out on a monthly bill.

This is the first stage where having the deciding layer live somewhere other than a head genuinely pays for itself

Ten to twenty-five people, and twenty services

Now the stack looks like the thing people give conference talks about.

You've got GitHub Actions doing CI, Terraform for the infrastructure that fits in Terraform, and the AWS console for the bits that didn't. There's a secrets manager, RDS, Datadog or Grafana so somebody knows when things break, and a Slack webhook so that somebody is actually told. Probably Kubernetes by now, possibly with ArgoCD sitting on top of it.

Infrastructure as code arrives properly at this stage, and for a good reason, which is that creating things by clicking around in a web console stops being survivable once more than two people are doing it.

Every one of those tools was right when it was decided. Terraform went in because of a resource nobody could account for. The secrets manager went in because of the incident. Datadog went in because something broke on a Saturday and nobody found out until Monday morning.

And it's still the same four jobs.

Here's the arithmetic that I think is the real point of all this. One service in one environment gives you four things you need to get right. Twenty services across three environments gives you 240 things you need to get right at the same time, each of which was configured in a different tool, by a different person, on a different day.

That's twenty sets of secrets, twenty health check definitions that were mostly copy-pasted from each other originally and are quietly different now, and twenty deploy pipelines that have drifted apart over two years in ways nobody has ever audited.

So you end up with seven tools, each holding one piece of the truth about your system, and nothing holding all of it. The failures mostly don't happen inside the tools. They happen between them, because the gap between two tools doesn't belong to anybody.

What a control plane is worth at this size is different from the last stage, and it isn't about unblocking a colleague any more. It's that there's one model of the system rather than seven fragments of one, so adding a twenty-first service means following a path that already exists instead of assembling one from scratch and hoping it matches the other twenty. The drift stops being inevitable, because there's only one place for the answer to live.

Twenty-five and up, where somebody finally builds one on purpose

Past this point the problem stops being technical and turns into governance.

Who's allowed to deploy to production, and can you prove it to an auditor? What's running in the EU versus the US, and can you show that to a regulator? When a region goes down, what fails over and how quickly? Which team's budget does that cluster actually belong to?

None of that is a new job. It's the same four, wrapped in proof.

And here's what companies at this size actually do about it: they build a control plane on purpose. They give it a team, a budget and a name, usually "internal developer platform" or "paved road," and they staff it with a handful of engineers whose entire job is making sure everyone else can ship without thinking about any of this. That work is real and those teams earn their keep.

Which is the point of the whole article, so I'll just say it. Everybody has a control plane at every stage. Solo, you have one and it's your memory. At three people it's one engineer plus a Notion page that's already out of date. At twenty it's seven tools and nobody in particular. The large company didn't invent anything here. It's just the only size that notices it already has one and decides to staff it properly.

The cost that never appears on an invoice

I want to talk about what building this yourself actually costs, because I think it's the most misunderstood thing in this whole area.

Week one is the part people budget for. You set it up, it works, everyone's pleased with themselves.

It's every week after that. Certificates expire, base images need patching, a Terraform provider upgrade renames a resource and breaks a plan, and Kubernetes supports each minor release for about fourteen months while shipping patches monthly, so there's always a clock running somewhere and always something worth looking at. Then a dependency bump breaks the build for a reason that takes three hours to find and thirty seconds to fix. None of it ships anything, and all of it has to happen.

And it lands on one person. Whoever wrote the pipeline ends up owning the pipeline, not because anyone decided that in a meeting, but because they're the only one who can debug it fast enough to be worth asking. That person knows they're the bus factor, and it isn't a nice thing to be.

The reason this is the most expensive option is the same reason it's the most popular one: it never appears on a bill. There's no invoice for it, and no budget line called "keeping the deploy process alive," so it comes quietly out of salaries you were paying for product work, and because nobody ever sees a number, nobody ever sits down and decides to stop paying it.

If you want the honest version for your own team, go and count. Work out how many hours went into infrastructure last quarter that wasn't an interesting architecture decision, just maintenance. Then ask whether anybody actually agreed to spend that.

Three options

You can keep building it yourself, which is a real answer and the right one if infrastructure is genuinely part of what makes your product good. Just make the cost visible: give it an owner and put it on a roadmap instead of pretending it's free.

You can use a managed platform. Railway, Render, Fly and Heroku are all good at this, and most of them will run your database too. The thing worth checking before you commit is the shape of the bill as you grow, because most of them meter the pieces separately, and separate meters are how a modest app turns into a number you have to explain to somebody.

Or you can stop owning the layer, which is the one we're building.

Choreon takes your source, whether that's a repo, a container image or a folder on your laptop, and runs it. You get community version Postgres or Redis if you need them, plus secrets, domains and basic gateway auth, with every service on one dashboard and git push to deploy. Same four jobs, we just streamline them.

The part we care most about is the bill. Database, bandwidth and auth are in the plan rather than metered separately, so the price on the pricing page is the price you pay, and it starts at $6.99. Isn't that fascinating? We care when you are starting small, and when you grow big, we still got you!

We're not a cloud and we don't own data centres. We run the layer above them, which is the provisioning and the wiring and the security plumbing, so that it stops being your job. Running the same setup tomorrow on your own AWS, GCP, Azure or on-premise is simply wiring them to the same cockpit. So, zero vendor lock-in.