Making CI 61% faster while test suite grew 5x

Conduit is a unified inbox for hospitality teams. Our app lives in one TypeScript monorepo: a Next.js frontend, a Convex backend and a few smaller packages, all tested with Vitest on GitHub Actions.

In May, a full CI run on our main branch took 13.6 minutes. We write all our code with coding agents, so changes were landing faster than CI could check them. Today the same run takes 5.3 minutes, and the backend test suite is almost five times bigger: 2,047 test files, about 30,000 tests. A lot of that came from asking why each step in our build and deploy existed, instead of keeping the setup everyone starts with.

The numbers

Most of the drop came from the first change in May. From June to early October, CI crept back up from 3.1 to 7.5 minutes while the suite nearly tripled. The October changes brought it down to 5.3.

Run every check before merge

This change came from questioning a default. In the standard Next.js on Vercel setup, CI checks a pull request, and then Vercel builds the whole thing again when it deploys. Every branch also gets its own preview build. You pay for the same work two or three times, and the copy that matters most runs after the code has already merged. If that build fails, main is broken and someone has to revert or patch it while everyone else waits.

That's how our deploys worked until August. A production build on Vercel took a median of 11.3 minutes, and about one in seven failed, half of them on type errors CI had never been required to catch.

So we asked what Vercel needed to do. Its job is to ship, and checking the code can happen once, in CI, before anything merges. We moved every check there:

• The web build. CI runs the same next build that Vercel runs, so a green pull request means the app compiles.
• Type checks. The web app and the backend are type-checked in CI with tsgo, and nowhere else.
• The backend functions. Convex can start a throwaway local backend with no account or credentials. CI generates the backend's code against it, pushes every function to it, and fails if the generated code committed in the pull request is out of date.

A GitHub ruleset on main requires all of these, plus every test suite and a review, before anything can merge. Nobody has to remember to wait for a green run, and the checks can't drift apart between people or teams.

Vercel no longer type-checks or regenerates code. It builds the frontend and pushes the backend in parallel, and pull request branches no longer get preview builds.

Deploys still fail about 5% of the time, almost always on the backend push itself, such as an index change that needs confirming on real data. CI runs against an empty backend, so it can't catch those.

Faster runners and a faster type checker

In May we moved our heaviest jobs, the backend tests and the type check, off GitHub's standard runners and onto Blacksmith, which runs the same workflows on faster machines. Its 4-vCPU runners were roughly twice as fast for our jobs, at about half the price per minute. At the same time we split the backend suite into four parallel shards. A full run went from 13.6 minutes to 4.2.

The type checker was the next slow step. We switched to tsgo, the Go rewrite of the TypeScript compiler, and our backend type check went from 205 seconds to 38. A month later we moved every other tsc call over.

Only run what changed

Not every change needs every check. In August we combined our seven workflow files into one and put a path filter in front of each job, so a job runs only when files it cares about change. A change that only touches docs now skips almost everything. Afterwards, a quarter of pull request runs finished in under a minute.

Having all of CI in one file also made duplicate work easy to see. next build runs its own type check by default, on top of the type check job we already had. Once the type check job had to pass before anything could merge, we turned off the copy inside the build.

Path filters can leave a pull request stuck. GitHub waits for every required check, and a check that a filter skips never reports. We require one small summary job per test suite instead, which always runs.

Less setup per job

Every job checks out the code and installs dependencies before it does any real work. For small checks, that setup was most of the job. Six of them spent about 395 seconds of runner time on 24 seconds of checking, so we merged them into one job. We also made each job install only the packages it needs.

The biggest setup cost was in the backend test job. It set 74 environment variables one CLI call at a time, which took 44 seconds. One bulk call takes 1.5.

More shards on bigger runners

Splitting tests into parallel shards kept CI fast while the suite grew, until early October, when a full run was back to 7.5 minutes.

Adding shards isn't free, because every shard repeats the setup. We ran the same commits through five combinations of shard count and machine size:

Six shards on 8-vCPU machines was the balance. Bigger machines saved under a minute more for about 45% more compute.

Defaults for agentic coding

None of these changes needed new technology. Each one came from looking at a step everyone runs by default and asking whether it still fit how we work. Most CI and deploy setups assume people write every change by hand and merge a few a day. We write our code with agents and merge far more often, so a slow or repeated check costs us many times a day, and a broken main blocks everyone at once.

A lot of the setup we all start with was built for a slower way of working. If your team is moving to agentic coding, go through every step between a pull request and production. Ask what each one proves, and whether something earlier already proved it.