Distributing monorepo tasks across CI agents

The problem

You have a monorepo and a CI pipeline that fans out across several machines. Something has to decide which packages run on which agent.

Nx's comparison of the two tools says Turborepo has no built-in task distribution, and that scaling across machines therefore means static binning: hand-assigning tasks to runners in your workflow file and rebalancing as the repo grows. It describes the failure mode as one agent finishing early and idling while another carries the long tail, getting worse with every machine you add.

That is a competitor's framing, so weigh it as one. Turborepo's own CI documentation is consistent with it, though, if indirectly: what it offers for multiple machines is a remote cache, letting agents reuse each other's results. A shared cache is not a scheduler. It changes whether a task runs, not which machine runs it, and deciding that is still yours.

Nx sells the other half as a cloud service that distributes tasks using historical timing data. If you are on Turborepo, or on plain npm workspaces and a GitHub Actions matrix, you are binning by hand. That is the gap this fills: you already have the timing data in your CI logs, and the assignment is 20 lines of Node.

The mapping

aequitas	Here
Bin	A CI agent or matrix job
Item	A package's task (build, test, lint)
Item's weight	Its recorded duration in seconds
Bin's max	The per-job timeout, if you have one
violations	The timeout cannot be met with this many agents

The code

Eight tasks, weighted by the durations your last green run recorded, across four agents:

import { suggest } from "aequitas";

// Durations in seconds, pulled from CI timing data.
const tasks = [
  { id: "@acme/e2e", weight: 420 },
  { id: "@acme/web", weight: 300 },
  { id: "@acme/api", weight: 240 },
  { id: "@acme/ui", weight: 180 },
  { id: "@acme/docs", weight: 120 },
  { id: "@acme/cli", weight: 90 },
  { id: "@acme/utils", weight: 60 },
  { id: "@acme/config", weight: 30 },
];

const agents = [
  { id: "agent-1" }, { id: "agent-2" }, { id: "agent-3" }, { id: "agent-4" },
];

const plan = suggest(tasks, agents);
plan.loads;
// → { "agent-1": 420, "agent-2": 360, "agent-3": 330, "agent-4": 330 }

That is 24 minutes of work across four machines. Against the two things people actually do by hand:

Strategy	Per-agent seconds	Slowest agent	Pipeline takes
Static bins, alphabetical, two each	330 / 150 / 600 / 360	600s	10m00s
Round-robin by declaration order	540 / 390 / 300 / 210	540s	9m00s
suggest(tasks, agents)	420 / 360 / 330 / 330	420s	7m00s

Three minutes of wall clock, on every run, for a build-time script. Alphabetical binning put @acme/e2e and @acme/ui on the same agent, which is exactly the long tail the Nx comparison describes.

Why 7 minutes and not 6

The total is 1440 seconds over four agents, so a perfect split would be 360 seconds each, 6 minutes. aequitas returned 420. That is not the solver falling short.

@acme/e2e weighs 420 seconds on its own. It has to run somewhere, and whichever agent takes it cannot finish sooner than 420 seconds. So the floor for any possible assignment is:

max(total / agents, heaviest single task)  =  max(360, 420)  =  420

aequitas hit that floor exactly. The remaining gap is not a scheduling problem, it is a 2-minute-longer end-to-end suite. Splitting that file is the next optimisation, and no bin packer can do it for you. Distributing the suite once it is split is a separate case, worked through in test sharding.

This is the useful habit when reading loads: compare against max(total / bins, heaviest item), not against the average. The average is often unreachable.

Feeding it to a CI matrix

assignments maps onto a GitHub Actions matrix. Each entry carries an itemId and a binIds array, which holds one agent here because there is no split in play.

It can also be empty. An item that landed nowhere still gets an entry, with binIds: [], and is additionally listed in unassigned. That cannot happen above, since these agents have no max, but it can as soon as you add one, so skip the empty entries rather than indexing straight into [0]:

const byAgent = {};
for (const a of plan.assignments) {
  if (a.binIds.length === 0) continue;  // unplaced; see plan.unassigned
  (byAgent[a.binIds[0]] ??= []).push(a.itemId);
}

console.log(JSON.stringify({
  include: Object.entries(byAgent).map(([agent, projects]) => ({
    agent,
    projects: projects.join(","),
  })),
}));

{
  "include": [
    { "agent": "agent-1", "projects": "@acme/e2e" },
    { "agent": "agent-2", "projects": "@acme/web,@acme/utils" },
    { "agent": "agent-3", "projects": "@acme/api,@acme/cli" },
    { "agent": "agent-4", "projects": "@acme/ui,@acme/docs,@acme/config" }
  ]
}

Emit that from a setup job, read it with fromJSON in the matrix, and each job runs turbo run test --filter=... over the packages it was given.

Using the timeout as a capacity band

If your jobs have a timeout, put it in max and let violations tell you whether the plan fits. With a 6-minute timeout:

const capped = agents.map((a) => ({ ...a, max: 360 }));

suggest(tasks, capped).violations;  // → 60

Non-zero, because @acme/e2e is 420 seconds and no arrangement fits it into 360. The 60 is exactly the overshoot. Cut the agents to three and it gets much worse:

const three = [
  { id: "agent-1", max: 360 },
  { id: "agent-2", max: 360 },
  { id: "agent-3", max: 360 },
];

const tight = suggest(tasks, three);
tight.loads;       // → { "agent-1": 510, "agent-2": 480, "agent-3": 450 }
tight.violations;  // → 360

1440 seconds of work cannot fit three 360-second slots however you arrange it. This is a usable CI check: if violations > 0, you need another agent or a faster suite, and you know that before burning the minutes.

Adding a package

A new package appears. You can re-solve from scratch, or start from the plan you are already running:

const withMobile = [...tasks, { id: "@acme/mobile", weight: 210 }];

rebalance(withMobile, agents, plan.assignments);
// slowest agent 450s, and 2 existing tasks moved

suggest(withMobile, agents);
// slowest agent 420s, and 4 existing tasks moved

For CI, prefer suggest

rebalance exists to keep churn down, which matters when moving an item costs something real, like migrating a tenant's data. Moving a package between CI agents costs nothing, so take the faster pipeline. Here that means suggest: 420 seconds against 450.

Reach for rebalance in this use case only if you cache per-agent and want assignments to stay stable across runs for cache-hit reasons.

Where this stops

Three honest limits before you wire this in.

Task dependencies are not modelled. This is the big one. If @acme/web cannot build until @acme/ui has, you cannot freely bin them, and aequitas knows nothing about a task graph. It fits when the tasks are genuinely independent: per-package test, lint or typecheck runs. For a dependency-ordered build, the ordering is Nx's or Turborepo's job and aequitas can only distribute within a layer that has no internal edges.

The objective is the range, not the makespan. aequitas minimizes maxLoad - minLoad, which is the spread term of the cost function. What you actually care about in CI is the slowest agent. With a fixed total those pull the same direction, and here the result was provably optimal, but they are not the same objective and a crafted input can separate them.

Durations drift. A weight is a snapshot. Tests get slower, packages get added, and a plan computed from last month's timings decays. Recompute from recent data on each run rather than committing the assignment, which is the same reason CircleCI and Nx keep re-reading their timing stores.

Caching changes what you should feed in. Turborepo's actual answer for CI is a remote cache: with a warm cache, only the tasks whose inputs changed execute, and the rest restore more or less instantly. So the items worth distributing are the cache misses, not every package in the repo.

Weighting all eight tasks when six of them will be cache hits produces a beautifully balanced plan for work that never runs, and the real pipeline is then decided by the two that do. Compute the affected set first, with whatever your tool offers for "changed since this ref", and weight only those. The set changes on every pull request, which is another reason to compute the assignment per run instead of committing it.