Sharding a Playwright or Jest suite by duration

The problem

Same shape as distributing monorepo tasks, one level down: not packages across agents, but spec files across shards of a single suite.

Playwright and Jest both ship --shard. Playwright's documentation is admirably blunt about what its version does: it shards test files by default, split by count, with no timing input. The docs then name the consequence directly: if your files are not evenly sized, "certain shards may end up running significantly more tests, while others may run fewer or even none."

Jest's --shard is less pinned down. Its CLI reference says the split is delegated to the configured testSequencer, which "has to implement a shard method", and does not state whether the default splits by file or by test, nor whether timing is considered. If you are on Jest, read your sequencer rather than assume it behaves like Playwright's.

Try the built-in fix first

Playwright's own suggestion is fullyParallel: true, which shards at the level of individual tests instead of files and distributes far more evenly. If your suite can run that way, use it and skip this section.

The reason people cannot is usually shared state: tests inside a file that depend on order, or a beforeAll that logs in once and is reused. When the file has to stay the indivisible unit, count-based sharding is what you get, and that is where weighting by duration helps.

The code

Twelve spec files, weighted by the durations your last green run reported:

import { suggest } from "aequitas";

const specs = [
  { id: "checkout.spec.ts", weight: 210 },
  { id: "billing.spec.ts", weight: 190 },
  { id: "admin-users.spec.ts", weight: 175 },
  { id: "onboarding.spec.ts", weight: 155 },
  { id: "search.spec.ts", weight: 140 },
  { id: "cart.spec.ts", weight: 120 },
  { id: "auth.spec.ts", weight: 95 },
  { id: "product-detail.spec.ts", weight: 85 },
  { id: "settings.spec.ts", weight: 70 },
  { id: "profile.spec.ts", weight: 60 },
  { id: "notifications.spec.ts", weight: 45 },
  { id: "smoke.spec.ts", weight: 30 },
];

const shards = [
  { id: "shard-1" }, { id: "shard-2" }, { id: "shard-3" }, { id: "shard-4" },
];

const plan = suggest(specs, shards);
plan.loads;
// → { "shard-1": 355, "shard-2": 355, "shard-3": 340, "shard-4": 325 }

That is 1375 seconds of tests, 22m55s if you ran them serially. Against what --shard does with four shards, splitting twelve files by count in path order:

	shard-1	shard-2	shard-3	shard-4	Suite takes
--shard, by count	460s	375s	300s	240s	7m40s
Weighted by duration	355s	355s	340s	325s	5m55s

1m45s per run. The count split put admin-users, auth and billing together, which is 460 seconds against a shard that finishes in 240 and then waits.

Then invoke each shard with its own file list rather than --shard:

npx playwright test checkout.spec.ts product-detail.spec.ts profile.spec.ts
npx playwright test billing.spec.ts auth.spec.ts settings.spec.ts
npx playwright test admin-users.spec.ts cart.spec.ts notifications.spec.ts
npx playwright test onboarding.spec.ts search.spec.ts smoke.spec.ts

How many shards are worth paying for

CI machines cost money, so this is a real question, and the floor answers it. Running the same suite at every shard count:

Shards	Floor	aequitas	Gain over previous
2	688s	695s	680s
3	458s	460s	235s
4	344s	355s	105s
5	275s	295s	60s
6	229s	240s	55s
7	210s	210s	30s
8	210s	210s	0s
9	210s	210s	0s

The eighth machine buys you nothing, and neither does the ninth. Once total / shards drops below the heaviest single file, checkout.spec.ts at 210 seconds becomes the binding constraint and no amount of parallelism helps. The crossover is total / heaviest, here 1375 / 210 = 6.55, so seven shards is the last one worth buying.

To go faster than 210 seconds you have to split checkout.spec.ts itself. That is a decision aequitas can tell you is necessary but cannot make for you.

aequitas is not exact, and you can see it here

At four shards the floor is 343.75 and aequitas returns 355, about 3% off. In the monorepo case it hit the floor exactly; here it does not. That is the hill-climb finding a strong local optimum rather than a proven global one, which is the documented trade.

3% of a CI run is usually a fine price for a 20-line script. If it is not, you want an exact bin-packing solver.

The new-file problem

This is the part that bites in practice. A spec file added this morning has no timing history, so what weight do you give it?

Measured, with the same twelve files plus one new reports.spec.ts, then checking what happens when the new file turns out to take 200 seconds:

Weight you guess	Planned slowest	Actual slowest at 200s
0, unknown	355s	525s (8m45s)
107.5, the median	380s	450s (7m30s)
210, the heaviest known	405s	405s (6m45s)

Weighting an unknown file 0 is worse than not balancing at all: 525 seconds against the count-based baseline's 460. The solver packs it onto the shard it thinks is emptiest, which is precisely the shard with no headroom to absorb a surprise.

Weight unknown files pessimistically. The heaviest known duration costs you a slightly lopsided plan on paper and buys the best real outcome. The median is a reasonable middle. Zero is a trap.

const known = timings.map((t) => t.duration);
const fallback = Math.max(...known);   // pessimistic, not optimistic

const specs = files.map((f) => ({
  id: f,
  weight: timings.find((t) => t.file === f)?.duration ?? fallback,
}));

Where this stops

Everything from the monorepo case still applies: the objective is the range rather than the makespan, and durations drift.

Sharding is per project, not per test. aequitas moves whole items. If one file dominates, as checkout.spec.ts does above, the answer is to split the file, and no allocator substitutes for that.

Retries and flakes are invisible. A weight is one number. A spec that passes in 40 seconds and fails-then-retries in 130 has no single honest weight, and the plan will be wrong on the runs that matter most. Use a high percentile of recent durations rather than the mean if your suite is flaky.