How big can a design get?

What we measured when we pushed KPD with oversized designs, and how big a pipeline can get before it slows down.

The short version

Keep each pipeline under about 300 assets. Past that, split the design into several pipelines joined by pipeline boxes. A project can hold 100 pipelines or more without trouble. It is the size of a single pipeline that slows KPD down, not the size of the project.

At a glance

How long an edit takes (adding a note, moving an asset, typing in a property) as one pipeline grows, on a recent Mac. Beyond the times, the rows also say what else changes at each size.

up to 150 assets ~0.1 s Instant. Panning and dragging are smooth.

300 assets ~0.2 s Fine. A noticeable moment per edit. The recommended maximum.

500 assets ~0.6 s Sluggish. Typing lags, dragging judders, a PNG export takes about half a minute.

1,000 assets ~1.5 s Falls apart. Dragging is a slideshow, and the pipeline is too big to export as one picture (KPD says so; before v0.64.0 the tab crashed).

2,000 assets ~5 s Unusable: about 10 seconds to save.

4,000 assets stuck Opens, then every pan freezes the page for ten seconds or more.

Bars use a stretched scale so the small times stay visible. Mac times are estimates from the measurements below (see How we measured).

What counts as an asset

An asset is anything from the palette on the canvas: a topic, connector, Flink statement, consumer group, external system and so on. Notes and group boxes are drawn too, so they add to the cost. The test designs carried about seven notes and one group box for every ten assets, which is typical of a well-documented client design. A pipeline of 300 assets with no notes is about as fast as one of 200 assets with them.

The Contents line on a pipeline's General tab and the status bar both show how many assets a pipeline has.

Keeping a big design fast

Split it. Break a large design into pipelines by stage or by domain, and connect them with pipeline boxes: drag one pipeline from the explorer onto another. Each pipeline stays quick to edit, and the design document reads better with one diagram per stage.
Turn off the moving dots. On a big pipeline, View → Animate data flow costs a lot. Turning it off roughly doubles how smoothly you can pan and drag at 500–1,000 assets, and makes edits up to 40% quicker.
Big pictures have a limit. Export PNG draws the whole pipeline as one picture. Large pipelines are drawn a little less sharply, and above about 1,200 assets, notes and group boxes KPD refuses with a message rather than let the browser run out of memory (before v0.64.0 the tab crashed, losing anything unsaved). Export to Detailed Design Template draws the open pipeline the same way.
Many pipelines are fine. A project of 200 pipelines (6,000 assets between them) opened in under two seconds, and Validate project, Find in project, Terraform and the design document all took a few seconds at most. Such a project is somewhat slower overall, because the explorer lists every schema and shared asset.

The results

These are the measurements behind the summary. They were taken in a browser on a modest test machine (2 processor cores, no graphics card), which is slower than a typical laptop: on a recent Mac expect about half these times, and smoother panning.

One pipeline of growing size

Times in milliseconds (1,000 ms is a second). Frames per second (fps): 60 is smooth, under 20 judders, under 5 is a slideshow. Animate data flow was on, which is the default.

Assets	Open	Pan (whole design)	Pan (100%)	Drag an asset	Edit	Undo	Save	Export PNG	Memory
30	616	53 fps	55 fps	—	95	96	112	3.8 s	15 MB
120	905	28 fps	39 fps	52 fps	252	242	253	14 s	18 MB
300	1,125	15 fps	20 fps	21 fps	402	503	784	36 s	42 MB
510	2,020	8 fps	13 fps	9 fps	1,194	1,252	1,203	61 s	60 MB
1,020	3,449	4 fps	4 fps	1 fps	3,098	4,713	5,077	tab crashed*	111 MB
2,010	6,512	2 fps	2 fps	0 fps	10,755	18,170	20,400	tab crashed	190 MB
4,020	17,041	1 fps	Each pan froze the page for about 13 seconds; the test was stopped after 20 minutes.

* Measured with 0.62.2. Since 0.64.0 KPD refuses to export pictures this big, with a message, instead of the tab crashing.

With Animate data flow off

The same pipelines with the moving dots off; the earlier figure (dots on) in brackets.

Assets	Pan (whole design)	Pan (100%)	Drag an asset	Edit	Save
300	21 fps (15)	31 fps (20)	37 fps (21)	472 (402)	556 (784)
510	14 fps (8)	20 fps (13)	22 fps (9)	904 (1,194)	882 (1,203)
1,020	8 fps (4)	10 fps (4)	3 fps (1)	1,820 (3,098)	2,229 (5,077)

Many pipelines in one project

Each pipeline had 30 assets; the canvas showed one of them.

Project	Open	Pan	Edit	Save
1 pipeline, 30 assets	616	53 fps	95	112
50 pipelines, 1,500 assets	1,006	42 fps	114	162
200 pipelines, 6,000 assets	1,839	23 fps	241	482

Whole-project work

The work KPD does without drawing anything, timed on a Mac. Checking the open pipeline happens on every edit; the others run when you ask for them.

Project	Check the pipeline (each edit)	Validate project	Find in project	Design document	Terraform
300 assets	16	70	8	72	16
1,020 assets	67	275	19	293	50
4,020 assets	498	1,722	66	2,313	202
200 pipelines, 6,000 assets	3	840	96	1,453	246

The checking and exporting stay quick even on huge designs. The time goes on drawing: after each edit the canvas redraws every asset, connection and note.

How we measured

The test designs were made from copies of the Inclined Labs sample, so every asset was fully set up as in a real client design: Flink SQL, schemas, data contracts, governance tags, dead letter topics, shared data products, notes and decisions. Each copy had its own topic names, schemas and compute pools, and every design passed Validate project with no errors or warnings. Sizes ran from 30 to 4,020 assets in one pipeline, and up to 200 pipelines in one project.

For each size, a script drove KPD in a real browser. It opened the project, panned, dragged an asset, added a note and undid it, saved, and exported a PNG, timing each step and recording the longest freeze and the memory used. The whole-project work was also timed on its own, on a Mac, where it ran about twice as fast as on the test machine. Measured with KPD 0.62.2 in October 2026; the developer write-up is docs/SCALE.md in the KPD folder.

Running KPD on your own computer?

You can build the same test designs and time them yourself. The scripts are in scripts/stress/ in the KPD folder, and docs/SCALE.md explains how to run them. Use a separate projects folder for them, not your client work.