Concepts

The handful of ideas KPD is built on.

Project

A project is a folder: one design for one client or initiative. It holds the project file (project.kpdproj) with everything shared by its pipelines (clusters, service accounts, compute pools, tags and business metadata, data contract rules, naming rules, shared assets), plus folders for pipelines, schemas and exports. See Project files and folders.

Pipeline

A pipeline is one diagram (a .kpd file): the assets, the connections between them, and their layout, groups and notes. Each pipeline runs on one Kafka cluster. Large designs are usually split into several pipelines, for example one per source system or data product.

Storyboard

A storyboard (a .kpdstory file) is the plain-words version of a pipeline: systems, data streams and steps, numbered from the links and read back as a story, with no technical settings. It is made in Storyboard mode and kept apart from pipelines; a pipeline can point at the storyboard it implements. See Storyboard mode.

Assets and palettes

An asset is one thing on the canvas: a topic, a connector, a Flink statement, a consumer group, an external system. Its palette is its type: it defines the asset's properties (the tabs in the properties panel), ports, validation rules, icon and colour. KPD ships 121 palettes. Colour tells you an asset's role at a glance:

Sources   Topics   Processing   Sinks   Producers and consumers   Dead letter topics   Governance and external systems

Ports and connections

Assets connect through ports: outputs on the right or bottom, inputs on the left or top. A connection has a type, such as data flow, join input, error route, sharing or external link, and a port only accepts the types that make sense. A Flink statement's join input accepts a topic, not a connector, and an error output goes to a topic. Invalid connections are refused with a reason.

The connection is the configuration: wiring a sink's error output to a dead letter topic is what sets its dead letter queue, so the picture and the settings can't disagree.

Shared assets

An asset used in more than one pipeline, typically a topic, can be made shared (right-click → Make shared). Its definition then lives in the project file, every pipeline that places it shows the same name, schema, tags and rules, and an edit in one place applies everywhere. Shared assets are listed in the explorer; drag one onto another pipeline to use it there.

Validation

Validation runs continuously against the open pipeline: palette rules (required fields, value ranges), ports and connections, naming, schemas, security, governance and Terraform readiness. Errors never stop you saving: a design is often incomplete on purpose. See Properties and validation.

What KPD does not do

• It never connects to Kafka or Confluent Cloud, and never stores credentials.
• It does not run or test Flink SQL; SQL in a design is a draft until your team runs it.
• It does not deploy. The Terraform it writes is a starting point for your pipeline to apply.