KPD closed beta Sign inRequest access

Flink SQL and compute pools

Design Confluent Cloud Flink SQL statements, the compute pools they run in, and what each statement reads and writes.

A Flink SQL statement

Drag Flink SQL from the Processing palettes, wire one or more topics into its Input, and wire its Output to the topic it writes. In Confluent Cloud every topic is also a Flink table, so there is no separate table asset: the topic is the table.

A Flink SQL statement selected, with its SQL tab underneath.
A Flink SQL statement selected, with its SQL tab underneath.

The statement's tabs:

Its ports are Main input, Join input (on the bottom edge, so the main flow stays left to right), UDFs / models, Output and Errors.

Errors and dead letter topics

Wire Errors to a Dead Letter topic for the records the statement can’t process. A statement can send errors to more than one Dead Letter topic:

With more than one, say on each Errors wire which records go there: select the wire and fill in Condition, such as orders: deserialisation failures. KPD reminds you until every wire says, the label shows on the canvas, and the design document’s DLQ table lists it as the reason. Streaming Agents, Kafka Streams apps and Stream Apps can send errors to several DLQs too. A connector has one DLQ, and so does a topic’s contract Rejected output, because Confluent gives each of those a single DLQ setting.

Statement kinds

KindUse it forOutputs
INSERT INTOA continuous query writing into an existing table (topic). The default.One
CREATE TABLE ASA query that creates its own table and topic.One
CREATE MATERIALIZED TABLEA table definition and its continuous query in one object, evolved in place.One (its backing topic)
EXECUTE STATEMENT SETSeveral INSERT INTO statements run as one job.Several

Writing to several topics: statement sets

One Flink job can write to more than one table. Set Statement kind to EXECUTE STATEMENT SET, wire the Output to each topic, and write the inserts in one block:

EXECUTE STATEMENT SET
BEGIN
  INSERT INTO `incl.prod.ent.booking.high-value.v1`
    SELECT * FROM bookings WHERE amount_gbp > 5000;
  INSERT INTO `incl.prod.ent.booking.standard.v1`
    SELECT * FROM bookings WHERE amount_gbp <= 5000;
END;

It runs as one statement, and the planner can read a shared source once. KPD warns when any other kind has more than one output, because each of those writes to a single table.

TipThe + beside a Flink statement’s Output stays available after the first connection, so quick add can keep adding output topics.

Materialized tables

With CREATE MATERIALIZED TABLE, the SQL tab offers START_MODE (with a start point for the timestamp and interval modes), the primary key, DISTRIBUTED BY … INTO n BUCKETS, changelog.mode, and whether the column list is declared explicitly (recommended). The topic wired to the Output is the table's backing topic: the materialized table creates and owns it. KPD warns if nothing is wired, or if anything else writes to that topic, and the node's subtitle reads Materialized.

NoteTopics created by Flink itself (by CREATE TABLE AS or a materialized table) are not created again by the Terraform export.

SQL is not checked

Not validatedKPD does not validate SQL. The SQL box is highlighted text, stored and exported exactly as you typed it. Nothing parses it, runs it or compares it with the rest of the design.

That means KPD will not tell you when:

What KPD does check is the design around the SQL: the wiring, the statement kind against the number of outputs, the materialized table options, the compute pool and the service account.

TipTreat SQL in KPD as a draft. Before relying on it, run it in a Confluent Cloud Flink workspace (or with EXPLAIN) against the real topics. The same applies to ksqlDB queries.

SQL in its own files

A statement can keep its SQL in a file of its own, in the project’s sql/ folder, instead of inside the pipeline. In Git a file reads, diffs and reviews as SQL: a one-word change shows as one changed line, not one long changed string in the pipeline file. It also opens in any editor, and SQL tools can run over it.

A Flink SQL statement's SQL tab with a SQL file attached: the file picker, a Shared marker saying two statements use the file, the Read-only tick, and the file's SQL.
A statement using a shared SQL file. Untick Read-only to edit the file here.

Inline SQL still works as before, so existing designs need no change: a statement uses its file when one is attached, and its inline SQL otherwise. As with inline SQL, KPD doesn’t check what’s in the file. The Terraform export deploys the file’s SQL exactly as written, and the design document shows it under the statement, saying which file it came from. A missing file is a warning on the statement, and a TODO in Terraform.

Compute pools

Statements run in a Flink compute pool. Define pools in the project explorer: right-click Flink compute pools → New Flink compute pool… and give it a name, cloud, region and maximum CFUs (5 to 50). The New project dialog can add them at the start, and the Data Product template adds one pool per stage. Then pick the pool on each statement's General tab.

A pool that already exists in Confluent Cloud can be given its Existing pool ID, so Terraform refers to it instead of creating it.

Pool capacity

Every running Flink statement takes at least 1 CFU from its pool, so KPD counts the statements on each pool:

Statements on the poolKPD says
Fewer than its max CFUsFine. The pool’s hint on a statement’s General tab shows how much it already claims, e.g. 4 statements in this pipeline: at least 4 of 10 CFU.
Exactly its max CFUsA warning: the statements run, but none can scale above 1 CFU.
More than its max CFUsAn error on each statement: they can't all run. Raise the pool's max CFUs or move statements to another pool.

A statement set or a materialized table counts as one statement. Badges and the Problems tab count the statements in the open pipeline; Validate project adds up every pipeline that shares the pool and lists each pool's usage.

Other processing assets