KPD closed beta Sign inRequest access

7. Data quality and dead letter topics

Two contract rules on the canonical topic, three dead letter topics, and a clean Validate.

Part 7 of 10 in the Hello World tutorial · Follow this part in the guided wizard · Printable version

Data contract rules check each message as it is written to a topic. A message that fails can be sent to a dead letter topic (DLQ) instead of stopping the pipeline. Flink statements have their own Errors output for records they can't process. By the end of this part every failure has somewhere to go, and the project validates.

Two rules for the canonical orders

  1. In the explorer, right-click Contract rules and choose New contract rule…
Right-click Contract rules → New contract rule…
Right-click Contract rules → New contract rule…
  1. Leave Start from as a blank rule (the list has ready-made templates to try another time). Fill in:
    • Rule name orderTotalPositive; Rule type Condition (CEL): it checks each whole message and fails when the expression is false.
    • Description An order must have a positive total. Failures go to the canonical order DLQ for the store team to check.
    • Runs On write (producer) · WRITE; On failure Send to dead letter topic · DLQ.
    • CEL expression message.totalPence > 0
  2. Click Create rule.
A CEL condition that sends failures to a dead letter topic.
A CEL condition that sends failures to a dead letter topic.
  1. Create a second rule the same way: customerNamePresent, Condition (CEL), description Every order must name its customer (the join found them). Failures go to the canonical order DLQ., WRITE, DLQ, expression size(message.customerName) > 0.
  2. Click the canonical topic hwc.prod.ent.order.cdm.v1, open Data contract, and in Add a rule… choose orderTotalPositive, then customerNamePresent. They run in the order listed.
The canonical topic’s Data contract tab, ready to add rules from the project’s library.
The canonical topic’s Data contract tab, ready to add rules from the project’s library.

Send rejected orders to a dead letter topic

  1. Once the topic has a rule that sends failures to a DLQ, it grows a Rejected output on its bottom edge (and a Contract · 2 badge). The orange ! is KPD reminding you the rejects have nowhere to go yet. Click the red + under the topic.
The Rejected output’s +.
The Rejected output’s +.
  1. Choose Dead Letter.
Only a Dead Letter topic can take rejected records.
Only a Dead Letter topic can take rejected records.
  1. On the dead letter topic's General tab: Name hwc.prod.ent.order.cdm.v1.dlq (the canonical topic's name plus .dlq); Description Canonical orders that broke a data quality rule, for the store team to check and fix at source. Holds customer details, so tagged like the canonical topic.; Replay strategy manual (someone fixes and resends them by hand).
  2. Governance tab: tag it Confidential and PII. A DLQ holds the same data as its source, so it needs the same protection.
The canonical DLQ, under the canonical topic. The red dashed line is labelled <i>contract rejects</i>.
The canonical DLQ, under the canonical topic. The red dashed line is labelled contract rejects.
  1. Click orders_raw_to_cdm. Its Errors output is the red + on its bottom edge. Click it and choose Dead Letter.
  2. Name it hwc.prod.sys.order.to-cdm.v1.dlq (sys, because it is a system topic), description Records orders_raw_to_cdm could not process (for example a value that would not convert), for the data team to inspect., replay manual, tags Confidential and PII.
Two dead letter topics in the Process box: Flink errors (left) and contract rejects (right).
Two dead letter topics in the Process box: Flink errors (left) and contract rejects (right).
  1. Now the two data product statements. Click order_history_product and click the red + on its Errors output.
The Errors + on order_history_product.
The Errors + on order_history_product.
  1. Choose Dead Letter: hwc.prod.sys.order.to-product.v1.dlq, description Records the two data product statements could not process, for the data team to inspect., replay manual, tags Confidential and PII.
The product DLQ. Its References list both statements once the second is wired.
The product DLQ. Its References list both statements once the second is wired.
  1. One DLQ can serve both product statements. Drag from the Errors dot under big_orders_product to the input dot on top of the new dead letter topic.
  2. Tidy up: drag the new DLQ into the bottom of the blue Surface box, under big_orders_product. If the line from order_history_product runs through big_orders_product, drag order_history_product a little to the right so its line drops straight down beside it.
Every failure has a home: Flink errors and contract rejects in Process, product errors in Surface.
Every failure has a home: Flink errors and contract rejects in Process, product errors in Surface.

Validate the project

  1. Click Validate in the toolbar.
The toolbar’s Validate.
The toolbar’s Validate.
  1. KPD checks the whole project. You should see Project validates: No errors or warnings, and Terraform will generate.
Project validates. The blue messages are information, not problems.
Project validates. The blue messages are information, not problems.

The messages are worth reading. Most say the clusters have no bootstrap server or ids yet: expected, because they don't exist yet. Two say how many CFUs each pool will use. The last ones are permission advice: the connector needs to create its topics (DeveloperManage), and Flink needs to write to the two sys dead letter topics, which its hwc.prod.ent. binding doesn't cover. You will accept that advice in Part 8.

Save. Next: 8. Export the picture, design document and Terraform.