KPD closed beta Sign inRequest access

1. Create the project

A new project from the Data Product (DPCF) template, with three clusters and two compute pools.

Part 1 of 10 in the Hello World tutorial · Follow this part in the guided wizard · Printable version

A project is a folder that holds everything for one piece of work: its pipelines, schemas, clusters, accounts, tags and rules. You start every design by creating one.

Open the New project dialog

  1. Open KPD. You see whichever project you had open last (here, the Inclined Labs sample), or the welcome screen if this is your first time.
  2. Click the File menu at the top left.
KPD as it opens. The File menu is top left.
KPD as it opens. The File menu is top left.
  1. Choose New project…
File → New project…
File → New project…

Name it and pick the template

  1. Project name: type Hello World Coffee. This is the only required field.
  2. Client: type Hello Coffee Co. It appears on the design document and the Terraform README.
  3. Click the Data Product (DPCF) card. The template gives you classification tags, a DataProduct catalogue metadata definition, the DPCF topic naming rule, two suggested compute pools, and a first pipeline laid out as Ingestion → Process → Surface.
  4. Organisation code: type hwc. Every topic name will start with it: the hint shows the pattern hwc.<env>.<raw|ent|sys>.<entity>.<function>.v<n>.
Name, client, the Data Product (DPCF) template and the organisation code hwc.
Name, client, the Data Product (DPCF) template and the organisation code hwc.
Reading a DPCF topic namehwc.prod.raw.order.shopdb.v1 means: organisation hwc, environment prod, layer raw (as captured; ent is the cleaned enterprise layer and sys is for system topics such as dead letters), entity order, function shopdb (where it came from), version 1. KPD checks every topic name against this rule as you type. See Topic naming standards.

Add three clusters and check the pools

Lower down the dialog you can describe your Kafka clusters and Flink compute pools now, so pipelines can use them straight away. Part 3 looks at them in more detail.

  1. Under Kafka clusters, click Add cluster three times to get three rows.
  2. Fill them in like this:
    EnvironmentCluster nameTypeCloudRegion
    devhwc-devStandardAWSeu-west-2
    testhwc-testStandardAWSeu-west-2
    prodhwc-prodStandardAWSeu-west-2
  3. Under Flink compute pools the template has already suggested two pools, one per processing stage: hwc-raw-to-cdm (raw to canonical) and hwc-cdm-to-surface (canonical to data products). Leave them at 10 max CFU. Once a cluster has a region, the pools take it.
Three Standard clusters on AWS in London (eu-west-2), and the two suggested compute pools.
Three Standard clusters on AWS in London (eu-west-2), and the two suggested compute pools.
  1. Click Create project.
Everything filled in. Create project is bottom right.
Everything filled in. Create project is bottom right.

A quick look around

KPD creates the project folder and opens its first pipeline, data-product.kpd, with the three layer boxes already drawn and a Start your pipeline card in the middle.

The new project. Left: the explorer. Middle: the canvas. Bottom: the properties panel. Right: the palettes.
The new project. Left: the explorer. Middle: the canvas. Bottom: the properties panel. Right: the palettes.

Press Ctrl/⌘+S to save. Next: 2. Draw the domain map.