datapatchbay

Data products as code. Declare owners, ports and freshness in YAML, then scaffold, validate and generate from one lightweight CLI.

The problem

Small and mid-size data teams hit the same four issues once more than one person or domain is involved:

  • Nobody owns a table. When a number is wrong, there is no name to call.

  • Everything is public. Consumers join straight into intermediate tables, so every refactor breaks a report.

  • The same concept has several names, and each gets a different count.

  • Access is handled by hand, so nobody can safely query the warehouse directly, and neither can an AI agent.

Data mesh gave teams the vocabulary for this but no implementation. The goal here is that creating a data product takes five minutes.

What datapatchbay does

datapatchbay is a file-based CLI that turns a data platform into a set of owned data products. Each data product is a folder of YAML files. They state who owns it, what it reads, what it publishes, how fresh each output is, and who may read it. The CLI scaffolds those folders, validates them in CI, and generates the orchestrator and warehouse files that put them into effect.

If the term is new to you, read What is a data product first.

The name comes from the patchbay behind a mixing console. Every input and output has a labelled jack, and you route by declaring connections. The command is dpb, and the project is licensed under Apache 2.0.

What it supports

Part of the platform

Supported

Warehouse

Postgres. DuckDB for local development and tests, where no access is enforced

Orchestrator

Airflow 3, Dagster, or a generated script that needs no orchestrator

Build code

Python functions and dbt models, chosen per table

Where build code runs

On the orchestrator’s worker. With Airflow, also in a Docker or Kubernetes container

datapatchbay is not released yet. It is not on PyPI, so you install it from GitHub, and the plugin interface may still change. Limits and guarantees lists what it does and does not do today, and Why not just dbt says when another tool fits better.

A first look

An output port is where a data product publishes data for others. It is a folder with a port.yaml, which states how fresh the data must be and who may read it:

# dataproducts/sa_jaffle_orders/outputs/sa_jaffle_orders/port.yaml
schema_version: 1
kind: schema
freshness: 24h
access:
  readers:
    - grp-shop-reporting
    - product: agg_order_turnaround
  pii_readers: [grp-shop-support]

Three commands take a data product from nothing to a DAG:

$ dpb new --name jaffle_orders --kind source-aligned --owner "Jane Doe" --source jaffle_shop --no-input
$ dpb validate
ok: 1 data product(s) valid
$ dpb generate

dpb new scaffolds the folder. dpb validate checks every file and the rules across files, and is meant for CI. dpb generate writes an Airflow DAG that builds the tables, audits them, and applies the views and grants.

Install

You need uv and Python 3.12 or newer:

uv tool install git+https://github.com/datapatchbay/patchbay

Installation lists the other ways to install it, and what the orchestrator needs.

Where to go next

New to datapatchbay

Read What is a data product, then follow the tutorial. It builds two data products from an empty folder, and runs one of them on your machine.

You evaluate it for your team

Why not just dbt compares it with dbt, a medallion architecture and other tools. How it works explains the model, and Limits and guarantees lists what to expect in production.

You have a task

The how-to guides cover one task each: run without an orchestrator, validate in CI, deploy the generated DAGs, give a reader access, build a table with dbt, import an existing dbt project, run build code in a container, run on Dagster and set up your editor. When a command fails, see Troubleshooting.

You look something up

The reference describes the commands, what dpb validate checks, what dpb generate writes, the four kinds of file, the settings, the dbt support and the terms.

Who it is for

  • A central data team on a single platform that wants ownership and contracts without buying a catalog.

  • A company where a second or third team starts building on the same warehouse.

  • Teams of 2 to 20 engineers, usually with one monorepo.

Principles

Lightweight

Three dependencies: Pydantic, Typer and ruamel.yaml. Managed and distributed with uv.

dpb never connects to the warehouse

It reads YAML and writes files. The orchestrator and the runtime do all the connecting.

Generated files stand alone

They do not import datapatchbay. If someone deletes datapatchbay, the DAGs still run.

No scheduler

The orchestrator schedules, and dpb generates what it runs.

CLI and files, no UI

Approvals happen in pull requests.

Defaults for everything

A team writes down only where it differs from the defaults.

Built for agents

--json output, a published JSON Schema, and a mode that fails instead of prompting.

Getting help