Jobs & Pipelines: from raw tables to reports, on a schedule | Docs

Jobs & Pipelines is where your data gets worked on while you are not looking. A pipeline is SQL that builds tables in your warehouse from your site databases; a job puts pipelines, SQL scripts, dashboard refreshes and notifications in order and runs them on a schedule. Both live in one list in Manage — sidebar group "Data Warehouse", next to Databases, Runs and Dashboards — and every run is on the Runs page. Included in every plan, at the budget of your warehouse tier.

A pipeline is SQL

There is no diagram to draw. A pipeline is a folder of up to 20 .sql files (64 KB each) that you edit in the same CodeMirror editor as the Databases workbench, with the databases you may read completing under their names (shop. completes). Each file holds any number of ;-separated statements, and there are exactly two forms:

CREATE OR REFRESH TABLE [warehouse.]name [KEY (col, col)] AS <select>;
CREATE OR REFRESH SOURCE VIEW name [FROM database] AS <select>;

TABLE is materialised in the warehouse (MariaDB or PostgreSQL — whichever the target database is). With KEY the rows are upserted on those columns, so a nightly run only rewrites what changed; without KEY the table is rebuilt every run. The SELECT may read source views and other tables of the pipeline, tables of the warehouse, and any table of a site database as database.table.
SOURCE VIEW runs on the site database, in its own dialect, through the same read-only login as the workbench, and is streamed into the warehouse as a staging table. Put your WHERE there so only the rows you need travel. It may not read anything the pipeline defines.

Sources are any site database, always read-only; the target is always a warehouse database. The dependency graph and lineage are derived by the server from the SQL: every name after FROM or JOIN that the pipeline defines is a dependency, so the build order follows from what you wrote, a loop is refused, and so is building in two warehouses. The panel under the editor shows Problems (as file:line, refreshed as you type), the Pipeline graph — "Source view" and "Materialised table" nodes — and Tables.

Header buttons: Run file previews the open file without writing anything — the first 50 rows per table, built from 200 rows per source; Dry run takes the whole pipeline through a run that writes nothing; Run pipeline builds it for real and shows on Runs. A job can run a pipeline with Full refresh, which rebuilds every incremental table that once. Up to 20 pipelines per account.

A job

A job is a list of steps that run one at a time, in order. Each step may depend on earlier steps (depends_on: run when that step succeeded, failed, or always), and there are eight kinds:

Run a pipeline — with the optional full-refresh toggle;
Run SQL — up to 50 statements against a warehouse database, in order, stopping at the first error;
Refresh a dashboard — runs every panel and keeps the answers warm, so the page opens instantly;
Send an email — to members of this account only; always, only when the job succeeds or only when it fails; subject and body may use {{job.name}}, {{run.status}}, {{run.duration}}, {{run.url}} and {{step.<key>.rows_written}} (also rows_read, status); optionally attach a CSV of a query, up to 5 MB or 50,000 rows;
Call a webhook — one of the endpoints on your Integrations page, on failure by default;
Wait — up to 300 seconds;
Run another job — and optionally wait for it;
Check — a yes/no question on something an earlier step reported (rows written, rows read, status, panels warmed…) or on the first cell of one read-only SELECT, with is, is not, is more than, is at least, is less than, is at most, is empty, is not empty — and the steps after it wired to the answer.

A job runs as the person who last saved it, and reaches exactly as far as they could by hand; a pipeline inside it runs as its last author. If that person leaves the account, the job stops until someone with access saves it again.

Triggers and concurrency

A job runs Only when I run it ("Runs when you press the button"), On a schedule, or After another job succeeds. Schedules come as presets — every few minutes or hours (with a from–to window and the weekdays it applies to), every hour, day, week or month — or as a custom cron expression, read in the timezone you choose. The shortest interval is 5 minutes on Pro and 1 minute on Business. One run at a time per account: when a scheduled run arrives while another is still going, the job either skips the new run (recorded as skipped) or queues it, up to three waiting. 500 runs a day per account.

Budgets per run

Every run is held to the account's ceiling — the budget of its warehouse tier; a job may lower its own budget, never raise it. On Starter (Warehouse XS) and Pro (S): 15 minutes, 500,000 rows, 512 MB extracted from sources, 512 MB of memory. On Business: 60 minutes, 5 million rows, 2 GB, 1 GB — its published budget, kept on top of Warehouse M. Warehouse L on Starter or Pro: 60 minutes, 5 million rows, 2 GB, 2 GB. A run that crosses its budget is stopped and says so. Each tier includes job minutes a month — 30 on XS, 120 on S, 600 on M, 2,000 on L — and runs above them are metered by compute time. Runs and their logs are kept for 90 days.

The Runs page

Runs lists "every unit of compute this account has used" — job runs, pipeline runs, dashboard refreshes, editor statements, API and MCP queries, imports — with the metered time added up for the window. Metered compute is time; "Nothing here is a bill." Filter by kind, status, who launched it and dates; open a run for its Steps, the Log drawer per step or for the whole run, and Stop one that is still going.

Limits

The published limits are the standard offer:

20 jobs and 20 pipelines per account, 20 steps per job, 20 files of 64 KB per pipeline;
schedules from every 5 minutes (Business and Warehouse L: every minute), one run at a time, three queued, 500 runs a day;
per run 15 min / 500,000 rows / 512 MB / 512 MB memory on Starter and Pro (Business: 60 min / 5 M rows / 2 GB / 1 GB; Warehouse L: 60 min / 5 M rows / 2 GB / 2 GB);
job minutes a month included: 30 on Starter, 120 on Pro, 600 on Business, 2,000 on Warehouse L — runs above them are metered by compute time;
runs kept 90 days; job runs are metered by compute time, shown on Runs before anything is billed.

Need more? Contact us — larger limits are available on request, for a fee.

From a script or your AI assistant

Jobs and pipelines are in the API and the MCP server, behind the opt-in data scope:

GET / POST /api/v1/jobs, GET / PUT / DELETE /api/v1/jobs/{id}, POST /api/v1/jobs/{id}/run, GET /api/v1/jobs/{id}/runs, GET /api/v1/jobs/runs, POST /api/v1/jobs/runs/{id}/cancel;
GET / POST /api/v1/pipelines, GET / PUT / DELETE /api/v1/pipelines/{id}, POST …/compile (pure and free — problems with file and line), POST …/preview (writes nothing), POST …/run, GET …/runs.

An assistant gets list_jobs, create_job, run_job, get_job_runs, list_pipelines, create_pipeline, preview_pipeline and run_pipeline — the whole pipeline grammar is in the create_pipeline tool's description, so a connected assistant can write a pipeline for you and check it compiles before saying it is done. A run answered as skipped or queued is the platform declining on purpose, not a failure.