Smarter Week

How to automate it

How to automate “build and maintain ingestion from new data sources”

Here are 2 ways to spend less time on this, best first. Each comes with steps you can follow today and, for AI fixes, a prompt to copy.

120 min
typically, once a week
60%
of the time can be automated
Some setup
to set up

Fix 1 of 2

Software featureBest fix

Use managed connectors instead of hand-built ingestion

Fivetran, Airbyte, dlt and the warehouse's own connectors handle API changes, schema drift and retries for common sources, so you maintain far less custom code.

Typically saves about 50% of the time4 h to set up
  1. 1List your custom ingestion jobs and how often each breaks.
  2. 2Check which sources have a managed connector in Fivetran, Airbyte, or your warehouse (Snowflake, Databricks Lakeflow, BigQuery Data Transfer).
  3. 3Move the most fragile ones first and run both in parallel for a week.
  4. 4For sources without a connector, use dlt or a coding agent to build from a template.

Tools: Fivetran · Airbyte · dlt · Databricks Lakeflow Connect · Snowflake connectors

Fix 2 of 2

AI

Use a coding agent on your dbt or pipeline repo

Claude Code, Codex, Cursor and Copilot can read the project, write models and tests, run dbt build and fix errors. You review the logic and the data it produces.

Typically saves about 30% of the time30 min to set up
  1. 1Add a short instructions file to the repo (CLAUDE.md or AGENTS.md) with naming conventions and how to run dbt and tests.
  2. 2Give the agent a well-scoped task with the prompt below.
  3. 3Let it run the build and tests against a dev target, never production.
  4. 4Review the SQL and check row counts against a known source before merging.
Prompt to copy
In this [DBT / AIRFLOW / DAGSTER] project, [TASK, e.g. add a model that calculates monthly active customers from events]. Follow the conventions in [EXAMPLE MODEL]. Add tests for [UNIQUENESS, NOT NULL, ACCEPTED VALUES] and a description for each column. Run the build against the dev target, fix any errors, and summarize what you changed plus anything you were unsure about. Do not run anything against production.

Tools: Claude Code, OpenAI Codex, Cursor or GitHub Copilot

Who does this task

Roles in our library that list this as one of their common tasks. Each guide covers the rest of that role’s week.

Data engineer

HourLeak · the 8-minute work audit

How many hours does this cost you?

The free 8-minute check works out where your week goes and gives you your top fixes. The team scan does the same for everyone and adds it up, so you know which leaks to fix first.

Take the free check Scan my team, $49

Answers are anonymous. Leaders only see team totals.

Other common tasks for Data engineers