Smarter Week

How to automate it

How to automate “check models in production for drift and broken inputs”

Here are 2 ways to spend less time on this, best first. Each comes with steps you can follow today.

45 min
typically, once a week
60%
of the time can be automated
Some setup
to set up

Fix 1 of 2

AutomationBest fix

Monitor models automatically and alert on drift

Monitoring tools check input data and predictions on a schedule and alert when something drifts or breaks, so you stop checking dashboards by hand.

Typically saves about 55% of the time4 h to set up
  1. 1Log model inputs and predictions to a table.
  2. 2Set up monitoring in Databricks Lakehouse Monitoring, SageMaker Model Monitor, Evidently or Arize.
  3. 3Define alerts for missing features, drift and performance drops.
  4. 4Send alerts to your team channel with a link to the runbook.

Tools: Databricks Lakehouse Monitoring · Amazon SageMaker Model Monitor · Evidently · Arize

Fix 2 of 2

Software feature

Catch broken data before stakeholders do with tests and observability

Data tests and observability tools spot freshness, volume and schema problems and point to the upstream cause, so failures are found and fixed faster.

Typically saves about 35% of the time4 h to set up
  1. 1Add freshness, not-null and uniqueness tests to your most-used models (dbt tests or Elementary).
  2. 2Turn on an observability tool (Monte Carlo, Metaplane, Elementary, Bigeye) for anomaly detection and lineage.
  3. 3Send alerts to a data-team channel with the owner tagged.
  4. 4Use lineage to tell affected dashboard owners before they notice.

Tools: dbt tests · Elementary · Monte Carlo · Metaplane · Bigeye

Quick wins

Have you tried…

Do you get an alert when data is late or broken before someone tells you a dashboard is wrong?
Data tests and observability tools like Monte Carlo or Elementary watch freshness, volume and schema changes and alert the right owner with the upstream cause.

Who does this task

Roles in our library that list this as one of their common tasks. Each guide covers the rest of that role’s week.

Data scientist

HourLeak · the 8-minute work audit

How many hours does this cost you?

The free 8-minute check works out where your week goes and gives you your top fixes. The team scan does the same for everyone and adds it up, so you know which leaks to fix first.

Take the free check Scan my team, $49

Answers are anonymous. Leaders only see team totals.

Other common tasks for Data scientists