Auto regular updates

ETL was initially conceived to maintain and publish datasets that are updated once a year. This is still the most common kind of update that we do this days.

However, sometimes we need to update datasets more frequently. This is the case for instance for the COVID-19 dataset, and other examples, which need weekly or monthly updates.

In such cases, the processing code remains the same, but the origin data needs to be updated. Put simply, the ETL process is the same, but with an updated snapshot of the data.

If you want to keep a dataset up-to-date with the latest data, follow the steps below.

Create the data pipeline using latest version¶

Firstly, create the necessary steps to build the dataset (i.e. snapshot, meadow, garden, etc.). Use version latest for all of them, to avoid adding duplicate code.

Make sure to add these steps to the DAG. For instance, in the example below, we want to keep the cases_deaths dataset up-to-date with the latest data.

# WHO - Cases and deaths
data://meadow/covid/latest/cases_deaths:
  - snapshot://covid/latest/cases_deaths.csv
data://garden/covid/latest/cases_deaths:
  - data://meadow/covid/latest/cases_deaths
  - data://garden/regions/2023-01-01/regions
  - data://garden/wb/2024-03-11/income_groups
  - data://garden/demography/2024-07-15/population
data://grapher/covid/latest/cases_deaths:
  - data://garden/covid/latest/cases_deaths

Create the update script¶

Create an update script and save it in the scripts/ directory. This script must be a bash script, which basically needs to run the necessary code to update the snapshot. In the example below, we user [].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/bin/bash
#
#  update-covid-cases-deaths.sh
#
#  Update COVID-19 cases and deaths dataset data://grapher/covid/latest/cases_deaths
#

set -e

source "$(dirname "$0")/commit-snapshots.sh"

start_time=$(date +%s)

echo '--- Update COVID-19 cases and deaths'
cd /home/owid/etl
uv run etls covid/latest/cases_deaths

# Files this job owns. commit_and_push_snapshots refuses to push a change to anything else.
snapshot_files=(
    snapshots/covid/latest/cases_deaths.csv.dvc
)

# commit to master will trigger ETL which is gonna run the step
echo '--- Commit and push changes'

commit_and_push_snapshots ":robot: update: covid-19 cases and deaths" "${snapshot_files[@]}"

end_time=$(date +%s)

echo "--- Done! ($(($end_time - $start_time))s)"

In the example above, replace the uv run etls line with your own snapshot, list the .dvc files it writes in snapshot_files, and set the commit message. Commit through commit_and_push_snapshots (from scripts/commit-snapshots.sh), not plain git commit and git push. It skips the pre-commit hook, refuses to push a change to any file outside snapshot_files, and fails the job if the commit or the push fails.

Schedule update in Buildkite.¶

Finally, schedule the update by adding a step to .buildkite/etl/automatic_dataset_updates.yml in owid/ops. The Buildkite pipeline loads it from there. Include *shared-checkout-lock: the job writes to the same checkout that ETL deploys reset, so the two must not run at the same time.

- label: "Update <step>"
  <<: [*retry-settings, *shared-checkout-lock]
  command:
    - "sudo su - owid -c 'bash /home/owid/etl/scripts/update-<step>.sh'"

See Also¶

• Automatic updates via metadata - An alternative approach using metadata in .dvc files to enable automatic dataset updates