Using the interactive walkthrough
The walkthrough is an interactive web UI for setting up the different ETL steps. It creates the base files to help you create the recipe for a dataset.
Get set up
Before you begin, make sure you've set up the ETL as described in Getting Started.
Walkthrough options
The walkthrough currently supports the following stages:
Option | Description |
---|---|
snapshot |
Create a Snapshot step: Insert upstream data to our catalog. |
walden |
(This step is deprecated. Use snapshot instead). Create a Walden step: Insert upstream data to our catalog. |
meadow |
Create a Meadow step: Format data. |
garden |
Create a Garden step: Harmonize and process data. |
grapher |
Create a Grapher step: Transform data to be Grapher-ready. |
explorers |
Create an Explorer step: Transform data to be Explorer-ready. |
charts |
Tool to update our charts. |
$ walkthrough --help
Usage: walkthrough [OPTIONS] {walden|snapshot|meadow|garden|grapher|explo
rers|charts}
Options:
--run-checks / --skip-checks Environment checks
--dummy-data Prefill form with dummy data, useful for
development
--auto-open / --no-auto-open Auto open browser on port 8082
--port INTEGER Application port
--help Show this message and exit.
Start the walkthrough
Walkthrough consists of a web app that will ask you for various metadata fields (such as namespace
, version
, source_url
, etc.). Based on your input,
it will generate the required files in the appropriate snapshots/
or etl/
directory. Walkthrough will point you to the commands that you need to run once you
have implemented the step.
A typical workflow would involve:
- Create a snapshot step:
poetry run walkthrough snapshot
. - Create a meadow step based on the output from 1:
poetry run walkthrough meadow
. - Create a garden step based on the output from 2:
poetry run walkthrough garden
. - Create a grapher step based on the output from 3:
poetry run walktrhough grapher
.