Walkthrough
Walkthrough is deprecated. Use Wizard instead!
The walkthrough is an interactive web UI for setting up the different ETL steps. It creates the base files to help you create the recipe for a dataset.
Get set up
Before you begin, make sure you've set up the ETL as described in Getting Started.
Walkthrough options
The walkthrough currently supports the following stages:
Option | Description |
---|---|
snapshot |
Create a Snapshot step: Insert upstream data to our catalog. |
walden |
(This step is deprecated. Use snapshot instead). Create a Walden step: Insert upstream data to our catalog. |
meadow |
Create a Meadow step: Format data. |
garden |
Create a Garden step: Harmonize and process data. |
grapher |
Create a Grapher step: Transform data to be Grapher-ready. |
explorers |
Create an Explorer step: Transform data to be Explorer-ready. |
charts |
Tool to update our charts. |
$ walkthrough --help
Usage: walkthrough [OPTIONS] {walden|snapshot|meadow|garden|grapher|explo
rers|charts}
Options:
--run-checks / --skip-checks Environment checks
--dummy-data Prefill form with dummy data, useful for
development
--auto-open / --no-auto-open Auto open browser on port 8082
--port INTEGER Application port
--help Show this message and exit.
Start the walkthrough
Walkthrough consists of a web app that will ask you for various metadata fields (such as namespace
, version
, source_url
, etc.). Based on your input,
it will generate the required files in the appropriate snapshots/
or etl/
directory. Walkthrough will point you to the commands that you need to run once you
have implemented the step.
A typical workflow would involve:
- Create a snapshot step:
poetry run walkthrough snapshot
. - Create a meadow step based on the output from 1:
poetry run walkthrough meadow
. - Create a garden step based on the output from 2:
poetry run walkthrough garden
. - Create a grapher step based on the output from 3:
poetry run walktrhough grapher
.