cowid

COVID-19 Data pipeline tool by Our World in Data.

cowid [OPTIONS] COMMAND [ARGS]...

Options

--parallel, --no-parallel

Parallelize process.

Default:

True

--n-jobs <n_jobs>

Number of threads to use.

Default:

-2

-S, --server, --no-server

Only critical log and final message to slack.

Default:

False

check

COVID-19 data pipeline checks.

cowid check [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

hosp

Upload dataset to DB.

cowid check hosp [OPTIONS]

jhu

Upload dataset to DB.

cowid check jhu [OPTIONS]

megafile

Upload dataset to DB.

cowid check megafile [OPTIONS]

test

Upload dataset to DB.

cowid check test [OPTIONS]

vax

Generate dataset.

cowid check vax [OPTIONS]

decoupling

COVID-19 Decoupling data pipeline.

cowid decoupling [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Download and generate our COVID-19 Decoupling dataset.

cowid decoupling generate [OPTIONS]

grapher-db

Download and generate our COVID-19 Decoupling dataset.

cowid decoupling grapher-db [OPTIONS]

gmobility

Google Mobility data pipeline.

cowid gmobility [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Download and generate our COVID-19 Hospitalization dataset.

cowid gmobility generate [OPTIONS]

grapher-db

Step 3: Update Grapher database with generated files.

cowid gmobility grapher-db [OPTIONS]

grapher-io

Step 2: Generate grapher-ready files.

cowid gmobility grapher-io [OPTIONS]

hosp

COVID-19 Hospitalization data pipeline.

cowid hosp [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Runs scraping scripts to collect the data from the primary sources, transforms it and exports the result to public/data/hospitalizations/.

By default, the default values for OPTIONS are those specified in the configuration file. The configuration file is a YAML file with the pipeline settings. Note that the environment variable OWID_COVID_CONFIG must be pointing to this file. We provide a default config file in the project folder scripts/config.yaml.

OPTIONS passed via command line will overwrite those from configuration file.

cowid hosp generate [OPTIONS] [COUNTRIES]...

Options

-s, --skip-countries <skip_countries>

List of countries to skip (comma-separated)

Arguments

COUNTRIES

Optional argument(s)

grapher-db

Step 3: Update Grapher database with generated files.

cowid hosp grapher-db [OPTIONS]

grapher-io

Step 2: Generate grapher-ready files.

cowid hosp grapher-io [OPTIONS]

jhu

COVID-19 Cases/Deaths data pipeline.

cowid jhu [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Step 2: Generate dataset.

cowid jhu generate [OPTIONS]

get

Downloads all JHU source files into project directory.

cowid jhu get [OPTIONS]

grapher-db

Step 3: Update Grapher database with generated files.

cowid jhu grapher-db [OPTIONS]

megafile

COVID-19 data integration pipeline (former megafile)

cowid megafile [OPTIONS]

oxcgrt

COVID-19 stringency index (by OxCGRT) data pipeline.

cowid oxcgrt [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

get

Downloads all OxCGRT source files into project directory.

cowid oxcgrt get [OPTIONS]

grapher-db

Step 3: Update Grapher database with generated files.

cowid oxcgrt grapher-db [OPTIONS]

grapher-io

Step 2: Generate grapher-ready files.

cowid oxcgrt grapher-io [OPTIONS]

sweden

COVID-19 Sweden data pipeline.

cowid sweden [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Generate our COVID-19 Sweden dataset.

cowid sweden generate [OPTIONS]

get

Download source file for Sweden dataset.

cowid sweden get [OPTIONS]

grapher-db

Download and generate our COVID-19 Decoupling dataset.

cowid sweden grapher-db [OPTIONS]

test

COVID-19 Testing data pipeline.

cowid test [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

get

Runs scraping scripts to collect the data from the primary sources of COUNTRIES. Data is exported to project folder scripts/output/testing/. By default, all countries are scraped.

By default, the default values for OPTIONS are those specified in the configuration file. The configuration file is a YAML file with the pipeline settings. Note that the environment variable OWID_COVID_CONFIG must be pointing to this file. We provide a default config file in the project folder scripts/config.yaml.

OPTIONS passed via command line will overwrite those from configuration file.

Examples:

Run the step using default values, from config.yaml file: cowid test get

Run the step only for Australia: cowid test get australia

Run the step for all countries except Australia: cowid test get -s australia all

Run the step for all incremental processes (can be also done using ‘batch’): cowid test get incremental

cowid test get [OPTIONS] [COUNTRIES]...

Options

-s, --skip-countries <skip_countries>

List of countries to skip (comma-separated).

Arguments

COUNTRIES

Optional argument(s)

uk-nations

COVID-19 UK Nations data pipeline.

cowid uk-nations [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Generate dataset.

cowid uk-nations generate [OPTIONS]

grapher-db

Upload dataset to DB.

cowid uk-nations grapher-db [OPTIONS]

variants

COVID-19 Variants data pipeline.

cowid variants [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Download and generate our COVID-19 Hospitalization dataset.

cowid variants generate [OPTIONS]

grapher-io

Step 2: Generate grapher-ready files.

cowid variants grapher-io [OPTIONS]

vax

COVID-19 Vaccination data pipeline.

cowid vax [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

export

Step 4: Export vaccination data and merge with global dataset.

cowid vax export [OPTIONS]

generate

Step 3: Generate vaccination dataset.

cowid vax generate [OPTIONS]

get

Runs scraping scripts to collect the data from the primary sources of COUNTRIES. Data is exported to project folder scripts/output/vaccinations/. By default, all countries are scraped.

By default, the default values for OPTIONS are those specified in the configuration file. The configuration file is a YAML file with the pipeline settings. Note that the environment variable OWID_COVID_CONFIG must be pointing to this file. We provide a default config file in the project folder scripts/config.yaml.

OPTIONS passed via command line will overwrite those from configuration file.

Examples:

Run the step using default values, from config.yaml file: cowid vax get

Run the step only for Australia: cowid vax get australia

Run the step for all countries except Australia: cowid vax get -s australia all

Run the step for all incremental processes (can be also done using ‘batch’): cowid vax get incremental

cowid vax get [OPTIONS] [COUNTRIES]...

Options

-s, --skip-countries <skip_countries>

List of countries to skip (comma-separated)

-O, --optimize, --no-optimize

Optimize processes based on older logging times.

Default:

False

Arguments

COUNTRIES

Optional argument(s)

icer

Download some specific country files. Useful when these are very large in size.

cowid vax icer [OPTIONS]

process

Process data in folder scripts/output/vaccinations/.

By default, the default values for OPTIONS are those specified in the configuration file. The configuration file is a YAML file with the pipeline settings. Note that the environment variable OWID_COVID_CONFIG must be pointing to this file. We provide a default config file in the project folder scripts/config.yaml.

OPTIONS passed via command line will overwrite those from configuration file.

Example: Process all country data.

cowid vax process

cowid vax process [OPTIONS]

track

Explore high-level analytics of vaccination dataset.

cowid vax track [OPTIONS]

Options

--output <output>

Output CSV file.

Default:

'cowid-vax-track.report.tmp.csv'

-d, --disable-export <disable_export>

Do not export CSV file, just logging results.

Default:

False

us-states

US vaccinations data pipeline.

cowid vax us-states [OPTIONS]

xm

COVID-19 Excess Mortality data pipeline.

cowid xm [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

generate

Download and generate our COVID-19 Hospitalization dataset.

cowid xm generate [OPTIONS]