Components
This feature is still in development and might change in patch releases. It’s not production ready, and the documentation may also evolve. Stay tuned for updates.
Welcome to Dagster Components.
Dagster Components is a new way to structure your Dagster projects. It aims to provide:
- An opinionated project layout that supports ongoing scaffolding from "Hello world" to the most advanced projects.
- A class-based interface (
Component) for dynamically constructing Dagster definitions from arbitrary data (such as third-party integration configuration files). - A toolkit for building YAML DSLs for
Components, allowing instances of components to be defined with little to no Python code. - A Dagster-provided set of component types that provide a simplified user experience for common integrations.
Installation
To use the Components framework, you must install the dg command line tool, which lives in the published Python package dagster-dg. dg is designed to be globally installed and has no dependency on dagster itself. dg allows you to quickly scaffold Dagster projects and populate them with components.
1. Install uv
We will use the Python package manager uv to install a globally available dg:
brew install uv
2. Install the dg command line tool
Once you have installed uv, use it to install dg.
uv tool install dagster-dg
uv tool install installs python packages from PyPI into isolated environments and exposes their executables on your shell path. This means the dg command should now be available. It will always execute in an isolated environment separate from any project environment.
If you have a local clone of the dagster repo, you can install a local version of dg with uv tool install -e $DAGSTER_GIT_REPO_DIR/python_modules/libraries/dagster-dg. This will create an isolated environment for dg just like the standard uv tool install, but the environment will contain an editable installation of dagster-dg.
dg API reference
Below is the help message for dg, which also serves as the API reference:
dg --help
Usage: dg [OPTIONS] COMMAND [ARGS]...
CLI for working with Dagster components.
╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
│ --clear-cache Clear the cache. │
│ --rebuild-component-registry Recompute and cache the set of available │
│ component types for the current environment. │
│ Note that this also happens automatically │
│ whenever the cache is detected to be stale. │
│ --install-completion Automatically detect your shell and install a │
│ completion script for the `dg` command. This │
│ will append to your shell startup file. │
│ --version -v Show the version and exit. │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Global options ───────────────────────────────────────────────────────────────────────╮
│ --require-local-venv --no-require-local-venv Require use of a local │
│ virtual environment │
│ (`.venv` found in │
│ ancestors of the working │
│ directory). │
│ --use-dg-managed-enviro… --no-use-dg-managed-en… Enable management of the │
│ virtual environment with │
│ uv. │
│ --builtin-component-lib TEXT Specify a builitin │
│ component library to use. │
│ --verbose Enable verbose output for │
│ debugging. │
│ --disable-cache Disable the cache.. │
│ --cache-dir PATH Specify a directory to │
│ use for the cache. │
╰────────────────────────────────────────────────────────────────────────── ──────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────╮
│ check Commands for checking the integrity of your Dagster code. │
│ dev Start a local instance of Dagster. │
│ docs Commands for generating docs from your Dagster code. │
│ init Initialize a new Dagster workspace and a first project within that │
│ workspace. │
│ list Commands for listing Dagster entities. │
│ scaffold Commands for scaffolding Dagster code. │
│ utils Assorted utility commands. │
╰────────────────────────────────────────────────────────────────────────────────────────╯