Skip to main content

Dagster & Omni (Component)

Preview feature

This feature is considered in a preview stage, and is under active development, and not considered ready for production use. You may encounter feature gaps, and the APIs may change. For more information, see the API lifecycle stages documentation.

The dagster-omni library provides an OmniComponent which can be used to easily represent Omni documents and queries as assets in Dagster.

info

OmniComponent is a state-backed component, which fetches and caches Omni workspace metadata. For information on managing component state, see Configuring state-backed components.

1. Prepare a Dagster project​

To begin, you'll need a Dagster project. You can use an existing components-ready project or create a new one:

dg project scaffold --name project my-project && cd my-project/src

Activate the project virtual environment:

source ../.venv/bin/activate

Finally, add the dagster-omni library to the project:

uv add dagster-omni

2. Scaffold an Omni component definition​

Now that you have a Dagster project, you can scaffold an Omni component definition:

dg scaffold defs dagster_omni.OmniComponent omni_ingest
Creating defs at /.../my-project/src/my_project/defs/omni_ingest.

The dg scaffold defs call will generate a defs.yaml file:

tree my_project/defs
my_project/defs
├── __init__.py
└── omni_ingest
    └── defs.yaml

2 directories, 2 files

3. Configure your Omni workspace​

Update the defs.yaml file with your Omni workspace connection details. You'll need to provide your Omni instance URL and API key. For more information on creating API credentials, see the Omni API documentation.

my_project/defs/omni_ingest/defs.yaml
type: dagster_omni.OmniComponent

attributes:
  workspace:
    base_url: https://your-company.omniapp.co
    api_key: "{{ env.OMNI_API_KEY }}"
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                              ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key                         ┃ Group     ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ Analytics/customer_analysis │ Analytics │      │ omni  │             │ │
│         │ ├─────────────────────────────┼───────────┼──────┼───────┼─────────────┤ │
│         │ │ Analytics/revenue_report    │ Analytics │      │ omni  │             │ │
│         │ ├─────────────────────────────┼───────────┼──────┼───────┼─────────────┤ │
│         │ │ Analytics/sales_dashboard   │ Analytics │      │ omni  │             │ │
│         │ └─────────────────────────────┴───────────┴──────┴───────┴─────────────┘ │
└─────────┴──────────────────────────────────────────────────────────────────────────┘

4. Customize Omni asset metadata​

You can customize the metadata and grouping of Omni assets using the translation key:

my_project/defs/omni_ingest/defs.yaml
type: dagster_omni.OmniComponent

attributes:
  workspace:
    base_url: https://your-company.omniapp.co
    api_key: "{{ env.OMNI_API_KEY }}"
  translation:
    group_name: analytics
    tags:
      domain: bi
dg list defs --columns name,group,tags
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ │
│         │ ┃ Key                         ┃ Group     ┃ Tags          ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │
│         │ │ Analytics/customer_analysis │ analytics │ "domain"="bi" │ │
│         │ ├─────────────────────────────┼───────────┼───────────────┤ │
│         │ │ Analytics/revenue_report    │ analytics │ "domain"="bi" │ │
│         │ ├─────────────────────────────┼───────────┼───────────────┤ │
│         │ │ Analytics/sales_dashboard   │ analytics │ "domain"="bi" │ │
│         │ └─────────────────────────────┴───────────┴───────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────┘