Dagster is flexible and allows incremental adoption. It provides add-on libraries to integrate with your existing tools and infrastructure.
This section includes guides on how to use Dagster with other tools.
| Name | Description |
|---|---|
| Dagster with dbt | Orchestrate dbt from Dagster. |
| Dagster with Great Expectations | Run data quality tests using Great Expectations in a Dagster pipeline. |
| Dagster with Spark | Define and execute Spark jobs in Dagster. |
| Dagster with Pandas | How Dagster works with Pandas. |
| Dagster with Pandera | Validate Pandas dataframes and visualize dataframe schemas with Pandera. |
| Dagster with Jupyter/Papermill | How to orchestrate Jupyter notebooks from Dagster. |
| Dagster with Airflow | Use Dagster in an Airflow cluster, or transform Airflow DAGs into Dagster pipelines. |
Here is a complete list of Dagster's integration libraries. See full documentation in API Reference.
| Integration | Library |
|---|---|
| Airbyte | dagster-airbyte |
| Airflow | dagster-airflow |
| AWS | dagster-aws |
| Azure | dagster-azure |
| Celery | dagster-celery |
| Celery + Docker | dagster-celery-docker |
| Dask | dagster-dask |
| Databricks | dagster-databricks |
| Datadog | dagster-datadog |
| Docker | dagster-docker |
| dbt | dagster-dbt |
| Fivetran | dagster-fivetran |
| GCP | dagster-gcp |
| Great Expectations | dagster-ge |
| Github | dagster-github |
| Kubernetes | dagster-k8s |
| Microsoft Teams | dagster-msteams |
| MLflow | dagster-mlflow |
| MySQL | dagster-mysql |
| PagerDuty | dagster-pagerduty |
| Pandas | dagster-pandas |
| Pandera | dagster-pandera |
| Papermill | dagstermill |
| Papertrail | dagster-papertrail |
| PostgreSQL | dagster-postgres |
| Prometheus | dagster-prometheus |
| Pyspark | dagster-pyspark |
| Shell | dagster-shell |
| Slack | dagster-slack |
| Snowflake | dagster-snowflake |
| Snowflake + Pandas | dagster-snowflake-pandas |
| Spark | dagster-spark |
| SSH / SFTP | dagster-ssh |
| Twilio | dagster-twilio |