From 44144220ed16f69767ea35d24b940b07af6964b0 Mon Sep 17 00:00:00 2001 From: Fernando Mercado Date: Wed, 26 Aug 2026 09:02:00 -0500 Subject: [PATCH 1/7] Document required Schema field for @task.datacoves_dbt service connections Users leaving Schema empty on a Datacoves service connection hit a dbt profile validation error ("None is not of type 'string'") that names their own dbt profile, leading them to wrongly suspect the repo's profiles.yml is being used instead of the generated temporary profile. Clarify that Schema is required for @task.datacoves_dbt, and that the Airflow Snowflake provider's "optional" schema applies to the Airflow connection only, not the generated dbt profile. --- docs/how-tos/datacoves/how_to_service_connections.md | 2 +- docs/reference/airflow/datacoves-decorators.md | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/how-tos/datacoves/how_to_service_connections.md b/docs/how-tos/datacoves/how_to_service_connections.md index b0b5d1a..2a62e5a 100644 --- a/docs/how-tos/datacoves/how_to_service_connections.md +++ b/docs/how-tos/datacoves/how_to_service_connections.md @@ -26,7 +26,7 @@ A Service Connection consists of the following fields: - **Environment Variables** The legacy method Datacoves used which would inject the connection credentials as environment variables into Airflow. The name of the service connection will be used to dynamically create [environment variables](/docs/reference/airflow/environment-service-connection-vars) which we inject into Airflow. ![Delivery Mode](assets/service_connection_delivery.jpg). - **Connection Template** The connection template to base this service connection on(i.e. the defaults) - Depending on the template selected, additional fields will be displayed with the default values entered in the connection template. These default values can be overridden by toggling the indicator next to the given value. Enter the appropriate user, schema, and password. This is commonly a service account created specifically for Airflow and may differ between the development and production environment. + Depending on the template selected, additional fields will be displayed with the default values entered in the connection template. These default values can be overridden by toggling the indicator next to the given value. Enter the appropriate user, schema, and password. This is commonly a service account created specifically for Airflow and may differ between the development and production environment. **Schema is required** if the connection will be used with [`@task.datacoves_dbt`](/docs/reference/airflow/datacoves-decorators#taskdatacoves_dbt); leaving it empty causes a dbt profile validation error at runtime. ![Service Connection Connection Details](./assets/serviceconnection_editnew_details.png) diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md index 8dbc000..7f8b04f 100644 --- a/docs/reference/airflow/datacoves-decorators.md +++ b/docs/reference/airflow/datacoves-decorators.md @@ -60,6 +60,16 @@ Datacoves dbt decorator supports all the [Datacoves dbt Operator params](/docs/r With the `connection_id` mentioned above, we create a temporary dbt profile (it only exists at runtime inside the Airflow DAG's worker). By default, this dbt profile contains the selected Service Credential connection details. +:::warning +The service connection's **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`. Leaving it empty causes an error like: + +``` +Credentials in profile "my_dw_profile", target "dev" invalid: None is not of type 'string' +``` + +The Airflow Snowflake provider docs list `schema` as optional. That applies to the Airflow connection, not to the dbt profile generated from it, which requires `schema`. +::: + The dbt profile `name` is defined either in Project or Environment settings, in their `Profile name` field. This can be overwritten by passing a custom `DATACOVES__DBT_PROFILE` environment variable to the decorator. Users can also customize this dbt profile's connection details and/or target with the following params: From 972902990d0d58ac030d5a0c199fad81a5b206fe Mon Sep 17 00:00:00 2001 From: Fernando Mercado Date: Wed, 26 Aug 2026 09:28:22 -0500 Subject: [PATCH 2/7] Restructure dbt profile generation section into defaults/overrides table Makes the existing defaults (connection details, profile name, target name) and their overrides explicit and scannable instead of prose, per request to clarify what @task.datacoves_dbt does out of the box vs. what can be customized. --- .../reference/airflow/datacoves-decorators.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md index 7f8b04f..2c97d3e 100644 --- a/docs/reference/airflow/datacoves-decorators.md +++ b/docs/reference/airflow/datacoves-decorators.md @@ -58,7 +58,13 @@ Datacoves dbt decorator supports all the [Datacoves dbt Operator params](/docs/r **dbt profile generation:** -With the `connection_id` mentioned above, we create a temporary dbt profile (it only exists at runtime inside the Airflow DAG's worker). By default, this dbt profile contains the selected Service Credential connection details. +With the `connection_id` mentioned above, we create a temporary dbt profile (it only exists at runtime inside the Airflow DAG's worker). + +By default: + +- **Connection details** (user, password, warehouse, database, schema, role, etc.) come from the selected service connection (`connection_id`). +- **Profile name** comes from the **Profile name** field in Project or Environment settings. +- **Target name** is `default`. :::warning The service connection's **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`. Leaving it empty causes an error like: @@ -70,12 +76,13 @@ Credentials in profile "my_dw_profile", target "dev" invalid: None is not of typ The Airflow Snowflake provider docs list `schema` as optional. That applies to the Airflow connection, not to the dbt profile generated from it, which requires `schema`. ::: -The dbt profile `name` is defined either in Project or Environment settings, in their `Profile name` field. This can be overwritten by passing a custom `DATACOVES__DBT_PROFILE` environment variable to the decorator. - -Users can also customize this dbt profile's connection details and/or target with the following params: +These can be overridden with the following params: -- `overrides`: a dictionary with override parameters such as warehouse, role, database, etc. -- `target`: the target name this temporary dbt profile will receive. Defaults to `default`. +| What | How | +|---|---| +| Profile name | Pass a custom `DATACOVES__DBT_PROFILE` environment variable, e.g. `env={"DATACOVES__DBT_PROFILE": "prod"}` | +| Target name | `target`: the target name this temporary dbt profile will receive | +| Connection details | `overrides`: a dictionary with override parameters such as warehouse, role, database, etc. | Basic example: From 1d59d3328477f35a33505b2f5c5f977d27128137 Mon Sep 17 00:00:00 2001 From: fmercadop Date: Mon, 12 Oct 2026 09:26:48 -0500 Subject: [PATCH 3/7] Remove Schema-required note from service connections how-to Schema is already mandatory in the service connection UI, so the extra note about it being required for @task.datacoves_dbt is redundant. --- docs/how-tos/datacoves/how_to_service_connections.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/datacoves/how_to_service_connections.md b/docs/how-tos/datacoves/how_to_service_connections.md index 2a62e5a..b0b5d1a 100644 --- a/docs/how-tos/datacoves/how_to_service_connections.md +++ b/docs/how-tos/datacoves/how_to_service_connections.md @@ -26,7 +26,7 @@ A Service Connection consists of the following fields: - **Environment Variables** The legacy method Datacoves used which would inject the connection credentials as environment variables into Airflow. The name of the service connection will be used to dynamically create [environment variables](/docs/reference/airflow/environment-service-connection-vars) which we inject into Airflow. ![Delivery Mode](assets/service_connection_delivery.jpg). - **Connection Template** The connection template to base this service connection on(i.e. the defaults) - Depending on the template selected, additional fields will be displayed with the default values entered in the connection template. These default values can be overridden by toggling the indicator next to the given value. Enter the appropriate user, schema, and password. This is commonly a service account created specifically for Airflow and may differ between the development and production environment. **Schema is required** if the connection will be used with [`@task.datacoves_dbt`](/docs/reference/airflow/datacoves-decorators#taskdatacoves_dbt); leaving it empty causes a dbt profile validation error at runtime. + Depending on the template selected, additional fields will be displayed with the default values entered in the connection template. These default values can be overridden by toggling the indicator next to the given value. Enter the appropriate user, schema, and password. This is commonly a service account created specifically for Airflow and may differ between the development and production environment. ![Service Connection Connection Details](./assets/serviceconnection_editnew_details.png) From fcac82094424ca20d61499f3b4aff2fa052f5445 Mon Sep 17 00:00:00 2001 From: fmercadop Date: Mon, 12 Oct 2026 09:50:05 -0500 Subject: [PATCH 4/7] Simplify Schema warning wording in decorators reference --- docs/reference/airflow/datacoves-decorators.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md index 2c97d3e..46c2cd6 100644 --- a/docs/reference/airflow/datacoves-decorators.md +++ b/docs/reference/airflow/datacoves-decorators.md @@ -67,7 +67,7 @@ By default: - **Target name** is `default`. :::warning -The service connection's **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`. Leaving it empty causes an error like: +The **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`. Leaving it empty causes an error like: ``` Credentials in profile "my_dw_profile", target "dev" invalid: None is not of type 'string' From b0679f01c97b3ef2709e96dce5900c6c67f24754 Mon Sep 17 00:00:00 2001 From: fmercadop Date: Wed, 14 Oct 2026 08:36:38 -0500 Subject: [PATCH 5/7] Document schema override options for @task.datacoves_dbt --- .../reference/airflow/datacoves-decorators.md | 57 +++++++++++++++++-- 1 file changed, 52 insertions(+), 5 deletions(-) diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md index 46c2cd6..6b3b361 100644 --- a/docs/reference/airflow/datacoves-decorators.md +++ b/docs/reference/airflow/datacoves-decorators.md @@ -63,11 +63,11 @@ With the `connection_id` mentioned above, we create a temporary dbt profile (it By default: - **Connection details** (user, password, warehouse, database, schema, role, etc.) come from the selected service connection (`connection_id`). -- **Profile name** comes from the **Profile name** field in Project or Environment settings. +- **Profile name** comes from the **Profile name** field in Project or Environment settings. It must match the `profile:` value in your `dbt_project.yml`. - **Target name** is `default`. :::warning -The **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`. Leaving it empty causes an error like: +The **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`, either in the connection or via `overrides`. Leaving it empty causes an error like: ``` Credentials in profile "my_dw_profile", target "dev" invalid: None is not of type 'string' @@ -80,9 +80,8 @@ These can be overridden with the following params: | What | How | |---|---| -| Profile name | Pass a custom `DATACOVES__DBT_PROFILE` environment variable, e.g. `env={"DATACOVES__DBT_PROFILE": "prod"}` | | Target name | `target`: the target name this temporary dbt profile will receive | -| Connection details | `overrides`: a dictionary with override parameters such as warehouse, role, database, etc. | +| Connection details | `overrides`: a dictionary with override parameters such as warehouse, role, database, schema, etc. | Basic example: @@ -104,7 +103,6 @@ def my_dbt_dag(): @task.datacoves_dbt( connection_id="main", overrides={"warehouse": "my_custom_wh"}, - env={"DATACOVES__DBT_PROFILE": "prod"}, target="testing" ) def dbt_test() -> str: @@ -113,6 +111,55 @@ def my_dbt_dag(): dag = my_dbt_dag() ``` +#### Overriding the schema + +`overrides` and `target` support Airflow templating, so the schema can be set in different ways: + +| Option | `overrides` value | +|---|---| +| Fixed value | `{"schema": "my_schema"}` | +| Chosen when triggering the DAG | `{"schema": "{{ params.schema }}"}` | +| Per environment, from an Airflow Variable | `{"schema": "{{ var.value.dbt_schema }}"}` | +| From an environment variable, resolved by dbt | `{"schema": "{% raw %}{{ env_var('DBT_SCHEMA') }}{% endraw %}"}` | + +**Chosen when triggering the DAG:** define a default in the DAG's `params`. When the DAG runs on its schedule, it uses the default. When you trigger it manually from the Airflow UI, Airflow asks for the schema, pre-filled with the default, and you can change it before starting the run. + +```python +@dag( + params={"schema": "my_default_schema"}, + # ... +) +def my_dbt_dag(): + @task.datacoves_dbt( + connection_id="main", + overrides={"schema": "{{ params.schema }}"} + ) + def dbt_run() -> str: + return "dbt run" + +dag = my_dbt_dag() +``` + +**Per environment, from an Airflow Variable:** create an Airflow Variable (e.g. `dbt_schema`) in each environment with the schema to use there. + +**From an environment variable, resolved by dbt:** pass the variable with `env` and let dbt resolve it at runtime. + +```python +@task.datacoves_dbt( + connection_id="main", + overrides={"schema": "{% raw %}{{ env_var('DBT_SCHEMA') }}{% endraw %}"}, + env={"DBT_SCHEMA": "my_schema"}, + append_env=True +) +def dbt_run() -> str: + return "dbt run" +``` + +:::note +- `{% raw %}...{% endraw %}` is required. Without it, Airflow tries to render `env_var` itself and the task fails with `'env_var' is undefined`. +- `append_env=True` keeps the rest of the worker's environment variables. Without it, only the variables in `env` are available to the task. +::: + The examples above use the Airflow connection `main` which is added automatically from the Datacoves Service Connection. ![Service Connection](assets/service_connection_main.jpg) From 4243cb04ea391181d39a44a569aedd20952bccbf Mon Sep 17 00:00:00 2001 From: fmercadop Date: Wed, 14 Oct 2026 09:02:56 -0500 Subject: [PATCH 6/7] Generalize templated overrides section and reorder dbt profile generation --- .../reference/airflow/datacoves-decorators.md | 40 +++++++++---------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md index 6b3b361..a3dc1b6 100644 --- a/docs/reference/airflow/datacoves-decorators.md +++ b/docs/reference/airflow/datacoves-decorators.md @@ -56,7 +56,7 @@ Datacoves dbt decorator supports all the [Datacoves dbt Operator params](/docs/r - `connection_id`: This is the [service connection](/docs/how-tos/datacoves/how_to_service_connections) which is automatically added to airflow if you select `Airflow Connection` as the `Delivery Mode`. -**dbt profile generation:** +#### dbt profile generation With the `connection_id` mentioned above, we create a temporary dbt profile (it only exists at runtime inside the Airflow DAG's worker). @@ -66,23 +66,23 @@ By default: - **Profile name** comes from the **Profile name** field in Project or Environment settings. It must match the `profile:` value in your `dbt_project.yml`. - **Target name** is `default`. +These can be overridden with the following params: + +| What | How | +|---|---| +| Target name | `target`: the target name this temporary dbt profile will receive | +| Connection details | `overrides`: a dictionary with override parameters such as warehouse, role, database, schema, etc. | + :::warning -The **Schema** field must be populated when the connection will be used with `@task.datacoves_dbt`, either in the connection or via `overrides`. Leaving it empty causes an error like: +A schema must be set, either in the connection's **Schema** field or via `overrides`. Leaving it empty causes an error like: ``` -Credentials in profile "my_dw_profile", target "dev" invalid: None is not of type 'string' +Credentials in profile "my_dw_profile", target "default" invalid: None is not of type 'string' ``` The Airflow Snowflake provider docs list `schema` as optional. That applies to the Airflow connection, not to the dbt profile generated from it, which requires `schema`. ::: -These can be overridden with the following params: - -| What | How | -|---|---| -| Target name | `target`: the target name this temporary dbt profile will receive | -| Connection details | `overrides`: a dictionary with override parameters such as warehouse, role, database, schema, etc. | - Basic example: ```python @@ -111,18 +111,22 @@ def my_dbt_dag(): dag = my_dbt_dag() ``` -#### Overriding the schema +The examples above use the Airflow connection `main` which is added automatically from the Datacoves Service Connection. + +![Service Connection](assets/service_connection_main.jpg) -`overrides` and `target` support Airflow templating, so the schema can be set in different ways: +##### Setting values dynamically -| Option | `overrides` value | +`target` and `overrides` support Airflow templating, so their values can be fixed or set dynamically. The examples below use `schema`, but the same works for any key in `overrides` and for `target`: + +| Option | Example `overrides` value | |---|---| | Fixed value | `{"schema": "my_schema"}` | | Chosen when triggering the DAG | `{"schema": "{{ params.schema }}"}` | | Per environment, from an Airflow Variable | `{"schema": "{{ var.value.dbt_schema }}"}` | | From an environment variable, resolved by dbt | `{"schema": "{% raw %}{{ env_var('DBT_SCHEMA') }}{% endraw %}"}` | -**Chosen when triggering the DAG:** define a default in the DAG's `params`. When the DAG runs on its schedule, it uses the default. When you trigger it manually from the Airflow UI, Airflow asks for the schema, pre-filled with the default, and you can change it before starting the run. +**Chosen when triggering the DAG:** define a default in the DAG's `params`. When the DAG runs on its schedule, it uses the default. When you trigger it manually from the Airflow UI, Airflow asks for the value, pre-filled with the default, and you can change it before starting the run. ```python @dag( @@ -140,9 +144,9 @@ def my_dbt_dag(): dag = my_dbt_dag() ``` -**Per environment, from an Airflow Variable:** create an Airflow Variable (e.g. `dbt_schema`) in each environment with the schema to use there. +**Per environment, from an Airflow Variable:** create an Airflow Variable (e.g. `dbt_schema`) in each environment with the value to use there. -**From an environment variable, resolved by dbt:** pass the variable with `env` and let dbt resolve it at runtime. +**From an environment variable, resolved by dbt:** pass the variable with `env` and let dbt resolve it at runtime. This option only applies to `overrides`, since dbt resolves `env_var` inside the profile's connection details, not in the target name. ```python @task.datacoves_dbt( @@ -160,10 +164,6 @@ def dbt_run() -> str: - `append_env=True` keeps the rest of the worker's environment variables. Without it, only the variables in `env` are available to the task. ::: -The examples above use the Airflow connection `main` which is added automatically from the Datacoves Service Connection. - -![Service Connection](assets/service_connection_main.jpg) - #### Uploading and downloading dbt results From Datacoves 3.4 onwards, the `datacoves_dbt` decorator allows users to upload and download dbt execution results and metadata to our `dbt API`. From 37303f5d099ff5b76e2c1a0e274a979db9b5f9fc Mon Sep 17 00:00:00 2001 From: fmercadop Date: Wed, 14 Oct 2026 10:30:27 -0500 Subject: [PATCH 7/7] Clarify schema warning scope and overrides wording in decorators reference --- docs/reference/airflow/datacoves-decorators.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md index a3dc1b6..5ed74d4 100644 --- a/docs/reference/airflow/datacoves-decorators.md +++ b/docs/reference/airflow/datacoves-decorators.md @@ -71,10 +71,10 @@ These can be overridden with the following params: | What | How | |---|---| | Target name | `target`: the target name this temporary dbt profile will receive | -| Connection details | `overrides`: a dictionary with override parameters such as warehouse, role, database, schema, etc. | +| Connection details | `overrides`: a dictionary of dbt profile fields to override, such as warehouse, role, database, schema, etc. | :::warning -A schema must be set, either in the connection's **Schema** field or via `overrides`. Leaving it empty causes an error like: +For Snowflake and Databricks connections, a schema must be set, either in the connection's **Schema** field or via `overrides`. Leaving it empty causes an error like: ``` Credentials in profile "my_dw_profile", target "default" invalid: None is not of type 'string' @@ -117,7 +117,7 @@ The examples above use the Airflow connection `main` which is added automaticall ##### Setting values dynamically -`target` and `overrides` support Airflow templating, so their values can be fixed or set dynamically. The examples below use `schema`, but the same works for any key in `overrides` and for `target`: +`target` and `overrides` support Airflow templating, so their values can be fixed or set dynamically. The examples below use `schema`, but the same works for any key in `overrides` and for `target` (except the last option): | Option | Example `overrides` value | |---|---| @@ -146,7 +146,7 @@ dag = my_dbt_dag() **Per environment, from an Airflow Variable:** create an Airflow Variable (e.g. `dbt_schema`) in each environment with the value to use there. -**From an environment variable, resolved by dbt:** pass the variable with `env` and let dbt resolve it at runtime. This option only applies to `overrides`, since dbt resolves `env_var` inside the profile's connection details, not in the target name. +**From an environment variable, resolved by dbt:** pass the environment variable with `env` and let dbt resolve it at runtime. This option only applies to `overrides`, since dbt resolves `env_var` inside the profile's connection details, not in the target name. ```python @task.datacoves_dbt(