[DCV-3942] Document DAG parse times via the Airflow REST API

docs/how-tos/airflow/dags/create-dag-level-docs.md

@@ -2,7 +2,7 @@
 title: Add Documentation to Airflow DAGs in Datacoves
 sidebar_label: Add Documentation
 description: "How to add Markdown documentation directly to Airflow DAGs in Datacoves so operators can read context in the Airflow UI and Grid view."
-sidebar_position: 20
+sidebar_position: 10
 ---
 # How to Add Docs at the DAG Level in Airflow
 

docs/how-tos/airflow/dags/dag-parse-times.md

@@ -0,0 +1,112 @@
+---
+title: "Find Slow-Parsing DAGs with the REST API"
+sidebar_label: "DAG parsing times"
+description: "Use the Airflow REST API to list DAG parse durations and find slow-parsing DAGs in Team Airflow and My Airflow on Datacoves."
+sidebar_position: 40
+---
+# Find slow-parsing DAGs with the REST API
+
+Every DAG file is re-parsed regularly by the dag-processor. A DAG that is slow to parse (top-level code doing imports, API calls, or file reads) delays scheduling for every DAG in the environment, so parse times are one of the first things to check when DAGs randomly disappear from the UI.
+
+On Airflow 3 environments, the dag-processor records each DAG's most recent parse duration and the REST API exposes it, so you can get the real parsing-time report with a simple API call: no access to the dag-processor pod is needed.
+
+:::info
+This guide targets **Team Airflow** on Airflow 3 environments. For **My Airflow**, use the [`datacoves my parse-logs`](/docs/reference/airflow/datacoves-commands#datacoves-my-parse-logs) command instead, which shows the same report plus the parse log of individual DAG files.
+
+Airflow 2 does not expose parse durations through its API; see [the note below](#airflow-2-environments).
+:::
+
+## What you need
+
+1. An Airflow API token for the target environment: in the launchpad, go to **Environments**, open your environment's **Keys**, and click **Generate New Airflow Token**. Copy the token immediately -- it is only shown once.
+2. The **Airflow API URL** shown on that same page (`https://airflow-<env>.<your-domain>/api/v2/`).
+3. Store both in a `.env` file (and add it to `.gitignore`):
+
+```env
+AIRFLOW_API_URL = "https://airflow-your-env.your-domain.com/api/v2/"
+AIRFLOW_API_TOKEN = "your-token-here"
+```
+
+The script works the same from CI, from Datacoves VSCode, or from any environment you use to reach Datacoves -- the Airflow API endpoints are reachable from all of them.
+
+## Get the parsing-time report
+
+The `GET /dags` endpoint returns `last_parse_duration` (seconds) and `last_parsed_time` for every DAG. The script below fetches all DAGs (paginated) and prints them slowest-first, the same shape as the `datacoves my parse-logs` report:
+
+```python
+# dag_parse_times.py
+import os
+
+import requests
+from dotenv import load_dotenv
+
+load_dotenv()
+API_URL = os.getenv("AIRFLOW_API_URL").rstrip("/")
+HEADERS = {"Authorization": f"Bearer {os.getenv('AIRFLOW_API_TOKEN')}"}
+
+
+def fetch_all_dags():
+    dags, offset = [], 0
+    while True:
+        response = requests.get(
+            f"{API_URL}/dags",
+            headers=HEADERS,
+            params={"limit": 100, "offset": offset},
+        )
+        response.raise_for_status()
+        payload = response.json()
+        dags.extend(payload["dags"])
+        offset += 100
+        if offset >= payload["total_entries"]:
+            return dags
+
+
+def print_parse_report(dags):
+    dags.sort(key=lambda d: d.get("last_parse_duration") or 0, reverse=True)
+    print(f"{'DAG':<50} {'Parse time':>12} {'Last parsed'}")
+    for dag in dags:
+        duration = dag.get("last_parse_duration")
+        duration = f"{duration:.3f}s" if duration is not None else "-"
+        parsed_at = dag.get("last_parsed_time") or "-"
+        print(f"{dag['dag_id']:<50} {duration:>12} {parsed_at}")
+
+
+if __name__ == "__main__":
+    print_parse_report(fetch_all_dags())
+```
+
+Example output:
+
+```
+DAG                                                  Parse time Last parsed
+daily_loan_run                                           2.153s 2026-08-18T14:02:11.480217Z
+variables_python_script                                  0.310s 2026-08-18T14:02:10.912304Z
+simple_bash_dag                                          0.052s 2026-08-18T14:02:10.598721Z
+```
+
+A parse time consistently above ~1 second is worth investigating: the usual cause is work done at the top level of the DAG file (imports of heavy libraries, `Variable.get` outside a task, reading files or calling APIs during parsing). See [Airflow's best practices on top-level code](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code) for how to restructure it.
+
+:::tip
+`last_parse_duration` reflects the most recent parse by the environment's dag-processor -- the number that actually matters for scheduling latency. Running `airflow dags report` inside a DAG task measures a worker's copy of the files under different conditions and can be misleading.
+:::
+
+## Alternative: authenticate as your own user
+
+The Airflow API token above belongs to the environment's service account and is a JWT with an expiration date, so long-lived automation would need re-generating it. As an alternative, you can authenticate with your personal Datacoves API key (generated under [User Settings > My Airflow API](/docs/how-tos/my_airflow/use-my-airflow-api) -- despite the name, it is a general Datacoves key): exchange it for a short-lived Airflow JWT at `POST /auth/token` using the special `__datacoves_token__` username. This authenticates you as **your own user** with your permissions, the key does not expire, and the same key works against **any environment you have access to**:
+
+```python
+def get_jwt(base_url, datacoves_api_key):
+    """Exchange a Datacoves API key for a short-lived Airflow JWT."""
+    response = requests.post(
+        f"{base_url}/auth/token",
+        json={"username": "__datacoves_token__", "password": datacoves_api_key},
+    )
+    response.raise_for_status()
+    return response.json()["access_token"]
+```
+
+Use the returned token in the same `Authorization: Bearer` header as above (`base_url` is the Airflow URL without `/api/v2`).
+
+## Airflow 2 environments
+
+Airflow 2 records when each DAG was last parsed (`last_parsed_time`) but not how long it took, and its REST API does not expose parse durations. This is a known limitation; parse times there only exist in the dag-processor manager logs. Since new Datacoves environments run Airflow 3, no Airflow 2-specific tooling is provided -- upgrading is the path to this report.

docs/how-tos/airflow/dags/dynamically-set-schedule.md

@@ -2,7 +2,7 @@
 title: Dynamically Set Airflow Schedule Intervals
 sidebar_label: Dynamic Schedule
 description: "Use environment variables or Airflow Variables to dynamically configure DAG schedule intervals in Datacoves for flexible pipeline orchestration."
-sidebar_position: 22
+sidebar_position: 30
 ---
 # How to Dynamically set the schedule Interval
 

docs/how-tos/airflow/dags/external-python-dag.md

@@ -2,7 +2,7 @@
 title: Call External Python Scripts from Airflow DAGs
 sidebar_label: External Python Scripts
 description: "Use DatacovesBashOperator to run external Python scripts from Airflow DAGs in Datacoves, with custom packages pre-installed in the worker environment."
-sidebar_position: 21
+sidebar_position: 20
 ---
 # External Python DAG 
 

docs/how-tos/airflow/dags/generate-dags-from-yml.md

@@ -2,7 +2,7 @@
 title: Generate Airflow DAGs from YAML with dbt-coves
 sidebar_label: Generate from YAML
 description: "Use dbt-coves to auto-generate Airflow Python DAG files from simple YAML definitions, reducing boilerplate and standardizing pipeline creation."
-sidebar_position: 23
+sidebar_position: 50
 ---
 # Generate DAGs from yml
 

docs/how-tos/airflow/dags/get-current-branch-name.md

@@ -2,7 +2,7 @@
 title: Get the Git Branch Name from an Airflow DAG Task
 sidebar_label: Get Git Branch Name
 description: "Retrieve the current git branch name from within an Airflow DAG task in Datacoves for branch-specific logic or conditional pipeline behavior."
-sidebar_position: 24
+sidebar_position: 60
 ---
 # Retrieving the Current Branch Name in a Git Repository
 

docs/how-tos/airflow/dags/retry-dbt-tasks.md

@@ -2,7 +2,7 @@
 title: Retry Failed dbt Tasks in Airflow DAGs
 sidebar_label: Retry dbt Tasks
 description: "Configure automatic retries for failed dbt tasks in Airflow DAGs in Datacoves, including retry count, delay settings, and email-on-retry behavior."
-sidebar_position: 29
+sidebar_position: 80
 ---
 # Retry a dbt task
 

docs/how-tos/airflow/dags/run-adf-pipeline.md

@@ -2,7 +2,7 @@
 title: Trigger ADF Pipelines from Airflow in Datacoves
 sidebar_label: Run ADF Pipeline
 description: "Use Azure Data Factory operators in Airflow DAGs within Datacoves to trigger and monitor ADF pipelines as part of your data orchestration workflow."
-sidebar_position: 26
+sidebar_position: 120
 ---
 # Use Microsoft Azure Data Factory Operators 
 

docs/how-tos/airflow/dags/run-airbyte-sync-jobs.md

@@ -2,7 +2,7 @@
 title: Trigger Airbyte Syncs from Airflow in Datacoves
 sidebar_label: Run Airbyte Syncs
 description: "Orchestrate Airbyte data syncs from Airflow DAGs in Datacoves using the Airbyte provider, with examples for triggering connections and monitoring status."
-sidebar_position: 27
+sidebar_position: 130
 ---
 # Run Airbyte sync jobs
 

docs/how-tos/airflow/dags/run-databricks-notebook.md

@@ -2,7 +2,7 @@
 title: Run Databricks Notebooks from Airflow DAGs
 sidebar_label: Run Databricks Notebook
 description: "Trigger and monitor Databricks Notebook runs from Airflow DAGs in Datacoves using the Databricks provider operator with cluster configuration."
-sidebar_position: 30
+sidebar_position: 90
 ---
 # Run Databricks Notebooks 
 

docs/how-tos/airflow/dags/run-dbt.md

@@ -2,7 +2,7 @@
 title: "Run dbt from Airflow in Datacoves: DAG Examples"
 sidebar_label: Run dbt
 description: "Use the Datacoves dbt decorator or operator to run dbt commands from Airflow tasks, with full examples for dbt run, dbt test, and dbt build."
-sidebar_position: 28
+sidebar_position: 100
 ---
 # How to run dbt from an Airflow worker
 

docs/how-tos/airflow/dags/run-fivetran-sync-jobs.md

@@ -2,7 +2,7 @@
 title: Trigger Fivetran Syncs from Airflow in Datacoves
 sidebar_label: Run Fivetran Syncs
 description: "Orchestrate Fivetran connector syncs from Airflow DAGs in Datacoves using the Fivetran provider, with connection ID configuration and monitoring."
-sidebar_position: 31
+sidebar_position: 140
 ---
 # Run Fivetran sync jobs
 

docs/how-tos/airflow/dags/s3-to-snowflake.md

@@ -2,7 +2,7 @@
 title: Load S3 Files into Snowflake with Airflow
 sidebar_label: S3 to Snowflake
 description: "Use Airflow in Datacoves to load files from Amazon S3 into Snowflake using the S3ToSnowflakeOperator, with staging area and copy options."
-sidebar_position: 25
+sidebar_position: 70
 ---
 # Loading S3 Files into Snowflake 
 

docs/how-tos/airflow/dags/test-dags.md

@@ -2,7 +2,7 @@
 title: Test Airflow DAGs in Your CI/CD Pipeline
 sidebar_label: Test DAGs
 description: "How to run automated Airflow DAG tests in your CI/CD pipeline using pytest and the Datacoves CLI to catch import errors before deploying to production."
-sidebar_position: 32
+sidebar_position: 110
 ---
 # Run DAG tests in your CI/CD 
 

docs/how-tos/airflow/dags/use-variables-and-connections.md

@@ -2,7 +2,7 @@
 title: Use Airflow Variables and Connections in DAGs
 sidebar_label: Variables & Connections
 description: "How to access and use Airflow Variables and Connections in Datacoves DAGs, with best practices for avoiding performance issues during DAG parsing."
-sidebar_position: 33
+sidebar_position: 150
 ---
 # Variables and Connections