bulkcopy() and bulkcopy_arrow() report 2x rows_copied and batch_count against Fabric Warehouse #711

Describe the bug

When the target is a Fabric Warehouse (SERVERPROPERTY('EngineEdition') = 11), both cursor.bulkcopy() and cursor.bulkcopy_arrow() return statistics that are exactly double the truth.

The data written is always correct. Only the returned statistics are wrong:

No exception is raised. Callers that trust rows_copied for reconciliation, logging, or progress reporting get a silently wrong number.

The doubled value also reaches the driver log:

INFO, cursor.rs:551, py-core, bulkcopy_arrow: Starting Arrow bulkcopy to table: dbo.logprobe
INFO, cursor.py:3657, Python, bulkcopy_arrow: completed - rows_copied=6000, batch_count=6, elapsed_time=2.3745249

That run copied 3,000 rows in 3 batches.

To reproduce

Needs pyarrow and azure-identity, and an az login session with write access to a Fabric Warehouse.

import pyarrow as pa
from azure.identity import AzureCliCredential
from mssql_python import connect

TABLE = "dbo.minrepro"
CONN_STR = (
    "Server=WORKSPACE_ID.datawarehouse.fabric.microsoft.com,1433;"
    "Database=WAREHOUSE_NAME;Encrypt=yes;TrustServerCertificate=no"
)

conn = connect(CONN_STR, token_provider=AzureCliCredential(process_timeout=60))
cur = conn.cursor()

cur.execute(f"IF OBJECT_ID('{TABLE}') IS NOT NULL DROP TABLE {TABLE}")
cur.execute(f"CREATE TABLE {TABLE} (id BIGINT)")
conn.commit()

result = cur.bulkcopy_arrow(TABLE, pa.table({"id": pa.array([1], pa.int64())}), timeout=600)
conn.commit()

cur.execute(f"SELECT COUNT(*) FROM {TABLE}")
print("reported:", result["rows_copied"], " actual:", cur.fetchone()[0])

Output:

reported: 2  actual: 1

Expected behavior

rows_copied equals the number of rows actually inserted, and batch_count equals the number of batches actually sent, on every engine.

Scope of the defect

Reproduced with one row, so no large dataset is needed. The 2x factor is stable and invariant to:

• row count, from 1 to 200,000
• API, bulkcopy and bulkcopy_arrow behave identically
• source type, pyarrow.Table, RecordBatchReader, DuckDBPyRelation, and list of tuples
• batch_size, including 0
• table_lock, keep_nulls, use_internal_transaction

Engine matrix, 5,000 rows, both APIs, batch_size 0 and 1000:

Sample from the Warehouse run:

Why this looks like a client-side bug rather than a server one

bcp was pointed at the same warehouse and the same table shape. It performs a TDS bulk load through the ODBC driver, so it exercises the same protocol path the Rust core does:

bcp dbo.bcpprobe in bcp_in.txt -S WORKSPACE_ID.datawarehouse.fabric.microsoft.com,1433 -d WAREHOUSE_NAME -c -G -U USER_UPN

Starting copy...
3 rows copied.

Actual row count afterwards was 3. So the correct count is derivable from what Warehouse sends back, and ODBC derives it correctly.

The Python layer is not involved. Cursor.bulkcopy and Cursor.bulkcopy_arrow in mssql_python/cursor.py return the dict from pycore_cursor.bulkcopy(...) unmodified, so this points at row-count accumulation in mssql_py_core.

Untested hypothesis

Warehouse is a distributed engine. bcp prints a Distributed request ID warning on the same load, which suggests the load is acknowledged by more than one layer. A factor of exactly 2 that is stable from 1 row to 200,000 rows would fit the core summing every row-count acknowledgement it sees rather than taking the final one. That would also explain batch_count doubling in lockstep.

I could not confirm this. RUST_LOG does not reach the core's log bridge, and TDS is encrypted, so I have no token-level trace. Worth checking how DONE token row counts are accumulated.

One detail that does not fit a naive "everything counted twice" model: with batch_size=0, rows_copied still doubles but batch_count reports 1 rather than 2, which suggests batch_count is derived rather than counted on the unbatched path.

Further technical details

Python version: 3.14.0 (amd64)
mssql-python: 1.13.0
mssql_py_core: 0.1.8
mssql-python-odbc: 18.6.2.1
SQL Server version: Fabric Warehouse, Microsoft Azure SQL Data Warehouse (RTM) - 12.0.2000.8, EngineEdition 11, collation Latin1_General_100_BIN2_UTF8
Operating system: Windows 11 ARM64
bcp control: 15.0.4298.1, ODBC Driver 17 for SQL Server

Additional context

Not a duplicate of #623, which was an authentication failure against Fabric Warehouse and is closed. Bulk copy connects and loads correctly here, only the returned statistics are wrong.