Skip to content

Fix COPY to STDOUT/from STDIN leaving the connection stuck - #1636

Open
anandghegde wants to merge 2 commits into
dbcli:mainfrom
anandghegde:fix-copy-stdout-transaction
Open

anandghegde wants to merge 2 commits into
dbcli:mainfrom
anandghegde:fix-copy-stdout-transaction

Conversation

@anandghegde

Copy link
Copy Markdown

Description

Running copy ... to stdout or copy ... from stdin (instead of \copy) broke the session. psycopg's cursor.execute() raises "COPY cannot be used with this method; use copy() instead", but only after the server has already started the COPY. The connection stays in the COPY state, so:

  • every later query fails with "sending query failed: another command is already in progress"
  • the transaction status stays ACTIVE, so quit asks about an ongoing transaction, and c/r fail for the same reason

Fix: in execute_normal_sql, when execute() raises a ProgrammingError and the connection is still ACTIVE, end the COPY on the connection. A COPY FROM STDIN is aborted with put_copy_end, and the rows of a COPY TO STDOUT are read and thrown away. Then the error is re-raised with a message that points to \copy:

postgres@localhost:postgres> copy (select 1) to stdout csv;
COPY to STDOUT or from STDIN is not supported, use \copy instead
Time: 0.004s
postgres@localhost:postgres> select 42;
+----------+
| ?column? |
|----------|
| 42       |
+----------+
SELECT 1

If the COPY runs inside BEGIN, a COPY TO STDOUT leaves the transaction open and usable. An aborted COPY FROM STDIN puts it in the failed state, the same as any other error.

How I tested it (Postgres 17 in Docker, psycopg 3.3.5):

  • New dbtest tests in tests/test_pgexecute.py cover COPY TO STDOUT (small and 10k rows), COPY FROM STDIN, and COPY TO STDOUT inside an open transaction. They fail on main and pass with this change.
  • pytest tests: 2783 passed, 1 xfailed, 1 xpassed.
  • behave tests/features: 33 scenarios passed, 0 failed.
  • ruff check and ruff format --diff are clean. docutils --halt=warning changelog.rst is clean.
  • Ran pgcli itself with pexpect. On main, copy (select 1) to stdout csv; followed by quit hits the "A transaction is ongoing" prompt, and any query in between fails. With this change, later queries work, copy ... from stdin and copy inside begin behave the same way, \copy still works, and quit exits right away.

Fixes #1505

Checklist

  • I've added this contribution to the changelog.rst.
  • I've added my name to the AUTHORS file (or it's already there).
  • I installed pre-commit hooks (pip install pre-commit && pre-commit install).
  • I verified that my changes work as expected (this may include manually testing them in your local environment, or in other available environments). Cross this out if not relevant (for example, if you're making a documentation change).
  • Please squash merge this pull request (uncheck if you'd like us to merge as multiple commits)

Running COPY ... TO STDOUT or COPY ... FROM STDIN (instead of \copy) made
psycopg raise "COPY cannot be used with this method" after the server had
already started the COPY. The connection stayed in the COPY state, so every
later query failed with "another command is already in progress" and
quitting asked about an ongoing transaction.

End the COPY on the connection (abort COPY FROM STDIN, drain COPY TO STDOUT)
and report an error that suggests \copy instead.

Fixes dbcli#1505
Comment thread pgcli/pgexecute.py
cur.execute(split_sql)
try:
cur.execute(split_sql)
except psycopg.ProgrammingError as e:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can psycopg.ProgrammingError be raised in situations other than copy, and if so, how do we handle that?

@anandghegde anandghegde Sep 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and those pass through unchanged. psycopg raises ProgrammingError in two broad cases here:

  • Server errors with SQLSTATE class 42, which psycopg maps to subclasses such as SyntaxError and UndefinedTable. By the time these are raised the server has already ended the statement, so the transaction status is IDLE or INERROR, never ACTIVE.
  • Client-side misuse (parameter count mismatches and similar). execute_normal_sql passes no parameters, and these are raised before anything is sent, so the status is not ACTIVE either.

The only result that leaves the connection ACTIVE is a COPY_IN/COPY_OUT status (_raise_for_result in psycopg's _cursor_base.py). That is why the handler checks transaction_status rather than the message, and re-raises the original exception otherwise.

To pin this down I added test_other_programming_errors_are_not_rewritten in ec70d50: a syntax error and an undefined table still raise their own exception classes without the \copy message, and the connection stays usable. With the ACTIVE check removed, both cases fail. The full suite passes against Postgres 16 (2785 passed).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MA8JGs7pCoXFL2W6jEJ8sD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use an incorrect copy command syntax copy instead of \copy and cannot quit app.

2 participants