Skip to content

FIX: read initializer scripts as UTF-8 in add-initializer - #2669

Open
MinYi Xie (ppcvote) wants to merge 1 commit into
microsoft:mainfrom
ppcvote:fix/add-initializer-utf8
Open

MinYi Xie (ppcvote) wants to merge 1 commit into
microsoft:mainfrom
ppcvote:fix/add-initializer-utf8

Conversation

@ppcvote

Copy link
Copy Markdown
Contributor

Description

add-initializer reads the local script without an encoding, so on Windows it is decoded with the ANSI code page instead of UTF-8: pyrit/cli/pyrit_scan.py:822 (aiofiles.open(script_path)) and pyrit/cli/pyrit_shell.py:399 (script_path.read_text()), both at b0dba3e. Under -X warn_default_encoding both reads report 'encoding' argument not specified.

Any initializer holding a non-ASCII character is affected wherever the OS default text encoding isn't UTF-8, which includes Windows. On a cp950 machine (Python 3.11.6), add-initializer on a Spanish initializer prints Registered initializer 'spanish_init' and returns 0, while the text passed to register_initializer_async is "Eres un asistente 繳til..." instead of "Eres un asistente útil...". Feeding that text to CustomInitializerStorage.save_script and reading it back keeps the damage, so the prompt the initializer sets changes while the CLI reports success. A file with Japanese text fails instead, with 'cp950' codec can't decode byte 0xe6 in position 76: illegal multibyte sequence, and so does a renamed copy of PyRIT's own pyrit/setup/initializers/targets.py, which contains an em dash. Read as cp1252, the same bytes give café, año, niño with no error at all.

Python source is UTF-8 by default (PEP 3120), and the rest of this path already agrees: pyrit/cli/_config_reader.py:68, pyrit/registry/custom_initializer_storage.py:70,80 and pyrit/registry/components/initializer_registry.py:485 all pass encoding="utf-8". The fix passes encoding="utf-8" at both reads.

Tests and Documentation

  • New test_handle_add_initializer_reads_source_as_utf8 (scan) and test_success_path_reads_source_as_utf8 (shell) register a UTF-8 file with accented Latin and Japanese text and assert the uploaded content matches the file.
  • New test_reads_script_with_explicit_utf8_encoding and the existing test_handle_add_initializer_reads_with_aiofiles assert the encoding argument, so they fail on any OS. That existing test pinned open(path) and is updated to open(path, encoding="utf-8").
  • The current tests miss this: test_main_add_initializer_success writes ASCII content, and test_handle_add_initializer_reads_with_aiofiles mocks aiofiles.open.
  • Reverting only the source change fails all four tests on this machine; with it tests/unit/cli is 242 passed. ruff 0.16.6 check and format --check pass on the changed files, and ty 0.0.78 passes on the two modules.
  • No doc change: no encoding is documented for initializer scripts. I did not open an issue first, happy to file one if you prefer.

pyrit_scan and pyrit_shell read the initializer script without an encoding, so
on Windows it is decoded with the ANSI code page. A non-ASCII initializer is
either silently transcoded (cp1252) or fails with UnicodeDecodeError
(cp932/936/949/950), while the server stores and loads the same scripts as
UTF-8.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant