FIX: read initializer scripts as UTF-8 in add-initializer - #2669
Open
MinYi Xie (ppcvote) wants to merge 1 commit into
Open
MinYi Xie (ppcvote) wants to merge 1 commit into
MinYi Xie (ppcvote) wants to merge 1 commit into
Conversation
pyrit_scan and pyrit_shell read the initializer script without an encoding, so on Windows it is decoded with the ANSI code page. A non-ASCII initializer is either silently transcoded (cp1252) or fails with UnicodeDecodeError (cp932/936/949/950), while the server stores and loads the same scripts as UTF-8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
add-initializerreads the local script without an encoding, so on Windows it is decoded with the ANSI code page instead of UTF-8:pyrit/cli/pyrit_scan.py:822(aiofiles.open(script_path)) andpyrit/cli/pyrit_shell.py:399(script_path.read_text()), both at b0dba3e. Under-X warn_default_encodingboth reads report'encoding' argument not specified.Any initializer holding a non-ASCII character is affected wherever the OS default text encoding isn't UTF-8, which includes Windows. On a cp950 machine (Python 3.11.6),
add-initializeron a Spanish initializer printsRegistered initializer 'spanish_init'and returns 0, while the text passed toregister_initializer_asyncis"Eres un asistente 繳til..."instead of"Eres un asistente útil...". Feeding that text toCustomInitializerStorage.save_scriptand reading it back keeps the damage, so the prompt the initializer sets changes while the CLI reports success. A file with Japanese text fails instead, with'cp950' codec can't decode byte 0xe6 in position 76: illegal multibyte sequence, and so does a renamed copy of PyRIT's ownpyrit/setup/initializers/targets.py, which contains an em dash. Read as cp1252, the same bytes givecafé, año, niñowith no error at all.Python source is UTF-8 by default (PEP 3120), and the rest of this path already agrees:
pyrit/cli/_config_reader.py:68,pyrit/registry/custom_initializer_storage.py:70,80andpyrit/registry/components/initializer_registry.py:485all passencoding="utf-8". The fix passesencoding="utf-8"at both reads.Tests and Documentation
test_handle_add_initializer_reads_source_as_utf8(scan) andtest_success_path_reads_source_as_utf8(shell) register a UTF-8 file with accented Latin and Japanese text and assert the uploaded content matches the file.test_reads_script_with_explicit_utf8_encodingand the existingtest_handle_add_initializer_reads_with_aiofilesassert theencodingargument, so they fail on any OS. That existing test pinnedopen(path)and is updated toopen(path, encoding="utf-8").test_main_add_initializer_successwrites ASCII content, andtest_handle_add_initializer_reads_with_aiofilesmocksaiofiles.open.tests/unit/cliis 242 passed. ruff 0.16.6checkandformat --checkpass on the changed files, and ty 0.0.78 passes on the two modules.