[copybara-service]

Add the verbatim and select features of the Python protocol, so code is not
escape-decoded.

PROBLEM
With protocol='python' a code field (shell script, Python program, file body)
is an inner Python string literal, and structure_from_python parses the
response with exec. Every \n, \" or \r the LM did not double-escape is
decoded before the code runs: print("\n=== x ===") reaches the shell with a
real line break and fails with SyntaxError: unterminated string literal. The
outer literal still parses, so autofix never sees the damage.

FIX
Two switches, both required: a str field opts in with
dict(verbatim=True), and a query opts in with protocol='python+verbatim'.
The agents also ask for select: protocol='python+verbatim+select', recorded
as python:2.0+verbatim:1.0+select:1.0. The marked fields then travel in
out-of-band blocks after the ```python fence. A block is copied byte for byte
and never unescaped:

output = NextStep(next_step=BashCode(bash_code=BASH_CODE))

<<<LF:BLOCK BASH_CODE 7f3a>>>
python3 -c 'print("a\nb")'
<<<LF:END BASH_CODE 7f3a>>>

• verbatim:1.0 supports the base python:2.0 only, whose example its prompt
  replaces. python:1.0+verbatim and json+verbatim raise ValueError. It
  does not imply select:1.0. Both are the default versions of their
  features, so python+verbatim+select keeps its meaning when a later version
  of either is added.
• For a schema with a verbatim field, the prompt replaces the 1 + 1 example
  of python:2.0 with block instructions and a worked example, and adds a hint
  comment above each verbatim field. A schema without one gets exactly the
  python:2.0 prompt and parse (plus select, if the name asks for it).
• select:1.0 adds no prompt text: it parses the python fence that builds the output object when the LM also echoes the class definitions. A line
  inside a string literal of the code does not end a fence, and if the fence
  that builds the object does not parse, the whole response is parsed as
  without select.
• Block names are bound as globals of the existing exec, so nothing
  downstream changes. Inline string literals still parse. A block never
  shadows a class of the schema, a name bound for every response (UNKNOWN,
  ...) or a builtin; a field named like one of them, e.g. db when the
  schema has a class DB anywhere, gets the block name VERBATIM_<NAME> in
  the hint and in few-shot examples.
• The checks after exec also cover outputs that are plain lists or dicts.
• When verbatim applies, lf.parse adds the block instructions to its
  preamble. A caller's preamble= still replaces the default preamble of
  lf.query or lf.parse, and gets the block instructions appended.
• When the features of a query's protocol that apply to its schema are not
  empty, its autofix correction query uses the protocol with just those
  features: under python+verbatim+select,
  python:2.0+verbatim:1.0+select:1.0 for a schema with a verbatim field,
  where CorrectedCode.corrected_code is a verbatim field, so a correction
  travels as a block as well, and python:2.0+select:1.0 for a schema without
  one. A correction reply that
  misuses the blocks counts as a failed attempt. Correction queries of other
  protocols are unchanged.
• Only VerbatimBlockError is exported from lf.structured.
• PythonPromptingProtocol runs the features by phase: cut the blocks, pick
  the fence, check the block references, exec, check the value. Within a
  phase, features run by rank: verbatim (1) cuts the blocks before select
  (2) picks the fence. The feature hooks are internal.

PARSE RULES of verbatim:1.0 (frozen; a change needs a new version)

• A block's text is the bytes strictly between the newline that ends the
  opening fence and the newline that starts the closing fence. If the opening
  fence ends with a CRLF, that newline is a CRLF too.
• Fence variants LMs emit are accepted: the keyword fused into the name, a
  closing fence without a name, indented fences, a quoted block name, and fence
  lines that end with \r. Any line that starts with <<<LF:END or <<LF:END
  is a closing fence, whatever follows its > (<<<LF:END NAME token>>,
  <<LF:END NAME token>>>, <<<LF:END NAME token>>>.); the text after END,
  up to a >, may name the block, repeat the token, both, or neither. One that
  names another block, or no block and another token (<<<LF:END OTHER t1>>>,
  <<<LF:END t2>>>, <<<LF:ENDPOINT x>>>), does not end the block; one that
  names the block with another token (<<<LF:END BASH_CODE t2>>>) counts as a
  closing fence without the token.
• A line inside a string literal of the Python code is never a fence, so an
  inline literal that quotes block syntax parses as under python:2.0. The
  code is in the , py and ```python fences, or, in a response without a
  fence, everywhere outside the blocks. A literal that is never closed does
  not count.
• The closing-fence search stops at the next opener that carries the block's
  token, whatever its name (without a token: the next opener without one),
  since every block of a response carries the same token. An opener with
  another token is payload, so a block can quote fences of another token;
  past one of its own name, only a closing fence that repeats the token ends
  the block.
• Before that point, the block ends at the first closing fence with its token
  (that names it or nothing) or its closing fence (same name and token) fused
  to the end of a line, keeping the text before the fence; only if there is
  none, at the first closing fence without the token. Closing fences without
  the token before that one are payload. The parser never guesses: it is
  unclear where the block ends if another closing fence, with or without the
  token, follows that one with text between them (a payload that quotes its
  own closing fence looks like prose that repeats it; between two closing
  fences with the token, one without it is text too), or follows it at all
  when that one is fused to a line of text, or if that one is fused to a line
  of text after a closing fence without the token that names the block, or if
  the block's only closing fences name another block or token; a later closing
  fence that names another block or token does not count. A used unclear block
  raises VerbatimBlockError; an unused one is ignored. Closing fences with
  only blank lines between them are a stutter: the first one ends the block.
• An indented opening fence: its indentation is removed only if every
  non-blank body line starts with it; otherwise the body is kept as written.
• A block with no closing fence runs to the next opening fence. A used block
  that the response ends inside raises VerbatimBlockError.
• A block whose own closing fence comes only after the next opener with its
  token quotes that block: its text would end at that opener, so a used one
  raises VerbatimBlockError; an unused one is ignored.
• A name defined twice with different text, a referenced block that is not
  defined, a reference to a block named like a schema class, constant or
  builtin, and misnamed blocks that match neither 1:1 nor by ordinal or name
  stem raise VerbatimBlockError before exec, so autofix is never asked to
  invent a payload. A lone unused block is not renamed to a lone undefined
  reference if its name agrees with a block that was already used, or only
  shares that block's name stem: the blocks are numbered off by one
  (references CODE_1, CODE_2; blocks CODE_2, CODE_3).
• After an autofix rewrite, every verbatim value must be the text of a block,
  a string literal of the response, a + chain of those that the response
  itself writes (evaluated with the blocks' texts), or the field's default.
  Other expressions, f-strings included, are not evaluated.
• A string literal that the executed code passes for a verbatim field and that
  names a block of that field (named after the field, or claimed by no other
  field) is replaced by the block's text; text that came from a block is never
  expanded. A quoted value that equals a block name derived from its field
  (BASH_CODE or BASH_CODE_2 for bash_code) raises VerbatimBlockError if
  no block has that name; other literals are kept.
• An inline string literal in a verbatim field is accepted, unless the
  response also defines blocks named after the field (BASH_CODE,
  BASH_CODE_2 for bash_code) that the object does not use and none of them
  holds the literal's text: that raises VerbatimBlockError. A literal that
  repeats such a block is accepted.

Opt-in: no default changes; existing protocols and callers behave as before.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.