Skip to content

Keep logging when json 3 rejects binary strings or duplicate keys - #411

Merged
johnnagro merged 3 commits into
masterfrom
fix-json3-encoding-and-duplicate-keys
Nov 14, 2026
Merged

johnnagro merged 3 commits into
masterfrom
fix-json3-encoding-and-duplicate-keys

Conversation

@johnnagro

@johnnagro johnnagro commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #409
Fixes #410

Problem

json 3 is stricter about what it will generate, and Lograge currently passes the error up from the formatter. ActiveSupport::LogSubscriber then swallows it and logs Could not log "process_action.action_controller" event, so the entire request log line is lost.

Changes

Formatters: Lograge::Formatters::Helpers::SafeJson (#409)

The Json and Cee formatters now serialize via SafeJson.dump:

  1. Try JSON.dump(data) as before. Well formed events produce exactly the same output and pay no extra cost.
  2. If that raises JSON::GeneratorError or an EncodingError, retry with a normalized copy of the data:
    • hash keys are stringified (the first key keeps its position and the later value wins if two collide, matching the subscriber merge below)
    • strings and symbols are converted to UTF-8; strings in other encodings are transcoded
    • ASCII-8BIT strings and strings in dummy encodings (e.g. UTF-7, which has no converter) are treated as UTF-8, so valid multibyte text survives
    • invalid or unconvertible bytes become U+FFFD
    • numbers, booleans and nil are kept as-is
    • other objects are kept if they can be generated on their own (so custom to_json output is preserved), and are otherwise replaced by their cleaned up to_s. This covers the README's exception_object pattern when the exception message holds binary bytes.

The input is never modified. If the retry also fails, the error is raised with the original error as its cause.

This is on by default instead of opt-in: it only applies to events that would otherwise be lost, and a log line with a few replacement characters is more useful than no line at all.

Log subscriber: merging custom data (#410)

custom_options and custom_payload are now merged so that a String key overrides an existing Symbol key of the same name (and vice versa) instead of being added alongside it. The value is stored under the key that was already present, so Lograge's own formatters, which read Symbol keys such as data[:path] and data[:status], see the override. Non-colliding keys keep their type, so before_format hooks and custom formatters that read String keys keep working. This is documented in the README.

As a side effect, the key=value style formatters (KeyValue, LTSV, L2met, ...) no longer print a key twice (status=200 status=ok). For JSON output this matches what most parsers did with json 2.x's duplicate output, where the last value won.

SafeJson's key normalization also covers duplicates introduced in other places, such as a before_format hook.

Out of scope

  • The Logstash formatter serializes via logstash-event's own serializer rather than the json gem; it already handles binary strings, so it is unchanged.
  • The string-building formatters (KeyValue etc.) can still hit Encoding::CompatibilityError when interpolating binary strings. That is a separate, pre-existing issue.

Testing

  • New specs for SafeJson, the Json/Cee formatters, and the subscriber merge behavior.
  • With json 3.1.0, the new specs fail without the library changes and all 160 specs pass with them.
  • The full suite also passes with json 2.15.2. There, valid UTF-8 binary strings still serialize on the first attempt, with json's own deprecation warning.
  • RuboCop is clean.

json 3.1 raises on non-ASCII ASCII-8BIT strings and json 3.0 raises on
hashes with keys that serialize to the same name. In both cases the
error escapes the formatter and the whole request log line is lost.

The JSON and Cee formatters now serialize through a SafeJson helper.
It tries JSON.dump first, so well formed events are unchanged. If
generation fails, it retries with a normalized copy of the data: keys
are stringified (later keys win), strings are converted to UTF-8,
binary strings are treated as UTF-8, and invalid bytes are replaced.
The original data is never modified.

The log subscriber now merges custom_options and custom_payload so
that a String key overrides a built-in Symbol key of the same name
(and vice versa) instead of adding a second entry. This also stops
the key=value style formatters from printing the key twice.

Fixes #409
Fixes #410
The JSON fallback only cleaned up strings, so an exception object or a
Symbol holding non-ASCII binary bytes still made the retry raise and
the log line was lost. Strings in dummy encodings such as UTF-7 also
raised Encoding::ConverterNotFoundError from inside the retry.

normalize now converts Symbols to strings, keeps other objects only if
they can be generated on their own, and otherwise uses their cleaned
up to_s. Strings in dummy encodings are treated like binary strings.

Also document how String and Symbol keys from custom_options and
custom_payload override Lograge's own keys.
On TruffleRuby, json 3's pure-Ruby generator serializes Symbols and
other non-native objects via to_s.to_json without passing its state.
With ActiveSupport loaded, that call goes through ActiveSupport's
encoder, which before Rails 8.1 passes the quirks_mode option that
json 3 rejects with ArgumentError. Treat that as a generation failure
so the normalized retry, which turns those objects into strings, runs.

Replace the Time-based spec, whose result depends on the platform,
with objects that define their own to_json.
@johnnagro
johnnagro merged commit 07485d2 into master Nov 14, 2026
173 of 186 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant