Repository navigation
Keep logging when json 3 rejects binary strings or duplicate keys - #411
Merged
Merged
Conversation
json 3.1 raises on non-ASCII ASCII-8BIT strings and json 3.0 raises on hashes with keys that serialize to the same name. In both cases the error escapes the formatter and the whole request log line is lost. The JSON and Cee formatters now serialize through a SafeJson helper. It tries JSON.dump first, so well formed events are unchanged. If generation fails, it retries with a normalized copy of the data: keys are stringified (later keys win), strings are converted to UTF-8, binary strings are treated as UTF-8, and invalid bytes are replaced. The original data is never modified. The log subscriber now merges custom_options and custom_payload so that a String key overrides a built-in Symbol key of the same name (and vice versa) instead of adding a second entry. This also stops the key=value style formatters from printing the key twice. Fixes #409 Fixes #410
The JSON fallback only cleaned up strings, so an exception object or a Symbol holding non-ASCII binary bytes still made the retry raise and the log line was lost. Strings in dummy encodings such as UTF-7 also raised Encoding::ConverterNotFoundError from inside the retry. normalize now converts Symbols to strings, keeps other objects only if they can be generated on their own, and otherwise uses their cleaned up to_s. Strings in dummy encodings are treated like binary strings. Also document how String and Symbol keys from custom_options and custom_payload override Lograge's own keys.
On TruffleRuby, json 3's pure-Ruby generator serializes Symbols and other non-native objects via to_s.to_json without passing its state. With ActiveSupport loaded, that call goes through ActiveSupport's encoder, which before Rails 8.1 passes the quirks_mode option that json 3 rejects with ArgumentError. Treat that as a generation failure so the normalized retry, which turns those objects into strings, runs. Replace the Time-based spec, whose result depends on the platform, with objects that define their own to_json.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #409
Fixes #410
Problem
json 3 is stricter about what it will generate, and Lograge currently passes the error up from the formatter.
ActiveSupport::LogSubscriberthen swallows it and logsCould not log "process_action.action_controller" event, so the entire request log line is lost.ASCII-8BITstrings (e.g. exception messages built from HTTP response bodies). Invalid UTF-8 (Better way to handle invalid charecter encoding in request params? #277, Failing to log JSON with UTF-8 chars #210) was already rejected before.custom_options/custom_payloadentry like"status" => ...collides with the built-in:status.Changes
Formatters:
Lograge::Formatters::Helpers::SafeJson(#409)The
JsonandCeeformatters now serialize viaSafeJson.dump:JSON.dump(data)as before. Well formed events produce exactly the same output and pay no extra cost.JSON::GeneratorErroror anEncodingError, retry with a normalized copy of the data:ASCII-8BITstrings and strings in dummy encodings (e.g. UTF-7, which has no converter) are treated as UTF-8, so valid multibyte text survivesnilare kept as-isto_jsonoutput is preserved), and are otherwise replaced by their cleaned upto_s. This covers the README'sexception_objectpattern when the exception message holds binary bytes.The input is never modified. If the retry also fails, the error is raised with the original error as its
cause.This is on by default instead of opt-in: it only applies to events that would otherwise be lost, and a log line with a few replacement characters is more useful than no line at all.
Log subscriber: merging custom data (#410)
custom_optionsandcustom_payloadare now merged so that a String key overrides an existing Symbol key of the same name (and vice versa) instead of being added alongside it. The value is stored under the key that was already present, so Lograge's own formatters, which read Symbol keys such asdata[:path]anddata[:status], see the override. Non-colliding keys keep their type, sobefore_formathooks and custom formatters that read String keys keep working. This is documented in the README.As a side effect, the key=value style formatters (
KeyValue,LTSV,L2met, ...) no longer print a key twice (status=200 status=ok). For JSON output this matches what most parsers did with json 2.x's duplicate output, where the last value won.SafeJson's key normalization also covers duplicates introduced in other places, such as abefore_formathook.Out of scope
Logstashformatter serializes vialogstash-event's own serializer rather than the json gem; it already handles binary strings, so it is unchanged.KeyValueetc.) can still hitEncoding::CompatibilityErrorwhen interpolating binary strings. That is a separate, pre-existing issue.Testing
SafeJson, theJson/Ceeformatters, and the subscriber merge behavior.