Repository navigation
fix(event_handler): harden multipart and JSON body validation - #8626
leandrodamascena merged 3 commits into
Conversation
- Return 415 when multipart/form-data has no boundary, and 422 for other multipart parsing errors (invalid base64, undecodable text) instead of letting them escape as 500/502. - Decode non-ASCII text fields in non-base64 multipart bodies correctly. - Only unwrap single-element lists for form and multipart bodies, so a JSON array sent to an object or scalar body parameter returns 422. - Parse Content-Disposition parameters so filename= before name= works.
leandrodamascena
left a comment
There was a problem hiding this comment.
Thanks @MTirnova, this is careful work and almost all of it checks out: I ran the suites and deployed it behind a real REST API, and café, ş€, the missing boundary, bad base64, filename= before name=, the JSON array cases and list[Item] all behave. Binary uploads with binary media types configured come through byte for byte.
One case needs a guard though. Without binary media types, API Gateway decodes the body as UTF-8 and replaces invalid bytes with U+FFFD before we ever see it. On develop that blew up in encode("latin-1") with a 502, which was accidental but at least loud. With the UTF-8 fallback we now return 200 with a corrupted file (a 2 KB random upload came back as 3730 bytes with a different md5). Could you make the fallback raise a 422 multipart_invalid when the body contains "�", with a message pointing at binary media types? ş€ keeps working since it has no replacement character, and mangled binaries fail clearly instead of silently. Everything else can stay exactly as is.
Without binary media types, API Gateway decodes the body as UTF-8 and replaces invalid bytes with U+FFFD. The UTF-8 fallback then passed a corrupted file on with a 200. Return 422 multipart_invalid instead, with a message pointing at Binary Media Types.
|
Good catch, thanks for testing it behind a real REST API! The UTF-8 fallback now raises a 422 |
leandrodamascena
left a comment
There was a problem hiding this comment.
Thanks @MTirnova, quick turnaround again. I redeployed behind a real REST API without binary media types: the 2 KB binary upload now gets a clear 422 instead of a corrupted 200, and café and ş€ still come through. The JPEG header test mirrors what API Gateway actually sends, nice.
Approving now. This goes out in 3.38.0, so it will sit here a bit before I merge. No need to rebase or touch anything, I'll take care of it and you'll get the credit in the release notes. Thank you!
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #8626 +/- ##
========================================
Coverage 96.93% 96.93%
========================================
Files 318 318
Lines 16058 16067 +9
Branches 1430 1430
========================================
+ Hits 15566 15575 +9
Misses 349 349
Partials 143 143 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience! |
|
Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience! |



Issue number: closes #8578
Summary
Changes
All three problems are in
middlewares/openapi_validation.py.1. Multipart errors escaping as 500/502
boundarynow raisesRequestUnsupportedContentType, so it returns 415 (missing_boundary) instead of a bareValueError.except ValueError: raisein_parse_multipart_data. It also re-raisedUnicodeDecodeErrorandbinascii.Error(both areValueErrorsubclasses), so they escaped as 500s. They now become the existing 422multipart_invaliderror.test_file_upload_non_base64_binary_contentrelies on it to keep raw file bytes intact. The body falls back to UTF-8 when it contains characters outside latin-1 (e.g.ş,€). Text fields and part headers are decoded as UTF-8, with a latin-1 fallback for bytes that came from the latin-1 round trip (e.g.café). If you'd rather switch fully to UTF-8 as suggested in the issue, I'm happy to change it, but that would change the bytes that non-base64 binary uploads get.2. JSON arrays accepted for object/scalar body parameters
The single-element list unwrapping in
_normalize_field_valueis now only applied to form and multipart bodies (_request_body_to_args(..., is_form_body=...)). A JSON array sent toitem: ItemorAnnotated[str, Body()]now returns 422. Query/header handling is unchanged.Note for reviewers: this changes the JSON case of
test_body_scalar_lists_keep_existing_normalization(added in #8509, where body normalization was intentionally left unchanged). I renamed it totest_body_scalar_lists_normalizationand parametrized the expectation: JSON now returns 422, and form data still returns the first value.3.
filename=beforename=_extract_header_paramnow matches the parameter only at the start of a;-separated segment, so looking upnameno longer matches insidefilename="...".Tests
New functional tests cover the missing boundary (415), an invalid base64 body (422), non-ASCII text in non-base64 bodies (latin-1 and non-latin-1 text), UTF-8 text in base64 bodies,
filename=beforename=, and JSON arrays for model and scalar body parameters. 8 of the 9 new tests fail ondevelop; the base64 UTF-8 one already passed and guards that path.tests/functional/event_handlerandtests/unit/event_handlershow the same failures before and after this change. The only failures locally come from optional dependencies missing in my environment (fastjsonschema).ruffandmypyare clean.User experience
Before:
cafésent without base64:UnicodeDecodeError→ 502boundary: unhandledValueError[{"name": "a", "price": 1}, {"name": "b", "price": 2}]sent todef create(item: Item): 200 with only the first itemContent-Disposition: form-data; filename="a.txt"; name="file": 422 missing fieldAfter:
café(andş,€) is received correctlyboundary: 415; other multipart parsing errors: 422filename=beforename=is parsed correctlyBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.