Repository navigation
[NAT.11] Image family: functional PNG/TIFF/EXR codecs over OpenImageIO and the managed ImageReader binding - #175
Conversation
Implements arc_image_open, arc_image_read and arc_image_close in native/arcimage-abi over OpenImageIO 3.1.14.0. Content is probed from the leading bytes (PNG, TIFF, EXR only); reads go through an arc_io_v1 proxy adapter; dimensions, pixel counts, tile memory and scanline bands are checked before any decode allocation; tile reads are capped; a 64-handle generation table backs the image handles; build info is a JSON capability manifest and the functional ABI minor is 1 (D2). Coverage (D3) is raster-order and non-overlapping; conversions and losses are reported in the metadata document (D4). Known gap, not resolved in this commit: OpenImageIO premultiplies 8-bit PNG and TIFF alpha as it reads, in 8-bit precision, and a reader configuration cannot be combined with a callback proxy in this build. Straight RGBA8 output of semi-transparent pixels is therefore inexact (worst delta 147 on the codec fixture). Two codec checks are expected to fail until the coordinator decides between direct libpng decode (native-libpng-r4 is admitted) and accepting the reported precision loss. Validated on win-x64-shim-static: library and test executables build with /W4 /WX; arcslate_image_abi.hello passes; arcslate_image_abi.codec runs 158 checks with 2 failures (round-trip byte exactness for PNG and TIFF with partial alpha). linux-x64 not run (WSL cmake/ninja missing). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Pass the callback proxy as the oiio:ioproxy pointer attribute, with oiio:UnassociatedAlpha = 1 for PNG and TIFF, in the ImageSpec config given to ImageInput::open. OpenImageIO 3.1.14.0 honours the request in its PNG and TIFF readers (png.imageio/pnginput.cpp, tiff.imageio/tiffinput.cpp) and returns straight alpha, so rgba8 output is byte exact without a premultiply and unpremultiply round trip. Associated sources (EXR, or a TIFF that marks its alpha associated) are unpremultiplied only where straight colour is needed, and rgba32f output premultiplies straight sources after any sRGB transfer. TIFF association is read from oiio:UnassociatedAlpha. With the request in place the reader reports that attribute only for unassociated alpha, so an absent attribute means associated. The previous rule treated absence as straight, which mislabelled associated TIFF files. Test fixtures: the PNG writer rewrites colour unless the spec marks alpha unassociated, and the TIFF writer labels alpha from the same attribute, so straight fixtures are encoded with that attribute. The three rgba8 metadata assertions that described the premultiply and unpremultiply path now assert the straight path. The byte-exact round-trip check is unchanged. Added an associated-TIFF passthrough check. native/arcimage-abi/README.md records the route and why no direct libpng or libtiff decode is needed. eng/packaging/native_consumer.py (ADP-07 binding, decision D5): ArcImageNative expects ABI 1.1 and checks its JSON capability manifest and identity field, because the functional build info embeds the identity as a field and no longer ends with the ;-prefixed suffix. Other libraries are unchanged. Validated on win-x64-shim-static with /W4 /WX: arcslate_image_abi.codec reports 164 checks and 0 failures. linux-x64 not run (WSL cmake/ninja missing). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Unit 2 of the image family. The native codec tests now cover bit depth and metadata round trips (PNG 8 and 16 bit, TIFF 8, 16 and float, EXR half and float), partial and one-pixel edge tiles, decompression bomb, memory and deadline refusals before decode, truncated and corrupt input (CORRUPT with no output), failed codec callbacks, cancellation at every tile and callback boundary (D9), coverage refusals (D3), one-byte-short buffers, unsupported and empty content (D10), and EXR multipart and mip subimages within the item limit. Each functional case is its own CTest target, arcslate_image_abi.codec.<name>. The CI CTest filter in native-abi.yml follows the new names (D5). A failed reader open now drains OpenImageIO's pending error, so the message does not accumulate in the calling thread. Validated on win-x64 with the CI-equivalent sequence: configure, build, install, the anchored CTest filter (36 of 36) and dotnet NativeAbiLayout. linux-x64 is recorded as not run (D8): the WSL cmake and ninja install is not present. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Unit 3 of the image family: the managed binding for arc_image_open, arc_image_read and arc_image_close, with the ABI 1.1 functional minor verified through the JSON build-info capability manifest. - ImageAbi.Functional: LibraryImport declarations kept internal; the library must advertise minor 1, image.open/read/close and png/tiff/exr. - ImageHandle: dedicated SafeHandle owning the generation token; the callback GCHandle is freed only after a successful native close. - ImageReader: IAsyncDisposable, bounded async open/read/complete with CancellationToken, per-reader serialised native calls, managed coverage accounting that mirrors D3, and CompleteAsync that refuses missing coverage with ImageCoverageIncompleteException. - ImageOptions, ImageResults, ImageMetadata: typed options and immutable result records; no native pointer or raw options dictionary is exposed. Output format Unspecified is refused rather than defaulting silently. - Callbacks never throw across the ABI; cancellation bridges through a pinned CancellationState for the duration of one native call. Tests (tests/NativeAbiTests, D5): PNG and TIFF straight-alpha byte exact round trips, float premultiply, coverage and completion, overlap and raster-order refusal without state change, corrupt and unsupported content, cancellation without coverage, dispose, invalid input and an oversized region refused before any read. Validated locally on win-x64 through the build slot: unlocked and locked restore of the test project, build with TreatWarningsAsErrors, NativeAbiLayout 3 of 3, NativeAbi 15 of 15, native CTest filter 36 of 36, full solution locked restore, managed licence-boundary evaluation (61 projects) and source and native provenance. linux-x64 is not run (D8, WSL cmake and ninja not installed). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Apply clang-format to the image shim sources (line wrapping only, no behaviour change). Rename tests/arcimage_codec_tests.cpp to tests/arcslate_image_codec_tests.cpp so the test source follows the arcslate_* naming of the target and the header (decision D6). The naming policy excludes the ArcImage product name, so the lowercase token is removed from the path and the fixture and summary strings. The summary now reads as ArcImageNative. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…T.11 project blobs Successor receipt nat-11-r1 chains from gov-30-r1 and names reviewer w-deku-20261008-rev-nat-11. The bound inputs that changed are the ArcForges.Native.Image csproj, the NativeAbiTests csproj and its lock (project reference only). The NuGet closure (58) and Python closure (10) are unchanged. The independent review is pending. Reconciliation active project pins move to the committed blobs of the three NAT.11 project files (the arcimage CMake file, the Native.Image csproj and the NativeAbiTests csproj). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
The guard already names the successor; it now names nat-11-r1 and checks that the successor chains from gov-30-r1. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Unit 5 of the image family ran the CI-equivalent win-x64 sequence from a clean shim-static build: CTest with the anchored CI filter (36 of 36), dotnet restore --locked-mode, NativeAbiLayout (3 of 3), the NativeAbi functional managed tests (15 of 15) through a manifest-checked local ArcImageNative.dll pair, a zero-warning managed build, and dotnet format --verify-no-changes. The local CMake (4.4.3) and Ninja (1.13.2) differ from the owned CMake 4.3.3 and Ninja 1.13.1, so eng/packaging/native.py stage refuses the owned generator. The staged packages, the managed Runtime pack and the package-guard tests therefore remain CI authority and are not run locally. linux-x64 is not run (D8). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Fix the two material findings from the independent review at 7ac7536. ImageReader.ReadRegionAsync: once the native read has returned OK, native coverage is committed. The managed read discarded its pixels and did not count the region when a cancellation arrived after that point, so the region could not be retried (native refuses the overlap) and CompleteAsync never succeeded. The managed count now follows the native success unconditionally, and the doc comment states the contract exactly: a region native code refuses or reports as cancelled changes no coverage; a region that native code returned OK for is counted and returned. arc_image_open: the metadata buffer was written before the handle was registered, so a refused 65th open (RESOURCE_LIMIT) wrote metadata while returning failure, against Annex 06 section 1. The handle is now registered first, into a local token. The metadata is copied only after that, and a short metadata buffer closes the new handle again, so a failed call writes no output and leaves no handle open. Tests: the lifecycle test checks that a short buffer is not written, that it leaves no handle open, and that a refused 65th open leaves a sentinel metadata buffer and its required field untouched. The managed suite gains CancellationAtEveryReadOrdinalKeepsCoverageConsistentWithOutcome, which sweeps the cancellation trigger across every source read of a region. The window after native success is not reachable deterministically through the source callback, so the sweep is the bounded approximation. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
Reviewed 9061e47 for [NAT.11] epoch 1: approved Reviewer: independent session w-deku-20261008-rev-nat-11, which authored none of this. Rounds: round 1 at 7ac7536 had two material findings:
Both were fixed in 9061e47. Round 2 approved. Verified at the exact head:
Non-blocking (recorded as follow-ups):
Not run locally (hosted CI authoritative): linux-x64 (D8), the PackageGuards tests, the packaging consumers, and a Native AOT consumer publish. Posted by the coordinator under the 2026-10-08 publication protocol. |
The repository-hooks job (pre-commit clang-format v22.1.8) on PR #175 reported one hunk in arcslate_image_codec_tests.cpp: a call that fits on one line. The diff is applied exactly as the hook printed it; no other file changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reviewed e8dff23 for [NAT.11] epoch 1: approved (delta review over the approved 9061e47). Reviewer: independent session w-deku-20261008-rev-nat-11.
Posted by the coordinator under the 2026-10-08 publication protocol. |
OpenImageIO refuses a header whose uncompressed size exceeds the process global limits:imagesize_MB, whose default is min(32 GiB, physical memory). The shim's answer for a 65535 x 65535 RGBA8 header therefore depended on host memory: RESOURCE_LIMIT on a large workstation, an OpenImageIO refusal at open on the smaller hosted runner. The shim now pins limits:imagesize_MB once, before the first OpenImageIO open, to 131072 MiB: hard_max_pixels x max_channels x 8 bytes, the largest uncompressed image the hard profile admits. limits:channels keeps the host-independent OpenImageIO default of 1024, which is above the shim's 64-channel bound. The codec test executable links its own OpenImageIO copy, so it cannot read the shim's global. oiio_size_pin checks the pin through arc_image_open with a 128 GiB, 64-channel double TIFF header, which must open. The bomb test is unchanged. The README records the pin, its reason and the test counts. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
Reviewed df3cf78 for [NAT.11] epoch 1: approved (delta review over the approved e8dff23). Reviewer: independent session w-deku-20261008-rev-nat-11. Defect fixed. Hosted native-win-x64 failed Fix. Test. The new case Verified:
Non-blocking, recorded as follow-ups:
Posted by the coordinator under the 2026-10-08 publication protocol. |
…ventory df3cf78 added the oiio_size_pin case to native/arcimage-abi/CMakeLists.txt, so the GOV.18 active-project pin drifted, and hosted reconciliation-policy failed with "pinned project blob drift". The row now holds the blob of that file at this commit (git rev-parse HEAD:native/arcimage-abi/CMakeLists.txt). No other row changes. Validation: eng/reconciliation.py and eng/dependency_policy.py pass; the reconciliation, dependency-policy and desktop-RID unit suites pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ackaging
The hosted native-win-x64 job failed at native.py stage with "Owned native
export set differs from the admitted ABI." ArcImageNative.dll now exports
arc_image_open, arc_image_read and arc_image_close (ABI 1.1, Annex 06)
alongside the three support probes, but the packaging checks admitted only
the probes. The same expectation was in packages.py verify
("Owned ABI exports mismatch.").
The admitted set is now one definition in native.py: every library keeps the
three probes, and ArcImageNative additionally admits _open, _read and _close.
native.require_owned_exports is used by both the stage check and the package
verify check, so the two cannot drift. A missing or extra export still fails.
No other library's expectation changes.
test_packages.py gains OwnedExportGuards: the exact six-export set is admitted,
each missing export and an extra export are rejected, probe-only is no longer
admitted for ArcImageNative, other libraries keep the probe-only set, and the
admitted set must equal the exports in native/arcimage-abi/exports/windows.def.
native_consumer.py already carries the ABI 1.1 expectation and has no export
set check, so it is unchanged.
ADP-07 supporting-file binding: eng/packaging/native.py and
eng/packaging/packages.py are outside NAT.11's recorded writes. They are bound
here for CI packaging of the delivered functional ABI. The matching unit suite
is eng/packaging/test_packages.py.
Validation: python -I -m unittest discover -s eng/packaging -p test_*.py runs
the 31 tests; OwnedExportGuards and ExternalDependencyGuards pass. The
PackageGuards fixtures need artifacts/packages from pack and are validated in
the follow-up gate run.
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
Reviewed 71a2aec for [NAT.11] epoch 1: approved (delta review over the approved df3cf78 and 0097442). Reviewer: independent session w-deku-20261008-rev-nat-11.
Non-blocking: the ADP-07 binding is now recorded in the PR body; secret-scan and dependency-review are hosted only. Posted by the coordinator under the 2026-10-08 publication protocol. |
…cture gate The hosted managed-packages run on 71a2aec failed the Platform validation step on two architecture rules for the ArcImageNative family. - RepositoryPolicyTests.ProductionNativeBindingsHaveOneCapabilityOwner listed ArcImageNative:arc_image_open as an unadmitted production binding. The closed map admitted only the three support probes. It now admits arc_image_open, arc_image_read and arc_image_close, each owned by src/Native/ArcForges.Native.Image with the flags ImageAbi.Functional.cs declares (no UTF-16 marshalling, no last-error capture). The rule itself is unchanged and no exemption is added. - EvaluatedPolicyTests reported RP-10 for the public members ImageByteSource.ReadAt, ImageByteSource.FromStream, ImageReader.OpenAsync, ProbeAsync, ReadRegionAsync, CompleteAsync and DisposeAsync. Each now maps in architecture-contract-tests.json to an existing [Fact] in tests/NativeAbiTests ImageReaderTests that drives that member through the native reader. No stub test is added. ADP-07 supporting-file binding: tests/ArchitectureTests/RepositoryPolicyTests.cs and eng/policy/architecture-contract-tests.json are outside NAT.11's recorded writes. They are bound here because the closed binding map and the public-API to contract-test correspondence are the policy data the managed-packages gate reads for the ABI 1.1 functional exports admitted in 71a2aec. Local validation on this tree (Windows, pinned SDK 10.0.400, build slot): ArchitectureTests 134/135; the single failure is EvaluatedPolicyTests with only RP-01, RP-08, RP-09 and AT-09, which need the hosted naming, secret and native configure receipts. RP-10 and the RepositoryPolicyTests failure are gone. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
Reviewed a68092e for [NAT.11] epoch 1: approved (delta review over the approved 71a2aec). The reviewer's single material condition, recording the ADP-07 binding of the two new files in the PR and handoff, is met in the PR body and the claim handoff. Reviewer: independent session w-deku-20261008-rev-nat-11. What a68092e changes:
Reproduced locally: every PR-job step that can run on this workstation or in WSL Debian (SDK 10.0.400):
Hosted-only: the native vcpkg and owned CMake and Ninja stage, pack, PackageGuards, the architecture evidence combine, and dependency-review. Non-blocking (follow-ups):
Posted by the coordinator under the 2026-10-08 publication protocol. |
|
Correction to the 71a2aec review record above: it says "delta review over the approved df3cf78 and 0097442", which is inaccurate. 0097442 (the CMakeLists blob re-pin) was not approved. Its delta review found that hosted run 37892380137 failed on the packaging export set ( |
Claim: NAT.11 epoch 1 (w-deku-20261008-nat-11)
Task record: native lane, task-nat-11. Authority: Annex 06 image clarification (planning repair 2026-10-09) and the coordinator decisions D1 to D12 (brief section 10, "NAT.11 decisions" and "NAT.11 straight alpha and packaging check").
What this changes
native/arcimage-abi(logical library ArcImageNative, ABI 1.1): implements the functionalarc_image_open/arc_image_read/arc_image_closefamily over OpenImageIO 3.1.14.0 with OpenEXR and Imath. Its behaviour:oiio:UnassociatedAlpha), with byte-exact 8-bit PNG and TIFF RGBA;arc_image_options_v1.formatat open;arcslate_image_abi.codec.*CTest cases plusarcslate_image_abi.hello. Fixtures are generated in test code, with no downloads. The cases are added to thenative-abi.ymlCTest filter (D5).src/Native/ArcForges.Native.Image:CancellationTokenon bounded async work, andIAsyncDisposable;ImageReadercompletion step that refuses missing coverage.It is AOT and trim clean with warnings as errors, and exposes no native pointer.
tests/NativeAbiTests: the functional oracle (15 ImageReaderTests) and the smoke and probe expectations for ABI 1.1 and the JSON build-info.CommonAbiLayoutTests.csis unchanged.nat-11-r1, chained fromgov-30-r1, names reviewerw-deku-20261008-rev-nat-11;eng/test_desktop_rids.pyactive-receipt assertion;eng/packaging/native_consumer.pyexpects ArcImageNative 1.1 and the JSON build-info.Validation actually performed (Windows, through the workstation build slot)
win-x64, preset win-x64-shim-static:
/W4 /WXwith zero warnings;The reviewer's fresh build differs from the author's staged DLL only in the PE and debug timestamps. Both embed source
9061e47d, dirty=false.Managed, SDK 10.0.400:
dotnet format --verify-no-changes.Python suites and policy scripts:
dependency_policy(58 NuGet, 10 Python, 281 inputs),reconciliation(67 projects, 7 native),native_provenance(21 components) andcheck_provenance(994 files).Not run locally:
native_consumer.pypackaging consumers;Hosted CI is authoritative for these.
Follow-up commits after the first review
e8dff231: one clang-format hunk printed by the repository-hooks job.df3cf780: pins OpenImageIOlimits:imagesize_MBto the shim's own bounds (131072 MiB = 2^28 pixels x 64 channels x 8 bytes). OIIO's default of min(32 GiB, physical memory) madecodec.limits_bombdepend on host memory, and it failed on the hosted runner. The new host-independent caseoiio_size_pincovers it.0097442f: re-pins thenative/arcimage-abi/CMakeLists.txtblob in the GOV.18 active inventory.71a2aec8: the packaging export-set checks admit the six ABI 1.1 exports for ArcImageNative only (arc_image_get_abi_version,_get_build_info,_get_last_error,_open,_read,_close). All other libraries keep the probe-only set.eng/packaging/native.pyandeng/packaging/packages.pyare outside the recorded NAT.11 writes. They are changed only so that CI can package the delivered functional ABI. No dependency receipt binds their hashes.a68092e2: hosted managed-packages "Platform validation" (tests/ArchitectureTests) failed twice.ProductionNativeBindingsHaveOneCapabilityOwnerlistedArcImageNative:arc_image_open, and RP-10 found no contract-test correspondence for the new publicArcForges.Native.ImageAPI. The resulting invariant-accounting "Malformed TRX" failure was downstream of the RP-10 message. The fix:arc_image_open,arc_image_readandarc_image_closewith the single ownersrc/Native/ArcForges.Native.Image, using the flags ofImageAbi.Functional.cs.ImageReaderTestsfacts.tests/ArchitectureTests/RepositoryPolicyTests.cs(the closed native binding map) andeng/policy/architecture-contract-tests.json(the RP-10 contract-test map) are outside the recorded NAT.11 writes. Their owner is the DesktopPlatform integration owner, and they are append-only rows. They are changed only so that the shared architecture policy accepts the delivered functional ABI and its public binding. No dependency receipt binds their hashes.Scope note (2026-10-09)
Under the user's scope correction of 2026-10-09, NAT.11 is finished as an in-progress DesktopPlatform delivery under the DesktopPlatform exception: review, CI, merge and publication closeout, and then it stops. It is not an ArcScope prerequisite, and no further native-image work (NAT.22, NAT.25, NAT.31, PLT.54, the image legs of NAT.28 and NAT.30) follows from it.
🤖 Generated with Claude Code