ci: resolve Arrow and Parquet from conda-forge on Unix legs

[abnobdoss]

Part of #799 (Cutting CI build times: prebuilt Arrow and build configuration). This is the conda-forge Arrow piece, Unix legs only.

What

Adds .github/actions/setup-conda-arrow, a composite action that installs libarrow and libparquet from conda-forge and exports CMAKE_PREFIX_PATH and the loader path.

Used on the Unix legs that build Arrow from source today: the ubuntu and macos jobs in Test, ASAN/UBSAN, SQL Catalog's non-Windows slices, and the AWS system-SDK slice.

Why not the rest

Windows stays vendored because conda-forge only ships Release builds, so a Debug consumer would hit a CRT mismatch (/MDd vs /MD). That is a packaging limitation rather than something to sequence around.

Hive also stays vendored, since prebuilt Arrow exports no thrift::thrift target and iceberg_hive requires one. I plan to raise that as a follow-up PR.

The AWS bundled-SDK slices cannot use conda at all. Bundling makes Arrow build the AWS SDK into its own arrow_bundled_dependencies archive, and that only exists when Arrow is built from source, so only the system-SDK slice is converted. The Meson legs are untouched because they do not use Arrow.

Notes

install-arrow.sh reads ICEBERG_ARROW_BUILD_VERSION from IcebergThirdpartyToolchain.cmake instead of repeating the version, so bumping the vendored Arrow moves the conda legs with it. The pin is exact (libarrow==24.0.0), since conda treats = as a prefix match and patch releases do get published.

ICEBERG_REQUIRE_SYSTEM_ARROW=ON tells build_iceberg.sh to fail the leg if Arrow was built from source anyway. That check is needed because fetchcontent_declare(VendoredArrow ...) passes FIND_PACKAGE_ARGS: a failed find_package falls back to a source build with no warning, and the leg would still pass.

Validation

Converted legs go from 931 to 596 compile requests, and the rest are unchanged.

All 20 checks pass on the fork with the guard armed, so a green run does mean conda Arrow was resolved. Upstream reports the checks as skipped rather than passing, because every job is gated on draft == false.

[wgtmac]

Sorry for the late review! Could you help rebase it? We have recently upgraded arrow to use 25.0.0.

[wgtmac]

Is it possible to fix the Windows CI to leverage it? We should also leave at least one CI job to use bundled build to make sure it does not break.

[abnobdoss]

Thanks for the review, @wgtmac. Apologies for the delay getting back to this. The branch has been rebased onto Arrow 25.0.0, and I've incorporated the Hive and C++ linter conversions from #854.

Builds that meet the intended outcome (build types are unchanged):

Build (type)	Main	This PR
AMD64 Ubuntu 26.04 (Debug)	Vendored	Conda
AMD64 Ubuntu 26.04 (RelWithDebInfo)	Vendored	Conda
AArch64 macOS 26 (Debug)	Vendored	Conda
AMD64 Ubuntu 26.04 Hive (Debug)	Vendored	Conda + system Thrift
ASAN and UBSAN Tests (Debug)	Vendored	Conda; Iceberg remains instrumented, Arrow is prebuilt without sanitizer instrumentation
SQL Catalog (AMD64 Ubuntu 26.04) (Debug)	Vendored	Conda
SQL Catalog (AArch64 macOS 26) (Debug)	Vendored	Conda
AWS (Ubuntu 26.04, S3 + SigV4, system AWS SDK) (Debug)	Vendored Arrow + vcpkg AWS SDK	Conda Arrow + AWS SDK
cpp-linter (Debug)	Vendored	Conda + system Thrift
Meson - AMD64 Ubuntu 26.04 (Debug)	No Arrow	No Arrow
Meson - AArch64 macOS 26 (Debug)	No Arrow	No Arrow
Meson - AMD64 Windows 2025 (Debug)	No Arrow	No Arrow
Verify (macos-26) (Release)	Vendored	Vendored
Verify (ubuntu-26.04) (Release)	Vendored	Vendored

Open exceptions:

Build (type)	Main	This PR	Question
AMD64 Windows 2025 (Debug)	Vendored	Vendored	See question 1.
SQL Catalog (AMD64 Windows 2025) (Debug)	Vendored	Vendored	See question 1.
AWS (Ubuntu 26.04, S3 + SigV4, bundled AWS SDK) (Debug)	Vendored	Vendored	See question 2.
AWS (macOS 26 ARM64, S3, bundled AWS SDK) (Debug)	Vendored	Vendored	See question 2.

1. If I understand correctly the only way to use conda's prebuilt Arrow on Windows, we would have to change our Windows CMake builds from Debug to RelWithDebInfo or Release. Is that something we would want to pursue?
2. Mixing conda Arrow with the vcpkg AWS SDK caused a runtime crash in one of tests of the Ubuntu job; using conda for both resolved it. Do we want to retain both bundled AWS jobs, or keep Ubuntu bundled and switch the macOS job to conda-provided Arrow and AWS SDK?
3. Both Hive-enabled jobs now use system Thrift. Given your point about bundled coverage, should we add bundled Hive/Thrift coverage to release verification, or revert one of those jobs to bundled dependencies?

More crucially, although this speeds up builds and reduces reliance on warm caches, the extra exceptions above feel like they make CI harder to maintain. I'm leaning toward dropping this change for now because I don't see a simple way to preserve coverage without adding additional complexity to the CI which is already slightly complex. What do you think: should we continue pursuing this change or drop it?