HTTP/2 Transport Benchmarks, Findings, and Production Server Tuning

[FrancescAlted]

HTTP/2 Transport Benchmarks, Findings, and Production Server Tuning

Summary

This branch implements the investigation outlined in plans/http2-optim2.md to determine whether HTTP/2 multiplexing should be adopted for peer-cache and chunk I/O in caterva2.c2cache and python-blosc2 (C2Array.aget_chunk), compared to the current pooled HTTP/1.1 approach.

Through local fixtures and live benchmarks against the production server (https://cat2.cloud/demo), we identified and resolved a critical reverse-proxy bottleneck, benchmarked multi-chunk workloads across varying concurrency levels, and established clear architectural guidelines for transport defaults.

---

Key Achievements & Findings

1. Benchmark Suite & Protocol Assertion Fixtures

Added reproducible benchmark harnesses under examples/benchmarks/http2/:

• peer_read.py: Cold sparse-cache multi-chunk retrieval with strict protocol assertion (HTTP/1.1 vs HTTP/2) and support for multi-dimensional --slice expressions.
• single_fetch.py: Isolates network transfer time from cframe deserialization and NumPy materialization for single large slices (api/fetch).
• simulated_latency.py & Caddyfile: Automated local test fixture using Caddy (TLS/HTTP2) and Toxiproxy (RTT simulation) with verified certificate authorities, integrated into caterva2/tests/test_peers.py.

2. Production Nginx Audit & 6x Slicing Acceleration (cat2.cloud)

Initial benchmarks on cat2.cloud showed HTTP/1.1 performing ~4.3x slower than HTTP/2 on single slices (1.36 s vs 0.32 s). Auditing the production Nginx configuration revealed two major bottlenecks:

1. Unix Socket Churn: Missing upstream keepalives forced Nginx to close and re-open the Uvicorn Unix domain socket on every request.
2. Disk Spooling: Default 32 KB buffers (proxy_buffers 8 4k) caused Nginx to spool the 2.68 MB compressed slice to disk files (/var/lib/nginx/proxy/...).

Fix Applied:

• Enabled upstream keepalive pool (upstream demo { server unix:...; keepalive 32; } + proxy_http_version 1.1; proxy_set_header Connection "";).
• Configured 2 MB in-memory ring buffers (proxy_buffers 16 128k; proxy_busy_buffers_size 256k;) to stream directly from RAM to the network card without disk I/O.
• Enabled tcp_nodelay on; and tcp_nopush on;.

Result: Single-slice download times dropped from 1.36 s down to 0.22 s (a 6x speedup), matching HTTP/2 (~0.22 s vs 0.24 s).

3. Multi-Chunk Benchmarks on gaia-3d.b2nd

Tested concurrent chunk retrieval on @public/large/gaia-3d.b2nd across 64 chunks (~20–50 KiB compressed per chunk) against the tuned cat2.cloud server:

Concurrency	HTTP/1.1 Median	HTTP/2 Median	Ratio (H1 / H2)	Result
1 (serial)	4.652 s	4.647 s	1.001	Dead tie
4	1.367 s	1.433 s	0.954	HTTP/1.1 is ~5% faster
8	0.838 s	0.970 s	0.863	HTTP/1.1 is ~15% faster
16	0.612 s	0.809 s	0.773	HTTP/1.1 is ~30% faster

Why pooled HTTP/1.1 wins for chunk I/O:

• Multiple TCP Congestion Windows: Pooled HTTP/1.1 opens $N$ independent TCP connections, each with its own kernel CWND, filling WAN bandwidth in parallel. HTTP/2 forces all streams through 1 TCP connection and 1 congestion window.
• No Head-of-Line Blocking: Packet loss on one connection does not stall the other 15 connections.
• Lower CPU Overhead: Avoids pure-Python h2 frame demultiplexing over active streams.

---

Architectural Conclusions & Decisions

1. caterva2.Client: Keep http2=True (default). For interactive user queries and single slices, HTTP/2 matches HTTP/1.1 throughput (~0.22 s) while minimizing server socket exhaustion under multi-user load and supporting clean RST_STREAM stream cancellation.
2. c2cache & python-blosc2 (C2Array.aget_chunk): Retain pooled HTTP/1.1 (http2=False). For bulk concurrent chunk downloads, connection pooling consistently outperforms HTTP/2 multiplexing by 15% to 30%. No code changes are required in either repository.