[cbrnr]

This picks up the proposals from #179 (after a critical review) plus a few things the issue missed. It differs in three ways:

• When a chunk's length doesn't match its samples, it falls back to the old per-sample reader instead of failing.
• It deletes the unused .pxd instead of editing it.
• It also adds lazy debug logging and a fix for truncated chunks.

Measured speedups were 2.5–5x (claimed: 2.7–7x), almost all from chunk parsing. Time series are identical; time stamps differ by up to about 1e-11 s when clock sync and dejittering are on.

Changes

• Read numeric samples chunks at once instead of sample by sample. Chunks where all or no samples have a time stamp are parsed in a single NumPy call; mixed chunks are walked in memory. If the chunk length does not match the samples (or the file ends early), we fall back to the previous per-sample reader, so files that loaded before still load.
• Only format per-chunk debug messages when they are logged (this previously also called f.tell() for every chunk).
• Closed-form line fit in _jitter_removal instead of lstsq (~6.7x faster, no large temporary arrays).
• Simplified ADMM iteration in _robust_fit (mathematically equivalent, ~2.5x faster, but only ~10 ms per stream).
• Fix: a chunk cut off at the end of the file is now dropped instead of returning a last sample filled with leftover bytes from the previous sample (readinto silently returned a short read).
• Remove pyxdf.pxd, which was never part of the build and no longer matched the code.

Performance

Synthetic recordings (64 channels, float32, 500 Hz, 30 min, 32-sample chunks; last row 8 channels, 1000 Hz, 10 min, 2-sample chunks), load_xdf with default settings:

Time stamps in file	main	this PR	Speedup
every sample	0.69 s	0.14 s	5.0x
none	0.63 s	0.19 s	3.4x
first sample per chunk	0.63 s	0.24 s	2.6x
alternating	0.65 s	0.26 s	2.5x
every sample, 2-sample chunks	1.05 s	0.79 s	1.3x

Time series are byte-identical to main. Time stamps are identical without clock synchronization and dejittering; with them, they differ by at most ~1e-11 s because the fits are computed in a different order.

Tests

New test/test_read_chunk.py covers all numeric formats with all time stamp patterns, wrong chunk lengths (first and last chunk), truncated files, empty chunks, flag/length mismatches, and checks _robust_fit against the original implementation.

[cbrnr]

Ping @sappelhoff - Gemini was not wrong! Could you review and test these changes, because you said another agent did not review it favorably?

[sappelhoff]

Thanks for picking this up, I am glad there was something salvageable.

Will dig out the review and will have another look at it (personally and with another Gemini agent)

[cbrnr]

Thanks! It would be helpful if you tried loading some of your real-world XDF files to see if nothing broke.

[sappelhoff]

This was the critical agent feedback and how you addressed it:

1. Bit-for-bit claim on time_stamps when clock sync / dejittering are enabled:

• Feedback on PERF: Speed up _read_chunk3, _robust_fit, and _jitter_removal (2.7x–7x faster load_xdf) #179: _read_chunk3 (Item 1) produces bit-for-bit identical time_series and raw time_stamps (np.array_equal), but _robust_fit (Item 2) and _jitter_removal (Item 3) reassociate floating-point sums and therefore differ by a few float64 ULPs when synchronize_clocks=True or dejitter_timestamps=True.
• How PR Speed up load_xdf and fix reading truncated sample chunks #180 handled it: Stated directly in the PR description ("Time series are byte-identical to main. Time stamps are identical without clock synchronization and dejittering; with them, they differ by at most ~1e-11 s because the fits are computed in a different order.").

2. Cython src/pyxdf/pyxdf.pxd incompatibility with np.cumsum(stamps, out=stamps):

• Feedback on PERF: Speed up _read_chunk3, _robust_fit, and _jitter_removal (2.7x–7x faster load_xdf) #179: pyxdf.pxd had @cython.locals(k=cython.Py_ssize_t, stamps=cython.double[:]) on _read_chunk3. If compiled with Cython, stamps would be typed as a memoryviewslice rather than an ndarray, causing np.cumsum(stamps, out=stamps) to fail with TypeError: output must be an array.
• How PR Speed up load_xdf and fix reading truncated sample chunks #180 handled it: Deleted the unused and out-of-date src/pyxdf/pyxdf.pxd file altogether.

3. Potential 64-bit integer overflow in n * (n * n - 1) / 12 in _jitter_removal:

• Feedback on PERF: Speed up _read_chunk3, _robust_fit, and _jitter_removal (2.7x–7x faster load_xdf) #179: If start_i and stop_i were ever np.int64 (or Cython Py_ssize_t), n * (n**2 - 1) would overflow 64-bit signed integers at n >= 2,097,153 (~35 min at 1 kHz).
• How PR Speed up load_xdf and fix reading truncated sample chunks #180 handled it: In pyxdf.py, _find_segment_indices constructs segments = list(zip(start_idx.tolist(), end_idx.tolist())), so start_i, stop_i, and n = stop_i - start_i + 1 are arbitrary-precision Python ints (<class 'int'>) and do not overflow. (In addition, PR Speed up load_xdf and fix reading truncated sample chunks #180 centers y - y_mean before computing np.dot(idx, y - y_mean), which improves numerical conditioning over PERF: Speed up _read_chunk3, _robust_fit, and _jitter_removal (2.7x–7x faster load_xdf) #179's np.dot(u, y) and main's uncentered lstsq.)

4. Updating s.last_timestamp across all fast paths:

• Feedback on PERF: Speed up _read_chunk3, _robust_fit, and _jitter_removal (2.7x–7x faster load_xdf) #179: Fast Path 3's prose in PERF: Speed up _read_chunk3, _robust_fit, and _jitter_removal (2.7x–7x faster load_xdf) #179 omitted mentioning the writeback to s.last_timestamp.
• How PR Speed up load_xdf and fix reading truncated sample chunks #180 handled it: Consolidated at the end of _read_chunk3 via if nsamples: s.last_timestamp = stamps[-1].

---

Generally I think this PR is great and it does its job.

Tested on a lot (nearly 100) of real life recordings I have here. Some numbers:

[cbrnr]

Very nice, thanks for testing!