Skip to content

Run region x metric tasks serially; cap convolution threads at CPU count - #66

Open
crvernon wants to merge 1 commit into
v3/packaging-repofrom
v3/perf-serial-passes
Open

crvernon wants to merge 1 commit into
v3/packaging-repofrom
v3/perf-serial-passes

Conversation

@crvernon

Copy link
Copy Markdown
Member

Stacked PR — part of the Demeter v3.0.0 upgrade. Base: v3/packaging-repo. Review and merge in order, bottom of the stack first.

Plan items: 3.4 (partial)

Benchmark: IDENTICAL. Global case 16.9 → 12.1 s (−28%), RSS −17%.

Commits

Run region x metric tasks serially; cap convolution threads at CPU count

The intensification and expansion tasks are Python bound, so the
ThreadPool only added GIL contention. Measured on the global case: 16.2 s
threaded, 11.1 s with serial passes. The pool was also sized to the
number of unique integers in the (region, metric) index pairs rather
than to the number of tasks. Each task modifies only its own region x
metric cells and target_change entries, and since 1.1 no shared input
is mutated, so serial execution gives bit-identical results.

The kernel density convolution keeps its thread pool, because
ndimage.convolve releases the GIL (all serial: 21.9 s). The pool is now
capped at the CPU count instead of one thread per land class.

Benchmark vs v3.0.0-b4 (all IDENTICAL):

  • global_0p5deg: 16.9 s -> 12.1 s (-28%), peak RSS -17%.
  • global_0p5deg_negcons: 17.1 s -> 11.7 s (-32%).
  • texas_0p5deg: 0.22 s -> 0.17 s (-20%).
    The global case has gone from 34.0 s at b0 to 12.1 s (-64%).

🤖 Generated with Claude Code

The intensification and expansion tasks are Python bound, so the
ThreadPool only added GIL contention. Measured on the global case: 16.2 s
threaded, 11.1 s with serial passes. The pool was also sized to the
number of unique integers in the (region, metric) index pairs rather
than to the number of tasks. Each task modifies only its own region x
metric cells and target_change entries, and since 1.1 no shared input
is mutated, so serial execution gives bit-identical results.

The kernel density convolution keeps its thread pool, because
ndimage.convolve releases the GIL (all serial: 21.9 s). The pool is now
capped at the CPU count instead of one thread per land class.

Benchmark vs v3.0.0-b4 (all IDENTICAL):
- global_0p5deg: 16.9 s -> 12.1 s (-28%), peak RSS -17%.
- global_0p5deg_negcons: 17.1 s -> 11.7 s (-32%).
- texas_0p5deg: 0.22 s -> 0.17 s (-20%).
The global case has gone from 34.0 s at b0 to 12.1 s (-64%).

Co-Authored-By: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant