Skip to content

test(cuda.core): keep graph-pool tests off the device default pool - #3496

Open
juenglin wants to merge 1 commit into
NVIDIA:mainfrom
juenglin:tests-avoid-default-pool-with-graph-pool
Open

juenglin wants to merge 1 commit into
NVIDIA:mainfrom
juenglin:tests-avoid-default-pool-with-graph-pool

Conversation

@juenglin

@juenglin juenglin commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #2381.

The device default mempool and the graph memory pool each reserve about twice the device memory of virtual address space for the lifetime of the process, and neither can be capped. A test that touches both needs two such reservations at once, which does not fit into a small virtual address space on a large-memory GPU (e.g. two ~180-190 GiB reservations on a 96 GB device vs. a 256 GiB address space).

Running each cuda_core test in its own process and probing both pools at session end, only two tests touch both:

  • graph/test_graph_memory_resource.py::test_graph_alloc (graph-capture params): DeviceMemoryResource(device) for the output buffer, plus GraphMemoryResource.
  • test_object_protocols.py::test_no_cross_type_equality: sample_buffer (Device.allocate) plus sample_alloc_node.

Both only need some device pool, so they now use a pool capped at POOL_SIZE:

  • test_graph_alloc creates a capped DeviceMemoryResource and releases the buffer and pool in a try/finally.
  • sample_buffer and sample_buffer_alt allocate from a new sample_memory_resource fixture (capped pool closed at teardown; falls back to the device's memory resource if pools are unsupported).

This deliberately departs from the cuda_core/tests/AGENTS.md rule against adding max_size to a call that passes no options, since that rule assumes the default pool already exists. AGENTS.md now documents the exception.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@juenglin juenglin added this to the cuda.core 1.3.1 milestone Nov 18, 2026
@juenglin juenglin added test Improvements or additions to tests cuda.core Everything related to the cuda.core module labels Nov 18, 2026
@juenglin juenglin self-assigned this Nov 18, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin
juenglin force-pushed the tests-avoid-default-pool-with-graph-pool branch from 50106a8 to 0ab7c6e Compare November 18, 2026 23:36
@juenglin
juenglin requested a review from Andy-Jost November 18, 2026 23:36
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 0ab7c6e

@juenglin
juenglin marked this pull request as ready for review November 18, 2026 23:38
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Tests
    • Updated memory-resource and object-protocol tests to use capped memory pools where supported, with a fallback to the device’s existing resource when pools are unavailable.
    • Added cleanup and device synchronization after pool use, and tested buffers of different sizes.
  • Documentation
    • Clarified testing guidance for cases that use both device default pools and graph memory pools.

The object protocol tests and graph allocation test now use device memory resources capped at POOL_SIZE. The graph allocation test also closes its device memory resource and synchronizes the device in a finally block.

Changes: Memory resource tests

Layer / File(s) Summary
Object protocol buffer fixtures
cuda_core/tests/test_object_protocols.py, cuda_core/tests/AGENTS.md
A fixture uses the device resource when memory pools are unsupported. Otherwise, it creates a resource capped at POOL_SIZE, then closes it and synchronizes the device. The 64-byte and 1024-byte buffers allocate through this fixture on the default stream. Test guidance describes this capped-pool pattern.
Graph allocation resource lifecycle
cuda_core/tests/graph/test_graph_memory_resource.py
test_graph_alloc caps its device memory resource at POOL_SIZE, uses a context-managed output buffer, then closes the resource and synchronizes the device in a finally block. The graph memory resource remains uncapped.

Change: Other

Merge Risk: Minimal · up to 80985

This PR caps the device memory pools used by two tests and releases them explicitly at the end of each test, so the tests no longer reserve address space for both the default and graph pools. The earlier cleanup concern is addressed. Only test code changes, so merge risk is minimal.

Security architecture risk: Minimal · up to 80985

The change uses capped, explicitly owned memory pools in tests without changing production interfaces or trust boundaries. No introduced or worsened architecture-level security risk was established.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0ab7c6e and 809855b.

Files selected for processing: 2

  • cuda_core/tests/AGENTS.md
  • cuda_core/tests/graph/test_graph_memory_resource.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 9b7a2cb and 0ab7c6e.

Files selected for processing: 2

  • cuda_core/tests/graph/test_graph_memory_resource.py
  • cuda_core/tests/test_object_protocols.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@github-actions

Copy link
Copy Markdown
Contributor

The device default mempool and the graph memory pool each reserve about
twice the device memory of virtual address space for the lifetime of the
process, and neither can be capped (NVIDIA#2381). A test that touches both needs
two such reservations at once, which does not fit in a small virtual
address space with a large-memory GPU.

Measured per test (one pytest process each, probing both pools at session
end), only two tests touch both pools:

- graph/test_graph_memory_resource.py::test_graph_alloc used
  DeviceMemoryResource(device), the default pool, for its output buffer.
- test_object_protocols.py::test_no_cross_type_equality combined
  sample_buffer (Device.allocate, default pool) with sample_alloc_node.

Both only need some device pool next to the graph pool. Use a pool capped
at POOL_SIZE instead. The object-protocol buffer fixtures now share a
capped pool fixture that is closed at teardown; on devices without memory
pool support it falls back to the device's memory resource as before.
@juenglin
juenglin force-pushed the tests-avoid-default-pool-with-graph-pool branch from 0ab7c6e to 809855b Compare November 19, 2026 00:41
@juenglin
juenglin requested a review from rwgk November 19, 2026 00:43
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 809855b

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module test Improvements or additions to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant