Skip to content

Add GraphDefinition.clone() - #3461

Open
martindurant wants to merge 5 commits into
NVIDIA:mainfrom
martindurant:graph_clone
Open

martindurant wants to merge 5 commits into
NVIDIA:mainfrom
martindurant:graph_clone

Conversation

@martindurant

Copy link
Copy Markdown
Contributor

Description

closes #2438

Allows cloning of a GraphDefinition, which duplicates the metadata/nodes, but they continue to refer
to the same underlying resources. The new graph can be mutated without changing the original.
Nodes from the original graph can be found on the clones one with .find_node() .
Resources are not cleaned up until both the source and clone have gone.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@martindurant martindurant self-assigned this Nov 13, 2026
@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Nov 13, 2026
@martindurant martindurant added enhancement Any code-related improvements feature New feature or request and removed cuda.core Everything related to the cuda.core module labels Nov 13, 2026
@martindurant martindurant added this to the cuda.core next milestone Nov 13, 2026
@martindurant
martindurant marked this pull request as ready for review November 18, 2026 16:06
@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Nov 18, 2026
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • New Features
    • Graph definitions can be cloned while retaining resources independently, and the cloned graph can be changed without affecting the original.
    • Look up a node in a clone that corresponds to a node in the source graph. Lookups reject synthetic entry nodes and invalid or destroyed source nodes.
  • Documentation
    • Added guidance on cloning graphs, node lookup constraints, and synchronizing access to shared graph hierarchies across host threads.

GraphDefinition adds public clone() and find_node() methods. Cloning copies graph structure and retains attachment resources independently. Tests cover mutation isolation, node lookup errors, resource lifetimes, and executable behavior. Documentation describes the APIs and graph concurrency requirements.

Changes: GraphDefinition cloning

Layer / File(s) Summary
Clone and node lookup API
cuda_core/cuda/core/graph/_graph_definition.pyi, cuda_core/cuda/core/graph/_graph_definition.pyx
Adds clone() and find_node(). The implementation clones the CUDA graph, copies attachment ownership, validates lookup inputs, and handles clone cleanup if setup fails.
Clone behavior and resource lifetime tests
cuda_core/tests/graph/test_graph_definition_lifetime.py
Tests independent edge changes, node lookup errors, attachment and callback lifetimes, and resource retention by cloned executables.
API and concurrency documentation
cuda_core/docs/source/api.rst, cuda_core/docs/source/concurrency.rst
Documents clone and lookup behavior, unsupported node types, and serialization requirements for shared graph hierarchies.

Assessment against linked issues

Objective Addressed Explanation
Expose cloning and node lookup, with independent graph mutations and resource lifetimes [#2438] ✅
Test successful source-to-clone node lookup [#2438] ❌ The summarized tests cover lookup errors, but do not show a successful source-to-clone lookup.
Add release notes for the cloning API [#2438] ❌ No release-note change is included in the supplied diff summaries.

Suggested reviewers: andy-jost

Change: Feature

Merge Risk: Minimal · up to cf4bb

GraphDefinition gains clone() and find_node(), with tests and documentation. The earlier documentation gap about graphs that cannot be cloned has been fixed. No actionable merge-blocking risk remains.

Architecture risk: Low · up to cf4bb

The change affects 1 system.

Changed systems: cuda_core

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — cuda_core (service) was modified; 5 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in cuda_core/cuda/core/graph/_graph_definition.pyi: GraphDefinition adds the clone() declaration. Its docstring describes an independent copy of nodes, edges, parameters, and child graphs; shared, independently retained resources; CUDA rejection of graphs with allocation, free, or conditional nodes; propagated driver errors; and required serialization with CUDA access.
  • observed — Modified behavior in cuda_core/cuda/core/graph/_graph_definition.pyi: GraphDefinition adds find_node(original), returning the corresponding operation-specific node in the clone. Its docstring specifies errors for the synthetic entry node, invalid graphs or destroyed source nodes, and CUDA lookup failures, and requires serialization with CUDA access to either hierarchy.
  • observed — Modified behavior in cuda_core/cuda/core/graph/_graph_definition.pyx: The module now imports GN_check_valid, attach_rollback_failure, graph_clone_attachments, and invalidate_root_graph_state for node validation and clone resource/state handling.
  • observed — Modified behavior in cuda_core/cuda/core/graph/_graph_definition.pyx: GraphDefinition.clone() is added. It validates the source, clones its CUDA graph, creates a graph handle, and copies attachment ownership. On failure, it attempts to destroy any created clone; successful destruction invalidates cloned root state, while failed destruction is recorded as a rollback failure before the original exception is re-raised. The documentation describes copied graph structure and independently retained shared resources, and notes CUDA rejection of allocation, free, or conditional nodes.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 7f603e5 and cf4bb07.

Files selected for processing: 1

  • cuda_core/tests/graph/test_graph_definition_lifetime.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 320532e and 80b1628.

Files selected for processing: 5

  • cuda_core/cuda/core/graph/_graph_definition.pyi
  • cuda_core/cuda/core/graph/_graph_definition.pyx
  • cuda_core/docs/source/api.rst
  • cuda_core/docs/source/release/1.3.0-notes.rst
  • cuda_core/tests/graph/test_graph_definition_lifetime.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment on lines +198 to +201
Nodes, edges, parameters, and nested child graphs are copied. Resources
referenced by nodes (buffers, kernels, events, and callbacks) are shared
and independently retained; their contents are not copied. Subsequent
graph mutations do not affect the other definition or its executables.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

important: Document which graphs clone() rejects.

The method help says it copies nodes, but cuGraphClone rejects otherwise valid graphs that contain allocation, free, or conditional nodes. The API guide lists these restrictions; the stub and runtime docstrings do not. Add the same limits to both docstrings so callers can check whether their graph is eligible before calling clone(). (docs.nvidia.com)

Comment thread cuda_core/docs/source/api.rst Resolved
@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module enhancement Any code-related improvements feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cuda.core: expose GraphDefinition cloning

1 participant