Skip to content

Add feature to calculate the number of parameters based on FSDP2 device mesh #326

Closed
Feature
#346
Closed
Add feature to calculate the number of parameters based on FSDP2 device mesh#326
Feature
#346
Assignees
le1nux
Labels
enhancementNew feature or request
Milestone
v0.4.0

Description

@le1nux
le1nux
opened on Mar 25, 2025

So far we only support the estimation of the number of parameters for FSDP1.

See

modalities/src/modalities/util.py

Line 154 in bd3e68b

return -1 # TODO We need to fix this by calculating the correct value based on the device-mesh!

Additionally, we need test coverage for the new functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Feature

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions