Accept pandas DataFrames in Python bindings #7

Accept pandas DataFrames in Python bindings

Goal

Allow the Python bindings to accept pandas DataFrames directly as input to cpplink, so users can pass their data from Python without having to write intermediate files.

The Python layer should convert the DataFrame into the native cpplink data structures, with the core C++ implementation remaining independent of pandas.

Tasks

• Define the Python API for passing one or more pandas DataFrames to cpplink.

• Implement conversion from pandas DataFrame columns to the corresponding C++ column types.

• Support the main cpplink column types, including:

  • String
  • Integer
  • Double
  • Boolean
  • Date/time, where supported

• Correctly handle missing values (NaN, pd.NA, NaT, etc.).

• Preserve column names and expose them to the cpplink schema.

• Avoid unnecessary copies where practical.

• Define behaviour for unsupported pandas dtypes.

• Provide useful Python exceptions for invalid or incompatible DataFrames.

• Add unit tests covering:

  • Different column types
  • Missing values
  • Empty DataFrames
  • Large DataFrames
  • Unsupported dtypes
  • Multiple DataFrames

• Add Python examples demonstrating the API.

• Document the conversion rules between pandas and cpplink types.

Design considerations

The binding should keep pandas-specific code at the Python/C++ interface. The cpplink core should operate on its existing native column/data structures and should not depend on pandas.

Investigate whether the conversion can use the pandas/NumPy buffer interfaces or other zero-copy mechanisms where appropriate, particularly for numerical columns, while keeping the API simple and robust.

Expected outcome

A Python user should be able to do something conceptually like:

import pandas as pd
import cpplink

df = pd.DataFrame({
    "first_name": ["John", "Jane"],
    "age": [35, 42],
    "active": [True, False],
})

linker = cpplink.Linker(schema, df)
model, estimate = linker.estimate()
predictions_df  = linker.predict(model)

without first exporting the DataFrame to CSV, Parquet, or another intermediate format.