OneHotEncoder loses a dummy name when it matches an encoded input column #1106

Describe the bug

OneHotEncoder mishandles a generated dummy name that matches another categorical input column being encoded. The input column names are unique, and the intended output names are also unique.

On current main, the generated indicator is silently removed, or transformation raises an error if the variables are encoded in the opposite order. On narwhals-migration, transformation preserves the indicator, but get_feature_names_out() still omits its name, so set_output(transform="pandas") raises a length mismatch.

To reproduce

import pandas as pd

from feature_engine.encoding import OneHotEncoder

X = pd.DataFrame(
    {"a_b": ["x", "x", "y", "y"], "a": ["b", "c", "b", "c"]}
)
encoder = OneHotEncoder()
result = encoder.fit_transform(X)

print("learned:", encoder.encoder_dict_)
print("columns:", result.columns.tolist())
print("names:", encoder.get_feature_names_out())
print("values:", result.to_numpy().tolist())

encoder.set_output(transform="pandas").transform(X)

On main at d866312874a66364c6dee3c247e69476b478395c:

learned: {'a_b': ['x', 'y'], 'a': ['b', 'c']}
columns: ['a_b_x', 'a_b_y', 'a_c']
names: ['a_b_x', 'a_b_y', 'a_c']
values: [[1, 0, 0], [1, 0, 1], [0, 1, 0], [0, 1, 1]]

The final call also returns only those three columns. With OneHotEncoder(variables=["a", "a_b"]), the same input instead raises:

ValueError: Data must be 1-dimensional, got ndarray of shape (4, 2) instead

On narwhals-migration at 91d01de9d1592a59b1c90c8cf0f74b452513eed7, the same snippet prints:

learned: {'a_b': ['x', 'y'], 'a': ['b', 'c']}
columns: ['a_b_x', 'a_b_y', 'a_b', 'a_c']
names: ['a_b_x', 'a_b_y', 'a_c']
values: [[1, 0, 1, 0], [1, 0, 0, 1], [0, 1, 1, 0], [0, 1, 0, 1]]

Its final call raises:

ValueError: Length mismatch: Expected axis has 4 elements, new values have 3 elements

Separate native Polars checks on the migration branch reproduce the same feature-name omission; its transformed columns and values are correct.

Expected behavior

With the default drop_last=False, all four learned indicators should be returned, and get_feature_names_out() should match:

['a_b_x', 'a_b_y', 'a_b', 'a_c']

There is no final-output naming ambiguity: the original a_b is replaced by a_b_x and a_b_y, leaving a_b available for the indicator of a == "b". Pandas output should return the same four-column result without an exception.

Additional context

The transform restructuring in #1028 is already merged into narwhals-migration and fixes the main-branch transformation problems above. This report also identifies the remaining metadata problem, rather than proposing that restructuring again.

In both versions, OneHotEncoder's own _add_new_feature_names() appends the generated names and then filters the combined list against variables_. This removes the newly generated a_b along with the original input name. On main, transform() similarly concatenates the dummies before dropping the original labels. The open column-order change #1068 does not change this name filtering.

All existing tests in tests/test_encoding/test_onehot_encoder.py pass on both refs: 29 on main and 49 on the migration branch. Separate local regressions reproduce the failures above. Renaming the original a_b column to other is a passing control on both branches, including unseen/missing categories in a later batch and pandas output with a nondefault index. The checks also cover an unrelated numeric column and input preservation.

Environment

macOS arm64; Python 3.13.16, pandas 3.0.6, NumPy 2.5.3, SciPy 1.18.1 and scikit-learn 1.9.1. Migration checks additionally use Narwhals 2.27.0 and Polars 2.0.0. Imports were verified against each unchanged source checkout at the exact commits above.

This investigation and report were prepared autonomously with AI assistance; no independent human review is claimed. No library code was changed, and no full-suite, build or other-platform validation is claimed.