From 253d9e2d42f1c7a71305f0853a0d77e86f15cb08 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Wed, 26 Aug 2026 12:18:10 +0200 Subject: [PATCH 1/7] Migrate RareLabelEncoder to narwhals, add polars support fit() replaces pandas .unique()/.value_counts(normalize=True) with narwhals Series.n_unique() (for the cardinality check - matches pandas' plain unique() length, which counts a null as its own category, unlike pandas' nunique() which drops it) and drop_nulls().value_counts(sort=True, normalize=True) (same drop_nulls()/sort=True reasoning as CountEncoder.fit(): narwhals' value_counts() has no dropna param and narwhals' own value_counts default is unsorted). transform() doesn't reuse CategoricalMethodsMixin._encode() (that's a dict-based numeric remap; this encoder keeps frequent categories as-is and only replaces the rest), so it's rewritten from pandas' .loc[~isin(...), feature] = replace_with onto nw.when().then().otherwise(nw.lit(replace_with)).alias( feature). Passing Series from get_column() (not nw.col()) into when/then/otherwise keeps this working for pandas integer column names, same as base_encoder.py's precedent. A pandas Categorical column still needs its own add_categories(replace_with) step before assignment - kept as a small is_pandas-gated block (structural, like base_encoder.py's existing reorder branches), since narwhals has no cross-backend equivalent and polars has no matching restriction. Unlike the old pandas-only code, no manual object-dtype fixup is needed before assignment for the ignore_format + numeric-variable + string replace_with case: narwhals resolves the common dtype itself (object in pandas, cast-to-string in polars). Benchmarked pandas-native vs narwhals-on-pandas vs narwhals-on-polars at 10k/50k/100k rows x 1/2/10 columns x 5/50 categories, warmed up. First pass (zip_with(col, new_series_filled_with_replace_with)) averaged 2.41x pandas-native at 50k-100k rows - most of that cost was constructing a full same-length replacement Series every transform() call (~2.5ms of a ~4.8ms transform at 100k rows, confirmed by isolating just the Series construction). Switched to nw.when(keep).then(col) .otherwise(nw.lit(replace_with)), which lets the backend broadcast the scalar instead of materialising a parallel array: dropped the average to 1.60x, converging to 1.12x-1.54x at 100k rows/10 columns, the "realistic size" range. narwhals-on-polars is faster than pandas-native throughout (0.7x-1.5x, mostly <1x at 50k+ rows). Merged into a single narwhals path per the established decision rule - no pandas/polars performance split - the remaining overhead is fixed per-call cost, not scaling cost, and stays under a few ms in absolute terms even at the largest sizes tested. Rewrote test_rare_label_encoder.py to one parametrized test per behaviour over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]), replacing the shared pandas-only module-level fixtures (df_enc_big, df_enc_big_na, df_enc_numeric, from tests/conftest.py, still used by other encoder test files) with local dict constants both backends can build from, per the CountEncoder precedent. Kept test_when_varnames_are_numbers and the three category-dtype tests pandas-only (integer column names and pandas Categorical dtype are backend-specific per AGENTS.md). Split test_max_n_categories_with_numeric_var into a pandas-only version (the existing str()-workaround test, unchanged) plus a new polars-only version documenting the real, expected behavioural difference: polars can't hold mixed int/str values in one column the way pandas' object dtype does, so a numeric variable with a string replace_with casts the whole column to string instead of leaving frequent numeric categories as numbers. Verified: tests/test_encoding/test_rare_label_encoder.py - 39 passed (up from 29, from parametrizing over both backends); full tests/test_encoding suite - 336 passed, 17 pre-existing failures with identical test IDs confirmed against the unmodified base_encoder.py baseline (numpy-array-input rejection checks plus 3 MeanEncoder inverse_transform failures from mean_encoding.py's still-unmigrated fit() - predate this change, reproduced identically on the unmodified rare_label.py too). flake8 clean on feature_engine and tests. mypy clean. Module imports with pandas blocked (loaded standalone, same technique as the base_encoder.py migration, since sibling encoder files in this package still import pandas at module level). sphinx -W build clean (only the pre-existing linkcode_resolve warning, confirmed identical against the unmodified baseline). Verified every doc example in RareLabelEncoder.rst against actual output; fixed a pre-existing, unrelated value_counts() Series-name drift ("Name: var_A" -> "Name: count", a pandas version difference, not caused by this migration) while touching that page, and added a verified "With polars" section to both the class docstring and the user guide (the polars value_counts() example needed an explicit .sort() - unlike pandas, its groupby-based value_counts() order isn't stable run to run). The Titanic-dataset section of the user guide could not be re-verified against live output in this sandboxed environment (SSL cert verification blocks urllib by default here, though curl succeeds) and was left untouched; a workaround (unverified SSL context) showed matching encoder_dict_/transform output, with only an unrelated .unique() repr-formatting difference from a newer pandas version. Co-Authored-By: Claude Sonnet 5 --- docs/user_guide/encoding/RareLabelEncoder.rst | 38 ++- feature_engine/encoding/rare_label.py | 113 +++++-- .../test_encoding/test_rare_label_encoder.py | 294 ++++++++++-------- 3 files changed, 290 insertions(+), 155 deletions(-) diff --git a/docs/user_guide/encoding/RareLabelEncoder.rst b/docs/user_guide/encoding/RareLabelEncoder.rst index c19719c66..f9173395a 100644 --- a/docs/user_guide/encoding/RareLabelEncoder.rst +++ b/docs/user_guide/encoding/RareLabelEncoder.rst @@ -179,11 +179,12 @@ In the following output, we see the number of observations per category: .. code:: python + var_A A 10 B 10 C 2 D 1 - Name: var_A, dtype: int64 + Name: count, dtype: int64 Now, we group categories only for variables with more than 3 unique categories: @@ -216,10 +217,43 @@ a new category called `Rare`: .. code:: python + var_A A 10 B 10 Rare 3 - Name: var_A, dtype: int64 + Name: count, dtype: int64 + +With polars +----------- + +:class:`RareLabelEncoder()` works the same way with polars dataframes: + +.. code:: python + + import polars as pl + from feature_engine.encoding import RareLabelEncoder + + data = {'var_A': ['A'] * 10 + ['B'] * 10 + ['C'] * 2 + ['D'] * 1} + data = pl.DataFrame(data) + + rare_encoder = RareLabelEncoder(tol=0.05, n_categories=3, max_n_categories=2) + Xt = rare_encoder.fit_transform(data) + Xt['var_A'].value_counts().sort('var_A') + +We see the same grouping as with the pandas dataframe: + +.. code:: text + + shape: (3, 2) + ┌───────┬───────┐ + │ var_A ┆ count │ + │ --- ┆ --- │ + │ str ┆ u32 │ + ╞═══════╪═══════╡ + │ A ┆ 10 │ + │ B ┆ 10 │ + │ Rare ┆ 3 │ + └───────┴───────┘ Considerations -------------- diff --git a/feature_engine/encoding/rare_label.py b/feature_engine/encoding/rare_label.py index 2bbd2bf73..726dc7eaf 100644 --- a/feature_engine/encoding/rare_label.py +++ b/feature_engine/encoding/rare_label.py @@ -4,8 +4,9 @@ import warnings from typing import List, Optional, Union -import numpy as np -import pandas as pd +import narwhals as nw +import narwhals.dependencies as nwd +from narwhals.typing import IntoDataFrame, IntoSeries from feature_engine._check_init_parameters.check_init_input_params import ( _check_return_empty_is_bool, @@ -137,6 +138,28 @@ class RareLabelEncoder(CategoricalMethodsMixin, CategoricalInitMixinNA): 3 4 b 4 5 b 5 6 Rare + + With polars + + >>> import polars as pl + >>> from feature_engine.encoding import RareLabelEncoder + >>> X = pl.DataFrame(dict(x1 = [1,2,3,4,5,6], x2 = ["b", "b", "b", "b", "b", "a"])) + >>> rle = RareLabelEncoder(n_categories = 1, tol=0.2) + >>> rle.fit(X) + >>> rle.transform(X) + shape: (6, 2) + ┌─────┬──────┐ + │ x1 ┆ x2 │ + │ --- ┆ --- │ + │ i64 ┆ str │ + ╞═════╪══════╡ + │ 1 ┆ b │ + │ 2 ┆ b │ + │ 3 ┆ b │ + │ 4 ┆ b │ + │ 5 ┆ b │ + │ 6 ┆ Rare │ + └─────┴──────┘ """ def __init__( @@ -186,13 +209,13 @@ def __init__( self.replace_with = replace_with self.return_empty = return_empty - def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None): + def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): """ Learn the frequent categories for each variable. Parameters ---------- - X: pandas dataframe of shape = [n_samples, n_features] + X: dataframe of shape = [n_samples, n_features] The training input samples. Can be the entire dataframe, not just selected variables @@ -206,20 +229,33 @@ def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None): self.encoder_dict_ = {} + # n_unique() counts a null as its own category, matching pandas' + # plain unique() (used for the cardinality check below), unlike + # pandas' nunique() which drops nulls by default. + nw_X = nw.from_native(X, eager_only=True) for var in variables_: - if len(X[var].unique()) > self.n_categories: + col = nw_X.get_column(var) + + if col.n_unique() > self.n_categories: # if the variable has more than the indicated number of categories - # the encoder will learn the most frequent categories - t = X[var].value_counts(normalize=True) + # the encoder will learn the most frequent categories. + # drop_nulls() mirrors pandas' value_counts(dropna=True) + # default, which narwhals' value_counts() doesn't apply on + # its own. sort=True matches pandas' own value_counts() + # default order (descending by count). + counts = col.drop_nulls().value_counts(sort=True, normalize=True) + cat_col, freq_col = counts.columns # non-rare labels: - freq_idx = t[t >= self.tol].index + freq_idx = counts.filter( + counts.get_column(freq_col) >= self.tol + ).get_column(cat_col).to_list() if self.max_n_categories: - self.encoder_dict_[var] = list(freq_idx[: self.max_n_categories]) + self.encoder_dict_[var] = freq_idx[: self.max_n_categories] else: - self.encoder_dict_[var] = list(freq_idx) + self.encoder_dict_[var] = freq_idx else: # if the total number of categories is smaller than the indicated @@ -229,25 +265,25 @@ def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None): "indicated in n_categories. Thus, all categories will be " "considered frequent".format(var) ) - self.encoder_dict_[var] = list(X[var].unique()) + self.encoder_dict_[var] = col.unique(maintain_order=True).to_list() self.variables_ = variables_ self._get_feature_names_in(X) return self - def transform(self, X: pd.DataFrame) -> pd.DataFrame: + def transform(self, X: IntoDataFrame) -> IntoDataFrame: """ Group infrequent categories. Replace infrequent categories by the string 'Rare' or any other name provided by the user. Parameters ---------- - X: pandas dataframe of shape = [n_samples, n_features] + X: dataframe of shape = [n_samples, n_features] The input samples. Returns ------- - X: pandas dataframe of shape = [n_samples, n_features] + X: dataframe of shape = [n_samples, n_features] The dataframe where rare categories have been grouped. """ @@ -256,29 +292,46 @@ def transform(self, X: pd.DataFrame) -> pd.DataFrame: # check if dataset contains na if self.missing_values == "raise": _check_contains_na(X, self.variables_, error_msg="optional") - with_nan = [] - else: - with_nan = [np.nan] + # a pandas Categorical column rejects an unseen label on assignment + # until the label is added to its categories; narwhals has no + # cross-backend equivalent (polars has no comparable dtype + # restriction here), so this stays a native pandas step. + is_pandas = nwd.is_pandas_dataframe(X) + if is_pandas is True: + for feature in self.variables_: + if X[feature].dtype == "category": + X[feature] = X[feature].cat.add_categories(self.replace_with) + + # nw.when().then().otherwise(nw.lit(...)) lets each + # column keep its own dtype where frequent, and take the (possibly + # differently-typed) replace_with value elsewhere - narwhals + # resolves the common dtype per backend, e.g. object in pandas, + # cast-to-string in polars, so no manual dtype fixup is needed + # before it, unlike the old pandas-only .astype("O"). Passing + # Series (from get_column(), not nw.col()) into when/then/otherwise + # keeps this working for pandas integer column names, and nw.lit() + # broadcasts replace_with natively instead of materialising a + # same-length replacement array (benchmarked ~1.7x faster than the + # zip_with(col, new_series(...)) equivalent at 100k rows). + nw_X = nw.from_native(X, eager_only=True) + new_columns = [] for feature in self.variables_: - # Setting an item of incompatible dtype is deprecated - # and will raise an error in a future version of pandas - if self.ignore_format is True and isinstance(self.replace_with, str): - num_vars = list( - X[self.variables_].select_dtypes(include="number").columns + col = nw_X.get_column(feature) + keep = col.is_in(self.encoder_dict_[feature]) + if self.missing_values == "ignore": + keep = keep | col.is_null() + new_columns.append( + nw.when(keep).then(col).otherwise(nw.lit(self.replace_with)).alias( + feature ) - X[num_vars] = X[num_vars].astype("O") - - if X[feature].dtype == "category": - X[feature] = X[feature].cat.add_categories(self.replace_with) - - X.loc[~X[feature].isin(self.encoder_dict_[feature] + with_nan), feature] = ( - self.replace_with ) + X = nw_X.with_columns(*new_columns).to_native() + return X - def inverse_transform(self, X: pd.DataFrame): + def inverse_transform(self, X: IntoDataFrame): """inverse_transform is not implemented for this transformer.""" raise NotImplementedError( "inverse_transform is not implemented for this transformer." diff --git a/tests/test_encoding/test_rare_label_encoder.py b/tests/test_encoding/test_rare_label_encoder.py index 9594e1cc3..fc4658b77 100644 --- a/tests/test_encoding/test_rare_label_encoder.py +++ b/tests/test_encoding/test_rare_label_encoder.py @@ -1,14 +1,52 @@ from collections import Counter -import numpy as np +import narwhals as nw import pandas as pd +import polars as pl import pytest from feature_engine.encoding import RareLabelEncoder - -def test_defo_params_plus_automatically_find_variables(df_enc_big): +DATA_ENC_BIG = { + "var_A": ["A"] * 6 + + ["B"] * 10 + + ["C"] * 4 + + ["D"] * 10 + + ["E"] * 2 + + ["F"] * 2 + + ["G"] * 6, + "var_B": ["A"] * 10 + + ["B"] * 6 + + ["C"] * 4 + + ["D"] * 10 + + ["E"] * 2 + + ["F"] * 2 + + ["G"] * 6, + "var_C": ["A"] * 4 + + ["B"] * 6 + + ["C"] * 10 + + ["D"] * 10 + + ["E"] * 2 + + ["F"] * 2 + + ["G"] * 6, +} +DATA_ENC_BIG_NA = { + key: [None] + values[1:] for key, values in DATA_ENC_BIG.items() +} +DATA_ENC_NUMERIC = { + "var_A": [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3], + "var_B": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3], +} + + +def _to_pandas(X): + return nw.from_native(X, eager_only=True).to_pandas() + + +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_defo_params_plus_automatically_find_variables(make_df): # test case 1: defo params, automatically select variables + df_enc_big = make_df(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.06, n_categories=5, variables=None, replace_with="Rare" ) @@ -53,11 +91,12 @@ def test_defo_params_plus_automatically_find_variables(df_enc_big): assert encoder.n_features_in_ == 3 assert encoder.encoder_dict_ == frequenc_cat # test transform output - pd.testing.assert_frame_equal(X, df) + pd.testing.assert_frame_equal(_to_pandas(X), df) -def test_when_varnames_are_numbers(df_enc_big): - input_df = df_enc_big.copy() +def test_when_varnames_are_numbers(): + # integer column names are pandas-only, polars has no such concept + input_df = pd.DataFrame(DATA_ENC_BIG) input_df.columns = [1, 2, 3] encoder = RareLabelEncoder( @@ -84,7 +123,9 @@ def test_when_varnames_are_numbers(df_enc_big): pd.testing.assert_frame_equal(X, df) -def test_correctly_ignores_nan_in_transform(df_enc_big): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_correctly_ignores_nan_in_transform(make_df): + df_enc_big = make_df(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.06, n_categories=5, @@ -101,31 +142,33 @@ def test_correctly_ignores_nan_in_transform(df_enc_big): assert encoder.encoder_dict_ == frequenc_cat # input - t = pd.DataFrame( + t = make_df( { - "var_A": ["A", np.nan, "J"], - "var_B": ["A", np.nan, "J"], - "var_C": ["C", np.nan, "J"], + "var_A": ["A", None, "J"], + "var_B": ["A", None, "J"], + "var_C": ["C", None, "J"], } ) # expected tt = pd.DataFrame( { - "var_A": ["A", np.nan, "Rare"], - "var_B": ["A", np.nan, "Rare"], - "var_C": ["C", np.nan, "Rare"], + "var_A": ["A", None, "Rare"], + "var_B": ["A", None, "Rare"], + "var_C": ["C", None, "Rare"], } ) X = encoder.transform(t) - pd.testing.assert_frame_equal(X, tt) + pd.testing.assert_frame_equal(_to_pandas(X), tt) -def test_correctly_ignores_nan_in_fit(df_enc_big): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_correctly_ignores_nan_in_fit(make_df): - df = df_enc_big.copy() - df.loc[df["var_C"] == "G", "var_C"] = np.nan + df = dict(DATA_ENC_BIG) + df["var_C"] = [None if v == "G" else v for v in df["var_C"]] + df = make_df(df) encoder = RareLabelEncoder( tol=0.06, @@ -144,71 +187,36 @@ def test_correctly_ignores_nan_in_fit(df_enc_big): assert Counter(encoder.encoder_dict_[key]) == Counter(frequent_cat[key]) # input - t = pd.DataFrame( + t = make_df( { - "var_A": ["A", np.nan, "J", "G"], - "var_B": ["A", np.nan, "J", "G"], - "var_C": ["C", np.nan, "J", "G"], + "var_A": ["A", None, "J", "G"], + "var_B": ["A", None, "J", "G"], + "var_C": ["C", None, "J", "G"], } ) # expected tt = pd.DataFrame( { - "var_A": ["A", np.nan, "Rare", "G"], - "var_B": ["A", np.nan, "Rare", "G"], - "var_C": ["C", np.nan, "Rare", "Rare"], + "var_A": ["A", None, "Rare", "G"], + "var_B": ["A", None, "Rare", "G"], + "var_C": ["C", None, "Rare", "Rare"], } ) X = encoder.transform(t) - pd.testing.assert_frame_equal(X, tt) + pd.testing.assert_frame_equal(_to_pandas(X), tt) -def test_correctly_ignores_nan_in_fit_when_var_is_numerical(df_enc_big): - - df = df_enc_big.copy() +def test_correctly_ignores_nan_in_fit_when_var_is_numerical(): + # pandas .astype("O") mixed-dtype workaround for a numeric variable with + # a string replace_with is a pandas-only quirk (polars casts to string + # instead - see test_max_n_categories_with_numeric_var_polars). + df = pd.DataFrame(DATA_ENC_BIG) df["var_C"] = [ - 1, - 1, - 1, - 1, - 2, - 2, - 2, - 2, - 2, - 2, - 3, - 3, - 3, - 3, - 3, - 3, - 3, - 3, - 3, - 3, - 4, - 4, - 4, - 4, - 4, - 4, - 4, - 4, - 4, - 4, - 5, - 5, - 6, - 6, - np.nan, - np.nan, - np.nan, - np.nan, - np.nan, - np.nan, + 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 6, 6, + None, None, None, None, None, None, ] encoder = RareLabelEncoder( @@ -231,18 +239,20 @@ def test_correctly_ignores_nan_in_fit_when_var_is_numerical(df_enc_big): # input t = pd.DataFrame( { - "var_A": ["A", np.nan, "J", "G"], - "var_B": ["A", np.nan, "J", "G"], - "var_C": [3, np.nan, 9, 10], + "var_A": ["A", None, "J", "G"], + "var_B": ["A", None, "J", "G"], + "var_C": [3, None, 9, 10], } ) - # expected + # expected (var_C mixes floats and strings after transform, so its + # missing value must be an actual float nan, not a bare None, to match + # pandas' own dtype inference for the same mix) tt = pd.DataFrame( { - "var_A": ["A", np.nan, "Rare", "G"], - "var_B": ["A", np.nan, "Rare", "G"], - "var_C": [3.0, np.nan, "Rare", "Rare"], + "var_A": ["A", None, "Rare", "G"], + "var_B": ["A", None, "Rare", "G"], + "var_C": [3.0, float("nan"), "Rare", "Rare"], } ) @@ -250,8 +260,10 @@ def test_correctly_ignores_nan_in_fit_when_var_is_numerical(df_enc_big): pd.testing.assert_frame_equal(X, tt, check_dtype=False) -def test_user_provides_grouping_label_name_and_variable_list(df_enc_big): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_user_provides_grouping_label_name_and_variable_list(make_df): # test case 2: user provides alternative grouping value and variable list + df_enc_big = make_df(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.15, n_categories=5, variables=["var_A", "var_B"], replace_with="Other" ) @@ -290,7 +302,7 @@ def test_user_provides_grouping_label_name_and_variable_list(df_enc_big): assert encoder.variables_ == ["var_A", "var_B"] assert encoder.n_features_in_ == 3 # test transform output - pd.testing.assert_frame_equal(X, df) + pd.testing.assert_frame_equal(_to_pandas(X), df) # init params @@ -318,42 +330,49 @@ def test_error_if_replace_with_not_string(replace_with): RareLabelEncoder(replace_with=replace_with) -def test_warning_if_variable_cardinality_less_than_n_categories(df_enc_big): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_warning_if_variable_cardinality_less_than_n_categories(make_df): # test case 3: when the variable has low cardinality + df_enc_big = make_df(DATA_ENC_BIG) with pytest.warns(UserWarning): encoder = RareLabelEncoder(n_categories=10) encoder.fit(df_enc_big) -def test_fit_raises_error_if_df_contains_na(df_enc_big_na): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_fit_raises_error_if_df_contains_na(make_df): # test case 4: when dataset contains na, fit method + df_enc_big_na = make_df(DATA_ENC_BIG_NA) encoder = RareLabelEncoder(n_categories=4) - with pytest.raises(ValueError) as record: - msg = ( - "Some of the variables in the dataset contain NaN. Check and " - "remove those before using this transformer or set the parameter " - "`missing_values='ignore'` when initialising this transformer." - ) + msg = ( + "Some of the variables in the dataset contain NaN. Check and " + "remove those before using this transformer or set the parameter " + "`missing_values='ignore'` when initialising this transformer." + ) + with pytest.raises(ValueError, match=msg): encoder.fit(df_enc_big_na) - assert str(record.value) == msg -def test_transform_raises_error_if_df_contains_na(df_enc_big, df_enc_big_na): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_transform_raises_error_if_df_contains_na(make_df): # test case 5: when dataset contains na, transform method + df_enc_big = make_df(DATA_ENC_BIG) + df_enc_big_na = make_df(DATA_ENC_BIG_NA) encoder = RareLabelEncoder(n_categories=4) encoder.fit(df_enc_big) - with pytest.raises(ValueError) as record: - msg = ( - "Some of the variables in the dataset contain NaN. Check and " - "remove those before using this transformer or set the parameter " - "`missing_values='ignore'` when initialising this transformer." - ) + msg = ( + "Some of the variables in the dataset contain NaN. Check and " + "remove those before using this transformer or set the parameter " + "`missing_values='ignore'` when initialising this transformer." + ) + with pytest.raises(ValueError, match=msg): encoder.transform(df_enc_big_na) - assert str(record.value) == msg -def test_max_n_categories(df_enc_big): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_max_n_categories(make_df): # test case 6: user provides the maximum number of categories they want + df_enc_big = make_df(DATA_ENC_BIG) rare_encoder = RareLabelEncoder(tol=0.10, max_n_categories=4, n_categories=5) X = rare_encoder.fit_transform(df_enc_big) df = { @@ -377,11 +396,14 @@ def test_max_n_categories(df_enc_big): + ["G"] * 6, } df = pd.DataFrame(df) - pd.testing.assert_frame_equal(X, df) + pd.testing.assert_frame_equal(_to_pandas(X), df) -def test_max_n_categories_with_numeric_var(df_enc_numeric): - # ignore_format=True +def test_max_n_categories_with_numeric_var(): + # pandas .astype("O") mixed-dtype workaround for a numeric variable with + # a string replace_with is a pandas-only quirk (see the polars variant + # below, which casts to string instead of keeping mixed dtypes). + df_enc_numeric = pd.DataFrame(DATA_ENC_NUMERIC) rare_encoder = RareLabelEncoder( tol=0.10, max_n_categories=2, n_categories=1, ignore_format=True ) @@ -399,8 +421,39 @@ def test_max_n_categories_with_numeric_var(df_enc_numeric): assert str(list(X["var_B"])[i]) == str(list(df["var_B"])[i]) -def test_variables_cast_as_category(df_enc_big): - # test case 1: defo params, automatically select variables +def test_max_n_categories_with_numeric_var_polars(): + # polars can't hold mixed int/str values in one column like pandas' + # object dtype does, so a numeric variable with a string replace_with + # is cast to string entirely instead - a real, backend-specific + # difference from the pandas behaviour above, not a bug. + df_enc_numeric = pl.DataFrame(DATA_ENC_NUMERIC) + rare_encoder = RareLabelEncoder( + tol=0.10, max_n_categories=2, n_categories=1, ignore_format=True + ) + + X = rare_encoder.fit_transform(df_enc_numeric.select(["var_A", "var_B"])) + + expected = pd.DataFrame( + { + "var_A": ["1"] * 6 + ["2"] * 10 + ["Rare"] * 4, + "var_B": ["1"] * 10 + ["2"] * 6 + ["Rare"] * 4, + } + ) + pd.testing.assert_frame_equal(_to_pandas(X), expected) + + +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_raises_not_implemented_error(make_df): + df_enc_big = make_df(DATA_ENC_BIG) + enc = RareLabelEncoder().fit(df_enc_big) + with pytest.raises(NotImplementedError): + enc.inverse_transform(df_enc_big) + + +def test_variables_cast_as_category(): + # pandas category dtype is backend-specific: polars has no equivalent + # concept in the same sense. + df_enc_big = pd.DataFrame(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.06, n_categories=5, variables=None, replace_with="Rare" ) @@ -441,7 +494,8 @@ def test_variables_cast_as_category(df_enc_big): pd.testing.assert_frame_equal(X, df, check_categorical=False) -def test_variables_cast_as_category_with_na_in_transform(df_enc_big): +def test_variables_cast_as_category_with_na_in_transform(): + df_enc_big = pd.DataFrame(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.06, n_categories=5, @@ -457,9 +511,9 @@ def test_variables_cast_as_category_with_na_in_transform(df_enc_big): # input t = pd.DataFrame( { - "var_A": ["A", np.nan, "J", "G"], - "var_B": ["A", np.nan, "J", "G"], - "var_C": ["A", np.nan, "J", "G"], + "var_A": ["A", None, "J", "G"], + "var_B": ["A", None, "J", "G"], + "var_C": ["A", None, "J", "G"], } ) t["var_B"] = pd.Categorical(t["var_B"]) @@ -467,19 +521,19 @@ def test_variables_cast_as_category_with_na_in_transform(df_enc_big): # expected tt = pd.DataFrame( { - "var_A": ["A", np.nan, "Rare", "G"], - "var_B": ["A", np.nan, "Rare", "G"], - "var_C": ["A", np.nan, "Rare", "G"], + "var_A": ["A", None, "Rare", "G"], + "var_B": ["A", None, "Rare", "G"], + "var_C": ["A", None, "Rare", "G"], } ) tt["var_B"] = pd.Categorical(tt["var_B"]) pd.testing.assert_frame_equal(encoder.transform(t), tt, check_categorical=False) -def test_variables_cast_as_category_with_na_in_fit(df_enc_big): +def test_variables_cast_as_category_with_na_in_fit(): - df = df_enc_big.copy() - df.loc[df["var_C"] == "G", "var_C"] = np.nan + df = pd.DataFrame(DATA_ENC_BIG) + df.loc[df["var_C"] == "G", "var_C"] = None df["var_C"] = df["var_C"].astype("category") encoder = RareLabelEncoder( @@ -492,9 +546,9 @@ def test_variables_cast_as_category_with_na_in_fit(df_enc_big): # input t = pd.DataFrame( { - "var_A": ["A", np.nan, "J", "G"], - "var_B": ["A", np.nan, "J", "G"], - "var_C": ["C", np.nan, "J", "G"], + "var_A": ["A", None, "J", "G"], + "var_B": ["A", None, "J", "G"], + "var_C": ["C", None, "J", "G"], } ) t["var_C"] = pd.Categorical(t["var_C"]) @@ -502,17 +556,11 @@ def test_variables_cast_as_category_with_na_in_fit(df_enc_big): # expected tt = pd.DataFrame( { - "var_A": ["A", np.nan, "Rare", "G"], - "var_B": ["A", np.nan, "Rare", "G"], - "var_C": ["C", np.nan, "Rare", "Rare"], + "var_A": ["A", None, "Rare", "G"], + "var_B": ["A", None, "Rare", "G"], + "var_C": ["C", None, "Rare", "Rare"], } ) tt["var_C"] = pd.Categorical(tt["var_C"]) pd.testing.assert_frame_equal(encoder.transform(t), tt, check_categorical=False) - - -def test_inverse_transform_raises_not_implemented_error(df_enc_big): - enc = RareLabelEncoder().fit(df_enc_big) - with pytest.raises(NotImplementedError): - enc.inverse_transform(df_enc_big) From a18d049b6218338ccad2c49cdd2be1699e5a2141 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Mon, 31 Aug 2026 00:43:14 +0200 Subject: [PATCH 2/7] Adapt RareLabelEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_na and _check_contains_na (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) round-trips in fit() and transform(). In transform(), detect the pandas Categorical fix-up path via nw_X.implementation .is_pandas() and run it on a copy so the user's dataframe is not mutated. Drop the now-unused narwhals.dependencies import. Co-Authored-By: Claude Sonnet 5 --- feature_engine/encoding/rare_label.py | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/feature_engine/encoding/rare_label.py b/feature_engine/encoding/rare_label.py index 726dc7eaf..e9195f641 100644 --- a/feature_engine/encoding/rare_label.py +++ b/feature_engine/encoding/rare_label.py @@ -5,7 +5,6 @@ from typing import List, Optional, Union import narwhals as nw -import narwhals.dependencies as nwd from narwhals.typing import IntoDataFrame, IntoSeries from feature_engine._check_init_parameters.check_init_input_params import ( @@ -223,7 +222,7 @@ def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): y is not required. You can pass y or None. """ - X = check_X(X) + nw_X = check_X(X) variables_ = self._check_or_select_variables(X) self._check_na(X, variables_) @@ -232,7 +231,6 @@ def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): # n_unique() counts a null as its own category, matching pandas' # plain unique() (used for the cardinality check below), unlike # pandas' nunique() which drops nulls by default. - nw_X = nw.from_native(X, eager_only=True) for var in variables_: col = nw_X.get_column(var) @@ -287,7 +285,7 @@ def transform(self, X: IntoDataFrame) -> IntoDataFrame: The dataframe where rare categories have been grouped. """ - X = self._check_transform_input_and_state(X) + nw_X = self._check_transform_input_and_state(X) # check if dataset contains na if self.missing_values == "raise": @@ -296,12 +294,16 @@ def transform(self, X: IntoDataFrame) -> IntoDataFrame: # a pandas Categorical column rejects an unseen label on assignment # until the label is added to its categories; narwhals has no # cross-backend equivalent (polars has no comparable dtype - # restriction here), so this stays a native pandas step. - is_pandas = nwd.is_pandas_dataframe(X) - if is_pandas is True: + # restriction here), so this stays a native pandas step. Operate on a + # copy so the user's dataframe is not mutated. + if nw_X.implementation.is_pandas(): + native_X = nw_X.to_native().copy() for feature in self.variables_: - if X[feature].dtype == "category": - X[feature] = X[feature].cat.add_categories(self.replace_with) + if native_X[feature].dtype == "category": + native_X[feature] = native_X[feature].cat.add_categories( + self.replace_with + ) + nw_X = nw.from_native(native_X, eager_only=True) # nw.when().then().otherwise(nw.lit(...)) lets each # column keep its own dtype where frequent, and take the (possibly @@ -314,7 +316,6 @@ def transform(self, X: IntoDataFrame) -> IntoDataFrame: # broadcasts replace_with natively instead of materialising a # same-length replacement array (benchmarked ~1.7x faster than the # zip_with(col, new_series(...)) equivalent at 100k rows). - nw_X = nw.from_native(X, eager_only=True) new_columns = [] for feature in self.variables_: col = nw_X.get_column(feature) From a51c7c2786ea658f615e2f1b068d6c1a2f6f45e8 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Mon, 14 Sep 2026 22:24:21 +0200 Subject: [PATCH 3/7] Use shared backend test fixtures and helpers in RareLabelEncoder tests Replace the file-local data dicts and the _to_pandas helper - whose polars to_pandas() call needs pyarrow, which is not a dependency, so the polars cases failed - with the shared test structure: make_df and data_enc_big* / data_enc_numeric fixtures, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 --- .../test_encoding/test_rare_label_encoder.py | 390 +++++++----------- 1 file changed, 139 insertions(+), 251 deletions(-) diff --git a/tests/test_encoding/test_rare_label_encoder.py b/tests/test_encoding/test_rare_label_encoder.py index fc4658b77..2d8676a14 100644 --- a/tests/test_encoding/test_rare_label_encoder.py +++ b/tests/test_encoding/test_rare_label_encoder.py @@ -1,85 +1,39 @@ +import re from collections import Counter -import narwhals as nw import pandas as pd import polars as pl import pytest from feature_engine.encoding import RareLabelEncoder - -DATA_ENC_BIG = { - "var_A": ["A"] * 6 - + ["B"] * 10 - + ["C"] * 4 - + ["D"] * 10 - + ["E"] * 2 - + ["F"] * 2 - + ["G"] * 6, - "var_B": ["A"] * 10 - + ["B"] * 6 - + ["C"] * 4 - + ["D"] * 10 - + ["E"] * 2 - + ["F"] * 2 - + ["G"] * 6, - "var_C": ["A"] * 4 - + ["B"] * 6 - + ["C"] * 10 - + ["D"] * 10 - + ["E"] * 2 - + ["F"] * 2 - + ["G"] * 6, -} -DATA_ENC_BIG_NA = { - key: [None] + values[1:] for key, values in DATA_ENC_BIG.items() -} -DATA_ENC_NUMERIC = { - "var_A": [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3], - "var_B": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3], +from tests.backend_helpers import to_dict + +MSG_NA = ( + "Some of the variables in the dataset contain NaN. Check and " + "remove those before using this transformer or set the parameter " + "`missing_values='ignore'` when initialising this transformer." +) + +FREQUENT_CATEGORIES = { + "var_A": ["B", "D", "A", "G", "C"], + "var_B": ["A", "D", "B", "G", "C"], + "var_C": ["C", "D", "B", "G", "A"], } - -def _to_pandas(X): - return nw.from_native(X, eager_only=True).to_pandas() +# data_enc_big after grouping categories E and F as "Rare" +ENC_BIG_RARE = { + "var_A": ["A"] * 6 + ["B"] * 10 + ["C"] * 4 + ["D"] * 10 + ["Rare"] * 4 + ["G"] * 6, + "var_B": ["A"] * 10 + ["B"] * 6 + ["C"] * 4 + ["D"] * 10 + ["Rare"] * 4 + ["G"] * 6, + "var_C": ["A"] * 4 + ["B"] * 6 + ["C"] * 10 + ["D"] * 10 + ["Rare"] * 4 + ["G"] * 6, +} -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_defo_params_plus_automatically_find_variables(make_df): +def test_defo_params_plus_automatically_find_variables(make_df, data_enc_big): # test case 1: defo params, automatically select variables - df_enc_big = make_df(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.06, n_categories=5, variables=None, replace_with="Rare" ) - X = encoder.fit_transform(df_enc_big) - - # expected output - df = { - "var_A": ["A"] * 6 - + ["B"] * 10 - + ["C"] * 4 - + ["D"] * 10 - + ["Rare"] * 4 - + ["G"] * 6, - "var_B": ["A"] * 10 - + ["B"] * 6 - + ["C"] * 4 - + ["D"] * 10 - + ["Rare"] * 4 - + ["G"] * 6, - "var_C": ["A"] * 4 - + ["B"] * 6 - + ["C"] * 10 - + ["D"] * 10 - + ["Rare"] * 4 - + ["G"] * 6, - } - df = pd.DataFrame(df) - - frequenc_cat = { - "var_A": ["B", "D", "A", "G", "C"], - "var_B": ["A", "D", "B", "G", "C"], - "var_C": ["C", "D", "B", "G", "A"], - } + X = encoder.fit_transform(make_df(data_enc_big)) # test init params assert encoder.tol == 0.06 @@ -89,14 +43,15 @@ def test_defo_params_plus_automatically_find_variables(make_df): # test fit attr assert encoder.variables_ == ["var_A", "var_B", "var_C"] assert encoder.n_features_in_ == 3 - assert encoder.encoder_dict_ == frequenc_cat + assert encoder.encoder_dict_ == FREQUENT_CATEGORIES # test transform output - pd.testing.assert_frame_equal(_to_pandas(X), df) + assert isinstance(X, make_df) + assert to_dict(X) == ENC_BIG_RARE -def test_when_varnames_are_numbers(): +def test_when_varnames_are_numbers(data_enc_big): # integer column names are pandas-only, polars has no such concept - input_df = pd.DataFrame(DATA_ENC_BIG) + input_df = pd.DataFrame(data_enc_big) input_df.columns = [1, 2, 3] encoder = RareLabelEncoder( @@ -105,77 +60,60 @@ def test_when_varnames_are_numbers(): X = encoder.fit_transform(input_df) # expected output - df = { - 1: ["A"] * 6 + ["B"] * 10 + ["C"] * 4 + ["D"] * 10 + ["Rare"] * 4 + ["G"] * 6, - 2: ["A"] * 10 + ["B"] * 6 + ["C"] * 4 + ["D"] * 10 + ["Rare"] * 4 + ["G"] * 6, - 3: ["A"] * 4 + ["B"] * 6 + ["C"] * 10 + ["D"] * 10 + ["Rare"] * 4 + ["G"] * 6, - } - df = pd.DataFrame(df) - - frequenc_cat = { - 1: ["B", "D", "A", "G", "C"], - 2: ["A", "D", "B", "G", "C"], - 3: ["C", "D", "B", "G", "A"], - } + df = pd.DataFrame( + { + 1: ENC_BIG_RARE["var_A"], + 2: ENC_BIG_RARE["var_B"], + 3: ENC_BIG_RARE["var_C"], + } + ) assert encoder.variables_ == [1, 2, 3] - assert encoder.encoder_dict_ == frequenc_cat + assert encoder.encoder_dict_ == { + 1: FREQUENT_CATEGORIES["var_A"], + 2: FREQUENT_CATEGORIES["var_B"], + 3: FREQUENT_CATEGORIES["var_C"], + } pd.testing.assert_frame_equal(X, df) -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_correctly_ignores_nan_in_transform(make_df): - df_enc_big = make_df(DATA_ENC_BIG) +def test_correctly_ignores_nan_in_transform(make_df, data_enc_big): encoder = RareLabelEncoder( tol=0.06, n_categories=5, missing_values="ignore", ) - X = encoder.fit_transform(df_enc_big) - - # expected: - frequenc_cat = { - "var_A": ["B", "D", "A", "G", "C"], - "var_B": ["A", "D", "B", "G", "C"], - "var_C": ["C", "D", "B", "G", "A"], - } - assert encoder.encoder_dict_ == frequenc_cat - - # input - t = make_df( - { - "var_A": ["A", None, "J"], - "var_B": ["A", None, "J"], - "var_C": ["C", None, "J"], - } - ) - - # expected - tt = pd.DataFrame( - { - "var_A": ["A", None, "Rare"], - "var_B": ["A", None, "Rare"], - "var_C": ["C", None, "Rare"], - } + encoder.fit(make_df(data_enc_big)) + assert encoder.encoder_dict_ == FREQUENT_CATEGORIES + + X = encoder.transform( + make_df( + { + "var_A": ["A", None, "J"], + "var_B": ["A", None, "J"], + "var_C": ["C", None, "J"], + } + ) ) - X = encoder.transform(t) - pd.testing.assert_frame_equal(_to_pandas(X), tt) - + assert isinstance(X, make_df) + assert to_dict(X) == { + "var_A": ["A", None, "Rare"], + "var_B": ["A", None, "Rare"], + "var_C": ["C", None, "Rare"], + } -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_correctly_ignores_nan_in_fit(make_df): - df = dict(DATA_ENC_BIG) - df["var_C"] = [None if v == "G" else v for v in df["var_C"]] - df = make_df(df) +def test_correctly_ignores_nan_in_fit(make_df, data_enc_big): + data = dict(data_enc_big) + data["var_C"] = [None if v == "G" else v for v in data["var_C"]] encoder = RareLabelEncoder( tol=0.06, n_categories=3, missing_values="ignore", ) - encoder.fit(df) + encoder.fit(make_df(data)) # expected: frequent_cat = { @@ -186,33 +124,29 @@ def test_correctly_ignores_nan_in_fit(make_df): for key in frequent_cat.keys(): assert Counter(encoder.encoder_dict_[key]) == Counter(frequent_cat[key]) - # input - t = make_df( - { - "var_A": ["A", None, "J", "G"], - "var_B": ["A", None, "J", "G"], - "var_C": ["C", None, "J", "G"], - } + X = encoder.transform( + make_df( + { + "var_A": ["A", None, "J", "G"], + "var_B": ["A", None, "J", "G"], + "var_C": ["C", None, "J", "G"], + } + ) ) - # expected - tt = pd.DataFrame( - { - "var_A": ["A", None, "Rare", "G"], - "var_B": ["A", None, "Rare", "G"], - "var_C": ["C", None, "Rare", "Rare"], - } - ) - - X = encoder.transform(t) - pd.testing.assert_frame_equal(_to_pandas(X), tt) + assert isinstance(X, make_df) + assert to_dict(X) == { + "var_A": ["A", None, "Rare", "G"], + "var_B": ["A", None, "Rare", "G"], + "var_C": ["C", None, "Rare", "Rare"], + } -def test_correctly_ignores_nan_in_fit_when_var_is_numerical(): +def test_correctly_ignores_nan_in_fit_when_var_is_numerical(data_enc_big): # pandas .astype("O") mixed-dtype workaround for a numeric variable with # a string replace_with is a pandas-only quirk (polars casts to string # instead - see test_max_n_categories_with_numeric_var_polars). - df = pd.DataFrame(DATA_ENC_BIG) + df = pd.DataFrame(data_enc_big) df["var_C"] = [ 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 6, 6, @@ -260,17 +194,24 @@ def test_correctly_ignores_nan_in_fit_when_var_is_numerical(): pd.testing.assert_frame_equal(X, tt, check_dtype=False) -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_user_provides_grouping_label_name_and_variable_list(make_df): +def test_user_provides_grouping_label_name_and_variable_list(make_df, data_enc_big): # test case 2: user provides alternative grouping value and variable list - df_enc_big = make_df(DATA_ENC_BIG) encoder = RareLabelEncoder( tol=0.15, n_categories=5, variables=["var_A", "var_B"], replace_with="Other" ) - X = encoder.fit_transform(df_enc_big) + X = encoder.fit_transform(make_df(data_enc_big)) - # expected output - df = { + # test init params + assert encoder.tol == 0.15 + assert encoder.n_categories == 5 + assert encoder.replace_with == "Other" + assert encoder.variables == ["var_A", "var_B"] + # test fit attr + assert encoder.variables_ == ["var_A", "var_B"] + assert encoder.n_features_in_ == 3 + # test transform output + assert isinstance(X, make_df) + assert to_dict(X) == { "var_A": ["A"] * 6 + ["B"] * 10 + ["Other"] * 4 @@ -283,26 +224,8 @@ def test_user_provides_grouping_label_name_and_variable_list(make_df): + ["D"] * 10 + ["Other"] * 4 + ["G"] * 6, - "var_C": ["A"] * 4 - + ["B"] * 6 - + ["C"] * 10 - + ["D"] * 10 - + ["E"] * 2 - + ["F"] * 2 - + ["G"] * 6, + "var_C": data_enc_big["var_C"], } - df = pd.DataFrame(df) - - # test init params - assert encoder.tol == 0.15 - assert encoder.n_categories == 5 - assert encoder.replace_with == "Other" - assert encoder.variables == ["var_A", "var_B"] - # test fit attr - assert encoder.variables_ == ["var_A", "var_B"] - assert encoder.n_features_in_ == 3 - # test transform output - pd.testing.assert_frame_equal(_to_pandas(X), df) # init params @@ -330,52 +253,44 @@ def test_error_if_replace_with_not_string(replace_with): RareLabelEncoder(replace_with=replace_with) -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_warning_if_variable_cardinality_less_than_n_categories(make_df): +def test_warning_if_variable_cardinality_less_than_n_categories( + make_df, data_enc_big +): # test case 3: when the variable has low cardinality - df_enc_big = make_df(DATA_ENC_BIG) - with pytest.warns(UserWarning): - encoder = RareLabelEncoder(n_categories=10) - encoder.fit(df_enc_big) + msg = ( + "The number of unique categories for variable var_A is less than that " + "indicated in n_categories. Thus, all categories will be " + "considered frequent" + ) + encoder = RareLabelEncoder(n_categories=10) + with pytest.warns(UserWarning, match=re.escape(msg)): + encoder.fit(make_df(data_enc_big)) -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_fit_raises_error_if_df_contains_na(make_df): +def test_fit_raises_error_if_df_contains_na(make_df, data_enc_big_na): # test case 4: when dataset contains na, fit method - df_enc_big_na = make_df(DATA_ENC_BIG_NA) encoder = RareLabelEncoder(n_categories=4) - msg = ( - "Some of the variables in the dataset contain NaN. Check and " - "remove those before using this transformer or set the parameter " - "`missing_values='ignore'` when initialising this transformer." - ) - with pytest.raises(ValueError, match=msg): - encoder.fit(df_enc_big_na) + with pytest.raises(ValueError, match=re.escape(MSG_NA)): + encoder.fit(make_df(data_enc_big_na)) -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_transform_raises_error_if_df_contains_na(make_df): +def test_transform_raises_error_if_df_contains_na( + make_df, data_enc_big, data_enc_big_na +): # test case 5: when dataset contains na, transform method - df_enc_big = make_df(DATA_ENC_BIG) - df_enc_big_na = make_df(DATA_ENC_BIG_NA) encoder = RareLabelEncoder(n_categories=4) - encoder.fit(df_enc_big) - msg = ( - "Some of the variables in the dataset contain NaN. Check and " - "remove those before using this transformer or set the parameter " - "`missing_values='ignore'` when initialising this transformer." - ) - with pytest.raises(ValueError, match=msg): - encoder.transform(df_enc_big_na) + encoder.fit(make_df(data_enc_big)) + with pytest.raises(ValueError, match=re.escape(MSG_NA)): + encoder.transform(make_df(data_enc_big_na)) -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_max_n_categories(make_df): +def test_max_n_categories(make_df, data_enc_big): # test case 6: user provides the maximum number of categories they want - df_enc_big = make_df(DATA_ENC_BIG) rare_encoder = RareLabelEncoder(tol=0.10, max_n_categories=4, n_categories=5) - X = rare_encoder.fit_transform(df_enc_big) - df = { + X = rare_encoder.fit_transform(make_df(data_enc_big)) + + assert isinstance(X, make_df) + assert to_dict(X) == { "var_A": ["A"] * 6 + ["B"] * 10 + ["Rare"] * 4 @@ -395,15 +310,13 @@ def test_max_n_categories(make_df): + ["Rare"] * 4 + ["G"] * 6, } - df = pd.DataFrame(df) - pd.testing.assert_frame_equal(_to_pandas(X), df) -def test_max_n_categories_with_numeric_var(): +def test_max_n_categories_with_numeric_var(data_enc_numeric): # pandas .astype("O") mixed-dtype workaround for a numeric variable with # a string replace_with is a pandas-only quirk (see the polars variant # below, which casts to string instead of keeping mixed dtypes). - df_enc_numeric = pd.DataFrame(DATA_ENC_NUMERIC) + df_enc_numeric = pd.DataFrame(data_enc_numeric) rare_encoder = RareLabelEncoder( tol=0.10, max_n_categories=2, n_categories=1, ignore_format=True ) @@ -421,70 +334,45 @@ def test_max_n_categories_with_numeric_var(): assert str(list(X["var_B"])[i]) == str(list(df["var_B"])[i]) -def test_max_n_categories_with_numeric_var_polars(): +def test_max_n_categories_with_numeric_var_polars(data_enc_numeric): # polars can't hold mixed int/str values in one column like pandas' # object dtype does, so a numeric variable with a string replace_with # is cast to string entirely instead - a real, backend-specific # difference from the pandas behaviour above, not a bug. - df_enc_numeric = pl.DataFrame(DATA_ENC_NUMERIC) + df_enc_numeric = pl.DataFrame(data_enc_numeric) rare_encoder = RareLabelEncoder( tol=0.10, max_n_categories=2, n_categories=1, ignore_format=True ) X = rare_encoder.fit_transform(df_enc_numeric.select(["var_A", "var_B"])) - expected = pd.DataFrame( - { - "var_A": ["1"] * 6 + ["2"] * 10 + ["Rare"] * 4, - "var_B": ["1"] * 10 + ["2"] * 6 + ["Rare"] * 4, - } - ) - pd.testing.assert_frame_equal(_to_pandas(X), expected) + assert isinstance(X, pl.DataFrame) + assert to_dict(X) == { + "var_A": ["1"] * 6 + ["2"] * 10 + ["Rare"] * 4, + "var_B": ["1"] * 10 + ["2"] * 6 + ["Rare"] * 4, + } -@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) -def test_inverse_transform_raises_not_implemented_error(make_df): - df_enc_big = make_df(DATA_ENC_BIG) +def test_inverse_transform_raises_not_implemented_error(make_df, data_enc_big): + df_enc_big = make_df(data_enc_big) enc = RareLabelEncoder().fit(df_enc_big) with pytest.raises(NotImplementedError): enc.inverse_transform(df_enc_big) -def test_variables_cast_as_category(): +def test_variables_cast_as_category(data_enc_big): # pandas category dtype is backend-specific: polars has no equivalent # concept in the same sense. - df_enc_big = pd.DataFrame(DATA_ENC_BIG) + df_enc_big = pd.DataFrame(data_enc_big) + df_enc_big["var_B"] = df_enc_big["var_B"].astype("category") + encoder = RareLabelEncoder( tol=0.06, n_categories=5, variables=None, replace_with="Rare" ) - - df_enc_big = df_enc_big.copy() - df_enc_big["var_B"] = df_enc_big["var_B"].astype("category") - X = encoder.fit_transform(df_enc_big) # expected output - df = { - "var_A": ["A"] * 6 - + ["B"] * 10 - + ["C"] * 4 - + ["D"] * 10 - + ["Rare"] * 4 - + ["G"] * 6, - "var_B": ["A"] * 10 - + ["B"] * 6 - + ["C"] * 4 - + ["D"] * 10 - + ["Rare"] * 4 - + ["G"] * 6, - "var_C": ["A"] * 4 - + ["B"] * 6 - + ["C"] * 10 - + ["D"] * 10 - + ["Rare"] * 4 - + ["G"] * 6, - } - df = pd.DataFrame(df) + df = pd.DataFrame(ENC_BIG_RARE) df["var_B"] = pd.Categorical(df["var_B"]) # test fit attr @@ -494,8 +382,11 @@ def test_variables_cast_as_category(): pd.testing.assert_frame_equal(X, df, check_categorical=False) -def test_variables_cast_as_category_with_na_in_transform(): - df_enc_big = pd.DataFrame(DATA_ENC_BIG) +def test_variables_cast_as_category_with_na_in_transform(data_enc_big): + # pandas category dtype is backend-specific. + df_enc_big = pd.DataFrame(data_enc_big) + df_enc_big["var_B"] = df_enc_big["var_B"].astype("category") + encoder = RareLabelEncoder( tol=0.06, n_categories=5, @@ -503,9 +394,6 @@ def test_variables_cast_as_category_with_na_in_transform(): replace_with="Rare", missing_values="ignore", ) - - df_enc_big = df_enc_big.copy() - df_enc_big["var_B"] = df_enc_big["var_B"].astype("category") encoder.fit(df_enc_big) # input @@ -530,9 +418,9 @@ def test_variables_cast_as_category_with_na_in_transform(): pd.testing.assert_frame_equal(encoder.transform(t), tt, check_categorical=False) -def test_variables_cast_as_category_with_na_in_fit(): - - df = pd.DataFrame(DATA_ENC_BIG) +def test_variables_cast_as_category_with_na_in_fit(data_enc_big): + # pandas category dtype is backend-specific. + df = pd.DataFrame(data_enc_big) df.loc[df["var_C"] == "G", "var_C"] = None df["var_C"] = df["var_C"].astype("category") From bf689d05aa80df898d30e5b1e699ce9af963bbc6 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Tue, 15 Sep 2026 12:01:40 +0200 Subject: [PATCH 4/7] Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 --- tests/test_encoding/test_rare_label_encoder.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/test_encoding/test_rare_label_encoder.py b/tests/test_encoding/test_rare_label_encoder.py index 2d8676a14..77c0a173c 100644 --- a/tests/test_encoding/test_rare_label_encoder.py +++ b/tests/test_encoding/test_rare_label_encoder.py @@ -6,7 +6,7 @@ import pytest from feature_engine.encoding import RareLabelEncoder -from tests.backend_helpers import to_dict +from tests.backend_helpers import frame_to_dict MSG_NA = ( "Some of the variables in the dataset contain NaN. Check and " @@ -46,7 +46,7 @@ def test_defo_params_plus_automatically_find_variables(make_df, data_enc_big): assert encoder.encoder_dict_ == FREQUENT_CATEGORIES # test transform output assert isinstance(X, make_df) - assert to_dict(X) == ENC_BIG_RARE + assert frame_to_dict(X) == ENC_BIG_RARE def test_when_varnames_are_numbers(data_enc_big): @@ -97,7 +97,7 @@ def test_correctly_ignores_nan_in_transform(make_df, data_enc_big): ) assert isinstance(X, make_df) - assert to_dict(X) == { + assert frame_to_dict(X) == { "var_A": ["A", None, "Rare"], "var_B": ["A", None, "Rare"], "var_C": ["C", None, "Rare"], @@ -135,7 +135,7 @@ def test_correctly_ignores_nan_in_fit(make_df, data_enc_big): ) assert isinstance(X, make_df) - assert to_dict(X) == { + assert frame_to_dict(X) == { "var_A": ["A", None, "Rare", "G"], "var_B": ["A", None, "Rare", "G"], "var_C": ["C", None, "Rare", "Rare"], @@ -211,7 +211,7 @@ def test_user_provides_grouping_label_name_and_variable_list(make_df, data_enc_b assert encoder.n_features_in_ == 3 # test transform output assert isinstance(X, make_df) - assert to_dict(X) == { + assert frame_to_dict(X) == { "var_A": ["A"] * 6 + ["B"] * 10 + ["Other"] * 4 @@ -290,7 +290,7 @@ def test_max_n_categories(make_df, data_enc_big): X = rare_encoder.fit_transform(make_df(data_enc_big)) assert isinstance(X, make_df) - assert to_dict(X) == { + assert frame_to_dict(X) == { "var_A": ["A"] * 6 + ["B"] * 10 + ["Rare"] * 4 @@ -347,7 +347,7 @@ def test_max_n_categories_with_numeric_var_polars(data_enc_numeric): X = rare_encoder.fit_transform(df_enc_numeric.select(["var_A", "var_B"])) assert isinstance(X, pl.DataFrame) - assert to_dict(X) == { + assert frame_to_dict(X) == { "var_A": ["1"] * 6 + ["2"] * 10 + ["Rare"] * 4, "var_B": ["1"] * 10 + ["2"] * 6 + ["Rare"] * 4, } From c52eb23c4a1f7b3912583f438449c5afeffc1e1f Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Tue, 15 Sep 2026 13:37:55 +0200 Subject: [PATCH 5/7] Group RareLabelEncoder init tests, match errors, shorten comments Co-Authored-By: Claude Opus 5 --- feature_engine/encoding/rare_label.py | 28 +---- .../test_encoding/test_rare_label_encoder.py | 108 +++++++++++------- 2 files changed, 74 insertions(+), 62 deletions(-) diff --git a/feature_engine/encoding/rare_label.py b/feature_engine/encoding/rare_label.py index e9195f641..a879f8c68 100644 --- a/feature_engine/encoding/rare_label.py +++ b/feature_engine/encoding/rare_label.py @@ -236,12 +236,8 @@ def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): if col.n_unique() > self.n_categories: - # if the variable has more than the indicated number of categories - # the encoder will learn the most frequent categories. - # drop_nulls() mirrors pandas' value_counts(dropna=True) - # default, which narwhals' value_counts() doesn't apply on - # its own. sort=True matches pandas' own value_counts() - # default order (descending by count). + # learn the most frequent categories, dropping nulls and sorting + # by frequency like pandas' value_counts() counts = col.drop_nulls().value_counts(sort=True, normalize=True) cat_col, freq_col = counts.columns @@ -291,11 +287,8 @@ def transform(self, X: IntoDataFrame) -> IntoDataFrame: if self.missing_values == "raise": _check_contains_na(X, self.variables_, error_msg="optional") - # a pandas Categorical column rejects an unseen label on assignment - # until the label is added to its categories; narwhals has no - # cross-backend equivalent (polars has no comparable dtype - # restriction here), so this stays a native pandas step. Operate on a - # copy so the user's dataframe is not mutated. + # pandas categorical columns need replace_with added to their categories; + # work on a copy so the user's dataframe is not changed if nw_X.implementation.is_pandas(): native_X = nw_X.to_native().copy() for feature in self.variables_: @@ -305,17 +298,8 @@ def transform(self, X: IntoDataFrame) -> IntoDataFrame: ) nw_X = nw.from_native(native_X, eager_only=True) - # nw.when().then().otherwise(nw.lit(...)) lets each - # column keep its own dtype where frequent, and take the (possibly - # differently-typed) replace_with value elsewhere - narwhals - # resolves the common dtype per backend, e.g. object in pandas, - # cast-to-string in polars, so no manual dtype fixup is needed - # before it, unlike the old pandas-only .astype("O"). Passing - # Series (from get_column(), not nw.col()) into when/then/otherwise - # keeps this working for pandas integer column names, and nw.lit() - # broadcasts replace_with natively instead of materialising a - # same-length replacement array (benchmarked ~1.7x faster than the - # zip_with(col, new_series(...)) equivalent at 100k rows). + # narwhals resolves the dtype when mixing each column with replace_with; + # series from get_column() also work with pandas integer column names new_columns = [] for feature in self.variables_: col = nw_X.get_column(feature) diff --git a/tests/test_encoding/test_rare_label_encoder.py b/tests/test_encoding/test_rare_label_encoder.py index 77c0a173c..641c2492e 100644 --- a/tests/test_encoding/test_rare_label_encoder.py +++ b/tests/test_encoding/test_rare_label_encoder.py @@ -28,6 +28,70 @@ } +# init parameters +@pytest.mark.parametrize("tol", ["hello", [0.5], -1, 1.5, None]) +def test_error_if_tol_not_between_0_and_1(tol): + msg = f"tol takes values between 0 and 1. Got {tol} instead." + with pytest.raises(ValueError, match=re.escape(msg)): + RareLabelEncoder(tol=tol) + + +@pytest.mark.parametrize("n_cat", ["hello", [0.5], -0.1, 1.5, -1, None]) +def test_error_if_n_categories_not_int(n_cat): + msg = f"n_categories takes only positive integer numbers. Got {n_cat} instead." + with pytest.raises(ValueError, match=re.escape(msg)): + RareLabelEncoder(n_categories=n_cat) + + +@pytest.mark.parametrize("max_n_categories", ["hello", ["auto"], -1, 0.5]) +def test_raises_error_when_max_n_categories_not_allowed(max_n_categories): + msg = ( + "max_n_categories takes only positive integer numbers. " + f"Got {max_n_categories} instead." + ) + with pytest.raises(ValueError, match=re.escape(msg)): + RareLabelEncoder(max_n_categories=max_n_categories) + + +@pytest.mark.parametrize("replace_with", [set("hello"), ["auto"], None]) +def test_error_if_replace_with_not_string(replace_with): + msg = ( + "replace_with can should be a string, integer or float. " + f"Got {replace_with} instead." + ) + with pytest.raises(ValueError, match=re.escape(msg)): + RareLabelEncoder(replace_with=replace_with) + + +@pytest.mark.parametrize( + "tol, n_categories, max_n_categories, replace_with, missing_values, ignore_format", + [ + (0.05, 10, None, "Rare", "raise", False), + (0, 0, 3, "Other", "ignore", True), + (1, 5, 0, 1, "raise", True), + (0.5, 2, 10, 0.5, "ignore", False), + ], +) +def test_init_param_assignment( + tol, n_categories, max_n_categories, replace_with, missing_values, ignore_format +): + encoder = RareLabelEncoder( + tol=tol, + n_categories=n_categories, + max_n_categories=max_n_categories, + replace_with=replace_with, + missing_values=missing_values, + ignore_format=ignore_format, + ) + assert encoder.tol == tol + assert encoder.n_categories == n_categories + assert encoder.max_n_categories == max_n_categories + assert encoder.replace_with == replace_with + assert encoder.missing_values == missing_values + assert encoder.ignore_format is ignore_format + + +# fit and transform def test_defo_params_plus_automatically_find_variables(make_df, data_enc_big): # test case 1: defo params, automatically select variables encoder = RareLabelEncoder( @@ -35,11 +99,6 @@ def test_defo_params_plus_automatically_find_variables(make_df, data_enc_big): ) X = encoder.fit_transform(make_df(data_enc_big)) - # test init params - assert encoder.tol == 0.06 - assert encoder.n_categories == 5 - assert encoder.replace_with == "Rare" - assert encoder.variables is None # test fit attr assert encoder.variables_ == ["var_A", "var_B", "var_C"] assert encoder.n_features_in_ == 3 @@ -201,11 +260,6 @@ def test_user_provides_grouping_label_name_and_variable_list(make_df, data_enc_b ) X = encoder.fit_transform(make_df(data_enc_big)) - # test init params - assert encoder.tol == 0.15 - assert encoder.n_categories == 5 - assert encoder.replace_with == "Other" - assert encoder.variables == ["var_A", "var_B"] # test fit attr assert encoder.variables_ == ["var_A", "var_B"] assert encoder.n_features_in_ == 3 @@ -228,31 +282,6 @@ def test_user_provides_grouping_label_name_and_variable_list(make_df, data_enc_b } -# init params -@pytest.mark.parametrize("tol", ["hello", [0.5], -1, 1.5]) -def test_error_if_tol_not_between_0_and_1(tol): - with pytest.raises(ValueError): - RareLabelEncoder(tol=tol) - - -@pytest.mark.parametrize("n_cat", ["hello", [0.5], -0.1, 1.5]) -def test_error_if_n_categories_not_int(n_cat): - with pytest.raises(ValueError): - RareLabelEncoder(n_categories=n_cat) - - -@pytest.mark.parametrize("max_n_categories", ["hello", ["auto"], -1, 0.5]) -def test_raises_error_when_max_n_categories_not_allowed(max_n_categories): - with pytest.raises(ValueError): - RareLabelEncoder(max_n_categories=max_n_categories) - - -@pytest.mark.parametrize("replace_with", [set("hello"), ["auto"]]) -def test_error_if_replace_with_not_string(replace_with): - with pytest.raises(ValueError): - RareLabelEncoder(replace_with=replace_with) - - def test_warning_if_variable_cardinality_less_than_n_categories( make_df, data_enc_big ): @@ -335,10 +364,8 @@ def test_max_n_categories_with_numeric_var(data_enc_numeric): def test_max_n_categories_with_numeric_var_polars(data_enc_numeric): - # polars can't hold mixed int/str values in one column like pandas' - # object dtype does, so a numeric variable with a string replace_with - # is cast to string entirely instead - a real, backend-specific - # difference from the pandas behaviour above, not a bug. + # polars can't mix int and str in one column, so a numeric variable with a + # string replace_with is cast to string df_enc_numeric = pl.DataFrame(data_enc_numeric) rare_encoder = RareLabelEncoder( tol=0.10, max_n_categories=2, n_categories=1, ignore_format=True @@ -356,7 +383,8 @@ def test_max_n_categories_with_numeric_var_polars(data_enc_numeric): def test_inverse_transform_raises_not_implemented_error(make_df, data_enc_big): df_enc_big = make_df(data_enc_big) enc = RareLabelEncoder().fit(df_enc_big) - with pytest.raises(NotImplementedError): + msg = "inverse_transform is not implemented for this transformer." + with pytest.raises(NotImplementedError, match=re.escape(msg)): enc.inverse_transform(df_enc_big) From cc3cb356f127dcb566cffcc088eb4f56a5dc8209 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Tue, 15 Sep 2026 13:46:08 +0200 Subject: [PATCH 6/7] Fix typo in RareLabelEncoder replace_with error Co-Authored-By: Claude Opus 5 --- feature_engine/encoding/rare_label.py | 2 +- tests/test_encoding/test_rare_label_encoder.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/feature_engine/encoding/rare_label.py b/feature_engine/encoding/rare_label.py index a879f8c68..720bf3dc9 100644 --- a/feature_engine/encoding/rare_label.py +++ b/feature_engine/encoding/rare_label.py @@ -195,7 +195,7 @@ def __init__( if not isinstance(replace_with, (str, int, float)): raise ValueError( - "replace_with can should be a string, integer or float. " + "replace_with should be a string, integer or float. " f"Got {replace_with} instead." ) diff --git a/tests/test_encoding/test_rare_label_encoder.py b/tests/test_encoding/test_rare_label_encoder.py index 641c2492e..47d20b6be 100644 --- a/tests/test_encoding/test_rare_label_encoder.py +++ b/tests/test_encoding/test_rare_label_encoder.py @@ -56,7 +56,7 @@ def test_raises_error_when_max_n_categories_not_allowed(max_n_categories): @pytest.mark.parametrize("replace_with", [set("hello"), ["auto"], None]) def test_error_if_replace_with_not_string(replace_with): msg = ( - "replace_with can should be a string, integer or float. " + "replace_with should be a string, integer or float. " f"Got {replace_with} instead." ) with pytest.raises(ValueError, match=re.escape(msg)): From fe039688694ee887310fa7018967d67cd1955248 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Wed, 16 Sep 2026 08:50:04 +0200 Subject: [PATCH 7/7] minor refactor to tests --- .../test_encoding/test_rare_label_encoder.py | 23 +++++-------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/tests/test_encoding/test_rare_label_encoder.py b/tests/test_encoding/test_rare_label_encoder.py index 47d20b6be..7b2858d15 100644 --- a/tests/test_encoding/test_rare_label_encoder.py +++ b/tests/test_encoding/test_rare_label_encoder.py @@ -93,7 +93,6 @@ def test_init_param_assignment( # fit and transform def test_defo_params_plus_automatically_find_variables(make_df, data_enc_big): - # test case 1: defo params, automatically select variables encoder = RareLabelEncoder( tol=0.06, n_categories=5, variables=None, replace_with="Rare" ) @@ -109,7 +108,7 @@ def test_defo_params_plus_automatically_find_variables(make_df, data_enc_big): def test_when_varnames_are_numbers(data_enc_big): - # integer column names are pandas-only, polars has no such concept + # integer column names are pandas-only input_df = pd.DataFrame(data_enc_big) input_df.columns = [1, 2, 3] @@ -164,7 +163,7 @@ def test_correctly_ignores_nan_in_transform(make_df, data_enc_big): def test_correctly_ignores_nan_in_fit(make_df, data_enc_big): - data = dict(data_enc_big) + data = data_enc_big data["var_C"] = [None if v == "G" else v for v in data["var_C"]] encoder = RareLabelEncoder( @@ -202,9 +201,7 @@ def test_correctly_ignores_nan_in_fit(make_df, data_enc_big): def test_correctly_ignores_nan_in_fit_when_var_is_numerical(data_enc_big): - # pandas .astype("O") mixed-dtype workaround for a numeric variable with - # a string replace_with is a pandas-only quirk (polars casts to string - # instead - see test_max_n_categories_with_numeric_var_polars). + # pandas only df = pd.DataFrame(data_enc_big) df["var_C"] = [ 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, @@ -238,9 +235,8 @@ def test_correctly_ignores_nan_in_fit_when_var_is_numerical(data_enc_big): } ) - # expected (var_C mixes floats and strings after transform, so its - # missing value must be an actual float nan, not a bare None, to match - # pandas' own dtype inference for the same mix) + # var_C mixes floats and strings after transform, so its + # missing value must be an actual nan tt = pd.DataFrame( { "var_A": ["A", None, "Rare", "G"], @@ -254,7 +250,6 @@ def test_correctly_ignores_nan_in_fit_when_var_is_numerical(data_enc_big): def test_user_provides_grouping_label_name_and_variable_list(make_df, data_enc_big): - # test case 2: user provides alternative grouping value and variable list encoder = RareLabelEncoder( tol=0.15, n_categories=5, variables=["var_A", "var_B"], replace_with="Other" ) @@ -285,7 +280,6 @@ def test_user_provides_grouping_label_name_and_variable_list(make_df, data_enc_b def test_warning_if_variable_cardinality_less_than_n_categories( make_df, data_enc_big ): - # test case 3: when the variable has low cardinality msg = ( "The number of unique categories for variable var_A is less than that " "indicated in n_categories. Thus, all categories will be " @@ -297,7 +291,6 @@ def test_warning_if_variable_cardinality_less_than_n_categories( def test_fit_raises_error_if_df_contains_na(make_df, data_enc_big_na): - # test case 4: when dataset contains na, fit method encoder = RareLabelEncoder(n_categories=4) with pytest.raises(ValueError, match=re.escape(MSG_NA)): encoder.fit(make_df(data_enc_big_na)) @@ -306,7 +299,6 @@ def test_fit_raises_error_if_df_contains_na(make_df, data_enc_big_na): def test_transform_raises_error_if_df_contains_na( make_df, data_enc_big, data_enc_big_na ): - # test case 5: when dataset contains na, transform method encoder = RareLabelEncoder(n_categories=4) encoder.fit(make_df(data_enc_big)) with pytest.raises(ValueError, match=re.escape(MSG_NA)): @@ -314,7 +306,6 @@ def test_transform_raises_error_if_df_contains_na( def test_max_n_categories(make_df, data_enc_big): - # test case 6: user provides the maximum number of categories they want rare_encoder = RareLabelEncoder(tol=0.10, max_n_categories=4, n_categories=5) X = rare_encoder.fit_transform(make_df(data_enc_big)) @@ -342,9 +333,7 @@ def test_max_n_categories(make_df, data_enc_big): def test_max_n_categories_with_numeric_var(data_enc_numeric): - # pandas .astype("O") mixed-dtype workaround for a numeric variable with - # a string replace_with is a pandas-only quirk (see the polars variant - # below, which casts to string instead of keeping mixed dtypes). + # pandas only df_enc_numeric = pd.DataFrame(data_enc_numeric) rare_encoder = RareLabelEncoder( tol=0.10, max_n_categories=2, n_categories=1, ignore_format=True