[Creylay]

Summary

Converters that create new columns (COLUMN_OPERATION = "add", e.g. Bag of Words, TF-IDF, One-Hot, DateFeatures) keep the columns they read next to the ones they create. Most of the time the source column is no longer useful: it clutters the notebook dataset and, in a session, it stays available as a model input that can be picked by mistake.

This PR adds a "Drop original columns" option to every column-adding converter. When it is on, the converter's scope columns are removed after it transforms, so only the new columns remain. It is off by default and behaves the same in notebooks and in session preprocessing.

• The option lives in the schema: a new base class, AddColumnsConverterSchema, declares drop_source_columns (translated to en, es, pt, de, zh), and the 16 column-adding schemas inherit it.
• The option never reaches a converter's constructor: it is taken out of the params before the converter is built.
• The columns are dropped in one place, rebuild_dataset_with_transformed_columns, which notebooks and sessions already share.
• In sessions, the structure estimate removes the dropped columns from the state, so a later step that uses one gets the existing missing_ref error.
• The target column is never dropped, and an empty column scope with the option on is refused.
• The option is hidden in document notebooks, where items are rewritten in place and there are no columns to drop.
• Saved notebooks, session preprocessing and pickled preprocessors need no migration: a missing key means off.

---

Type of Change

Check all that apply like this [x]:

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Option and helpers

• DashAI/back/converters/drop_source_columns.py (new):
  • AddColumnsConverterSchema with the drop_source_columns field. It has a real False default, so params saved before this change keep validating.
  • split_converter_params, which takes the option out of the params before a converter is built.
  • drops_source_columns, the shared rule: the option is on and the converter is "add".
  • reject_output_named_like_scope.
• DashAI/back/config_object.py: get_schema moves drop_source_columns to the end of the form.
• 16 converter schemas switch their base class to AddColumnsConverterSchema:
  • clustering.py, image_embedding.py, summarizer.py, translation/base.py
  • text_vectorizer.py, binarizer.py, cca.py, label_encoder.py, missing_indicator.py, one_hot_encoder.py, ordinal_encoder.py
  • sam3_segment_converter.py, column_arithmetic.py, column_concat.py, date_features.py, numeric_expansion.py
• DashAI/back/converters/dataset_columns.py: rebuild_dataset_with_transformed_columns takes columns_to_drop. New column names are planned against the columns that remain.

Notebooks and DAG units

• DashAI/back/units/converter_scope.py:
  • _merge_transformed drops the scope columns, except the target.
  • _check_drop_scope refuses an empty scope.
• DashAI/back/units/apply_converter_unit.py, fit_converter_unit.py: build the converter without the option and publish it as fitted_converter_options.
• DashAI/back/units/transform_dataset_unit.py: requires fitted_converter_options, so fitting on one dataset and transforming another drops the same columns. The DAG engine only moves the keys two nodes agree on.
• DashAI/back/job/converter_job.py: the document path strips the option and otherwise ignores it.

Session preprocessing

• DashAI/back/preprocessing/structure.py: a dropping step removes its scope from the state, like an expand step, and frees those names for later steps.
• DashAI/back/preprocessing/session_preprocessor.py:
  • drops the scope columns in every split and in transform_only;
  • records the step's group under the names the dataset really has;
  • reads the option from each step's own params, so preprocessors pickled before this change never drop.

Bug fix

• DashAI/back/converters/simple_converters/column_arithmetic.py, column_concat.py: reject an output column name equal to one of the scope columns.
  • Before, the result overwrote the source column while the structure estimate predicted <name>_1, so later references failed at training.
  • With the drop option on, the result would have been deleted.

Frontend

• DashAI/front/src/components/shared/FormSchemaWithSelectedModel.jsx, notebooks/converterCreation/ParameterStepConverter.jsx: accept and forward excludeFields.
• DashAI/front/src/components/notebooks/converterCreation/FormConverterSection.jsx: hides drop_source_columns in document notebooks, and skips the parameter step when it was the only field.

Tests

• New: tests/back/converters/test_drop_source_columns.py and FormConverterSection.test.jsx.
• Extended:
  • test_dataset_columns.py
  • test_apply_converter_unit.py, test_converter_fit_transform_split.py
  • test_session_preprocessor.py, test_structure.py
  • test_engine.py
• Updated to build converters without the option: test_sklearn_option_validity.py, test_structure_matches_runtime.py.
• Updated for the new field order: test_translation_converters.py.

---

Testing (optional)

• Notebook (dataset): apply Bag of Words to a text column with "Drop original columns" on. Only the term columns should remain, and the text column should be gone.
• Notebook (document): Summarizer or a translator should not show the option.
• Session: add the same Bag of Words step in preprocessing with the option on. The structure panel should drop the text column after that step, and training and prediction should work.
• Session: a later step that uses the dropped column should show an error on that step.
• ColumnArithmetic with an output name equal to one of the selected columns should be rejected with a clear message.
• An existing notebook or session created before this branch should behave exactly as before.