Project Overview

This repository is a focused data pre-processing utility for cleaning bilingual (Nepali-English) Excel files. The main script is preprocessing.py, which reads an input Excel file, applies a series of text normalization and filtering heuristics, and writes a cleaned output Excel file. The workflow is designed to preserve alignment between language pairs while removing noise and invalid data.

Key Files

• preprocessing.py: Main script for all data processing logic.
• input.xlsx: Example input file (bilingual, Nepali-English).
• output.xlsx: Output file generated by the script.

Core Workflow

• Run the script from the command line, specifying input and output Excel file paths and (optionally) column names and filtering parameters.
• The script expects columns named Nepal (Nepali text) and English (English text) by default, but these can be overridden with arguments.
• Data is filtered by:
  • Removing rows with blank or nonsense text (using Unicode heuristics)
  • Removing rows where Nepali contains English letters
  • Removing rows with disallowed characters in Nepali
  • Dropping duplicate sentence pairs (optionally case-insensitive)
• Output preserves the original column order and writes to Excel using openpyxl.

Usage Example

python preprocessing.py input.xlsx output.xlsx --nepali_col=Nepal --english_col=English

Project-Specific Patterns

• All text normalization and filtering logic is in preprocessing.py.
• Uses regular expressions for Unicode and language-specific filtering.
• No external configuration files; all parameters are via CLI arguments.
• No test suite or build system; run the script directly.
• Only dependency is pandas (and openpyxl for Excel I/O).

Conventions

• Nepali text is expected in Devanagari Unicode range.
• English text is expected to be Latin script.
• All data cleaning is non-destructive to column order and non-target columns.

Integration Points

• No external APIs or services; all processing is local and file-based.
• Designed for batch, not interactive, use.

Extending

• Add new filters or normalization steps in preprocessing.py.
• To support new file formats, extend the I/O logic in the main script.

---

For questions about project-specific logic, see comments in preprocessing.py.