Telugu TN: Cardinal Semiotic Class

[nharidas]

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

• Have you signed your commits? Use git commit -s to sign.
• Do all unittests finish successfully before sending PR?
  1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
  2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
• If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
• Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
• Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
• Have you added the correct license header Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
• If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
• Remove import guards (try import: ... except: ...) if not already done.
• If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
• Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

• New Feature
• Bugfix
• Documentation
• Test

If you haven't finished some of the above items you can still open "Draft" PR.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[folivoramanh]

make sure both pytest and sparrowhawk test pass, and fix the problem of blank space

[nharidas]

Thanks for the review. I’ve addressed the comments on this PR.

• Removed Telugu post-processing (no PostProcessingFst for te; tests use post_process=False)
• Dropped custom add_weight; competing *1 forms are split instead of using a -0.2 bias
• Deleted teens_and_ties_thousand.tsv; combining ఒక is composed from ties
• magnitudes.tsv is loaded with load_labels
• Digit/zero classes come from digit.tsv / zero.tsv (no hardcoded NEMO_TE_DIGIT)
• Word tagger simplified to NEMO_NOT_SPACE
• Jenkinsfile: removed the commented TE ITN stage
• Tests: assert pred == expected (no .strip())

pytest for Telugu cardinals is passing. Sparrowhawk tests are still pending; I’ll update this PR once they are run.

[folivoramanh]

• add init file in data/ (currently only in data/numbers/)

[folivoramanh]

check this example and see that if you can build a reusable component for thousand/million/billion/trillion
https://github.com/NVIDIA/NeMo-text-processing/blob/main/nemo_text_processing/text_normalization/es/taggers/cardinal.py

[nharidas]

Thanks for the pointer. I reviewed, es/taggers/cardinal.py.

Spanish works with one reusable block because it uses the Western scale: each step adds 6 zeros (thousand → million → billion → trillion) and reuses the same 3-digit pattern with a new magnitude word.

Telugu uses the Indian numbering system, which is structured differently:

• 1,000 → thousand
• 1,00,000 → lakh (not “hundred thousand”)
• 1,00,00,000 → crore
• higher tiers (arab, kharab, nil, padma, shankh) with different digit groupings

On top of that, the spoken form is not one fixed word per magnitude. It depends on position in the number and what follows, for example:

• singular vs plural / “before” forms (e.g. వేల vs వేలు, కోటి vs కోట్లు)
• special “one” form when combining (ఒక in 21, 31, … vs ఒకటి for standalone 1)
• prefix variants (e.g. నూట vs వంద for hundreds)
• special lakh form (లక్షా)

So we cannot drop in one ES-style thousand/million/billion component, both the scale and the word forms change at each tier.

This current code already reuses shared logic where the grammar allows it:

• build_group, create_graph_suffix, create_larger_number_graph
• magnitude ladders (thousand → lakh → crore → …)
• shared strings in magnitudes.tsv

The longer code reflects real Telugu morphology, not unnecessary duplication. Collapsing it into one Spanish-style block would risk wrong readings (e.g. 21 → ఇరవై ఒక, not ఇరవై ఒకటి).

[mgrafu]

if it uses the Indian numbering system, are we building the graph in the same way that the Hindi graph does?

[nharidas]

@mgrafu Yes, same Indian numbering system and the same overall graph structure as Hindi.
Both use the tiered scale (hundred → thousand → lakh → crore → arab/kharab/nil/…) with the same create_graph_suffix / create_larger_number_graph pattern and the same 2+3 digit grouping.
Telugu extends that base for language-specific morphology: singular/plural/before magnitude forms, special “one” handling (ఒక vs ఒకటి), and hundred prefixes (నూట vs వంద), driven by TSVs and build_group ladders. Hindi uses a simpler one-suffix-per-tier model.

[mgrafu]

ok, thanks! one more comment though: graph_arabs / graph_kharabs / graph_padmas / graph_shankhs name tiers that produce no such word. The file already uses crore-relative naming for its prefix helpers (hundred_crore_prefix, lakh_crore_count_prefix, ten_lakh_crore_prefix), so this is internally inconsistent too. Renaming to that existing convention — graph_hundred_crores (10^9), graph_ten_thousand_crores (10^11), graph_lakh_crores (10^13), graph_crore_crores (10^15) — would make the structure self-describing.

[nharidas]

@mgrafu Done. Renamed higher tiers to the crore-relative convention already used in the file:
graph_arabs → graph_hundred_crores
graph_ten_arabs → graph_thousand_crores
graph_kharabs → graph_ten_thousand_crores
graph_ten_kharabs → graph_lakh_crores
graph_ten_nils → graph_ten_lakh_crores
graph_padmas → graph_crore_crores
graph_ten_padmas → graph_ten_crore_crores
graph_shankhs → graph_hundred_crore_crores

[nharidas]

Thanks for the follow-up review.

Changes Made:

• Added nemo_text_processing/text_normalization/te/data/init.py
• Simplified Jenkinsfile to a single L0: TE TN grammars stage (removed the parallel Create TE TN/ITN Grammars wrapper)
• Expanded cardinal tests to cover large magnitudes, including 999999999999999 (15 digits) and up to 19-digit shankh-scale numbers, plus Telugu digits, in-sentence, leading zeros, and negatives
• Negative numbers: verbalizer now outputs “మైనస్” for the sign (Telugu localized form, same pattern as Spanish “menos”), instead of English “minus”. Test cases updated accordingly.

Testing:

• Cardinal pytest test cases passed
• Sparrowhawk “testTNCardinal” passed

Re: Spanish reusable component: replied in the thread with details.

[nharidas]

Thanks for the review.

Changes Made:

• Expanded test_cases_cardinal.txt with internal-zero and separator coverage (Indian and Western grouping)
• Added separator handling in cardinal.py (indian_grouping / western_grouping) to strip commas before normalization
• Moved hardcoded inputs to TSV files.
• Removed unused code in test_sparrowhawk_normalization.sh

Testing:

• Cardinal pytest test cases passed
• Sparrowhawk testTNCardinal passed

[nharidas]

@mgrafu Thanks for the review.

Changes Made:

• Documented max magnitude coverage in the CardinalFst docstring (up to 19 digits / shankh / శంఖ, 10¹⁷; larger values via composed కోటి groups)
• Fixed singular/plural కోటి vs కోట్ల for large numbers.
• Expanded test_cases_cardinal.txt with the suggested large-number, mixed-script, separator, and edge cases
• Updated Jenkins: guarded TE L0 grammar build (when + failFast + parallel) and L1 TE pytest stage with TE_TN_CACHE
• Added word and punctuation unit tests (test_word.py, test_punctuation.py + corresponding test_cases_*.txt)
• Updated test_sparrowhawk_normalization.sh to run word and punctuation Sparrowhawk suites

Testing:

• Cardinal, word, and punctuation pytest cases passed
• Sparrowhawk testTNCardinal, testTNWord, and testTNPunctuation passed

[mgrafu]

please follow the style of English word test cases

[mgrafu]

also, are specific English test cases necessary?

[nharidas]

@mgrafu Done. updated the TE word tests to follow the English style (empty/whitespace, leading/trailing space, sentences with punct, hyphenated words, digit+punct spacing, etc.), with Telugu examples.
EN cases that need measure/money grammars, or English-only letter+digit splits, aren’t covered yet since those aren’t in TE.

[mgrafu]

let's test just Telugu for now instead of English. Please remove lines like $ and 5% or %~$ and ఐదు % or % or X!~X!

[nharidas]

@mgrafu Done.

[nharidas]

@mgrafu Thanks for the review.

Changes Made:

• Fixed cardinal edge cases: …01 → ఒక కోటి, X1 లక్షల → ఒక (not ఒకటి), mixed-script leading zeros pass-through
• Renamed arab/kharab/padma/shankh tiers to crore-relative names.
• Fixed FST ambiguity so Sparrowhawk and Pytest agree on large numbers (e.g. 3164010000000)
• Added EN-style word and punctuation tests (Telugu examples), plus punctuation match-input via punct_post_process=True

Testing:

• Cardinal, word, and punctuation pytest cases passed
• Sparrowhawk testTNCardinal, testTNWord, and testTNPunctuation passed

[mgrafu]

let's include these test cases

1000010000000~లక్ష ఒక కోటి
2000010000000~రెండు లక్షల ఒక కోటి
5000010000000~ఐదు లక్షల ఒక కోటి
9000010000000~తొమ్మిది లక్షల ఒక కోటి
11000010000000~పదకొండు లక్షల ఒక కోటి
99000010000000~తొంభై తొమ్మిది లక్షల ఒక కోటి
1000014261933~లక్ష ఒక కోటి నలభై రెండు లక్షల అరవై ఒక వేల తొమ్మిది వందల ముప్పై మూడు
10000010000000~పది లక్షల ఒక కోటి
50000010000000~యాభై లక్షల ఒక కోటి
90000010701028~తొంభై లక్షల ఒక కోటి ఏడు లక్షల వెయ్యి ఇరవై ఎనిమిది
1010010000000~లక్ష వెయ్యి ఒక కోటి
3010010000000~మూడు లక్షల వెయ్యి ఒక కోటి
5010010000000~ఐదు లక్షల వెయ్యి ఒక కోటి
1010012900770~లక్ష వెయ్యి ఒక కోటి ఇరవై తొమ్మిది లక్షల ఏడు వందల డెబ్బై
5010012900770~ఐదు లక్షల వెయ్యి ఒక కోటి ఇరవై తొమ్మిది లక్షల ఏడు వందల డెబ్బై
5001010000000~ఐదు లక్షల నూట ఒక కోటి
90410010000205~తొంభై లక్షల నలభై ఒక వేల ఒక కోటి రెండు వందల ఐదు
30920010793149~ముప్పై లక్షల తొంభై రెండు వేల ఒక కోటి ఏడు లక్షల తొంభై మూడు వేల నూట నలభై తొమ్మిది
10601010405114~పది లక్షల అరవై వేల నూట ఒక కోటి నాలుగు లక్షల ఐదు వేల నూట పద్నాలుగు
1000020000000~లక్ష రెండు కోట్లు
10000020000000~పది లక్షల రెండు కోట్లు
50000020000000~యాభై లక్షల రెండు కోట్లు

[nharidas]

@mgrafu Done. added these test cases.

[nharidas]

@mgrafu Thanks for the review.

Changes Made:

• Added these test cases, fixed the related failed readings
• Additionally cleaned up the cardinal grammar by collapsing repeated band wiring into shared helpers (band, oka_count, crore_of, kotlu, oka_koti, etc.).
• Removed the English related test cases from word test cases.

Testing:

• Cardinal, word, and punctuation pytest cases passed
• Sparrowhawk testTNCardinal, testTNWord, and testTNPunctuation passed