Skip to content

Validate and escape table identifiers in JDBC sink operators - #839

Open
AliRana30 wants to merge 1 commit into
apache:mainfrom
AliRana30:fix/validate-escape-table-identifiers-836
Open

AliRana30 wants to merge 1 commit into
apache:mainfrom
AliRana30:fix/validate-escape-table-identifiers-836

Conversation

@AliRana30

Copy link
Copy Markdown
Contributor

Key Changes

  • SqlIdentifierUtils: Added a utility in wayang-basic to strictly validate single-part and multipart table/column identifiers using regex patterns (^[a-zA-Z_][a-zA-Z0-9_$]*(?:\.[a-zA-Z_][a-zA-Z0-9_$]*)*$) and escape/quote identifiers according to database dialect.
  • TableSink: Integrated validation in the constructor and setter so any invalid or malicious table/column identifier is rejected early with an IllegalArgumentException.
  • JdbcTableSinkOperator & JdbcExecutor: Quoted table names in generated DROP TABLE IF EXISTS, CREATE TABLE ... AS, and INSERT INTO clauses. Multipart identifiers (such as Trino catalog.schema.table) are quoted segment-by-segment ("catalog"."schema"."table").
  • JavaTableSink & SparkTableSink: Updated to use SqlIdentifierUtils for safe identifier quoting.
  • Unit Tests: Added SqlIdentifierUtilsTest and updated JdbcTableSinkExecutorTest to assert proper quoting and rejection of SQL injection payloads.

Fixes : #836.

Verification

  • SqlIdentifierUtilsTest (10 tests passed)
  • JdbcTableSinkExecutorTest (8 tests passed)
  • JavaTableSinkTest (7 tests passed)
  • SparkTableSinkTest (10 tests passed)
  • apache-rat:check (0 unapproved licenses)

Copilot AI balanced review requested due to automatic review settings October 10, 2026 23:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@AliRana30

Copy link
Copy Markdown
Contributor Author

@zkaoudi PR is up for your review.

@novatechflow

Copy link
Copy Markdown
Member

Thanks for the contribution. Pls always add regression tests.

  • Quoting changes identifier case handling. On HSQLDB, an existing unquoted target_append becomes TARGET_APPEND, but the new SQL targets "target_append". Append fails, and overwrite leaves the original table untouched.
  • The shared validation rejects valid BigQuery names such as my-project.sales.orders, including the documented backtick-quoted form.

Reproduced both with focused Java 17 checks. Preserve identifier lookup semantics & handle dialect-specific names.

@AliRana30
AliRana30 force-pushed the fix/validate-escape-table-identifiers-836 branch from 908cf6c to 76fecd5 Compare October 21, 2026 22:39
@AliRana30

Copy link
Copy Markdown
Contributor Author

@novatechflow I have addressed both issues and added regression tests covering both case folding and BigQuery formats. Verified on Java 17

@novatechflow

Copy link
Copy Markdown
Member

Two small things, reproduced on Java 17:

  • Mixed quoting breaks: "PUBLIC".target_append becomes """PUBLIC"""."TARGET_APPEND", and causes lookup failures.
  • Case folding depends on system locale: under Turkish locale, input_table becomes "İNPUT_TABLE" and fails. Use Locale.ROOT, that should avoid that.

I think the quoting utility needs per-segment quote handling and locale-independent folding before approval. Thoughts?

@AliRana30
AliRana30 force-pushed the fix/validate-escape-table-identifiers-836 branch from 76fecd5 to d928af5 Compare October 24, 2026 22:25
@AliRana30

Copy link
Copy Markdown
Contributor Author

Thanks for the review @novatechflow . Addressed both issues and added regression tests:

  • Case-folding semantics: Preserved standard identifier lookup via DatabaseMetaData case folding (folding unquoted identifiers to uppercase on HSQLDB/H2/Oracle/Derby and lowercase on Postgres, while preserving already-quoted names as-is). Unquoted tables like target_append now properly match catalog tables on append and overwrite.
  • BigQuery names: Added support for hyphenated identifiers (my-project.sales.orders) and documented backtick forms (whole-quoted and per-segment).
  • Regression tests: Added live HSQLDB tests covering append, overwrite, and quoted case-preservation semantics, as well as BigQuery validation test cases.

@novatechflow

Copy link
Copy Markdown
Member

Ran a short test together with the other review, also great progress. Mixed quoting is fixed; the locale bug remains:

the code still uses parameterless toUpperCase() / toLowerCase(). Both need Locale.ROOT, plus a Turkish-locale regression test. My locale test found only Turkish, not sure if we also hit others. So locale.ROOT seems the best solution. We can't assume that users set their console to English.

@AliRana30
AliRana30 force-pushed the fix/validate-escape-table-identifiers-836 branch from d928af5 to a36bac8 Compare October 25, 2026 17:53
@AliRana30

Copy link
Copy Markdown
Contributor Author

@novatechflow Thanks for the review and catching the locale issue!

  • Updated all identifier case folding in SqlIdentifierUtils and JDBC URL detection in SqlTypeUtils to use Locale.ROOT.
  • Added Turkish locale (tr-TR) regression tests in SqlIdentifierUtilsTest, SqlTypeUtilsTest, and JavaTableSinkTest (verifying live HSQLDB table operations under Turkish locale).
  • Resolved the CI failures in Backend / Build and Docker / Build caused by dummy pattern templates in TableSinkMapping (new TableSink<Record>(null, null, "", new String[0])) crashing during pattern registration; validation is strictly enforced during sink operator creation and query generation.

@novatechflow novatechflow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java 17 verification passes:

  • Turkish-locale HSQLDB lookup now uses "INPUT_TABLE" correctly.
  • Mixed quoting, append/overwrite, and BigQuery validation still pass.
  • All 22 identifier/type utility tests pass.

thanks @AliRana30 for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Validate and escape table identifiers in JDBC sink operators

3 participants