Skip to content

feat: run text value set operations as SQL where order and repeats don't count (#485) - #493

Closed
CaffeinatedCoder wants to merge 4 commits into
mainfrom
feat/485-text-set-forms
Closed

CaffeinatedCoder wants to merge 4 commits into
mainfrom
feat/485-text-set-forms

Conversation

@CaffeinatedCoder

Copy link
Copy Markdown
Owner

Closes #485

Stacked on #492 (#483). It relies on #483's guarantee that a string set column compares bytes, and it touches the same guard file.

Problem

On StringSet and StringSet<T>, PostgreSQL can't sort a set operation's result as memory does: it sorts text by the database collation (#479). So since #478, every set type but text got canonical Union/Add/Intersect/Except in SQL, while on text:

  • Intersect, Except and Add ran in memory only;
  • Count, == and != over a server-side Union threw.

#479 can change the order only in 11.0.0, so on 10.x this would stay for the line's whole life.

Change

  • Translator: text Add/Intersect/Except now emit ValueSetSqlGuard.UnguardedAdd/UnguardedIntersect/UnguardedExcept (inside the NULL-over-NULL CASE), as Union already emitted UnguardedUnion.
  • Walker: a value derived from any of the four may now feed cardinality, =/<> and NULLIF, besides the multiset uses it already allowed. GROUP BY, DISTINCT, ORDER BY, deduplicating set operations, IN and UPDATE setters stay refused, with the messages reworded for all four operations.
  • Expander:
    • It builds the four unsorted: array_cat, array_cat with a one-element array, or a filter of the first array.
    • It rewrites cardinality(derived) as (SELECT count(DISTINCT x)::int FROM unnest(…)), NULL for a NULL set.
    • It rewrites =/<>/NULLIF with a derived side as mutual containment, a @> b AND b @> a. A stored-column = keeps = and its index.
    • It recognizes a derived value structurally, plus the walker's derived subquery columns. UnionNamer folds into it, because a separate renaming pass would rebuild the tree before the expander sees it.
  • ValueSetsMemberTranslator no longer declines Count over a visible union; MayBeConcatenation is gone.
  • No COLLATE "C": measured on Text value set operators follow a nondeterministic column collation: on a case-insensitive column, Contains("a") is true for {A} and Remove("a") deletes A #483, it costs a GIN index its use. Text value set operators follow a nondeterministic column collation: on a case-insensitive column, Contains("a") is true for {A} and Remove("a") deletes A #483 makes the column's equality byte equality instead.
  • Docs: in the EF guide, Combine sets with Union is rewritten, plus both what-runs-where tables, Good to know, Rows written by other tools and the value set section. Also value-sets.md, getting-started.md, both READMEs, AGENTS.md (the intro and the text rule), architecture.md and testing.md. Changelogs (root, EF Core) under 11.0.0-preview.1 Added.
  • Oracles:
    • translation-support.json: text Intersect/Except/Add are now server.
    • Claims E17–E20, E35 and E82 follow the new sentences.
    • New seeds text-set-count-counts-repeats and text-set-equality-compares-arrays.
    • union-equality-not-refused is retired: its refusal no longer exists, and the second new seed holds the rewrite.

Tests

  • CanonicalSetOperationParityTests.Text_AnswersAsInMemory_WhereOrderAndRepeatsDontCount (new sweep).
    • Data: StringSet and StringSet<T>, every pair of subsets of a, 😀 and 。 (the pair UTF-16 and code points order apart), stored canonical, out of order and twice, and with a NULL element.
    • Asked: Count on every row; ==/!=, the projection, containment and IsEmpty on the rows without a NULL element.
    • Refused: storing a result through ExecuteUpdate.
  • UnionCanonicalityTests: the equality and Count cases now run and compare with memory, including through a UNION ALL column, a conditional and a coalesce. The ordering, grouping, DISTINCT, set-operation, IN and ExecuteUpdate refusals stay. SetIntegrationTests and EfCore11RetargetTests likewise.
  • No-database: 13 tests that asserted a refusal now assert the SQL; DeclaredNullabilityTests treats the ARRAY(…) constructor as non-strict.
  • Proved by reverting (tools/oracles.cs seeds):
    • leaving Count as cardinality fails the text sweep and 2 UnionCanonicalityTests;
    • rewriting equality only when both sides are derived fails the sweep and 4 UnionCanonicalityTests.
  • Full suite green: unit, conventions, ASP.NET Core and EF no-database projects, and both integration projects on PostgreSQL 16 and 18.

🤖 Generated with Claude Code

CaffeinatedCoder and others added 2 commits October 26, 2026 12:59
…n't count (#485)

PostgreSQL can't sort text as memory does (#479), so on StringSet and
StringSet<T> Intersect, Except and Add ran in memory only, and Count, == and
!= over a server-side text Union threw. Most uses don't depend on order or
repeats, so the translator now emits unguarded placeholders for all four and
ValueSetSqlGuard builds them unsorted (array_cat, or a filter of the first
array) where the walker allows: @>, <@, &&, IsEmpty, a null test and the
outermost projection, as for Union before. Two more uses are rewritten:
- cardinality over one becomes (SELECT count(DISTINCT x) FROM unnest(...)),
  NULL for a NULL set;
- =, <> and EF Core 11's NULLIF with one on a side become mutual
  containment, a @> b AND b @> a. A stored-column = keeps = and its index.

No COLLATE "C": a collated operand costs a GIN index its use, and #483 makes
a string set column compare bytes. Grouping, DISTINCT, ORDER BY, deduplicating
set operations, IN over several sets and UPDATE setters stay refused.

UnionNamer's renaming moves into the expander, which recognizes a derived
value structurally plus the walker's derived subquery columns, and
ValueSetsMemberTranslator no longer declines Count over a visible union.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusal seed (#485)

text-set-count-counts-repeats leaves Count as cardinality, and
text-set-equality-compares-arrays rewrites == only when both sides are a text
operation; CanonicalSetOperationParityTests and UnionCanonicalityTests catch
both. union-equality-not-refused removed an equality refusal that no longer
exists: equality over a text Union is rewritten now, and the second seed holds
that rewrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CaffeinatedCoder CaffeinatedCoder added this to the 10.4.0 milestone Oct 26, 2026
@CaffeinatedCoder CaffeinatedCoder added enhancement New feature or request area/value-sets Value set families and wrapper arities package/efcore Affects CodoMetis.ValueRanges.EFCore.PostgreSQL stacked Base is another open PR; merge in order, CI does not run until it targets main labels Oct 26, 2026
@CaffeinatedCoder
CaffeinatedCoder changed the base branch from fix/483-reject-nondeterministic-string-set-collation to main October 26, 2026 13:22
# Conflicts:
#	CHANGELOG.md
#	docs/efcore.md
#	src/CodoMetis.ValueRanges.EFCore.PostgreSQL/CHANGELOG.md
#	test/CodoMetis.ValueRanges.EFCore.PostgreSQL.IntegrationTests/DeclaredNullabilityTests.cs
@CaffeinatedCoder CaffeinatedCoder removed the stacked Base is another open PR; merge in order, CI does not run until it targets main label Oct 26, 2026
feat: run text value set operations as SQL where order and repeats don't count (#485)
@CaffeinatedCoder

Copy link
Copy Markdown
Owner Author

Merged into main as 40c81da ("Merge pull request #493 …", parents 3776504 and this PR's head 304c3d7). GitHub's merge call failed with an internal GraphQL error after it had written the merge commit, so the PR was never marked merged. Fast-forwarding the branch to main let GitHub close it, but as closed rather than merged, and its Closes #485 didn't fire. #485 is closed by hand with the same pointer. The 10.x port is #497 (PR #501).

This branch was successfully deployed

1 active deployment
github-pages — 40c81da7 Deployed Oct 26, 2026 by CaffeinatedCoder via Publish #80
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/value-sets Value set families and wrapper arities enhancement New feature or request package/efcore Affects CodoMetis.ValueRanges.EFCore.PostgreSQL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Translate text value set Intersect, Except, Add, Count and equality where the result needs no order (StringSet, StringSet<T>)

1 participant