Skip to content

fix(queryset): skip auto GROUP BY when values() selects only aggregates - #2294

Open
d3vyce wants to merge 1 commit into
tortoise:developfrom
d3vyce:fix/aggregate-only-group-by
Open

d3vyce wants to merge 1 commit into
tortoise:developfrom
d3vyce:fix/aggregate-only-group-by

Conversation

@d3vyce

@d3vyce d3vyce commented Oct 5, 2026

Copy link
Copy Markdown

Description

Skips the automatic GROUP BY on the model columns when values()/values_list() select only aggregates and the only joins come from filters:

await BillInfo.filter(order__pay_time__gt=0).annotate(total=Sum("amount")).values("total")
# before: [{'total': 2}, {'total': 4}, {'total': 6}, {'total': 8}, {'total': 10}]
# after:  [{'total': 30}]

Without the relation filter, the same query already returns one row. Joins from the annotations, the selected fields or the ordering still group, so Author.annotate(n=Count("books")).values("n") stays per author. Model querysets, count(), exists(), and queries with a HAVING or an ORDER BY keep the current grouping. ValuesQuery now passes its selected fields to resolve_filters(), like ValuesListQuery already does.

Through a reverse FK or m2m filter, the join repeats each object once per matching row, so the aggregate now sees those rows: Event.filter(participants__id__in=[...]).annotate(n=Count("event_id")).values("n") returns [{'n': 4}] instead of [{'n': 3}, {'n': 1}], and Count("event_id", distinct=True) counts the events ([{'n': 2}]). This is what the same SQL returns without the ORM.

Motivation and Context

Closes #613, closes #1149. A total over rows filtered on a relation came back as one row per object, because the filter's join triggered the GROUP BY meant for per-object aggregates over a relation.

How Has This Been Tested?

New tests in tests/test_aggregation.py, run on SQLite and PostgreSQL 18 (asyncpg). The full suite passes on SQLite and asyncpg. MySQL and MSSQL are left to CI.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added the changelog accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

This fix has been partially implemented by Claude Opus 5.5. All modifications made by the AI have been reviewed by me.

@d3vyce
d3vyce force-pushed the fix/aggregate-only-group-by branch from dbaf23b to 6e653c4 Compare October 5, 2026 17:12
@codspeed

codspeed Bot commented Oct 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 24 untouched benchmarks


Comparing d3vyce:fix/aggregate-only-group-by (6e653c4) with develop (1f896c0)

Open in CodSpeed

@waketzheng

Copy link
Copy Markdown
Contributor

Two things I'd like to confirm, plus a test gap.

1. _make_query: removed selects filtering — is resolve_filters() responsible for narrowing selects?

The old code did:

self.resolve_filters()
self.query._selects = [
    select for select in self.query._selects if select.alias in self._fields_for_select
]

Now that block is replaced with self.resolve_filters(self._fields_for_select.keys()). If resolve_filters() only adds selects for fields_for_select but doesn't remove selects that are already in self.query._selects, model fields that were selected earlier could remain. That would be a problem for two reasons:

  • values("total") could return extra model columns alongside total.

  • The GROUP BY decision in resolve_filters() uses self.query._selects:

    has_joins = fields_for_select is None or not all(
    select.is_aggregate for select in self.query._selects
    )

If a non-aggregate model select is still present, all(select.is_aggregate ...) is False, has_joins becomes True, and the GROUP BY is added — the fix silently stops working.

Could you confirm whether resolve_filters() narrows self.query._selects to fields_for_select, or whether the explicit filtering needs to stay? A test asserting values("total") returns a dict with only the total key would lock this in.

2. Test gaps

The current tests cover relation-filter and m2m-filter aggregates, plus the per-object case. Missing:

  • values("model_field", "aggregate") — mixed selection should still GROUP BY per object.
  • HAVING / ORDER BY — the PR description says these keep the current grouping, but there's no test locking it in.
  • values("total") returning only total (see point 1).

Minor: readability of the has_joins logic

The timing works because has_joins is captured after _resolve_annotate but before the for node in self._q_objects loop adds filter joins. A one-line comment noting that ordering would help future readers see why not has_joins and self._joined_tables can be true at that point.

@d3vyce
d3vyce force-pushed the fix/aggregate-only-group-by branch from 6e653c4 to 5c7b4e1 Compare October 7, 2026 16:07
@d3vyce

d3vyce commented Oct 7, 2026 •

Copy link
Copy Markdown
Author

Thanks for the review!

  1. resolve_filters() doesn't need to narrow _selects, because nothing else is in there. ValuesQuery._make_query starts from a fresh copy(basequery) with no selects, add_field_to_select_query only selects the values() fields, and _resolve_annotate(fields_for_select) only selects annotations listed in fields_for_select. resolve_ordering never adds selects. The removed block only dropped annotations that weren't requested, and passing fields_for_select now keeps them out. I added values("count") == [{"count": 3}] with a second annotation that isn't selected. The == comparisons on dicts in the other tests also fail on extra keys.

  2. Added test_aggregate_values_with_related_filter_stays_per_object: values("name", "count"), a HAVING (filter(count__gte=1)), and an order_by("count") all still return one row per book.

Minor. Reworded the comment on has_joins to say it is captured before the filter joins are added.

Ready for another look whenever you have time @waketzheng !

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to achieve SELECT count(distinct step_id)... annotate group_by confused

2 participants