Skip to content

fix(plugins): memoise BigQuery analytics readiness across re-initialisation - #7020

Open
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7017-bqaa-readiness-reruns
Open

fix(plugins): memoise BigQuery analytics readiness across re-initialisation#7020
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7017-bqaa-readiness-reruns

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Summary

Closes #7017.

BigQueryAgentAnalyticsPlugin re-ran its full table/view readiness pass on every re-initialisation instead of once per process. Each pass issues one CREATE OR REPLACE VIEW statement per _EVENT_VIEW_DEFS entry (25 at HEAD), and it is awaited from before_run_callback, so the DDL sits on the agent request path.

Any deployment that shares one plugin instance across short-lived Runners (Runner.close()PluginManager.close()plugin.close() clears _started) re-runs all the view DDL per request. The reporter measured ~19k view-DDL jobs/hour, per-table BigQuery quota exhausted in ~2h, and ~25s added to median latency — a 2.3.0 → 2.8.0 regression.

In 2.3.0 the readiness pass was gated behind the schema cache, so it ran at most once per process. At HEAD that gate was deliberately removed so a failed first attempt would still retry — but the comment's load-bearing assumption ("once _started is True the steady state pays no extra RPC") only holds when the plugin is initialised once per process, which is false whenever it is closed and reused.

Fix

  • Memoise readiness success in a new _schema_ready flag that, like _schema, survives close()/shutdown(), and gate _ensure_schema_exists on it in _lazy_setup.
    • A failed attempt raises before the flag is set, so it is still retried on the next setup (preserves the 2.8.0 intent).
    • A successful attempt is memoised, so it is not repeated per re-initialisation (restores the 2.3.0 cost profile). No new steady-state RPC.
  • Secondary: emit exactly one WARNING when a generation mismatch aborts an otherwise-successful setup, so a plugin churning through full setups is no longer completely silent (the issue observed 12 log lines against ~2000 setup runs).

Scope is limited to the readiness memoisation and the one diagnostic log line. The view SQL, _EVENT_VIEW_DEFS, and the _ensure_started backoff are untouched (the issue's non-goals).

Testing Plan

Added TestReadinessMemoisedAcrossReinit with two tests:

  • test_readiness_pass_runs_once_across_reinit_cycles — 3 _ensure_started() / shutdown() cycles over one shared plugin issue the view DDL once total (client.query called len(_EVENT_VIEW_DEFS) times), not once per cycle.
  • test_failed_readiness_is_retried_and_not_memoised — a first readiness attempt that raises leaves _schema_ready False and is retried on the next setup, which then succeeds and memoises.

Evidence the tests prove the bug: with the source change stashed (test kept), both new tests fail (AttributeError: ... _schema_ready / would-be 75 vs 25 DDL statements); with the fix restored they pass.

# without the fix (source change stashed):
2 failed in 24.20s

# with the fix:
tests/unittests/plugins/test_bigquery_agent_analytics_plugin.py  505 passed, 6 skipped in 482.90s

Lint/format on the changed files: pyink (2 files unchanged), isort (exit 0), ruff check (all checks passed).

AI assistance disclosure

This change was prepared with AI assistance (Claude). A human reviewed the diff, the tests, and the failing-without-fix evidence before submission.

…sation

BigQueryAgentAnalyticsPlugin re-ran its full table/view readiness pass on
every re-initialisation instead of once per process. Each pass issues one
CREATE OR REPLACE VIEW statement per _EVENT_VIEW_DEFS entry, awaited from
before_run_callback, so the DDL sat on the request path. A host that builds
a short-lived Runner per request over one shared plugin closes the plugin
after every request (Runner.close() -> PluginManager.close() ->
plugin.close()), which clears _started, so the next request re-ran all the
view DDL. In production this exhausted the per-table BigQuery quota and
added ~25s to median latency (2.3.0 -> 2.8.0 regression).

Remember that readiness succeeded in a flag (_schema_ready) that, like
_schema, survives close()/shutdown(), and gate the readiness pass on it. A
*failed* attempt raises before the flag is set and is still retried on the
next setup (the 2.8.0 intent); a *successful* one is not repeated
(restoring the 2.3.0 cost profile). Also log one WARNING when a generation
mismatch aborts an otherwise-successful setup, so a plugin churning through
full setups is no longer silent.

Closes google#7017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] BigQueryAgentAnalyticsPlugin re-runs all view DDL on every re-initialisation, on the request path (2.3.0 -> 2.8.0 regression)

2 participants