fix(plugins): memoise BigQuery analytics readiness across re-initialisation - #7020
Open
chelsealong wants to merge 1 commit into
Open
fix(plugins): memoise BigQuery analytics readiness across re-initialisation#7020chelsealong wants to merge 1 commit into
chelsealong wants to merge 1 commit into
Conversation
…sation BigQueryAgentAnalyticsPlugin re-ran its full table/view readiness pass on every re-initialisation instead of once per process. Each pass issues one CREATE OR REPLACE VIEW statement per _EVENT_VIEW_DEFS entry, awaited from before_run_callback, so the DDL sat on the request path. A host that builds a short-lived Runner per request over one shared plugin closes the plugin after every request (Runner.close() -> PluginManager.close() -> plugin.close()), which clears _started, so the next request re-ran all the view DDL. In production this exhausted the per-table BigQuery quota and added ~25s to median latency (2.3.0 -> 2.8.0 regression). Remember that readiness succeeded in a flag (_schema_ready) that, like _schema, survives close()/shutdown(), and gate the readiness pass on it. A *failed* attempt raises before the flag is set and is still retried on the next setup (the 2.8.0 intent); a *successful* one is not repeated (restoring the 2.3.0 cost profile). Also log one WARNING when a generation mismatch aborts an otherwise-successful setup, so a plugin churning through full setups is no longer silent. Closes google#7017
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #7017.
BigQueryAgentAnalyticsPluginre-ran its full table/view readiness pass on every re-initialisation instead of once per process. Each pass issues oneCREATE OR REPLACE VIEWstatement per_EVENT_VIEW_DEFSentry (25 at HEAD), and it is awaited frombefore_run_callback, so the DDL sits on the agent request path.Any deployment that shares one plugin instance across short-lived
Runners (Runner.close()→PluginManager.close()→plugin.close()clears_started) re-runs all the view DDL per request. The reporter measured ~19k view-DDL jobs/hour, per-table BigQuery quota exhausted in ~2h, and ~25s added to median latency — a 2.3.0 → 2.8.0 regression.In 2.3.0 the readiness pass was gated behind the schema cache, so it ran at most once per process. At HEAD that gate was deliberately removed so a failed first attempt would still retry — but the comment's load-bearing assumption ("once
_startedis True the steady state pays no extra RPC") only holds when the plugin is initialised once per process, which is false whenever it is closed and reused.Fix
_schema_readyflag that, like_schema, survivesclose()/shutdown(), and gate_ensure_schema_existson it in_lazy_setup.WARNINGwhen a generation mismatch aborts an otherwise-successful setup, so a plugin churning through full setups is no longer completely silent (the issue observed 12 log lines against ~2000 setup runs).Scope is limited to the readiness memoisation and the one diagnostic log line. The view SQL,
_EVENT_VIEW_DEFS, and the_ensure_startedbackoff are untouched (the issue's non-goals).Testing Plan
Added
TestReadinessMemoisedAcrossReinitwith two tests:test_readiness_pass_runs_once_across_reinit_cycles— 3_ensure_started()/shutdown()cycles over one shared plugin issue the view DDL once total (client.querycalledlen(_EVENT_VIEW_DEFS)times), not once per cycle.test_failed_readiness_is_retried_and_not_memoised— a first readiness attempt that raises leaves_schema_readyFalse and is retried on the next setup, which then succeeds and memoises.Evidence the tests prove the bug: with the source change stashed (test kept), both new tests fail (
AttributeError: ... _schema_ready/ would-be 75 vs 25 DDL statements); with the fix restored they pass.Lint/format on the changed files:
pyink(2 files unchanged),isort(exit 0),ruff check(all checks passed).AI assistance disclosure
This change was prepared with AI assistance (Claude). A human reviewed the diff, the tests, and the failing-without-fix evidence before submission.