feat(clp): speed up logtype-insights with a one-command bootstrap and model2vec clustering - #7
Open
robinlee09201 wants to merge 1 commit into
Open
robinlee09201 wants to merge 1 commit into
robinlee09201 wants to merge 1 commit into
Conversation
… model2vec clustering Cuts the skill's two dominant costs — kick-start latency and classification wall time: - bin/logtype-insights-bootstrap: one command for schema sampling, per-field value distributions, the logtype dictionary dump, and the classification-cache probe (KEY=VALUE summary). Handles the templatize fallback for pre-shapes-API binaries, converting the frequency table to the canonical NDJSON so the cache probe works there too. SIGPIPE-safe on high-distinct fields. - bin/logtype-cluster (setup/cluster/expand): model2vec (potion-base-8M) pre-clustering so the classifier labels one representative per cluster by id; expand re-attaches member logtypes byte-exact from the cluster file, which keeps GROWTH cache merging safe by construction. - Classification runs in ONE haiku-default subagent (sonnet retry on validation failure) instead of pasting every template into sonnet. - skills-claude/logtype-insights: 621-line monolith split into a thin SKILL.md plus on-demand references (logtype-baseline, logtype-classify, logtype-insight). skills-codex stays a monolith per convention, trimmed and using the same scripts. Both keep the mandatory scoped semantic cross-check from #5. - Both variants now narrate every step to the user, including the fallback and retry branches. - Docs: helper lists (README, CONTRIBUTING), LOCAL_TESTING smoke block, release-testing walkthrough, dev preflight.
jackluo923
approved these changes
Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cuts the
logtype-insightsskill's two dominant costs — kick-start latency and classification wall time — and makes every step visible to the user.New helpers (shared
bin/)logtype-insights-bootstrap— one command replaces the ~6 hand-composed pipelines of the old steps 3–5: schema sample, per-field value distributions (DISTlines), thestats.log_shapesdictionary dump, and the classification-cache probe, printed as a grep-ableKEY=VALUEsummary (LOGTYPE_COUNT,FALLBACK,CACHE_MODE,APP_KEY,TO_CLASSIFY, file paths). Handles the templatize fallback for pre-shapes-API binaries and converts the frequency table to canonical NDJSON so the cache probe works there too (previously it silently probed an empty file on 0.12.x). SIGPIPE-safe on high-distinct fields (ahead -8mid-pipeline used to kill the script with exit 141 underset -e+pipefail).logtype-cluster(setup/cluster/expand) — model2vec (potion-base-8M, numpy-only) pre-clustering. The classifier labels one representative per cluster by id;expandre-attaches member logtypes byte-exact from the cluster file, so GROWTH cache merging is safe by construction — the LLM never echoes logtype strings.Skill changes
logtype-baseline.md,logtype-classify.md,logtype-insight.md) following the plugin's progressive-disclosure pattern.Test plan
bash -n+ shellcheck on the bootstrap;py_compileon the cluster script;claude plugin validate+scripts/validate-codex-plugin.shpassLOGTYPE_COUNT=934,FALLBACK=SHAPES_OK, cache probe modes NEW → UPTODATE verified--messagere-run), freqs → NDJSON feeds the cache probeexpandoutput diff-identical to the input NDJSON; GROWTH merge on a grown archive returns UPTODATE on re-runclaude --plugin-dir: id-only assignments, 8-section report