Skip to content

feat(clp): speed up logtype-insights with a one-command bootstrap and model2vec clustering - #7

Open
robinlee09201 wants to merge 1 commit into
mainfrom
feat(clp)--logtype-insights-fast-path
Open

robinlee09201 wants to merge 1 commit into
mainfrom
feat(clp)--logtype-insights-fast-path

Conversation

@robinlee09201

Copy link
Copy Markdown
Contributor

Summary

Cuts the logtype-insights skill's two dominant costs — kick-start latency and classification wall time — and makes every step visible to the user.

New helpers (shared bin/)

  • logtype-insights-bootstrap — one command replaces the ~6 hand-composed pipelines of the old steps 3–5: schema sample, per-field value distributions (DIST lines), the stats.log_shapes dictionary dump, and the classification-cache probe, printed as a grep-able KEY=VALUE summary (LOGTYPE_COUNT, FALLBACK, CACHE_MODE, APP_KEY, TO_CLASSIFY, file paths). Handles the templatize fallback for pre-shapes-API binaries and converts the frequency table to canonical NDJSON so the cache probe works there too (previously it silently probed an empty file on 0.12.x). SIGPIPE-safe on high-distinct fields (a head -8 mid-pipeline used to kill the script with exit 141 under set -e + pipefail).
  • logtype-cluster (setup / cluster / expand) — model2vec (potion-base-8M, numpy-only) pre-clustering. The classifier labels one representative per cluster by id; expand re-attaches member logtypes byte-exact from the cluster file, so GROWTH cache merging is safe by construction — the LLM never echoes logtype strings.

Skill changes

  • skills-claude: the 621-line monolithic SKILL.md becomes a thin ~170-line workflow plus three on-demand references (logtype-baseline.md, logtype-classify.md, logtype-insight.md) following the plugin's progressive-disclosure pattern.
  • skills-codex: stays a monolith per convention, trimmed to use the same two scripts with inline classification.
  • Classification runs in one haiku-default subagent (sonnet retry on validation failure) instead of pasting every template into sonnet.
  • Both variants keep the mandatory scoped semantic cross-check from Migrate clp skills to the clp-core 0.13 shapes API (stats.log_shapes) #5 and now narrate every step to the user, including all fallback/retry branches.
  • Docs updated: helper lists (README, CONTRIBUTING), LOCAL_TESTING smoke block, release-testing walkthrough, dev preflight.

Test plan

  • bash -n + shellcheck on the bootstrap; py_compile on the cluster script; claude plugin validate + scripts/validate-codex-plugin.sh pass
  • Bootstrap on the vLLM demo archive: LOGTYPE_COUNT=934, FALLBACK=SHAPES_OK, cache probe modes NEW → UPTODATE verified
  • Templatize fallback exercised on a 0.12.x binary (two-phase --message re-run), freqs → NDJSON feeds the cache probe
  • Byte-exactness: expand output diff-identical to the input NDJSON; GROWTH merge on a grown archive returns UPTODATE on re-run
  • SIGPIPE regression: bootstrap exits 0 on an archive with a ~20k-distinct timestamp field
  • End-to-end claude --plugin-dir: id-only assignments, 8-section report

… model2vec clustering

Cuts the skill's two dominant costs — kick-start latency and
classification wall time:

- bin/logtype-insights-bootstrap: one command for schema sampling,
  per-field value distributions, the logtype dictionary dump, and the
  classification-cache probe (KEY=VALUE summary). Handles the templatize
  fallback for pre-shapes-API binaries, converting the frequency table to
  the canonical NDJSON so the cache probe works there too. SIGPIPE-safe
  on high-distinct fields.
- bin/logtype-cluster (setup/cluster/expand): model2vec (potion-base-8M)
  pre-clustering so the classifier labels one representative per cluster
  by id; expand re-attaches member logtypes byte-exact from the cluster
  file, which keeps GROWTH cache merging safe by construction.
- Classification runs in ONE haiku-default subagent (sonnet retry on
  validation failure) instead of pasting every template into sonnet.
- skills-claude/logtype-insights: 621-line monolith split into a thin
  SKILL.md plus on-demand references (logtype-baseline, logtype-classify,
  logtype-insight). skills-codex stays a monolith per convention, trimmed
  and using the same scripts. Both keep the mandatory scoped semantic
  cross-check from #5.
- Both variants now narrate every step to the user, including the
  fallback and retry branches.
- Docs: helper lists (README, CONTRIBUTING), LOCAL_TESTING smoke block,
  release-testing walkthrough, dev preflight.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants