Repository navigation
docs: rebuild the docs bot index before the release, re-record its evals - #628
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs-index:checkwarns about 14 pages that changed since the index was last built in #517. Three of them are new: the site indexing guide (#537), the test-before-launch guide (#570) and the React page (#583). Until they're in the index, the docs bot answers from the old pages and doesn't know@chatcops/reactexists, so this goes in before the release (#627).This rebuilds the index (32 pages, 347 chunks,
openai:text-embedding-3-small@512) and re-records both eval files, since the new chunks change what the bot gets as context.How I tested this
pnpm docs-index:check: up to date.pnpm --filter @chatcops/website evals(replay, as in CI): 9 of 9 tests, 12 of 12 assertions.pnpm --filter @chatcops/website evals:citations: 48 of 50 cite the right page (96%, needs 85%), the same as in docs: rebuild the docs bot index for the 1.0 snippets, re-record its evals #501. The misses are the Cloudflare cleanup question (cited/server/storage/, which covers it) and the lead callback question (cited/server/express/and/examples/cloudflare-full/).Bearerand?key=matches are docs text the bot quoted.