Skip to content

tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379

Open

Description

@jiaran-king

Current state: #380 两轮对照已完成:短指引未达到预定推广门槛;现有目录与自主搜索的效率仍有不确定性。#370 已通过双 NPU 预检并开展正式工具对照;#344 仍待确定首版范围与负责人。
Decision needed: 请求者及维护者复核 #380 的产品决定,并决定是否追加验证预算;仓库级验收由维护者确认。
Next action: 参考 #380 第二轮结果判断是否进行新的有界验证;#370 与 #344 按各自职责推进。

目标与产品决定

让推理优化 Agent 在历史经验可能改变实验或验证决策时,能用已有线索找到、判断和使用适用材料;信息足够时也能直接继续。暂不将 #380 所测短指引推广到产品入口,临时保留现有目录和 rg;这不是现有方式已足够高效的结论。第二轮已用完 12 个受测会话,唯一建议追加的问题及所需预算见完整结果。

工作分工与证据

请求者与仓库维护者负责目标验收和关闭;此跟踪不授予 #370 或 #344 额外实施权限。

Activity

  1. amy-why-3459 commented on Nov 10, 2026

    @amy-why-3459
    Collaborator

    感谢建立这个跟踪入口。“只给目录”对照“同目录加简短检索指引”这个首轮方向和 #370 改变的变量是分开的,我同意先做这一组。冻结 A/B 用例之前,有几点建议,想先对齐一下:

    1. 三类决策点的判定标准:“适用先例”“相似但不适用”“无需搜索”这三类,建议冻结用例时就写明每个用例的期望决策和判定依据,最好由不参与执行的人预先标注。不然结果出来后容易事后归类,准确性这一项就没法和效率分开评估。
    2. “采用”的可观测定义:正文已经强调命中不等于采用,这点我很认同。建议首轮就把“读取了某材料”“依据该材料改变了实验或验证决策”分成两个可从日志里核验的字段,并说明怎么从 Agent 轨迹中判定后者(例如决策理由里引用了材料 id/revision)。如果 task: 支持每个优化迭代的知识库命中归因统计 #344 的逐轮记录格式还没定,首轮能否先用一个测试内的最小记录,避免等 task: 支持每个优化迭代的知识库命中归因统计 #344?
    3. 知识版本固定方式:建议在用例里记录所用知识目录的 revision(例如 KnowledgeReader 返回的 sha256: revision 或 commit),保证两组对照读到的是同一份材料。
    4. 成本口径:“可得成本”建议事先列明具体指标(例如工具调用次数、读取的字节数或 token、总耗时),并说明指引文本本身占用的上下文怎么计入,这样“指引增加成本”的负结果才有可比性。
    5. 样本量与重复:同一模型、同一推理强度下,每个用例跑几次?如果只跑一次,建议结论明确限定为描述性,和正文的表述保持一致。

    另外想确认一下:核心改进任务创建后,会作为本 issue 的子任务关联,还是单独开 issue 再回链?方便后续 review 时对照范围。

  2. added
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    kind:trackingTracking umbrella that follows child work
    on Nov 10, 2026
  3. jiaran-king commented on Nov 10, 2026

    @jiaran-king
    ContributorAuthor

    回应 Amy 在此的建议,执行细节和首个超限样本已同步到 #380。

    • **任务关系:**核心任务是单独的 task: 改进真实优化任务中的材料发现与使用 #380,与本跟踪 issue 双向引用;目前没有建立 GitHub 原生子 issue 关系。
    • **用例与版本:**四个源自公开工程记录的决策片段、语料版本、条件与预算已在首次运行前冻结;task: 改进真实优化任务中的材料发现与使用 #380 记录了语料 commit 和评测侧冻结计划摘要。判据由执行者预先整理,尚无独立标注。
    • **判断与采用:**先按判据评价最终决策,再依据实际工具输出和决策理由分别记录读取、引用和应用;只有命中或引用不能证明决定正确,也不能单凭轨迹声称因果收益。当前执行者参与评价,尚无独立盲评,结果将说明这一限制。
    • **对照与成本:**A、B 都收到目录、文件搜索入口及使用材料时说明依据的要求;B 仅额外收到“何时查、判断适用性”的短指引。记录实际 shell 调用、可得工具输出/读取量、耗时和 token;无法观测的读入字节或费用记未知,B 的输入 token 包含指引。
    • **次数与异常:**正式对照每条件每例一次。首个 A 组在第 13 次 shell 调用后中止,超出 12 次预算,记为超预算、未完成;不会被排除或用补跑替代。已启动的同条件会话仅作追加诊断。单次轨迹只支持描述性结论。
    • **轨迹:**完整原始记录保留在本地;公开报告提供经隐私核对的可复核摘录,并说明原始记录的可访问范围。这里记录的是 Agent 的决策建议,不表示已执行工程改动。
  4. jiaran-king commented on Nov 10, 2026

    @jiaran-king
    ContributorAuthor

    搜索域基线复盘:先例发现、可见送达和决策边界

    承接 roadmap 第 4 项,范围是整个搜索域:材料覆盖、选取/排序、上下文送达、继续查询和适用条件判断;“何时停止”是其中一个假设。当前产品入口先维持原状:#380 所测短指引未达到推广门槛,本次新 case 也未使默认 package Indexer 在两个开发查询进入 top 3。这不是目录/rg 已足够省成本的证明。

    具体决策点 可核实的材料和观察 边界
    Qwen Ascend 首次组合三种优化 历史三项组合初始化 OOM 可改变首次检查的优先级。实际案例中第 16、18、19 行均送达,决定明确引用第 18 行;第 19 行是不同繁忙主机记录,只有送达证据。 历史决策片段,没有新设备实测;不能把第 19 行写成已采用或已误解。#380 S2 · #384 归档
    Qwen 已复核质量门失败 现有事实足以拒绝“已验收”;第二轮一次 A 直接决定、B 仍搜索 5 次,首轮方向相反。 两条第二轮答复均漏写算子级复验,按完整冻结判据是部分满足;不能把不搜索等同于质量完整。#380
    Drama3 双侧 WER 失败 两组可见地读到并使用先例,建议用受控 capture / 动态重放区分共同环境和候选。 仅给诊断建议,不能将历史原因直接移植到本次;不构成 Indexer 对 rg 对照。#380
    真实 MoE 候选 同模型和知识条件下各一条轨迹:Indexer final 2.176×、rg 2.374×;完整会话 token 分别为 8,015,641 / 14,597,699。 完整会话含检索以外活动和大量缓存输入;第 3 轮当时具体查询、正文送达及采用无法由公开摘要确认,不能归因性能差异。#370
    shared-digests,方向明确后的失败先例 用户提供的本地真实轮次:外层工具返回可核实读取 limit-sizing 报告,但旧的资源缓存失败报告无可核实完整读取;#384 机器归档因复合命令给出 events=[]、counts=null,不意味着未读取。新候选在该单轮 CPU Collect 协议内通过。 实际决策引用 profile;旧报告的事后自述不能证明因果。历史 #255 v1 整体 FAIL、单独确认不覆盖原失败,且 CLI 尾部变化原因未知;该关切并未被新一轮协议重新测量。#255;原始私有 trace 留本地。

    离线范围检查:默认 package relevance 只排序已登记的 skill owner,相关 case 正文在 owner 选中后才有机会交付;rg 可在真实仓库范围内搜 evidence/,因此默认入口差异不能伪装成共同语料下的排序差异。共同语料的 75 个 common/ascend owner 中,Qwen/Drama3 casebook 分别排第 23/51;前三未交付目标。rg 对同一资源集可命中相应案例,但行输出与完整上下文是不同交付量,不能直接比较 token 或任务效果。

    经请求者明确授权,新的三入口、三个决策点、各两次 Agent 对照已在 #380 冻结并开始。它将分别报告决策、材料送达/采用与成本;旧 #255 通用准确度未达成的结论、#370 工具对照以及 #344/#384 记录能力各归原 issue,不在此宣称完成。

  5. jiaran-king commented on Nov 16, 2026

    @jiaran-king
    ContributorAuthor

    建议接受 #466 的入口切换,请请求者和维护者在 #379 记录决定。常规知识发现先用 find,再用结果给出的 read 命令读材料。保留旧命令作为兼容和回退入口。 PR #466

    #380 当时比较的是在已有目录和 rg 的环境中增加一段短检索指引。暂不推广的决定针对那次指引试验。#380 第二轮结果

    用 #466 的基础 wheel 在无 PyYAML 的 Python 3.11 环境复测了 ascend graph mode。限定 Skill 时,find 命中 ascend-graph-mode 并报告缺少 PyYAML。遇到这个情况,保留 collect --relevance 'ascend graph mode' --limit 1 --material --json 回退。实测它能找到打包 Skill,随后可用返回的 ref、revision 和 skill_documents.original 通过 resource 读回完整 SKILL.md,字节与安装包一致。 David 的评审说明

    这次提议的验收范围是知识入口。#466 已验证固定查询的命中和案例正文读回。Agent 是否实际采用材料,以及是否带来优化收益,仍待验证。

  6. jiaran-king commented on Nov 17, 2026

    @jiaran-king
    ContributorAuthor

    由 Codex 代为记录 #379 请求者 jiaran-king 确认的范围决定。接受 #466 将 find → read 设为 README、quickstart 和插件的默认知识检索流程。原有命令保留为兼容和回退入口。无 PyYAML 时,可以用 collect --relevance 重新搜索打包的 Skill,判断新结果是否适用后再读回正文。 PR #466

    #380 暂不推广的决定针对当时测试的短检索指引。#466 验证了固定查询的命中和案例正文读回,单次 Qwen-Image 对照也记录了两份材料的具体采用。稳定的决策质量改善和优化收益仍待验证,#379 的后续目标验收按原有职责推进。 #380 第二轮结果

  7. Dong1017 commented on Nov 17, 2026

    @Dong1017
    Collaborator

    Posted by an agent on behalf of @Dong1017 (incremental only; points already raised by @david6666666 and the prior reviewers are not repeated).

    Incremental finding on the current #466 head: the accepted default workflow is still missing from the WorkSwarm plugin instructions.

    The accepted #379 scope decision says to make find → read the default knowledge-retrieval flow in the README, quickstart, and plugins (decision). The InferRSI plugin was updated, but the WorkSwarm skill still presents the old discovery path as its research workflow: knowledge research, knowledge search, and collect --relevance are the only discovery commands in plugins/workswarm/inferrsi/skills/inferrsi/SKILL.md:42-49, followed by an exact read. A user following that installed plugin will therefore not get the accepted cross-kind find → displayed read command, casebook results, or the documented PyYAML fallback.

    Please update that WorkSwarm “Research and diagnose” section to lead with find "<problem or symptom>" and its displayed read arguments, and carry over the PyYAML fallback guidance. Keep the existing package-only commands available as compatibility/fallback operations, as the #379 decision requires.

    Scope: read #379 and all five comments, #380, #370, PR #466 metadata/comments/reviews, and the current pr466 head diff. I did not run the test suite or modify code.

  8. Dong1017 commented on Nov 17, 2026

    @Dong1017
    Collaborator

    Posted by an agent on behalf of @Dong1017 (authorized steward while hosted Dong-bot is on usage_limit).

    Maintainer decision — accept find → read as default discovery for #466

    Accepted. Agree with the requester confirmation (comment):

    Implementer note for #466: WorkSwarm’s Research section still needs the same find → read default (InferRSI plugin already updated). Steward Request changes on that gap: #466 (review)

  9. david6666666 commented on Nov 17, 2026

    @david6666666
    Collaborator

    Could the tracker record default-entry delivery separately from evidence of decision quality? The frozen #380 comparison uses collect, collect plus case lookup, and directory/rg; none of its arms runs find → read. Its results can inform the older retrieval designs, but should not count as validation of #466’s default. Please keep that distinction in the acceptance record when the results arrive.

  10. zengchuang-hw commented on Nov 17, 2026

    @zengchuang-hw
    Contributor

    Independent code/evidence review at main 61303fa8150472dfbda62ec8d77908bc05c73f3b.

    Finding: useful discovery and attribution infrastructure is present; the tracking goal is not yet a demonstrated retrieval-efficiency or optimization-quality result. I reviewed the material-discovery/use goal, not linked discussion conclusions or product/budget decisions.

    • src/inferrsi/knowledge/knowledge_find.py:732-783 ranks skills, records and cases together, exposes matched items and checks whether selected material can be opened. Its own contract correctly says that a lexical match is not proof of applicability. This supports finding historical material from partial clues, not an automatic adoption decision.
    • knowledge/knowledge_operations.py:786-813 retains the actual collect request, ranking method, returned references/revisions, counts and assembly state. harness/optimization.py:145-157 retains delivered context by path and digest. These are auditable retrieval/delivery facts, not evidence that an agent read or used a resource.
    • apps/views/search_attribution.py:270-339 supports round-bounded search/read observations, caller-reviewed use and candidate-result links. Missing/partial coverage produces null counts rather than a misleading zero. harness/records/optimization_log.py:150-174,208-224 separately records purpose, takeaways, used/rejected references and round context links. The delivery tests also require a matching versioned resource block in a model-facing return. These distinctions are important: hit, delivered/read, adopted and successful outcome must not be collapsed into one number.
    • Continuing without a fresh search is supported by the workflow contract (docs/optimization-workflow-unification.md:196); it does not require retrieval on every round. A comparison should preserve that valid no-new-search case rather than count it as an automatic failure.

    Remaining evidence/acceptance gap: the clean requirement supplied for this tracking item does not define an original quantitative threshold, frozen tool-comparison protocol or repository-level closure criterion. The code/tests above cannot supply those missing decisions or establish which retrieval tool improves real optimization. A defensible comparison would need a pinned corpus and workload/query set, tool versions/configuration, equal budgets and knowledge conditions, an explicit applicability/read/adoption rubric, and round-level links from the cited material to the changed experiment or validation decision and its measured outcome. These are review recommendations, not accepted scope or a commitment to implement/run a campaign.

    Validation actually run: PYTHONPATH=$PWD/src <existing-test-venv>/bin/python -m pytest -q tests/test_knowledge_find.py tests/test_search_capture_delivery.py tests/test_search_attribution.py tests/test_optimization_story.py tests/test_optimization_trace.py: 150 tests passed, exit 0, against the pinned checkout. Collection independently confirmed 60 + 1 + 18 + 41 + 30 tests. No repository changes. No full-suite/CI verification, external retrieval-tool comparison, paid/model/device experiment, or inference performance/quality measurement was run.

    Conclusion: focused scaffold behavior is verified; end-to-end improvement and Issue closure remain INCONCLUSIVE, not PASS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:trackingTracking umbrella that follows child workstatus:proposedWaiting for the decision named in the status block

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions