Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
257e71f
fix(structurize): строить запись для привязки с фильтром колонтитулов…
fUS1ONd Oct 10, 2026
7c5133f
fix(catalog): сообщение об ошибке батча называет реальный максимум
fUS1ONd Oct 10, 2026
75792d7
refactor(slides): удалить неиспользуемые AnchorResponse и промпт docu…
fUS1ONd Oct 10, 2026
2e223c1
feat(transcribe): кеш транскриптов для перезапуска без повторной тран…
fUS1ONd Oct 10, 2026
cf953af
fix(structurize): справочник написаний только со слайдов раздела ±1
fUS1ONd Oct 10, 2026
3f21854
fix(structurize): резать подтемы по границе реплики, без дублей на ст…
fUS1ONd Oct 10, 2026
f748243
fix(slides): непривязанный слайд со слабым доказательством идёт в гал…
fUS1ONd Oct 10, 2026
7183560
fix(transcribe): уникальный tmp при записи в кеш транскриптов, тест д…
fUS1ONd Oct 10, 2026
3f29a02
fix(slides): слайд, пропущенный путём по порядку колоды, идёт в галер…
fUS1ONd Oct 10, 2026
a4ca7b5
fix(slides): причина неупомянутого слайда в диагностике
fUS1ONd Oct 10, 2026
f447912
Merge ticket smf-01: перезапуск без повторной транскрипции
fUS1ONd Oct 10, 2026
4afb402
test(slides): формула силы доказательства и маршрут непривязанного сл…
fUS1ONd Oct 10, 2026
122a75c
Merge branch 'feat/slide-matcher-fixes' into impl/smf-05-unbound-gallery
fUS1ONd Oct 10, 2026
1727416
fix(structurize): справочник написаний после фрагмента транскрипта
fUS1ONd Oct 10, 2026
8b93a75
test(slides): краевые случаи маршрута непривязанных слайдов
fUS1ONd Oct 10, 2026
04ff2c2
fix(structurize): устойчивая к краевым случаям нарезка подтем по репл…
fUS1ONd Oct 10, 2026
44ff655
Merge branch 'feat/slide-matcher-fixes' into impl/smf-02-subtopic-bou…
fUS1ONd Oct 10, 2026
b158c80
Merge ticket smf-02: подтемы режутся по границе реплики
fUS1ONd Oct 10, 2026
f5a9046
fix(structurize): хвост транскрипта в последнюю подтему, смежные инте…
fUS1ONd Oct 10, 2026
1ad9384
Merge ticket smf-02: хвост транскрипта и смежные интервалы подтем
fUS1ONd Oct 10, 2026
5df9346
Merge ticket smf-09-debt
fUS1ONd Oct 10, 2026
0e123cb
Merge ticket smf-05-unbound-gallery
fUS1ONd Oct 10, 2026
33bdee3
Merge ticket smf-08-spelling-glossary
fUS1ONd Oct 10, 2026
4c0e243
feat(worker): TTL-очистка workspace упавших задач с marker и инжектир…
fUS1ONd Oct 10, 2026
54c0293
feat(slides): отрицательный margin делает слайд непривязанным
fUS1ONd Oct 10, 2026
6ca7aec
feat(slides): допуск margin −5 по умолчанию до калибровки
fUS1ONd Oct 10, 2026
cf0c67b
fix(worker): marker из будущего уступает mtime; 410 на транскрипт упа…
fUS1ONd Oct 10, 2026
9dd8899
Merge ticket workspace-ttl/01: TTL-очистка workspace упавших задач
fUS1ONd Oct 10, 2026
16b3367
fix: пропущенный путём слайд с отрицательным competition_margin уходи…
fUS1ONd Oct 10, 2026
43d0313
Merge branch 'feat/slide-matcher-fixes' into impl/smf-06-negative-margin
fUS1ONd Oct 10, 2026
868b569
style: ruff
fUS1ONd Oct 10, 2026
80ce1b7
Merge ticket smf-06: отрицательный margin делает слайд непривязанным
fUS1ONd Oct 10, 2026
a34895e
feat(slides): визуальное описание слайда как доказательство привязки
fUS1ONd Oct 10, 2026
12f19c3
fix(slides): визуальное описание не даёт verified без судьи и не совп…
fUS1ONd Oct 10, 2026
adc1dac
Merge ticket smf-07: визуальное описание слайда как доказательство
fUS1ONd Oct 10, 2026
db420d6
docs(glossary): статус unbound и причины статуса слайда
fUS1ONd Oct 10, 2026
4bf50b3
docs(alignment): русский докстринг _lexical_ground
fUS1ONd Oct 10, 2026
3074e20
refactor(alignment): AlignmentWeights и профиль визуального описания …
fUS1ONd Oct 10, 2026
ff3a930
refactor(alignment): одна формула силы доказательства для DP и compet…
fUS1ONd Oct 10, 2026
ec0f47b
refactor(alignment): reason_code неупомянутого из status_reason, един…
fUS1ONd Oct 10, 2026
ebf144b
refactor(alignment): UnboundReason заменён SlideStatusReason, min_mar…
fUS1ONd Oct 10, 2026
17840d6
refactor(srt): snap_sections_to_cues переехал в infrastructure/srt.py
fUS1ONd Oct 10, 2026
543910b
feat(alignment): честные status_reason для alignment_failed, legacy_m…
fUS1ONd Oct 10, 2026
c6a3093
fix(grounding): короткие слова-описатели сравниваются по целым словам…
fUS1ONd Oct 10, 2026
1ed3ea1
refactor(worker): одна проверка task_id для cleanup и sweeper
fUS1ONd Oct 10, 2026
de5cf2d
test(api): 410 transcript_unavailable для упавшей задачи без транскрипта
fUS1ONd Oct 10, 2026
aed307e
feat(worker): sweeper чистит кеш транскриптов по сроку TRANSCRIPT_CAC…
fUS1ONd Oct 10, 2026
b12ebd4
feat(deploy): именованный volume под кеш транскриптов
fUS1ONd Oct 10, 2026
50ddbf0
Merge исправлений по code-review этапа 1
fUS1ONd Oct 10, 2026
b5953b2
docs(benchmark): раскладка новых лекторов по validation/holdout и мат…
fUS1ONd Oct 11, 2026
1f4ce6d
docs(benchmark): базовая линия этапа 1 на validation — v2 и legacy
fUS1ONd Oct 11, 2026
5ec2da4
fix(alignment): калибровка порога и краевые случаи финальной охоты эт…
fUS1ONd Oct 11, 2026
3b18ff9
docs(benchmark): замер «после» этапа 1 и проверка порога min_margin
fUS1ONd Oct 11, 2026
5066f23
Merge remote-tracking branch 'origin/dev' into feat/slide-matcher-fixes
fUS1ONd Oct 11, 2026
ee83e1e
docs(benchmark): проверка min_margin −2.5 на validation-лекциях swe
fUS1ONd Oct 11, 2026
9bb6b0c
fix(alignment): маркер слайда у абзаца с репликой-доказательством, пр…
fUS1ONd Oct 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,33 @@ _Не путать:_ с размещением — привязка может
галерею раздела с самым сильным доказательством.
_Не путать:_ с неупомянутым — «не уверены, куда поставить» не значит «лектор не обсуждал».

**Сборка**
Соседние страницы колоды, где следующая повторяет текст предыдущей и добавляет к нему — один слайд,
показанный по шагам. Шаги одной сборки — одна страница в нескольких видах: общее доказательство у
них не считается коллизией. Колонтитулы при сравнении текстов не учитываются.

**Колонтитул**
Строка, повторяющаяся на многих страницах колоды (название курса, автор, номер) — не содержание
слайда; не участвует ни в поиске доказательств, ни в распознавании сборки.

**Статус слайда**
Итог выравнивания для страницы колоды: `discussed` (привязан к месту речи), `unbound`
(непривязанный: раздел известен, места речи нет), `unmentioned` (неупомянутый), `duplicate`
(точный дубль другой страницы), `deck_mismatch` (колода признана посторонней).

**Причина статуса**
Почему слайд получил статус `unbound`, `unmentioned` или `deck_mismatch` (поле `status_reason`):
- `weak_evidence` (`unbound`) — доказательство есть, но слишком слабое для вставки в абзац;
- `deck_order_conflict` (`unbound`) — путь по порядку колоды пропустил слайд, хотя доказательство есть;
- `stronger_competitor` (`unbound`) — другой раздел совпадает с речью сильнее выбранного больше чем на порог
(`min_margin`, мера без бонуса вердикта судьи); слайд уходит в галерею этого раздела и не служит
опорой пути по порядку колоды;
- `no_candidates` (`unmentioned`) — ни одного кандидата с доказательством;
- `service_role` (`unmentioned`) — служебная пустая страница;
- `alignment_failed` (`unmentioned`) — выравнивание упало, слайд не оценивался;
- `legacy_mode` (`unmentioned`) — старый режим не привязал слайд, выравнивание по доказательствам не запускалось;
- `deck_mismatch` (`deck_mismatch`) — колода признана посторонней, слайд не оценивался.

**Галерея раздела**
Блок слайдов в конце раздела для слайдов, чья привязка к разделу есть, а к абзацу — недостаточно уверенная.

Expand Down
189 changes: 189 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-after-anf-v2.md

Large diffs are not rendered by default.

250 changes: 250 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-after-db-v2.md

Large diffs are not rendered by default.

285 changes: 285 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-after2-anf-v2.md

Large diffs are not rendered by default.

219 changes: 219 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-after2-db-v2.md

Large diffs are not rendered by default.

161 changes: 161 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-baseline-anf-legacy.md

Large diffs are not rendered by default.

162 changes: 162 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-baseline-anf-v2.md

Large diffs are not rendered by default.

225 changes: 225 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-baseline-db-v2.md

Large diffs are not rendered by default.

204 changes: 204 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-val-swe-03-26-v2.md

Large diffs are not rendered by default.

225 changes: 225 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-val-swe-04-09-v2.md

Large diffs are not rendered by default.

230 changes: 230 additions & 0 deletions benchmarks/lecture-quality/2026-10-11-val-swe-04-23-v2.md

Large diffs are not rendered by default.

9 changes: 8 additions & 1 deletion benchmarks/lecture-quality/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,15 @@ not read these baselines until they have completed both passes of their own audi
| `2026-07-29-judge-i-proper-nouns.md` | 2026-02-12 | I (runs G and H were not judged) |
| `2026-07-29-judge-j-02-26.md` | 2026-02-26 | J, same configuration as I |
| `2026-07-29-judge-k-03-12.md` | 2026-03-12 | K, first validation lecture |
| `2026-10-11-baseline-db-v2.md` | db-2026-09-24 | stage 1 baseline, v2 |
| `2026-10-11-baseline-anf-v2.md` | compilers-anf | stage 1 baseline, v2 |
| `2026-10-11-baseline-anf-legacy.md` | compilers-anf | stage 1 baseline, legacy |
| `2026-10-11-val-swe-03-26-v2.md` | swe-2026-03-26 | stage 1 validation of `min_margin` −2.5, v2 |
| `2026-10-11-val-swe-04-09-v2.md` | swe-2026-04-09 | stage 1 validation of `min_margin` −2.5, v2 |
| `2026-10-11-val-swe-04-23-v2.md` | swe-2026-04-23 | stage 1 validation of `min_margin` −2.5, v2 |

Run configurations and cross-run analysis live in
`docs/progress/2026-07-27-matcher-model-experiments.md`. Dataset splits and the release
`docs/progress/2026-07-27-matcher-model-experiments.md`; the stage 1 baseline summary is
`docs/progress/2026-10-11-stage1-baseline.md`. Dataset splits and the release
gates live in `generalization-roadmap.md`.

55 changes: 55 additions & 0 deletions benchmarks/lecture-quality/generalization-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,46 @@ wrong-topic rate 11.9%, high-confidence error rate 13.0%, verified precision 100
(26/26). This is a snapshot, not a passed gate: no revision has been evaluated against
unseen data since `2026-03-12` was retired.

### Split of the new lecturers (2026-10-11, fixed before any run)

Two new lecturers were added: a databases course (video recording + PPTX deck) and a
compilers course (audio recording + PDF deck). Rule, chosen after download and before any
pipeline run, judging or listening: for each lecturer the earlier lecture goes to validation and the later
one to the hidden holdout, so each style is checked in both sets.

| Lecture | Lecturer / modality | Split | Files under `test-data/` |
| --- | --- | --- | --- |
| `db-2026-09-24` | databases, video + PPTX | development (was validation) | `document-slide-alignment/db-2026-09-24/` |
| `compilers-anf` | compilers, audio + PDF | development (was validation) | `document-slide-alignment/compilers-anf/` |
| `db-2026-10-07` | databases, video + PPTX | holdout | `holdout/db-2026-10-07/` |
| `compilers-riscv` | compilers, audio + PDF | holdout | `holdout/compilers-riscv/` |
| `swe-2026-03-26` | software engineering (same lecturer as 02-12…05-07), audio + PDF | development (was validation) | `document-slide-alignment/swe-2026-03-26/` |
| `swe-2026-04-09` | same | development (was validation) | `document-slide-alignment/swe-2026-04-09/` |
| `swe-2026-04-23` | same | development (was validation) | `document-slide-alignment/swe-2026-04-23/` |

- 2026-10-11: both validation lectures retired to development. Stage 1 (ticket 10)
calibrated `min_margin` on their slide-by-slide margins (`docs/progress/2026-10-11-stage1-after.md`),
so validation is empty again; the calibrated threshold is provisional until it is checked on
new validation lectures.
- 2026-10-11: validation refilled with the three lectures of the development lecturer that were
never processed (03-26, 04-09, 04-23; 02-12, 02-26, 03-12 and 05-07 were already seen). Same
lecturer, so they check the threshold against overfitting to db/anf, not across lecturers.
- 2026-10-11: the three swe lectures were judged once (`2026-10-11-val-swe-*-v2.md`) and
did not refute `min_margin` −2.5; no threshold separates right and wrong path choices, and the
owner kept −2.5 (`docs/progress/2026-10-11-stage1-after.md`). Their per-slide failures are now
read and feed the next tickets, so — as with 03-12 — they are retired to development.
Validation is empty again and must be refilled (preferably with a new lecturer) before the
next matcher revision is accepted; holdout is untouched.
- Video lectures keep both `lecture.mp4` (stages 2 and 3) and the extracted
`lecture.m4a` (document mode); `test-data/` is not tracked by git.
- Holdout lectures live in a separate directory and must not be processed, judged or
listened to before stage 4. Opening one retires it to development.
- Not taken from the source folder: a deck without a recording (`090parsing.pdf`) and
recordings without a deck (`Комп4.m4a`, `Комп5.m4a`).
- This supersedes the 2026-07-29 consequences above: validation is no longer empty
(2 lectures) and the holdout is assembled (2 lectures); both are still below the
targets in "Target dataset size".

## Coverage matrix

Select lectures by matcher behaviour, not only by academic subject. The benchmark should
Expand All @@ -101,6 +141,21 @@ eventually contain at least 2–3 independent examples of each high-risk class:
A lecture may cover several classes. A large collection of nearly identical sequential
lectures does not substitute for this coverage.

### Coverage so far (2026-10-11)

Behaviour classes are assigned from judge labels, never by guessing. Holdout lectures
stay unlabelled until stage 4.

| Dimension | Covered by | Gap |
| --- | --- | --- |
| Lecturers | 3: the original single-lecturer series, databases, compilers | — |
| Recording | audio (all), video with a separate deck (`db-*`) | — |
| Deck format | PDF, PPTX (`db-*`) | — |
| Behaviour classes on new validation | not labelled yet | judge labels from the baseline runs (stage 1, ticket 04) |
| Mostly visual deck | none confirmed | still needed |
| Slide and speech languages differ | none | still needed |
| Irrelevant or wrong deck | none | still needed |

## Target dataset size

The first useful generalization benchmark should contain:
Expand Down
Loading
Loading