Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ build*/

# Data
/data/
/cache/
/results/
/input/
/output/

Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,8 @@ and it then falls back to its default. [engine/configs/geolife.json](engine/conf
| `where_at` | Probes per trajectory and max time gap (s) between points to probe |
| `where_at_scoring` | Tolerance range (m) for the primary average-hit score, and the hit radius (m) for the within-radius diagnostic |
| `frechet` | Fréchet variant for kNN, join and clustering: `Discrete` (kept vertices only, the default) or `Continuous` (polyline distance, unaffected by adding a vertex on an edge) |
| `clustering` | Route clustering metric: `enabled`, neighbour threshold `eps_m` (m) and `min_size` per cluster. Reuses the self join when `join.mode` is `Self` with the same eps |
| `join` | `mode`, threshold `eps_m` (m), `query_count` and `self_join`. `HeldOut` (the default) joins `query_count` original trajectories from the query split (`0` = all) against the simplified database. `Self` joins the database with itself, both sides simplified. `self_join: true` also scores the self join next to a held-out join, as a diagnostic outside the combined score. `score_eps_m` lists more thresholds: the combined score's join term then averages the normalised join at `eps_m` and at each of them |
| `clustering` | Route clustering metric: `enabled`, neighbour threshold `eps_m` (m) and `min_size` per cluster. Reuses the self join when one is scored with the same eps |
| `allocation_alphas` | Allocation exponents to test: budgets follow size^α (`1` = size-proportional, `0` = equal). Exports for α ≠ 1 get an `_a<α>` suffix |

## Tests
Expand Down
80 changes: 80 additions & 0 deletions TEACHER_NEXT_STEPS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Teacher: status and next steps

Updated 2026-11-17 on branch `teacher`. The teacher is described in
[engine/src/teacher/README.md](engine/src/teacher/README.md); this file says what is settled and
what is open.

## The teacher

`BUDGET_LOOP`: budgets searched on value curves, the best candidate recipe per trajectory, then the
selection loop, which re-picks each trajectory's recipe against its neighbours' current compressed
state until nothing switches. kNN is scored as a ranking among the compressed database
(`teacher.ranking_knn`, the default).

| Arm | Role |
|---|---|
| `EQUAL_CHOICE`, `EQUAL_CHOICE_RDP_PARTNERS`, `EQUAL_LOOP` | the same steps at equal budgets |
| `BUDGET_RDP` | searched budgets with RDP's points |
| `BUDGET_CHOICE` | searched budgets and recipe choice, no loop |
| `BUDGET_LOOP_REALLOCATE` | budgets re-searched after the loop, so they react to neighbours too |
| `BUDGET_LOOP_NO_UPDATE` | control: one look at the start, no reacting |
| `BUDGET_LOOP_SHUFFLED` | control: join partners and kNN neighbours relabelled |
| `RECIPE_<recipe>` (baselines in `teacher_eval`) | one recipe for every trajectory |

## Settled (Geolife, 0.25 % unless stated)

- **Budgets per trajectory:** `BUDGET_RDP` 0.553 J4 against `RDP_a0` 0.456, every seed.
- **Recipe per trajectory:** `EQUAL_CHOICE` beats the best single recipe (`RECIPE_GREEDY_RDP2`) by
0.014 to 0.035 J4 at 0.25 % to 1 %, every seed.
- **Relations, database self join:** trained on the self join, the loop lifts the 150 m self join
well above the start; the shuffled control falls far below it. Trained on several thresholds
(`teacher.extra_join_eps_m`) the gain holds at thresholds not trained on, at small budgets.
- **Relations, kNN on held-out queries:** the ranking loop adds 0.01 to 0.05 held-out kNN over the
same loop with the per-trajectory score, every seed and budget tested; shuffled neighbours turn
it into a loss.
- **Overall:** `BUDGET_LOOP` J4 0.600 against `RDP_a0` 0.458 at 0.25 %, 0.876 against 0.795 at 1 %
(seed 42).

## Open

1. **Held-out kNN and join at larger budgets.** Above 0.25 % the teacher trails `RDP_a0` on
held-out kNN, and at 1 % on the held-out join. Its kNN losses carry over to new queries while
most of its gains do not.
2. **Value curves still use the per-trajectory kNN check.** A ranking version against a fixed
state that only rescored each query's true neighbours made no difference; one that also rescores
intruders is far too slow as is.
3. **Term weights** (`teacher.term_weights`, `--term-weights`) trade range and where-at for kNN.
Seeds 17, 42 and 43 at 0.25 %: with kNN weighted 1×, 2× or 4× `BUDGET_LOOP` beats `RDP_a0` on
all four held-out terms on every seed; at 4× held-out kNN is 0.719 against 0.669, for 0.019
J4. At 0.5 % (seeds 17 and 43) 4× brings kNN level with `RDP_a0` (0.827 against 0.828) for
0.006 J4. At 1 % (seed 42) it barely moves kNN: there the gap comes from the candidates.
4. **Held-out join thresholds.** `teacher_eval` now scores both joins at every `--eval-eps`
threshold. Trained at 150 m only, the held-out join lead at 0.25 % (seeds 42 and 17) peaks at
150 m (+0.10 over the best baseline) and falls to −0.01 at 100 m. Trained on 100, 125, 150, 200
and 300 m (`teacher.extra_join_eps_m`) it is +0.07 to +0.10 at every threshold from 100 to
300 m, including 175 and 250 m, which neither trained on; J4 −0.006. `engine/configs/geolife.json` now trains on these thresholds, and
`teacher_eval` scores both joins at 100 to 300 m by default. At 1 % it narrows the gap
to the baselines from −0.16 to −0.08 at 100 m and from −0.03 to −0.01 at 300 m, J4 +0.002.
5. **Budget moves between trajectories scored exactly** with `KnnState` and `JoinState`, beyond
what `BUDGET_LOOP_REALLOCATE` does.
6. **For a paper:** published baselines, a second dataset, the 2 % budget, more seeds.

## Evaluation

With `engine/configs/geolife.json`, J4's join term is the held-out join averaged over 100, 150, 200
and 300 m (`join.score_eps_m`), each normalised against its own floor, so a teacher cannot win it
by fitting 150 m. `results.csv` keeps the 150 m value alone as `norm_join_main`, and
`thresholds.csv` has both joins at every `--eval-eps` threshold, including 175 and 250 m, which
nothing trains on. Numbers above from before 2026-11-17 use the 150 m join alone.

## Run

```bash
cmake --build build-release && ctest --test-dir build-release
./build-release/engine/teacher_eval --config engine/configs/geolife.json \
--seeds 42 --shares 0.0025 --out results/<new_dir>
python3 engine/apps/teacher_stats.py results/<new_dir>
```

Use a new `--out` directory each time: the CSVs are appended to. Runs may rewrite the cache files
in `cache/`, which are tracked although `.gitignore` lists them.
Binary file added cache/teacher_cache_seed17.bin
Binary file not shown.
Binary file added cache/teacher_cache_seed42.bin
Binary file not shown.
15 changes: 15 additions & 0 deletions engine/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ add_library(engine
src/io/workload_export.cpp
src/pipeline/config.cpp
src/pipeline/pipeline.cpp
src/pipeline/scorer.cpp
src/pipeline/test_workloads.cpp
src/selection/rdp.cpp
src/selection/sed.cpp
src/selection/uniform.cpp
Expand All @@ -40,6 +42,11 @@ add_library(engine
src/evaluator/combined.cpp
src/query/route_clustering.cpp
src/evaluator/route_clustering.cpp
src/teacher/workloads.cpp
src/teacher/objective.cpp
src/teacher/curves.cpp
src/teacher/greedy.cpp
src/teacher/teacher.cpp
)

target_include_directories(engine PUBLIC include)
Expand All @@ -52,6 +59,12 @@ add_executable(run

target_link_libraries(run PRIVATE engine)

add_executable(teacher_eval
apps/teacher_eval.cpp
)

target_link_libraries(teacher_eval PRIVATE engine)

add_executable(engine_tests
tests/parquet_loader_test.cpp
tests/projection_test.cpp
Expand All @@ -74,6 +87,8 @@ add_executable(engine_tests
tests/combined_test.cpp
tests/route_clustering_test.cpp
tests/continuous_frechet_test.cpp
tests/teacher/curves_test.cpp
tests/teacher/teacher_test.cpp
)
target_link_libraries(engine_tests PRIVATE engine GTest::gtest_main Arrow::arrow_shared Parquet::parquet_shared)
gtest_discover_tests(engine_tests)
Loading
Loading