diff --git a/TEACHER_NEXT_STEPS.md b/TEACHER_NEXT_STEPS.md index 6bcba7d..876356c 100644 --- a/TEACHER_NEXT_STEPS.md +++ b/TEACHER_NEXT_STEPS.md @@ -1,71 +1,52 @@ # Teacher: status and next steps -Implementation update, 2026-11-20: the range objective and endpoint floor now use macro F1 like -J4, with exact incremental updates in the loop and F1 marginals for candidate and budget scoring. -`RDP_GLOBAL`, a database-wide top-down PED baseline, runs alongside the other baselines in -`teacher_eval` and is enabled in `geolife.json`. It is included in the best-cheap-baseline comparison. -Continuous kNN now shares a deterministic millimetre-grid ranking between the evaluator and -tracker; V6 caches rebuild old reference rankings. Limited watch lists remain approximate. -The numerical findings below predate these changes and need to be checked with fresh runs. - -Updated 2026-11-17 on branch `teacher`. The teacher is described in +Updated 2026-11-22. The teacher is described in [engine/src/teacher/README.md](engine/src/teacher/README.md); this file says what is settled and -what is open. +what is open. Findings from before 2026-11-22, including the removed arms and the self-trained +teacher, are in git history. ## The teacher -`BUDGET_LOOP`: budgets searched on value curves, the best candidate recipe per trajectory, then the -selection loop, which re-picks each trajectory's recipe against its neighbours' current compressed -state until nothing switches. kNN is scored as a ranking among the compressed database -(`teacher.ranking_knn`, the default). +Trained for held-out queries only: database kNN queries and join queries are originals against +the simplified database, as in J4. J_T is J4's four terms on the database's own workloads plus two +query-free terms on each trajectory's Fréchet deviation bound (shape 0.5, margin 1, in +`engine/configs/geolife.json`). | Arm | Role | |---|---| -| `EQUAL_CHOICE`, `EQUAL_CHOICE_RDP_PARTNERS`, `EQUAL_LOOP` | the same steps at equal budgets | -| `BUDGET_RDP` | searched budgets with RDP's points | -| `BUDGET_CHOICE` | searched budgets and recipe choice, no loop | -| `BUDGET_LOOP_REALLOCATE` | budgets re-searched after the loop, so they react to neighbours too | -| `BUDGET_LOOP_NO_UPDATE` | control: one look at the start, no reacting | -| `BUDGET_LOOP_SHUFFLED` | control: join partners and kNN neighbours relabelled | +| `BUDGET_RDP` | budgets from value curves, RDP's points | +| `BUDGET_CHOICE` | budgets and recipe choice per trajectory | +| `BUDGET_LOOP` | then best response on the tracked J_T | +| `BUDGET_LOOP_TRADE` | then points traded between trajectories: the teacher | | `RECIPE_` (baselines in `teacher_eval`) | one recipe for every trajectory | -## Settled (Geolife, 0.25 % unless stated) +## Settled (Geolife, held-out J4, seeds 42, 7, 99, 123) + +Research notes: `Teacher pseudo held out selection 2026-11-21`, `Teacher fresh seed confirmation +2026-11-22` in `~/code/qdts-research/research_notes/`. + +| Share | Arm | kNN | join | range | where-at | J4 | +|---|---|---|---|---|---|---| +| 0.25 % | RDP_a0 | 0.669 | 0.433 | 0.401 | 0.340 | 0.461 | +| 0.25 % | BUDGET_LOOP_TRADE | 0.734 | 0.614 | 0.656 | 0.445 | 0.612 | +| 1 % | RDP_a0 | 0.915 | 0.875 | 0.705 | 0.698 | 0.798 | +| 1 % | BUDGET_LOOP_TRADE | 0.911 | 0.882 | 0.943 | 0.836 | 0.893 | -- **Budgets per trajectory:** `BUDGET_RDP` 0.553 J4 against `RDP_a0` 0.456, every seed. -- **Recipe per trajectory:** `EQUAL_CHOICE` beats the best single recipe (`RECIPE_GREEDY_RDP2`) by - 0.014 to 0.035 J4 at 0.25 % to 1 %, every seed. -- **Relations, database self join:** trained on the self join, the loop lifts the 150 m self join - well above the start; the shuffled control falls far below it. Trained on several thresholds - (`teacher.extra_join_eps_m`) the gain holds at thresholds not trained on, at small budgets. -- **Relations, kNN on held-out queries:** the ranking loop adds 0.01 to 0.05 held-out kNN over the - same loop with the per-trajectory score, every seed and budget tested; shuffled neighbours turn - it into a loss. -- **Overall:** `BUDGET_LOOP` J4 0.600 against `RDP_a0` 0.458 at 0.25 %, 0.876 against 0.795 at 1 % - (seed 42). +- Shape and margin weights were chosen on pseudo-held-out splits of the database (the real query + pool never read), then confirmed once on four seeds. +- At 1 % the teacher trails RDP on held-out kNN by 0.001 to 0.006 per seed; the gap is near-tie + swaps (recall with 5 % distance tolerance 0.993 against 0.991). +- Most of the J4 gain comes from allocation, then recipe choice; the loop and trades add little + J4 but win back kNN and join. ## Open -1. **Held-out kNN and join at larger budgets.** Above 0.25 % the teacher trails `RDP_a0` on - held-out kNN, and at 1 % on the held-out join. Its kNN losses carry over to new queries while - most of its gains do not. -2. **Value curves still use the per-trajectory kNN check.** A ranking version against a fixed - state that only rescored each query's true neighbours made no difference; one that also rescores - intruders is far too slow as is. -3. **Term weights** (`teacher.term_weights`, `--term-weights`) trade range and where-at for kNN. - Seeds 17, 42 and 43 at 0.25 %: with kNN weighted 1×, 2× or 4× `BUDGET_LOOP` beats `RDP_a0` on - all four held-out terms on every seed; at 4× held-out kNN is 0.719 against 0.669, for 0.019 - J4. At 0.5 % (seeds 17 and 43) 4× brings kNN level with `RDP_a0` (0.827 against 0.828) for - 0.006 J4. At 1 % (seed 42) it barely moves kNN: there the gap comes from the candidates. -4. **Held-out join thresholds.** `teacher_eval` now scores both joins at every `--eval-eps` - threshold. Trained at 150 m only, the held-out join lead at 0.25 % (seeds 42 and 17) peaks at - 150 m (+0.10 over the best baseline) and falls to −0.01 at 100 m. Trained on 100, 125, 150, 200 - and 300 m (`teacher.extra_join_eps_m`) it is +0.07 to +0.10 at every threshold from 100 to - 300 m, including 175 and 250 m, which neither trained on; J4 −0.006. `engine/configs/geolife.json` now trains on these thresholds, and - `teacher_eval` scores both joins at 100 to 300 m by default. At 1 % it narrows the gap - to the baselines from −0.16 to −0.08 at 100 m and from −0.03 to −0.01 at 300 m, J4 +0.002. -5. **Budget moves between trajectories scored exactly** with `KnnState` and `JoinState`, beyond - what `BUDGET_LOOP_REALLOCATE` does. -6. **For a paper:** published baselines, a second dataset, the 2 % budget, more seeds. +1. **The student:** learn the frozen teacher's kept sets or budgets. +2. **Value curves still use the per-trajectory kNN check** and score the join on i's own slot, not + i's effect on its partners' F1. +3. **For a paper:** published baselines, a second dataset, the 2 % budget, more seeds. The loop + controls (shuffled partners, no update) were removed; restore them from git history if a claim + about the loop needs them. ## Evaluation diff --git a/engine/apps/teacher_eval.cpp b/engine/apps/teacher_eval.cpp index 209ffaf..6efe830 100644 --- a/engine/apps/teacher_eval.cpp +++ b/engine/apps/teacher_eval.cpp @@ -32,9 +32,12 @@ #include #include #include +#include #include #include #include +#include +#include #include namespace { @@ -55,14 +58,13 @@ struct Args { std::optional> shares; // config: budget_shares std::vector arms{teacher::kArms.begin(), teacher::kArms.end()}; bool baselines = true; - std::optional teacher_join; // config: teacher.join, else join.mode std::optional> train_eps; // config: teacher.extra_join_eps_m - std::optional held_out_term; // config: teacher.held_out_term - std::optional ranking_knn; // config: teacher.ranking_knn - std::optional compressed_knn_query; // config: teacher.knn_query std::optional> term_weights; // config: teacher.term_weights std::vector eval_eps{kEvalEps.begin(), kEvalEps.end()}; bool knn_check = false; + bool diagnose = false; + std::size_t knn_outside = 0; // ranked outsiders per query for the margin term + std::optional pseudo_split; // re-split the database: tuning without the pool }; template std::vector parse_list(const std::string &text) { @@ -106,33 +108,24 @@ Args parse_args(int argc, char **argv) { args.eval_eps = parse_list(value()); else if (key == "--term-weights") { args.term_weights = parse_list(value()); - if (args.term_weights->size() != teacher::kTermCount) - throw std::invalid_argument("--term-weights takes kNN,range,where-at,join"); - } else if (key == "--independent-knn") - args.ranking_knn = false; - else if (key == "--held-out-term") - args.held_out_term = true; - else if (key == "--knn-query") { - const auto query = value(); - if (query != "Original" && query != "Compressed") - throw std::invalid_argument("--knn-query must be Original or Compressed"); - args.compressed_knn_query = query == "Compressed"; + if (args.term_weights->size() < 4 || args.term_weights->size() > teacher::kTermCount) + throw std::invalid_argument( + "--term-weights takes kNN,range,where-at,join[,shape[,margin]]"); } else if (key == "--knn-check") args.knn_check = true; - else if (key == "--teacher-join") { - const auto mode = value(); - if (mode != "HeldOut" && mode != "Self") - throw std::invalid_argument("--teacher-join must be HeldOut or Self"); - args.teacher_join = mode == "Self" ? query::JoinMode::Self : query::JoinMode::HeldOut; - } else if (key == "--no-baselines") + else if (key == "--diagnose") + args.diagnose = true; + else if (key == "--knn-outside") + args.knn_outside = std::stoull(value()); + else if (key == "--pseudo-split") + args.pseudo_split = std::stoull(value()); + else if (key == "--no-baselines") args.baselines = false; else throw std::invalid_argument("unknown argument " + key + " (--config --data --out --cache --seeds --shares --arms " - "--teacher-join --train-eps --held-out-term " - "--independent-knn --knn-query --term-weights --eval-eps " - "--knn-check " - "--no-baselines)"); + "--train-eps --term-weights --eval-eps --knn-check " + "--diagnose --knn-outside --pseudo-split --no-baselines)"); } teacher::check_arms(args.arms); return args; @@ -146,18 +139,12 @@ pipeline::Config effective_config(const Args &args) { config.teacher.cache_dir = *args.cache; if (args.shares) config.budget_shares = *args.shares; - if (args.teacher_join) - config.teacher.join = *args.teacher_join; if (args.train_eps) config.teacher.extra_join_eps_m = *args.train_eps; - if (args.held_out_term) - config.teacher.held_out_term = *args.held_out_term; - if (args.ranking_knn) - config.teacher.ranking_knn = *args.ranking_knn; - if (args.compressed_knn_query) - config.teacher.compressed_knn_query = *args.compressed_knn_query; - if (args.term_weights) + if (args.term_weights) { + config.teacher.term_weights.fill(0.0); std::ranges::copy(*args.term_weights, config.teacher.term_weights.begin()); + } config.self_join = true; return config; } @@ -275,6 +262,149 @@ std::vector baseline_arms(const core::TrajectoryDb &db, return arms; } +// Per trajectory: a bound on its own Fréchet deviation, and the held-out kNN and join answers it +// lost (in the truth, not answered) or gained falsely. Join counts sum over every join workload +// Held-out kNN: how far each true neighbour's distance moved, as a share of its deviation bound, +// and recall when a false neighbour within (1 + eps) of the true k-th distance counts as a hit +void diagnose_knn(std::ofstream &out, std::uint64_t seed, const Arm &arm, + const core::TrajectoryDb &db, const core::TrajectoryDb &compressed, + const query::KnnWorkload &workload, std::span deviation, + std::span> found) { + constexpr std::array kEps{0.01, 0.05, 0.1}; + const std::size_t m = workload.entries.size(); + std::vector> ratios(m); + std::vector> tolerant(m); + std::vector recall(m, kNan); + core::parallel_for(m, [&](std::size_t q) { + const auto &truth = workload.entries[q].reference; + if (truth.empty()) + return; + const auto query = workload.queries.trajectory(workload.entries[q].query_index); + auto distance = [&](const core::TrajectoryView &t) { + return query::frechet_distance(query, t, workload.frechet); + }; + for (const auto i : truth) + if (deviation[i] > 0) + ratios[q].push_back( + (distance(compressed.trajectory(i)) - distance(db.trajectory(i))) / + deviation[i]); + const double kth = distance(db.trajectory(truth.back())); + std::size_t hits = 0; + std::array near{}; + for (const auto j : found[q]) { + if (std::ranges::find(truth, j) != truth.end()) { + ++hits; + continue; + } + const double d = distance(db.trajectory(j)); + for (std::size_t e = 0; e < kEps.size(); ++e) + near[e] += d <= (1.0 + kEps[e]) * kth; + } + const auto k = static_cast(truth.size()); + recall[q] = static_cast(hits) / k; + for (std::size_t e = 0; e < kEps.size(); ++e) + tolerant[q][e] = static_cast(hits + near[e]) / k; + }); + std::vector all; + for (const auto &r : ratios) + all.insert(all.end(), r.begin(), r.end()); + std::ranges::sort(all); + auto quantile = [&](double p) { + return all.empty() ? kNan : all[static_cast(p * (all.size() - 1))]; + }; + double mean = 0, positive = 0; + for (const double r : all) { + mean += r; + positive += r > 0; + } + if (!all.empty()) { + mean /= static_cast(all.size()); + positive /= static_cast(all.size()); + } + double sum = 0; + std::array sums{}; + std::size_t count = 0; + for (std::size_t q = 0; q < m; ++q) { + if (std::isnan(recall[q])) + continue; + ++count; + sum += recall[q]; + for (std::size_t e = 0; e < kEps.size(); ++e) + sums[e] += tolerant[q][e]; + } + const auto c = static_cast(std::max(count, 1)); + out << std::format("{},{},{},{},{:.4f},{:.4f},{:.4f},{:.4f},{:.4f},{:.4f},{:.4f},{:.4f},{:.4f}," + "{:.4f}\n", + seed, arm.share, arm.name, all.size(), quantile(0.1), quantile(0.5), + quantile(0.9), mean, positive, sum / c, sums[0] / c, sums[1] / c, + sums[2] / c, quantile(0.99)); + out.flush(); +} + +void diagnose(std::ofstream &out, std::ofstream &knn_out, std::uint64_t seed, const Arm &arm, + const core::TrajectoryDb &db, const core::TrajectoryDb &compressed, + const evaluator::Workloads &test) { + const std::size_t n = db.size(); + std::vector deviation(n); + core::parallel_for(n, [&](std::size_t i) { + deviation[i] = teacher::simplification_bound(db.trajectory(i), compressed.trajectory(i)); + }); + std::vector knn_truth(n), knn_lost(n), knn_false(n); + if (test.knn) { + std::unordered_map index; + for (std::size_t i = 0; i < n; ++i) + index[db.trajectory(i).trajectory_id] = i; + const auto answers = evaluator::evaluate_knn(compressed, *test.knn).answers; + std::vector> found(answers.size()); + for (std::size_t q = 0; q < answers.size(); ++q) { + const auto &truth = test.knn->entries[q].reference; + for (const auto id : answers[q].neighbors) + found[q].push_back(index.at(id)); + for (const auto i : truth) { + ++knn_truth[i]; + knn_lost[i] += std::ranges::find(found[q], i) == found[q].end(); + } + for (const auto i : found[q]) + knn_false[i] += std::ranges::find(truth, i) == truth.end(); + } + diagnose_knn(knn_out, seed, arm, db, compressed, *test.knn, deviation, found); + } + std::vector join_truth(n), join_lost(n), join_false(n); + std::vector joins; + if (test.join) + joins.push_back(test.join); + for (const auto &workload : test.join_grid) + if (&workload != test.join) + joins.push_back(&workload); + for (const auto *workload : joins) { + if (workload->mode != query::JoinMode::HeldOut) + continue; + std::vector> matched; + evaluator::evaluate_join(compressed, *workload, &matched); + std::ranges::sort(matched); + for (std::size_t e = 0; e < workload->entries.size(); ++e) { + const auto &entry = workload->entries[e]; + for (const auto i : entry.reference) { + ++join_truth[i]; + join_lost[i] += !std::ranges::binary_search( + matched, std::pair{entry.query_index, i}); + } + } + std::unordered_map entry_of; + for (std::size_t e = 0; e < workload->entries.size(); ++e) + entry_of[workload->entries[e].query_index] = e; + for (const auto &[q, i] : matched) + join_false[i] += + !std::ranges::binary_search(workload->entries[entry_of.at(q)].reference, i); + } + for (std::size_t i = 0; i < n; ++i) + out << std::format("{},{},{},{},{},{},{:.3f},{},{},{},{},{},{}\n", seed, arm.share, + arm.name, i, db.trajectory(i).size(), arm.kept[i].size(), deviation[i], + knn_truth[i], knn_lost[i], knn_false[i], join_truth[i], join_lost[i], + join_false[i]); + out.flush(); +} + void validate(const Arm &arm, const core::TrajectoryDb &db) { if (arm.kept.size() != db.size()) throw std::runtime_error(arm.name + ": wrong number of trajectories"); @@ -311,16 +441,11 @@ std::uint64_t kept_hash(const KeptSets &kept) { return hash; } -// Whether the teacher trained on the join at eps_m in mode: one of its thresholds in its own join -// mode, or its held-out term at join.eps_m -bool trained_on(const pipeline::Config &config, const teacher::ObjectiveConfig &objective, - double eps_m, query::JoinMode mode) { +// Whether the teacher trained on the join at eps_m in mode: one of its held-out thresholds +bool trained_on(const pipeline::Config &config, double eps_m, query::JoinMode mode) { const auto extra = pipeline::teacher_extra_join_eps_m(config); - const bool teacher_eps = - eps_m == config.join.eps_m || std::ranges::find(extra, eps_m) != extra.end(); - const bool held_out_term = - objective.held_out_term && mode == query::JoinMode::HeldOut && eps_m == config.join.eps_m; - return (teacher_eps && mode == objective.join_mode) || held_out_term; + return mode == query::JoinMode::HeldOut && + (eps_m == config.join.eps_m || std::ranges::find(extra, eps_m) != extra.end()); } std::ofstream open_csv(const std::filesystem::path &file, const std::string &header) { @@ -345,9 +470,8 @@ int main(int argc, char **argv) try { "seed,share,arm,budget,knn_recall,range_f1,where_at_avg_hit,join_f1,clustering," "where_at_within_100,join_micro_f1,knn_tau,norm_knn,norm_range,norm_where_at,norm_join," "norm_clustering,J4,J_T,predicted_clusters," - "eval_s,join_mode,self_join_f1,norm_self_join,teacher_join_mode,norm_join_main," - "norm_clustering_main,self_knn_recall,norm_self_knn,norm_self_join_mean,J4_self," - "teacher_knn_query"); + "eval_s,join_mode,self_join_f1,norm_self_join,norm_join_main," + "norm_clustering_main,self_knn_recall,norm_self_knn,norm_self_join_mean,J4_self,J_T4"); auto floors = open_csv(args.out / "floors.csv", "seed,knn_recall,range_f1,where_at_avg_hit,join_f1,clustering,self_join_f1," @@ -355,9 +479,10 @@ int main(int argc, char **argv) try { auto thresholds = open_csv(args.out / "thresholds.csv", "seed,share,arm,join,eps_m,join_f1,norm_join,recall,teacher_threshold"); - auto terms = open_csv(args.out / "terms.csv", - "seed,share,arm,teacher_knn,teacher_range,teacher_where_at,teacher_join," - "norm_knn,norm_range,norm_where_at,norm_join,norm_self_join"); + auto terms = open_csv( + args.out / "terms.csv", + "seed,share,arm,teacher_knn,teacher_range,teacher_where_at,teacher_join," + "norm_knn,norm_range,norm_where_at,norm_join,norm_self_join,teacher_shape,teacher_margin"); auto budgets = open_csv(args.out / "budgets.csv", "seed,share,arm,points,budget"); auto timings = open_csv(args.out / "timings.csv", "seed,step,seconds"); auto hashes = open_csv(args.out / "kept_hashes.csv", "seed,share,arm,hash"); @@ -366,19 +491,47 @@ int main(int argc, char **argv) try { knn_check = open_csv(args.out / "knn_check.csv", "seed,share,arm,proxy_recall,db_recall,test_recall,norm_proxy," "norm_db,norm_test"); + std::ofstream diagnostics; + if (args.diagnose) + diagnostics = open_csv(args.out / "trajectories.csv", + "seed,share,arm,trajectory,points,kept,deviation,knn_truth," + "knn_lost,knn_false,join_truth,join_lost,join_false"); + std::ofstream knn_diagnostics; + if (args.diagnose) + knn_diagnostics = + open_csv(args.out / "knn_diagnostics.csv", + "seed,share,arm,pairs,shift_p10,shift_p50,shift_p90,shift_mean,shift_positive," + "recall,recall_eps01,recall_eps05,recall_eps10,shift_p99"); for (const auto seed : args.seeds.value_or(std::vector{base_config.seed})) { auto config = base_config; config.seed = seed; - const auto parts = split::split(all, config.split, seed); - const auto db = core::subset(all, parts.database); - std::cerr << std::format("seed {}: database {} trajectories, {} points\n", seed, db.size(), - db.num_points()); + // Pseudo split: the database is split again and stands in for everything; the real query + // pool is never read. Its own cache, as the database differs + const auto outer = split::split(all, config.split, seed); + std::optional inner; + if (args.pseudo_split) + inner = core::subset(all, outer.database); + const auto &source = inner ? *inner : all; + const auto parts = + args.pseudo_split ? split::split(source, config.split, *args.pseudo_split) : outer; + if (args.pseudo_split) + config.teacher.cache_dir /= std::format("pseudo_{}", *args.pseudo_split); + const auto db = core::subset(source, parts.database); + std::cerr << std::format( + "seed {}: database {} trajectories, {} points{}\n", seed, db.size(), db.num_points(), + args.pseudo_split ? std::format(" (pseudo split {})", *args.pseudo_split) : ""); // Database only auto start = std::chrono::steady_clock::now(); auto workloads = teacher::load_or_build_workloads( db, pipeline::teacher_workload_config(config), seed, config.teacher.cache_dir); + const std::size_t outside = args.knn_outside > 0 ? args.knn_outside + : config.teacher.term_weights[teacher::Objective::Margin] > 0 + ? teacher::kMarginOutside + : 0; + if (outside > 0) + teacher::add_ranked_neighbours(db, workloads, outside, seed, config.teacher.cache_dir); const auto pairs = workloads.positives; const teacher::Teacher model(db, std::move(workloads), pipeline::teacher_objective_config(config), seed); @@ -408,7 +561,7 @@ int main(int argc, char **argv) try { // Held-out data from here on start = std::chrono::steady_clock::now(); - const auto pool = core::subset(all, parts.queries); + const auto pool = core::subset(source, parts.queries); const pipeline::TestWorkloads test_workloads( db, pool, config, teacher::self_knn_workload(model.objective().workloads(), config.knn.p)); @@ -460,9 +613,20 @@ int main(int argc, char **argv) try { const auto scores = evaluator::Evaluator(db, compressed, test, scoring).evaluate().scores; const auto normalized = evaluator::normalize_scores(scores, floor); + // The teacher's own estimate of each metric, next to the test's + const auto teacher_metrics = objective.metrics(arm.kept); + const auto &teacher_floors = objective.floors(); + auto norm = [&](teacher::Objective::Term t) { + return (teacher_metrics[t] - teacher_floors[t]) / (1.0 - teacher_floors[t]); + }; + // J_T without the query-free terms: the training counterpart of J4 + const double j_t4 = + (norm(teacher::Objective::Knn) + norm(teacher::Objective::Range) + + norm(teacher::Objective::WhereAt) + norm(teacher::Objective::Join)) / + 4.0; results << std::format( "{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{:.2f}," - "{},{},{},{},{},{},{},{},{},{},{}\n", + "{},{},{},{},{},{},{},{},{},{}\n", seed, arm.share, arm.name, allocation::total_budget(db, arm.share), scores.knn.recall_at_k, scores.range_f1, scores.where_at.average_hit, scores.join.f1, scores.clustering.score, scores.where_at.within_radius, @@ -471,16 +635,13 @@ int main(int argc, char **argv) try { normalized.combined, objective.total(arm.kept), scores.clustering.predicted_clusters, core::seconds_since(start), mode_name(config.join.mode), scores.self_join.f1, normalized.self_join, - mode_name(objective.config().join_mode), normalized.join_f1, - normalized.clustering_score, scores.self_knn.recall_at_k, normalized.self_knn, - normalized.self_join_mean, normalized.combined_self, - objective.config().compressed_knn_query ? "Compressed" : "Original"); + normalized.join_f1, normalized.clustering_score, scores.self_knn.recall_at_k, + normalized.self_knn, normalized.self_join_mean, normalized.combined_self, j_t4); for (std::size_t k = 0; k < other_joins.size(); ++k) { const auto &workload = other_joins[k]; const auto join_scores = evaluator::evaluate_join(compressed, workload); - const bool teacher_threshold = - trained_on(config, objective.config(), workload.eps_m, workload.mode); + const bool teacher_threshold = trained_on(config, workload.eps_m, workload.mode); thresholds << std::format("{},{},{},{},{},{},{},{},{}\n", seed, arm.share, arm.name, mode_name(workload.mode), workload.eps_m, join_scores.f1, (join_scores.f1 - other_floor[k]) / @@ -491,25 +652,18 @@ int main(int argc, char **argv) try { const auto &workload = other_clusters[k]; const double score = evaluator::evaluate_route_clustering(compressed, workload).score; - // Clustering is built on the self join at its threshold - const bool teacher_threshold = - trained_on(config, objective.config(), workload.eps_m, query::JoinMode::Self); - thresholds << std::format("{},{},{},Clustering,{},{},{},,{}\n", seed, arm.share, + // Built on the self join, which the teacher never trains on + thresholds << std::format("{},{},{},Clustering,{},{},{},,0\n", seed, arm.share, arm.name, workload.eps_m, score, - (score - cluster_floor[k]) / (1.0 - cluster_floor[k]), - teacher_threshold ? 1 : 0); + (score - cluster_floor[k]) / (1.0 - cluster_floor[k])); } - // The teacher's own estimate of each metric, next to the test's - const auto teacher_metrics = objective.metrics(arm.kept); - const auto &teacher_floors = objective.floors(); - auto norm = [&](teacher::Objective::Term t) { - return (teacher_metrics[t] - teacher_floors[t]) / (1.0 - teacher_floors[t]); - }; - terms << std::format("{},{},{},{},{},{},{},{},{},{},{},{}\n", seed, arm.share, arm.name, - norm(teacher::Objective::Knn), norm(teacher::Objective::Range), - norm(teacher::Objective::WhereAt), norm(teacher::Objective::Join), - normalized.knn, normalized.range, normalized.where_at, - normalized.join, normalized.self_join); + terms << std::format("{},{},{},{},{},{},{},{},{},{},{},{},{},{}\n", seed, arm.share, + arm.name, norm(teacher::Objective::Knn), + norm(teacher::Objective::Range), norm(teacher::Objective::WhereAt), + norm(teacher::Objective::Join), normalized.knn, normalized.range, + normalized.where_at, normalized.join, normalized.self_join, + teacher_metrics[teacher::Objective::Shape], + teacher_metrics[teacher::Objective::Margin]); if (args.knn_check) { const double db_knn = objective.knn_recall(compressed); knn_check << std::format("{},{},{},{},{},{},{},{},{}\n", seed, arm.share, arm.name, @@ -519,6 +673,9 @@ int main(int argc, char **argv) try { normalized.knn); knn_check.flush(); } + if (args.diagnose && (teacher::is_arm(arm.name) || arm.name == "RDP_a0" || + arm.name.starts_with("PARTNER"))) + diagnose(diagnostics, knn_diagnostics, seed, arm, db, compressed, test); if (teacher::is_arm(arm.name)) for (std::size_t i = 0; i < arm.kept.size(); ++i) budgets << std::format("{},{},{},{},{}\n", seed, arm.share, arm.name, diff --git a/engine/apps/teacher_stats.py b/engine/apps/teacher_stats.py index 6b3c535..bcdc5e5 100644 --- a/engine/apps/teacher_stats.py +++ b/engine/apps/teacher_stats.py @@ -1,7 +1,7 @@ """Summarises a teacher_eval run: python3 teacher_stats.py [summary.md] Per share: means and SDs over seeds, paired J4 differences with t intervals, the gate against -the best cheap baseline, the coordination checks, each step's gain in J_T (training) against J4 +the best cheap baseline, the loop and trade gains, each step's gain in J_T (training) against J4 and J4-self (test), and the self and held-out joins at join.eps_m and at every threshold in thresholds.csv (teacher_eval --eval-eps), with the thresholds the teacher trained on marked. @@ -21,14 +21,11 @@ "norm_self_join_mean", "norm_clustering"] # Each step of the teacher, from the arm before it to the arm after it STEPS = [("budget search", "RDP_a0", "BUDGET_RDP"), - ("recipe choice", "RDP_a0", "EQUAL_CHOICE"), ("recipe choice at searched budgets", "BUDGET_RDP", "BUDGET_CHOICE"), - ("loop, equal budgets", "EQUAL_CHOICE_RDP_PARTNERS", "EQUAL_LOOP"), ("loop", "BUDGET_CHOICE", "BUDGET_LOOP"), - ("looking once", "BUDGET_CHOICE", "BUDGET_LOOP_NO_UPDATE"), - ("reacting", "BUDGET_LOOP_NO_UPDATE", "BUDGET_LOOP")] + ("trades", "BUDGET_LOOP", "BUDGET_LOOP_TRADE")] CHEAP = ["RDP_a0", "RDP_a025", "RDP_GLOBAL", "SED_a0", "UNI_a0", "PARTNER_l0.5", "PARTNER_l1"] -EXTRA = ["J_T", "norm_self_join", "norm_join_main"] +EXTRA = ["J_T", "J_T4", "norm_self_join", "norm_join_main"] JOINS = [("Self", "Normalised self-join F1"), ("HeldOut", "Normalised held-out join F1")] @@ -122,25 +119,21 @@ def main(): if best and "BUDGET_CHOICE" in arms: gate = paired(arms["BUDGET_CHOICE"]["J4"], arms[best]["J4"]) out.append(f"| **BUDGET_CHOICE − {best} (gate: ≥ .03, CI above 0)** | {fmt(gate)} |") - if all(a in arms for a in ["RDP_a0", "EQUAL_CHOICE_RDP_PARTNERS", "BUDGET_RDP", "BUDGET_CHOICE"]): - j = {a: arms[a]["J4"] for a in ["RDP_a0", "EQUAL_CHOICE_RDP_PARTNERS", "BUDGET_RDP", "BUDGET_CHOICE"]} - seeds = set.intersection(*(set(v) for v in j.values())) - inter = {s: j["BUDGET_CHOICE"][s] - j["EQUAL_CHOICE_RDP_PARTNERS"][s] - j["BUDGET_RDP"][s] + j["RDP_a0"][s] for s in seeds} - out.append(f"| Interaction (BUDGET_CHOICE − EQUAL_CHOICE_RDP_PARTNERS − BUDGET_RDP + RDP_a0) | {fmt(paired(inter, {s: 0.0 for s in seeds}))} |") - for a, b in [("BUDGET_LOOP", "BUDGET_CHOICE"), ("BUDGET_LOOP_REALLOCATE", "BUDGET_LOOP"), - ("EQUAL_LOOP", "EQUAL_CHOICE_RDP_PARTNERS"), ("BUDGET_LOOP", "BUDGET_LOOP_SHUFFLED")]: + for a, b in [("BUDGET_LOOP", "BUDGET_CHOICE"), ("BUDGET_LOOP_TRADE", "BUDGET_LOOP")]: if a in arms and b in arms: out.append(f"| {a} − {b} | {fmt(paired(arms[a]['J4'], arms[b]['J4']))} |") out.append("") - steps = [st for st in STEPS if st[1] in arms and st[2] in arms and arms[st[1]]["J_T"]] + # J_T4 leaves out the query-free shape and margin terms; older runs have only J_T + train_key = "J_T4" if any(arms[a].get("J_T4") for a in arms) else "J_T" + steps = [st for st in STEPS if st[1] in arms and st[2] in arms and arms[st[1]][train_key]] if steps: - out.append("Gain of each step in training (J_T) against test (J4, J4-self), " + out.append(f"Gain of each step in training ({train_key}) against test (J4, J4-self), " "mean over seeds. Kept = test gain / training gain:\n") - out.append("| Step | ΔJ_T | ΔJ4 | kept | ΔJ4_self | kept |") + out.append(f"| Step | Δ{train_key} | ΔJ4 | kept | ΔJ4_self | kept |") out.append("|---|---:|---:|---:|---:|---:|") for name, before, after in steps: row = [f"{name} ({before} → {after})"] - train = paired(arms[after]["J_T"], arms[before]["J_T"]) + train = paired(arms[after][train_key], arms[before][train_key]) row.append(fmt(train)) for m in ["J4", "J4_self"]: test = paired(arms[after][m], arms[before][m]) diff --git a/engine/configs/geolife.json b/engine/configs/geolife.json index 0933424..d67d93f 100644 --- a/engine/configs/geolife.json +++ b/engine/configs/geolife.json @@ -49,8 +49,6 @@ "cache_dir": "cache", "arms": ["BUDGET_CHOICE", "BUDGET_LOOP"], "range_query_count": 128000, - "held_out_term": false, - "ranking_knn": true, - "term_weights": [1.0, 1.0, 1.0, 1.0] + "term_weights": [1.0, 1.0, 1.0, 1.0, 0.5, 1.0] } } diff --git a/engine/include/backend/engine/teacher/objective.hpp b/engine/include/backend/engine/teacher/objective.hpp index 9cdd710..f7b795d 100644 --- a/engine/include/backend/engine/teacher/objective.hpp +++ b/engine/include/backend/engine/teacher/objective.hpp @@ -2,9 +2,9 @@ #include "backend/engine/core/simplification.hpp" #include "backend/engine/core/trajectory_db.hpp" -#include "backend/engine/query/join.hpp" #include "backend/engine/teacher/workloads.hpp" +#include #include #include #include @@ -47,24 +47,22 @@ class PreparedTrajectories { // keeping a subsequence of its points. Infinity if kept is not such a subsequence double simplification_bound(const core::TrajectoryView &original, const core::TrajectoryView &kept); -inline constexpr std::size_t kTermCount = 4; // kNN, range, where-at, join +inline constexpr std::size_t kTermCount = 6; // kNN, range, where-at, join, shape, margin using TermWeights = std::array; struct ObjectiveConfig { double min_tolerance_m = 25.0; // where-at ramp, as in the evaluator double max_tolerance_m = 500.0; - // Self: both sides compressed. HeldOut: each trajectory also queries its partners as original - query::JoinMode join_mode = query::JoinMode::Self; - bool held_out_term = false; // with Self: add the held-out form at join_eps_m as a term - // kNN ranked on the compressed database; false: per-trajectory check. Curves always use the - // latter - bool ranking_knn = false; - // The kNN query compressed too, the in-database setting; needs ranking_knn. Curves then share - // each hit between the query and its neighbour, both against the fixed partners - bool compressed_knn_query = false; - TermWeights term_weights{1.0, 1.0, 1.0, 1.0}; + double min_shape_m = 25.0; // shape ramp on each trajectory's deviation bound + double max_shape_m = 500.0; + TermWeights term_weights{1.0, 1.0, 1.0, 1.0, 0.0, 0.0}; }; +// 1 up to min, 0 from max, linear between: the where-at and shape scores +inline double ramp(double value, double min, double max) { + return std::clamp((max - value) / (max - min), 0.0, 1.0); +} + // Throws std::invalid_argument unless finite, not negative and not all zero. Returns their sum double check_term_weights(const TermWeights &weights); @@ -75,37 +73,44 @@ struct JoinLink { std::uint8_t term = 0; // join term the pair belongs to }; -// J_T: weighted mean of normalised kNN, range, where-at and join on the teacher's workloads, -// (metric - floor) / (1 - floor) with floor = endpoints only, like J4. +// J_T: weighted mean of normalised terms on the teacher's workloads, (metric - floor) / (1 - floor) +// with floor = endpoints only, like J4. Queries are original, as in the held-out test. +// kNN recall@k of the database kNN queries ranked on the compressed database. Value curves +// use a per-trajectory check instead: original neighbour within the query's cutoff // range macro F1 over nonempty reference boxes, like J4 // where-at average-hit ramp over i's probes -// join macro F1 over candidate pairs per join term (threshold, held-out form), each -// normalised by its own floor, then averaged: J4's join grid mean -// Where-at and the per-trajectory kNN add up over trajectories. Range, join and ranking kNN +// join held-out macro F1 per threshold: each trajectory queries its partners as original +// against their compressed versions; each threshold normalised by its own floor, then +// averaged: J4's join grid mean +// shape query-free: mean over trajectories of a ramp on simplification_bound, a prior +// for queries the workloads do not contain. Off unless weighted +// margin query-free per hit: for each true neighbour of a database kNN query, the chance it +// stays inside the top k if its distance moves anywhere within its deviation bound, +// against its gap to the (k + 1)th neighbour, and for each outsider ranked past k +// the chance it stays out, against its gap to the k-th. Needs workloads.ranked +// (add_ranked_neighbours). Off unless weighted +// Where-at, shape, margin and the per-trajectory kNN add up over trajectories. Range, join and kNN // are tracked under one-at-a-time changes by RangeState, JoinState and KnnState class Objective { public: - enum Term : std::size_t { Knn, Range, WhereAt, Join, TermCount }; + enum Term : std::size_t { Knn, Range, WhereAt, Join, Shape, Margin, TermCount }; using Terms = std::array; // Keeps references to db and workloads Objective(const core::TrajectoryDb &db, const TeacherWorkloads &workloads, const ObjectiveConfig &config = {}); - // i's scaled separable terms, 0 with endpoints only + // i's scaled where-at, shape and margin, 0 with endpoints only double separable(std::size_t i, std::span kept) const; double separable(std::size_t i, const core::TrajectoryView &candidate) const; - // Approximate score of i alone: separable, marginal range F1, and i's join F1 against fixed - // partners (empty = original; the join ignores partners when held out). Exact above beat, - // otherwise some value below beat + // Approximate score of i alone: separable, per-trajectory kNN, marginal range F1 against fixed + // partners (empty = original), and i's held-out join F1. Exact above beat, otherwise some value + // below beat double independent_value(std::size_t i, const core::TrajectoryView &candidate, std::span partners, double beat = -std::numeric_limits::infinity(), std::span range_weights = {}) const; - // independent_value with every kNN check a hit - double independent_bound(std::size_t i, const core::TrajectoryView &candidate, - std::span partners) const; // A bound followed by scoring without recomputing. Borrows the candidate's coordinates struct IndependentScore { @@ -131,7 +136,7 @@ class Objective { // Unscaled metrics Terms metrics(const KeptSets &kept) const; const Terms &floors() const { return m_floor; } - // Recall@k of the database kNN queries ranked on compressed, as in metrics with ranking_knn + // Recall@k of the database kNN queries ranked on compressed, as in metrics double knn_recall(const core::TrajectoryDb &compressed) const; double scale(Term m) const { return m_scale[m]; } @@ -145,7 +150,6 @@ class Objective { // The join metric from each term's macro F1 double join_value(std::span f1s) const; std::size_t truth(std::size_t slot) const { return m_truth[slot]; } - bool held_out(std::size_t term) const { return m_term_held_out[term]; } struct RangeMembership { std::uint32_t box; @@ -175,12 +179,9 @@ class Objective { double cutoff; }; - Terms raw_separable(std::size_t i, std::span kept) const; - Terms raw_separable(std::size_t i, const core::TrajectoryView &candidate, - bool knn = true) const; + // Knn slot left 0 + Terms raw_separable(std::size_t i, const core::TrajectoryView &candidate) const; double independent_sum(const IndependentScore &score, double knn) const; - std::size_t knn_checks(std::size_t i) const; - double knn_check_weight() const; double join_metric(std::span state) const; std::vector join_f1s(std::span state) const; @@ -191,14 +192,17 @@ class Objective { std::vector> m_knn; double m_knn_weight = 0; // 1 / (k * valid queries) std::vector> m_range; + std::size_t m_scored_boxes = 0; // boxes with a nonempty reference std::vector> m_probes; // where-at query indices per trajectory double m_where_at_weight = 0; // 1 / trajectories with probes + double m_shape_weight = 0; // 1 / trajectories + std::vector> m_margins; // per trajectory, one per kNN membership + double m_margin_weight = 0; // 1 / memberships std::vector m_pair_i, m_pair_j; // pairs used, positives first std::vector m_pair_term; std::vector m_pair_upper; // held-out positives: bound on the originals' distance std::vector m_eps; // threshold of each join term - std::vector m_term_held_out; std::size_t m_positive_count = 0; std::vector> m_links; // per trajectory std::vector m_truth; // positives per slot @@ -241,7 +245,8 @@ class RangeState { double m_sum = 0.0; }; -// The exact join metric under one-at-a-time changes; changing i only rescores i's pairs +// The exact held-out join metric under one-at-a-time changes. A pair has one outcome per direction, +// each found by its original query; changing i only rescores the directions with i compressed class JoinState { public: JoinState(const Objective &objective, std::span state); @@ -253,13 +258,9 @@ class JoinState { }; // Valid until the next apply() - Proposal propose(std::size_t i, const core::TrajectoryView &candidate, - std::span state) const; - // Outcomes of i's links, in links(i) order. Held-out links ignore the state; with - // held_out_known their outcomes are taken from it instead of retested - std::vector outcomes(std::size_t i, const core::TrajectoryView &candidate, - std::span state, - std::span held_out_known = {}) const; + Proposal propose(std::size_t i, const core::TrajectoryView &candidate) const; + // Outcomes of i's links with i as candidate, in links(i) order. They do not depend on the state + std::vector outcomes(std::size_t i, const core::TrajectoryView &candidate) const; Proposal propose(std::size_t i, std::vector matched) const; void apply(const Proposal &proposal); @@ -268,16 +269,13 @@ class JoinState { private: bool counted(std::size_t t, std::size_t found) const; double f1(std::size_t t, std::size_t hits, std::size_t found) const; - std::size_t count(std::size_t t, const JoinLink &link) const { - return m_objective.slot(t, link.term); + // The query's counts: link.other queries i + std::size_t count(const JoinLink &link) const { + return m_objective.slot(link.other, link.term); } - // Where the outcome of i's link is stored. Held out: the direction with i compressed + // Where the outcome of i's link is stored: the direction with i compressed std::size_t slot(std::size_t i, const JoinLink &link) const { - return 2 * std::size_t{link.pair} + (m_objective.held_out(link.term) && i > link.other); - } - // Held out: only the query's counts change - std::size_t sides(const JoinLink &link) const { - return m_objective.held_out(link.term) ? 1 : 2; + return 2 * std::size_t{link.pair} + (i > link.other); } const Objective &m_objective; @@ -288,16 +286,14 @@ class JoinState { }; // The ranking kNN metric under one-at-a-time changes. Each query watches its true top k and its -// nearest others in state and watch_states; unwatched trajectories are assumed to stay outside +// nearest others in state; unwatched trajectories are assumed to stay outside class KnnState { public: - KnnState(const Objective &objective, const KeptSets &state, - std::span watch_states, std::size_t watch); + KnnState(const Objective &objective, const KeptSets &state, std::size_t watch); + // Whether i's choice can change the metric bool watched(std::size_t i) const { return !m_watchers[i].empty(); } - // Whether i's choice can change the metric: watched, or with compressed queries a query - bool active(std::size_t i) const; - // Original queries only. From every query watching i, in the order propose and apply expect + // From every query watching i, in the order propose and apply expect std::vector distances(std::size_t i, const core::TrajectoryView &candidate, bool parallel = false) const; // Change in J_T @@ -305,44 +301,26 @@ class KnnState { // The same change from i becoming candidate, computing a distance only near a query's k-th // place double propose_bounded(std::size_t i, const core::TrajectoryView &candidate) const; - // Original queries only. Adds trajectory i to the list of every query that one of versions[i] - // comes within the query's watch radius of, the distance of its watch-th nearest in state + // Adds trajectory i to the list of every query that one of versions[i] comes within the query's + // watch radius of, the distance of its watch-th nearest in state void watch(std::span> versions, std::span state); void apply(std::size_t i, std::span distances); - // Either query kind. With compressed queries a distance depends on both sides, so a move also - // rescores i's own list, against the current state - struct Move { - std::size_t trajectory = 0; - std::vector watched; // as distances() - std::vector own; // i's own list; empty with original queries - }; - Move move(std::size_t i, const core::TrajectoryView &candidate, - std::span state) const; - double propose(const Move &move) const; - void apply(const Move &move); - double metric() const; // recall@k over the valid queries private: struct Watcher { std::uint32_t query, position; // position in the query's list }; + // Change in J_T from a change in total hits + double value(long change) const; // True neighbours in e's top k, with the distance at position replaced. Reads e's ranking std::size_t hits(std::size_t e, std::size_t position, double distance) const; // Re-ranks e after its row changed: m_rank[e] and m_hits[e] void refresh(std::size_t e); bool ranks_before(std::size_t e, double distance, std::size_t position, std::size_t other) const; - // The same with e's whole row replaced - std::size_t hits(std::size_t e, std::span row) const; - // True neighbours in the top k of order (distance, position) - std::size_t ranked_hits(std::size_t e, - std::vector> &order) const; - // Distances from query, e in some state, to e's list in state - std::vector own_row(std::size_t e, const core::TrajectoryView &query, - std::span state) const; const Objective &m_objective; std::size_t m_valid = 0; diff --git a/engine/include/backend/engine/teacher/teacher.hpp b/engine/include/backend/engine/teacher/teacher.hpp index ce4c2ef..013bdc3 100644 --- a/engine/include/backend/engine/teacher/teacher.hpp +++ b/engine/include/backend/engine/teacher/teacher.hpp @@ -17,29 +17,12 @@ namespace teacher { -// "Loop" is best response against the neighbours' current state until nothing switches. -// EQUAL_CHOICE equal budgets, best candidate each, partners original -// EQUAL_CHOICE_RDP_PARTNERS the same, partners RDP -// EQUAL_LOOP the loop from EQUAL_CHOICE_RDP_PARTNERS -// BUDGET_RDP budgets from value curves, RDP -// BUDGET_CHOICE budgets from value curves, best candidate each -// BUDGET_LOOP the loop from BUDGET_CHOICE: the teacher -// BUDGET_LOOP_REALLOCATE then curves, budgets and loop again while J_T rises. With only -// held-out join terms the curves ignore partners, so this only reruns -// the loop in a new visit order -// BUDGET_LOOP_TRADE then moves points between trajectories while exact J_T rises -// BUDGET_LOOP_SHUFFLED control: loop on relabelled partners and neighbours -// BUDGET_LOOP_NO_UPDATE control: one pass against the starting state -inline constexpr std::array kArms{"EQUAL_CHOICE", - "EQUAL_CHOICE_RDP_PARTNERS", - "EQUAL_LOOP", - "BUDGET_RDP", - "BUDGET_CHOICE", - "BUDGET_LOOP", - "BUDGET_LOOP_REALLOCATE", - "BUDGET_LOOP_TRADE", - "BUDGET_LOOP_SHUFFLED", - "BUDGET_LOOP_NO_UPDATE"}; +// BUDGET_RDP budgets from value curves, RDP +// BUDGET_CHOICE budgets from value curves, best candidate each +// BUDGET_LOOP then best response against the others' current state until nothing switches +// BUDGET_LOOP_TRADE then moves points between trajectories while exact J_T rises: the teacher +inline constexpr std::array kArms{"BUDGET_RDP", "BUDGET_CHOICE", "BUDGET_LOOP", + "BUDGET_LOOP_TRADE"}; bool is_arm(std::string_view name); // Throws std::invalid_argument on an unknown or repeated arm @@ -50,16 +33,9 @@ struct TeacherConfig { std::filesystem::path cache_dir = "cache"; std::vector arms{"BUDGET_CHOICE", "BUDGET_LOOP"}; std::size_t range_query_count = kRangeQueryCount; - // Unset: join.mode - std::optional join; // Unset: join.score_eps_m, the thresholds J4 averages over std::optional> extra_join_eps_m; - bool held_out_term = false; - bool ranking_knn = true; - // ObjectiveConfig::compressed_knn_query. Unset: compressed with a Self join, the in-database - // setting - std::optional compressed_knn_query; - TermWeights term_weights{1.0, 1.0, 1.0, 1.0}; + TermWeights term_weights{1.0, 1.0, 1.0, 1.0, 0.0, 0.0}; }; struct Arm { @@ -68,7 +44,7 @@ struct Arm { double objective = 0.0; // J_T }; -// Sees only the database it is given and workloads built from it +// Sees only the database it is given and workloads built from it. Trained for held-out queries class Teacher { public: Teacher(const core::TrajectoryDb &db, TeacherWorkloads workloads, const ObjectiveConfig &config, diff --git a/engine/include/backend/engine/teacher/workloads.hpp b/engine/include/backend/engine/teacher/workloads.hpp index 2a87bdf..1f15af9 100644 --- a/engine/include/backend/engine/teacher/workloads.hpp +++ b/engine/include/backend/engine/teacher/workloads.hpp @@ -17,7 +17,6 @@ namespace teacher { inline constexpr std::uint64_t kRangeSeedOffset = 7001; inline constexpr std::uint64_t kWhereAtSeedOffset = 7002; inline constexpr std::uint64_t kOrderSeedOffset = 7004; -inline constexpr std::uint64_t kShuffleSeedOffset = 7006; inline constexpr std::size_t kRangeQueryCount = 128000; @@ -38,6 +37,11 @@ struct KnnEntry { double cutoff = 0.0; // Fréchet distance to the (k + 1)th neighbour }; +struct RankedNeighbour { + std::uint32_t index; + double distance; +}; + struct JoinPair { std::uint32_t i, j; // i < j }; @@ -56,6 +60,9 @@ struct TeacherWorkloads { std::vector positives; // self-join pairs std::vector negatives; // endpoint candidates that are not pairs std::vector extra_joins; + // Optional, per kNN query: its k + depth nearest others, closest first, the top k included + // (add_ranked_neighbours) + std::vector> ranked; query::RangeWorkload range; query::WhereAtWorkload where_at; }; @@ -66,6 +73,14 @@ TeacherWorkloads load_or_build_workloads(const core::TrajectoryDb &db, const Wor std::uint64_t seed, const std::filesystem::path &cache_dir); +inline constexpr std::size_t kMarginOutside = 5; // outsiders per query the margin term scores + +// Each kNN query's k + depth nearest others with their distances, from +// cache_dir/teacher_ranked_seed_d.bin when it matches, otherwise built and saved +void add_ranked_neighbours(const core::TrajectoryDb &db, TeacherWorkloads &workloads, + std::size_t depth, std::uint64_t seed, + const std::filesystem::path &cache_dir); + // The self-kNN test workload from the cached lists: every valid database query with its k nearest // others. They depend on the database alone, so they are what a fresh build would give query::KnnWorkload self_knn_workload(const TeacherWorkloads &workloads, double p); diff --git a/engine/src/pipeline/config.cpp b/engine/src/pipeline/config.cpp index 731bcdc..0f1b640 100644 --- a/engine/src/pipeline/config.cpp +++ b/engine/src/pipeline/config.cpp @@ -10,6 +10,7 @@ #include #include #include +#include namespace pipeline { @@ -115,28 +116,25 @@ void parse_clustering_config(Config &config, const json &clustering) { } void parse_teacher_config(Config &config, const json §ion) { - validate_keys(section, - {"enabled", "cache_dir", "arms", "range_query_count", "join", "extra_join_eps_m", - "held_out_term", "ranking_knn", "knn_query", "term_weights"}, - "teacher"); + validate_keys( + section, + {"enabled", "cache_dir", "arms", "range_query_count", "extra_join_eps_m", "term_weights"}, + "teacher"); auto &t = config.teacher; t.enabled = section.value("enabled", t.enabled); t.cache_dir = section.value("cache_dir", t.cache_dir.string()); t.arms = section.value("arms", t.arms); t.range_query_count = section.value("range_query_count", t.range_query_count); - if (section.contains("join")) - t.join = parse_enum(section["join"]); if (section.contains("extra_join_eps_m")) t.extra_join_eps_m = section["extra_join_eps_m"].get>(); - t.held_out_term = section.value("held_out_term", t.held_out_term); - t.ranking_knn = section.value("ranking_knn", t.ranking_knn); - if (section.contains("knn_query")) { - const auto query = section["knn_query"].get(); - if (query != "Original" && query != "Compressed") - throw std::invalid_argument("config: teacher.knn_query must be Original or Compressed"); - t.compressed_knn_query = query == "Compressed"; + if (section.contains("term_weights")) { + // kNN, range, where-at, join, then optionally shape and margin (default 0) + const auto weights = section["term_weights"].get>(); + if (weights.size() < 4 || weights.size() > teacher::kTermCount) + throw std::invalid_argument("config: teacher.term_weights takes 4 to 6 weights"); + t.term_weights.fill(0.0); + std::ranges::copy(weights, t.term_weights.begin()); } - t.term_weights = section.value("term_weights", t.term_weights); teacher::check_term_weights(t.term_weights); if (t.extra_join_eps_m) query::check_extra_thresholds(*t.extra_join_eps_m, config.join.eps_m, @@ -232,15 +230,8 @@ teacher::WorkloadConfig teacher_workload_config(const Config &config) { } teacher::ObjectiveConfig teacher_objective_config(const Config &config) { - // Trains on the kind of join it is scored on, unless told otherwise - const auto join_mode = config.teacher.join.value_or(config.join.mode); return {.min_tolerance_m = config.where_at_scoring.min_tolerance_m, .max_tolerance_m = config.where_at_scoring.max_tolerance_m, - .join_mode = join_mode, - .held_out_term = config.teacher.held_out_term, - .ranking_knn = config.teacher.ranking_knn, - .compressed_knn_query = config.teacher.compressed_knn_query.value_or( - join_mode == query::JoinMode::Self && config.teacher.ranking_knn), .term_weights = config.teacher.term_weights}; } diff --git a/engine/src/pipeline/pipeline.cpp b/engine/src/pipeline/pipeline.cpp index 8750f3a..a8ce415 100644 --- a/engine/src/pipeline/pipeline.cpp +++ b/engine/src/pipeline/pipeline.cpp @@ -51,11 +51,13 @@ std::unique_ptr make_teacher(const core::TrajectoryDb &databas const Config &config) { if (!config.teacher.enabled) return nullptr; - return std::make_unique( - database, - teacher::load_or_build_workloads(database, teacher_workload_config(config), config.seed, - config.teacher.cache_dir), - teacher_objective_config(config), config.seed); + auto workloads = teacher::load_or_build_workloads(database, teacher_workload_config(config), + config.seed, config.teacher.cache_dir); + if (config.teacher.term_weights[teacher::Objective::Margin] > 0) + teacher::add_ranked_neighbours(database, workloads, teacher::kMarginOutside, config.seed, + config.teacher.cache_dir); + return std::make_unique(database, std::move(workloads), + teacher_objective_config(config), config.seed); } } // namespace diff --git a/engine/src/teacher/README.md b/engine/src/teacher/README.md index 4898ea0..f93b8f7 100644 --- a/engine/src/teacher/README.md +++ b/engine/src/teacher/README.md @@ -5,8 +5,14 @@ and which candidate keeps them: RDP, SED, UNIFORM or one of three greedy variant meant to be a target that a learned model can later imitate. It sees **only the database** (the 85% split) and workloads built from it with their own seeds. It -never sees the held-out query pool that the pipeline scores on. The fixed-budget arms use equal -shares (α = 0); the allocation arms redistribute the same total point budget. +never sees the held-out query pool that the pipeline scores on, and it trains for that setting: +its kNN and join queries are original trajectories against the simplified database. Budgets start +from equal shares (α = 0) and are redistributed within the same total. + +Self-trained options (compressed kNN queries, a `Self` join term, the held-out term next to it, the +per-trajectory kNN term) and the arms `EQUAL_CHOICE`, `EQUAL_CHOICE_RDP_PARTNERS`, `EQUAL_LOOP`, +`BUDGET_LOOP_REALLOCATE`, `BUDGET_LOOP_SHUFFLED` and `BUDGET_LOOP_NO_UPDATE` were removed on +2026-11-22; they are in git history before that date. Range scoring changed on 2026-11-20: J_T now uses macro F1, including its endpoint floor, matching J4. Historical teacher selections and J_T values need a fresh run. `RDP_GLOBAL` is the @@ -24,19 +30,17 @@ distances and join decisions are unchanged. Limited kNN watch lists remain an ap ``` pipeline::run (pipeline.cpp) ├─ load_or_build_workloads(database, ...) workloads.cpp: kNN, pairs, boxes, probes; cache + ├─ add_ranked_neighbours workloads.cpp: k + 5 nearest per query, for margin ├─ Teacher(database, workloads, ...) teacher.cpp: builds the Objective once │ └─ Objective(db, workloads) objective.cpp: inverts workloads, computes floors └─ per budget share: Teacher::run(budget, arms) - ├─ make_candidates six candidate recipes at equal budgets - ├─ EQUAL_CHOICE: independent_choice best candidate, partners original - ├─ EQUAL_CHOICE_RDP_PARTNERS / EQUAL_LOOP equal budgets: partners guessed as RDP / the loop - ├─ build_curves → allocate partners fixed at equal-budget RDP - │ ├─ BUDGET_RDP searched budgets, RDP's points - │ ├─ BUDGET_CHOICE → BUDGET_LOOP best candidates, then the loop: the teacher - │ ├─ BUDGET_LOOP_REALLOCATE rebuild curves, re-search budgets, loop again - │ ├─ BUDGET_LOOP_NO_UPDATE control: one look at the start, no reacting - │ └─ BUDGET_LOOP_SHUFFLED control: partners and neighbours relabelled + ├─ make_candidates RDP at equal budgets: the curves' partners + ├─ build_curves → allocate value curves, concave-envelope allocation + │ ├─ BUDGET_RDP allocated budgets, RDP's points + │ └─ BUDGET_CHOICE allocated budgets, best candidate each + │ └─ BUDGET_LOOP best response on the tracked J_T + │ └─ BUDGET_LOOP_TRADE points traded between trajectories: the teacher → each arm evaluated and exported like RDP and SED ``` @@ -76,41 +80,29 @@ Files: on the database alone. Without a teacher, `run` reports it as NaN. - **J5:** the earlier headline, J4 plus clustering. Results quoted below from earlier runs use it; `teacher_eval` no longer writes it. -- **J_T:** the teacher's own score on its training workloads. It has the same shape as J4. kNN - uses a training proxy during value-curve construction; range uses macro F1 like J4. -- **J_T's kNN term:** with `teacher.ranking_knn` (the default) it is recall@k of the teacher's - database queries answered on the compressed database (`knn_recall`), so competitors are - compressed too, like the test. The loop tracks it with `KnnState`. Value curves still use the per-trajectory check - against original competitors, which `teacher_eval --independent-knn` restores everywhere as the - ablation. The per-trajectory check misleads the teacher: it credited the teacher with beating - RDP on kNN where held-out queries showed it did not. - With a `Self` join the query is compressed too (`compressed_knn_query`, `teacher.knn_query` or - `teacher_eval --knn-query Original|Compressed` to override), as J4-self scores it. A - trajectory's choice then changes its own answer as well as the lists it is in, so `KnnState` - tracks both against the current state (`move`, `propose`, `apply`), and the curves share each - (query, neighbour) hit between the two sides, half each, against the fixed partners. -- **J_T's join term:** follows `join.mode`, so the teacher trains on the kind of join it is - scored on. `Self`: both sides of a pair compressed, the self join. `HeldOut`: every database - trajectory also acts as an original query against its compressed partners, so each pair is - tested once in each direction and a trajectory's own choice only affects its partners' F1. - This is the database-only stand-in for the held-out join, as the kNN proxy is for held-out - kNN; the teacher still never sees the query pool. `teacher.join` (or `teacher_eval - --teacher-join Self|HeldOut`) overrides it, e.g. to train on the self join for the self-join - experiment. With held-out join terms the join scoring is independent of compressed partners, - but range F1 still couples trajectories that answer the same box. Everything below describing the join term as "both - sides compressed" is the `Self` mode. -- **Join terms:** the join term can pool several terms. By default the teacher trains on - `join.eps_m` and `join.score_eps_m`, the thresholds J4 averages over, so the join gain is not - fitted to one threshold. `teacher.extra_join_eps_m` (or `teacher_eval --train-eps`) overrides - the extra thresholds. `teacher.held_out_term` (or `--held-out-term`) adds the held-out - form of the join at `join.eps_m` next to a `Self` join term, so one teacher keeps both the self - join and the join from outside queries. Like J4's join grid, each term's macro F1 is normalised - by its own endpoints-only floor and the terms are averaged equally. A term without true pairs is - left out, as J4 skips NaN entries. +- **J_T:** the teacher's own score on its training workloads: J4's four terms on the database's + own queries, plus the query-free shape and margin terms, weighted by `term_weights`. `teacher_eval` + also writes `J_T4`, the equal mean of the four J4-like terms, so training gains can be compared + with J4 without the priors. +- **J_T's kNN term:** recall@k of the teacher's database queries, as originals, answered on the + compressed database (`knn_recall`), so competitors are compressed too, like the test. The loop + and trades track it with `KnnState`. Value curves use a per-trajectory check instead (is the + simplified neighbour still within the query's original cutoff?), since they score trajectories + one at a time. +- **J_T's join term:** held out. Every database trajectory acts as an original query against its + compressed partners, so each pair is tested once in each direction and a trajectory's own choice + only affects its partners' F1. This is the database-only stand-in for the held-out join; the + teacher still never sees the query pool. A trajectory's join outcomes do not depend on the + others' choices, so only range F1 and ranking kNN couple trajectories. +- **Join terms:** by default the teacher trains on `join.eps_m` and `join.score_eps_m`, the + thresholds J4 averages over, so the join gain is not fitted to one threshold. + `teacher.extra_join_eps_m` (or `teacher_eval --train-eps`) overrides the extra thresholds. Like + J4's join grid, each term's macro F1 is normalised by its own endpoints-only floor and the terms + are averaged equally. A term without true pairs is left out, as J4 skips NaN entries. - **Join pairs:** the teacher uses every pair at each of its thresholds, because whoever compresses the database knows all of it. Whether the join gain generalises to thresholds the teacher never saw is tested in `teacher_eval` at 125, 175 and 250 m, between the trained ones. -- **Arm:** one named output of the teacher, e.g. `EQUAL_CHOICE_RDP_PARTNERS`. +- **Arm:** one named output of the teacher, e.g. `BUDGET_LOOP_TRADE`. --- @@ -122,8 +114,7 @@ Files: SED or from RDP. A variant seeded with a quarter from SED was removed: at small budgets it kept the same points as unseeded greedy. Seeded candidates combine classic and greedy point selection. Different recipes can produce identical kept sets. -- The fixed-budget arms use equal shares (α = 0). Allocation arms search per-trajectory budgets - using value curves, while keeping the total fixed. +- Per-trajectory budgets come from value curves, with the total fixed. - For example, the three classic candidates for a trajectory with budget 4 might be: ``` @@ -137,49 +128,25 @@ Files: ### Why a trajectory's choice depends on other trajectories Range macro F1 depends on how many trajectories still answer each box, so the marginal reward for recovering one trajectory depends on the others. Ranking kNN also depends on competing -trajectories. In a self join, both sides of each pair are simplified. These dependencies make the -best method for i depend on the current choices elsewhere in the database. - -### Two stages -The fixed-budget arms illustrate two stages, which differ in how the join partners are seen: - -``` -Stage 1, EQUAL_CHOICE_RDP_PARTNERS: one round, every trajectory on its own, partners ASSUMED to be RDP -Stage 2, EQUAL_LOOP: starts from stage 1, partners are their ACTUAL current choices - pass 1, pass 2, ... until a pass makes no switches -``` - -| | Stage 1 (EQUAL_CHOICE_RDP_PARTNERS) | Stage 2 (BR) | -|---|---|---| -| Points per trajectory | fixed budget | fixed budget | -| Methods tried for i | all 6 recipes | other distinct kept sets | -| Terms scored | all 4 (kNN, range, where-at, join) | all 4, as the change from i's current method | -| i's join partners seen as | RDP (a guess) | their actual current methods | -| Join change counted for | i only | i and every partner whose pair flips | -| Order | all trajectories at once, independently | one at a time; each switch is visible to the next | -| Runs | once | until no switches, at most 10 passes | - -Stage 1 is not needed for correctness: BR could start from all-RDP and still climb. It gives BR a -good starting point, and it is a result of its own (the independent, one-pass teacher), which -shows how much stage 2 adds. - -### Stage 1: EQUAL_CHOICE_RDP_PARTNERS (`independent_choice`, or `curve_methods`) -For each trajectory i, independently, score its candidates at its equal budget with -`independent_value`, with join partners fixed at equal-budget RDP. Choose the highest score; -ties go to the earlier candidate. If allocation arms also need curves, read the same choice from -the curve at that budget instead. - -The independent scores use a fixed partner reference. Their best choices need not maximise the -actual J_T once both sides of every join pair use their chosen candidates. - -### Allocation arms -`build_curves` scores budgets from 2 up to a per-trajectory cap. `allocate` fills the total budget -with greedy steps along each curve's upper concave envelope. `BUDGET_RDP` uses RDP-only curves; `BUDGET_CHOICE` uses the best of all candidates. -`BUDGET_LOOP` applies best response at those allocated budgets. `BUDGET_LOOP_SHUFFLED` is a control that -searches with relabelled partners and kNN neighbours, then scores the result against the real -workloads. `BUDGET_LOOP_NO_UPDATE` is the second control: one pass from the same start in which -every trajectory picks its best candidate against the others' starting state and never sees their -switches. The gap between it and `BUDGET_LOOP` is what reacting adds to looking once. +trajectories: a neighbour drops out of a query's top k when a competitor moves closer. The +held-out join does not couple trajectories. These dependencies make the best method for i depend +on the current choices elsewhere in the database. + +### The steps +1. **Value curves** (`build_curves`): for each trajectory and each budget up to its cap, score + every candidate with `independent_value` against fixed partners (RDP at equal budgets) and keep + the best. `BUDGET_RDP` uses RDP-only curves. +2. **Allocation** (`allocate`): fill the total budget with greedy steps along each curve's upper + concave envelope. +3. **Recipe choice** (`BUDGET_CHOICE`): each trajectory takes its curve's best candidate at its + allocated budget. +4. **The loop** (`BUDGET_LOOP`, class `Search`): best response on the tracked J_T, against the + others' current choices. +5. **Trades** (`BUDGET_LOOP_TRADE`, `trade_budgets`): move 1, 2 or 4 points from one trajectory to + another while the exact J_T rises. + +On held-out runs most of the J4 gain comes from allocation, then recipe choice; the loop and +trades add little J4 but win back kNN and join (research notes, 2026-11-22). `Teacher::single_recipes` gives the matching baselines: each candidate recipe for every trajectory at equal budgets (`RECIPE_RDP`, …, `RECIPE_GREEDY_RDP2`). `teacher_eval` writes them with the @@ -187,88 +154,47 @@ other baselines. `RECIPE_RDP`, `RECIPE_SED` and `RECIPE_UNIFORM` equal `RDP_a0`, `UNI_a0`; the greedy recipes are the query-aware single recipes that choosing per trajectory must beat. -`BUDGET_LOOP_REALLOCATE` starts from `BUDGET_LOOP`. Each round rebuilds curves against the current partner -state, reallocates and runs best response. A round is accepted only if the exact J_T improves by -more than 1e-12. Rounds continue until J_T stops rising, at most `kMaxAllocationRounds` (10). -Even with only held-out join terms, rebuilding curves can change budgets: marginal range F1 -reacts to the current coverage of each box. - -### Stage 2: BR (`best_response`) +### The loop **In one sentence:** go through the trajectories one at a time, and let each switch to whichever -of its other methods makes the *real* J_T highest, given what everyone else currently uses; +of its other methods makes the tracked J_T highest, given what everyone else currently uses; repeat until nobody wants to switch. -1. **Who is visited:** trajectories with a range membership, a join link or an active kNN query. -2. **Starting state:** stage 1's method for every trajectory, plus a `JoinState` holding every - pair's outcome, a `RangeState` holding box coverage counts, and the ranking kNN tracker. +1. **Who is visited:** every trajectory with a range membership, a join link, a watching kNN + query, or any trajectory when shape or margin is weighted. +2. **Setup:** every distinct candidate is prepared once, with its separable value, kNN distances + to the queries watching it, box coverage and join outcomes; none depend on the others. A + `JoinState`, a `RangeState` and a `KnnState` hold the current state. 3. **One pass:** visit those trajectories in random order. For trajectory i and each of its other distinct candidate kept sets c: ``` - gain(c) = [separable(i, c) − separable(i, current)] where-at and independent kNN, if enabled - + range.propose(i, c).gain exact change in range macro F1 - + join.propose(i, c, state).gain exact change in the join F1 - + knn.propose(...) ranking kNN change, if enabled + gain(c) = [separable(i, c) − separable(i, current)] where-at, shape, margin + + join.propose(i, outcomes).gain exact change in the join F1 + + range.propose(i, coverage).gain exact change in range macro F1 + + knn.propose(i, distances) ranking kNN change ``` - `join.propose` rescores **all** of i's pairs against the partners' current kept points, and - updates the F1 of i and of every partner whose pair flips. - - **It switches when the best gain is positive** (above 1e-12). The terms can pull in - opposite directions: a switch that costs i a little where-at can still win if it fixes a join - pair. After a switch, the trackers store the new outcomes, and later visits see i's new - method. + **It switches when the best gain is positive** (above 1e-12), then applies the move to all + trackers, so later visits see i's new method. 4. **Repeat passes** until a pass has no switches (at most 10). -**Why it repeats.** A switch changes every pair with that trajectory in it, so its partners may -now prefer a different method too. If a partner comes later in the same pass, it is checked -against the new choice right away; otherwise the next pass picks it up. One switch can ripple -through connected trajectories. - -### Example -A reduced example using only the three classic candidates: trajectories 1–4; (1, 4) is a true -join pair, and 1 is also paired with 2. - -``` -stage 1 (EQUAL_CHOICE_RDP_PARTNERS): 1=RDP 2=RDP 3=SED 4=SED - 1 chose RDP assuming its partner 4 would be RDP, but 4 actually chose SED, - so the pair (1, 4) doesn't match. - -stage 2, pass 1 (random order): - visit 3: try RDP, UNIFORM → no gain 1=RDP 2=RDP 3=SED 4=SED - visit 1: try SED, UNIFORM, against 4 as SED: - SED makes (1, 4) match; it also rescores (1, 2); - total gain > 0 → switch 1=SED 2=RDP 3=SED 4=SED - visit 4: try RDP, UNIFORM → would break (1, 4): stays - visit 2: scored against 1 as SED (its new method): - SED makes (1, 2) match again → switch 1=SED 2=SED 3=SED 4=SED -pass 1 had switches → pass 2 -pass 2: nobody gains → stop -``` - -1's switch only became worthwhile once the teacher used 4's actual method, and 2's switch only -became worthwhile after 1 switched. Neither could be found by scoring each trajectory on its own. +The kNN tracker only watches each query's top k plus its nearest others in the start state and +the candidates, so a trajectory outside those lists can enter a top k unseen. `checked_loop` +therefore compares the result's exact J_T with the start's and keeps the start if it lost. ### Why it always stops, and where -Every switch raises J_T by a strictly positive amount, and there are finitely many choices, so it -must stop. Requiring a strict gain also means no trajectory flips back and forth on a tie. In -practice there is also a limit of 10 passes. +Every switch raises the tracked J_T by a strictly positive amount, and there are finitely many +choices, so it must stop. Requiring a strict gain also means no trajectory flips back and forth on +a tie. In practice there is also a limit of 10 passes. If a pass has no switches, **no single trajectory can raise J_T by switching on its own** within -the current candidate sets and improvement tolerance. Reaching the pass limit does not guarantee -this. Such a local optimum need not be global: switching two trajectories together might still -help. The random visiting order can change which local optimum it lands in. In +the current candidate sets and improvement tolerance. Such a local optimum need not be global. In optimisation terms this is coordinate ascent; in game-theory terms, best-response dynamics. ### Why it is fast -`JoinState` keeps every pair's outcome and every trajectory's hit and found counts. Testing one -switch only rescores that trajectory's own pairs (usually a handful) instead of all -pairs, and the result is exact: the tests check it against a full recompute. - -### What it shows -In an earlier three-candidate run, seed 42 at 0.25%: EQUAL_CHOICE_RDP_PARTNERS 0.506 → EQUAL_LOOP 0.544 J5, with the same -methods and budgets. The gain is in join (0.522 → 0.621) and clustering. The only difference is using the -partners' actual choices. +The trackers keep every pair's outcome, every box's count and every query's ranking, so testing +one switch only rescores that trajectory's own pairs, boxes and watchers, and the result is exact +for range and join: the tests check it against a full recompute. --- @@ -308,7 +234,8 @@ not. They matter because the join metric is F1: simplification can make negative are rebuilt once because their metadata-only fingerprints cannot verify the point contents. 3. Otherwise, it builds everything with `build_knn` and `collect_pairs`, then `save`s it. 4. Always regenerates the range boxes and where-at probes. They are fast, and use their own seeds - (`seed + 7001` and `seed + 7002`). + (`seed + 7001` and `seed + 7002`). Range draws its acceptance stream from its seed + 1, the + same stream as where-at's; harmless, and kept so existing workloads and results stay valid. 5. Always regenerates the extra joins, one self join per extra threshold. They are not cached, so changing the thresholds does not invalidate the kNN cache. - **Output:** a `TeacherWorkloads`. It prints whether the cache was hit, and the counts. @@ -365,21 +292,30 @@ not. They matter because the join metric is F1: simplification can make negative ### Why -This is the score the teacher maximises on its training workloads. Range, where-at and join -use J4's metric definitions; the value curves retain an independent kNN proxy. Improvements -on training workloads must still be checked against held-out J4. +This is the score the teacher maximises on its training workloads. kNN, range, where-at and join +use J4's metric definitions on database queries; the value curves use a per-trajectory kNN check +instead. Improvements on training workloads must still be checked against held-out J4. -The four terms, each normalised `(metric − floor) / (1 − floor)` and then averaged: +Six terms, each normalised `(metric − floor) / (1 − floor)`, then averaged with +`term_weights` (Geolife: 1, 1, 1, 1, 0.5, 1): | Term | Per trajectory i, simplified | Mirrors | |---|---|---| -| kNN | for each query e that has i in its top k: Fréchet(original e, simplified i) ≤ cutoff(e) | recall@k | +| kNN | recall@k of each database query, as original, ranked on the compressed database (`knn_recall`, tracked by `KnnState`) | recall@k | | Range | average `2 * found / (reference size + found)` over boxes with nonempty references | range macro F1 | | Where-at | average-hit ramp over i's probes: `clamp((max − err) / (max − min), 0, 1)` | where-at average-hit | -| Join | macro F1 over the pairs per threshold, both sides simplified, each normalised by its floor, then averaged | J4's join grid mean, computed the same way | +| Join | held-out macro F1 per threshold (original query, compressed partner, both directions), each normalised by its floor, then averaged | J4's join grid mean, computed the same way | +| Shape | ramp 25–500 m on `simplification_bound(original i, simplified i)`, averaged over trajectories | none: a prior for queries the workloads lack | +| Margin | for each query e ranking i among its k + 5 nearest (`workloads.ranked`): 1 if i's deviation bound is within its gap to e's k-th place, else ½(1 + gap / deviation); averaged over memberships | held-out kNN: whether i's rank side survives any query moving it by its deviation | -Where-at and independent kNN are **separable**. Range macro F1, join and ranking kNN depend -on other trajectories and use incremental state trackers during best response. +`simplification_bound` is an upper bound on the continuous Fréchet distance between a trajectory +and its simplification (`certificate_test.cpp`), so for any query Q, |d(Q, simplified i) − d(Q, i)| +is at most it. Shape and margin need no query results, so they also cover held-out queries: without +them the teacher starved trajectories its training workloads never touch. The weights were chosen +on pseudo-held-out splits of the database and confirmed on four seeds (research notes, 2026-11-22). + +Where-at, shape and margin are **separable**. Range macro F1 and ranking kNN depend on other +trajectories; the join and kNN are tracked incrementally during the loop and trades. Why range has no false positives: kept points are a subset of the original points, so a simplified trajectory can't have a point in a box the original had none in. @@ -396,9 +332,8 @@ trajectory can't have a point in a box the original had none in. - **Output:** `view()`, valid while the object lives and is unmodified. `assign` replaces the retained points and rebuilds bounds while reusing coordinate storage between budgets. `PreparedTrajectories` owns a whole set and provides borrowed views. Fixed partners are prepared - once for curve scoring; best response prepares each unique candidate once and shares duplicate - views. Trajectories without range memberships, join links or active kNN queries only need - their selected candidate prepared. + once for curve scoring; the loop prepares every distinct candidate of every trajectory once and + shares duplicate views. ### Range and join helpers — internal @@ -429,29 +364,31 @@ trajectory can't have a point in a box the original had none in. `m_join_weight[k] = 1 / (terms with true pairs · (1 − floor_k))`, so the join metric is already normalised and 0 at the floor. Then `metrics(endpoints)` gives each metric's floor. If a floor is already 1, it throws, because that metric can't be normalised. - 6. **Scales:** `m_scale[m] = 1 / (4 · (1 − floor_m))`. This applies the normalisation and the - equal average in one factor. + 6. **Scales:** `m_scale[m] = weight_m / (Σ weights · (1 − floor_m))`. This applies the + normalisation and the weighted average in one factor. 7. **`m_endpoint_separable[i]`:** trajectory i's scaled separable terms with endpoints only. Subtracting it makes `separable` exactly 0 at the floor. - **Output:** a ready `Objective`. It keeps references to `db` and `workloads`, so both must outlive it (`Teacher` guarantees this). -### `raw_separable(i, kept)` — private +### `raw_separable(i, candidate)` — private -- **Why:** the independent kNN and where-at checks for one trajectory. -- **Input:** trajectory index and kept set. +- **Why:** the where-at, shape and margin terms for one trajectory. The kNN slot stays 0: the + per-trajectory kNN check is done in `independent_value`, the ranking term in `metrics`. +- **Input:** trajectory index and a prepared candidate view. - **Computation:** - - **kNN:** for each membership (e, cutoff), test `frechet_within(original e, simplified i, - cutoff)` and add `m_knn_weight` if it holds. Competitors stay original. - **Where-at:** for each probe, interpolate the simplified position at the probe's time (`core::position_at`), take the distance to the true position, apply the ramp, and average over i's probes times `m_where_at_weight`. + - **Shape and margin**, when weighted: `simplification_bound` once, then the shape ramp and each + of i's margins against it. - **Output:** `Terms`, with i's raw separable contributions (range and join slots 0). ### `separable(i, kept)` — public - **Input:** trajectory index and kept set. -- **Computation:** `Σ m_scale[m] · raw_separable[m] − m_endpoint_separable[i]` over independent kNN and where-at. +- **Computation:** `Σ m_scale[m] · raw_separable[m] − m_endpoint_separable[i]` over where-at, + shape and margin. - **Output:** i's separable part of J_T, 0 at endpoints only. Summed over all trajectories, this equals those terms' share of J_T exactly. @@ -459,44 +396,42 @@ trajectory can't have a point in a box the original had none in. - **Why:** the value curves need one number per (trajectory, candidate), computed without knowing what the other trajectories will choose. -- **Input:** trajectory index, a prepared candidate view, and fixed partner views. An empty - partner span means original partners (the `EQUAL_CHOICE` arm); otherwise the views contain RDP - at equal budgets for `EQUAL_CHOICE_RDP_PARTNERS`. -- **Computation:** `separable(i, candidate)`, plus the marginal range F1 of covering i against - fixed partners, plus i's own join F1 against those partners: count - `found` and `hits` over i's links using `match`, then add +- **Input:** trajectory index, a prepared candidate view, and fixed partner views (empty means + original partners). Partners only enter the range marginals. +- **Computation:** `separable(i, candidate)`, plus the per-trajectory kNN check over i's + memberships, plus the marginal range F1 of covering i against the partners, plus i's held-out + join F1: count `found` and `hits` over i's links using `held_out_match`, then add `m_scale[Join] · Σ_k m_join_weight[k] · F1_ik / (trajectories with truth in term k)`. -- **Output:** an approximate per-trajectory score. It's approximate because the real join depends - on partners' actual choices and counts each pair on both sides. That is exactly what - `best_response` corrects. +- **Output:** an approximate per-trajectory score: range is marginal against fixed partners and + kNN ignores competitors. The loop corrects both. - `prepare_independent` computes the range, where-at and join terms once, together with the upper bound used to order candidates. `independent_value(score, beat)` reuses those terms while checking kNN, so a candidate is not rescored just because its bound was promising. The prepared score borrows its candidate's coordinates and must be used before they change. -### `match(a, b)` — public +### `match(a, b)` and `held_out_match(i, link, candidate)` — public -- **Input:** two prepared trajectory views (or an original view). -- **Computation:** `query::frechet_within` on the simplified views, with the workload's threshold - and Fréchet variant. -- **Output:** whether the pair would be reported by the join. +- `match`: `query::frechet_within` with the term's threshold and the workload's Fréchet variant. +- `held_out_match`: whether original `link.other` finds candidate i. A positive pair whose + originals' discrete Fréchet distance plus `simplification_bound(i)` is below the threshold is + certified without the test (`certificate_test.cpp`). ### `join_metric(state)` — private - **Input:** a whole simplified database. -- **Computation:** - 1. In parallel, `match` every pair used. - 2. For each matched pair, increment `found` on both sides, and `hits` too if it's a positive. - 3. Average `f1_of(hits, found, truth)` over trajectories with any truth or any match. This is the - same counting rule as the engine's evaluator. -- **Output:** macro join F1 on the pairs used. +- **Computation:** for every pair, in parallel, original i against compressed j and original j + against compressed i; each outcome counts for its query. Then `f1_of(hits, found, truth)` per + trajectory and term, averaged over trajectories with any truth or any match, like the evaluator. +- **Output:** the normalised join metric. ### `metrics(kept)` — public - **Input:** a whole simplified database. - **Computation:** prepares the trajectories once, sums `raw_separable` over their views (in - parallel), and reuses those views in `RangeState::metric` and `join_metric`. -- **Output:** the four raw metric values. Used for the floors, and in tests. + parallel), reuses those views in `RangeState::metric` and `join_metric`, and ranks kNN on the + materialised database with `knn_recall`. +- **Output:** the six raw metric values. Used for the floors, `teacher_eval`'s `terms.csv`, and + in tests. ### `total(kept)` — public @@ -525,32 +460,23 @@ The result is exact, not approximate: the tests check it against a full recomput #### `JoinState(objective, state)` — constructor - **Input:** the objective and a starting simplified database. -- **Computation:** matches every pair used once (from its smaller side, in parallel). It fills - `m_matched`, `m_hits[t]` and `m_found[t]`, then sums F1 over counted slots into `m_sum` and - `m_counted`, per join term. +- **Computation:** each trajectory's outcomes as the compressed side (`outcomes`, in parallel). + Each outcome is stored per pair and direction in `m_matched` and counted for the query, then F1 + is summed over counted slots into `m_sum` and `m_counted`, per join term. - **Output:** a state whose `metric()` equals `join_metric(state)`. -#### `propose(i, candidate, state)` +#### `outcomes(i, candidate)` and `propose(i, outcomes)` -- **Why:** "how much would J_T's join term change if i switched to `kept`?", without changing - anything. -- **Input:** trajectory index, prepared candidate view, and current partner views. -- **Computation:** - 1. For each of i's links, compare the pair's current outcome with its outcome under the - candidate. - 2. For every pair that flips, record the changes to hits and found on both trajectories. - 3. For each affected trajectory, remove its old F1 from the sum and add its new one, adjusting - the counted trajectories when one starts or stops being counted. -- **Output:** a `Proposal` with the trajectory index, pair outcomes, and - `gain = m_scale[Join] · (new join metric − current join metric)`. Added to the cached `separable` - difference and range/kNN gains, this is the change in J_T. Proposals remain valid until the next `apply`. +- `outcomes`: i's link outcomes with i as candidate. They do not depend on the state, so the loop + computes them once per candidate. +- `propose`: for every link whose outcome flips, the change in hits and found at the query, then + each affected slot's F1 swapped in the sum. Returns a `Proposal` with + `gain = m_scale[Join] · (new join metric − current join metric)`, valid until the next `apply`. #### `apply(proposal)` -- **Input:** the winning proposal evaluated against the current state. -- **Computation:** writes its saved outcomes into `m_matched`, the counts and the sum, without - repeating Fréchet tests. The caller also switches the current view to the winning candidate. -- **Output:** nothing; the state now matches the new solution. +Writes the saved outcomes into `m_matched`, the counts and the sum, without repeating Fréchet +tests. #### `metric()`, `counted(t, found)`, `f1(t, hits, found)` @@ -589,14 +515,14 @@ simplified. | Name | Meaning | |---|---| | `CandidateIndex`, `kCandidates` | Named indices and labels for the six candidate recipes. The index is the "method". | -| `kArms` | the nine valid arm names, listed in `teacher.hpp` | -| `TeacherConfig` | `enabled`, `cache_dir`, `arms`, `range_query_count` (default 128000), `join` (optional, default: the pipeline's `join.mode`), `extra_join_eps_m` (default: `join.score_eps_m`), `held_out_term`, `ranking_knn` (default true), `term_weights` (kNN, range, where-at, join; equal by default) | +| `kArms` | the four arm names, listed in `teacher.hpp` | +| `TeacherConfig` | `enabled`, `cache_dir`, `arms`, `range_query_count` (default 128000), `extra_join_eps_m` (default: `join.score_eps_m`), `term_weights` (kNN, range, where-at, join, shape, margin; 4 to 6 values, missing ones 0; default 1, 1, 1, 1, 0, 0) | | `Arm` | one result: `name`, `simplification` (for the pipeline), `objective` (its J_T) | | `Candidates` | `candidates[c][i]` = trajectory i simplified with method c | | `Methods` | one method index per trajectory, i.e. a selection | | `kMaxResponsePasses` | at most 10 best-response passes | -| `kKnnWatchFactor` | each kNN query watches its 3k nearest trajectories under each candidate state | -| `kMaxAllocationRounds` | at most 10 iterative reallocation rounds; they stop once J_T stops rising | +| `kKnnWatchFactor` | each kNN query watches its 3k nearest trajectories in the start state | +| `kTradeSteps`, `kMaxTradeRounds` | trades move 1, 2 or 4 points; at most 3 rounds | | `kMinImprovement` | strict gain threshold, 1e-12 | ### `Teacher(db, workloads, config, seed)` — constructor @@ -615,16 +541,13 @@ simplified. 2. `equal_budgets` = equal budgets per trajectory (the engine's size^α allocator with α = 0). Curves are capped at about four times the mean interior budget, at least six interior points, never below the equal budget and never above trajectory length. - 3. `make_candidates` at those budgets, building only RDP if no fixed-budget arm needs alternatives. - 4. **`EQUAL_CHOICE`** (if requested): `independent_choice` with `partners = nullptr`. - 5. **`EQUAL_CHOICE_RDP_PARTNERS` and `EQUAL_LOOP`** (if either is requested): - - score candidates directly against equal-budget RDP partners, or reuse curves if allocation - arms also need them, to obtain `EQUAL_CHOICE_RDP_PARTNERS`; - - `best_response` starting from `EQUAL_CHOICE_RDP_PARTNERS`, which gives `EQUAL_LOOP`. - 6. Allocation arms build curves against equal-budget RDP, allocate the total budget, and select - candidates. Response arms refine that selection; iterative rounds rebuild against the current - partners and keep only improvements in exact J_T. - 7. `keep` assembles each arm's kept sets, computes its exact J_T, and prints how many + 3. RDP at those budgets (`make_candidates`): the fixed partners for the value curves. + 4. `build_curves`, RDP only unless a later arm needs the best candidate. + 5. **`BUDGET_RDP`:** `allocate` on the RDP curves. + 6. **`BUDGET_CHOICE`:** `allocate` on the best curves; each trajectory takes its curve's method. + 7. **`BUDGET_LOOP`:** `Search(...).run(seed)` from `BUDGET_CHOICE`, then `checked_loop`. + 8. **`BUDGET_LOOP_TRADE`:** `trade_budgets` from `BUDGET_LOOP`. + 9. `keep` assembles each arm's kept sets, computes its exact J_T, and prints how many trajectories each method won. `timed` prints step durations. - **Output:** one `Arm` per requested name, in the requested order. Each keeps exactly `total_budget` points, because every candidate keeps exactly its trajectory's budget. @@ -643,21 +566,11 @@ simplified. - **Computation:** for each trajectory, picks the kept set of its chosen method. - **Output:** a whole simplified database (`KeptSets`). -### `independent_choice(db, objective, candidates, partners)` — internal - -- **Why:** the simplest teacher. Each trajectory picks its best method on its own, against a fixed - partner state. -- **Input:** candidates, and the partner state for `independent_value`. -- **Computation:** in parallel, for each trajectory, the method with the highest - `independent_value`. Ties go to the earlier method, so RDP wins when nothing differs. -- **Output:** `Methods`. With `partners = nullptr` this gives `EQUAL_CHOICE`. Original partners - hide the effect of simplifying both sides of a join pair. - ### `build_curves(db, objective, partners, caps, rdp_only)` — internal - **Why:** the value curves: for each trajectory and each budget, the best method and its value. -- **Input:** the partner reference state (initially RDP at equal budgets, or the current state in - iterative rounds), curve caps, and whether only RDP is needed. +- **Input:** the partner reference state (RDP at equal budgets), curve caps, and whether only RDP + is needed. - **Computation:** in parallel per trajectory, build the RDP addition sequence and reversed SED removal sequence once, then use sorted prefixes for each budget. Candidate coordinate buffers are reused between budgets. For b = 2 … max: @@ -665,11 +578,8 @@ simplified. - score each with `independent_value` against the fixed partner reference; - record the best value and method. - **Output:** RDP-only and best-candidate curves per trajectory. -- Runs requesting only `EQUAL_CHOICE_RDP_PARTNERS` and/or `EQUAL_LOOP` score their fixed budgets directly and skip full - curve construction. -- **Why partners as RDP:** a fixed, plausible guess at how partners will look. It still scores each - trajectory on its own, but it rewards matching how partners will likely be simplified. This took - J5 from 0.446 (`EQUAL_CHOICE`) to 0.506 (`EQUAL_CHOICE_RDP_PARTNERS`) in the earlier three-candidate experiment. +- **Why partners as RDP:** a fixed, plausible guess at how partners will look, so range marginals + reward covering boxes the others will likely miss. ### `curve_methods(curves, budgets)` — internal @@ -677,23 +587,20 @@ simplified. - **Computation:** `method[b − 2]` for each trajectory. Throws if a budget is off its curve. - **Output:** `Methods` at the supplied budgets. -### `best_response(db, objective, candidates, method, seed)` — internal +### `Search(db, objective, candidates, method)` and `run(seed)` — internal -- **Why:** this step uses the partners' **actual** choices instead of a guess. A trajectory's best - method depends on what its join partners chose, so the teacher revisits trajectories until no - single change helps. -- **Input:** candidates, a starting selection (`EQUAL_CHOICE_RDP_PARTNERS` or `BUDGET_CHOICE`), and the seed. -- **Computation:** - 1. `order` = trajectories with a range membership, a join link or an active ranking kNN query. - 2. Prepares candidate views, caches separable scores, and builds range, join and kNN trackers. - 3. For up to 10 passes, in a seeded random order: - - for each trajectory i, and each other method c: gain = `separable(c) − separable(current)` - + `range.propose(c).gain` + `join.propose(c).gain` + `knn.propose(c)`. That's the exact change in J_T, up to - trajectories outside a query's watch list entering its top k; - - switches to the best c only if its gain beats 1e-12 (strictly positive, so ties can't - cycle), and applies the move to all trackers. - 4. Stops after a pass with no switches. -- **Output:** the improved `Methods`. Stops after a pass with no switches or the 10-pass limit. +The loop described above. The constructor prepares every distinct candidate and builds the +trackers; `run` updates them in place, so it runs once. + +### `trade_budgets(db, objective, caps, kept, value)` — internal + +- **Why:** allocation scores trajectories against fixed partners; trades correct budgets against + the actual state. +- **Computation:** each round, every trajectory's best recipe at `b ± 1, 2, 4` is scored against + the current state (kNN by `propose_bounded`, computing a distance only near a query's k-th + place). The kNN lists then watch every offered version. Buyers are paired with the cheapest + sellers; each pair is rescored exactly as applied and kept only if the sum gains. +- **Output:** the traded kept sets and their exact J_T, or the input if the exact J_T fell. ### `wins`, `is_arm` — helpers @@ -714,20 +621,6 @@ simplified. goes through the same `evaluate` lambda: materialise, score on the held-out workloads, print, and export as `_pct`. -## Earlier three-candidate results (Geolife, 0.25%, seed 42) - -These predate J4 and use J5, which includes clustering. Arm names are the old ones: `GATEORIG_a0` is now `EQUAL_CHOICE`, `GATE_a0` is `EQUAL_CHOICE_RDP_PARTNERS` and `BR_a0` is `EQUAL_LOOP`. - -| Arm | J_T | J5 | -|---|---:|---:| -| RDP_a0 (baseline) | — | 0.492 | -| GATEORIG_a0 | 0.458 | 0.446 | -| GATE_a0 | 0.503 | 0.506 | -| BR_a0 | 0.548 | 0.544 | - -Historical run time: about 7 s for `GATEORIG_a0`, 16 s for the curves and 66 s for best response. -These timings predate the seven-candidate teacher, allocation search and cache changes. - ## Tests `engine/tests/teacher/teacher_test.cpp` uses a small synthetic database: @@ -737,18 +630,22 @@ These timings predate the seven-candidate teacher, allocation search and cache c - `ObjectiveIsZeroAtTheFloorAndTracksJoinExactly`: J_T = 0 at endpoints only. After random switches, the tracked range and join F1 equal full recomputations, and their gains plus separable predict the real change in J_T. -- `SeveralJoinTermsAreTrackedExactly`: the same exactness with two extra thresholds, with and - without the held-out term. +- `SeveralJoinTermsAreTrackedExactly`: the same exactness with two extra thresholds. - `ContinuousKnnNearTiesMatchFullScoringThroughSwitches`: the former 23/24 versus 1 regression, - checked through original/compressed query switches and bounded proposals. + checked through switches and bounded proposals. - `RankingKnnIsTrackedExactly`: when every query watches every trajectory, `KnnState`'s tracked recall equals a full recompute after random switches, and the kNN, range, join and separable changes together predict the real change in J_T. - `RankingKnnArmsKeepTheBudget`: the arms keep exactly the budget with the ranking kNN term. - `SingleRecipesKeepTheBudget`: the six single-recipe baselines each keep exactly the budget. - `ArmsKeepTheBudgetAndBestResponseOnlyImproves`: every arm keeps exactly the budget, endpoints - kept, indices sorted; and `EQUAL_LOOP`'s J_T ≥ `EQUAL_CHOICE_RDP_PARTNERS`'s. + kept, indices sorted; `BUDGET_LOOP`'s J_T ≥ `BUDGET_CHOICE`'s and the trades' ≥ the loop's. - `JoinTermsAreNormalisedApartAndAveragedLikeJ4`: the join metric equals the mean over thresholds of each threshold's macro F1, normalised by its own endpoints-only floor. +- `ShapeTermRampsTheDeviationBound`, `TwoSidedMarginScoresOutsidersToo`: shape and margin are 1 + with every point, 0 at the floor, separable, and the margin term needs ranked neighbours. + +`engine/tests/teacher/certificate_test.cpp` checks `simplification_bound` never falls below the +continuous Fréchet distance, alone and added to a discrete pair distance. `engine/tests/config_test.cpp` has the three teacher config tests. diff --git a/engine/src/teacher/greedy.cpp b/engine/src/teacher/greedy.cpp index 73e6749..243ad30 100644 --- a/engine/src/teacher/greedy.cpp +++ b/engine/src/teacher/greedy.cpp @@ -76,10 +76,6 @@ std::vector Greedy::sequence(std::size_t limit, std::span(trajectory.t[from]), @@ -88,7 +84,8 @@ std::vector Greedy::sequence(std::size_t limit, std::span t_from ? std::clamp((m_probe_t[p] - t_from) / (t_to - t_from), 0.0, 1.0) : 0.0; const double x = trajectory.x[from] + fraction * (trajectory.x[to] - trajectory.x[from]); const double y = trajectory.y[from] + fraction * (trajectory.y[to] - trajectory.y[from]); - return ramp(std::hypot(x - m_probe_x[p], y - m_probe_y[p])); + return ramp(std::hypot(x - m_probe_x[p], y - m_probe_y[p]), m_min_tolerance, + m_max_tolerance); }; // Probes from t_a up to t_c, the last segment including its end auto probes_between = [&](std::size_t from, std::size_t to) { diff --git a/engine/src/teacher/objective.cpp b/engine/src/teacher/objective.cpp index 476cbe6..a39e483 100644 --- a/engine/src/teacher/objective.cpp +++ b/engine/src/teacher/objective.cpp @@ -157,9 +157,10 @@ Objective::Objective(const core::TrajectoryDb &db, const TeacherWorkloads &workl throw std::invalid_argument("objective: kNN workload does not match the database"); if (!(config.min_tolerance_m < config.max_tolerance_m)) throw std::invalid_argument("objective: min_tolerance_m must be below max_tolerance_m"); + if (!(config.min_shape_m < config.max_shape_m)) + throw std::invalid_argument("objective: min_shape_m must be below max_shape_m"); + m_shape_weight = n > 0 ? 1.0 / static_cast(n) : 0.0; const double weight_sum = check_term_weights(config.term_weights); - if (config.compressed_knn_query && !config.ranking_knn) - throw std::invalid_argument("objective: compressed kNN queries need the ranking kNN term"); // kNN: invert each query's top k m_knn.resize(n); @@ -174,17 +175,37 @@ Objective::Objective(const core::TrajectoryDb &db, const TeacherWorkloads &workl } m_knn_weight = valid ? 1.0 / (static_cast(workloads.k) * static_cast(valid)) : 0.0; + m_margins.resize(n); + const std::size_t k = workloads.k; + if (config.term_weights[Margin] > 0) { + // Both sides of each query's k-th place: a true neighbour against the (k + 1)th, an + // outsider against the k-th + if (workloads.ranked.size() != n) + throw std::invalid_argument("objective: the margin term needs ranked neighbours"); + for (std::size_t e = 0; e < n; ++e) { + const auto &list = workloads.ranked[e]; + if (!workloads.knn[e].valid || list.size() <= k) + continue; + const double inside = list[k - 1].distance, outside = list[k].distance; + for (std::size_t r = 0; r < list.size(); ++r) + m_margins[list[r].index].push_back( + std::max(0.0, r < k ? outside - list[r].distance : list[r].distance - inside)); + } + } + std::size_t memberships = 0; + for (const auto &margins : m_margins) + memberships += margins.size(); + m_margin_weight = memberships > 0 ? 1.0 / static_cast(memberships) : 0.0; // Range: invert each box's reference m_range.resize(n); - std::size_t scored = 0; for (const auto &query : workloads.range.queries) - scored += !query.reference.empty(); + m_scored_boxes += !query.reference.empty(); for (std::size_t b = 0; b < workloads.range.queries.size(); ++b) { const auto &ref = workloads.range.queries[b].reference; for (const auto i : ref) m_range[i].push_back({static_cast(b), - range_marginal(ref.size() - 1, ref.size(), scored)}); + range_marginal(ref.size() - 1, ref.size(), m_scored_boxes)}); } // Where-at: probes per trajectory @@ -196,24 +217,16 @@ Objective::Objective(const core::TrajectoryDb &db, const TeacherWorkloads &workl std::ranges::count_if(m_probes, [](auto &probes) { return !probes.empty(); }); m_where_at_weight = with_probes ? 1.0 / static_cast(with_probes) : 0.0; - // Join terms: join_eps_m, the extra thresholds, then the held-out form of the first. Pairs are - // stored positives first, then negatives + // Join terms: join_eps_m, then the extra thresholds. Pairs are stored positives first, then + // negatives struct TermPairs { const std::vector *positives, *negatives; }; - const bool held_out = config.join_mode == query::JoinMode::HeldOut; std::vector terms{{&workloads.positives, &workloads.negatives}}; m_eps = {workloads.join_eps_m}; - m_term_held_out = {held_out}; for (const auto &extra : workloads.extra_joins) { terms.push_back({&extra.positives, &extra.negatives}); m_eps.push_back(extra.eps_m); - m_term_held_out.push_back(held_out); - } - if (config.held_out_term && !held_out) { - terms.push_back(terms.front()); - m_eps.push_back(workloads.join_eps_m); - m_term_held_out.push_back(true); } if (terms.size() > kMaxJoinTerms) throw std::invalid_argument("objective: too many join terms"); @@ -243,11 +256,11 @@ Objective::Objective(const core::TrajectoryDb &db, const TeacherWorkloads &workl for (std::size_t t = 0; t < m_truth.size(); ++t) m_with_truth[t / n] += m_truth[t] > 0; - // Held-out positives: the originals' discrete Fréchet distance bounds the continuous one + // Positives: the originals' discrete Fréchet distance bounds the continuous one m_pair_upper.assign(m_pair_i.size(), std::numeric_limits::infinity()); core::parallel_for(m_positive_count, [&](std::size_t p) { const auto k = m_pair_term[p]; - if (!m_term_held_out[k] || m_workloads.frechet != query::FrechetVariant::Continuous) + if (m_workloads.frechet != query::FrechetVariant::Continuous) return; std::vector prev, curr; const double cutoff = m_eps[k] * m_eps[k]; @@ -279,9 +292,9 @@ Objective::Objective(const core::TrajectoryDb &db, const TeacherWorkloads &workl } m_endpoint_separable.assign(n, 0.0); core::parallel_for(n, [&](std::size_t i) { - const auto endpoint_terms = raw_separable( - i, PreparedTrajectory(db.trajectory(i), endpoints[i]).view(), !config.ranking_knn); - for (const auto m : {Knn, WhereAt}) + const auto endpoint_terms = + raw_separable(i, PreparedTrajectory(db.trajectory(i), endpoints[i]).view()); + for (const auto m : {WhereAt, Shape, Margin}) m_endpoint_separable[i] += m_scale[m] * endpoint_terms[m]; }); } @@ -303,33 +316,33 @@ bool Objective::match(const core::TrajectoryView &a, const core::TrajectoryView return query::frechet_within(a, b, m_eps[term], m_workloads.frechet); } -Objective::Terms Objective::raw_separable(std::size_t i, std::span kept) const { - return raw_separable(i, PreparedTrajectory(m_db.trajectory(i), kept).view()); -} - -Objective::Terms Objective::raw_separable(std::size_t i, const core::TrajectoryView &candidate, - bool knn) const { +Objective::Terms Objective::raw_separable(std::size_t i, + const core::TrajectoryView &candidate) const { Terms terms{}; - - if (knn) { - for (const auto &membership : m_knn[i]) - if (query::frechet_within(m_db.trajectory(membership.query), candidate, - membership.cutoff, m_workloads.frechet)) - terms[Knn] += m_knn_weight; - } - if (!m_probes[i].empty()) { double sum = 0; for (const auto k : m_probes[i]) { const auto &probe = m_workloads.where_at.queries[k]; const double error = core::distance(core::position_at(candidate, probe.t), probe.reference); - sum += std::clamp((m_config.max_tolerance_m - error) / - (m_config.max_tolerance_m - m_config.min_tolerance_m), - 0.0, 1.0); + sum += ramp(error, m_config.min_tolerance_m, m_config.max_tolerance_m); } terms[WhereAt] = m_where_at_weight * sum / static_cast(m_probes[i].size()); } + + const bool shape = m_config.term_weights[Shape] > 0; + const bool margin = m_config.term_weights[Margin] > 0 && !m_margins[i].empty(); + if (shape || margin) { + const double deviation = simplification_bound(m_db.trajectory(i), candidate); + if (shape) + terms[Shape] = + m_shape_weight * ramp(deviation, m_config.min_shape_m, m_config.max_shape_m); + // The distance moves uniformly within +-deviation; a hit survives up to its margin + if (margin) + for (const double gap : m_margins[i]) + terms[Margin] += + m_margin_weight * (deviation <= gap ? 1.0 : 0.5 * (1.0 + gap / deviation)); + } return terms; } @@ -338,16 +351,15 @@ double Objective::separable(std::size_t i, std::span kept) co } double Objective::separable(std::size_t i, const core::TrajectoryView &candidate) const { - const auto t = raw_separable(i, candidate, !m_config.ranking_knn); - return m_scale[Knn] * t[Knn] + m_scale[WhereAt] * t[WhereAt] - m_endpoint_separable[i]; + const auto t = raw_separable(i, candidate); + return m_scale[WhereAt] * t[WhereAt] + m_scale[Shape] * t[Shape] + m_scale[Margin] * t[Margin] - + m_endpoint_separable[i]; } std::vector Objective::range_weights(std::size_t i, std::span partners) const { if (m_range[i].empty()) return {}; - const auto scored = std::ranges::count_if(m_workloads.range.queries, - [](const auto &q) { return !q.reference.empty(); }); std::vector weights; weights.reserve(m_range[i].size()); for (const auto &membership : m_range[i]) { @@ -356,7 +368,7 @@ std::vector Objective::range_weights(std::size_t i, for (const auto j : q.reference) if (j != i && (partners.empty() || query::intersects(partners[j], q.box))) ++others; - weights.push_back(range_marginal(others, q.reference.size(), scored)); + weights.push_back(range_marginal(others, q.reference.size(), m_scored_boxes)); } return weights; } @@ -375,7 +387,7 @@ Objective::prepare_independent(std::size_t i, const core::TrajectoryView &candid score.trajectory = i; score.candidate = candidate; score.partners = partners; - score.terms = raw_separable(i, candidate, false); + score.terms = raw_separable(i, candidate); const auto owned_weights = range_weights.empty() ? this->range_weights(i, partners) : std::vector{}; if (range_weights.empty()) @@ -388,13 +400,8 @@ Objective::prepare_independent(std::size_t i, const core::TrajectoryView &candid std::array hits{}, found{}; double own_error = -1.0; for (const auto &link : m_links[i]) { - // Held out: the partner is the original query and i the database side - const bool matched = - m_term_held_out[link.term] - ? held_out_match(i, link, candidate, own_error) - : match(candidate, - partners.empty() ? m_db.trajectory(link.other) : partners[link.other], - link.term); + // The partner is the original query and i the database side + const bool matched = held_out_match(i, link, candidate, own_error); found[link.term] += matched; hits[link.term] += matched && link.positive; } @@ -404,70 +411,36 @@ Objective::prepare_independent(std::size_t i, const core::TrajectoryView &candid join += m_join_weight[k] * f1_of(hits[k], found[k], m_truth[slot(i, k)]) / static_cast(m_with_truth[k]); score.join = m_scale[Join] * join; - score.bound = independent_sum(score, score.terms[Knn]) + - m_scale[Knn] * knn_check_weight() * static_cast(knn_checks(i)); + score.bound = independent_sum(score, 0.0) + + m_scale[Knn] * m_knn_weight * static_cast(m_knn[i].size()); return score; } double Objective::independent_sum(const IndependentScore &score, double knn) const { // Range is i's marginal F1 against fixed partners; the loop tracks the actual joint F1 return m_scale[Knn] * knn + m_scale[Range] * score.terms[Range] + - m_scale[WhereAt] * score.terms[WhereAt] - m_endpoint_separable[score.trajectory] + + m_scale[WhereAt] * score.terms[WhereAt] + m_scale[Shape] * score.terms[Shape] + + m_scale[Margin] * score.terms[Margin] - m_endpoint_separable[score.trajectory] + score.join; } double Objective::independent_value(const IndependentScore &score, double beat) const { - const auto i = score.trajectory; - double knn = score.terms[Knn]; + double knn = 0.0; auto value = [&] { return independent_sum(score, knn); }; // Stop once even all remaining hits cannot beat beat - const double weight = knn_check_weight(); - const double per_hit = m_scale[Knn] * weight; - std::size_t left = knn_checks(i); - auto check = [&](const core::TrajectoryView &a, const core::TrajectoryView &b, double cutoff) { + const double per_hit = m_scale[Knn] * m_knn_weight; + std::size_t left = m_knn[score.trajectory].size(); + for (const auto &m : m_knn[score.trajectory]) { if (value() + per_hit * static_cast(left) < beat - kBoundSlack) - return false; + break; --left; - if (query::frechet_within(a, b, cutoff, m_workloads.frechet)) - knn += weight; - return true; - }; - if (!m_config.compressed_knn_query) { - for (const auto &m : m_knn[i]) - if (!check(m_db.trajectory(m.query), score.candidate, m.cutoff)) - return value(); - return value(); + if (query::frechet_within(m_db.trajectory(m.query), score.candidate, m.cutoff, + m_workloads.frechet)) + knn += m_knn_weight; } - // Compressed queries: i is found by the queries it belongs to, and finds its own neighbours - auto partner = [&](std::size_t j) { - return score.partners.empty() ? m_db.trajectory(j) : score.partners[j]; - }; - for (const auto &m : m_knn[i]) - if (!check(partner(m.query), score.candidate, m.cutoff)) - return value(); - const auto &own = m_workloads.knn[i]; - if (own.valid) - for (const auto j : own.top) - if (!check(score.candidate, partner(j), own.cutoff)) - return value(); return value(); } -std::size_t Objective::knn_checks(std::size_t i) const { - const auto &own = m_workloads.knn[i]; - return m_knn[i].size() + (m_config.compressed_knn_query && own.valid ? own.top.size() : 0); -} - -double Objective::knn_check_weight() const { - // Compressed queries check each (query, neighbour) hit from both sides, so each counts half - return m_config.compressed_knn_query ? m_knn_weight / 2.0 : m_knn_weight; -} - -double Objective::independent_bound(std::size_t i, const core::TrajectoryView &candidate, - std::span partners) const { - return prepare_independent(i, candidate, partners).bound; -} - double Objective::join_value(std::span f1s) const { double value = 0; for (std::size_t k = 0; k < f1s.size(); ++k) @@ -480,18 +453,13 @@ double Objective::join_metric(std::span state) const } std::vector Objective::join_f1s(std::span state) const { - // matched[p]: the pair's outcome, counted for i. reverse[p]: counted for j, the same outcome - // unless held out, where each side is in turn the original query + // matched[p]: original i queries compressed j, counted for i. reverse[p]: the other way std::vector matched(m_pair_i.size()), reverse(m_pair_i.size()); core::parallel_for(m_pair_i.size(), [&](std::size_t p) { const auto i = m_pair_i[p], j = m_pair_j[p]; const auto k = m_pair_term[p]; - if (m_term_held_out[k]) { - matched[p] = match(m_db.trajectory(i), state[j], k); - reverse[p] = match(m_db.trajectory(j), state[i], k); - } else { - matched[p] = reverse[p] = match(state[i], state[j], k); - } + matched[p] = match(m_db.trajectory(i), state[j], k); + reverse[p] = match(m_db.trajectory(j), state[i], k); }); std::vector hits(m_truth.size(), 0), found(m_truth.size(), 0); for (std::size_t p = 0; p < m_pair_i.size(); ++p) { @@ -525,17 +493,14 @@ Objective::Terms Objective::metrics(const KeptSets &kept) const { const PreparedTrajectories prepared(m_db, kept); const auto views = prepared.views(); std::vector per(kept.size()); - core::parallel_for(kept.size(), [&](std::size_t i) { - per[i] = raw_separable(i, views[i], !m_config.ranking_knn); - }); + core::parallel_for(kept.size(), [&](std::size_t i) { per[i] = raw_separable(i, views[i]); }); Terms total{}; for (const auto &terms : per) - for (const auto m : {Knn, WhereAt}) + for (const auto m : {WhereAt, Shape, Margin}) total[m] += terms[m]; total[Range] = RangeState(*this, views).metric(); total[Join] = join_metric(views); - if (m_config.ranking_knn) - total[Knn] = knn_recall(core::materialize(m_db, kept)); + total[Knn] = knn_recall(core::materialize(m_db, kept)); return total; } @@ -546,10 +511,8 @@ double Objective::knn_recall(const core::TrajectoryDb &compressed) const { const auto &entry = m_workloads.knn[e]; if (!entry.valid) return; - const auto query = - m_config.compressed_knn_query ? compressed.trajectory(e) : m_db.trajectory(e); - auto found = - query::knn(compressed, query, std::min(k + 1, m_db.size()), m_workloads.frechet); + auto found = query::knn(compressed, m_db.trajectory(e), std::min(k + 1, m_db.size()), + m_workloads.frechet); std::erase(found, e); found.resize(std::min(found.size(), k)); for (const auto i : entry.top) @@ -663,30 +626,20 @@ JoinState::JoinState(const Objective &objective, std::span>> outcome(n); - core::parallel_for(n, [&](std::size_t i) { - double own_error = -1.0; - for (const auto &link : objective.links(i)) { - if (objective.held_out(link.term)) - outcome[i].push_back( - {slot(i, link), objective.held_out_match(i, link, state[i], own_error)}); - else if (i < link.other) - outcome[i].push_back( - {slot(i, link), objective.match(state[i], state[link.other], link.term)}); - } - }); - for (std::size_t i = 0; i < n; ++i) - for (const auto &[at, matched] : outcome[i]) - m_matched[at] = matched; - for (std::size_t i = 0; i < n; ++i) - for (const auto &link : objective.links(i)) - if (m_matched[slot(i, link)]) { - // Held out, the match is found by the query, not by the compressed side - const std::size_t t = count(objective.held_out(link.term) ? link.other : i, link); + // Each direction from its compressed side + std::vector> outcome(n); + core::parallel_for(n, [&](std::size_t i) { outcome[i] = outcomes(i, state[i]); }); + for (std::size_t i = 0; i < n; ++i) { + const auto links = objective.links(i); + for (std::size_t k = 0; k < links.size(); ++k) { + m_matched[slot(i, links[k])] = outcome[i][k]; + if (outcome[i][k]) { + const std::size_t t = count(links[k]); ++m_found[t]; - m_hits[t] += link.positive; + m_hits[t] += links[k].positive; } + } + } for (std::size_t t = 0; t < objective.slots(); ++t) if (counted(t, m_found[t])) { m_sum[t / n] += f1(t, m_hits[t], m_found[t]); @@ -704,29 +657,16 @@ double JoinState::f1(std::size_t t, std::size_t hits, std::size_t found) const { double JoinState::metric() const { return mean_f1(m_objective, m_sum, m_counted); } -JoinState::Proposal JoinState::propose(std::size_t i, const core::TrajectoryView &candidate, - std::span state) const { - return propose(i, outcomes(i, candidate, state)); +JoinState::Proposal JoinState::propose(std::size_t i, const core::TrajectoryView &candidate) const { + return propose(i, outcomes(i, candidate)); } -std::vector JoinState::outcomes(std::size_t i, const core::TrajectoryView &candidate, - std::span state, - std::span held_out_known) const { - const auto links = m_objective.links(i); - if (!held_out_known.empty() && held_out_known.size() != links.size()) - throw std::invalid_argument("join outcomes: one known outcome per link needed"); +std::vector JoinState::outcomes(std::size_t i, const core::TrajectoryView &candidate) const { std::vector matched; - matched.reserve(links.size()); + matched.reserve(m_objective.links(i).size()); double own_error = -1.0; - for (std::size_t k = 0; k < links.size(); ++k) { - const auto &link = links[k]; - if (!m_objective.held_out(link.term)) - matched.push_back(m_objective.match(candidate, state[link.other], link.term)); - else if (!held_out_known.empty()) - matched.push_back(held_out_known[k]); - else - matched.push_back(m_objective.held_out_match(i, link, candidate, own_error)); - } + for (const auto &link : m_objective.links(i)) + matched.push_back(m_objective.held_out_match(i, link, candidate, own_error)); return matched; } @@ -735,7 +675,7 @@ JoinState::Proposal JoinState::propose(std::size_t i, std::vector matched) const auto links = m_objective.links(i); if (proposal.matched.size() != links.size()) throw std::invalid_argument("join proposal: outcome count does not match links"); - // The hits/found changes the new outcomes cause on each side + // The hits/found changes the new outcomes cause at each query std::unordered_map> change; // t -> (hits, found) for (std::size_t k = 0; k < links.size(); ++k) { const auto &link = links[k]; @@ -744,12 +684,9 @@ JoinState::Proposal JoinState::propose(std::size_t i, std::vector matched) if (now == then) continue; const long delta = then ? 1 : -1; - const std::array touched{count(link.other, link), count(i, link)}; - for (const auto t : std::span(touched).first(sides(link))) { - auto &[hits, found] = change[t]; - found += delta; - hits += link.positive ? delta : 0; - } + auto &[hits, found] = change[count(link)]; + found += delta; + hits += link.positive ? delta : 0; } if (change.empty()) return proposal; @@ -792,26 +729,23 @@ void JoinState::apply(const Proposal &proposal) { continue; m_matched[slot(i, link)] = then; const long delta = then ? 1 : -1; - const std::array touched{count(link.other, link), count(i, link)}; - for (const auto t : std::span(touched).first(sides(link))) { - const auto term = t / m_objective.db().size(); - if (counted(t, m_found[t])) { - m_sum[term] -= f1(t, m_hits[t], m_found[t]); - --m_counted[term]; - } - m_found[t] = static_cast(static_cast(m_found[t]) + delta); - if (link.positive) - m_hits[t] = static_cast(static_cast(m_hits[t]) + delta); - if (counted(t, m_found[t])) { - m_sum[term] += f1(t, m_hits[t], m_found[t]); - ++m_counted[term]; - } + const auto t = count(link); + const auto term = t / m_objective.db().size(); + if (counted(t, m_found[t])) { + m_sum[term] -= f1(t, m_hits[t], m_found[t]); + --m_counted[term]; + } + m_found[t] = static_cast(static_cast(m_found[t]) + delta); + if (link.positive) + m_hits[t] = static_cast(static_cast(m_hits[t]) + delta); + if (counted(t, m_found[t])) { + m_sum[term] += f1(t, m_hits[t], m_found[t]); + ++m_counted[term]; } } } -KnnState::KnnState(const Objective &objective, const KeptSets &state, - std::span watch_states, std::size_t watch) +KnnState::KnnState(const Objective &objective, const KeptSets &state, std::size_t watch) : m_objective(objective), m_watch(watch) { const auto &db = objective.db(); const auto &workloads = objective.workloads(); @@ -820,24 +754,17 @@ KnnState::KnnState(const Objective &objective, const KeptSets &state, m_distance.resize(n); m_hits.assign(n, 0); m_watchers.resize(n); - std::vector compressed; - compressed.push_back(core::materialize(db, state)); - for (const auto &watch_state : watch_states) - compressed.push_back(core::materialize(db, watch_state)); + const auto compressed = core::materialize(db, state); core::parallel_for(n, [&](std::size_t e) { const auto &entry = workloads.knn[e]; if (!entry.valid) return; auto &list = m_list[e]; list.assign(entry.top.begin(), entry.top.end()); - for (const auto &compressed_db : compressed) - for (const auto j : - query::knn(compressed_db, - objective.config().compressed_knn_query ? compressed_db.trajectory(e) - : db.trajectory(e), - std::min(watch + 1, n), workloads.frechet)) - if (j != e && std::ranges::find(list, j) == list.end()) - list.push_back(static_cast(j)); + for (const auto j : + query::knn(compressed, db.trajectory(e), std::min(watch + 1, n), workloads.frechet)) + if (j != e && std::ranges::find(list, j) == list.end()) + list.push_back(static_cast(j)); }); for (std::size_t e = 0; e < n; ++e) { m_valid += workloads.knn[e].valid; @@ -848,16 +775,11 @@ KnnState::KnnState(const Objective &objective, const KeptSets &state, } const PreparedTrajectories prepared(db, state); const auto views = prepared.views(); - if (objective.config().compressed_knn_query) { - core::parallel_for(n, [&](std::size_t e) { m_distance[e] = own_row(e, views[e], views); }); - } else { - std::vector> own(n); - core::parallel_for(n, [&](std::size_t i) { own[i] = distances(i, views[i]); }); - for (std::size_t i = 0; i < n; ++i) - for (std::size_t watch_state = 0; watch_state < m_watchers[i].size(); ++watch_state) - m_distance[m_watchers[i][watch_state].query][m_watchers[i][watch_state].position] = - own[i][watch_state]; - } + std::vector> own(n); + core::parallel_for(n, [&](std::size_t i) { own[i] = distances(i, views[i]); }); + for (std::size_t i = 0; i < n; ++i) + for (std::size_t s = 0; s < m_watchers[i].size(); ++s) + m_distance[m_watchers[i][s].query][m_watchers[i][s].position] = own[i][s]; m_rank.resize(n); core::parallel_for(n, [&](std::size_t e) { if (!m_list[e].empty()) @@ -883,63 +805,11 @@ std::vector KnnState::distances(std::size_t i, const core::TrajectoryVie return result; } -bool KnnState::active(std::size_t i) const { - return watched(i) || (m_objective.config().compressed_knn_query && !m_list[i].empty()); -} - -std::vector KnnState::own_row(std::size_t e, const core::TrajectoryView &query, - std::span state) const { - std::vector row; - row.reserve(m_list[e].size()); - for (const auto j : m_list[e]) - row.push_back(query::knn_distance(query, state[j], m_objective.workloads().frechet)); - return row; -} - -KnnState::Move KnnState::move(std::size_t i, const core::TrajectoryView &candidate, - std::span state) const { - if (!m_objective.config().compressed_knn_query) - return {i, distances(i, candidate), {}}; - Move result{i, {}, own_row(i, candidate, state)}; - result.watched.reserve(m_watchers[i].size()); - for (const auto &watcher : m_watchers[i]) - result.watched.push_back( - query::knn_distance(state[watcher.query], candidate, m_objective.workloads().frechet)); - return result; -} - -double KnnState::propose(const Move &move) const { - long change = 0; - for (std::size_t s = 0; s < m_watchers[move.trajectory].size(); ++s) { - const auto &watcher = m_watchers[move.trajectory][s]; - change += static_cast(hits(watcher.query, watcher.position, move.watched[s])) - - static_cast(m_hits[watcher.query]); - } - if (!move.own.empty() && !m_list[move.trajectory].empty()) - change += static_cast(hits(move.trajectory, move.own)) - - static_cast(m_hits[move.trajectory]); +double KnnState::value(long change) const { return m_objective.scale(Objective::Knn) * static_cast(change) / static_cast(m_objective.workloads().k * m_valid); } -void KnnState::apply(const Move &move) { - apply(move.trajectory, move.watched); - if (move.own.empty() || m_list[move.trajectory].empty()) - return; - const auto i = move.trajectory; - m_distance[i] = move.own; - m_hit_sum -= m_hits[i]; - refresh(i); - m_hit_sum += m_hits[i]; -} - -std::size_t KnnState::hits(std::size_t e, std::span row) const { - std::vector> order(m_list[e].size()); - for (std::size_t p = 0; p < order.size(); ++p) - order[p] = {row[p], static_cast(p)}; - return ranked_hits(e, order); -} - bool KnnState::ranks_before(std::size_t e, double distance, std::size_t position, std::size_t other) const { return query::knn_rank_less(distance, m_list[e][position], m_distance[e][other], @@ -988,26 +858,8 @@ void KnnState::refresh(std::size_t e) { [&](std::uint32_t p) { return p < members; })); } -std::size_t KnnState::ranked_hits(std::size_t e, - std::vector> &order) const { - const auto &list = m_list[e]; - const std::size_t k = std::min(m_objective.workloads().k, list.size()); - const std::size_t members = m_objective.workloads().knn[e].top.size(); - // Ties go to the lower database index, like query::knn - std::ranges::partial_sort( - order, order.begin() + static_cast(k), [&](const auto &a, const auto &b) { - return query::knn_rank_less(a.first, list[a.second], b.first, list[b.second]); - }); - std::size_t count = 0; - for (std::size_t p = 0; p < k; ++p) - count += order[p].second < members; - return count; -} - void KnnState::watch(std::span> versions, std::span state) { - if (m_objective.config().compressed_knn_query) - throw std::logic_error("knn state: watch needs original queries"); const auto &db = m_objective.db(); const auto frechet = m_objective.workloads().frechet; const std::size_t n = db.size(); @@ -1057,8 +909,6 @@ void KnnState::watch(std::span> versions } double KnnState::propose_bounded(std::size_t i, const core::TrajectoryView &candidate) const { - if (m_objective.config().compressed_knn_query) - throw std::logic_error("knn state: propose_bounded needs original queries"); const auto &db = m_objective.db(); const auto frechet = m_objective.workloads().frechet; const std::size_t k = m_objective.workloads().k; @@ -1090,8 +940,7 @@ double KnnState::propose_bounded(std::size_t i, const core::TrajectoryView &cand } change += static_cast(now) - static_cast(m_hits[e]); } - return m_objective.scale(Objective::Knn) * static_cast(change) / - static_cast(k * m_valid); + return value(change); } double KnnState::propose(std::size_t i, std::span distances) const { @@ -1101,8 +950,7 @@ double KnnState::propose(std::size_t i, std::span distances) const change += static_cast(hits(watcher.query, watcher.position, distances[s])) - static_cast(m_hits[watcher.query]); } - return m_objective.scale(Objective::Knn) * static_cast(change) / - static_cast(m_objective.workloads().k * m_valid); + return value(change); } void KnnState::apply(std::size_t i, std::span distances) { diff --git a/engine/src/teacher/teacher.cpp b/engine/src/teacher/teacher.cpp index 8c74e2f..74d42cc 100644 --- a/engine/src/teacher/teacher.cpp +++ b/engine/src/teacher/teacher.cpp @@ -19,7 +19,6 @@ #include #include #include -#include #include namespace teacher { @@ -42,7 +41,6 @@ constexpr std::array kCandidates{ "RDP", "SED", "UNIFORM", "GREEDY", "GREEDY_SED2", "GREEDY_RDP2"}; constexpr std::size_t kMaxResponsePasses = 10; constexpr std::size_t kKnnWatchFactor = 3; -constexpr std::size_t kMaxAllocationRounds = 10; constexpr std::array kTradeSteps{1, 2, 4}; // points moved per trade constexpr std::size_t kMaxTradeRounds = 3; constexpr double kMinImprovement = 1e-12; @@ -207,29 +205,6 @@ Pick pick(const Objective &objective, std::size_t i, return best; } -Methods independent_choice(const core::TrajectoryDb &db, const Objective &objective, - const Candidates &candidates, const KeptSets *partners) { - Methods method(db.size()); - std::optional prepared; - std::vector partner_views; - if (partners) { - prepared.emplace(db, *partners); - partner_views = prepared->views(); - } - const RangeState range(objective, partner_views); - core::parallel_for(db.size(), [&](std::size_t i) { - std::vector owned; - std::vector> options; - owned.reserve(kCandidates.size()); - for (std::size_t c = 0; c < kCandidates.size(); ++c) { - owned.emplace_back(db.trajectory(i), candidates[c][i]); - options.push_back({c, owned.back().view()}); - } - method[i] = pick(objective, i, options, partner_views, range.weights(i)).method; - }); - return method; -} - // Value curves up to caps[i], best candidate and RDP alone, partners fixed struct Curves { std::vector best, rdp; @@ -289,13 +264,12 @@ Methods curve_methods(const std::vector &curves, const std::vector{}, - watch); - std::vector> versions(db.size()); - for (std::size_t i = 0; i < db.size(); ++i) - for (std::size_t c = 0; c < kCandidates.size(); ++c) - if (m_representative[i][c] == c) - versions[i].push_back(m_views[c][i]); - m_knn->watch(versions, m_state); - } - } + // Lists from the start, then every query also watches any candidate within the distance of + // its watch-th nearest there + m_knn.emplace(objective, assemble(candidates, method), + kKnnWatchFactor * objective.workloads().k); + std::vector> versions(db.size()); + for (std::size_t i = 0; i < db.size(); ++i) + for (std::size_t c = 0; c < kCandidates.size(); ++c) + if (m_representative[i][c] == c) + versions[i].push_back(m_views[c][i]); + m_knn->watch(versions, m_state); const double knn_s = core::seconds_since(start); start = std::chrono::steady_clock::now(); const auto active = [&](std::size_t i) { return !objective.boxes(i).empty() || !objective.links(i).empty() || - (m_knn && m_knn->active(i)); + m_knn->watched(i) || objective.scale(Objective::Shape) > 0 || + objective.scale(Objective::Margin) > 0; }; for (std::size_t i = 0; i < db.size(); ++i) if (active(i)) m_order.push_back(i); - m_separable.resize(db.size()); - m_knn_distances.resize(db.size()); - core::parallel_for(db.size(), [&](std::size_t i) { - if (!active(i)) - return; - for (std::size_t c = 0; c < kCandidates.size(); ++c) { - const auto first = m_representative[i][c]; - if (first == c) { - m_separable[i][c] = objective.separable(i, m_views[c][i]); - // Compressed queries depend on the partners' state, so are computed later - if (m_knn && !m_compressed_queries) - m_knn_distances[i][c] = m_knn->distances(i, m_views[c][i]); - } else { - m_separable[i][c] = m_separable[i][first]; - } - } - }); std::cout << std::format("teacher: loop setup: prepare {:.1f} s\n", prepare_s); m_join.emplace(objective, m_state); m_range.emplace(objective, m_state); - // Box coverage and held-out join outcomes do not depend on the others, so are tested once + // Every candidate's separable value, kNN distances, box coverage and join outcomes; none + // depend on the others + m_separable.resize(db.size()); + m_knn_distances.resize(db.size()); m_coverage.resize(db.size()); - m_held_out.resize(db.size()); + m_outcomes.resize(db.size()); core::parallel_for(db.size(), [&](std::size_t i) { if (!active(i)) return; - const auto links = objective.links(i); for (std::size_t c = 0; c < kCandidates.size(); ++c) { - if (m_representative[i][c] != c) + const auto first = m_representative[i][c]; + m_separable[i][c] = + first == c ? objective.separable(i, m_views[c][i]) : m_separable[i][first]; + if (first != c) continue; + m_knn_distances[i][c] = m_knn->distances(i, m_views[c][i]); m_coverage[i][c] = m_range->coverage(i, m_views[c][i]); - auto &known = m_held_out[i][c]; - known.assign(links.size(), 0); - double own_error = -1.0; - for (std::size_t k = 0; k < links.size(); ++k) - if (objective.held_out(links[k].term)) - known[k] = objective.held_out_match(i, links[k], m_views[c][i], own_error); + m_outcomes[i][c] = m_join->outcomes(i, m_views[c][i]); } }); std::cout << std::format("teacher: loop setup: kNN lists {:.1f} s, candidates {:.1f} s\n", @@ -394,26 +346,23 @@ class Search { Search(const Search &) = delete; Search &operator=(const Search &) = delete; - // update = false: one pass against the starting state - Methods run(std::uint64_t seed, bool update = true) const { + // Once: the trackers are updated in place + Methods run(std::uint64_t seed) { auto method = m_start; - auto state = m_state; - auto join = *m_join; - auto range = *m_range; - auto knn = m_knn; - auto order = m_order; + auto &join = *m_join; + auto &range = *m_range; + auto &knn = *m_knn; std::mt19937_64 rng(seed + kOrderSeedOffset); for (std::size_t pass = 1; pass <= kMaxResponsePasses; ++pass) { const auto start = std::chrono::steady_clock::now(); - std::ranges::shuffle(order, rng); + std::ranges::shuffle(m_order, rng); std::size_t switches = 0; - for (const auto i : order) { + for (const auto i : m_order) { const double here = m_separable[i][method[i]]; std::size_t best = method[i]; JoinState::Proposal best_proposal; RangeState::Proposal best_range; - KnnState::Move best_move; double best_gain = kMinImprovement; // strict gains only, so ties cannot cycle for (std::size_t c = 0; c < kCandidates.size(); ++c) { // Each distinct kept set is tried once, and never the one i already has @@ -421,46 +370,31 @@ class Search { const bool current = m_representative[i][c] == m_representative[i][method[i]]; if (duplicate || current) continue; - auto proposal = - join.propose(i, join.outcomes(i, m_views[c][i], state, m_held_out[i][c])); + auto proposal = join.propose(i, m_outcomes[i][c]); auto range_proposal = range.propose(i, m_coverage[i][c]); - KnnState::Move move; - double knn_gain = 0.0; - if (knn && m_compressed_queries) { - move = knn->move(i, m_views[c][i], state); - knn_gain = knn->propose(move); - } else if (knn) { - knn_gain = knn->propose(i, m_knn_distances[i][c]); - } - const double gain = - m_separable[i][c] - here + proposal.gain + range_proposal.gain + knn_gain; + const double gain = m_separable[i][c] - here + proposal.gain + + range_proposal.gain + knn.propose(i, m_knn_distances[i][c]); if (gain > best_gain) { best = c; best_gain = gain; best_proposal = std::move(proposal); best_range = std::move(range_proposal); - best_move = std::move(move); } } if (best == method[i]) continue; method[i] = best; ++switches; - if (!update) - continue; - state[i] = m_views[best][i]; + m_state[i] = m_views[best][i]; join.apply(best_proposal); range.apply(best_range); - if (knn && m_compressed_queries) - knn->apply(best_move); - else if (knn) - knn->apply(i, m_knn_distances[i][best]); + knn.apply(i, m_knn_distances[i][best]); } - std::cout << std::format( - "teacher: pass {}: {} switches, join F1 {:.4f}{}, {:.1f} s\n", pass, switches, - join.metric(), knn ? std::format(", knn {:.4f}", knn->metric()) : "", - core::seconds_since(start)); - if (switches == 0 || !update) + std::cout << std::format("teacher: pass {}: {} switches, join F1 {:.4f}, knn {:.4f}, " + "{:.1f} s\n", + pass, switches, join.metric(), knn.metric(), + core::seconds_since(start)); + if (switches == 0) break; } return method; @@ -468,28 +402,21 @@ class Search { private: Methods m_start; - bool m_compressed_queries; std::vector m_order; std::vector> m_prepared; std::array, kCandidates.size()> m_views; std::vector> m_separable; std::vector> m_representative; std::vector, kCandidates.size()>> m_knn_distances; - std::vector, kCandidates.size()>> m_coverage, m_held_out; + std::vector, kCandidates.size()>> m_coverage, m_outcomes; std::optional m_knn; std::vector m_state; std::optional m_join; std::optional m_range; }; -Methods best_response(const core::TrajectoryDb &db, const Objective &objective, - const Candidates &candidates, Methods method, std::uint64_t seed, - bool update = true) { - return Search(db, objective, candidates, method).run(seed, update); -} - // The loop accepts switches on the tracked J_T, whose kNN term only watches each query's top k -// plus its nearest others in the start and single-recipe states. In a mixed state an unwatched +// plus its nearest others in the start state and the candidates. In a mixed state an unwatched // trajectory can enter a top k unseen, so the result is checked exactly against the start. // before is the start's exact J_T when the caller already has it struct Checked { @@ -521,8 +448,6 @@ struct Traded { Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, const std::vector &caps, KeptSets kept, double start_value) { const std::size_t n = db.size(); - const bool ranking = objective.config().ranking_knn; - const bool compressed = objective.config().compressed_knn_query; std::vector owned; owned.reserve(n); for (std::size_t i = 0; i < n; ++i) @@ -534,7 +459,8 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, RangeState range(objective, state); std::vector separable(n); core::parallel_for(n, [&](std::size_t i) { separable[i] = objective.separable(i, state[i]); }); - std::optional knn; + // Built once: the trades keep it current + KnnState knn(objective, kept, kKnnWatchFactor * objective.workloads().k); struct Pending { std::size_t i; @@ -543,36 +469,26 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, double separable = 0.0, gain = 0.0; JoinState::Proposal join; RangeState::Proposal range; - KnnState::Move move; std::vector distances; }; auto pending = [&](std::size_t i, std::vector next) { - Pending p{ - i, PreparedTrajectory(db.trajectory(i), next), std::move(next), 0.0, 0.0, {}, {}, {}, - {}}; + Pending p{i, PreparedTrajectory(db.trajectory(i), next), std::move(next), 0.0, 0.0, {}, {}, + {}}; const auto view = p.prepared.view(); p.separable = objective.separable(i, view); - p.join = join.propose(i, view, state); + p.join = join.propose(i, view); p.range = range.propose(i, view); p.gain = p.separable - separable[i] + p.join.gain + p.range.gain; - if (knn && compressed) { - p.move = knn->move(i, view, state); - p.gain += knn->propose(p.move); - } else if (knn) { - // One trade at a time, so the distances themselves go in parallel - p.distances = knn->distances(i, view, true); - p.gain += knn->propose(i, p.distances); - } + // One trade at a time, so the distances themselves go in parallel + p.distances = knn.distances(i, view, true); + p.gain += knn.propose(i, p.distances); return p; }; auto commit = [&](Pending &p) { const auto i = p.i; join.apply(p.join); range.apply(p.range); - if (knn && compressed) - knn->apply(p.move); - else if (knn) - knn->apply(i, p.distances); + knn.apply(i, p.distances); owned[i] = std::move(p.prepared); state[i] = owned[i].view(); kept[i] = std::move(p.kept); @@ -583,13 +499,9 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, double gain = -std::numeric_limits::infinity(); std::vector kept; }; - const std::size_t watch = kKnnWatchFactor * objective.workloads().k; const auto start_kept = kept; double value = start_value; std::cout << std::format("teacher: trading from J_T {:.6f}\n", value); - // Built once: the trades keep it current - if (ranking) - knn.emplace(objective, kept, std::span{}, watch); // Sequences depend only on the trajectory and its cap, so are built once for every round std::vector> makers(n); for (std::size_t round = 1; round <= kMaxTradeRounds; ++round) { @@ -617,11 +529,8 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, const PreparedTrajectory prepared(db.trajectory(i), candidate); const auto view = prepared.view(); double gain = objective.separable(i, view) - separable[i] + - join.propose(i, view, state).gain + range.propose(i, view).gain; - if (knn && compressed) - gain += knn->propose(knn->move(i, view, state)); - else if (knn) - gain += knn->propose_bounded(i, view); + join.propose(i, view).gain + range.propose(i, view).gain; + gain += knn.propose_bounded(i, view); if (gain > best.gain) best = {gain, std::move(candidate)}; } @@ -639,8 +548,8 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, // The trades below try only these states, so the kNN lists watch them. Either way the // lists can now see trajectories the old ones missed, which changes the tracked metric // of the current state; that change is carried into the tracked total below - const double before_watch = ranking ? knn->metric() : 0.0; - if (ranking && !compressed) { + const double before_watch = knn.metric(); + { std::vector> prepared(n); std::vector> versions(n); core::parallel_for(n, [&](std::size_t i) { @@ -651,20 +560,10 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, for (const auto &version : prepared[i]) versions[i].push_back(version.view()); }); - knn->watch(versions, state); - } else if (ranking) { - std::vector offered; - for (const auto *offers : {&buy, &sell}) - for (const auto &by_trajectory : *offers) { - auto &with = offered.emplace_back(kept); - for (std::size_t i = 0; i < n; ++i) - if (std::isfinite(by_trajectory[i].gain)) - with[i] = by_trajectory[i].kept; - } - knn.emplace(objective, kept, offered, watch); + knn.watch(versions, state); } const double knn_correction = - ranking ? objective.scale(Objective::Knn) * (knn->metric() - before_watch) : 0.0; + objective.scale(Objective::Knn) * (knn.metric() - before_watch); value += knn_correction; const double watch_s = core::seconds_since(start); start = std::chrono::steady_clock::now(); @@ -726,74 +625,14 @@ Traded trade_budgets(const core::TrajectoryDb &db, const Objective &objective, std::cout << std::format( "teacher: trading J_T tracked {:.6f}, exact {:.6f}; tracked/exact kNN {:.6f}/{:.6f}, " "range {:.6f}/{:.6f}, join {:.6f}/{:.6f}\n", - value, exact, knn ? knn->metric() : terms[Objective::Knn], terms[Objective::Knn], - range.metric(), terms[Objective::Range], join.metric(), terms[Objective::Join]); + value, exact, knn.metric(), terms[Objective::Knn], range.metric(), terms[Objective::Range], + join.metric(), terms[Objective::Join]); // The kNN tracker is not exact everywhere, so the trades could in principle lose if (exact < start_value) return {start_kept, start_value}; return {std::move(kept), exact}; } -// The workloads with trajectories relabelled among those sharing 1 km start and end cells, in -// every join pair and kNN neighbour list: wrong but plausible partners -TeacherWorkloads shuffled_partners(const core::TrajectoryDb &db, const TeacherWorkloads &workloads, - std::uint64_t seed) { - constexpr double kCell = 1000.0; - std::map, std::vector> buckets; - for (std::size_t i = 0; i < db.size(); ++i) { - const auto trajectory = db.trajectory(i); - auto cell = [&](double coordinate) { - return static_cast(std::floor(coordinate / kCell)); - }; - buckets[{cell(trajectory.x.front()), cell(trajectory.y.front()), cell(trajectory.x.back()), - cell(trajectory.y.back())}] - .push_back(static_cast(i)); - } - std::vector label(db.size()); - std::mt19937_64 rng(seed + kShuffleSeedOffset); - for (auto &[key, members] : buckets) { - auto shuffled = members; - std::ranges::shuffle(shuffled, rng); - for (std::size_t k = 0; k < members.size(); ++k) - label[members[k]] = shuffled[k]; - } - - auto result = workloads; - std::set> real; - for (const auto &pair : workloads.positives) - real.insert({pair.i, pair.j}); - std::size_t kept_real = 0; - std::vector *> lists{&result.positives, &result.negatives}; - for (auto &extra : result.extra_joins) { - lists.push_back(&extra.positives); - lists.push_back(&extra.negatives); - } - for (auto *pairs : lists) - for (auto &pair : *pairs) { - const auto first = label[pair.i], second = label[pair.j]; - pair.i = std::min(first, second); - pair.j = std::max(first, second); - kept_real += pairs == &result.positives && real.contains({pair.i, pair.j}); - } - std::size_t kept_neighbours = 0, neighbours = 0; - for (std::size_t e = 0; e < result.knn.size(); ++e) { - std::vector relabelled; - for (const auto j : result.knn[e].top) { - const auto to = label[j] == e ? j : label[j]; - if (std::ranges::find(relabelled, to) == relabelled.end()) - relabelled.push_back(to); - kept_neighbours += to == j; - ++neighbours; - } - result.knn[e].top = std::move(relabelled); - } - std::cout << std::format( - "teacher: shuffled in {} buckets, {} of {} positive pairs and {} of {} " - "kNN neighbours still real\n", - buckets.size(), kept_real, result.positives.size(), kept_neighbours, neighbours); - return result; -} - std::string wins(const Methods &method) { std::array count{}; for (const auto candidate : method) @@ -832,10 +671,11 @@ Teacher::Teacher(const core::TrajectoryDb &db, TeacherWorkloads workloads, m_seed(seed) { const auto &floors = m_objective.floors(); std::cout << std::format("teacher: {} join pairs ({} positive), floors knn {:.4f}, range " - "{:.4f}, where-at {:.4f}, join {:.4f}\n", + "{:.4f}, where-at {:.4f}, join {:.4f}, shape {:.4f}, margin {:.4f}\n", m_objective.pair_count(), m_objective.positive_count(), floors[Objective::Knn], floors[Objective::Range], - floors[Objective::WhereAt], floors[Objective::Join]); + floors[Objective::WhereAt], floors[Objective::Join], + floors[Objective::Shape], floors[Objective::Margin]); } std::vector Teacher::run(std::size_t total_budget, std::span arms) const { @@ -843,20 +683,14 @@ std::vector Teacher::run(std::size_t total_budget, std::span Teacher::run(std::size_t total_budget, std::span guess_value; - if (want_guess) - guess_value = keep("EQUAL_CHOICE_RDP_PARTNERS", equal_candidates, guess); - if (want_equal_loop) { - const auto looped = checked_loop( - objective, equal_candidates, guess, guess_value, - best_response(db, objective, equal_candidates, guess, m_seed), "EQUAL_LOOP"); - keep("EQUAL_LOOP", equal_candidates, looped.methods, looped.value); - timed("EQUAL_LOOP"); - } - } auto report = [&](std::string_view what, double value, const std::vector &budgets) { std::size_t at_cap = 0; @@ -929,87 +727,48 @@ std::vector Teacher::run(std::size_t total_budget, std::span choice_value; - if (want_budget_choice) - choice_value = keep("BUDGET_CHOICE", candidates, choice); - // The loop and its no-update control start from the same state and share the setup - std::optional search; - if (want_loop || want_reallocate || want_trade || want_no_update) - search.emplace(db, objective, candidates, choice); - if (want_loop || want_reallocate || want_trade) { - const auto checked = checked_loop(objective, candidates, choice, choice_value, - search->run(m_seed), "BUDGET_LOOP"); - const auto &looped = checked.methods; - timed("BUDGET_LOOP"); - if (want_loop) - keep("BUDGET_LOOP", candidates, looped, checked.value); - if (want_trade) { - auto traded = - trade_budgets(db, objective, caps, assemble(candidates, looped), checked.value); - outputs["BUDGET_LOOP_TRADE"] = {"BUDGET_LOOP_TRADE", - core::to_simplification(traded.kept), traded.value}; - timed("BUDGET_LOOP_TRADE"); - } - if (want_reallocate) { - // Each round is kept only if J_T rises - auto current = assemble(candidates, looped); - auto current_methods = looped; - double current_value = checked.value; - for (std::size_t round = 1; round <= kMaxAllocationRounds; ++round) { - const auto round_curves = build_curves(db, objective, current, caps, false); - double round_value = 0; - const auto round_budgets = - allocate(round_curves.best, total_budget, &round_value); - const auto next_candidates = make_candidates(objective, round_budgets, caps); - const auto round_start = curve_methods(round_curves.best, round_budgets); - auto round_loop = checked_loop( - objective, next_candidates, round_start, std::nullopt, - best_response(db, objective, next_candidates, round_start, m_seed + round), - std::format("reallocation round {}", round)); - auto next_methods = std::move(round_loop.methods); - auto next = assemble(next_candidates, next_methods); - const double next_value = round_loop.value; - std::cout << std::format( - "teacher: reallocation round {}, J_T {:.4f} -> {:.4f}\n", round, - current_value, next_value); - if (next_value <= current_value + kMinImprovement) - break; - current_value = next_value; - current = std::move(next); - current_methods = std::move(next_methods); - } - std::cout << std::format("teacher BUDGET_LOOP_REALLOCATE: {}, J_T {:.4f}\n", - wins(current_methods), current_value); - outputs["BUDGET_LOOP_REALLOCATE"] = { - "BUDGET_LOOP_REALLOCATE", core::to_simplification(current), current_value}; - timed("BUDGET_LOOP_REALLOCATE"); - } - } - if (want_no_update) { - keep("BUDGET_LOOP_NO_UPDATE", candidates, search->run(m_seed, false)); - timed("BUDGET_LOOP_NO_UPDATE"); - } - if (want_shuffled) { - // Searched on fake partners, scored on the real ones - const auto shuffled = shuffled_partners(db, m_workloads, m_seed); - const Objective fake(db, shuffled, objective.config()); - keep("BUDGET_LOOP_SHUFFLED", candidates, - best_response(db, fake, candidates, choice, m_seed)); - timed("BUDGET_LOOP_SHUFFLED"); - } + if (!needs_candidates) + return collect(outputs, arms); + + double value = 0; + const auto budgets = allocate(curves.best, total_budget, &value); + report("allocation", value, budgets); + const auto choice = curve_methods(curves.best, budgets); + const auto candidates = + make_candidates(objective, budgets, caps, needs_loop ? nullptr : &choice); + std::optional choice_value; + if (want_choice) + choice_value = keep("BUDGET_CHOICE", candidates, choice); + if (!needs_loop) + return collect(outputs, arms); + + const auto checked = + checked_loop(objective, candidates, choice, choice_value, + Search(db, objective, candidates, choice).run(m_seed), "BUDGET_LOOP"); + timed("BUDGET_LOOP"); + if (want_loop) + keep("BUDGET_LOOP", candidates, checked.methods, checked.value); + if (want_trade) { + auto traded = trade_budgets(db, objective, caps, assemble(candidates, checked.methods), + checked.value); + outputs["BUDGET_LOOP_TRADE"] = {"BUDGET_LOOP_TRADE", core::to_simplification(traded.kept), + traded.value}; + timed("BUDGET_LOOP_TRADE"); } return collect(outputs, arms); } diff --git a/engine/src/teacher/workloads.cpp b/engine/src/teacher/workloads.cpp index a468940..16e6ab4 100644 --- a/engine/src/teacher/workloads.cpp +++ b/engine/src/teacher/workloads.cpp @@ -20,6 +20,7 @@ namespace { // Bump when the layout or workload generation changes. V6 uses fixed-grid continuous kNN. constexpr std::uint64_t kMagicV6 = 0x5431'4341'4348'4536ull; +constexpr std::uint64_t kMagicRanked = 0x5431'5241'4e4b'4431ull; constexpr std::uint64_t kFnvOffsetBasis = 14695981039346656037ull; constexpr std::uint64_t kFnvPrime = 1099511628211ull; @@ -222,6 +223,69 @@ TeacherWorkloads load_or_build_workloads(const core::TrajectoryDb &db, const Wor return workloads; } +void add_ranked_neighbours(const core::TrajectoryDb &db, TeacherWorkloads &workloads, + std::size_t depth, std::uint64_t seed, + const std::filesystem::path &cache_dir) { + const std::size_t want = workloads.k + depth; + const auto db_fingerprint = fingerprint(db); + const auto file = cache_dir / std::format("teacher_ranked_seed{}_d{}.bin", seed, depth); + auto &ranked = workloads.ranked; + { + std::ifstream in(file, std::ios::binary); + if (in && get(in) == kMagicRanked && get(in) == seed && + get(in) == db_fingerprint && get(in) == want && + get(in) == static_cast(workloads.frechet)) { + ranked.resize(get(in)); + for (auto &list : ranked) { + list.resize(get(in)); + for (auto &neighbour : list) { + neighbour.index = get(in); + neighbour.distance = get(in); + } + } + if (in && ranked.size() == db.size()) { + std::cout << "teacher: ranked neighbours loaded from " << file.string() << '\n'; + return; + } + } + } + const auto start = std::chrono::steady_clock::now(); + ranked.assign(db.size(), {}); + core::parallel_for(db.size(), [&](std::size_t e) { + const auto query = db.trajectory(e); + auto neighbours = query::knn(db, query, want + 1, workloads.frechet); + std::erase(neighbours, e); + neighbours.resize(std::min(neighbours.size(), want)); + for (const auto j : neighbours) + ranked[e].push_back( + {static_cast(j), + query::frechet_distance(query, db.trajectory(j), workloads.frechet)}); + }); + std::cout << std::format("teacher: ranked neighbours {:.1f} s\n", core::seconds_since(start)); + if (file.has_parent_path()) + std::filesystem::create_directories(file.parent_path()); + const auto tmp = file.string() + ".tmp"; + { + std::ofstream out(tmp, std::ios::binary); + put(out, kMagicRanked); + put(out, seed); + put(out, db_fingerprint); + put(out, want); + put(out, static_cast(workloads.frechet)); + put(out, ranked.size()); + for (const auto &list : ranked) { + put(out, list.size()); + for (const auto &neighbour : list) { + put(out, neighbour.index); + put(out, neighbour.distance); + } + } + if (!out) + throw std::runtime_error("teacher ranked cache: write failed " + tmp); + } + std::filesystem::rename(tmp, file); +} + query::KnnWorkload self_knn_workload(const TeacherWorkloads &workloads, double p) { query::KnnWorkload result{.queries = core::TrajectoryDb({0}, {}, {}, {}, {}), .entries = {}, diff --git a/engine/tests/config_test.cpp b/engine/tests/config_test.cpp index 59e8bfc..7e1711d 100644 --- a/engine/tests/config_test.cpp +++ b/engine/tests/config_test.cpp @@ -1,4 +1,5 @@ #include "backend/engine/pipeline/config.hpp" +#include "backend/engine/teacher/objective.hpp" #include @@ -259,56 +260,37 @@ TEST(Config, TeacherIsOffByDefault) { "teacher_grid.json", R"({"data_path": "d", "join": {"score_eps_m": [100, 300]}})")); EXPECT_EQ(pipeline::teacher_workload_config(grid).extra_join_eps_m, (std::vector{100.0, 300.0})); - EXPECT_FALSE(pipeline::teacher_objective_config(config).held_out_term); - EXPECT_TRUE(pipeline::teacher_objective_config(config).ranking_knn); - // The teacher's join term follows the join it is scored on - EXPECT_FALSE(config.teacher.join); - EXPECT_EQ(pipeline::teacher_objective_config(config).join_mode, query::JoinMode::HeldOut); - const auto self = pipeline::load_config( - write_config("teacher_self.json", R"({"data_path": "d", "join": {"mode": "Self"}})")); - EXPECT_EQ(pipeline::teacher_objective_config(self).join_mode, query::JoinMode::Self); - // The kNN query is compressed in the in-database setting only, unless set - EXPECT_FALSE(pipeline::teacher_objective_config(config).compressed_knn_query); - EXPECT_TRUE(pipeline::teacher_objective_config(self).compressed_knn_query); - const auto original = pipeline::load_config(write_config( - "teacher_original.json", - R"({"data_path": "d", "join": {"mode": "Self"}, "teacher": {"knn_query": "Original"}})")); - EXPECT_FALSE(pipeline::teacher_objective_config(original).compressed_knn_query); - EXPECT_THROW(pipeline::load_config( - write_config("teacher_bad_query.json", - R"({"data_path": "d", "teacher": {"knn_query": "Sometimes"}})")), - std::invalid_argument); + EXPECT_EQ(pipeline::teacher_objective_config(config).term_weights, + (teacher::TermWeights{1.0, 1.0, 1.0, 1.0, 0.0, 0.0})); } TEST(Config, ReadsTheTeacherSection) { const auto config = pipeline::load_config(write_config("teacher.json", R"({ "data_path": "d", - "teacher": {"enabled": true, "cache_dir": "my_cache", "arms": ["EQUAL_CHOICE", "EQUAL_LOOP"], - "range_query_count": 1000, - "join": "Self", "extra_join_eps_m": [100, 300], "held_out_term": true, - "ranking_knn": false, "term_weights": [2, 1, 1, 0]} + "teacher": {"enabled": true, "cache_dir": "my_cache", "arms": ["BUDGET_RDP", "BUDGET_LOOP"], + "range_query_count": 1000, "extra_join_eps_m": [100, 300], + "term_weights": [2, 1, 1, 0]} })")); - EXPECT_EQ(pipeline::teacher_objective_config(config).join_mode, query::JoinMode::Self); - EXPECT_TRUE(pipeline::teacher_objective_config(config).held_out_term); - EXPECT_FALSE(pipeline::teacher_objective_config(config).ranking_knn); EXPECT_EQ(pipeline::teacher_objective_config(config).term_weights, - (std::array{2.0, 1.0, 1.0, 0.0})); + (teacher::TermWeights{2.0, 1.0, 1.0, 0.0, 0.0, 0.0})); EXPECT_EQ(pipeline::teacher_workload_config(config).extra_join_eps_m, (std::vector{100.0, 300.0})); EXPECT_TRUE(config.teacher.enabled); EXPECT_EQ(config.teacher.cache_dir, "my_cache"); - EXPECT_EQ(config.teacher.arms, (std::vector{"EQUAL_CHOICE", "EQUAL_LOOP"})); + EXPECT_EQ(config.teacher.arms, (std::vector{"BUDGET_RDP", "BUDGET_LOOP"})); EXPECT_EQ(config.teacher.range_query_count, 1000u); } TEST(Config, RejectsBadTeacherSections) { for (const auto §ion : {R"({"enabled": true, "arms": ["DP"]})", R"({"enabled": true, "arms": []})", - R"({"arms": ["EQUAL_CHOICE_RDP_PARTNERS", "EQUAL_CHOICE_RDP_PARTNERS"]})", + R"({"arms": ["BUDGET_LOOP", "BUDGET_LOOP"]})", R"({"arms": ["EQUAL_LOOP"]})", R"({"cache_dir": ""})", R"({"range_query_count": 0})", R"({"shape_weight": 0.25})", R"({"term_weights": [0, 0, 0, 0]})", R"({"term_weights": [-1, 1, 1, 1]})", - R"({"seed": 7})", R"({"join": "Both"})", R"({"extra_join_eps_m": [0]})", - R"({"extra_join_eps_m": [-100]})"}) { + R"({"term_weights": [1, 1, 1]})", R"({"term_weights": [1, 1, 1, 1, 1, 1, 1]})", + R"({"seed": 7})", R"({"join": "Self"})", R"({"held_out_term": true})", + R"({"ranking_knn": true})", R"({"knn_query": "Original"})", + R"({"extra_join_eps_m": [0]})", R"({"extra_join_eps_m": [-100]})"}) { SCOPED_TRACE(section); const auto contents = std::string(R"({"data_path": "d", "teacher": )") + section + "}"; EXPECT_THROW(pipeline::load_config(write_config("bad_teacher.json", contents)), diff --git a/engine/tests/teacher/teacher_test.cpp b/engine/tests/teacher/teacher_test.cpp index 59e7c97..83a1f9a 100644 --- a/engine/tests/teacher/teacher_test.cpp +++ b/engine/tests/teacher/teacher_test.cpp @@ -3,6 +3,7 @@ #include "backend/engine/core/projection.hpp" #include "backend/engine/core/trajectory_db.hpp" #include "backend/engine/evaluator/range.hpp" +#include "backend/engine/selection/rdp.hpp" #include "backend/engine/teacher/greedy.hpp" #include "backend/engine/teacher/objective.hpp" #include "backend/engine/teacher/teacher.hpp" @@ -219,9 +220,9 @@ TEST_F(TeacherTest, CacheForOtherSettingsIsRebuilt) { TEST_F(TeacherTest, ObjectiveIsZeroAtTheFloorAndTracksJoinExactly) { const auto db = random_db(2, 24, 20); const auto workloads = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); - for (const auto mode : {query::JoinMode::Self, query::JoinMode::HeldOut}) { - SCOPED_TRACE(mode == query::JoinMode::Self ? "Self" : "HeldOut"); - const teacher::Objective objective(db, workloads, {.join_mode = mode}); + { + // kNN off: its ranking term is tracked by KnnState, tested below + const teacher::Objective objective(db, workloads, {.term_weights = {0, 1, 1, 1}}); const auto floor = endpoints(db); EXPECT_NEAR(objective.total(floor), 0.0, 1e-12); @@ -245,10 +246,10 @@ TEST_F(TeacherTest, ObjectiveIsZeroAtTheFloorAndTracksJoinExactly) { const double before = objective.total(state); const auto next_view = state[i].size() == 2 ? full_views[i] : floor_views[i]; const double old_metric = join.metric(); - const auto proposal = join.propose(i, next_view, views); + const auto proposal = join.propose(i, next_view); const auto range_proposal = range.propose(i, next_view); // Proposing must not change the state or this proposal - const auto stay = join.propose(i, views[i], views); + const auto stay = join.propose(i, views[i]); EXPECT_DOUBLE_EQ(stay.gain, 0.0); EXPECT_DOUBLE_EQ(join.metric(), old_metric); const double predicted = proposal.gain; @@ -369,12 +370,10 @@ TEST_F(TeacherTest, SeveralJoinTermsAreTrackedExactly) { EXPECT_LE(workloads.extra_joins[0].positives.size(), workloads.positives.size()); EXPECT_GE(workloads.extra_joins[1].positives.size(), workloads.positives.size()); - const teacher::Objective single(db, workloads, {.join_mode = query::JoinMode::Self}); - for (const bool held_out_term : {false, true}) { - SCOPED_TRACE(held_out_term ? "with held-out term" : "self terms"); - const teacher::Objective objective( - db, workloads, {.join_mode = query::JoinMode::Self, .held_out_term = held_out_term}); - EXPECT_EQ(objective.slots(), db.size() * (held_out_term ? 4u : 3u)); + const teacher::Objective single(db, workloads); + { + const teacher::Objective objective(db, workloads, {.term_weights = {0, 1, 1, 1}}); + EXPECT_EQ(objective.slots(), db.size() * 3u); EXPECT_NEAR(objective.total(endpoints(db)), 0.0, 1e-12); EXPECT_DOUBLE_EQ(objective.metrics(every_point(db))[teacher::Objective::Join], 1.0); @@ -392,7 +391,7 @@ TEST_F(TeacherTest, SeveralJoinTermsAreTrackedExactly) { const auto &next = grow ? full[i] : floor[i]; const auto next_view = grow ? full_views[i] : floor_views[i]; const double before = objective.total(state); - const auto proposal = join.propose(i, next_view, views); + const auto proposal = join.propose(i, next_view); const auto range_proposal = range.propose(i, next_view); const double separable = objective.separable(i, next) - objective.separable(i, state[i]); @@ -408,7 +407,7 @@ TEST_F(TeacherTest, SeveralJoinTermsAreTrackedExactly) { // Without extra terms the workloads' extra joins are the only difference teacher::TeacherWorkloads plain = workloads; plain.extra_joins.clear(); - const teacher::Objective base(db, plain, {.join_mode = query::JoinMode::Self}); + const teacher::Objective base(db, plain); EXPECT_EQ(base.slots(), db.size()); EXPECT_GT(single.pair_count(), base.pair_count()); } @@ -421,7 +420,7 @@ TEST_F(TeacherTest, JoinTermsAreNormalisedApartAndAveragedLikeJ4) { ASSERT_FALSE(workloads.positives.empty()); for (const auto &extra : workloads.extra_joins) ASSERT_FALSE(extra.positives.empty()); - const teacher::Objective objective(db, workloads, {.join_mode = query::JoinMode::Self}); + const teacher::Objective objective(db, workloads); ASSERT_EQ(objective.join_terms(), 3u); auto macro_f1 = [&](const teacher::KeptSets &kept, std::size_t term) { @@ -434,7 +433,8 @@ TEST_F(TeacherTest, JoinTermsAreNormalisedApartAndAveragedLikeJ4) { for (const auto &link : objective.links(i)) { if (link.term != term) continue; - const bool matched = objective.match(views[i], views[link.other], term); + // i is the original query + const bool matched = objective.match(db.trajectory(i), views[link.other], term); found += matched; hits += matched && link.positive; } @@ -470,16 +470,14 @@ TEST_F(TeacherTest, RankingKnnIsTrackedExactly) { // Discrete distances order exactly, without the continuous ranking's 1 mm ties config.frechet = query::FrechetVariant::Discrete; const auto workloads = teacher::load_or_build_workloads(db, config, 9, m_dir); - const teacher::Objective objective(db, workloads, - {.join_mode = query::JoinMode::Self, .ranking_knn = true}); + const teacher::Objective objective(db, workloads); const auto floor = endpoints(db), full = every_point(db); EXPECT_NEAR(objective.total(floor), 0.0, 1e-12); EXPECT_DOUBLE_EQ(objective.metrics(full)[teacher::Objective::Knn], 1.0); // Every query watches every trajectory, so the tracked value must equal a full recompute auto state = floor; - const std::vector watch{full}; - teacher::KnnState knn(objective, state, watch, db.size()); + teacher::KnnState knn(objective, state, db.size()); EXPECT_NEAR(knn.metric(), objective.metrics(state)[teacher::Objective::Knn], 1e-12); const teacher::PreparedTrajectories prepared_floor(db, floor), prepared_full(db, full); const auto floor_views = prepared_floor.views(), full_views = prepared_full.views(); @@ -496,7 +494,7 @@ TEST_F(TeacherTest, RankingKnnIsTrackedExactly) { const auto distances = knn.distances(i, next_view); const double knn_gain = knn.propose(i, distances); EXPECT_DOUBLE_EQ(knn.propose(i, knn.distances(i, views[i])), 0.0); - const auto proposal = join.propose(i, next_view, views); + const auto proposal = join.propose(i, next_view); const auto range_proposal = range.propose(i, next_view); const double separable = objective.separable(i, next) - objective.separable(i, state[i]); state[i] = next; @@ -510,60 +508,6 @@ TEST_F(TeacherTest, RankingKnnIsTrackedExactly) { } } -TEST_F(TeacherTest, CompressedKnnQueriesAreTrackedExactly) { - const auto db = random_db(2, 24, 20); - auto config = small_config(); - config.frechet = query::FrechetVariant::Discrete; - const auto workloads = teacher::load_or_build_workloads(db, config, 9, m_dir); - const teacher::Objective objective( - db, workloads, - {.join_mode = query::JoinMode::Self, .ranking_knn = true, .compressed_knn_query = true}); - const auto floor = endpoints(db), full = every_point(db); - EXPECT_NEAR(objective.total(floor), 0.0, 1e-12); - EXPECT_DOUBLE_EQ(objective.metrics(full)[teacher::Objective::Knn], 1.0); - - // A switch changes i as a neighbour and as a query, both against the others' current state - auto state = floor; - const std::vector watch{full}; - teacher::KnnState knn(objective, state, watch, db.size()); - EXPECT_NEAR(knn.metric(), objective.metrics(state)[teacher::Objective::Knn], 1e-12); - const teacher::PreparedTrajectories prepared_floor(db, floor), prepared_full(db, full); - const auto floor_views = prepared_floor.views(), full_views = prepared_full.views(); - auto views = floor_views; - teacher::JoinState join(objective, views); - teacher::RangeState range(objective, views); - std::mt19937_64 rng(5); - for (int step = 0; step < 40; ++step) { - const std::size_t i = rng() % db.size(); - const bool grow = state[i].size() == 2; - const auto &next = grow ? full[i] : floor[i]; - const auto next_view = grow ? full_views[i] : floor_views[i]; - const double before = objective.total(state); - const auto move = knn.move(i, next_view, views); - const double knn_gain = knn.propose(move); - EXPECT_DOUBLE_EQ(knn.propose(knn.move(i, views[i], views)), 0.0); - const auto proposal = join.propose(i, next_view, views); - const auto range_proposal = range.propose(i, next_view); - const double separable = objective.separable(i, next) - objective.separable(i, state[i]); - state[i] = next; - views[i] = next_view; - join.apply(proposal); - range.apply(range_proposal); - knn.apply(move); - EXPECT_NEAR(knn.metric(), objective.metrics(state)[teacher::Objective::Knn], 1e-12); - EXPECT_NEAR(objective.total(state) - before, - knn_gain + proposal.gain + range_proposal.gain + separable, 1e-9); - } -} - -TEST_F(TeacherTest, CompressedKnnQueriesNeedTheRankingTerm) { - const auto db = random_db(2, 24, 20); - const auto workloads = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); - EXPECT_THROW( - teacher::Objective(db, workloads, {.ranking_knn = false, .compressed_knn_query = true}), - std::invalid_argument); -} - TEST_F(TeacherTest, ContinuousKnnNearTiesMatchFullScoringThroughSwitches) { // The first trajectory is a line; the next two backtrack by almost the same amount. // The old evaluator tied overlapping distance intervals, while the tracker sorted their @@ -596,63 +540,32 @@ TEST_F(TeacherTest, ContinuousKnnNearTiesMatchFullScoringThroughSwitches) { const auto full = every_point(db), floor = endpoints(db); const teacher::PreparedTrajectories prepared_full(db, full), prepared_floor(db, floor); const auto full_views = prepared_full.views(), floor_views = prepared_floor.views(); - for (const bool compressed : {false, true}) { - SCOPED_TRACE(compressed ? "compressed queries" : "original queries"); - const teacher::Objective objective(db, workloads, - {.join_mode = query::JoinMode::HeldOut, - .ranking_knn = true, - .compressed_knn_query = compressed}); - auto kept = full; - auto views = full_views; - teacher::KnnState knn(objective, kept, {}, db.size()); - EXPECT_DOUBLE_EQ(objective.knn_recall(core::materialize(db, kept)), 1.0); - EXPECT_DOUBLE_EQ(knn.metric(), 1.0); - for (const auto i : {1u, 2u, 0u, 5u, 1u, 0u, 2u, 5u}) { - const bool shrink = kept[i].size() != 2; - const auto candidate = shrink ? floor_views[i] : full_views[i]; - const double before = objective.knn_recall(core::materialize(db, kept)); - const auto move = knn.move(i, candidate, views); - const double gain = knn.propose(move); - if (!compressed) - EXPECT_NEAR(knn.propose_bounded(i, candidate), gain, 1e-12); - kept[i] = shrink ? floor[i] : full[i]; - views[i] = candidate; - knn.apply(move); - const double after = objective.knn_recall(core::materialize(db, kept)); - EXPECT_NEAR(knn.metric(), after, 1e-12); - EXPECT_NEAR(gain, objective.scale(teacher::Objective::Knn) * (after - before), 1e-12); - } - } -} - -TEST_F(TeacherTest, CompressedKnnArmsKeepTheBudgetAndLoopOnlyImproves) { - const auto db = random_db(3, 30, 25); - auto workloads = teacher::load_or_build_workloads(db, small_config(), 11, m_dir); - const teacher::Teacher model( - db, std::move(workloads), - {.join_mode = query::JoinMode::Self, .ranking_knn = true, .compressed_knn_query = true}, - 11); - const std::size_t budget = 2 * db.size() + 40; - const std::vector arms{"EQUAL_CHOICE", "EQUAL_CHOICE_RDP_PARTNERS", - "EQUAL_LOOP", "BUDGET_RDP", - "BUDGET_CHOICE", "BUDGET_LOOP"}; - const auto result = model.run(budget, arms); - for (const auto &arm : result) { - SCOPED_TRACE(arm.name); - EXPECT_EQ(arm.simplification.num_points(), budget); + const teacher::Objective objective(db, workloads); + auto kept = full; + teacher::KnnState knn(objective, kept, db.size()); + EXPECT_DOUBLE_EQ(objective.knn_recall(core::materialize(db, kept)), 1.0); + EXPECT_DOUBLE_EQ(knn.metric(), 1.0); + for (const auto i : {1u, 2u, 0u, 5u, 1u, 0u, 2u, 5u}) { + const bool shrink = kept[i].size() != 2; + const auto candidate = shrink ? floor_views[i] : full_views[i]; + const double before = objective.knn_recall(core::materialize(db, kept)); + const auto distances = knn.distances(i, candidate); + const double gain = knn.propose(i, distances); + EXPECT_NEAR(knn.propose_bounded(i, candidate), gain, 1e-12); + kept[i] = shrink ? floor[i] : full[i]; + knn.apply(i, distances); + const double after = objective.knn_recall(core::materialize(db, kept)); + EXPECT_NEAR(knn.metric(), after, 1e-12); + EXPECT_NEAR(gain, objective.scale(teacher::Objective::Knn) * (after - before), 1e-12); } - // Watch lists hold every trajectory here, so the loop's tracked gains are exact - EXPECT_GE(result[2].objective, result[1].objective - 1e-12); - EXPECT_GE(result[5].objective, result[4].objective - 1e-12); } TEST_F(TeacherTest, RankingKnnArmsKeepTheBudget) { const auto db = random_db(3, 30, 25); auto workloads = teacher::load_or_build_workloads(db, small_config(), 11, m_dir); - const teacher::Teacher model(db, std::move(workloads), - {.join_mode = query::JoinMode::HeldOut, .ranking_knn = true}, 11); + const teacher::Teacher model(db, std::move(workloads), {}, 11); const std::size_t budget = 2 * db.size() + 40; - const std::vector arms{"EQUAL_CHOICE", "EQUAL_LOOP", "BUDGET_RDP", "BUDGET_LOOP"}; + const std::vector arms{"BUDGET_RDP", "BUDGET_LOOP"}; for (const auto &arm : model.run(budget, arms)) { SCOPED_TRACE(arm.name); EXPECT_EQ(arm.simplification.num_points(), budget); @@ -682,6 +595,58 @@ TEST_F(TeacherTest, TermWeightsScaleTheTerms) { std::invalid_argument); } +TEST_F(TeacherTest, ShapeTermRampsTheDeviationBound) { + const auto db = random_db(2, 24, 20); + const auto workloads = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); + using T = teacher::Objective; + const teacher::Objective off(db, workloads); + EXPECT_DOUBLE_EQ(off.scale(T::Shape), 0.0); + const teacher::Objective shape(db, workloads, {.term_weights = {0, 0, 0, 0, 1}}); + EXPECT_NEAR(shape.total(every_point(db)), 1.0, 1e-9); + EXPECT_NEAR(shape.total(core::endpoints_only(db)), 0.0, 1e-9); + // Separable: J_T is the sum of each trajectory's gain over endpoints only + const auto kept = core::to_kept(selection::rdp::simplify_database(db, db.size() * 4)); + double sum = 0; + for (std::size_t i = 0; i < db.size(); ++i) + sum += shape.separable(i, kept[i]); + EXPECT_NEAR(shape.total(kept), sum, 1e-9); +} + +TEST_F(TeacherTest, TwoSidedMarginScoresOutsidersToo) { + const auto db = random_db(2, 24, 20); + auto workloads = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); + teacher::add_ranked_neighbours(db, workloads, 3, 9, m_dir); + for (std::size_t e = 0; e < db.size(); ++e) { + if (!workloads.knn[e].valid) + continue; + ASSERT_GT(workloads.ranked[e].size(), workloads.k); + for (std::size_t r = 0; r < workloads.k; ++r) + EXPECT_EQ(workloads.ranked[e][r].index, workloads.knn[e].top[r]); + } + // A second call reads the cache + auto again = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); + teacher::add_ranked_neighbours(db, again, 3, 9, m_dir); + EXPECT_EQ(again.ranked.size(), workloads.ranked.size()); + using T = teacher::Objective; + const teacher::Objective margin(db, workloads, {.term_weights = {0, 0, 0, 0, 0, 1}}); + EXPECT_NEAR(margin.total(every_point(db)), 1.0, 1e-9); + EXPECT_NEAR(margin.total(core::endpoints_only(db)), 0.0, 1e-9); + // Every surviving chance is at least a half + EXPECT_GE(margin.floors()[T::Margin], 0.5 - 1e-12); + const auto kept = core::to_kept(selection::rdp::simplify_database(db, db.size() * 4)); + double sum = 0; + for (std::size_t i = 0; i < db.size(); ++i) + sum += margin.separable(i, kept[i]); + EXPECT_NEAR(margin.total(kept), sum, 1e-9); + const double rdp = margin.total(kept); + EXPECT_GE(rdp, 0.0); + EXPECT_LE(rdp, 1.0 + 1e-9); + // Without ranked neighbours the term has no margins + const auto plain = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); + EXPECT_THROW(teacher::Objective(db, plain, {.term_weights = {0, 0, 0, 0, 0, 1}}), + std::invalid_argument); +} + TEST_F(TeacherTest, ExtraJoinThresholdsMustBeDistinctAndPositive) { const auto db = random_db(2, 24, 20); for (const auto &extra : @@ -696,7 +661,7 @@ TEST_F(TeacherTest, ExtraJoinThresholdsMustBeDistinctAndPositive) { TEST_F(TeacherTest, HeldOutJoinScoresOriginalQueriesAgainstCompressedPartners) { const auto db = random_db(2, 24, 20); const auto workloads = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); - const teacher::Objective objective(db, workloads, {.join_mode = query::JoinMode::HeldOut}); + const teacher::Objective objective(db, workloads); // Every point kept is the original database: each query finds exactly its positives EXPECT_DOUBLE_EQ(objective.metrics(every_point(db))[teacher::Objective::Join], 1.0); @@ -735,11 +700,12 @@ TEST_F(TeacherTest, PreparedIndependentScoresMatchFullScoringAndSurvivePruning) const auto db = random_db(2, 24, 20); const auto workloads = teacher::load_or_build_workloads(db, small_config(), 9, m_dir); std::mt19937_64 rng(74); - for (const bool held_out : {false, true}) { + std::size_t valid = 0; + for (const auto &entry : workloads.knn) + valid += entry.valid; + { { - const teacher::Objective objective( - db, workloads, - {.join_mode = held_out ? query::JoinMode::HeldOut : query::JoinMode::Self}); + const teacher::Objective objective(db, workloads); std::size_t with_truth = 0; for (std::size_t i = 0; i < db.size(); ++i) with_truth += objective.truth(i) != 0; @@ -762,15 +728,21 @@ TEST_F(TeacherTest, PreparedIndependentScoresMatchFullScoringAndSurvivePruning) for (std::size_t i = 0; i < db.size(); ++i) { std::size_t hits = 0, found = 0; for (const auto &link : objective.links(i)) { - const auto other = - original_partners ? db.trajectory(link.other) : views[link.other]; - // Held out, i is the database side of an original partner's query + // i is the database side of an original partner's query const bool matched = - held_out ? objective.match(db.trajectory(link.other), views[i]) - : objective.match(views[i], other); + objective.match(db.trajectory(link.other), views[i]); found += matched; hits += matched && link.positive; } + // Per-trajectory kNN: each query with i in its top k, i within its cutoff + double knn = 0.0; + for (const auto &entry : workloads.knn) + if (entry.valid && std::ranges::find(entry.top, i) != entry.top.end() && + query::frechet_within(db.trajectory(static_cast( + &entry - workloads.knn.data())), + views[i], entry.cutoff, workloads.frechet)) + knn += objective.scale(teacher::Objective::Knn) / + static_cast(workloads.k * valid); const double f1 = found + objective.truth(i) ? 2.0 * static_cast(hits) / static_cast(found + objective.truth(i)) @@ -788,7 +760,7 @@ TEST_F(TeacherTest, PreparedIndependentScoresMatchFullScoringAndSurvivePruning) workloads.range.queries[boxes[b].box].box)) range_gain += weights[b]; const double expected = - objective.separable(i, views[i]) + + objective.separable(i, views[i]) + knn + objective.scale(teacher::Objective::Range) * range_gain + join; const auto score = objective.prepare_independent(i, views[i], partners); EXPECT_GE(score.bound + 1e-12, expected); @@ -809,54 +781,35 @@ TEST_F(TeacherTest, PreparedIndependentScoresMatchFullScoringAndSurvivePruning) TEST_F(TeacherTest, ArmsKeepTheBudgetAndBestResponseOnlyImproves) { const auto db = random_db(3, 30, 25); - for (const auto mode : {query::JoinMode::Self, query::JoinMode::HeldOut}) - for (const bool ranking : {false, true}) { - SCOPED_TRACE(std::string(mode == query::JoinMode::Self ? "Self" : "HeldOut") + - (ranking ? ", ranking kNN" : "")); - auto workloads = teacher::load_or_build_workloads(db, small_config(), 11, m_dir); - const teacher::Teacher model( - db, std::move(workloads), - {.join_mode = mode, - .ranking_knn = ranking, - .compressed_knn_query = ranking && mode == query::JoinMode::Self}, - 11); - const std::vector arms(teacher::kArms.begin(), teacher::kArms.end()); - const std::size_t budget = 2 * db.size() + 40; - const auto result = model.run(budget, arms); + auto workloads = teacher::load_or_build_workloads(db, small_config(), 11, m_dir); + const teacher::Teacher model(db, std::move(workloads), {}, 11); + const std::vector arms(teacher::kArms.begin(), teacher::kArms.end()); + const std::size_t budget = 2 * db.size() + 40; + const auto result = model.run(budget, arms); - ASSERT_EQ(result.size(), arms.size()); - double rdp_partners_jt = 0, equal_loop_jt = 0, budget_choice_jt = 0, budget_loop_jt = 0, - reallocate_jt = 0, trade_jt = 0; - for (const auto &arm : result) { - SCOPED_TRACE(arm.name); - ASSERT_EQ(arm.simplification.size(), db.size()); - EXPECT_EQ(arm.simplification.num_points(), budget); - for (std::size_t i = 0; i < db.size(); ++i) { - const auto kept = arm.simplification.trajectory(i); - ASSERT_GE(kept.size(), 2u); - EXPECT_EQ(kept.front(), 0u); - EXPECT_EQ(kept.back(), db.trajectory(i).size() - 1); - EXPECT_TRUE(std::ranges::is_sorted(kept)); - EXPECT_EQ(std::ranges::adjacent_find(kept), kept.end()); - } - if (arm.name == "EQUAL_CHOICE_RDP_PARTNERS") - rdp_partners_jt = arm.objective; - if (arm.name == "EQUAL_LOOP") - equal_loop_jt = arm.objective; - if (arm.name == "BUDGET_CHOICE") - budget_choice_jt = arm.objective; - if (arm.name == "BUDGET_LOOP") - budget_loop_jt = arm.objective; - if (arm.name == "BUDGET_LOOP_REALLOCATE") - reallocate_jt = arm.objective; - if (arm.name == "BUDGET_LOOP_TRADE") - trade_jt = arm.objective; - } - EXPECT_GE(equal_loop_jt, rdp_partners_jt - 1e-12); - EXPECT_GE(budget_loop_jt, budget_choice_jt - 1e-12); - EXPECT_GE(reallocate_jt, budget_loop_jt - 1e-12); - EXPECT_GE(trade_jt, budget_loop_jt - 1e-12); + ASSERT_EQ(result.size(), arms.size()); + double choice_jt = 0, loop_jt = 0, trade_jt = 0; + for (const auto &arm : result) { + SCOPED_TRACE(arm.name); + ASSERT_EQ(arm.simplification.size(), db.size()); + EXPECT_EQ(arm.simplification.num_points(), budget); + for (std::size_t i = 0; i < db.size(); ++i) { + const auto kept = arm.simplification.trajectory(i); + ASSERT_GE(kept.size(), 2u); + EXPECT_EQ(kept.front(), 0u); + EXPECT_EQ(kept.back(), db.trajectory(i).size() - 1); + EXPECT_TRUE(std::ranges::is_sorted(kept)); + EXPECT_EQ(std::ranges::adjacent_find(kept), kept.end()); } + if (arm.name == "BUDGET_CHOICE") + choice_jt = arm.objective; + if (arm.name == "BUDGET_LOOP") + loop_jt = arm.objective; + if (arm.name == "BUDGET_LOOP_TRADE") + trade_jt = arm.objective; + } + EXPECT_GE(loop_jt, choice_jt - 1e-12); + EXPECT_GE(trade_jt, loop_jt - 1e-12); } // With 200 trajectories and k = 3 the kNN watch lists cover a small part of the database, so the @@ -864,16 +817,13 @@ TEST_F(TeacherTest, ArmsKeepTheBudgetAndBestResponseOnlyImproves) { TEST_F(TeacherTest, LoopsNeverEndBelowTheirStartOnALargeDatabase) { const auto db = random_db(5, 200, 14); auto workloads = teacher::load_or_build_workloads(db, small_config(), 13, m_dir); - const teacher::Teacher model(db, std::move(workloads), - {.join_mode = query::JoinMode::HeldOut, .ranking_knn = true}, 13); - const std::vector arms{"EQUAL_CHOICE_RDP_PARTNERS", "EQUAL_LOOP", "BUDGET_CHOICE", - "BUDGET_LOOP"}; + const teacher::Teacher model(db, std::move(workloads), {}, 13); + const std::vector arms{"BUDGET_CHOICE", "BUDGET_LOOP"}; for (const std::size_t extra : {100u, 400u}) { SCOPED_TRACE(extra); const auto result = model.run(2 * db.size() + extra, arms); ASSERT_EQ(result.size(), arms.size()); EXPECT_GE(result[1].objective, result[0].objective - 1e-12); - EXPECT_GE(result[3].objective, result[2].objective - 1e-12); } } @@ -939,7 +889,7 @@ TEST_F(TeacherTest, BoundedKnnProposalsMatchFullDistances) { config.extra_join_eps_m.clear(); config.k = 2; const auto workloads = teacher::load_or_build_workloads(db, config, 3, m_dir); - const teacher::Objective objective(db, workloads, {.ranking_knn = true}); + const teacher::Objective objective(db, workloads); std::mt19937_64 rng(8); auto random_kept = [&](std::size_t i) { const std::size_t n = db.trajectory(i).size(); @@ -955,7 +905,7 @@ TEST_F(TeacherTest, BoundedKnnProposalsMatchFullDistances) { if (!start_full) for (std::size_t i = 0; i < db.size(); ++i) state[i] = random_kept(i); - const teacher::KnnState knn(objective, state, {}, db.size()); + const teacher::KnnState knn(objective, state, db.size()); std::size_t changed = 0; for (std::size_t i = 0; i < db.size(); ++i) for (int trial = 0; trial < 8; ++trial) { @@ -974,7 +924,7 @@ TEST_F(TeacherTest, WatchingVersionsMakesKnnProposalsExact) { auto config = small_config(); config.extra_join_eps_m.clear(); const auto workloads = teacher::load_or_build_workloads(db, config, 3, m_dir); - const teacher::Objective objective(db, workloads, {.ranking_knn = true}); + const teacher::Objective objective(db, workloads); std::mt19937_64 rng(9); auto random_kept = [&](std::size_t i, std::size_t every) { const std::size_t n = db.trajectory(i).size(); @@ -996,7 +946,7 @@ TEST_F(TeacherTest, WatchingVersionsMakesKnnProposalsExact) { for (std::size_t i = 0; i < db.size(); ++i) { // One spare place for whoever replaces a leaver, so versions entering a top k are the only // ones unwatched until watch() - teacher::KnnState knn(objective, state, {}, workloads.k + 1); + teacher::KnnState knn(objective, state, workloads.k + 1); const auto kept = random_kept(i, 1); const teacher::PreparedTrajectory version(db.trajectory(i), kept); std::vector> versions(db.size()); @@ -1080,7 +1030,7 @@ TEST_F(TeacherTest, SingleRecipesKeepTheBudget) { } TEST(Teacher, RejectsUnknownArms) { - EXPECT_TRUE(teacher::is_arm("EQUAL_CHOICE_RDP_PARTNERS")); + EXPECT_FALSE(teacher::is_arm("EQUAL_CHOICE_RDP_PARTNERS")); EXPECT_FALSE(teacher::is_arm("gate_a0")); EXPECT_TRUE(teacher::is_arm("BUDGET_CHOICE")); EXPECT_FALSE(teacher::is_arm("TEACHER_A0")); @@ -1088,14 +1038,13 @@ TEST(Teacher, RejectsUnknownArms) { TEST_F(TeacherTest, IndividualArmsMatchCombinedRun) { const auto db = random_db(3, 30, 25); - for (const auto [variant, ranking] : {std::pair{query::FrechetVariant::Discrete, false}, - std::pair{query::FrechetVariant::Continuous, false}, - std::pair{query::FrechetVariant::Continuous, true}}) { - SCOPED_TRACE(ranking ? "ranking kNN" : "per-trajectory kNN"); + for (const auto variant : + {query::FrechetVariant::Discrete, query::FrechetVariant::Continuous}) { + SCOPED_TRACE(variant == query::FrechetVariant::Discrete ? "discrete" : "continuous"); auto config = small_config(); config.frechet = variant; auto workloads = teacher::load_or_build_workloads(db, config, 11, m_dir); - const teacher::Teacher model(db, std::move(workloads), {.ranking_knn = ranking}, 11); + const teacher::Teacher model(db, std::move(workloads), {}, 11); const std::vector arms(teacher::kArms.begin(), teacher::kArms.end()); const std::size_t budget = 2 * db.size() + 40; const auto combined = model.run(budget, arms);