OIE: parse event pages and embed auto-filed nodes

[northdpole]

Summary

Stacked on #1135 (tip of the OIE / OWASP agent stack: #1132 -> #1135). Merge the whole stack, not this PR alone.

• Event pages are metadata, so harvesting them is opt-in. The scheduled owasp job skips kind: event repos unless OIE_HARVEST_EVENTS=1 (default off). --harvester_kinds event on the CLI still works.
• Event page parsing. event_page.prepare_event_page turns a Jekyll index.md (front matter + schema.org JSON-LD + Liquid) into a title/dates/location/description header plus the markdown body. Pages with no content are skipped and do not block the checkpoint. When enabled, event chunks flow through noise filter and Librarian like other chunks; the filing confidence floor still decides what reaches the graph.
• Auto-filed nodes get embeddings. New embed_nodes stage (last step of the daily cre_expansion job) embeds oie-auto-filed nodes with no embedding. The pending set is derived from the graph, so a failed run is retried next time. A failure degrades the job but never touches filed nodes/edges. OIE_EMBED_FILED_NODES=0 turns it off.
• Filed event nodes are embedded from the cleaned page text (hook in in_memory_embeddings.get_content, plain-text and Playwright paths).
• Page parsing is linear-time on untrusted content (adversarial-input tests included).
• Migration fix for the stack: f8a9b0c1d2e3 (admin tables, Admin showcase panel for signed-in users #1135) and b8c9d0e1f2a3 (oie_run, OIE: scheduled runs (OWASP every 10 min, CRE expansion daily) #1132) both descended from 967016ee10fa, leaving two alembic heads. The admin migration now follows b8c9d0e1f2a3 (pushed to Admin showcase panel for signed-in users #1135's branch together with a Black fix for its lint).

Verification

• black --check on changed files: pass.
• Full unittest suite: 1624 tests, 1 failure (librarian.dataset_test.test_build_check_matches_committed_dataset, also fails on the base branch).
• Local Postgres: single alembic head, upgrade from scratch, guardrail OK; real harvest of www-event-2020-07-virtual; real embed of an auto-filed node via the default path.

Notes

• Anyone who already applied f8a9b0c1d2e3 on a DB that lacks b8c9d0e1f2a3 should run flask db upgrade again after pulling.
• Commits are unsigned (agent cannot sign).

  

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: OWASP/OpenCRE/.coderabbit.yml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 91ee2cff-799c-4bd5-bfbc-d112cc050401

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}