Skip to content

feat: AEGIS write pipeline — ingest recent conversation threads from dreaming/conversation-facts cycle #4

Closed

Description

@stackbilt-admin

Background

MindSpring currently has 3,007 conversation vectors (verified 2026-05-19) from a historical bulk export. AEGIS has no scheduled task that writes new conversations into MindSpring — every integration that reads from it (grounding RRF, curiosity Source 8, PRISM Step 4.5) queries a static snapshot that grows staler as new conversations accumulate in D1.

What's needed

A write path from AEGIS's scheduled pipeline into MindSpring's ingest API. The natural home is the conversation-facts task (runs every 2h in aegis-oss/web/src/kernel/scheduled/conversation-facts.ts) which already iterates recent D1 conversation threads. After processing a batch, it should serialize the threads as NDJSON and POST to POST /api/uploads/simple.

Expected format (from queue.ts)

One JSON object per line:

{"id": "<thread_id>", "title": "<first user message or thread summary>", "create_time": <unix_seconds>, "messages": [{"author": {"role": "user"}, "content": {"parts": ["..."]}}, ...]}

MindSpring's built-in processed:<id> KV deduplication means re-uploading already-ingested threads is safe.

Integration context

Three AEGIS systems now depend on MindSpring being current:

  1. Grounding RRF (recall.ts) — Rank 4, limit=5, threshold=0.4
  2. Curiosity Source 8 (curiosity.ts, shipped 2026-05-19) — thin MW topic seeds → MindSpring gap scan
  3. PRISM Step 4.5 (synthesis.ts, shipped 2026-05-19) — dedicated conversation context block for synthesis prompt

All three read-only. None write.

Auth

AEGIS has a read-scoped API key (ms_aegis_*) for search. The ingest call would need an ingest-scoped key — either a new key or an upgrade of the existing one. Key provisioning is on the AEGIS side once the scope is available.

Notes

  • File size for a 2h conversation batch should stay well under the 5MB simple upload limit
  • AEGIS's D1 messages table stores role and content per message, keyed by conversation_id — straightforward to serialize
  • The conversation-facts task already has a watermark pattern for tracking processed threads; the MindSpring upload can reuse the same batch window

Activity

  1. stackbilt-admin commented on May 19, 2026

    @stackbilt-admin
    MemberAuthor

    Shipped in main via commit 3309996.

    Implemented MindSpring-side support for the AEGIS write pipeline described in this issue:

    • Added NDJSON ingestion support for /api/uploads/simple queue processing.
      • Queue now auto-detects .ndjson uploads and uses streaming line-by-line batch parsing.
    • Added parser support for AEGIS thread object shape:
      • messages[] with author.role + content.parts[] now validates and extracts correctly.
    • Expanded simple-upload content type acceptance for NDJSON payloads:
      • application/x-ndjson
      • text/plain
    • Updated OpenAPI for /api/uploads/simple request body to include NDJSON media types.
    • Added tests:
      • NDJSON streaming batch parser coverage
      • AEGIS thread extraction/validation coverage

    Validation run:

    • npm run type-check passed
    • npm test passed (12 tests)

    This addresses the core write-path compatibility needed for AEGIS conversation-facts to POST NDJSON batches into MindSpring safely (with existing dedupe semantics via processed:<id>).

  2. stackbilt-admin commented on May 19, 2026

    @stackbilt-admin
    MemberAuthor

    Closing as shipped in main via commit 3309996 (NDJSON + AEGIS thread ingest compatibility). Validation passed: npm run type-check, npm test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions