Skip to content

knowledge-graph.md

Latest commit

 

History

History
351 lines (299 loc) · 10.1 KB

knowledge-graph.md

File metadata and controls

351 lines (299 loc) · 10.1 KB
Raw

Knowledge Graph Documentation

Overview

The Photobook Knowledge Graph (KG) is a claim-centric graph database that stores entities extracted from photos along with semantic relationships, visual evidence, and provenance information. It supports human-in-the-loop verification and temporal reasoning.

Core Concepts

1. Nodes

Generic graph nodes representing entities in the photo domain:

Node Type Description Example
IMAGE Uploaded photo photo_vacation_001.jpg
PERSON Human individual John Doe
DETECTION Object detection instance car in photo_001
ENTITY Named entity (non-person) Golden Gate Bridge
PLACE Location/venue Central Park, NYC
EVENT Detected/inferred event Birthday Party
ATTRIBUTE Property/characteristic wearing sunglasses
SCENE Scene classification beach, restaurant
TAG User/AI-assigned tag vacation, family
TEXT OCR-detected text "Welcome Home"
UNKNOWN Unclassified -

2. Claims

Reified edges representing statements about the world:

"According to [source], with [confidence], [subject] [predicate] [object], 
valid during [time_range], supported by [evidence]."

Example Claims:

  • Image_001 --depicts_person--> Person_Alice (confidence: 0.92, source: qwen3-vl)
  • Person_Alice --is_wearing--> sunglasses (confidence: 0.85, source: qwen3-vl)
  • Image_001 --taken_at--> Place_Beach (confidence: 0.95, source: google_places)
  • Person_Alice --same_as--> Person_Photo3 (confidence: 0.88, source: insightface)

3. Evidence

Visual grounding linking claims to image regions:

{
  "claim_id": "...",
  "image_node_id": "...",
  "bbox": {"x": 100, "y": 50, "w": 200, "h": 300},
  "mask_path": "masks/claim_123.png",
  "model_output": {"raw_response": "..."},
  "confidence": 0.92
}

4. Status Lifecycle

Claims progress through a verification workflow:

                    ┌─────────────┐
                    │  PROPOSED   │  (AI-generated, unverified)
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
              ▼            ▼            ▼
    ┌─────────────┐  ┌─────────┐  ┌──────────┐
    │ WEAKLY_     │  │ ACCEPTED │  │ REJECTED │
    │ SUPPORTED   │  │ (user    │  │ (user    │
    │ (multiple   │  │ verified)│  │ denied)  │
    │ evidence)   │  └──────────┘  └──────────┘
    └──────┬──────┘
           │
           ▼
    ┌──────────────┐
    │  INFERRED    │  (derived from accepted claims)
    └──────────────┘
           │
           ▼
    ┌──────────────┐
    │  SUPERSEDED  │  (replaced by newer claim)
    └──────────────┘

5. Source Types

Provenance tracking for claim origins:

Source Description
AI_VISION VLM-generated (Qwen3-VL)
AI_FACE Face recognition (InsightFace)
USER User-created or verified
SYSTEM System-inferred
EXTERNAL External API (Google Places)

Database Schema

Node Table (kg_nodes)

CREATE TABLE kg_nodes (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    node_type VARCHAR(50) NOT NULL,  -- NodeType enum
    canonical_name VARCHAR(255),
    canonical_type VARCHAR(100),
    properties JSONB,            -- Flexible metadata
    embedding JSONB,             -- Vector for similarity
    user_id INTEGER REFERENCES users(id),
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ix_kg_nodes_user_type ON kg_nodes(user_id, node_type);

Claim Table (kg_claims)

CREATE TABLE kg_claims (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    subject_id UUID NOT NULL REFERENCES kg_nodes(id),
    predicate VARCHAR(255) NOT NULL,
    object_id UUID REFERENCES kg_nodes(id),     -- Either object_id
    literal_value TEXT,                          -- OR literal_value
    confidence FLOAT DEFAULT 0.5,
    status VARCHAR(50) DEFAULT 'proposed',
    valid_from TIMESTAMPTZ,
    valid_to TIMESTAMPTZ,
    source_type VARCHAR(50) DEFAULT 'SYSTEM',
    source_model VARCHAR(100),
    source_version VARCHAR(50),
    user_id INTEGER REFERENCES users(id),
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ix_kg_claims_subject_predicate ON kg_claims(subject_id, predicate);
CREATE INDEX ix_kg_claims_user_status ON kg_claims(user_id, status);
CREATE INDEX ix_kg_claims_temporal ON kg_claims(valid_from, valid_to);

Evidence Table (kg_evidence)

CREATE TABLE kg_evidence (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    claim_id UUID NOT NULL REFERENCES kg_claims(id),
    image_node_id UUID NOT NULL REFERENCES kg_nodes(id),
    bbox JSONB,              -- {x, y, w, h}
    mask_path VARCHAR(500),
    model_output JSONB,
    confidence FLOAT DEFAULT 1.0,
    created_at TIMESTAMPTZ DEFAULT now()
);

User Action Table (kg_user_actions)

CREATE TABLE kg_user_actions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    claim_id UUID NOT NULL REFERENCES kg_claims(id),
    user_id INTEGER REFERENCES users(id),
    action_type VARCHAR(50) NOT NULL,  -- ACCEPT, REJECT, CORRECT
    previous_status VARCHAR(50),
    new_status VARCHAR(50),
    correction_data JSONB,
    created_at TIMESTAMPTZ DEFAULT now()
);

KG Service API

Python Service Layer

from kg.service import KGService

async with session:
    kg = KGService(session)
    
    # Create node
    person = await kg.create_node(
        user_id=1,
        node_type=NodeType.PERSON,
        canonical_name="Alice",
        properties={"is_identified": True}
    )
    
    # Create claim
    claim = await kg.create_claim(
        user_id=1,
        subject_id=image_id,
        predicate="depicts_person",
        object_id=person.id,
        confidence=0.92,
        source_type=SourceType.AI_VISION,
        source_model="qwen3-vl-32b"
    )
    
    # Add evidence
    evidence = await kg.add_evidence(
        claim_id=claim.id,
        image_node_id=image_id,
        bbox={"x": 100, "y": 50, "w": 200, "h": 300},
        confidence=0.92
    )
    
    # Find or create (deduplication)
    tag = await kg.find_or_create_node(
        user_id=1,
        node_type=NodeType.TAG,
        canonical_name="vacation"
    )

REST API Endpoints

Endpoint Method Description
/api/v1/kg/nodes GET List nodes with filters
/api/v1/kg/nodes/{id} GET Get node with claims
/api/v1/kg/nodes/{id} PUT Update node properties
/api/v1/kg/claims GET List claims with filters
/api/v1/kg/claims/{id} PUT Update claim
/api/v1/kg/claims/{id}/accept POST Accept claim
/api/v1/kg/claims/{id}/reject POST Reject claim
/api/v1/kg/graph GET Get visualization data

Graph Visualization Data

GET /api/v1/kg/graph?include_rejected=false&min_confidence=0.5&limit_nodes=500

Response:

{
  "nodes": [
    {"id": "...", "type": "IMAGE", "label": "photo_001.jpg", "properties": {...}},
    {"id": "...", "type": "PERSON", "label": "Alice", "properties": {...}}
  ],
  "edges": [
    {
      "id": "...",
      "source": "image_id",
      "target": "person_id",
      "predicate": "depicts_person",
      "confidence": 0.92,
      "status": "accepted",
      "source_type": "AI_VISION"
    }
  ],
  "stats": {
    "total_nodes": 150,
    "total_edges": 320,
    "node_type_counts": {"IMAGE": 50, "PERSON": 10, "DETECTION": 80},
    "claim_status_counts": {"proposed": 200, "accepted": 100, "rejected": 20}
  }
}

Common Predicates

Image → Entity Relations

Predicate Description
depicts_person Image contains person
contains_object Image contains detected object
contains_entity Image contains named entity
taken_at Image location
has_tag Image has tag
depicts_scene Image scene classification
contains_text Image contains OCR text

Person Relations

Predicate Description
same_as Person identity matching
is_wearing Clothing/accessory
is_holding Object interaction
is_near Spatial proximity

Semantic Relations (from Scene Graph)

Predicate Description
hugging Physical interaction
playing_with Activity
standing_next_to Spatial
sitting_on Position
looking_at Gaze

Query Patterns

Find all claims for an image

claims = await kg.get_claims_for_node(
    node_id=image_uuid,
    as_subject=True,
    as_object=False,
    status_filter=[ClaimStatus.PROPOSED, ClaimStatus.ACCEPTED]
)

Find related nodes

# Find all people in an image
people = await kg.find_related_nodes(
    node_id=image_uuid,
    predicate="depicts_person",
    direction="outgoing"
)

Human-in-the-loop verification

# User accepts a claim
claim = await kg.transition_claim(
    claim_id=claim_uuid,
    user_id=1,
    action=ActionType.ACCEPT
)
# Stores UserAction for audit trail

Visualization

The frontend uses Sigma.js with graphology for interactive KG exploration:

Node Colors by Type:

  • IMAGE: Blue (#3b82f6)
  • PERSON: Purple (#8b5cf6)
  • DETECTION: Orange (#f97316)
  • PLACE: Green (#10b981)
  • TAG: Yellow (#eab308)

Edge Colors by Status:

  • proposed: Gray
  • weakly_supported: Light blue
  • accepted: Green
  • rejected: Red

Interactions:

  • Click node → View properties and claims
  • Click edge → View claim details, accept/reject
  • Filter by node type, claim status, confidence
  • Search by node label