ipmach · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -84,7 +84,7 @@ detectors:
             - pos: pid          # named wildcard label OR integer position in ParserSchema.variables[]
               name: pid
           header_variables:
-            - pos: Type         # key in ParserSchema.logFormatVariables{}
+            - pos: Type         # key in ParserSchema.headerVariables{}
               params: {}
     global:                     # event-ID-independent instances (GLOBAL_EVENT_ID = "*")
       global_monitor:
@@ -185,7 +185,7 @@ class MyDetector(TrackerDetector):
         super().__init__(name=name, config=config)
 
     def train(self, input_: ParserSchema) -> None:
-        self._ingest(input_, variables={...}, event_id=input_["EventID"])
+        self._ingest(input_, variables={...}, event_id=input_["eventID"])
 ```
 
 Pass tracker kwargs by overriding `_event_data_kwargs()` / `_auto_conf_kwargs()`; build an extra store with `self._new_store()`. Keep `__init__` callable as `MyDetector(name=..., config=...)`: `from_binary()` rebuilds the detector that way, so any other constructor argument needs a default.

diff --git a/README.md b/README.md
@@ -1,106 +1,24 @@
-# DetectMateLibrary
-
-Main library to run the different components in DetectMate.
-
-## Main structure
-
-The library contains the next components:
-
-* **Parsers**: parse the logs received from the reader.
-* **Detectors**: return alerts if anomalies are detected.
-* **Alert Aggregation**: aggregate the alerts produced by the detectors.
-* **Schemas**: standard data classes used in DetectMate.
-```
- +--------+     +-----------+    +-------------------+
- | Parser | --> |  Detector | -> | Alert Aggregation |
- +--------+     +-----------+    +-------------------+
-```
-## Developer setup:
-
-### Step 1: Install python dependencies
-Set up the dev environment and install pre-commit hooks:
-
-```bash
-uv sync --dev
-uv run prek install
-```
-
-### Step 2: Install Protobuf dependencies
-
-To install in Linux do:
-
-```bash
-sudo apt install -y protobuf-compiler
-protoc --version
-```
-
-This dependency is only needed if a proto file is modified. To compile the proto file do:
-```bash
-protoc --proto_path=src/detectmatelibrary/schemas/ --python_out=src/detectmatelibrary/schemas/ src/detectmatelibrary/schemas/schemas.proto
-```
 
-### Step 3: Run unit tests
-Run the tests:
-
-```bash
-uv run --dev pytest -q
-```
-
-Run the tests with coverage (add --cov-report=html to generate an HTML report):
-
-```bash
-uv run --dev pytest --cov=. --cov-report=term-missing
-```
-
-## Workspace generator (`mate create`)
-
-DetectMateLibrary includes a small CLI helper to bootstrap standalone workspaces
-for custom parsers and detectors. This is useful if you want to develop and test
-components in isolation while still using the same library and schemas.
-
-### Usage
-
-The CLI entry point is `mate` with a `create` command:
-
-```bash
-mate create --type <parser|detector> --name <workspace_name> --dir <target_dir>
-```
+# DetectMateLibrary
 
-| Option   | Description                                                                                                                                                                   |
-|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `--type` | Component type to generate:<br>- `parser`: CoreParser-based template<br>- `detector`: CoreDetector-based template                                                             |
-| `--name` | Name of the component and package:<br>- Creates package dir: `<target_dir>/<name>/`<br>- Creates main file: `<name>.py`<br>- Derives class names: `<Name>` and `<Name>Config` |
-| `--dir`  | Directory where the workspace will be created                                                                                                                                 |
+![UML](docs/img/logo.png)
 
+DetectMate is a flexible, component-based library for parsing and log anomaly detection.
 
-### What gets generated
+## Installation
 
-For example:
+Download us from `Pip`!!
 
 ```bash
-mate create --type parser --name custom_parser --dir ./workspaces/custom_parser
-```
-
-will create:
-
-```text
-workspaces/custom_parser/          # workspace root
-├── custom_parser/                 # Python package
-│   ├── __init__.py
-│   └── custom_parser.py           # CoreParser-based template
-├── tests/
-│   └── test_custom_parser.py      # generated from template to test custom_parser
-├── data.json                      # example data to run the code
-├── LICENSE.md                     # copied from main project
-├── .gitignore                     # copied from main project
-├── .pre-commit-config.yaml        # copied from main project
-├── pyproject.toml                 # minimal project + dev extras
-└── README.md                      # setup instructions
+pip install detectmatelibrary
 ```
 
 ## Documentation
 
-- [Project Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/)
+Main documentations:
+
+- [Library Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/) (this repo)
+- [Service Documentation](https://github.com/ait-detectmate/DetectMateService/tree/main)
 
 ## Contribution
 

diff --git a/docs/alert_aggregator.md b/docs/alert_aggregator.md
@@ -19,6 +19,7 @@ This document explains the expected API and how to implement an alert aggregator
   With the default `buffer_size` of 1, it is called for every alert with a list of one.
 - Use a typed `Config` class (subclass of `CoreAlertAggregatorConfig`) to hold runtime parameters.
 
+
 ## CoreAlertAggregator: minimal API
 
 ```python
@@ -52,16 +53,46 @@ Before calling `aggregate_alerts()`, `CoreAlertAggregator.run()` collects these
 alerts in the window:
 
 - `detectorIDs`, `detectorTypes`, `alertIDs`: one entry per alert
-- `logIDs`, `extractedTimestamps`: the entries of all alerts, combined into one list
+- `logIDs`, `logTimestamps`: the entries of all alerts, combined into one list
 
-After `aggregate_alerts()` returns `True`, `run()` also sets `outputTimestamp`. So the only
-fields left for `aggregate_alerts()` to fill are `description` and `alertsObtain`.
+After `aggregate_alerts()` returns `True`, `run()` also sets `aggregatedTimestamp`. So the only
+fields left for `aggregate_alerts()` to fill are `description` and `alertDetails`.
 
 !!! warning "The alerts' details are not carried over"
     `run()` copies only the fields listed above. Everything else in the incoming alerts is dropped
-    unless your `aggregate_alerts()` keeps it, including their `detectionTimestamp` and
+    unless your `aggregate_alerts()` keeps it, including their `detectedTimestamp` and
     `receivedTimestamp`.
 
-## Available alert aggregators
+## Alert aggregator methods
+
+The alert aggregator are numbered from simplest to most complex, and the sidebar lists them in the same order.
+
+| # | Detector | What it detects |
+|---|---|---|
+| 00 | [Basic Concat Aggregator](alert_aggregators/basic_concatenation.md) | Concat a set of alerts. |
+| 01 | [LogID Count Aggregator](alert_aggregators/logid_count.md) | Count the amount of times a logID is in alerts. |
+| 02 | [Density Cluster](alert_aggregators/density_cluster.md) | Base class for clustering alerts using a distance function. |
+
+
+### Common parameters (all alert aggregators)
+
+There are some parameters, that **every** alert aggregator inherits from `CoreAlertAggregatorConfig`/`CoreConfigCoreAlertAggregator`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on.
+
+<!-- Start common_arguments -->
+???+ note "Top level"
+
+    | Field | Type | Default | Description |
+    |---|---|---|---|
+    | `auto_config` | boolean | False | Runs the configuration step before the training process. |
+
+???+ note "params"
 
-- [Basic Concat](alert_aggregators/basic_concatenation.md)
+    | Field | Type | Default | Description |
+    |---|---|---|---|
+    | `start_id` | integer | 10 | Number used to start the unique ID generator. |
+    | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. |
+    | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. |
+    | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+    | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+<!-- End common_arguments -->
diff --git a/docs/alert_aggregators/basic_concatenation.md b/docs/alert_aggregators/basic_concatenation.md
@@ -1,26 +1,71 @@
 # Basic Concat Alert Aggregation
 
-The basic concat alert aggregation approach aggregates x number of alerts into one output schema.
+The Basic Concat alert aggregation combines the most recent alerts into one aggregated record by
+concatenating their IDs and timestamps.
+
+## In/out
+
+Input and output schemas in the pipeline
+
+## In/out
 
 |            | Schema                     | Description        |
 |------------|----------------------------|--------------------|
 | **Input**  | [DetectorSchema](../schemas.md)    | Alerts from detectors  |
 | **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts    |
 
+## Description
 
-# Configuration example
+The aggregator keeps a sliding window of the last `buffer_size` alerts. The first aggregated record
+is emitted once the window is full, and then one record for every new alert, so consecutive records
+overlap. Alerts are combined in the order they arrive, not grouped by log, and the alerts' own
+messages are not carried over (see [What `run()` fills in for you](../alert_aggregator.md#what-run-fills-in-for-you)).
+The [tutorial](../user_guide/tutorial.md#4-add-an-alert-aggregator) shows the overlapping output.
 
-```yaml
-alert_aggregators:
-    BasicConcatAggregator:
-        method_type: "basic_concat_aggregator"
-        buffer_size: 3
-        deduplicate_values: False
-        auto_config: False
-```
+## At a glance
 
-## Example usage
+| Learns from training data | Auto-configuration | Needs `events` | Federation |
+|---|---|---|---|
+| ❌ | ❌ | ❌ | ❌ |
+
+
+## Example
 
 ```python
 --8<-- "docs/examples/alert_aggregators/basic_concat.py:example_1"
 ```
+
+## Configuration file
+
+The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)).
+
+```yaml
+--8<-- "docs/examples/alert_aggregators/basic_concat.yaml"
+```
+
+Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`.
+
+## Configuration arguments
+
+All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators.
+
+<!-- Start arguments -->
+??? note "Top level"
+
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `method_type` | string | basic_concat_aggregator | shared | Indicates what type of method it is. |
+    | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. |
+
+???+ note "params"
+
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. |
+    | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
+    | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. |
+    | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
+    | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+    | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+<!-- End arguments -->
diff --git a/docs/alert_aggregators/density_cluster.md b/docs/alert_aggregators/density_cluster.md
@@ -0,0 +1,89 @@
+# Density Cluster Alert Aggregation
+
+The density-clustering-based alert aggregator is a base class to implement alert aggregation by clustering alerts according a given distance function of alerts.
+
+## In/out
+
+|            | Schema                     | Description        |
+|------------|----------------------------|--------------------|
+| **Input**  | [DetectorSchema](../schemas.md)    | Alerts from detectors  |
+| **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts    |
+
+## At a glance
+
+| Learns from training data | Auto-configuration | Needs `events` | Federation |
+|---|---|---|---|
+| ❌ | ❌ | ❌ | ❌ |
+
+## Description
+
+This class implements alert aggregation by density clustering.
+
+### Implementation Details
+
+Creating a custom density clustering aggregator requires the implementation of 2 methods in subclasses:
+
+1. One has to implement either the single_metric or the metric function.
+The single_metric function computes the distance between 2 alerts,
+while the metric function computes the distance matrix between a list of alerts.
+If both functions are implemented, the metric function will be used.
+
+2. The discard_alert_fn function must be implemented to determine which alerts
+should no longer be considered for aggregation.
+To this end, after each aggregation step, the discard_alert_fn
+function is called with the list of relevant alerts.
+It should return a list of booleans indicating which alerts should be discarded (True)
+and which should be kept (False).
+
+### Labelling of Alerts
+
+The group labels assigned to alerts are published in the output schema as a
+dictionary mapping alert IDs to group IDs in the "alertDetails" field.
+Every alert is assigned a group ID immediately after it is received,
+but as the aggregation progresses, groups may be merged.
+This is published in the output schema as a list of messages in the "alertDetails" field.
+Likewise, when a group is finished and no longer reachable from any active alert,
+this is also published in the output schema as a list of messages in the "alertDetails" field.
+
+## Example
+
+```python
+--8<-- "docs/examples/alert_aggregators/density_cluster.py:example_1"
+```
+
+## Configuration file
+
+The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)).
+
+```yaml
+--8<-- "docs/examples/alert_aggregators/density_cluster.yaml"
+```
+
+Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`.
+
+## Configuration arguments
+
+All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators.
+
+<!-- Start arguments -->
+??? note "Top level"
+
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `method_type` | string | density_clustering_aggregator | shared | The kind of aggregation method. |
+    | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. |
+
+???+ note "params"
+
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `buffer_size` | integer | 3 | specific | The size of the buffer. |
+    | `buffer_mode` | object | BufferMode.BATCH | specific | The mode of the buffer. |
+    | `delta` | number | 0.5 | specific | The threshold for the clustering. |
+    | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
+    | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. |
+    | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
+    | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+    | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+<!-- End arguments -->