add abstract base class for density-clustering-based alert aggregation

docs/alert_aggregator.md

@@ -71,6 +71,7 @@ The alert aggregator are numbered from simplest to most complex, and the sidebar
 |---|---|---|
 | 00 | [Basic Concat Aggregator](alert_aggregators/basic_concatenation.md) | Concat a set of alerts. |
 | 01 | [LogID Count Aggregator](alert_aggregators/logid_count.md) | Count the amount of times a logID is in alerts. |
+| 02 | [Density Cluster](alert_aggregators/density_cluster.md) | Base class for clustering alerts using a distance function. |
 
 
 ### Common parameters (all alert aggregators)

docs/alert_aggregators/density_cluster.md

@@ -0,0 +1,70 @@
+# Density Cluster Alert Aggregation
+
+The density-clustering-based alert aggregator is a base class to implement alert aggregation by clustering alerts according a given distance function of alerts.
+
+## In/out
+
+|            | Schema                     | Description        |
+|------------|----------------------------|--------------------|
+| **Input**  | [DetectorSchema](../schemas.md)    | Alerts from detectors  |
+| **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts    |
+
+## At a glance
+
+| Learns from training data | Auto-configuration | Needs `events` | Federation |
+|---|---|---|---|
+| ❌ | ❌ | ❌ | ❌ |
+
+## Description
+
+This class implements alert aggregation by density clustering.
+
+### Implementation Details
+
+Creating a custom density clustering aggregator requires the implementation of 2 methods in subclasses:
+
+1. One has to implement either the single_metric or the metric function.
+The single_metric function computes the distance between 2 alerts,
+while the metric function computes the distance matrix between a list of alerts.
+If both functions are implemented, the metric function will be used.
+
+2. The discard_alert_fn function must be implemented to determine which alerts
+should no longer be considered for aggregation.
+To this end, after each aggregation step, the discard_alert_fn
+function is called with the list of relevant alerts.
+It should return a list of booleans indicating which alerts should be discarded (True)
+and which should be kept (False).
+
+### Labelling of Alerts
+
+The group labels assigned to alerts are published in the output schema as a
+dictionary mapping alert IDs to group IDs in the "alertDetails" field.
+Every alert is assigned a group ID immediately after it is received,
+but as the aggregation progresses, groups may be merged.
+This is published in the output schema as a list of messages in the "alertDetails" field.
+Likewise, when a group is finished and no longer reachable from any active alert,
+this is also published in the output schema as a list of messages in the "alertDetails" field.
+
+## Example
+
+```python
+--8<-- "docs/examples/alert_aggregators/density_cluster.py:example_1"
+```
+
+## Configuration file
+
+The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)).
+
+```yaml
+--8<-- "docs/examples/alert_aggregators/density_cluster.yaml"
+```
+
+Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`.
+
+## Configuration arguments
+
+All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators.
+
+<!-- Start arguments -->
+
+<!-- End arguments -->

docs/alert_aggregators/logid_count.md

@@ -41,20 +41,21 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc
 <!-- Start arguments -->
 ??? note "Top level"
 
-| Field | Type | Default | Scope | Description |
-|---|---|---|---|---|
-| `method_type` | string | basic_concat_aggregator | shared | Indicates what type of method it is. |
-| `auto_config` | boolean | False | shared | Runs the configuration step before the training process. |
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `method_type` | string | logid_count_aggregator | shared | Indicates what type of method it is. |
+    | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. |
 
 ???+ note "params"
 
-| Field | Type | Default | Scope | Description |
-|---|---|---|---|---|
-| `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. |
-| `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
-| `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. |
-| `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. |
-| `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
-| `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
-| `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `max_count` | integer | 4 | specific | Max count of alerts in a specific logID. |
+    | `max_memory` | integer | 5000 | specific | Max number of logIDs store in the count memory. |
+    | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
+    | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. |
+    | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
+    | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+    | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
 <!-- End arguments -->

docs/examples/alert_aggregators/density_cluster.py

@@ -0,0 +1,63 @@
+# --8<-- [start:example_1]
+import yaml
+
+from detectmatelibrary.alert_aggregation.density_clustering import (
+    DensityClusteringAggregation,
+)
+from detectmatelibrary.schemas import AggregateSchema, DetectorSchema
+
+with open("docs/examples/alert_aggregators/density_cluster.yaml") as f:
+    config = yaml.safe_load(f)
+
+
+def make_alert(alert_id: str, score: float) -> DetectorSchema:
+    return DetectorSchema(
+        {
+            "detectorID": "detector",
+            "detectorType": "dummy",
+            "alertID": alert_id,
+            "detectedTimestamp": 0,
+            "logIDs": [f"log-{alert_id}"],
+            "score": score,
+            "logTimestamps": [0],
+            "description": "test alert",
+            "receivedTimestamp": 0,
+            "alertDetails": {},
+        }
+    )
+
+
+class ConcreteDensityClusteringAggregation(DensityClusteringAggregation):
+    def single_metric(
+        self,
+        alert1: DetectorSchema,
+        alert2: DetectorSchema
+    ) -> float:
+        return abs(alert1["score"] - alert2["score"])
+
+    def discard_alert_fn(self, alerts: list[DetectorSchema]) -> list[bool]:
+        return [False for alert in alerts]
+
+
+aggregator = ConcreteDensityClusteringAggregation(config)
+first = make_alert("a", 0.0)
+second = make_alert("b", 6.0)
+bridge = make_alert("c", 3.0)
+
+first_output = AggregateSchema()
+aggregator.aggregate_alerts(first, first_output)
+
+second_output = AggregateSchema()
+aggregator.aggregate_alerts(second, second_output)
+
+bridge_output = AggregateSchema()
+aggregator.aggregate_alerts(bridge, bridge_output)
+
+print(first_output["alertDetails"]["a"] == "0")  # True
+print(second_output["alertDetails"]["b"] == "1")  # True
+print(bridge_output["alertDetails"]["c"] == "0")  # True
+
+# group labels 0 and 1 are merged into a single group
+print(aggregator.active_groups[1] == {0, 1})  # True
+
+# --8<-- [end:example_1]

docs/examples/alert_aggregators/density_cluster.yaml

@@ -0,0 +1,4 @@
+alert_aggregators:
+    DensityClusteringAggregator:
+        method_type: density_clustering_aggregator
+        delta: 4.0

docs/examples/config/update.py

@@ -28,6 +28,8 @@
 from detectmatelibrary.common.parser import CoreParserConfig
 from detectmatelibrary.common.alert_aggregator import CoreAlertAggregatorConfig
 from detectmatelibrary.alert_aggregation.basic_concat import BasicConcatAggregationConfig
+from detectmatelibrary.alert_aggregation.density_clustering import DensityClusteringAggregationConfig
+from detectmatelibrary.alert_aggregation.logid_count import LogIDCountAggegationConfig
 from detectmatelibrary.base_detectors import TrackerDetectorConfig
 from detectmatelibrary.base_detectors import VariableDetectorConfig
 from detectmatelibrary.base_detectors import DeepLearningDetectorConfig
@@ -271,7 +273,17 @@ def update_shared_args_aggregation(doc_path: str) -> None:
         BasicConcatAggregationConfig(),
         CoreAlertAggregatorConfig,
         "docs/alert_aggregators/basic_concatenation.md"
-    )
+    ),
+    (
+        LogIDCountAggegationConfig(),
+        CoreAlertAggregatorConfig,
+        "docs/alert_aggregators/logid_count.md"
+    ),
+    (
+      DensityClusteringAggregationConfig(),
+      CoreAlertAggregatorConfig,
+      "docs/alert_aggregators/density_cluster.md"
+    ),
 ]
 
 update_shared_args_detectors("docs/detectors.md")

mkdocs.yml

@@ -68,6 +68,7 @@ nav:
     - Alert Aggregation methods:
       - 00_Basic Concat: alert_aggregators/basic_concatenation.md
       - 01_LogID Count: alert_aggregators/logid_count.md
+      - 02_Density Cluster: alert_aggregators/density_cluster.md
     - Helpers:
       - FromTo: helper/from_to.md
       - Select Component: helper/select_component.md

src/detectmatelibrary/alert_aggregation/__init__.py

@@ -1,9 +1,15 @@
 from .basic_concat import BasicConcatAggregation, BasicConcatAggregationConfig
+from .density_clustering import (
+    DensityClusteringAggregation,
+    DensityClusteringAggregationConfig,
+)
 from .logid_count import LogIDCountAggegation, LogIDCountAggegationConfig
 
 __all__ = [
     "BasicConcatAggregation",
     "BasicConcatAggregationConfig",
+    "DensityClusteringAggregation",
+    "DensityClusteringAggregationConfig",
     "LogIDCountAggegation",
     "LogIDCountAggegationConfig",
 ]