diff --git a/docs/alert_aggregator.md b/docs/alert_aggregator.md index 0c2edd83..0d1a4738 100644 --- a/docs/alert_aggregator.md +++ b/docs/alert_aggregator.md @@ -71,6 +71,7 @@ The alert aggregator are numbered from simplest to most complex, and the sidebar |---|---|---| | 00 | [Basic Concat Aggregator](alert_aggregators/basic_concatenation.md) | Concat a set of alerts. | | 01 | [LogID Count Aggregator](alert_aggregators/logid_count.md) | Count the amount of times a logID is in alerts. | +| 02 | [Density Cluster](alert_aggregators/density_cluster.md) | Add description... ADD_DENSITY_CLUSTER| ### Common parameters (all alert aggregators) diff --git a/docs/alert_aggregators/density_cluster.md b/docs/alert_aggregators/density_cluster.md new file mode 100644 index 00000000..b9b16771 --- /dev/null +++ b/docs/alert_aggregators/density_cluster.md @@ -0,0 +1,48 @@ +# Density Cluster Alert Aggregation + +add stuff, dont remove start and end arguments. + +## In/out + +Input and output schemas in the pipeline + +## In/out + +| | Schema | Description | +|------------|----------------------------|--------------------| +| **Input** | [DetectorSchema](../schemas.md) | Alerts from detectors | +| **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts | + +## Description + + +## At a glance + +| Learns from training data | Auto-configuration | Needs `events` | Federation | +|---|---|---|---| +| ❌ | ❌ | ❌ | ❌ | + + +## Example + +```python +--8<-- "docs/examples/alert_aggregators/density_cluster.py:example_1" +``` + +## Configuration file + +The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)). + +```yaml +--8<-- "docs/examples/alert_aggregators/density_cluster.yaml" +``` + +Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`. + +## Configuration arguments + +All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators. + + + + diff --git a/docs/alert_aggregators/logid_count.md b/docs/alert_aggregators/logid_count.md index 1a868e22..94e178e3 100644 --- a/docs/alert_aggregators/logid_count.md +++ b/docs/alert_aggregators/logid_count.md @@ -41,20 +41,21 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc ??? note "Top level" -| Field | Type | Default | Scope | Description | -|---|---|---|---|---| -| `method_type` | string | basic_concat_aggregator | shared | Indicates what type of method it is. | -| `auto_config` | boolean | False | shared | Runs the configuration step before the training process. | + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `method_type` | string | logid_count_aggregator | shared | Indicates what type of method it is. | + | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. | ???+ note "params" -| Field | Type | Default | Scope | Description | -|---|---|---|---|---| -| `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. | -| `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | -| `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | -| `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | -| `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | -| `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | -| `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `max_count` | integer | 4 | specific | Max count of alerts in a specific logID. | + | `max_memory` | integer | 5000 | specific | Max number of logIDs store in the count memory. | + | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | + | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | + | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | + | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/examples/alert_aggregators/density_cluster.py b/docs/examples/alert_aggregators/density_cluster.py new file mode 100644 index 00000000..3c86f9a3 --- /dev/null +++ b/docs/examples/alert_aggregators/density_cluster.py @@ -0,0 +1,9 @@ +# --8<-- [start:example_1] + +print("Hello World") + +# Check BasicConcat of LogID count documentation to add your example here +# Use the example config -> density_cluster.yaml from same folder +# ADD_DENSITY_CLUSTER + +# --8<-- [end:example_1] diff --git a/docs/examples/alert_aggregators/density_cluster.yaml b/docs/examples/alert_aggregators/density_cluster.yaml new file mode 100644 index 00000000..b690caf5 --- /dev/null +++ b/docs/examples/alert_aggregators/density_cluster.yaml @@ -0,0 +1,3 @@ +alert_aggregators: + NAME: + method_type: "ADD_DENSITY_CLUSTER" diff --git a/docs/examples/config/update.py b/docs/examples/config/update.py index 6c104a2f..85f1f2ae 100644 --- a/docs/examples/config/update.py +++ b/docs/examples/config/update.py @@ -28,6 +28,7 @@ from detectmatelibrary.common.parser import CoreParserConfig from detectmatelibrary.common.alert_aggregator import CoreAlertAggregatorConfig from detectmatelibrary.alert_aggregation.basic_concat import BasicConcatAggregationConfig +from detectmatelibrary.alert_aggregation.logid_count import LogIDCountAggegationConfig from detectmatelibrary.base_detectors import TrackerDetectorConfig from detectmatelibrary.base_detectors import VariableDetectorConfig from detectmatelibrary.base_detectors import DeepLearningDetectorConfig @@ -271,7 +272,18 @@ def update_shared_args_aggregation(doc_path: str) -> None: BasicConcatAggregationConfig(), CoreAlertAggregatorConfig, "docs/alert_aggregators/basic_concatenation.md" - ) + ), + ( + LogIDCountAggegationConfig(), + CoreAlertAggregatorConfig, + "docs/alert_aggregators/logid_count.md" + ), + # ADD_DENSITY_CLUSTER + # ( + # DensityClusteringAggregationConfig, + # CoreAlertAggregatorConfig, + # "docs/alert_aggregators/logid_count.md" + # ), ] update_shared_args_detectors("docs/detectors.md") diff --git a/mkdocs.yml b/mkdocs.yml index 8b0464f7..b0e19095 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -68,6 +68,7 @@ nav: - Alert Aggregation methods: - 00_Basic Concat: alert_aggregators/basic_concatenation.md - 01_LogID Count: alert_aggregators/logid_count.md + - 02_Density Cluster: alert_aggregators/density_cluster.md - Helpers: - FromTo: helper/from_to.md - Select Component: helper/select_component.md