diff --git a/AGENTS.md b/AGENTS.md index c499600f..ea9581b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,7 +84,7 @@ detectors: - pos: pid # named wildcard label OR integer position in ParserSchema.variables[] name: pid header_variables: - - pos: Type # key in ParserSchema.logFormatVariables{} + - pos: Type # key in ParserSchema.headerVariables{} params: {} global: # event-ID-independent instances (GLOBAL_EVENT_ID = "*") global_monitor: @@ -185,7 +185,7 @@ class MyDetector(TrackerDetector): super().__init__(name=name, config=config) def train(self, input_: ParserSchema) -> None: - self._ingest(input_, variables={...}, event_id=input_["EventID"]) + self._ingest(input_, variables={...}, event_id=input_["eventID"]) ``` Pass tracker kwargs by overriding `_event_data_kwargs()` / `_auto_conf_kwargs()`; build an extra store with `self._new_store()`. Keep `__init__` callable as `MyDetector(name=..., config=...)`: `from_binary()` rebuilds the detector that way, so any other constructor argument needs a default. diff --git a/README.md b/README.md index 72a284da..20ae0647 100644 --- a/README.md +++ b/README.md @@ -1,106 +1,24 @@ -# DetectMateLibrary - -Main library to run the different components in DetectMate. - -## Main structure - -The library contains the next components: - -* **Parsers**: parse the logs received from the reader. -* **Detectors**: return alerts if anomalies are detected. -* **Alert Aggregation**: aggregate the alerts produced by the detectors. -* **Schemas**: standard data classes used in DetectMate. -``` - +--------+ +-----------+ +-------------------+ - | Parser | --> | Detector | -> | Alert Aggregation | - +--------+ +-----------+ +-------------------+ -``` -## Developer setup: - -### Step 1: Install python dependencies -Set up the dev environment and install pre-commit hooks: - -```bash -uv sync --dev -uv run prek install -``` - -### Step 2: Install Protobuf dependencies - -To install in Linux do: - -```bash -sudo apt install -y protobuf-compiler -protoc --version -``` - -This dependency is only needed if a proto file is modified. To compile the proto file do: -```bash -protoc --proto_path=src/detectmatelibrary/schemas/ --python_out=src/detectmatelibrary/schemas/ src/detectmatelibrary/schemas/schemas.proto -``` -### Step 3: Run unit tests -Run the tests: - -```bash -uv run --dev pytest -q -``` - -Run the tests with coverage (add --cov-report=html to generate an HTML report): - -```bash -uv run --dev pytest --cov=. --cov-report=term-missing -``` - -## Workspace generator (`mate create`) - -DetectMateLibrary includes a small CLI helper to bootstrap standalone workspaces -for custom parsers and detectors. This is useful if you want to develop and test -components in isolation while still using the same library and schemas. - -### Usage - -The CLI entry point is `mate` with a `create` command: - -```bash -mate create --type --name --dir -``` +# DetectMateLibrary -| Option | Description | -|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `--type` | Component type to generate:
- `parser`: CoreParser-based template
- `detector`: CoreDetector-based template | -| `--name` | Name of the component and package:
- Creates package dir: `//`
- Creates main file: `.py`
- Derives class names: `` and `Config` | -| `--dir` | Directory where the workspace will be created | +![UML](docs/img/logo.png) +DetectMate is a flexible, component-based library for parsing and log anomaly detection. -### What gets generated +## Installation -For example: +Download us from `Pip`!! ```bash -mate create --type parser --name custom_parser --dir ./workspaces/custom_parser -``` - -will create: - -```text -workspaces/custom_parser/ # workspace root -├── custom_parser/ # Python package -│ ├── __init__.py -│ └── custom_parser.py # CoreParser-based template -├── tests/ -│ └── test_custom_parser.py # generated from template to test custom_parser -├── data.json # example data to run the code -├── LICENSE.md # copied from main project -├── .gitignore # copied from main project -├── .pre-commit-config.yaml # copied from main project -├── pyproject.toml # minimal project + dev extras -└── README.md # setup instructions +pip install detectmatelibrary ``` ## Documentation -- [Project Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/) +Main documentations: + +- [Library Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/) (this repo) +- [Service Documentation](https://github.com/ait-detectmate/DetectMateService/tree/main) ## Contribution diff --git a/docs/alert_aggregator.md b/docs/alert_aggregator.md index deac6763..086eaf72 100644 --- a/docs/alert_aggregator.md +++ b/docs/alert_aggregator.md @@ -19,6 +19,7 @@ This document explains the expected API and how to implement an alert aggregator With the default `buffer_size` of 1, it is called for every alert with a list of one. - Use a typed `Config` class (subclass of `CoreAlertAggregatorConfig`) to hold runtime parameters. + ## CoreAlertAggregator: minimal API ```python @@ -52,16 +53,60 @@ Before calling `aggregate_alerts()`, `CoreAlertAggregator.run()` collects these alerts in the window: - `detectorIDs`, `detectorTypes`, `alertIDs`: one entry per alert -- `logIDs`, `extractedTimestamps`: the entries of all alerts, combined into one list +- `logIDs`, `logTimestamps`: the entries of all alerts, combined into one list -After `aggregate_alerts()` returns `True`, `run()` also sets `outputTimestamp`. So the only -fields left for `aggregate_alerts()` to fill are `description` and `alertsObtain`. +After `aggregate_alerts()` returns `True`, `run()` also sets `aggregatedTimestamp`. So the only +fields left for `aggregate_alerts()` to fill are `description` and `alertDetails`. !!! warning "The alerts' details are not carried over" `run()` copies only the fields listed above. Everything else in the incoming alerts is dropped - unless your `aggregate_alerts()` keeps it, including their `detectionTimestamp` and + unless your `aggregate_alerts()` keeps it, including their `detectedTimestamp` and `receivedTimestamp`. -## Available alert aggregators +## Alert aggregator methods + +The alert aggregator are numbered from simplest to most complex, and the sidebar lists them in the same order. + +| # | Detector | What it detects | +|---|---|---| +| 00 | [Basic Concat Aggregator](alert_aggregators/basic_concatenation.md) | Concat a set of alerts. | +| 01 | [LogID Count Aggregator](alert_aggregators/logid_count.md) | Count the amount of times a logID is in alerts. | +| 02 | [Time Delta](alert_aggregators/time_delta.md) | Group alerts by their time difference. | + + +### Common parameters (all alert aggregators) + +There are some parameters, that **every** alert aggregator inherits from `CoreAlertAggregatorConfig`/`CoreConfigCoreAlertAggregator`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on. + + +???+ note "Top level" + + | Field | Type | Default | Description | + |---|---|---|---| + | `auto_config` | boolean | False | Runs the configuration step before the training process. | + +???+ note "params" + + | Field | Type | Default | Description | + |---|---|---|---| + | `start_id` | integer | 10 | Number used to start the unique ID generator. | + | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. | + | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. | + | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | + | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | + + +### Density Clustering Aggregator + +The alert aggregators that are based on density clustering ([Time Delta](alert_aggregators/time_delta.md)) accept also the following arguments: + + +???+ note "params" -- [Basic Concat](alert_aggregators/basic_concatenation.md) + | Field | Type | Default | Description | + |---|---|---|---| + | `buffer_size` | integer | 3 | The size of the buffer. | + | `buffer_mode` | object | BufferMode.BATCH | The mode of the buffer. | + | `delta` | number | 0.5 | The threshold for the clustering. | + diff --git a/docs/alert_aggregators/basic_concatenation.md b/docs/alert_aggregators/basic_concatenation.md index d7c60c1c..bca3916b 100644 --- a/docs/alert_aggregators/basic_concatenation.md +++ b/docs/alert_aggregators/basic_concatenation.md @@ -1,26 +1,71 @@ # Basic Concat Alert Aggregation -The basic concat alert aggregation approach aggregates x number of alerts into one output schema. +The Basic Concat alert aggregation combines the most recent alerts into one aggregated record by +concatenating their IDs and timestamps. + +## In/out + +Input and output schemas in the pipeline + +## In/out | | Schema | Description | |------------|----------------------------|--------------------| | **Input** | [DetectorSchema](../schemas.md) | Alerts from detectors | | **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts | +## Description -# Configuration example +The aggregator keeps a sliding window of the last `buffer_size` alerts. The first aggregated record +is emitted once the window is full, and then one record for every new alert, so consecutive records +overlap. Alerts are combined in the order they arrive, not grouped by log, and the alerts' own +messages are not carried over (see [What `run()` fills in for you](../alert_aggregator.md#what-run-fills-in-for-you)). +The [tutorial](../user_guide/tutorial.md#4-add-an-alert-aggregator) shows the overlapping output. -```yaml -alert_aggregators: - BasicConcatAggregator: - method_type: "basic_concat_aggregator" - buffer_size: 3 - deduplicate_values: False - auto_config: False -``` +## At a glance -## Example usage +| Learns from training data | Auto-configuration | Needs `events` | Federation | +|---|---|---|---| +| ❌ | ❌ | ❌ | ❌ | + + +## Example ```python --8<-- "docs/examples/alert_aggregators/basic_concat.py:example_1" ``` + +## Configuration file + +The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)). + +```yaml +--8<-- "docs/examples/alert_aggregators/basic_concat.yaml" +``` + +Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`. + +## Configuration arguments + +All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators. + + +??? note "Top level" + + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `method_type` | string | basic_concat_aggregator | shared | Indicates what type of method it is. | + | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. | + +???+ note "params" + + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. | + | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | + | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | + | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | + | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | + diff --git a/docs/alert_aggregators/logid_count.md b/docs/alert_aggregators/logid_count.md new file mode 100644 index 00000000..94e178e3 --- /dev/null +++ b/docs/alert_aggregators/logid_count.md @@ -0,0 +1,61 @@ +# LogID Count Alert Aggregation + +The logID count alert aggregation tracks how many times each log ID appears in the alerts, and if it exceeds a specified threshold, it triggers the aggregation. + +## In/out + +| | Schema | Description | +|------------|----------------------------|--------------------| +| **Input** | [DetectorSchema](../schemas.md) | Alerts from detectors | +| **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts | + +## At a glance + +| Learns from training data | Auto-configuration | Needs `events` | Federation | +|---|---|---|---| +| ❌ | ❌ | ❌ | ❌ | + +## Description + +A log ID counter stores, in a fixed-size table, the counts for different log IDs. If the count for any log ID from the Detector Schemas exceeds a defined threshold, it returns an alert aggregation. + +The fixed-size table operates as a FIFO: once it is full, it removes the oldest entry and inserts the new log ID. +## Example + +```python +--8<-- "docs/examples/alert_aggregators/logid_count.py:example_1" +``` + +## Configuration file + +The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)). + +```yaml +--8<-- "docs/examples/alert_aggregators/logid_count.yaml" +``` + +## Configuration arguments + +All parameters this detector accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this detector or `shared` with other detectors (see the [Alert Aggregation overview](../alert_aggregator.md#common-parameters-all-alert-aggregators)). Where this detector changes a shared default, the shared value is shown in brackets. + + +??? note "Top level" + + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `method_type` | string | logid_count_aggregator | shared | Indicates what type of method it is. | + | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. | + +???+ note "params" + + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `max_count` | integer | 4 | specific | Max count of alerts in a specific logID. | + | `max_memory` | integer | 5000 | specific | Max number of logIDs store in the count memory. | + | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | + | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | + | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | + | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | + diff --git a/docs/alert_aggregators/time_delta.md b/docs/alert_aggregators/time_delta.md new file mode 100644 index 00000000..f65fba67 --- /dev/null +++ b/docs/alert_aggregators/time_delta.md @@ -0,0 +1,67 @@ +# Time Delta Alert Aggregation + +The time-delta alert aggregator implements alert aggregation by grouping alerts according their time-difference as proposed by [Landauer et al.](https://dl.acm.org/doi/pdf/10.1145/3510581). + +## In/out + +| | Schema | Description | +|------------|----------------------------|--------------------| +| **Input** | [DetectorSchema](../schemas.md) | Alerts from detectors | +| **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts | + +## At a glance + +| Learns from training data | Auto-configuration | Needs `events` | Federation | +|---|---|---|---| +| ❌ | ❌ | ❌ | ❌ | + +## Description + +This class implements alert aggregation by considering the differences between the timestamps of alerts. + +Two alerts will be assigned to the same group if the time difference between them is below a certain threshold delta. +The time difference is computed based on the timestamps of the last logs associated with each alert as these determine the earliest time at which the alert was detected. +Important: It is assumed that the alerts arrive in order of their timestamps! + +## Example + +```python +--8<-- "docs/examples/alert_aggregators/time_delta.py:example_1" +``` + +## Configuration file + +The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)). + +```yaml +--8<-- "docs/examples/alert_aggregators/time_delta.yaml" +``` + +Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`. + +## Configuration arguments + +All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators. + + +??? note "Top level" + + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `method_type` | string | time_delta_aggregator | shared | The kind of aggregation method. | + | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. | + +???+ note "params" + + | Field | Type | Default | Scope | Description | + |---|---|---|---|---| + | `buffer_size` | integer | 3 | specific | The size of the buffer. | + | `buffer_mode` | object | BufferMode.BATCH | specific | The mode of the buffer. | + | `delta` | number | 10.0 | specific | The threshold for the clustering. | + | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | + | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | + | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | + | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | + diff --git a/docs/auxiliar/input_buffer.md b/docs/auxiliar/input_buffer.md index 09a53397..e1d352b2 100644 --- a/docs/auxiliar/input_buffer.md +++ b/docs/auxiliar/input_buffer.md @@ -1,4 +1,4 @@ -# Data Buffer +# Data Buffer The data buffer is an auxiliary method that can be used in all the components. It takes the stream data and formats it to the given specifications. diff --git a/docs/auxiliar/persistency.md b/docs/auxiliar/persistency.md index 808af5c0..27653386 100644 --- a/docs/auxiliar/persistency.md +++ b/docs/auxiliar/persistency.md @@ -1,7 +1,7 @@ # Persistency The persistency module gives a detector a place to remember things about the -events it sees. State is keyed by `EventID` and survives across training, the +events it sees. State is keyed by `eventID` and survives across training, the detection loop, and (optionally) restarts on disk. This page is structured to read top-to-bottom: first the mental model, then a @@ -14,10 +14,10 @@ rest of the page much easier to follow. ### 1. Events -Logs are grouped by `EventID`. Two events with the same ID share a template +Logs are grouped by `eventID`. Two events with the same ID share a template but have their own variable values. Persistency stores **one independent state -object per event ID**, so an `EventStabilityTracker` for `EventID=4733` does -not interfere with one for `EventID=4624`. +object per event ID**, so an `EventStabilityTracker` for `eventID=4733` does +not interfere with one for `eventID=4624`. ### 2. Backends (`EventDataStructure`) @@ -239,10 +239,10 @@ class MyDetector(TrackerDetector): super().__init__(name=name, config=config) def train(self, input_): - self._ingest(input_, variables={...}, event_id=input_["EventID"]) + self._ingest(input_, variables={...}, event_id=input_["eventID"]) def detect(self, input_, output_): - tracker = self.persistency.get_events_data().get(input_["EventID"]) + tracker = self.persistency.get_events_data().get(input_["eventID"]) # compare against tracker to produce alerts ``` diff --git a/docs/contribution.md b/docs/contribution.md index 64b81925..dc6ccfa0 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -11,9 +11,13 @@ Bug reports are very helpful to us in improving the software, and therefore, the at least the following information in a bug report: 1. Description of the bug. Describe the problem clearly. -2. Steps to reproduce. With the following configuration, go to.., click.., see error -3. Expected behavior. What should happen? -4. Environment. What was the environment for the test(version, browser, etc..) +2. Steps to reproduce. A minimal code snippet, plus the configuration (YAML or dict) and, if + possible, a few log lines that trigger the problem. +3. Expected behavior. What should happen, and what happens instead (including the full error + message, if any)? +4. Environment. Your Python version, your DetectMate version + (`python -c "import detectmatelibrary; print(detectmatelibrary.__version__)"`) and your + operating system. *Please don't include any private/sensitive information in your issue! For reporting security-related issues, see [SECURITY.md](https://github.com/ait-detectmate/DetectMateLibrary/blob/main/SECURITY.md)* diff --git a/docs/detectors.md b/docs/detectors.md index f3d9e5fe..b1eca29d 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -8,7 +8,7 @@ Detectors process structured logs from Parsers and emit alerts when anomalies ar | **Input** | [ParserSchema](schemas.md) | Structured log | | **Output** | [DetectorSchema](schemas.md)| Alert / finding | -This document describes the minimal API, implementation guidance, a short example detector and a unit test pattern. +This document describes the minimal API, implementation guidance and a short example detector. ## CoreDetector: minimal API @@ -145,7 +145,7 @@ example configurations set `auto_config: false` anyway, to make this explicit. I `auto_config` is `True` *and* `data_use_configure` is set, the configure phase replaces the `events` block with its own selection. -When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables, that are not bound to events (`header_variables` can but don't have to be event bound): +When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables that are not bound to events (`header_variables` can but don't have to be event bound): ```yaml detectors: @@ -172,14 +172,14 @@ detectors: ### Common parameters (all detectors) -There are some parameters, that **every** detector inherits from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on. +Some parameters are inherited by **every** detector from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The parameters that are **specific** to a detector are explained on that detector's own page. ???+ note "Top level" | Field | Type | Default | Description | |---|---|---|---| - | `auto_config` | boolean | True | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | Events configuration dict keyed by event_id. | | `global` | object | {} | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -188,8 +188,8 @@ There are some parameters, that **every** detector inherits from `CoreDetectorCo | Field | Type | Default | Description | |---|---|---|---| | `start_id` | integer | 10 | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | @@ -207,13 +207,13 @@ The detectors that keep their model in persistency stores ([New Event](detectors | Field | Type | Default | Description | |---|---|---|---| - | `persist` | object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" | Field | Type | Default | Description | |---|---|---|---| - | `allow_fed` | boolean | False | Allow to do the federation | + | `allow_fed` | boolean | False | Builds the stores needed to take part in federation. | ### Per-variable model detectors @@ -255,7 +255,7 @@ The two neural detectors ([DeepLog](detectors/deeplog.md), [LogBERT](detectors/l ### Configuration semantics (preliminary) -**`events` key**: the key is the `EventID` (or `event_id`) to monitor: an integer as the parser emits it (see the [Template Matcher](parsers/template_matcher.md) docs for how the EventID is assigned), or a name (see [Named EventIDs and variables](#named-eventids-and-variables) below). +**`events` key**: the key is the `eventID` (or `event_id`) to monitor: an integer as the parser emits it (see the [Template Matcher](parsers/template_matcher.md) docs for how the EventID is assigned), or a name (see [Named EventIDs and variables](#named-eventids-and-variables) below). **`global` key** - This one has a similar functionality as the `events` key but refers to variables, that are not bound to events (thus can only contain `header_variables`). @@ -272,7 +272,7 @@ pid=<*> uid=<*> auid=<*> ses=<*> msg='op=<*> acct=<*> exe=<*> hostname=<*> addr= ### Named EventIDs and variables -Instead of numbers, the `events` block can use names: a name for the `EventID`, and +Instead of numbers, the `events` block can use names: a name for the `eventID`, and the name of a wildcard for `variables[].pos`. The names come from the [Template Matcher](parsers/template_matcher.md)'s template file: @@ -285,10 +285,10 @@ the name of a wildcard for `variables[].pos`. The names come from the --8<-- "docs/examples/data/named_templates.csv" ``` -The parser still emits numbers (`EventID` 1 and position 0 for `` in the second +The parser still emits numbers (`eventID` 1 and position 0 for `` in the second template), so the names must be translated before the detector runs. Do this once, after creating both components, with `compile_detector_config()`. **Without this step -the detector monitors nothing**; it only logs a warning that the named `EventID` was +the detector monitors nothing**; it only logs a warning that the named `eventID` was never observed in training. ```python @@ -368,8 +368,8 @@ axes: | `slope_index` | change centroid vs. `slope_threshold` | index positions | | `slope_time` | change centroid vs. `slope_threshold` | normalized timestamps | -Any subset of the four may be enabled, and any single one may stand alone. The -default (`index` alone) is the historical behaviour: each segment's mean rate +Any subset of the four may be enabled, and any single one may stand alone. By +default only `index` is enabled: each segment's mean rate of change is compared against its entry in `segment_thresholds` (four segments by default, one threshold per segment), and the segments are **equal-count**: each holds the same number of observations, regardless of how much time they @@ -379,7 +379,7 @@ traffic looks stable, because the flood supplies enough samples to dominate the later segments. Enabling `time` cuts the same segments at **equal durations** instead, so each segment covers the same amount of wall-clock time; the detector then needs an event time per record, which it reads from the log's -named variables (`logFormatVariables`, i.e. the fields declared in the parser's +named variables (`headerVariables`, i.e. the fields declared in the parser's `log_format`) under the name given by `timestamp_variable`. `slope_index` and `slope_time` ask a different question (whether the change centroid sits early or late in the series), on the index axis and the time axis respectively. @@ -410,12 +410,11 @@ detectors: timestamp_format: "%y%m%d %H%M%S" ``` -Defaults reproduce the historical behaviour exactly: `index: True`, the other -three `False`, `segment_thresholds: [1.1, 0.3, 0.1, 0.01]`, `decision: consensus`, +The defaults are `index: True`, the other three `False`, +`segment_thresholds: [1.1, 0.3, 0.1, 0.01]`, `decision: consensus` and `slope_threshold: -0.05`. A config that sets nothing under `classification` -classifies identically to before these changes on every series with at least -four observations; the one exception is the segment floor described under -"Fields" below. +uses exactly these. A variable with fewer observations than segments is +classified `INSUFFICIENT_DATA`; see the segment floor under "Fields" below. #### The decision rule @@ -448,7 +447,7 @@ All of these live in the detector's `auto_config_params` block. | `use_stable_vars` | `bool` | `true` | Include variables classified `STABLE` in the generated configuration. | | `use_static_vars` | `bool` | `true` | Include variables classified `STATIC`. Defaults to `false` on `NewValueComboDetector`. | | `classification` | `ClassificationMethods` | see below | Which classification methods run and how their verdicts combine. | -| `timestamp_variable` | `str \| null` | `null` | Name of the field in `logFormatVariables` holding the record's event time. Required for `time` and `slope_time` to have any effect. Only named log-format fields are consulted, never the positional `variables` list. | +| `timestamp_variable` | `str \| null` | `null` | Name of the field in `headerVariables` holding the record's event time. Required for `time` and `slope_time` to have any effect. Only named log-format fields are consulted, never the positional `variables` list. | | `timestamp_format` | `str \| null` | `null` | Explicit [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) pattern for parsing that field. When unset, `TimeFormatHandler` auto-detects the format (ISO 8601, Apache, syslog, numeric epoch seconds/milliseconds, and other common layouts). | Set `timestamp_format` when the source uses a layout the auto-detection does not diff --git a/docs/detectors/bigram_frequency.md b/docs/detectors/bigram_frequency.md index 5834fc32..434c4e22 100644 --- a/docs/detectors/bigram_frequency.md +++ b/docs/detectors/bigram_frequency.md @@ -50,10 +50,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | bigram_frequency_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -63,13 +63,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | `default_freqs` | boolean | False | specific | Initialize the probabilities with default values from https://github.com/markbaggett/freq. | | `skip_repetitions` | boolean | True | specific | Only use distinct values for character-pair counting. This counteracts the problem of imbalanced word frequencies that distort the frequency table generated in a single run. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/charset.md b/docs/detectors/charset.md index 11f9de65..dfc97c37 100644 --- a/docs/detectors/charset.md +++ b/docs/detectors/charset.md @@ -1,6 +1,6 @@ # Charset Detector -The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect novelty, configuration drift, or the appearance of new actors in the environment. +The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect malformed or injected values in fields with a predictable format, for example shell characters such as `;` in a user name. ## In/out @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed characters per monitored field and emits an alert when a character not present in the set is seen for the first time (subject to configuration). +This detector keeps the set of characters observed during training for each monitored field and emits an alert whenever a value contains a character that is not in the set. Alerts don't add the character to the set, so the same unseen character raises an alert every time it appears. ## Example @@ -50,23 +50,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | charset_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/combo.md b/docs/detectors/combo.md index fca4ae37..ebd26f6a 100644 --- a/docs/detectors/combo.md +++ b/docs/detectors/combo.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed combination of values per monitored fields and emits an alert when a combination is not present in the set seen for the first time (subject to configuration). +This detector keeps the set of value combinations observed during training for each group of monitored fields and emits an alert whenever a combination that is not in the set appears. Alerts don't add the combination to the set, so the same unseen combination raises an alert every time it appears. A combination needs at least two variables in the same instance: with a single variable the detector behaves like the [New Value Detector](new_value.md). @@ -54,23 +54,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | new_value_combo_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/deeplog.md b/docs/detectors/deeplog.md index 16a0d78f..4cabdbae 100644 --- a/docs/detectors/deeplog.md +++ b/docs/detectors/deeplog.md @@ -50,7 +50,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | deeplog_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -59,8 +59,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/ecvc_detector.md b/docs/detectors/ecvc_detector.md index af292b69..dcfe68d5 100644 --- a/docs/detectors/ecvc_detector.md +++ b/docs/detectors/ecvc_detector.md @@ -52,10 +52,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | ecvc_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -66,11 +66,11 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | `seed` | integer | 0 | specific | Random seed used to shuffle count vectors into train/validation splits. | | `threshold_method` | string | mean | specific | Method used to derive the anomaly threshold from the validation split: 'mean' averages the distance scores, 'default' uses a fixed threshold of 0. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | diff --git a/docs/detectors/event_sequence.md b/docs/detectors/event_sequence.md index a8751bf4..374283b4 100644 --- a/docs/detectors/event_sequence.md +++ b/docs/detectors/event_sequence.md @@ -78,10 +78,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | event_sequence_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -89,13 +89,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `fixed_window_size` | integer, null | None | specific | Length of the sliding EventID window. A window whose exact EventID sequence was not seen during training is reported as an anomaly. When set it overrides the `auto_config_params` window range and skips auto-configuration; auto-configuration writes its own choice here. While it is None the detector is unconfigured and neither trains nor alerts. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/logbert.md b/docs/detectors/logbert.md index 510a8eac..aecaf0a7 100644 --- a/docs/detectors/logbert.md +++ b/docs/detectors/logbert.md @@ -50,7 +50,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | logbert_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -59,8 +59,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/new_event.md b/docs/detectors/new_event.md index 03ccf041..c2c60022 100644 --- a/docs/detectors/new_event.md +++ b/docs/detectors/new_event.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed event IDs and emits an alert when an event ID not present in the set is seen for the first time (subject to configuration). +This detector keeps the set of event IDs observed during training and emits an alert whenever an event ID that is not in the set appears. Alerts don't add the event ID to the set, so the same unseen event ID raises an alert every time it appears. ## Example @@ -50,21 +50,21 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | new_event_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | diff --git a/docs/detectors/new_value.md b/docs/detectors/new_value.md index 62a5e10d..95157284 100644 --- a/docs/detectors/new_value.md +++ b/docs/detectors/new_value.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed values per monitored field and emits an alert when a value not present in the set is seen for the first time (subject to configuration). +This detector keeps the set of values observed during training for each monitored field and emits an alert whenever a value that is not in the set appears. Alerts don't add the value to the set, so the same unseen value raises an alert every time it appears. ## Example @@ -50,23 +50,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | new_value_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/random_detector.md b/docs/detectors/random_detector.md index 64840cc0..80e05d44 100644 --- a/docs/detectors/random_detector.md +++ b/docs/detectors/random_detector.md @@ -50,7 +50,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | random_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -59,8 +59,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/rule_based.md b/docs/detectors/rule_based.md index ace49d2e..7be844ae 100644 --- a/docs/detectors/rule_based.md +++ b/docs/detectors/rule_based.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -The detector analyzes parsed logs one by one and checks which rules are triggered. When alerts are produced, the triggered rules and their messages are recorded in the `alertsObtain` field of the output schema. The `score` field contains the number of rules that triggered. +The detector analyzes parsed logs one by one and checks which rules are triggered. When alerts are produced, the triggered rules and their messages are recorded in the `alertDetails` field of the output schema. The `score` field contains the number of rules that triggered. ### Available rules @@ -66,7 +66,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | rule_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -76,8 +76,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `rules` | array | [{'rule': 'R001 - TemplateNotFound'}, {'rule': 'R003 - CheckForExceptions'}, {'rule': 'R004 - ErrorLevelFound'}] | specific | List of rules to evaluate, each a dict with a 'rule' name and an optional 'args' list. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/scvs_detector.md b/docs/detectors/scvs_detector.md index ce75278a..5918394d 100644 --- a/docs/detectors/scvs_detector.md +++ b/docs/detectors/scvs_detector.md @@ -52,10 +52,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | scvs_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -63,11 +63,11 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `window_size` | integer | 10 | specific | Length of the event-ID window a count vector is built over. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | diff --git a/docs/detectors/value_range.md b/docs/detectors/value_range.md index 21220d98..60040bcd 100644 --- a/docs/detectors/value_range.md +++ b/docs/detectors/value_range.md @@ -50,10 +50,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | value_range_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -61,13 +61,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `ignore_non_numerical_val` | boolean | True | specific | Drop non-numeric values instead of raising when a configured variable cannot be cast to a number. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/development.md b/docs/development.md index 97ba47ce..00a5195d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -20,9 +20,9 @@ uv sync --dev ## Use prek to run code checks -Every code contributer must use [`prek`](https://github.com/j178/prek) to run basic checks at commit time. +Every code contributor must use [`prek`](https://github.com/j178/prek) to run basic checks at commit time. `prek` is configured via the existing `.pre-commit-config.yaml` -and can be installed as part of the `dev` extras. To ensure pre-commit hooks run before each commit, run: +and is installed with the `dev` dependency group. To ensure pre-commit hooks run before each commit, run: ```bash uv run prek install @@ -66,10 +66,10 @@ The `configure()` method ingests all available variables (not just configured on ```python def configure(self, input_): self.auto_conf_persistency.ingest_event( - event_id=input_["EventID"], + event_id=input_["eventID"], event_template=input_["template"], variables=input_["variables"], - named_variables=input_["logFormatVariables"], + named_variables=input_["headerVariables"], ) ``` diff --git a/docs/examples/alert_aggregators/basic_concat.py b/docs/examples/alert_aggregators/basic_concat.py index ce66e7c6..34ca5b4d 100644 --- a/docs/examples/alert_aggregators/basic_concat.py +++ b/docs/examples/alert_aggregators/basic_concat.py @@ -2,37 +2,28 @@ # --8<-- [start:example_1] from detectmatelibrary.alert_aggregation.basic_concat import BasicConcatAggregation -import detectmatelibrary.schemas as schemas +import detectmatelibrary.schemas as sc +import yaml -aggregations_config = { - "alert_aggregators": { - "BasicConcatAggregator": { - "method_type": "basic_concat_aggregator", - "buffer_size": 3, - "deduplicate_values": False, - "auto_config": False, - } - } -} +with open("docs/examples/alert_aggregators/basic_concat.yaml") as f: + config = yaml.safe_load(f) -input_ = schemas.DetectorSchema({ +input_ = sc.DetectorSchema({ "detectorID": "1", "detectorType": "dummy", "alertID": "2", - "detectionTimestamp": 0, + "detectedTimestamp": 0, "logIDs": ["logID1"], "score": 0.2, - "extractedTimestamps": [-1], + "logTimestamps": [-1], "description": "hello there", "receivedTimestamp": 1, - "alertsObtain": {"99 problems": "but logs aint one"} + "alertDetails": {"99 problems": "but logs aint one"} }) -alert_aggregator = BasicConcatAggregation( - "BasicConcatAggregator", aggregations_config -) +alert_aggregator = BasicConcatAggregation("BasicConcatAggregator", config) alert_aggregator.process(input_) # --8<-- [end:example_1] diff --git a/docs/examples/alert_aggregators/basic_concat.yaml b/docs/examples/alert_aggregators/basic_concat.yaml new file mode 100644 index 00000000..9d653635 --- /dev/null +++ b/docs/examples/alert_aggregators/basic_concat.yaml @@ -0,0 +1,6 @@ +alert_aggregators: + BasicConcatAggregator: + method_type: basic_concat_aggregator + auto_config: false + params: + buffer_size: 3 # aggregate the 3 most recent alerts diff --git a/docs/examples/alert_aggregators/logid_count.py b/docs/examples/alert_aggregators/logid_count.py new file mode 100644 index 00000000..cb9d012a --- /dev/null +++ b/docs/examples/alert_aggregators/logid_count.py @@ -0,0 +1,35 @@ +# --8<-- [start:example_1] +from detectmatelibrary.alert_aggregation.logid_count import LogIDCountAggegation + +import detectmatelibrary.schemas as schemas + +import yaml + + +with open("docs/examples/alert_aggregators/logid_count.yaml") as f: + config = yaml.safe_load(f) + + +def generate_schema(log_id: str | list[str]) -> schemas.DetectorSchema: + log_id = log_id if isinstance(log_id, list) else [log_id] + return schemas.DetectorSchema({ + "detectorID": "1", + "detectorType": "dummy", + "alertID": "2", + "detectedTimestamp": 0, + "logIDs": log_id, + "score": 0.2, + "logTimestamps": [-1], + "description": "hello there", + "receivedTimestamp": 1, + "alertDetails": {"99 problems": "but logs aint one"} + }) + + +alert_aggregator = LogIDCountAggegation("LogIDCountAggregator", config=config) + +print(alert_aggregator.process(generate_schema("a")) is None) # True +print(alert_aggregator.process(generate_schema("b")) is None) # True +print(alert_aggregator.process(generate_schema("a")) is None) # False + +# --8<-- [end:example_1] diff --git a/docs/examples/alert_aggregators/logid_count.yaml b/docs/examples/alert_aggregators/logid_count.yaml new file mode 100644 index 00000000..bc646f55 --- /dev/null +++ b/docs/examples/alert_aggregators/logid_count.yaml @@ -0,0 +1,5 @@ +alert_aggregators: + LogIDCountAggregator: + method_type: "logid_count_aggregator" + max_count: 2 + max_memory: 5000 diff --git a/docs/examples/alert_aggregators/time_delta.py b/docs/examples/alert_aggregators/time_delta.py new file mode 100644 index 00000000..2b01157a --- /dev/null +++ b/docs/examples/alert_aggregators/time_delta.py @@ -0,0 +1,53 @@ +# --8<-- [start:example_1] +import yaml + +from detectmatelibrary.alert_aggregation.time_delta import ( + TimeDeltaAggregation, +) +from detectmatelibrary.schemas import AggregateSchema, DetectorSchema + +with open("docs/examples/alert_aggregators/time_delta.yaml") as f: + config = yaml.safe_load(f) + + +def make_alert(alert_id: str, timestamp: float) -> DetectorSchema: + return DetectorSchema( + { + "detectorID": "detector", + "detectorType": "dummy", + "alertID": alert_id, + "detectedTimestamp": 0, + "logIDs": [f"log-{alert_id}"], + "score": 0.0, + "logTimestamps": [timestamp], + "description": "test alert", + "receivedTimestamp": 0, + "alertDetails": {}, + } + ) + + +aggregator = TimeDeltaAggregation(config) +first = make_alert("a", 0) +second = make_alert("b", 6) +third = make_alert("c", 9) +fourth = make_alert("d", 15) + +first_output = AggregateSchema() +aggregator.aggregate_alerts(first, first_output) + +second_output = AggregateSchema() +aggregator.aggregate_alerts(second, second_output) + +third_output = AggregateSchema() +aggregator.aggregate_alerts(third, third_output) + +fourth_output = AggregateSchema() +aggregator.aggregate_alerts(fourth, fourth_output) + +print(first_output["alertDetails"]["a"] == "0") # True +print(second_output["alertDetails"]["b"] == "1") # True +print(third_output["alertDetails"]["c"] == "1") # True +print(fourth_output["alertDetails"]["d"] == "2") # True + +# --8<-- [end:example_1] diff --git a/docs/examples/alert_aggregators/time_delta.yaml b/docs/examples/alert_aggregators/time_delta.yaml new file mode 100644 index 00000000..b00bb97b --- /dev/null +++ b/docs/examples/alert_aggregators/time_delta.yaml @@ -0,0 +1,4 @@ +alert_aggregators: + TimeDeltaAggregator: + method_type: time_delta_aggregator + delta: 4.0 diff --git a/docs/examples/config/update.py b/docs/examples/config/update.py index 12b8d8aa..4abb5b06 100644 --- a/docs/examples/config/update.py +++ b/docs/examples/config/update.py @@ -26,10 +26,16 @@ from detectmatelibrary.common.core import CoreConfig from detectmatelibrary.common.detector import CoreDetectorConfig from detectmatelibrary.common.parser import CoreParserConfig +from detectmatelibrary.common.alert_aggregator import CoreAlertAggregatorConfig +from detectmatelibrary.base_alert_aggregation.density_clustering import DensityClusteringAggregationConfig +from detectmatelibrary.alert_aggregation.basic_concat import BasicConcatAggregationConfig +from detectmatelibrary.alert_aggregation.time_delta import TimeDeltaAggregationConfig +from detectmatelibrary.alert_aggregation.logid_count import LogIDCountAggegationConfig from detectmatelibrary.base_detectors import TrackerDetectorConfig from detectmatelibrary.base_detectors import VariableDetectorConfig from detectmatelibrary.base_detectors import DeepLearningDetectorConfig + from typing import Any @@ -184,6 +190,36 @@ def update_shared_args_parsers(doc_path: str) -> None: raise Exception(f"While updating {doc_path} -> {str(e)}") +def update_shared_args_aggregation(doc_path: str) -> None: + """Fill docs/alert_aggregation.md's tables of parameters shared across several + alert_aggregator: fields every parser has (CoreAlertAggregatorConfig).""" + try: + with open(doc_path, "r") as f: + docs = f.readlines() + + docs = append_docs( + docs=docs, + start_cmd="\n", + end_cmd="\n", + add=get_arguments(CoreAlertAggregatorConfig().get_docs(), with_scope=False), + ) + docs = append_docs( + docs=docs, + start_cmd="\n", + end_cmd="\n", + add=get_arguments( + _family_rows( + DensityClusteringAggregationConfig(), base=CoreAlertAggregatorConfig + ), + with_scope=False, + ), + ) + with open(doc_path, "w") as f: + f.writelines(docs) + except Exception as e: + raise Exception(f"While updating {doc_path} -> {str(e)}") + + # %% Documentation update # # Every config below drives its own doc page: whenever a Field is added, @@ -244,11 +280,33 @@ def update_shared_args_parsers(doc_path: str) -> None: ), ] +AGGREGATOR_DOCS: list[tuple[CoreConfig, type[CoreConfig], str]] = [ + ( + BasicConcatAggregationConfig(), + CoreAlertAggregatorConfig, + "docs/alert_aggregators/basic_concatenation.md" + ), + ( + LogIDCountAggegationConfig(), + CoreAlertAggregatorConfig, + "docs/alert_aggregators/logid_count.md" + ), + ( + TimeDeltaAggregationConfig(), + CoreAlertAggregatorConfig, + "docs/alert_aggregators/time_delta.md" + ) +] + update_shared_args_detectors("docs/detectors.md") update_shared_args_parsers("docs/parsers.md") +update_shared_args_aggregation("docs/alert_aggregator.md") for detector_config, shared_base, doc_path in DETECTOR_DOCS: update_docs(detector_config, doc_path=doc_path, shared_base=shared_base) for parser_config, shared_base, doc_path in PARSER_DOCS: update_docs(parser_config, doc_path=doc_path, shared_base=shared_base) + +for aggr_config, shared_base, doc_path in AGGREGATOR_DOCS: + update_docs(aggr_config, doc_path=doc_path, shared_base=shared_base) diff --git a/docs/examples/detectors/bigram_frequency.py b/docs/examples/detectors/bigram_frequency.py index 0291faad..d6c2d183 100644 --- a/docs/examples/detectors/bigram_frequency.py +++ b/docs/examples/detectors/bigram_frequency.py @@ -11,7 +11,7 @@ def failed_login(user: str) -> schemas.ParserSchema: """What a parser emits for 'Failed password for from 10.0.0.1 port 22'.""" - return schemas.ParserSchema({"EventID": 1, "variables": [user, "10.0.0.1", "22"]}) + return schemas.ParserSchema({"eventID": 1, "variables": [user, "10.0.0.1", "22"]}) # the first 4 logs train the detector (data_use_training: 4) @@ -20,6 +20,6 @@ def failed_login(user: str) -> schemas.ParserSchema: print(detector.process(failed_login("alia"))) # None: familiar character pairs alert = detector.process(failed_login("xq7zv9k")) # random-looking user name -print(dict(alert["alertsObtain"])) +print(dict(alert["alertDetails"])) # {'EventID 1 - user': 'Bigram frequency anomaly with value xq7zv9k, critical_val 0.0 and threshold 0.05.'} # --8<-- [end:example] diff --git a/docs/examples/detectors/charset.py b/docs/examples/detectors/charset.py index 78f8abe8..e81f4982 100644 --- a/docs/examples/detectors/charset.py +++ b/docs/examples/detectors/charset.py @@ -11,7 +11,7 @@ def failed_login(user: str) -> schemas.ParserSchema: """What a parser emits for 'Failed password for from 10.0.0.1 port 22'.""" - return schemas.ParserSchema({"EventID": 1, "variables": [user, "10.0.0.1", "22"]}) + return schemas.ParserSchema({"eventID": 1, "variables": [user, "10.0.0.1", "22"]}) # the first 3 logs train the detector (data_use_training: 3) @@ -20,6 +20,6 @@ def failed_login(user: str) -> schemas.ParserSchema: print(detector.process(failed_login("carla"))) # None: only known characters alert = detector.process(failed_login("bob;rm -rf")) -print(dict(alert["alertsObtain"])) +print(dict(alert["alertDetails"])) # {'EventID 1 - user': "Unknown character(s): ' ', '-', ';', 'f', 'm'"} # --8<-- [end:example] diff --git a/docs/examples/detectors/combo.py b/docs/examples/detectors/combo.py index d7f5a544..ac490bbf 100644 --- a/docs/examples/detectors/combo.py +++ b/docs/examples/detectors/combo.py @@ -11,7 +11,7 @@ def login(user: str, ip: str) -> schemas.ParserSchema: """What a parser emits for 'Accepted password for from port 22'.""" - return schemas.ParserSchema({"EventID": 0, "variables": [user, ip, "22"]}) + return schemas.ParserSchema({"eventID": 0, "variables": [user, ip, "22"]}) # the first 3 logs train the detector (data_use_training: 3) @@ -21,6 +21,6 @@ def login(user: str, ip: str) -> schemas.ParserSchema: print(detector.process(login("bob", "10.0.0.2"))) # None: known user/IP pair # alice and 10.0.0.2 are both known, but never appeared together alert = detector.process(login("alice", "10.0.0.2")) -print(dict(alert["alertsObtain"])) +print(dict(alert["alertDetails"])) # {"EventID 0 - ('user', 'src_ip')": "Unknown value combination: ('alice', '10.0.0.2')"} # --8<-- [end:example] diff --git a/docs/examples/detectors/deeplog.py b/docs/examples/detectors/deeplog.py index 78d8a03e..7155b35b 100644 --- a/docs/examples/detectors/deeplog.py +++ b/docs/examples/detectors/deeplog.py @@ -10,7 +10,7 @@ def event(event_id: int) -> schemas.ParserSchema: """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"EventID": event_id}) + return schemas.ParserSchema({"eventID": event_id}) # a repeating workflow of 4 events; the model trains once 60 windows are collected diff --git a/docs/examples/detectors/ecvc_detector.py b/docs/examples/detectors/ecvc_detector.py index 09eaf186..fd5b458a 100644 --- a/docs/examples/detectors/ecvc_detector.py +++ b/docs/examples/detectors/ecvc_detector.py @@ -10,7 +10,7 @@ def event(event_id: int) -> schemas.ParserSchema: """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"EventID": event_id}) + return schemas.ParserSchema({"eventID": event_id}) # normal session: 0 = login accepted, 1 = session opened, 2 = session closed. diff --git a/docs/examples/detectors/event_sequence.py b/docs/examples/detectors/event_sequence.py index 9a9027ea..54a4b5d5 100644 --- a/docs/examples/detectors/event_sequence.py +++ b/docs/examples/detectors/event_sequence.py @@ -1,4 +1,6 @@ # --8<-- [start:example] +from itertools import count + import yaml from detectmatelibrary import schemas from detectmatelibrary.detectors.event_sequence_detector import EventSequenceDetector @@ -8,9 +10,13 @@ detector = EventSequenceDetector(name="LoginSequenceDetector", config=config) +log_ids = count() + + def event(event_id: int) -> schemas.ParserSchema: - """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"EventID": event_id}) + """A parsed log with a running logID; only its EventID (the matched + template) matters for detection.""" + return schemas.ParserSchema({"eventID": event_id, "logID": str(next(log_ids))}) # normal session: 0 = login accepted, 1 = session opened, 2 = session closed. @@ -23,8 +29,8 @@ def event(event_id: int) -> schemas.ParserSchema: detector.process(event(0)) alert = detector.process(event(2)) # session closed without being opened -print(dict(alert["alertsObtain"])) -# {'Sequence (0, 2)': 'EventID sequence of length 2 ending at logID was not seen during training.'} +print(dict(alert["alertDetails"])) +# {'Sequence (0, 2)': 'EventID sequence of length 2 ending at logID 10 was not seen during training.'} print(detector.get_known_sequences()) # {(0, 1), (1, 2), (2, 0)} # --8<-- [end:example] diff --git a/docs/examples/detectors/logbert.py b/docs/examples/detectors/logbert.py index b460d1c6..8fed4c03 100644 --- a/docs/examples/detectors/logbert.py +++ b/docs/examples/detectors/logbert.py @@ -10,7 +10,7 @@ def event(event_id: int) -> schemas.ParserSchema: """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"EventID": event_id}) + return schemas.ParserSchema({"eventID": event_id}) # a repeating workflow of 4 events; the model trains once 60 windows are collected diff --git a/docs/examples/detectors/named_events.py b/docs/examples/detectors/named_events.py index 1b8cbece..1aa70302 100644 --- a/docs/examples/detectors/named_events.py +++ b/docs/examples/detectors/named_events.py @@ -35,7 +35,7 @@ ] for line in lines: alert = detector.process(parser.process(schemas.LogSchema({"log": line}))) - print(dict(alert["alertsObtain"]) if alert else None) + print(dict(alert["alertDetails"]) if alert else None) # None # None # {'EventID 1 - user': "Unknown value: 'mallory'"} diff --git a/docs/examples/detectors/new_event.py b/docs/examples/detectors/new_event.py index 22c44bd3..67dd0dff 100644 --- a/docs/examples/detectors/new_event.py +++ b/docs/examples/detectors/new_event.py @@ -10,7 +10,7 @@ def event(event_id: int) -> schemas.ParserSchema: """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"EventID": event_id}) + return schemas.ParserSchema({"eventID": event_id}) # the first 3 logs train the detector (data_use_training: 3): @@ -20,5 +20,5 @@ def event(event_id: int) -> schemas.ParserSchema: print(detector.process(event(1))) # None: known event alert = detector.process(event(2)) # a template never seen during training -print(dict(alert["alertsObtain"])) # {'EventID 2 - {}': "Unknown event ID: '2'"} +print(dict(alert["alertDetails"])) # {'EventID 2 - {}': "Unknown event ID: '2'"} # --8<-- [end:example] diff --git a/docs/examples/detectors/new_value.py b/docs/examples/detectors/new_value.py index d9ecb77c..1d64461f 100644 --- a/docs/examples/detectors/new_value.py +++ b/docs/examples/detectors/new_value.py @@ -11,7 +11,7 @@ def login(user: str, ip: str) -> schemas.ParserSchema: """What a parser emits for 'Accepted password for from port 22'.""" - return schemas.ParserSchema({"EventID": 0, "variables": [user, ip, "22"]}) + return schemas.ParserSchema({"eventID": 0, "variables": [user, ip, "22"]}) # the first 3 logs train the detector (data_use_training: 3) @@ -20,5 +20,5 @@ def login(user: str, ip: str) -> schemas.ParserSchema: print(detector.process(login("bob", "10.0.0.9"))) # None: bob is a known user alert = detector.process(login("mallory", "10.0.0.1")) -print(dict(alert["alertsObtain"])) # {'EventID 0 - user': "Unknown value: 'mallory'"} +print(dict(alert["alertDetails"])) # {'EventID 0 - user': "Unknown value: 'mallory'"} # --8<-- [end:example] diff --git a/docs/examples/detectors/random_detector.py b/docs/examples/detectors/random_detector.py index 90cac0b3..94d3a81e 100644 --- a/docs/examples/detectors/random_detector.py +++ b/docs/examples/detectors/random_detector.py @@ -8,7 +8,7 @@ config = yaml.safe_load(f) detector = RandomDetector(name="RandomDetector", config=config) -login = schemas.ParserSchema({"EventID": 0, "variables": ["alice", "10.0.0.1", "22"]}) +login = schemas.ParserSchema({"eventID": 0, "variables": ["alice", "10.0.0.1", "22"]}) np.random.seed(0) # only to make this example reproducible diff --git a/docs/examples/detectors/rule_based.py b/docs/examples/detectors/rule_based.py index 4740114c..3fe9339d 100644 --- a/docs/examples/detectors/rule_based.py +++ b/docs/examples/detectors/rule_based.py @@ -10,7 +10,7 @@ def parsed(event_id: int, log: str, level: str = "INFO") -> schemas.ParserSchema: """A parsed log; EventID -1 means the parser found no matching template.""" - return schemas.ParserSchema({"EventID": event_id, "log": log, "logFormatVariables": {"Level": level}}) + return schemas.ParserSchema({"eventID": event_id, "log": log, "headerVariables": {"Level": level}}) logs = [ @@ -21,7 +21,7 @@ def parsed(event_id: int, log: str, level: str = "INFO") -> schemas.ParserSchema ] for log in logs: alert = detector.process(log) - print(dict(alert["alertsObtain"]) if alert else None) + print(dict(alert["alertDetails"]) if alert else None) # None # {'R001 - TemplateNotFound': 'No template found by parser'} # {'R002 - SpecificKeyword': "Found word 'sudo' in the logs"} diff --git a/docs/examples/detectors/scvs_detector.py b/docs/examples/detectors/scvs_detector.py index cddacbeb..4fbda79e 100644 --- a/docs/examples/detectors/scvs_detector.py +++ b/docs/examples/detectors/scvs_detector.py @@ -10,7 +10,7 @@ def event(event_id: int) -> schemas.ParserSchema: """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"EventID": event_id}) + return schemas.ParserSchema({"eventID": event_id}) # normal session: 0 = login accepted, 1 = session opened, 2 = session closed. diff --git a/docs/examples/detectors/value_range.py b/docs/examples/detectors/value_range.py index 38f70b9b..0bfb3c80 100644 --- a/docs/examples/detectors/value_range.py +++ b/docs/examples/detectors/value_range.py @@ -11,7 +11,7 @@ def download(path: str, bytes_read: int) -> schemas.ParserSchema: """What a parser emits for 'close bytes read written 0'.""" - return schemas.ParserSchema({"EventID": 2, "variables": [path, str(bytes_read), "0"]}) + return schemas.ParserSchema({"eventID": 2, "variables": [path, str(bytes_read), "0"]}) # the first 3 logs train the detector (data_use_training: 3) @@ -20,5 +20,5 @@ def download(path: str, bytes_read: int) -> schemas.ParserSchema: print(detector.process(download("/srv/d.pdf", 200))) # None: inside the learned range alert = detector.process(download("/srv/db.dump", 99999)) -print(dict(alert["alertsObtain"])) # {'EventID 2 - bytes_read': "Out of range value: '99999' (100 - 250)"} +print(dict(alert["alertDetails"])) # {'EventID 2 - bytes_read': "Out of range value: '99999' (100 - 250)"} # --8<-- [end:example] diff --git a/docs/examples/others/pipeline.py b/docs/examples/others/pipeline.py new file mode 100644 index 00000000..045e1648 --- /dev/null +++ b/docs/examples/others/pipeline.py @@ -0,0 +1,46 @@ +# --8<-- [start:example_1] +from detectmatelibrary.alert_aggregation.logid_count import LogIDCountAggegationConfig + +from detectmatelibrary.parsers.tree_matcher import TemplateCppTreeMatcherConfig + +from detectmatelibrary.detectors.charset_detector import CharsetDetectorConfig + +from detectmatelibrary.pipeline import PipelineConfig, Pipeline +import detectmatelibrary.schemas as schemas + +config = PipelineConfig( + do_parsing=True, # Use the parser + do_aggregation=True, # Use the aggregator + parser=TemplateCppTreeMatcherConfig( + path_templates="docs/examples/data/tutorial_templates.txt", + log_format="type= msg=audit(