diff --git a/docs/alert_aggregator.md b/docs/alert_aggregator.md index c0deb337..0c2edd83 100644 --- a/docs/alert_aggregator.md +++ b/docs/alert_aggregator.md @@ -89,8 +89,8 @@ There are some parameters, that **every** alert aggregator inherits from `CoreAl | Field | Type | Default | Description | |---|---|---|---| | `start_id` | integer | 10 | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/alert_aggregators/basic_concatenation.md b/docs/alert_aggregators/basic_concatenation.md index 6334c7ef..bca3916b 100644 --- a/docs/alert_aggregators/basic_concatenation.md +++ b/docs/alert_aggregators/basic_concatenation.md @@ -1,6 +1,11 @@ # Basic Concat Alert Aggregation -The basic concat alert aggregation approach aggregates x number of alerts into one output schema. +The Basic Concat alert aggregation combines the most recent alerts into one aggregated record by +concatenating their IDs and timestamps. + +## In/out + +Input and output schemas in the pipeline ## In/out @@ -9,16 +14,20 @@ The basic concat alert aggregation approach aggregates x number of alerts into o | **Input** | [DetectorSchema](../schemas.md) | Alerts from detectors | | **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts | +## Description + +The aggregator keeps a sliding window of the last `buffer_size` alerts. The first aggregated record +is emitted once the window is full, and then one record for every new alert, so consecutive records +overlap. Alerts are combined in the order they arrive, not grouped by log, and the alerts' own +messages are not carried over (see [What `run()` fills in for you](../alert_aggregator.md#what-run-fills-in-for-you)). +The [tutorial](../user_guide/tutorial.md#4-add-an-alert-aggregator) shows the overlapping output. + ## At a glance | Learns from training data | Auto-configuration | Needs `events` | Federation | |---|---|---|---| | ❌ | ❌ | ❌ | ❌ | -## Description - -A basic approach that simply concatenates the alerts without verifying anything specific. - ## Example @@ -34,9 +43,11 @@ The configuration used by the example above. It sets only what this use case nee --8<-- "docs/examples/alert_aggregators/basic_concat.yaml" ``` +Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`. + ## Configuration arguments -All parameters this detector accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this detector or `shared` with other detectors (see the [Alert Aggregation overview](../alert_aggregator.md#common-parameters-all-alert-aggregators)). Where this detector changes a shared default, the shared value is shown in brackets. +All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators. ??? note "Top level" @@ -52,8 +63,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/auxiliar/input_buffer.md b/docs/auxiliar/input_buffer.md index 09a53397..e1d352b2 100644 --- a/docs/auxiliar/input_buffer.md +++ b/docs/auxiliar/input_buffer.md @@ -1,4 +1,4 @@ -# Data Buffer +# Data Buffer The data buffer is an auxiliary method that can be used in all the components. It takes the stream data and formats it to the given specifications. diff --git a/docs/contribution.md b/docs/contribution.md index 64b81925..dc6ccfa0 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -11,9 +11,13 @@ Bug reports are very helpful to us in improving the software, and therefore, the at least the following information in a bug report: 1. Description of the bug. Describe the problem clearly. -2. Steps to reproduce. With the following configuration, go to.., click.., see error -3. Expected behavior. What should happen? -4. Environment. What was the environment for the test(version, browser, etc..) +2. Steps to reproduce. A minimal code snippet, plus the configuration (YAML or dict) and, if + possible, a few log lines that trigger the problem. +3. Expected behavior. What should happen, and what happens instead (including the full error + message, if any)? +4. Environment. Your Python version, your DetectMate version + (`python -c "import detectmatelibrary; print(detectmatelibrary.__version__)"`) and your + operating system. *Please don't include any private/sensitive information in your issue! For reporting security-related issues, see [SECURITY.md](https://github.com/ait-detectmate/DetectMateLibrary/blob/main/SECURITY.md)* diff --git a/docs/detectors.md b/docs/detectors.md index e483fcb8..b1eca29d 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -8,7 +8,7 @@ Detectors process structured logs from Parsers and emit alerts when anomalies ar | **Input** | [ParserSchema](schemas.md) | Structured log | | **Output** | [DetectorSchema](schemas.md)| Alert / finding | -This document describes the minimal API, implementation guidance, a short example detector and a unit test pattern. +This document describes the minimal API, implementation guidance and a short example detector. ## CoreDetector: minimal API @@ -145,7 +145,7 @@ example configurations set `auto_config: false` anyway, to make this explicit. I `auto_config` is `True` *and* `data_use_configure` is set, the configure phase replaces the `events` block with its own selection. -When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables, that are not bound to events (`header_variables` can but don't have to be event bound): +When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables that are not bound to events (`header_variables` can but don't have to be event bound): ```yaml detectors: @@ -172,14 +172,14 @@ detectors: ### Common parameters (all detectors) -There are some parameters, that **every** detector inherits from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on. +Some parameters are inherited by **every** detector from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The parameters that are **specific** to a detector are explained on that detector's own page. ???+ note "Top level" | Field | Type | Default | Description | |---|---|---|---| - | `auto_config` | boolean | True | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | Events configuration dict keyed by event_id. | | `global` | object | {} | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -188,8 +188,8 @@ There are some parameters, that **every** detector inherits from `CoreDetectorCo | Field | Type | Default | Description | |---|---|---|---| | `start_id` | integer | 10 | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | @@ -207,13 +207,13 @@ The detectors that keep their model in persistency stores ([New Event](detectors | Field | Type | Default | Description | |---|---|---|---| - | `persist` | object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" | Field | Type | Default | Description | |---|---|---|---| - | `allow_fed` | boolean | False | Allow to do the federation | + | `allow_fed` | boolean | False | Builds the stores needed to take part in federation. | ### Per-variable model detectors @@ -368,8 +368,8 @@ axes: | `slope_index` | change centroid vs. `slope_threshold` | index positions | | `slope_time` | change centroid vs. `slope_threshold` | normalized timestamps | -Any subset of the four may be enabled, and any single one may stand alone. The -default (`index` alone) is the historical behaviour: each segment's mean rate +Any subset of the four may be enabled, and any single one may stand alone. By +default only `index` is enabled: each segment's mean rate of change is compared against its entry in `segment_thresholds` (four segments by default, one threshold per segment), and the segments are **equal-count**: each holds the same number of observations, regardless of how much time they @@ -410,12 +410,11 @@ detectors: timestamp_format: "%y%m%d %H%M%S" ``` -Defaults reproduce the historical behaviour exactly: `index: True`, the other -three `False`, `segment_thresholds: [1.1, 0.3, 0.1, 0.01]`, `decision: consensus`, +The defaults are `index: True`, the other three `False`, +`segment_thresholds: [1.1, 0.3, 0.1, 0.01]`, `decision: consensus` and `slope_threshold: -0.05`. A config that sets nothing under `classification` -classifies identically to before these changes on every series with at least -four observations; the one exception is the segment floor described under -"Fields" below. +uses exactly these. A variable with fewer observations than segments is +classified `INSUFFICIENT_DATA`; see the segment floor under "Fields" below. #### The decision rule diff --git a/docs/detectors/bigram_frequency.md b/docs/detectors/bigram_frequency.md index 5834fc32..434c4e22 100644 --- a/docs/detectors/bigram_frequency.md +++ b/docs/detectors/bigram_frequency.md @@ -50,10 +50,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | bigram_frequency_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -63,13 +63,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | `default_freqs` | boolean | False | specific | Initialize the probabilities with default values from https://github.com/markbaggett/freq. | | `skip_repetitions` | boolean | True | specific | Only use distinct values for character-pair counting. This counteracts the problem of imbalanced word frequencies that distort the frequency table generated in a single run. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/charset.md b/docs/detectors/charset.md index 11f9de65..dfc97c37 100644 --- a/docs/detectors/charset.md +++ b/docs/detectors/charset.md @@ -1,6 +1,6 @@ # Charset Detector -The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect novelty, configuration drift, or the appearance of new actors in the environment. +The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect malformed or injected values in fields with a predictable format, for example shell characters such as `;` in a user name. ## In/out @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed characters per monitored field and emits an alert when a character not present in the set is seen for the first time (subject to configuration). +This detector keeps the set of characters observed during training for each monitored field and emits an alert whenever a value contains a character that is not in the set. Alerts don't add the character to the set, so the same unseen character raises an alert every time it appears. ## Example @@ -50,23 +50,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | charset_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/combo.md b/docs/detectors/combo.md index fca4ae37..ebd26f6a 100644 --- a/docs/detectors/combo.md +++ b/docs/detectors/combo.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed combination of values per monitored fields and emits an alert when a combination is not present in the set seen for the first time (subject to configuration). +This detector keeps the set of value combinations observed during training for each group of monitored fields and emits an alert whenever a combination that is not in the set appears. Alerts don't add the combination to the set, so the same unseen combination raises an alert every time it appears. A combination needs at least two variables in the same instance: with a single variable the detector behaves like the [New Value Detector](new_value.md). @@ -54,23 +54,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | new_value_combo_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/deeplog.md b/docs/detectors/deeplog.md index 16a0d78f..4cabdbae 100644 --- a/docs/detectors/deeplog.md +++ b/docs/detectors/deeplog.md @@ -50,7 +50,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | deeplog_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -59,8 +59,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/ecvc_detector.md b/docs/detectors/ecvc_detector.md index af292b69..dcfe68d5 100644 --- a/docs/detectors/ecvc_detector.md +++ b/docs/detectors/ecvc_detector.md @@ -52,10 +52,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | ecvc_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -66,11 +66,11 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | `seed` | integer | 0 | specific | Random seed used to shuffle count vectors into train/validation splits. | | `threshold_method` | string | mean | specific | Method used to derive the anomaly threshold from the validation split: 'mean' averages the distance scores, 'default' uses a fixed threshold of 0. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | diff --git a/docs/detectors/event_sequence.md b/docs/detectors/event_sequence.md index a8751bf4..374283b4 100644 --- a/docs/detectors/event_sequence.md +++ b/docs/detectors/event_sequence.md @@ -78,10 +78,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | event_sequence_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -89,13 +89,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `fixed_window_size` | integer, null | None | specific | Length of the sliding EventID window. A window whose exact EventID sequence was not seen during training is reported as an anomaly. When set it overrides the `auto_config_params` window range and skips auto-configuration; auto-configuration writes its own choice here. While it is None the detector is unconfigured and neither trains nor alerts. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/logbert.md b/docs/detectors/logbert.md index 510a8eac..aecaf0a7 100644 --- a/docs/detectors/logbert.md +++ b/docs/detectors/logbert.md @@ -50,7 +50,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | logbert_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -59,8 +59,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/new_event.md b/docs/detectors/new_event.md index 03ccf041..c2c60022 100644 --- a/docs/detectors/new_event.md +++ b/docs/detectors/new_event.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed event IDs and emits an alert when an event ID not present in the set is seen for the first time (subject to configuration). +This detector keeps the set of event IDs observed during training and emits an alert whenever an event ID that is not in the set appears. Alerts don't add the event ID to the set, so the same unseen event ID raises an alert every time it appears. ## Example @@ -50,21 +50,21 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | new_event_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | diff --git a/docs/detectors/new_value.md b/docs/detectors/new_value.md index 62a5e10d..95157284 100644 --- a/docs/detectors/new_value.md +++ b/docs/detectors/new_value.md @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed values per monitored field and emits an alert when a value not present in the set is seen for the first time (subject to configuration). +This detector keeps the set of values observed during training for each monitored field and emits an alert whenever a value that is not in the set appears. Alerts don't add the value to the set, so the same unseen value raises an alert every time it appears. ## Example @@ -50,23 +50,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | new_value_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ??? note "params" | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/detectors/random_detector.md b/docs/detectors/random_detector.md index 64840cc0..80e05d44 100644 --- a/docs/detectors/random_detector.md +++ b/docs/detectors/random_detector.md @@ -50,7 +50,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | random_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -59,8 +59,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/rule_based.md b/docs/detectors/rule_based.md index 174cbf87..7be844ae 100644 --- a/docs/detectors/rule_based.md +++ b/docs/detectors/rule_based.md @@ -66,7 +66,7 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | rule_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | @@ -76,8 +76,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `rules` | array | [{'rule': 'R001 - TemplateNotFound'}, {'rule': 'R003 - CheckForExceptions'}, {'rule': 'R004 - ErrorLevelFound'}] | specific | List of rules to evaluate, each a dict with a 'rule' name and an optional 'args' list. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | diff --git a/docs/detectors/scvs_detector.md b/docs/detectors/scvs_detector.md index ce75278a..5918394d 100644 --- a/docs/detectors/scvs_detector.md +++ b/docs/detectors/scvs_detector.md @@ -52,10 +52,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | scvs_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -63,11 +63,11 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `window_size` | integer | 10 | specific | Length of the event-ID window a count vector is built over. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | diff --git a/docs/detectors/value_range.md b/docs/detectors/value_range.md index 21220d98..60040bcd 100644 --- a/docs/detectors/value_range.md +++ b/docs/detectors/value_range.md @@ -50,10 +50,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc | Field | Type | Default | Scope | Description | |---|---|---|---|---| | `method_type` | string | value_range_detector | shared | Indicates what type of method it is. | - | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. | + | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. | | `events` | object | {} | shared | Events configuration dict keyed by event_id. | | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. | - | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. | + | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. | ???+ note "params" @@ -61,13 +61,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc |---|---|---|---|---| | `ignore_non_numerical_val` | boolean | True | specific | Drop non-numeric values instead of raising when a configured variable cannot be cast to a number. | | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. | - | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. | - | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. | + | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. | + | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. | | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. | | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. | | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. | | `parser` | string | PARSER | shared | Name of the parser used. | - | `allow_fed` | boolean | False | shared | Allow to do the federation | + | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. | ??? note "auto_config_params (read only while auto_config is true)" diff --git a/docs/development.md b/docs/development.md index e48c1f29..00a5195d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -20,9 +20,9 @@ uv sync --dev ## Use prek to run code checks -Every code contributer must use [`prek`](https://github.com/j178/prek) to run basic checks at commit time. +Every code contributor must use [`prek`](https://github.com/j178/prek) to run basic checks at commit time. `prek` is configured via the existing `.pre-commit-config.yaml` -and can be installed as part of the `dev` extras. To ensure pre-commit hooks run before each commit, run: +and is installed with the `dev` dependency group. To ensure pre-commit hooks run before each commit, run: ```bash uv run prek install diff --git a/docs/examples/alert_aggregators/basic_concat.yaml b/docs/examples/alert_aggregators/basic_concat.yaml index 8eef83ca..9d653635 100644 --- a/docs/examples/alert_aggregators/basic_concat.yaml +++ b/docs/examples/alert_aggregators/basic_concat.yaml @@ -1,5 +1,6 @@ alert_aggregators: - BasicConcatAggregator: - method_type: "basic_concat_aggregator" - buffer_size: 3 - auto_config: False + BasicConcatAggregator: + method_type: basic_concat_aggregator + auto_config: false + params: + buffer_size: 3 # aggregate the 3 most recent alerts diff --git a/docs/examples/config/update.py b/docs/examples/config/update.py index c6479a2b..6c104a2f 100644 --- a/docs/examples/config/update.py +++ b/docs/examples/config/update.py @@ -25,13 +25,13 @@ from detectmatelibrary.common.core import CoreConfig from detectmatelibrary.common.detector import CoreDetectorConfig -from detectmatelibrary.common.alert_aggregator import CoreAlertAggregatorConfig from detectmatelibrary.common.parser import CoreParserConfig +from detectmatelibrary.common.alert_aggregator import CoreAlertAggregatorConfig +from detectmatelibrary.alert_aggregation.basic_concat import BasicConcatAggregationConfig from detectmatelibrary.base_detectors import TrackerDetectorConfig from detectmatelibrary.base_detectors import VariableDetectorConfig from detectmatelibrary.base_detectors import DeepLearningDetectorConfig -from detectmatelibrary.alert_aggregation.basic_concat import BasicConcatAggregationConfig from typing import Any diff --git a/docs/examples/detectors/event_sequence.py b/docs/examples/detectors/event_sequence.py index 2d3a74ca..54a4b5d5 100644 --- a/docs/examples/detectors/event_sequence.py +++ b/docs/examples/detectors/event_sequence.py @@ -1,4 +1,6 @@ # --8<-- [start:example] +from itertools import count + import yaml from detectmatelibrary import schemas from detectmatelibrary.detectors.event_sequence_detector import EventSequenceDetector @@ -8,9 +10,13 @@ detector = EventSequenceDetector(name="LoginSequenceDetector", config=config) +log_ids = count() + + def event(event_id: int) -> schemas.ParserSchema: - """A parsed log; only its EventID (the matched template) matters here.""" - return schemas.ParserSchema({"eventID": event_id}) + """A parsed log with a running logID; only its EventID (the matched + template) matters for detection.""" + return schemas.ParserSchema({"eventID": event_id, "logID": str(next(log_ids))}) # normal session: 0 = login accepted, 1 = session opened, 2 = session closed. @@ -24,7 +30,7 @@ def event(event_id: int) -> schemas.ParserSchema: detector.process(event(0)) alert = detector.process(event(2)) # session closed without being opened print(dict(alert["alertDetails"])) -# {'Sequence (0, 2)': 'EventID sequence of length 2 ending at logID was not seen during training.'} +# {'Sequence (0, 2)': 'EventID sequence of length 2 ending at logID 10 was not seen during training.'} print(detector.get_known_sequences()) # {(0, 1), (1, 2), (2, 0)} # --8<-- [end:example] diff --git a/docs/other/log_formats.md b/docs/other/log_formats.md index 3c2f605a..a828d6e8 100644 --- a/docs/other/log_formats.md +++ b/docs/other/log_formats.md @@ -53,7 +53,7 @@ Linux auditd log. Produced by the kernel audit subsystem; records system calls, type=USER_ACCT msg=audit(1642723741.072:375): pid=10125 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting acct="root" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success' ``` -**`log_format`:** `type= msg=audit(