Docs/minor fixes 2

docs/alert_aggregator.md

@@ -89,8 +89,8 @@ There are some parameters, that **every** alert aggregator inherits from `CoreAl
     | Field | Type | Default | Description |
     |---|---|---|---|
     | `start_id` | integer | 10 | Number used to start the unique ID generator. |
-    | `data_use_training` | integer, null | None | Data used for training, if None, training is not done. |
-    | `data_use_configure` | integer, null | None | Data used for configuration, if None, configuration is not done. |
+    | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. |
     | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. |
     | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
     | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |

docs/alert_aggregators/basic_concatenation.md

@@ -1,6 +1,11 @@
 # Basic Concat Alert Aggregation
 
-The basic concat alert aggregation approach aggregates x number of alerts into one output schema.
+The Basic Concat alert aggregation combines the most recent alerts into one aggregated record by
+concatenating their IDs and timestamps.
+
+## In/out
+
+Input and output schemas in the pipeline
 
 ## In/out
 
@@ -9,16 +14,20 @@ The basic concat alert aggregation approach aggregates x number of alerts into o
 | **Input**  | [DetectorSchema](../schemas.md)    | Alerts from detectors  |
 | **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts    |
 
+## Description
+
+The aggregator keeps a sliding window of the last `buffer_size` alerts. The first aggregated record
+is emitted once the window is full, and then one record for every new alert, so consecutive records
+overlap. Alerts are combined in the order they arrive, not grouped by log, and the alerts' own
+messages are not carried over (see [What `run()` fills in for you](../alert_aggregator.md#what-run-fills-in-for-you)).
+The [tutorial](../user_guide/tutorial.md#4-add-an-alert-aggregator) shows the overlapping output.
+
 ## At a glance
 
 | Learns from training data | Auto-configuration | Needs `events` | Federation |
 |---|---|---|---|
 | ❌ | ❌ | ❌ | ❌ |
 
-## Description
-
-A basic approach that simply concatenates the alerts without verifying anything specific.
-
 
 ## Example
 
@@ -34,9 +43,11 @@ The configuration used by the example above. It sets only what this use case nee
 --8<-- "docs/examples/alert_aggregators/basic_concat.yaml"
 ```
 
+Load it with `yaml.safe_load` and pass the dict as `config=`, as in the example. The key under `alert_aggregators:` must match the aggregator's `name`.
+
 ## Configuration arguments
 
-All parameters this detector accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this detector or `shared` with other detectors (see the [Alert Aggregation overview](../alert_aggregator.md#common-parameters-all-alert-aggregators)). Where this detector changes a shared default, the shared value is shown in brackets.
+All parameters this alert aggregator accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this aggregator or `shared` with other alert aggregators.
 
 <!-- Start arguments -->
 ??? note "Top level"
@@ -52,8 +63,8 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc
     |---|---|---|---|---|
     | `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. |
     | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
-    | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. |
-    | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. |
+    | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. |
     | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
     | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
     | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |

docs/auxiliar/input_buffer.md

@@ -1,4 +1,4 @@
-#  Data Buffer
+# Data Buffer
 
 The data buffer is an auxiliary method that can be used in all the components. It takes the stream data and formats it to the given specifications.
 

docs/contribution.md

@@ -11,9 +11,13 @@ Bug reports are very helpful to us in improving the software, and therefore, the
 at least the following information in a bug report:
 
 1. Description of the bug. Describe the problem clearly.
-2. Steps to reproduce. With the following configuration, go to.., click.., see error
-3. Expected behavior. What should happen?
-4. Environment. What was the environment for the test(version, browser, etc..)
+2. Steps to reproduce. A minimal code snippet, plus the configuration (YAML or dict) and, if
+   possible, a few log lines that trigger the problem.
+3. Expected behavior. What should happen, and what happens instead (including the full error
+   message, if any)?
+4. Environment. Your Python version, your DetectMate version
+   (`python -c "import detectmatelibrary; print(detectmatelibrary.__version__)"`) and your
+   operating system.
 
 *Please don't include any private/sensitive information in your issue! For reporting security-related issues, see [SECURITY.md](https://github.com/ait-detectmate/DetectMateLibrary/blob/main/SECURITY.md)*
 

docs/detectors.md

@@ -8,7 +8,7 @@ Detectors process structured logs from Parsers and emit alerts when anomalies ar
 | **Input**  | [ParserSchema](schemas.md) | Structured log     |
 | **Output** | [DetectorSchema](schemas.md)| Alert / finding    |
 
-This document describes the minimal API, implementation guidance, a short example detector and a unit test pattern.
+This document describes the minimal API, implementation guidance and a short example detector.
 
 ## CoreDetector: minimal API
 
@@ -145,7 +145,7 @@ example configurations set `auto_config: false` anyway, to make this explicit. I
 `auto_config` is `True` *and* `data_use_configure` is set, the configure phase replaces the
 `events` block with its own selection.
 
-When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables, that are not bound to events (`header_variables` can but don't have to be event bound):
+When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables that are not bound to events (`header_variables` can but don't have to be event bound):
 
 ```yaml
 detectors:
@@ -172,14 +172,14 @@ detectors:
 
 ### Common parameters (all detectors)
 
-There are some parameters, that **every** detector inherits from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on.
+Some parameters are inherited by **every** detector from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The parameters that are **specific** to a detector are explained on that detector's own page.
 
 <!-- Start common_arguments -->
 ???+ note "Top level"
 
     | Field | Type | Default | Description |
     |---|---|---|---|
-    | `auto_config` | boolean | True | Runs the configuration step before the training process. |
+    | `auto_config` | boolean | True | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. |
     | `events` | object | {} | Events configuration dict keyed by event_id. |
     | `global` | object | {} | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. |
 
@@ -188,8 +188,8 @@ There are some parameters, that **every** detector inherits from `CoreDetectorCo
     | Field | Type | Default | Description |
     |---|---|---|---|
     | `start_id` | integer | 10 | Number used to start the unique ID generator. |
-    | `data_use_training` | integer, null | None | Data used for training, if None, training is not done. |
-    | `data_use_configure` | integer, null | None | Data used for configuration, if None, configuration is not done. |
+    | `data_use_training` | integer, null | None | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | Number of logs used for the configuration step. None skips it. |
     | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. |
     | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
     | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
@@ -207,13 +207,13 @@ The detectors that keep their model in persistency stores ([New Event](detectors
 
     | Field | Type | Default | Description |
     |---|---|---|---|
-    | `persist` | object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. |
+    | `persist` | object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. |
 
 ???+ note "params"
 
     | Field | Type | Default | Description |
     |---|---|---|---|
-    | `allow_fed` | boolean | False | Allow to do the federation |
+    | `allow_fed` | boolean | False | Builds the stores needed to take part in federation. |
 <!-- End tracker_arguments -->
 
 ### Per-variable model detectors
@@ -368,8 +368,8 @@ axes:
 | `slope_index` | change centroid vs. `slope_threshold` | index positions |
 | `slope_time` | change centroid vs. `slope_threshold` | normalized timestamps |
 
-Any subset of the four may be enabled, and any single one may stand alone. The
-default (`index` alone) is the historical behaviour: each segment's mean rate
+Any subset of the four may be enabled, and any single one may stand alone. By
+default only `index` is enabled: each segment's mean rate
 of change is compared against its entry in `segment_thresholds` (four segments
 by default, one threshold per segment), and the segments are **equal-count**:
 each holds the same number of observations, regardless of how much time they
@@ -410,12 +410,11 @@ detectors:
       timestamp_format: "%y%m%d %H%M%S"
 ```
 
-Defaults reproduce the historical behaviour exactly: `index: True`, the other
-three `False`, `segment_thresholds: [1.1, 0.3, 0.1, 0.01]`, `decision: consensus`,
+The defaults are `index: True`, the other three `False`,
+`segment_thresholds: [1.1, 0.3, 0.1, 0.01]`, `decision: consensus` and
 `slope_threshold: -0.05`. A config that sets nothing under `classification`
-classifies identically to before these changes on every series with at least
-four observations; the one exception is the segment floor described under
-"Fields" below.
+uses exactly these. A variable with fewer observations than segments is
+classified `INSUFFICIENT_DATA`; see the segment floor under "Fields" below.
 
 #### The decision rule
 

docs/detectors/bigram_frequency.md

@@ -50,10 +50,10 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc
     | Field | Type | Default | Scope | Description |
     |---|---|---|---|---|
     | `method_type` | string | bigram_frequency_detector | shared | Indicates what type of method it is. |
-    | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. |
+    | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. |
     | `events` | object | {} | shared | Events configuration dict keyed by event_id. |
     | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. |
-    | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. |
+    | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. |
 
 ???+ note "params"
 
@@ -63,13 +63,13 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc
     | `default_freqs` | boolean | False | specific | Initialize the probabilities with default values from https://github.com/markbaggett/freq. |
     | `skip_repetitions` | boolean | True | specific | Only use distinct values for character-pair counting. This counteracts the problem of imbalanced word frequencies that distort the frequency table generated in a single run. |
     | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
-    | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. |
-    | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. |
+    | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. |
     | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
     | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
     | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
     | `parser` | string | PARSER | shared | Name of the parser used. |
-    | `allow_fed` | boolean | False | shared | Allow to do the federation |
+    | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. |
 
 ??? note "auto_config_params (read only while auto_config is true)"
 

docs/detectors/charset.md

@@ -1,6 +1,6 @@
 # Charset Detector
 
-The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect novelty, configuration drift, or the appearance of new actors in the environment.
+The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect malformed or injected values in fields with a predictable format, for example shell characters such as `;` in a user name.
 
 ## In/out
 
@@ -19,7 +19,7 @@ Input and output schemas in the pipeline
 
 ## Description
 
-This detector maintains a lightweight set of observed characters per monitored field and emits an alert when a character not present in the set is seen for the first time (subject to configuration).
+This detector keeps the set of characters observed during training for each monitored field and emits an alert whenever a value contains a character that is not in the set. Alerts don't add the character to the set, so the same unseen character raises an alert every time it appears.
 
 ## Example
 
@@ -50,23 +50,23 @@ All parameters this detector accepts, grouped by the YAML block they go in. **Sc
     | Field | Type | Default | Scope | Description |
     |---|---|---|---|---|
     | `method_type` | string | charset_detector | shared | Indicates what type of method it is. |
-    | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. |
+    | `auto_config` | boolean | True | shared | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. |
     | `events` | object | {} | shared | Events configuration dict keyed by event_id. |
     | `global` | object | {} | shared | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML. |
-    | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See the Persistency page. |
+    | `persist` | object, null | None | shared | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. |
 
 ??? note "params"
 
     | Field | Type | Default | Scope | Description |
     |---|---|---|---|---|
     | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
-    | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. |
-    | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. |
+    | `data_use_training` | integer, null | None | shared | Number of logs used for training. None skips training. |
+    | `data_use_configure` | integer, null | None | shared | Number of logs used for the configuration step. None skips it. |
     | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
     | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
     | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
     | `parser` | string | PARSER | shared | Name of the parser used. |
-    | `allow_fed` | boolean | False | shared | Allow to do the federation |
+    | `allow_fed` | boolean | False | shared | Builds the stores needed to take part in federation. |
 
 ??? note "auto_config_params (read only while auto_config is true)"