[njoubert-cleverage]

Description

Fixes #230.

DoctrineProcessHandler / LogRecord: a log record with an invalid UTF-8 sequence in its message or context (e.g. a Latin-1 file imported), or with a message longer than 512 code points once truncated to 512 graphemes (emojis, combining marks), made the insertion of the logs fail. The whole batch (up to 500 records) was lost, the process failed on the 500th record (even with error_strategy: skip), the records then accumulated (500 === count() never true again), and at the end of the process success() / fail() were not called: the execution stayed started and the command exited with 1.

• The message and the channel are scrubbed (mb_scrub(), NUL characters removed) and truncated in code points (mb_substr()), as the column length is counted by MySQL and PostgreSQL.
• A context which cannot be encoded in JSON is stored with its invalid UTF-8 sequences replaced (U+FFFD) and its non-encodable values replaced by null; an encodable context is unchanged.
• When the insertion of the logs fails anyway, its records are dropped (and detached from a still open entity manager, a closed one is reset right away), and the failure is logged as an error on the cleverage_process channel (log file of the execution, not stored in database); the process goes on and the status of its execution is saved. The batch is flushed at >= 500 records.
• The entity manager is shared with the tasks: a failure of a flush that also writes pending changes of the tasks (or of another execution) is not caught, as before. Checking it costs a change set computation every 500 records when entities of the tasks are managed (documented).

Requirements

• Documentation updates
  • Reference
  • Cookbooks
  • Changelog
• Unit tests

Breaking changes

None.

🤖 Generated with Claude Code