Repository navigation
Conversation
A bad task (a wrong hashcat command, a syntax error) used to deactivate every agent that picked it up, one by one, until the whole fleet went offline. Nodes are untrusted (a cracking contest), so a single node failing is a driver or hardware problem on that node, not proof the task is broken. On a client error the agent is no longer deactivated on its first error. AgentErrorUtils now decides between the two failure modes: - if brokenTaskThreshold DISTINCT agents have failed on one task, the task is marked broken (recorded in the new BrokenTask table) and is not handed out again until an operator clears it; the reporting agents stay active; - if an agent has failed on brokenAgentThreshold distinct tasks, the agent itself is deactivated. An apiv2 endpoint (/api/v2/ui/brokentasks, GET + DELETE) lists broken tasks and clears one, which is a manual retry after the operator has looked at it. Three thresholds are configurable (Cracking/Tasks section): - brokenTaskThreshold (default 2, distinct agents; 0 disables) - brokenAgentThreshold (default 3, distinct tasks; 0 disables) - brokenErrorWindow (default 3600s; 0 counts the whole history) Setting brokenTaskThreshold to 0 and brokenAgentThreshold to 1 reproduces the previous deactivate-on-first-error behaviour. Implements the server side of the hashtopolis#884 epic (and the hashtopolis#2487 duplicate).
Add an isBroken aggregate to the task and taskwrapperdisplay resources, and a brokenReason aggregate to the task resource. The task list (taskwrapperdisplays) flags broken tasks and the task edit page shows why, without a separate request or a dedicated page.
Marking a task broken counted only the agents that failed. If two nodes share a driver fault while a third node is cracking the same task fine, the working node proves the command runs, so the failures are agent faults, not a task fault. Skip markTaskBroken when any chunk on the task has non-zero progress; a genuinely broken command never progresses, so real detection is unaffected.
With fewer active agents than the configured threshold a task could never reach it, so an agent would loop on a broken task forever. Cap the threshold at the number of active agents (recomputed per error): a single-agent setup faults a task on the first distinct failure, while a larger fleet keeps the full threshold. Adds a test for the single-active-agent case.
… on edit brokenReason now returns the actual agent error that broke the task (its most recent agent error) rather than the generic auto-marked text, so the UI explains what went wrong. Editing the attack command clears the task's broken flag, since that is an admin fixing it; if it is still bad the agents mark it broken again.
thatux
changed the title
…the mark-broken race Review fixes for the broken-task detection: - Cap the effective threshold at the number of agents ELIGIBLE for the task (same access group, trusted for any secret hashlist or file), not the whole active fleet. A task reachable by only a subset smaller than the configured threshold no longer loops forever on its eligible agents. - markTaskBroken swallows a UNIQUE(taskId) violation from a concurrent error on the same task instead of surfacing a 500 to the reporting agent. - Rely on changeAttackCmd -> purgeTask to clear the BrokenTask row when the command is edited; drop the redundant clearBrokenTask call, so resubmitting the same command no longer clears a broken task. - Note in a comment that a task with any chunk progress is never auto-faulted. Tests put agents in the task's access group so they count as eligible, and a new test asserts the threshold caps at eligible agents rather than the active fleet.
|
Changing the attackcmd removes the 'broken' mark from the task, or you can use the 'mark not broken button'. 
|
The broken-task work added an API (BrokenTaskAPI) and the isBroken/brokenReason aggregates but did not regenerate the artifacts CI checks against: - openapi.json regenerated (ci/tools/generate-openapi.php) so the committed schema matches the one the server serves (openapi-lint). - BrokenTask model and factory regenerated with generator.php so the generated DBA sources match (dba-generator); the diff was whitespace only. - Mark isTaskBroken() @phpstan-impure: it reads the database, so the second check in markTaskBroken's race guard is not dead code.
mockConfig replaced the whole SConfig singleton with a DataSet holding only the three broken-task thresholds, so every other config lookup returned null for the duration of the test. In particular Util::createLogEntry (reached via markTaskBroken -> DServerLog::log) reads numLogEntries for its log rotation; with it null the rotation deleted the oldest LogEntry rows, including id 1. That broke the unrelated apiv2 test_logentry (LogEntry.get(pk=1)) later in the same CI run on the synthetic dataset. Build the mock from the real config (SConfig::getInstance()->getAllValues()) and override only the three thresholds, so unrelated config keeps its real values.
|
CI should work now |
|
I really like the feature, but I am not really sure if checking whether a task is broken by a n amount of agents is the way to go. Preferably I would like it if hashcat could return a status code for wrong user input. If I read https://github.com/hashcat/hashcat/blob/master/docs/exit_status_code.txt I think it could be -1. So if we could validate and use that, that could be perfect. Another solution could be to parse the exact error message hashcat throws on - - keyspace. If the error is "error during keyspace calculation", I think we can fairly surely say it was an user input error in the task. |
|
Agreed, this feature is really nice to have and heavily requested. I also would propose to change the implementation, though. @jessevz already elaborated on the "trigger logic" and how to implement that. As for the implementation regarding the application model, I'd suggest to introduce an additional task status It'd be appreciated if you find time to explore these alternative implementations. |
|
Maybe it makes sense to wait a moment before implementing further too much. Currently we have the stack open which brings background jobs and local downloads of hashcat to the server. This would then allow calculating Or are there other use cases, where a task would be considered broken but does work normally on the keyspace calculation? |
|
The reason for testing with multiple agents instead of feedback from the cracker is twofold:
@s3inlc can you link me to the background task implementation (out of interest) |
|
The background job lives currently here: #2502 For the moment let's wait with actual development on this feature and wait for the stacked PRs to be merged and then do it in a heuristic way that on an error either the task gets marked as broken or the agent gets deactivated. We can then see if we continue on this branch or use it as a basis for a new one. What would help to already have is a conceptual heuristic, how (based on the exit code of hashcat and potential error messages to stderr) to determine if an error is likely the reason to a faulty task or a faulty agent. |
|
Hashcat master now has dedicated exit codes that help distinguish these cases: command/task errors and bridge/plugin errors. I'd suggest this routing:
The paired positive numbers are POSIX shell exit statuses. Cracked/exhausted ( The exit code would be the primary classifier, with stderr providing the reason and resolving ambiguous cases. Server-side checks with the matching binary avoid trusting a single agent's report, although a successful For an MVP, I'd require a hashcat build containing these changes and leave older binaries on the existing handling. That avoids adding stderr-pattern classification for older versions. |
fixes #884
A bad task command makes every agent that picks it up fail. The old behaviour
deactivated them one by one until the fleet was offline. This marks the task
broken instead and exposes it to the UI. Detection works with the stock agent
(it already reports failures); the companion #746 makes the reported error
useful instead of a bare "Keyspace measure failed!".
Reproduction
Give a task an invalid mask so its keyspace measure fails, register one agent,
let it poll:
Changes
AgentErrorUtils):brokenTaskThresholddistinct agents failingmarks the task broken (not the agents); an agent failing across
brokenAgentThresholdtasks is deactivated instead. Broken tasks are skippedin
getCandidateTasks. Thresholds configurable.min(configured, active)),recomputed per error, so a single agent faults immediately instead of looping.
faulted.
isBrokenaggregate ontaskandtaskwrapperdisplay,brokenReasonontask(the causing agent error).BrokenTaskAPI(GET/DELETE) already exists.Testing