Add real transformer batch inference

[Mayukhga83]

Summary

• replace per-example transformer inference with padded whole-list batching and a single model forward pass
• preserve input order while handling uneven sequence lengths, token and span outputs, confidence filtering, taxonomy typing, and empty inputs
• reject mismatched prompt/answer lengths for both transformer and LLM batch paths
• keep benchmark memory reporting portable on platforms without the Unix resource module

Related issue

Closes #23.

Type of change

• Bug fix
• Feature
• Documentation
• Tests
• Refactor or maintenance

Testing

• ruff format --check lettucedetect/ lettucedetect_api/ tests/
• ruff check lettucedetect/ lettucedetect_api/ tests/ --extend-exclude lettucedetect/integrations/
• python -m pytest
• Other: ruff check scripts/benchmark_detectors.py

Checklist

• I kept the PR focused on one change.
• I added or updated tests/docs when needed.
• I checked that no secrets, API keys, or credentials are included.

Rights & sign-off (required)

• I certify that I have the right to submit this code and that it may be
  distributed under the repository's MIT license
  (see CONTRIBUTING).

[adaamko]

Thanks for the work here, @Mayukhga83. The batched path is careful, and it matched one-by-one output on the real ModernBERT model at every batch size I tried. #113 implemented the same change for #23, was opened first, and has now been merged; our policy for duplicate PRs is that the earlier one lands when it is correct, so I'm closing this one. For context on one difference: with a tokenizer set to padding_side="left", labels[row, answer_start:sequence_length] (transformer.py:529 in this PR) assumes right padding and silently returns empty results for all but the longest row; #113 raises an explicit error instead.

A few pieces here did not land with #113 and would be welcome as small, separate PRs on top of main:

1. scripts/benchmark_detectors.py portability on platforms without the resource module, together with its test update. A useful fix in its own right.
2. Truncation warning parity. Your warning is computed from the untruncated length and uses the same message as predict_prompt, which is nicer than the current "reached max length" check in _predict_batch.
3. A mismatch error message that includes both counts (got 7 and 6); this could be folded into (2).

Thanks again, and we'd be glad to review any of these.