Skip to content

Keep the 0.9.39 cdxt command lines working (deprecated) - #80

Merged
malteos merged 3 commits into
feat/cli-refactorfrom
feat/cli-legacy-compat
Oct 16, 2026
Merged

malteos merged 3 commits into
feat/cli-refactorfrom
feat/cli-legacy-compat

Conversation

@malteos

@malteos malteos commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Keeps the command lines of cdx_toolkit 0.9.39 working on top of #79. Without this PR, every documented old command line fails with "unrecognized arguments", for example cdxt --cc --limit 10 iter 'commoncrawl.org/*', --crawl 3 … iter, size and warc. This includes the examples in Common Crawl's own whirlwind docs.

What changes

cdxt recognizes an old command line in either of two cases:

  • the command is iter, size or warc;
  • options other than -h, -V or -v come before the command.

Such command lines are parsed with the 0.9.39 grammar, so mistakes still exit with status 2, and they run with the new code. stderr gets one line with a suggested new command:

$ cdxt --cc --limit 10 iter 'commoncrawl.org/*'
cdxt: this command line uses the old syntax, which is deprecated and will be removed in a future version; suggested replacement: cdxt index cc --url 'commoncrawl.org/*' --limit 10 --output-format csv --fields status,timestamp,url (see …)
status 200, timestamp 20250804103329, url https://www.commoncrawl.org/
  • iter:
    • The old text, --jsonl and --csv output and the default fields url,status,timestamp are unchanged.
    • Captures come from the CDX API backend, which sends the same requests as before.
    • --get uses CDXFetcher.get() (default limit 1000).
    • --closest without --get uses paged requests, as before.
  • size: runs CDXFetcher.get_size_estimate(). --details works, and the result is not capped by --limit.
  • warc:
    • The captures stream into repackage() with sort=False, so memory use stays flat and index paging overlaps with fetching.
    • --url-fgrep / --url-fgrepv apply after the limit.
    • --ia replays from the Internet Archive without --wb.
  • The new-arguments cdxt iter alias is removed (it was never released), and so is cli/deprecated.py.

Accepted differences from 0.9.39

  • warc:
    • writes TEST-001.warc.gz instead of TEST-000000.extracted.warc.gz;
    • copies records byte-for-byte, without the WARC-Source-* headers;
    • writes no file when nothing is selected;
    • exits with status 1 if a capture could not be fetched.
  • iter --all-fields --csv is now a usage error. In 0.9.39 it failed with a traceback.

Tests

  • tests/test_cli_legacy.py: the 0.9.39 command lines, run against the responses that were recorded for them in tests/data/mock_responses/test_cli/. They check the output format, --get, size, size --details, and the WARC files and warcinfo for --cc, --ia, the prefix options and an empty selection.
  • tests/unit/test_cli_legacy.py covers:
    • detection of old command lines;
    • strict errors, with no requests made;
    • source precedence;
    • the formatters;
    • the fetcher paths;
    • the warc content source, sort=False and exit status 1;
    • that every suggested replacement parses with the new parser.

…mpatibility adapter

The command lines of cdx_toolkit 0.9.39 (options before the command, iter, size, warc) failed: the iter alias only
renamed the command, size and warc only printed a hint. They are now parsed with the old grammar and run with the
current code, with a deprecation line and a suggested new command line on stderr:

- iter: the old text, --jsonl and --csv output and default fields; --get through CDXFetcher.get() (default limit
  1000), --closest without --get paged as before; --all-fields --csv is a usage error (it failed with a traceback)
- size: CDXFetcher.get_size_estimate() with --details, not capped by --limit
- warc: the captures streamed into repackage() (sort=False), --url-fgrep/--url-fgrepv after the limit, replays
  from the Internet Archive for --ia; TEST-001.warc.gz, exit status 1 if captures could not be fetched

The unreleased `cdxt iter` alias with the new arguments is removed.
@malteos
malteos marked this pull request as ready for review October 16, 2026 09:59
@malteos
malteos merged commit e2ff0d7 into feat/cli-refactor Oct 16, 2026
@malteos malteos mentioned this pull request Oct 16, 2026
2 of 4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant