Repository navigation
Keep the 0.9.39 cdxt command lines working (deprecated) - #80
Merged
Merged
Conversation
…mpatibility adapter The command lines of cdx_toolkit 0.9.39 (options before the command, iter, size, warc) failed: the iter alias only renamed the command, size and warc only printed a hint. They are now parsed with the old grammar and run with the current code, with a deprecation line and a suggested new command line on stderr: - iter: the old text, --jsonl and --csv output and default fields; --get through CDXFetcher.get() (default limit 1000), --closest without --get paged as before; --all-fields --csv is a usage error (it failed with a traceback) - size: CDXFetcher.get_size_estimate() with --details, not capped by --limit - warc: the captures streamed into repackage() (sort=False), --url-fgrep/--url-fgrepv after the limit, replays from the Internet Archive for --ia; TEST-001.warc.gz, exit status 1 if captures could not be fetched The unreleased `cdxt iter` alias with the new arguments is removed.
… docstrings of the legacy commands
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keeps the command lines of cdx_toolkit 0.9.39 working on top of #79. Without this PR, every documented old command line fails with "unrecognized arguments", for example
cdxt --cc --limit 10 iter 'commoncrawl.org/*',--crawl 3 … iter,sizeandwarc. This includes the examples in Common Crawl's own whirlwind docs.What changes
cdxtrecognizes an old command line in either of two cases:iter,sizeorwarc;-h,-Vor-vcome before the command.Such command lines are parsed with the 0.9.39 grammar, so mistakes still exit with status 2, and they run with the new code. stderr gets one line with a suggested new command:
iter:--jsonland--csvoutput and the default fieldsurl,status,timestampare unchanged.--getusesCDXFetcher.get()(default limit 1000).--closestwithout--getuses paged requests, as before.size: runsCDXFetcher.get_size_estimate().--detailsworks, and the result is not capped by--limit.warc:repackage()withsort=False, so memory use stays flat and index paging overlaps with fetching.--url-fgrep/--url-fgrepvapply after the limit.--iareplays from the Internet Archive without--wb.cdxt iteralias is removed (it was never released), and so iscli/deprecated.py.Accepted differences from 0.9.39
warc:TEST-001.warc.gzinstead ofTEST-000000.extracted.warc.gz;WARC-Source-*headers;iter --all-fields --csvis now a usage error. In 0.9.39 it failed with a traceback.Tests
tests/test_cli_legacy.py: the 0.9.39 command lines, run against the responses that were recorded for them intests/data/mock_responses/test_cli/. They check the output format,--get,size,size --details, and the WARC files and warcinfo for--cc,--ia, the prefix options and an empty selection.tests/unit/test_cli_legacy.pycovers:warccontent source,sort=Falseand exit status 1;