Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
- unreleased
+ new command line interface: `cdxt index` and `cdxt repackage` (see README)
+ `cdxt iter` is a deprecated alias of `cdxt index` (same arguments, prints a deprecation warning)
+ removed `cdxt size` and `cdxt warc`; they print the equivalent new command
+ the command lines of 0.9.39 (`cdxt --cc iter URL`, `size`, `warc`) are deprecated (to be removed in a future version): they run
with the new code and print a suggested new command line; `iter` output, `--get` and `size --details` are kept;
`warc` writes `TEST-001.warc.gz` (was `TEST-000000.extracted.warc.gz`) with the records copied byte-for-byte
(no `WARC-Source-URI`/`WARC-Source-Range` headers), no file without captures, and exits with status 1 if
captures could not be fetched; `iter --all-fields --csv` is a usage error (was a traceback)
+ `cdxt index`: `--hostname`, `--domain`, repeatable `--url`, `-o/--output` (local or remote),
JSONL/CDXJ/CSV/TSV output, all fields by default, `--count` replaces `size`
+ `cdxt repackage`: reads captures from a file or stdin, human-readable `--size`, `--is-part-of`, `--description`
Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,10 +317,12 @@ upload (an S3 lifecycle rule for incomplete uploads removes it).

### Migrating from the old commands

`cdxt iter` is a deprecated alias of `cdxt index` with the same (new)
arguments; it prints a deprecation warning. The `size` and `warc` commands
have been removed; running them prints the equivalent new command. See the [migration guide](docs/cli-migration-guide.md)
for all changes and more examples.
The command lines of cdx_toolkit 0.9.39 (`cdxt --cc iter ...`, `size`,
`warc`) still work, but are deprecated and will be removed in a future
version; they print a suggested new command line. The old `warc` now writes
`TEST-001.warc.gz` and exits with status 1 if captures could not be fetched.
See the [migration guide](docs/cli-migration-guide.md) for all changes and
more examples.

| Old | New |
|---|---|
Expand Down
15 changes: 6 additions & 9 deletions cdx_toolkit/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@

from . import index, repackage
from .common import IncompleteError, add_verbose_arg, set_loglevel
from .deprecated import DEPRECATED_COMMANDS, REMOVED_COMMANDS, exit_removed_command, get_command
from .deprecated import rewrite_deprecated_command
from .legacy import build_legacy_parser, deprecation_message, is_legacy_command_line


def build_parser() -> ArgumentParser:
Expand All @@ -28,15 +27,13 @@ def main(args=None):
if args is None: # pragma: no cover
args = sys.argv[1:]

command = get_command(args)
if command in REMOVED_COMMANDS:
exit_removed_command(command, args)
if command in DEPRECATED_COMMANDS:
args = rewrite_deprecated_command(args)

parser = build_parser()
# the command lines of cdx_toolkit 0.9.39 (deprecated): `cdxt --cc iter URL`, `size`, `warc`
legacy = is_legacy_command_line(args)
parser = build_legacy_parser() if legacy else build_parser()
cmd = parser.parse_args(args=args)
set_loglevel(cmd.verbose)
if legacy:
sys.stderr.write(deprecation_message(cmd))
configure_fsspec_s3() # Common Crawl's S3 defaults for s3:// paths, unless configured by the user
cmdline = ' '.join(['cdxt'] + list(args))
if uses_hf(args):
Expand Down
104 changes: 0 additions & 104 deletions cdx_toolkit/cli/deprecated.py

This file was deleted.

Loading
Loading