Repository navigation
Conversation
Replace the old `iter`/`size`/`warc` commands with two commands that can be piped: `cdxt index` selects captures from an index, `cdxt repackage` writes them to WARC files. cdxt index - indexes: CDX API (cc, ia, URLs), CDX files (local/remote, globs), Common Crawl's columnar URL index with DuckDB (cc-url-index-s3, cc-url-index-hf, *.parquet) or Athena (athena://...); detected from the INDEX argument or --index-format - selection: --url (CDX API patterns, matched as SURT keys for CDX files and SQL), --hostname, --domain, --url-file/--url-column/--url-file-type (CDX files), --crawl, --from/--to, --closest, --filter, --language (SQL), --url-fgrep(v), --limit, --sql (custom query), --count, --dry-run, cost confirmation (--yes) - output: -o (local or remote), JSONL, CDX/CDXJ, CSV, TSV, Parquet (optional pyarrow), --fields; one output per CDX file for a glob input and an -o path without extension (--processes) cdxt repackage - reads captures written by `cdxt index` (file, glob or stdin), copies the WARC records byte-for-byte with the async WARC fetcher from feat/warc-range-sources (http, s3 and hf:// reads; local, s3 and hf:// output; --processes, --parallel-readers, --add-metadata-record), --wb for wayback replays - multi-process workers share one file counter, so files get their final names (and warcinfo records) directly; the HF reader resolves each WARC file once - default --prefix REPACKAGE, human readable --size, no file without captures Other changes - `cdxt iter` is a deprecated alias of `cdxt index`; `size` and `warc` exit with the equivalent new command line - fix pad_timestamp_up() for timestamps with a day and for leap years - remove the deprecated cdx_iter and cdx_size scripts - new extras: athena, duckdb, hf, parquet; fsspec, aiofiles and url-is-in are required
- README: `cdxt index` (index types, selection, output, SQL and CDX file options) and `cdxt repackage`, new extras, migration table - docs/cli-migration-guide.md: old `iter`/`size`/`warc` commands and the feat/warc-range-sources `repackage` options mapped to the new CLI, with examples - docs/notes/warc-fetcher-performance.md: WARC fetcher benchmarks and tuning notes from feat/warc-range-sources - CHANGELOG entry
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #79 +/- ##
==========================================
+ Coverage 97.24% 98.40% +1.16%
==========================================
Files 9 44 +35
Lines 943 3575 +2632
==========================================
+ Hits 917 3518 +2601
- Misses 26 57 +31 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
CI - .gitattributes: no line-ending conversion for tests/data, so fixtures (e.g. the metadata record with a fixed payload digest) are byte-identical on Windows - compare relative paths with as_posix() in the per-file output test (Windows) - `make test` no longer runs examples/iter-and-warc.py against the live Common Crawl index (connections from CI runners time out; it also took ~45 minutes in the minimal job), it is now `make example` Fixes found by the new tests - WARC files: only rotate a file that already has records; a record larger than the remaining --size no longer leaves a WARC file with only the warcinfo record - DuckDB: retry S3 requests longer (8 retries, 1s, 2s, 4s, ...) on 503 SlowDown - is_s3_url() accepts None (WARCFilter without warc_download_prefix) Tests (coverage of the new code) - Athena source and `cdxt index athena://...` with a fake boto3 client - HF reader: resolve without redirect, resolve/read errors, 429s, cache eviction - DuckDB source with a local parquet file and a fake connection (HF filesystem) - WARC pipeline and multi-process mode in-process (fake HTTP/HF reads, memory:// output), reader/writer errors, uvloop - usage errors and edge cases of index/repackage Cleanup: remove dead code (fsspec fallbacks in record_io, the old --jsonl/--csv translation, the cost estimate of the repackage sources, Athena get_databases, the Python < 3.9 check, DuckDbSource's query builder)
…ests cdxt's S3 clients (the WARC fetcher's aioboto3 clients and s3fs for all other s3:// paths) use region us-east-1, retry_mode adaptive, max_attempts 100000, addressing_style virtual, payload_signing_enabled false and 2 concurrent requests (s3fs max_concurrency). The user stays in control: a default is only used for a setting that is not configured in the environment (AWS_REGION, AWS_DEFAULT_REGION, AWS_RETRY_MODE, AWS_MAX_ATTEMPTS), the AWS config file of the active profile or the fsspec configuration; CDXT_AWS_DEFAULTS=0 turns them off. The s3fs defaults are only set by the cdxt command (and its worker processes), not for library users. The multipart_threshold/multipart_chunksize settings are not applied: they are for whole-file transfers, cdxt reads WARC records with range requests and index files sequentially (and S3 parts are limited to 5 GiB).
- Athena client: the same botocore defaults (adaptive retries, 100000 attempts); the region of the athena:// location still wins - DuckDB (httpfs) on S3, as far as it has equivalents: region -> s3_region and the region of the credential secret, max_attempts -> http_retries (attempts - 1, with a constant 1s wait, because DuckDB's exponential backoff has no maximum), addressing style -> s3_url_style; settings the user configured (environment, AWS config file) are used instead, CDXT_AWS_DEFAULTS=0 keeps DuckDB's own defaults - fix DuckDB's http_timeout: it is in seconds, 120000 was ~33 hours, now 120s
- the S3 writer keeps a part's bytes until the part is uploaded; a part that cannot be uploaded aborts the multipart upload, and close() then raises instead of completing an object with missing bytes - any writer failure is fatal: the current file is aborted (local file removed, S3 upload aborted), the files written before stay complete - cancellation and failures cancel all tasks without waiting for queue space; the producer thread stops via an event instead of blocking on a full queue - the separate writer shutdown coordinator (a polling loop) is gone
…tched Captures that cannot be fetched are counted instead of only logged; the others are still written. The pipeline returns the requested, written and failed counts (summed over the worker processes), which the CLI logs; any failed capture makes it exit with status 1. The same applies to --wb. Ctrl-C exits with status 130.
An HTTP range response is only written if it holds exactly the requested bytes: status 206 with the requested Content-Range, or 200 for a range that is the whole file, and as many bytes as requested. A server that ignores the range, a redirect (not followed), a wrong range or a truncated body makes the capture fail. The HF reader uses the same check, the S3 reader checks the number of bytes. The tests serve WARC files with realistic range responses (tests/fake_warc_server.py).
The S3 client and the HF reader were only created if --warc-download-prefix was s3:// or hf://, so captures whose filename is a full s3:// or hf:// URL failed with the default https prefix. The readers are now created on first use, by the URL scheme of each range job, so mixed input works; the download prefix only resolves relative filenames and is no longer passed to the fetcher or the worker processes.
The URL filters read record['url'] even when neither --url-fgrep nor --url-fgrepv was given, so a --sql query that selects only the WARC location crashed with a KeyError. Records now pass through untouched without URL filters; with a URL filter, a record without url is a usage error that names the missing column.
… have The columns were the fields of the first capture, and other fields of later captures were silently dropped. Now: - SQL results have a fixed schema: NULL columns stay in the record as None (empty in CSV/TSV, left out in JSONL/CDXJ as before), so the first record has all columns - for the CDX API and CDX files, the captures are first written to a temporary file to find all fields - an explicit --fields still selects the columns - the CSV writer raises instead of dropping an unexpected field
--domain blog.example.com matched url_host_registered_domain = 'blog.example.com', which no capture has (their registered domain is example.com), while CDX files selected the captures below blog.example.com. The SQL query now matches the same SURT prefixes as CDX files (with a host boundary), and keeps a registered-domain condition (one of the domain's suffixes) so the engines can still prune. A shared test runs the same selectors against CDX files and DuckDB.
The results of --url, --hostname and --domain were concatenated, so a capture selected by two of them was written twice and used up the limit twice. With several selectors, duplicates are now skipped (by URL, timestamp and WARC location, so distinct captures of the same URL stay), and --limit is applied after that instead of by the server. The help of --count says that the API estimate counts overlaps more than once.
… capped by --limit - CDX files applied --url-fgrep/--url-fgrepv while reading, before --limit, contrary to the documented order; they are now applied after the limit like for the other backends (one output per CDX file, which rejects --limit, still filters while reading) - --count is at most --limit on every backend (the CDX API estimate and the CDX file count ignored it) - --count with --url-fgrep/--url-fgrepv is a usage error (they are client-side filters) - --limit below 1 is a usage error
The HTTP reader called the synchronous request helper from the event loop, so network waits and retry sleeps blocked the other readers and the writer, and --parallel-readers had no effect for http(s). The request now runs in a worker thread. The minimum interval between requests to a host (e.g. 0.55s for data.commoncrawl.org) is kept across threads: each request reserves its start time under a lock.
…y bounds - botocore ignores AWS_REGION, so with only AWS_REGION set the client had no region (or the default); it is now passed to botocore explicitly (an explicit region still wins, then AWS_REGION, AWS_DEFAULT_REGION, the profile, the default) - an explicit retries.max_attempts (e.g. in the fsspec configuration) was merged with the default total_max_attempts=100000, which botocore prefers; an explicit bound of either form now replaces the default one The tests check the configuration of real botocore clients (no requests).
--to 20240801 became fetch_time <= '2024-08-01 23:59:59', which excluded captures at e.g. 23:59:59.500. The end is now exclusive at the next second (fetch_time < '2024-08-02 00:00:00'); an end beyond year 9999 is a usage error.
…e themselves The files were staged as warc-001.warc.gz and renamed when copied, so their warcinfo record named the staging file. They are now staged under the basename of the prefix, so the WARC-Filename is the final name from the start.
… reads index records only - the SQL query building (escaping, guided WHERE clause, crawl names) is now cdx_toolkit.indexes.sql_query; running queries is cdx_toolkit.indexes.athena and cdx_toolkit.indexes.duckdb, which yield rows; the range-job paths of the Athena and DuckDB sources, build_athena_query and validate_result_columns are gone (cdxt index writes the rows, cdxt repackage reads them like any index file) - confirm_cost takes the number of crawls and the engine (no CostEstimate type, no dependency of indexes on repackage); the unused SqlBackend._prepared is gone - repackage/sources/ is now repackage/records_source.py (RecordsSource, join_warc_url); the multi-process shards are JSONL files written and read with record_io instead of a separate range-job CSV reader/writer; RangeJob.extra is gone
… S3 and metadata paths offline - the local and S3 writer tests check behaviour (buffering, exact bytes, part sizes, close, abort) with a fake S3 client instead of repeating constructor assignments - offline tests for S3 output with metadata records, a failing metadata record, S3 read lengths, crawl name resolution and URL filters with one output per CDX file - LocalFileWriter.close closes the file in a finally
…oto) and DuckDB (local index) - the WARC records the repackage tests download are recorded in tests/data/repackage/warc_records.jsonl and served for data.commoncrawl.org range requests (responses; DISABLE_MOCK_RESPONSES=1 still uses the real server) - the S3 tests run against a local moto server with s3://commoncrawl holding the same records (configured with environment variables, so worker processes use it too); with CDXT_LIVE_TESTS=1 they run against AWS - the DuckDB end-to-end tests query a local columnar index built from the same captures (local_url_index fixture); the tests of Common Crawl's index on S3 and of Athena only run with CDXT_LIVE_TESTS=1, and the AWS access checks no longer run at collection time - the two remaining live requests of main (example.com, a non-existent host) are mocked - CI sets CDXT_DISABLE_ATHENA_TESTS (it set DISABLE_ATHENA_TESTS, which no test read) - moto is in the all extra (not in test: it would install boto3 for the minimal tests)
- tests: CDXT_DISABLE_MOCK_RESPONSES, CDXT_SAVE_MOCK_RESPONSES, CDXT_FAKE_GITHUB_ACTION - the log level: CDXT_LOGLEVEL (LOGLEVEL still works)
…e output per CDX file) The warcinfo record named the whole local path on Windows (split at '/' only), and one output per CDX file used the whole path as the output name; both use os.path.basename for local paths now. The S3 metadata test writes its file in binary mode (no CRLF on Windows).
…may have real AWS settings) Since Python 3.14, Linux starts worker processes from a forkserver, which an earlier test may have started with the environment of that time: the workers then read the real s3://commoncrawl (with CI's AWS credentials) instead of the moto server. Spawned workers get the current environment, which points to the moto server.
The pull request CI needs no AWS credentials anymore (S3 is mocked); CI-slow logs in to AWS and runs the tests with CDXT_LIVE_TESTS=1 (and installs .[all] for the S3 and DuckDB tests).
…overwriting an earlier run)
…d from the README; removed the dead link to the moved benchmark notes
… CDX file needs a new or empty directory and an -o path without extension - the output was opened before the lazy input, so -o equal to the input (also ./, symlinks, hard links, a glob member, the --url-file, a Parquet index) truncated it; now exit 2 before anything is opened - a rerun into the same directory left the outputs of the earlier run (e.g. a run that selected nothing) - -o out.txt silently wrote one output per file; an unknown extension now needs --output-format
…names, domains and URL patterns include ports - example.com/blog/* selected /blogger and /blog?x (the SURT drops the trailing slash); now like pywb's prefix search range - CDX files: --hostname and --domain missed captures with a port (com,example:8443)/), which SQL selected - a URL pattern with a port and without scheme (example.com:8080/*) became an invalid SURT key - the selector parity tests use SURT keys written out, not computed by the code under test
matchType=host selects the SURT host, the same for example.com and www.example.com; the captures of the other host are now dropped before --limit, which is then not sent to the server. --count stays an estimate of the SURT host (documented).
…or Ctrl-C The source ran in asyncio.to_thread, whose default executor asyncio.run() waits for at shutdown: with stdin from a process that keeps running (or a blocking generator), a failed writer or Ctrl-C hung. The source now runs in a daemon thread that nothing waits for (run_in_daemon_thread, as for the HTTP reads), and stops before its next record once the pipeline stops.
ex.map() returned the results in order and the executor waited for all workers, so the other processes ran their shards to the end. The workers now share a cancel event that each pipeline watches (Repackager(cancel=...) aborts the current file and raises RepackageCancelled); the parent raises the first failure (or Ctrl-C) as soon as it happens, after setting the event, cancelling the pending workers and terminating those that have not stopped after 60 s.
… the S3 defaults too A spawned process (macOS, Windows) does not inherit the fsspec configuration set by the main process; only the repackage workers configured it. Both process pools now come from aws.process_pool(), which applies configure_fsspec_s3() in every new process.
…er-host rate limiter The slot was reserved once per request, so with parallel readers a retry could start together with another reader's request. The limiter also uses time.monotonic() now.
…lear error for older versions 0.30 (the old minimum) and 1.5.0 (the Buckets API) treat hf://buckets/ paths as model repos; 1.6.0 and 1.7.0/1.7.1 raise EntryNotFoundError for a missing path, which broke the existing-output checks. Verified with the live HF tests on Python 3.9: 1.7.2 passes, 1.7.1 fails. An installed older version (which satisfied >=0.30) is now reported before any work (exit 2); installing the extra upgrades it. New live test: cdxt index -o hf://.
…s than cc They were passed to open_index() and silently ignored by the CDX files, DuckDB and Athena backends, and by ia and custom CDX API URLs (CDXFetcher only uses them for cc).
…ery backend --to 202413 raised an uncaught IllegalMonthError for SQL indexes (the padding was outside the error handling), and CDX files accepted impossible dates such as 20240230. timeutils.timestamp_range() checks the padded first and last second of a timestamp; IndexBackend.validate() uses it for all backends (exit 2 before any request), and the SQL bounds are built from it.
…] as a glob) Also in the error messages that suggest an install, and in the CI workflows.
The README, the help texts and the migration guide used a homepages.sql that was nowhere: the homepages of .va in one crawl, with the columns cdxt repackage needs; tested on a local copy of the index's columns, and run once on s3://commoncrawl (50 captures in 41 s).
…cstrings _output_filename split an S3 URL only to rebuild it (generate_warc_filename takes the whole prefix); removed the stale TODO about .extracted.warc and the docstring that said subprefixes distinguish the processes.
…ction error
index.commoncrawl.org answered with the text of an exception ("Error: ('Connection aborted.', RemoteDisconnected(...")
when it could not read its index, with a status that was not retried, so the response could not be decoded
(seen in the live tests with --closest). Such answers are now retried like a 503; the decode error names the status.
… reader (flaky CI) - Before 3.11, asyncio.run() raises KeyboardInterrupt wherever the main thread is, e.g. after the writer created a file and before it registered it, so abort() missed it (Linux/3.9: 15 of 40 runs left a file, 10 hung). Repackager.run() now cancels the pipeline on SIGINT, like asyncio.run() does since 3.11. - If cdxt stops while the source thread still waits for input (stdin from a process that keeps running), Python can hang at exit (on Windows, closing stdin waits for the pending read): the CLI then writes its message and exits without finalization (os._exit).
…s run The check looked at every thread named cdxt-source, so a thread left waiting by another run in the same process (in CI: an earlier test) made an unrelated error exit with status 1, or end the whole process with os._exit. The CLI now only considers source threads that were not running before the command.
…tart times On a slow CI runner (macOS) a thread that wakes up late for its turn starts close to the next one; the test now checks that every attempt, also the retry, reserves its own turn, the minimum interval from the others.
The child's stderr (log lines and a traceback, about 3 KB, more with Windows paths) went to a pipe that was only read after the child exited; a full pipe (a few KB on Windows) blocks the child, so the test timed out on Windows. stderr goes to a file now, and the child dumps the stacks of its threads if it hangs.
…hread It did not fix the Windows timeout (the test's full stderr pipe did, see 9e96e67), and finding the thread of the current run by name is racy: a late thread of an earlier run made an in-process CLI call (in CI: pytest) end with os._exit. The Ctrl-C handling for Python < 3.11 stays.
malteos
marked this pull request as ready for review
October 15, 2026 19:42
wumpus
self-requested a review
October 15, 2026 20:04
…d warc in the migration guide
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR refactors the CLI according to the specs defined in the internal Google doc.
The old
iter/size/warccommands are replaced by two commands that can be piped:The old command lines (
cdxt --cc --limit 10 iter 'commoncrawl.org/*',size,warc) keep working, deprecated(#80): they are parsed with the 0.9.39 grammar, run with the new code and print a suggested new command line on
stderr (see Breaking changes).
cdxt index— select captures from an indexcc,ia,https://…(no file extension)CDXFetcher, unchanged)s3://commoncrawl/cc-index/collections/CC-MAIN-2024-30/indexes/cdx-*.gzurl-is-in)cc-url-index-s3,cc-url-index-hf,*.parquet, a directory with--index-backend duckdbathena://athena.<region>.amazonaws.com/<db>?S3OutputLocation=s3://…The backend is detected from INDEX;
--index-backend cdx-api|cdx-files|duckdb|athenaoverrides it.--url(CDX API pattern syntax for every backend),--hostname,--domain,--crawl,--from/--to,--limit,--url-fgrep(v);backend-specific:
--filter/--closest(CDX API),--language/--sql(SQL),--url-file/--url-column/--url-file-type(CDX files).Options a backend doesn't support are rejected with a usage error, also
--cc-mirror/--cc-sortfor other indexes than
cc. For--url-filewith SQL, the error explains how to write a joinquery instead.
--from/--to/--closestare validated as calendar dates by every backend./or?as aboundary like pywb (
example.com/blog/*does not select/blogger);--hostnameis the exact host,also with the CDX API (whose
matchType=hostmergeswww.; the other host's captures are droppedbefore
--limit); hostnames, domains and URL patterns include captures with a port.-o(stdout by default, local or remote), JSONL / CDXJ (cdx) / CSV / TSV /Parquet (
[parquet]extra),--fields. For CDX files, a glob with an-opath without fileextension (or ending with
/) writes one output per CDX file (lines unchanged,--processes N)into a new or empty directory; an unknown extension (
out.txt) needs--output-format.An output that would overwrite an input of the query (the CDX/Parquet files,
--url-file; alsovia
./, symlinks or hard links) is refused before anything is opened.--count(CDX API: cheap page-based estimate; SQL/CDX files: exact),--dry-run(prints the SQL /API requests / file list), cost guard for SQL scans over >10 or unknown crawls (
--yes).url_host_name_reversed, which follows the sort order of the columnarindex, so DuckDB and Athena can skip row groups:
--domainand--hostnamematch on it, and--urladds it as a hint to theurl_surtkeycondition (IP addresses keep the old predicates).On CC-MAIN-2026-17 over S3,
--domainon one Parquet file went from 87 s / 45 MB to 3 s / 0.8 MB,and
--domain commoncrawl.org --countover the whole crawl takes 149 s (the old predicate timed out).cdxt repackage— write the captures to WARC filescdxt indexoutput (file, glob or stdin; format detected from the first line, so CC's.cdxfiles work directly, or set with--input-format). It also accepts the columnar index'swarc_filename/warc_record_*columns, with full URLs or names relative to
--warc-download-prefix.https://,s3://and
hf://, output to local,s3://orhf://,--processes,--parallel-readers,--add-metadata-record.--wb(replay of captures without a WARC location, e.g. IA captures) runsthrough the same pipeline and writer, also with
--parallel-readersand--add-metadata-record(not
--processes). It needsurl,timestampandstatus, and refuses Common Crawl captures.warcinforecords right away. The shard renumbering/merge step (merge.py,--keep-shards)is gone.
rate-limited
/resolve/request per WARC file instead of per record. The slowerHfFileSystemreader (--hf-reader) was removed, based on the benchmarks.--prefixisREPACKAGE.--prefix/--subprefixwhose WARC files exist already (exit 2), so runs arenever mixed.
--processesall processes (a sharedcancel event; a process that has not stopped after 60 s is terminated); the files being written
are removed or their S3 uploads aborted, the completed files are kept. A source that blocks
(e.g. stdin from a running process) no longer delays this.
Network and AWS settings
See docs/settings.md. Requests to a host keep a minimum interval (e.g. 0.55 s for
data.commoncrawl.org), also across--parallel-readersand for every retry. S3, Athena and DuckDBuse Common Crawl's recommended AWS defaults (region
us-east-1, adaptive retries, virtual addressing,...) for whatever the user has not configured (environment, AWS config file, fsspec configuration),
in every worker process too;
CDXT_AWS_DEFAULTS=offdisables them.Usage guides
docs/usage/: HTTP (no account), AWS: Athena and S3,
Hugging Face (Storage Buckets, HF Jobs); linked from the README.
Python API
cdxt indexandcdxt repackagecall the same code that Python users call:CDXFetcherandCaptureObjectstay as the low-level CDX API client.cdx_toolkit.warc.get_writer()/CDXToolkitWARCWriterare deprecated (FutureWarning, to be removed in a future version).Breaking changes
cdx_toolkit/cli/legacy.py): options beforethe command,
iter(same text /--jsonl/--csvoutput,--get),size(also--details) andwarc(streamed into
repackage()). Each run prints one stderr line: deprecated, will be removed in a future version,and a suggested new command line.
iter --all-fields --csvis a usage error (it was a traceback).--urlinstead of a positional argument,and output defaults to JSONL with all fields (no
k v, k vtext format).repackageand of the oldwarc(also with--wb) are<prefix>-001.warc.gz(was…-000000.extracted.warc.gz), without the addedWARC-Source-URI/WARC-Source-Rangeheaders; no file iswritten without captures, and failed captures give exit status 1.
scripts/cdx_iterandscripts/cdx_sizescripts were removed.fsspec,aiofilesandurl-is-inare now required. New extras:athena,duckdb,hf(huggingface_hub>=1.7.2, the first version whoseHfFileSystemhandles StorageBuckets; an older installed version gets a clear error),
parquet;s3now also includesaioboto3.pad_timestamp_up()replaced a given day with the last day of the month and ignored leapyears. This also affected which crawls a CDX API
--to <day>query covered.See docs/cli-migration-guide.md for old → new examples, including the #73
repackage/filter_cdxoptions.Reviewer notes
IndexBackend.records()/count()/explain() validate the query first, then call the backend's select()/select_count().
CDXT_LIVE_TESTS=http,s3,athena,hf, AWS credentials and theHF_TOKENsecret; each live test still skips when its dependency or permission check fails (e.g. Athena needsathena:ListDatabases).CDXT_LIVE_TESTS=http,s3,athena,hf(orall) replacesCDXT_DISABLE_MOCK_RESPONSES,CDXT_DISABLE_S3_TESTSandCDXT_DISABLE_ATHENA_TESTS; the tests that can run live are markedliveandlive_<service>, e.g.CDXT_LIVE_TESTS=s3 pytest -m live_s3 tests. CONTRIBUTING.md lists the services. The HF tests read from hf://buckets/commoncrawl/commoncrawl and write to hf://buckets/commoncrawl/commoncrawl-ci-temp (CDXT_TEST_HF_BUCKET).~/.cache/cdx_toolkit/.Live validation (separate from the mocked suite)
pytest -m live_hf, incl.cdxt index -o hf://)pytest -m live_s3)test_index_athena_e2e)ccindexexamples/homepages.sqlvia DuckDB on s3://commoncrawl--hostnameOpen tasks
--hostnamecheck is pending.HF_TOKENsecret for ci-slow.yaml.