Skip to content

fix(crawl): send obeyRobots=0 / useProxies=0 when explicitly False - #13

Merged
shixish merged 1 commit into
mainfrom
fix/crawl-falsy-flags
Nov 18, 2026
Merged

shixish merged 1 commit into
mainfrom
fix/crawl-falsy-flags

Conversation

@shixish

@shixish shixish commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

In 3.4.0, crawl(obey_robots=False) omitted the obeyRobots param, so an explicit False was silently ignored and robots.txt was still obeyed (the API default). use_proxies=False had the same "falsy flag dropped" bug.

  • obey_robots / use_proxies are now tri-state (Optional[bool] = None), like use_canonical: None omits the param (unchanged behaviour), True sends 1 (unchanged), False now sends 0. Sync and async share _build_crawl_params, so both are fixed.
  • CLI db crawl: --obey-robots/--no-obey-robots and --use-proxies/--no-use-proxies (default: omitted → API default), and --restrict-domain/--no-restrict-domain (it was is_flag with default=True, so it could never be turned off).
  • Audited the other boolean params: restrict_domain already sends 0; use_canonical already tri-state; DQL non_canonical_facts/no_dedup_articles only filter None. The SDK has no bulk endpoint.

Tests

  • tests/test_crawl.py: parametrized None/True/False for obeyRobots and useProxies, explicit obey_robots=False → obeyRobots=0, restrict_domain=False → 0, async variant.
  • tests/test_cli_output.py: CLI defaults (None/None/True), --no-* → False, --* → True.

🤖 Generated with Claude Code

crawl(obey_robots=False) and crawl(use_proxies=False) dropped the param,
so the API default silently applied. Both are now tri-state (None omits,
True/False send 1/0), matching use_canonical. The CLI gains
--no-obey-robots, --no-use-proxies and --no-restrict-domain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shixish
shixish merged commit 3fed32d into main Nov 18, 2026
8 checks passed
@shixish
shixish deleted the fix/crawl-falsy-flags branch November 18, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant