Skip to content

avoid exponential :not() XPath in _xpath_match_condition - #219

Open
itecz26 wants to merge 2 commits into
scrapy:masterfrom
itecz26:not-reverse-path
Open

itecz26 wants to merge 2 commits into
scrapy:masterfrom
itecz26:not-reverse-path

Conversation

@itecz26

@itecz26 itecz26 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

_xpath_match_condition() turns each combinator of a complex :not() argument into a reverse axis nested in the previous step's predicate (:not(x * * *) becomes not(ancestor::*[ancestor::*[ancestor::*[self::x]]])), and libxml2 evaluates a nested predicate again for every node the outer axis reaches, so the generated XPath takes time that grows like C(depth, k) for k descendant combinators, or C(siblings, k) with ~. that's an algorithmic-complexity DoS for anything that runs user-supplied selectors against a page: on 3.13 with lxml 6.1.3, :not(x * * * * * * * *) (23 chars) takes 4.35 s on a 32-element document nested 30 deep, and tr:not(x ~ * ~ * ~ *) takes 5.25 s on a plain 200-row table. emitting the chain as one location path (parent::*[self::b]/parent::*[self::a]) means the same thing in a boolean context but lets libxml2 merge the node-set at each step, which brings those two down to 0.5 ms and 146 ms. single-combinator output doesn't change; the e:not(a > b > c) expectation in test_translation now has the path form, the two longer chains added there fail without the change, two multi-combinator selects go into test_select, and old vs new translations select the same elements on ~40k random complex :not() selectors over random documents.

_xpath_match_condition() nested each reverse combinator step inside the
previous step's predicate, and libxml2 evaluates a nested predicate again
for every node the outer axis reaches, so a complex :not() argument with
k combinators took time exponential in k on deep or wide documents
(4.4 s for :not(x * * * * * * * *) on a 30-deep chain). Emit the steps
as one location path instead, which selects the same elements and lets
each step's node-set be merged.
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (0a0c197) to head (1c389d3).

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #219   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines         1019      1023    +4     
  Branches       170       170           
=========================================
+ Hits          1019      1023    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Looks good to me. Please, drop the code comment and the test comment about nested predicates though, that reasoning is already in the commit message, and the code reads fine without them.

@itecz26

itecz26 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

makes sense, dropped both comments in 1c389d3.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants