Skip to content

chore(sources): read http client targets on every interval tick - #27156

Open
JinchengCao wants to merge 7 commits into
masterfrom
JinchengCao-http-client-dynamic-targets
Open

JinchengCao wants to merge 7 commits into
masterfrom
JinchengCao-http-client-dynamic-targets

Conversation

@JinchengCao

@JinchengCao JinchengCao commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

call() in src/sources/util/http_client.rs is the scrape loop shared by the http_client and prometheus_scrape sources. It used to take a fixed list of URLs. It now takes a function that returns the targets for each interval tick. A target can also carry source-specific data, which the source's context builder receives along with the URL.

With this change, Kubernetes Pod discovery (#26013) can reuse the shared loop with a target list that changes between ticks, instead of keeping its own copy of the request logic. Existing callers pass a fixed list through fixed_targets() and behave as before.

References

Vector configuration

NA

How did you test this PR?

Added targets_are_read_on_every_tick: the targets change after the first tick, and each event must carry the data of the target it was scraped from. Existing http_client and prometheus_scrape tests pass unchanged.

make test FEATURES="sources-http_client sources-prometheus_scrape" SCOPE="-E 'test(/sources::util::http_client/) or test(/sources::http_client::tests/) or test(/sources::prometheus::scrape/)'"

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
    For a new deprecation announcement, follow the deprecation policy and add a deprecation.d/ notice instead.
  • No. A maintainer will apply the no-changelog label to this PR.

@JinchengCao JinchengCao added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Nov 12, 2026
@github-actions github-actions Bot added the domain:sources Anything related to Vector's sources label Nov 12, 2026
@JinchengCao
JinchengCao marked this pull request as ready for review November 12, 2026 18:41
@JinchengCao
JinchengCao requested a review from a team as a code owner November 12, 2026 18:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T21:59:43.657115Z 446d904 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadoghq-integration datadoghq-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The fixed-target adapter preserves existing callers’ endpoint lists, and each request context receives the same target that supplies its URL. The completed static review identified no reportable defects.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit e8d0074

Base automatically changed from JinchengCao-http-client-body-timeout to master November 12, 2026 22:06

@datadoghq-integration datadoghq-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Static review supports that target URLs remain paired with their source-specific context as targets change between ticks, and that the timeout bounds response-body reads. Runtime behavior was not exercised.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 446d904

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:sources Anything related to Vector's sources no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant