Skip to content

Require search-tool-sourced DCIDs in observation and metadata tool instructions - #219

Draft
keyurva wants to merge 1 commit into
datacommonsorg:mainfrom
keyurva:dcid-search-only-tool-instructions
Draft

keyurva wants to merge 1 commit into
datacommonsorg:mainfrom
keyurva:dcid-search-only-tool-instructions

Conversation

@keyurva

@keyurva keyurva commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Added a constraint to the observation and metadata tool descriptions stating that variable DCIDs must come from the search tools (search_indicators or search_child_indicators) and must not be guessed, assumed, or constructed from memory.

Updated tool instruction files:

  • tools/get_observations.md
  • tools/get_child_observations.md
  • tools/get_multi_entity_observations.md (also updated the variable_dcid parameter description)
  • tools/get_variable_metadata.md

Added the same DCID sourcing rule to skills/data-commons-multi-entity-researcher/SKILL.md, which did not have one. The single-place and child-places researcher skills already state this rule.

Why

server.md and the researcher skills already prohibit guessing DCIDs, but those are delivered at session initialization and via opt-in resource reads respectively. Tool descriptions are always present in the tool schema at the point where the model constructs the variable_dcid argument. Hallucinated DCIDs return empty or wrong results rather than a loud error, so the constraint is stated at the point of use as well.

Scope

Markdown instruction files only. No code or behavior changes.

…structions

Added a constraint to the get_observations, get_child_observations,
get_multi_entity_observations, and get_variable_metadata tool descriptions
stating that variable DCIDs must come from search_indicators or
search_child_indicators and must not be guessed.

Added the same DCID sourcing rule to the multi-entity researcher skill, which
did not have one. The single-place and child-places researcher skills already
state this rule.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates several instruction and tool markdown files to explicitly mandate that only DCIDs returned by search tools should be used, preventing the model from guessing or assuming them. The review feedback correctly identifies an inaccuracy in the SKILL.md file, where the instructions suggest that entity DCIDs are returned by tools that actually only return variable DCIDs or accept entity DCIDs as inputs, and provides a suggestion to clarify this rule.

3. **Retrieval (`get_multi_entity_observations`)**: Fetch the observation tables using the mapped entity properties.

### CRITICAL: Only use DCIDs returned by tools
* Only use variable and entity DCIDs returned by `search_indicators` or `get_variable_metadata` - never guess or assume DCIDs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The instruction states to only use variable and entity DCIDs returned by search_indicators or get_variable_metadata. However, search_indicators only returns statistical variable (indicator) DCIDs, and get_variable_metadata is an assessment tool that takes entity_dcids as an input parameter rather than returning them. This phrasing could confuse the LLM since it cannot obtain entity DCIDs from these tools. It is better to restrict this rule specifically to variable DCIDs.

Suggested change
* Only use variable and entity DCIDs returned by `search_indicators` or `get_variable_metadata` - never guess or assume DCIDs.
* Only use variable DCIDs returned by search_indicators - never guess or assume variable DCIDs.

This branch is waiting to be deployed

1 waiting deployment
evals-and-secrets — c95c2fed Waiting Sep 14, 2026 by keyurva via agent-evals #496
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants