When a sidecar annotation references a value column with curly braces
("ball": "{response_time}, Green"), the column mapper removes that column from the per-column
assembly and folds it into the referencing annotation. Rows whose annotation does not reference it
(here the face rows) then never have their response_time value validated at all. An invalid value
in such a row, for example 7,3 substituted into Parameter-value/#, passes silently. Without the
brace reference the same data reports TAG_INVALID.
Where it shows. hed-tests validation_tests.json, record sidecar-braces-self-reference,
combo_tests.passes[0]: the last events row has response_time = "7,3" with event_code = "face",
and face does not reference {response_time}. The case passes only because of this gap. (The test
data is being changed to 7.3; the validator gap remains.)
import io, json
import pandas as pd
from hed.errors import ErrorHandler, get_printable_issue_string
from hed.models import Sidecar, TabularInput
from hed.schema import load_schema_version
schema = load_schema_version("8.4.0")
events = pd.DataFrame({
"onset": ["4.5", "5.0", "5.5"],
"response_time": ["3.4", "6.8", "7,3"],
"event_code": ["face", "ball", "face"],
})
def run(sidecar_dict, label):
sidecar = Sidecar(io.StringIO(json.dumps(sidecar_dict)), name="events.json")
tabular = TabularInput(file=events, sidecar=sidecar, name="events.tsv")
print(f"--- {label}")
print("assembled per column:", tabular.dataframe_a.values.tolist())
issues = tabular.validate(schema, error_handler=ErrorHandler(check_for_warnings=False))
print("issues:", [(i["code"], i.get("ec_row")) for i in issues])
if issues:
print(get_printable_issue_string(issues))
run({"event_code": {"HED": {"face": "(Red, Blue)", "ball": "{response_time}, Green"}},
"response_time": {"HED": "Parameter-value/#"}},
"response_time referenced by {response_time} in the 'ball' annotation")
run({"event_code": {"HED": {"face": "(Red, Blue)", "ball": "Green"}},
"response_time": {"HED": "Parameter-value/#"}},
"same data, no brace reference")Output:
--- response_time referenced by {response_time} in the 'ball' annotation
assembled per column: [['(Red, Blue)'], ['Parameter-value/6.8, Green'], ['(Red, Blue)']]
issues: []
--- same data, no brace reference
assembled per column: [['(Red, Blue)', 'Parameter-value/3.4'], ['Green', 'Parameter-value/6.8'], ['(Red, Blue)', 'Parameter-value/7,3']]
issues: [('TAG_INVALID', 4)]
Issues in row 4:
Issues in column response_time:
hed string: Parameter-value/7,3
TAG_INVALID: '3' in 3 is not a valid base HED tag. Problem spans string indexes: 18, 19
In the first run the assembled dataframe has one column: response_time has been consumed by the
reference and appears only in the ball row. Its value in the two face rows (3.4 and 7,3) is
never substituted into Parameter-value/# and never validated.
Expected. Every non-n/a value of a value column is substituted into its template and validated,
whether or not the row's other annotations reference the column. A reference changes where the
expansion is placed in the assembled string, not whether the value is checked. At minimum the value
should satisfy the value class of the placeholder tag (Parameter-value/# is textClass, which
forbids a comma) in every row.
Why it matters beyond this case. A dataset that references a value column from one category
({response_time} only for response events) is the normal use of the brace syntax. Every row of the
other categories carries a value that is never checked, so a malformed value there is invisible to
hed-validator and to BIDS validation.
Where to look. The column-reference handling in the column mapper / df_util that removes a
referenced column from the standalone expansion (dataframe_a above shows the removal). One fix:
keep the per-column expansion of a referenced column for rows whose referencing annotation does not
apply, or validate the substituted value for every row before the reference is resolved.
When a sidecar annotation references a value column with curly braces
(
"ball": "{response_time}, Green"), the column mapper removes that column from the per-columnassembly and folds it into the referencing annotation. Rows whose annotation does not reference it
(here the
facerows) then never have theirresponse_timevalue validated at all. An invalid valuein such a row, for example
7,3substituted intoParameter-value/#, passes silently. Without thebrace reference the same data reports
TAG_INVALID.Where it shows. hed-tests
validation_tests.json, recordsidecar-braces-self-reference,combo_tests.passes[0]: the last events row hasresponse_time = "7,3"withevent_code = "face",and
facedoes not reference{response_time}. The case passes only because of this gap. (The testdata is being changed to
7.3; the validator gap remains.)Output:
In the first run the assembled dataframe has one column:
response_timehas been consumed by thereference and appears only in the
ballrow. Its value in the twofacerows (3.4and7,3) isnever substituted into
Parameter-value/#and never validated.Expected. Every non-
n/avalue of a value column is substituted into its template and validated,whether or not the row's other annotations reference the column. A reference changes where the
expansion is placed in the assembled string, not whether the value is checked. At minimum the value
should satisfy the value class of the placeholder tag (
Parameter-value/#istextClass, whichforbids a comma) in every row.
Why it matters beyond this case. A dataset that references a value column from one category
(
{response_time}only for response events) is the normal use of the brace syntax. Every row of theother categories carries a value that is never checked, so a malformed value there is invisible to
hed-validatorand to BIDS validation.Where to look. The column-reference handling in the column mapper /
df_utilthat removes areferenced column from the standalone expansion (
dataframe_aabove shows the removal). One fix:keep the per-column expansion of a referenced column for rows whose referencing annotation does not
apply, or validate the substituted value for every row before the reference is resolved.