DFO Protocol Update

[eflumerf]

Description

See issue #428 for details

This PR introduces a "DataflowStatusModule", which lives in each Dataflow application and tracks the state of that application (i.e. TriggerRecords being built and written). The DFOModule now requests the state from each Dataflow application before making a decision. Statuses are locked to the decision/iteration number, and a Dataflow application will always respond with the same status to the same request. Likewise, the DFOModule's selection algorithm is stable when given the same set of statuses, so multiple DFOModules can run concurrently as they will always arrive at the same decisions.

Type of change

• Documentation (non-breaking change that adds or improves the documentation)
• New feature or enhancement (non-breaking change which adds functionality)
• Optimization (non-breaking change that improves code/performance)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (whatever its nature)

Testing checklist

• Unit tests pass (e.g. dbt-build --unittest)
• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)
• Python tests pass if applicable (e.g. python -m pytest)
• Pre-commit hooks run successfully if applicable (e.g. pre-commit run --all-files)

Comments here on the testing

Further checks

• Code is commented where needed, particularly in hard-to-understand areas
• Code style is correct (dbt-build --lint, and/or see https://dune-daq-sw.readthedocs.io/en/latest/packages/styleguide/)
• If applicable, new tests have been added or an issue has been opened to tackle that in the future.

Here are sample instructions for testing that these changes work as intended:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260804_A9 ${DATE_PREFIX}DFOProtocol_${TIME_SUFFIX}
cd ${DATE_PREFIX}DFOProtocol_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/dfmodules.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/daqconf.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/appmodel.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/trigger.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/daqconf.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/dfmessage.git -b eflumerf/DFOProtocol_StatusModule

git clone https://github.com/DUNE-DAQ/appfwk.git -b eflumerf/AddErrorMessage
git clone https://github.com/DUNE-DAQ/ipm.git -b eflumerf/SendRetryUsleep

git clone https://github.com/DUNE-DAQ/datahandlinglibs.git -b develop
git clone https://github.com/DUNE-DAQ/fdreadoutlibs.git -b develop
git clone https://github.com/DUNE-DAQ/fdreadoutmodules.git -b develop
git clone https://github.com/DUNE-DAQ/hdf5libs.git -b develop
git clone https://github.com/DUNE-DAQ/hsilibs.git -b develop
git clone https://github.com/DUNE-DAQ/iomanager.git -b develop
git clone https://github.com/DUNE-DAQ/asiolibs.git -b develop
git clone https://github.com/DUNE-DAQ/crtmodules.git -b develop
git clone https://github.com/DUNE-DAQ/ctbmodules.git -b develop
git clone https://github.com/DUNE-DAQ/daphnemodules.git -b develop
git clone https://github.com/DUNE-DAQ/dpdklibs.git -b develop
git clone https://github.com/DUNE-DAQ/flxlibs.git -b develop
git clone https://github.com/DUNE-DAQ/snbmodules.git -b develop
git clone https://github.com/DUNE-DAQ/hermesmodules.git -b develop
git clone https://github.com/DUNE-DAQ/listrev.git -b develop
git clone https://github.com/DUNE-DAQ/sspmodules.git -b develop
git clone https://github.com/DUNE-DAQ/tdemodules.git -b develop
git clone https://github.com/DUNE-DAQ/timinglibs.git -b develop
git clone https://github.com/DUNE-DAQ/wibmod.git -b develop
git clone https://github.com/DUNE-DAQ/trgtools.git -b develop
git clone https://github.com/DUNE-DAQ/restcmd.git -b develop
cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b eflumerf/DFOProtocol_StatusModule
git clone https://github.com/DUNE-DAQ/drunc.git # For nightly releases before 260821
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -r all -k 'graceful|listrev|disabled_tpg|tpstream'

echo ""
echo -e "\U1F535 \U2705 Note that the integtests ran successfully. \U2705 \U1F535"
echo ""
echo ""

[jcfreeman2]

I'm running into problems with the long_window_readout_test, getting output like this:

2026-Aug-13 13:31:13,898 ERROR [void dunedaq::dfmodules::DFOModule::do_stop(const dunedaq::appfwk::DAQModule::CommandData_t&) at /home/nfs/jcfree/daqbuild_test_DFO_PR/sourcecode/dfmodules/plugins/DFOModule.cpp:202] TriggerDecision 1 didn't complete within timeout in run 102 DAQModule: DFO-dfo-01

F
🚨 An incorrect number of raw data files was found, expected 4, found 2 🚨

...using a work area based on today's nightly, NFD_DEV_260813_A9 (not the August 4th nightly, haven't tried it yet(*)), with dbt-info sourcecode's output as follows:

Source code for locally checked out packages:
dfmessages: eflumerf/DFOProtocol_StatusModule [b3b783f]
dfmodules: eflumerf/DFOProtocol_StatusModule [c9fb938]
trigger: eflumerf/DFOProtocol_StatusModule [484af92]
appmodel: eflumerf/DFOProtocol_StatusModule [fa82b25]
daqconf: eflumerf/DFOProtocol_StatusModule [1daf2cc]
daqsystemtest: eflumerf/DFOProtocol_StatusModule [4c3b24b]
appfwk: eflumerf/AddErrorMessage [18bbe5d]
ipm: eflumerf/SendRetryUsleep [121eba2]
integrationtest: eflumerf/DFOProtocol_StatusModule [d550c00]

Source code for packages not checked out locally is in
/cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFD_DEV_260813_A9/sourcecode

(*) using NFD_DEV_260804_A9 I see the same thing

[MRiganSUSX]

Notes from testing at EHN1 (@eflumerf @mroda88):

SETUP:

• following your notes/recipe above but with latest nightly
• area: dunedaq-NFD_DEV_261009_A9-dev-DFO-test
• for the configs, I made a mock release called mrigan/DFOTest and pushed branches there (starting from develop + merging in your branch)
• session: main-np02-DFO
• grafana: link
• there were issues from integtesting, but unrelated (I think)

WHAT WAS SEEN:

• generally, there are trigger decisions being made, data requests follow, data is written without any errors
• no associated ERS messages, issued triggers increase, written TRs increase
• there are actual data files written offline
• for the monitoring, I do see data flowing with the new/updated metrics (the new DFOInfo and DataflowStatusInfo metrics arrive and are consistent)
• configs seem right, in the sense that I can run all the generators and there were no issues/errors

PROBLEMS:

• TPStreamWriterInfo is never stored. This is because tardy_timeslice_max_seconds was changed from uint64 to double. The existing field is integer and influxdb rejects it:

2026-10-09T16:53:56.542574967+02:00 2026-10-09 14:53:56 ERROR    InfluxDB client error occurred
2026-10-09T16:53:56.542586747+02:00 Traceback (most recent call last):
2026-10-09T16:53:56.542590608+02:00   File "/opt/app/microservices/opmon-protobuf-dbwriter/./dbwriter.py", line 238, in send_batch
2026-10-09T16:53:56.542595194+02:00     influx.write_points(batch)
2026-10-09T16:53:56.542598659+02:00   File "/opt/venv/lib64/python3.12/site-packages/influxdb/client.py", line 614, in write_points
2026-10-09T16:53:56.542602371+02:00     return self._write_points(points=points,
2026-10-09T16:53:56.542606073+02:00            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-10-09T16:53:56.542609310+02:00   File "/opt/venv/lib64/python3.12/site-packages/influxdb/client.py", line 692, in _write_points
2026-10-09T16:53:56.542612622+02:00     self.write(
2026-10-09T16:53:56.542615946+02:00   File "/opt/venv/lib64/python3.12/site-packages/influxdb/client.py", line 424, in write
2026-10-09T16:53:56.542619427+02:00     self.request(
2026-10-09T16:53:56.542622832+02:00   File "/opt/venv/lib64/python3.12/site-packages/influxdb/client.py", line 389, in request
2026-10-09T16:53:56.542626065+02:00     raise InfluxDBClientError(err_msg, response.status_code)
2026-10-09T16:53:56.542630579+02:00 influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: field type conflict: input field \"tardy_timeslice_max_seconds\" on measurement \"dunedaq.dfmodules.opmon.TPStreamWriterInfo\" is type float, already exists as type integer dropped=1"}

Not sure we want to wait for the DB to fix itself (it can fix types periodically), but then people can still run older releases... Otherwise, either change name or type back to int ?

• pending_trigger_decisions can't show backlogs. It reads m_assigned_trigger_decisions.size() once at publish time. At low rates it's always 0, a short backlog that clears between two publishes is never seen. Maybe adding max_pending_trigger_decisions peak value?
• There was a burst at run start from statuses_received (think because the DF apps start counting heartbeats before DFO ?)
• requests_received is 0 until the first trigger, then exactly 1 per 10 s per DF app, even for apps that got no decision in that window
• Many plots on the Dataflow dashboard that showed data before are now empty: because things like waiting_for_token/processing_token were removed but were used. Combined with that, nothing is yet using DataflowStatusInfo.
• Data Writers Information on the main dashboard is also empty now, due to a rename outside of this PR? (@mroda88)
• Global Trigger Rate: the data for tokens_received is also empty, uses ``DFOInfo.tokens_received` which was removed. Clearly there are changes even outside of Dataflow dashboard.

FROM TESTING:

• integtests:

running:
(new monitoring data is flowing)

(several plots now empty / missing data, using the old names)