Skip to content

StripEnergyThresholdFinder for per-strip slow and fast threshold extraction with diagnostics - #166

Open
JarredMRoberts wants to merge 44 commits into
cositools:develop/emfrom
JarredMRoberts:feature/strip-threshold-finder
Open

JarredMRoberts wants to merge 44 commits into
cositools:develop/emfrom
JarredMRoberts:feature/strip-threshold-finder

Conversation

@JarredMRoberts

Copy link
Copy Markdown

Adds a standalone application, StripEnergyThresholdFinder, for computing per-strip slow and fast energy thresholds. Slow thresholds are determined from ADC spectra using a noise peak and trough method, while fast thresholds are determined from dt0/dt1 timing crossover. The tool reads calibrated HDF5 data via MModuleLoaderMeasurementsHDF, applies strip mapping and energy calibration from a YAML configuration, and produces ROOT diagnostic outputs (energy spectra with thresholds, dt0 vs dt1 per strip, and threshold distributions) along with CSV export files. The implementation is self-contained under apps/ and does not modify existing modules. Tested on COSI datasets with consistent threshold behavior and expected diagnostic results. Target branch is develop/em.

@JarredMRoberts

Copy link
Copy Markdown
Author

I still need to fix all of the code style issues and work on some optimizations to speed the code up a bit.

@fhagemann

Copy link
Copy Markdown

Is this different from #143 or making #143 obsolete?

Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
@fhagemann
fhagemann marked this pull request as draft June 23, 2026 21:50
JarredMRoberts and others added 3 commits June 30, 2026 07:33
Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>
Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>
…nventions and integrate energy calibration improvements
@JarredMRoberts
JarredMRoberts marked this pull request as ready for review July 28, 2026 08:40
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved

@fhagemann fhagemann left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Jarred,
I had an extensive look at this app and finally got this to run.
I replied to all comments in this PR, and opened a separate PR onto your fork/branch, addressing some code changes to

  1. get this app running
  2. remove the helper classes by replacing them with existing nuclearizer/megalib code

Here is the PR onto your branch with detailed code changes: JarredMRoberts#1

Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
@zoglauer

zoglauer commented Aug 17, 2026 •

Copy link
Copy Markdown
Collaborator

I second not to have an additional parser linked.
You can use the nuclearizer/megalib XML file format instead; your file then would be:

<strip-threshold-finder>
    <input>
        <data_files>
            <file>/path/to/data/file.hdf5</file>
        </data_files>
        <calibration_file>/path/to/calibration/file.ecal</calibration_file>
        <tac_calibration_file>/path/to/tac/calibration/file.csv</tac_calibration_file>
        <strip_map>/path/to/strip/map/file.map</strip_map>
    </input>

    <analysis>
        <!-- number of skipped strips with minimum statistics -->
        <min_entries>10</min_entries> 
        
        <!-- If a threshold cannot be determined set threshold to default value -->
        <fallback_threshold_keV>20</fallback_threshold_keV> 
        
        <!-- limit for locating the low-energy noise peak -->
        <!-- most noise peaks should be between 100 and 250 -->
        <!-- Worst case the ADC max should be set to ~1000 -->
        <noise_search_max_adc>1800</noise_search_max_adc>
    </analysis>
    
    <output>
        <prefix>output_file_prefix</prefix>
    </output>
</strip-threshold-finder>

@fhagemann fhagemann left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more remark: I got this app to run and got reasonable results for the fast and slow thresholds only when using a dataset taken with NN off.
We might want to always filter out NN events, for this app to also run using datasets taken with NN on.

Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
@fhagemann
fhagemann force-pushed the feature/strip-threshold-finder branch from bd7161b to 20de738 Compare September 15, 2026 22:50
Comment on lines +1711 to +1714
if (n1 > n0) {
crossoverADC = ADC_val;
break;
}

@fhagemann fhagemann Sep 15, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (n1 > n0) {
crossoverADC = ADC_val;
break;
}
if (n1 > n0) {
crossoverADC = ADC_val;
// Make sure that this is an actual crossover and not just the first bin
// with more than FAST_MIN_CROSSOVER_COUNTS where n1 was above n0
while (n1 > n0 && ADC_val >= first_nonzero) {
crossoverADC = ADC_val;
ADC_val--;
if (ADCMap.find(ADC_val) != ADCMap.end()) {
n0 = ADCMap[ADC_val].first;
n1 = ADCMap[ADC_val].second;
}
}

Adding this backward search seems to resolve the issue above:
Image

@fhagemann fhagemann Sep 15, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

The vimdiff between the fast threshold file created (left) with and (right) without the backward search suggests that this might impact that results for the fast thresholds for most HV strips

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this specific measurement, it looks like the HV spectra seem to feature this dip with <50 counts/bin, such that we might want to employ the backward search:

HV_fast_thresholds

The LV spectra seem fine, because they exceed 50 counts/bin in the region where we have the crossover:

LV_fast_thresholds

@fhagemann
fhagemann dismissed their stale review September 16, 2026 00:26

All major changes I requested are implemented.

Comment on lines +1694 to +1736
for (auto& a : ADCMap) {
int ADC_val = a.first;
int n0 = a.second.first;
int n1 = a.second.second;

// Require minimum statistics to avoid noise triggers
if (n0 + n1 < FAST_MIN_CROSSOVER_COUNTS) {
continue;
}

if (n1 > n0) {
crossoverADC = ADC_val;
// Make sure that this is an actual crossover and not just the first bin
// with more than FAST_MIN_CROSSOVER_COUNTS where n1 was above n0
while (n1 > n0 && ADC_val >= first_nonzero) {
crossoverADC = ADC_val;
ADC_val--;
if (ADCMap.find(ADC_val) != ADCMap.end()) {
n0 = ADCMap[ADC_val].first;
n1 = ADCMap[ADC_val].second;
}
}
break;
}
}

const int nbins = FAST_CROSSOVER_WINDOW_BINS;

// Extend search window for stability
int searchMax = first_nonzero + FAST_SEARCH_RANGE_ADC;

for (int ADC = first_nonzero; ADC < searchMax; ADC++) {
int n_dt0 = 0;
int n_dt1 = 0;

for (int a = ADC; a < ADC + nbins; a++) {
if (ADCMap.find(a) == ADCMap.end()) {
continue;
}

n_dt0 += ADCMap[a].first;
n_dt1 += ADCMap[a].second;
}

@fhagemann fhagemann Oct 3, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When trying this out for Cs-137 data to explore the possibility to use Cs-137 data in payload calibrations for threshold determination, I found that the current implementation would give wrong results if there are a lot of events with fast timing in the noise peak (I guess these are NN events featuring a dominant peak-dip structure, that might give a fast-shaper signal that exceeds the threshold, but in the slow shaper average out to a small energy value).

In this case, the condition n1 .> n0 is reached early:
image

A fix for this might be to start the crossover search only for ADC bins right of the noise peak, i.e. the slow threshold (plus some tolerance):

Suggested change
for (auto& a : ADCMap) {
int ADC_val = a.first;
int n0 = a.second.first;
int n1 = a.second.second;
// Require minimum statistics to avoid noise triggers
if (n0 + n1 < FAST_MIN_CROSSOVER_COUNTS) {
continue;
}
if (n1 > n0) {
crossoverADC = ADC_val;
// Make sure that this is an actual crossover and not just the first bin
// with more than FAST_MIN_CROSSOVER_COUNTS where n1 was above n0
while (n1 > n0 && ADC_val >= first_nonzero) {
crossoverADC = ADC_val;
ADC_val--;
if (ADCMap.find(ADC_val) != ADCMap.end()) {
n0 = ADCMap[ADC_val].first;
n1 = ADCMap[ADC_val].second;
}
}
break;
}
}
const int nbins = FAST_CROSSOVER_WINDOW_BINS;
// Extend search window for stability
int searchMax = first_nonzero + FAST_SEARCH_RANGE_ADC;
for (int ADC = first_nonzero; ADC < searchMax; ADC++) {
int n_dt0 = 0;
int n_dt1 = 0;
for (int a = ADC; a < ADC + nbins; a++) {
if (ADCMap.find(a) == ADCMap.end()) {
continue;
}
n_dt0 += ADCMap[a].first;
n_dt1 += ADCMap[a].second;
}
for (auto& a : ADCMap) {
int ADC_val = a.first;
int n0 = a.second.first;
int n1 = a.second.second;
// Require minimum statistics to avoid noise triggers
if (n0 + n1 < FAST_MIN_CROSSOVER_COUNTS || ADC_val <= m_SlowThresholdsADC[R] + FAST_FIRST_ADC_OFFSET) {
continue;
}
if (n1 > n0) {
crossoverADC = ADC_val;
// Make sure that this is an actual crossover and not just the first bin
// with more than FAST_MIN_CROSSOVER_COUNTS where n1 was above n0
while (n1 > n0 && ADC_val >= m_SlowThresholdsADC[R] + FAST_FIRST_ADC_OFFSET) {
crossoverADC = ADC_val;
ADC_val--;
if (ADCMap.find(ADC_val) != ADCMap.end()) {
n0 = ADCMap[ADC_val].first;
n1 = ADCMap[ADC_val].second;
}
}
break;
}
}
const int nbins = FAST_CROSSOVER_WINDOW_BINS;
// Extend search window for stability
int searchMax = m_SlowThresholdsADC[R] + FAST_SEARCH_RANGE_ADC;
for (int ADC = m_SlowThresholdsADC[R]; ADC < searchMax; ADC++) {
int n_dt0 = 0;
int n_dt1 = 0;
for (int a = ADC; a < ADC + nbins; a++) {
if (ADCMap.find(a) == ADCMap.end()) {
continue;
}
n_dt0 += ADCMap[a].first;
n_dt1 += ADCMap[a].second;
}

(and I also had to set FAST_FIRST_ADC_OFFSET to 20 for this specific dataset I was looking at).

This is what I get with the updated code (but note that FAST_FIRST_ADC_OFFSET might need to be optimized):
image

Comment on lines +2108 to +2134
// Fill graphs
for (const auto& kv : m_SlowHardwareThresholds) {

MReadOutElementDoubleStrip R = kv.first;
int Strip = R.GetStripID();
double Threshold = kv.second;

// Ignore invalid hardware thresholds
if (Threshold < 0.0) {
continue;
}

if (R.IsLowVoltageStrip() == true) {

gSlowHardwareThresh_LV->SetPoint(
gSlowHardwareThresh_LV->GetN(),
Strip,
Threshold);

} else {

gSlowHardwareThresh_HV->SetPoint(
gSlowHardwareThresh_HV->GetN(),
Strip,
Threshold);
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They way the plotting code is written right now does ignore detector ID.
When running this threshold app on payload calibration data, all detectors will be plotted into one plot:

Image

Can this be split into multiple plots (maybe one per "active" detector)?

@fhagemann

Copy link
Copy Markdown

I had to run this app now several times on the same data, and the part that takes the longest is reading the HDF5 files and applying the strip map / energy calibration. Would it be possible to also support reading ROA files as input (and not just HDF5) to avoid this first step in the threshold app having to be re-run every time I want to change settings or fix bugs in the app.
This becomes especially relevant for payload data, where there is a LOT of data (and we might be able to speed this up per detector by creating detector specific ROA files (#231)).

@fhagemann

fhagemann commented Oct 11, 2026 •

Copy link
Copy Markdown

A lot of the plots and the analysis are hard-coded to the Am-241 energy range, which makes the fast threshold extraction from the FM calibration Cs-137 partyhat measurement challenging. It would be nice to have the option to set some of the constants (e.g. constexpr int FAST_SEARCH_RANGE_ADC = 2000;) and some of the member class variables (like double m_NoiseStartMinCounts = 8.0;), (or preferably all?) through the config file or terminal commands.

@fhagemann

Copy link
Copy Markdown

This PR has merge conflicts. Can you resolve them?
Also, it looks like the #217 commit was cherry-picked into this PR, even though it already lives on develop/em (?)

@fhagemann

Copy link
Copy Markdown

And would this be the threshold app in its final form (ready for final review), or are you still working on something?

@fhagemann
fhagemann force-pushed the feature/strip-threshold-finder branch from 83e2912 to b97f78e Compare November 17, 2026 01:15
@fhagemann

Copy link
Copy Markdown

My last push was to fix the merge conflict in the strip map file.

@fhagemann
fhagemann requested a balanced review from Copilot November 17, 2026 01:16

This comment was marked as spam.

Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
Comment thread apps/StripEnergyThresholdFinder.cxx Resolved
@fhagemann

Copy link
Copy Markdown

I'm re-running everything with the latest version of this app right now, stay tuned :)
This is my current XML config file:

<?xml version="1.0" encoding="UTF-8"?>
<StripEnergyThresholdFinder>
  <Input>
    <DataFiles>
      <DataFile>/volumes/metis/COSI/SMEX/fm/config2b/raw/20261014/gse_20261014T124603.hdf5</DataFile>
      <DataFile>/volumes/metis/COSI/SMEX/fm/config2b/raw/20261014/gse_20261014T124603_1.hdf5</DataFile>
      <DataFile>/volumes/metis/COSI/SMEX/fm/config2b/raw/20261014/gse_20261014T124603_2.hdf5</DataFile>
      <DataFile>/volumes/metis/COSI/SMEX/fm/config2b/raw/20261014/gse_20261014T124603_3.hdf5</DataFile>
      <DataFile>/volumes/metis/COSI/SMEX/fm/config2b/raw/20261014/gse_20261014T124603_4.hdf5</DataFile>
      <DataFile>/volumes/metis/COSI/SMEX/fm/config2b/raw/20261014/gse_20261014T124603_5.hdf5</DataFile>
    </DataFiles>
    <CalibrationFile>20261013_FMPayload_config2b_prequick_PR235.ecal</CalibrationFile>
    <StripMap>strip-map-FM-payload.map</StripMap>
  </Input>

  <Output>
    <Prefix>/volumes/mobius/users/felix/Analysis/FM/PR166_Cs137_partyhat_20261014_124603</Prefix>
    <SaveDiagnosticPlots>true</SaveDiagnosticPlots>
  </Output>

  <Analysis>
    <ProcessAllDetectors>true</ProcessAllDetectors>
    <MinEntries>50</MinEntries>
    <FallbackThresholdKeV>-1</FallbackThresholdKeV>
    <NoiseSearchMaxKeV>50</NoiseSearchMaxKeV>
    <FastFallbackKeV>-1</FastFallbackKeV>
    <NoiseStartMinCounts>5</NoiseStartMinCounts>
    <FastSearchRangeADC>2000</FastSearchRangeADC>
  </Analysis>
</StripEnergyThresholdFinder>

@fhagemann fhagemann left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is my review after running the newest version of the app with the XML config file I posted earlier:

  • The new panel overview plots are nice, and make it way easier checking if the thresholds were determined correctly or not :)
  • This is the output when running cSlowSpectraADC_Det1_LV->Draw():
    image
    and cSlowSpectraADC_Det1_HV->Draw():
    image
    It looks like these histograms are still smoothed (which is also what is fitted in the end), but I think this is OK.
  • Could these 8x8 panels also be created for the fast threshold determination?
  • I can see you added the detector names to the titles of those plots (which would make this an FM threshold app, but show non-accurate detector names, e.g. when running this app on unit-level data or data from the EM - I would favor removing them here to keep the app as generic as possible).
  • For the guard rings: they tend to have more counts than the strips, which makes it hard finding a good set of parameters (especially NoiseStartMinCounts) that works well for both the strips and the GR contacts:
    image
    Might be worth having the option to set them separately. For now, I've just been running the threshold app twice, with different settings for NoiseStartMinCounts to extract the strip and the GR thresholds separately.
  • The summary plots for the thresholds look nice now that they are split by detector ID. However, their display arrangement differ from the arrangement that we use in the other nuclearizer expos, e.g. in #244:
    image
    Same applies to the slow hardware plots:
    image
  • The individual detector plots look nice, but I would again vote for not having the hard-coded detector names in them:
    image

Comment on lines +2017 to +2034
const map<unsigned int, pair<string, string>> DetectorInfo = {
{0, {"Q0L0", "HP52366-2"}},
{1, {"Q0L1", "HP52358-2"}},
{2, {"Q0L2", "HP52432-1"}},
{3, {"Q0L3", "M61617 (FM2)"}},
{4, {"Q1L0", "HP52542-1"}},
{5, {"Q1L1", "HP52358-3"}},
{6, {"Q1L2", "HP52434-1"}},
{7, {"Q1L3", "HP52406-1"}},
{8, {"Q2L0", "HP52542-2"}},
{9, {"Q2L1", "HP52407-3"}},
{10, {"Q2L2", "HP52379-1"}},
{11, {"Q2L3", "HP52358-1"}},
{12, {"Q3L0", "HP52434-2"}},
{13, {"Q3L1", "HP52361-1"}},
{14, {"Q3L2", "M61618 (FM-3)"}},
{15, {"Q3L3", "HP52406-3"}}
};

@fhagemann fhagemann Nov 17, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this is nice to include in the title of the plots, this is only true for the FM payload. I can envision the threshold app to also be used for other purposes (unit-level testing data, the EM model, other experiments like GRIPS) - not sure if we want this here. I'd be fine with just using the detector ID in the titles instead of all of this information.

Comment on lines +2460 to +2474
// -------------------------------------------------------------
// Canvas with LV and HV overlaid
// -------------------------------------------------------------

TCanvas* cSlowHardwareThresh =
new TCanvas(
"cSlowHardwareThresh",
"Slow Hardware Threshold per Strip",
800,
600);

cSlowHardwareThresh->cd();

gSlowHardwareThresh_LV->Draw("AP");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The slow hardware thresholds still seem to plotted into the same plot instead of split by detectors:
image

Comment on lines +268 to +276
cout << "Paneled raw ADC or Energy spectra with slow thresholds:" << endl;
cout << " One 8x8 canvas per detector and detector side (strips 0-63)" << endl;
cout << " Red solid line = software slow threshold" << endl;
cout << " Blue dashed line = hardware slow threshold" << endl;
cout << " Examples:" << endl;
cout << " cSlowSpectraADC_Det1_LV->Draw()" << endl;
cout << " cSlowSpectraADC_Det1_HV->Draw()" << endl;
cout << " cSlowSpectraEnergy_Det1_LV->Draw()" << endl;
cout << " cSlowSpectraEnergy_Det1_HV->Draw()" << endl;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be easy to also add these paneled plots for the fast threshold determination?

int m_HistogramBins = 512;
double m_HistogramMaxADC = 4096.0;
double m_NoiseSearchMaxKeV = 40.0;
double m_NoiseStartMinCounts = 50.0;

@fhagemann fhagemann Nov 17, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be nice being able to set different numbers for strips and GR contacts, as the latter tend to have way more counts (but not super urgent, as this app can just be run twice with different values for NoiseStartMinCounts)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants