Repository navigation
Supera run-time is excessively long for MC overlay productions #656
Description
Activity
Francois traced the issue back to a problem in LArCV2. The issue was not simulation vs. data, but rather in the mixing of space point sources inherent in an overlay MC dataset. The code within LArCV2 that handles the mixed indexing required by overlays was highly inefficient, and thus very slow for overlay datasets. This explains why it was not a problem for pure simulation (old ICARUS and current SBND) or real data.
The fix is implemented in LArCV2 2.4.1. A sub-leading optimization is present also in Supera v1.0.0+. To incorporate this fix into sbncode (and therefore into sbndcode or icaruscode), it is necessary to update the LArCV2 dependence in sbncode and point the submodule dependence to Supera v1.0.0+.
The ICARUS Overlay MC production has highlighted the fact that Supera is taking significantly longer than all other modules in stage1. Moreover, the time distribution has a long tail extending from the central Gaussian core. Processing the full dataset requires a problematic amount of resources due to this long tail.
Issue must be debugged by instrumenting Supera module calls and looking for hot spots.