Repository navigation
DAOS-19591 container: recover EC boundaries after reintegration - #19321
Conversation
|
Ticket title is 'Aurora daos_user: engine printing continuously in log "cont_agg_eph_sync() 66e9ae82/2416cc43: Sluggish EC boundary report from rank 117, 165933 Seconds." ' |
Reintegrated or extended targets can miss the last EC aggregation boundary and wait indefinitely for a refresh. Their zero epochs may be suppressed by the rank's previous report, or they may have no local VOS container and therefore never report. Refreshing only when the minimum epoch changes can leave the container's boundary and space reclamation stuck. Track such waiting ranks with ea_need_refresh, set when a rank's zero report is suppressed by its earlier report or when some non-failed rank has reported nothing yet. While it is set, and rebuild is inactive and reclaim is enabled, resend the unchanged boundary every 30 seconds; the flag is cleared before each push so steady-state traffic drops back to zero. Allow leader pushes to create missing VOS containers, using deep-stack thread collectives and tolerating concurrent creation. Mark pushes using the otherwise unused boundary payload rank; clear that marker on fetch copies so cached boundaries cannot recreate containers. The IV wire layout remains unchanged. When destroying a container, wait up to 10 seconds for an in-flight boundary push and return DER_BUSY if it has not finished, rather than proceeding after a timeout. Stop tracking before the destroy broadcast, including orphan cleanup, and check retired tracking entries as well. Skip refreshes after RDB errors or deletion during an RDB operation, and retire entries for nonexistent containers. Also suppress sluggish-boundary warnings for epoch jumps larger than the current leader's tracking age. Features: container rebuild Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
6e3f708 to
c00065d
Compare
|
Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19321/2/testReport/ |
|
@mchaarawi the test hit a NLT failure, I never know how to find its log or how to debug that kind of failure. artifact seems cannot find where the log is. |
NLT on 2.6 has been failing consistently with those valgrind errors. unfortunately we stopped updating NLT and supressions for 2.6. you can just ignore those failures and it should not block the PR |
|
Imagine an engine evicted a SSD, which was originally alive, then the reporting ULT should have reported a valid epoch boundary of containers. Now the SSD is reintegrated again, the epoch boundary of those containers shards will stay zero, ds_cont_tgt_ec_eph_query_ult() can't refresh and report value anymore? I think a change like this can avoid this. |
…tion When a target is excluded and reintegrated on a running engine (for example an evicted SSD), its container shards start with no EC aggregation boundary and wait for the leader to resend it. The rank's reporting ULT however skipped a zero minimum once it had reported a non-zero epoch before, so the leader never learned that a shard was waiting, kept the rank's stale epoch and pinned the pool-wide boundary. Report a zero minimum even after earlier non-zero reports. The leader ignores the value itself but marks the container for a boundary resend, which validates the boundary on the reintegrated target and lets it aggregate and report again. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Thanks, that is a valid concern. I pushed the 2nd commit to fix that case, same as what you suggested. |
|
Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19321/3/testReport/ |
OK. Now all tests complete and only failure is the NLT case, so seems we are fine. thx |
…) (#19350) Reintegrated or extended targets can miss the last EC aggregation boundary and wait indefinitely for a refresh. Their zero epochs may be suppressed by the rank's previous report, or they may have no local VOS container and therefore never report. Refreshing only when the minimum epoch changes can leave the container's boundary and space reclamation stuck. Track such waiting ranks with ea_need_refresh, set when a rank's zero report is suppressed by its earlier report or when some non-failed rank has reported nothing yet. While it is set, and rebuild is inactive and reclaim is enabled, resend the unchanged boundary every 30 seconds; the flag is cleared before each push so steady-state traffic drops back to zero. Allow leader pushes to create missing VOS containers, using deep-stack thread collectives and tolerating concurrent creation. Mark pushes using the otherwise unused boundary payload rank; clear that marker on fetch copies so cached boundaries cannot recreate containers. The IV wire layout remains unchanged. When destroying a container, wait up to 10 seconds for an in-flight boundary push and return DER_BUSY if it has not finished, rather than proceeding after a timeout. Stop tracking before the destroy broadcast, including orphan cleanup, and check retired tracking entries as well. Skip refreshes after RDB errors or deletion during an RDB operation, and retire entries for nonexistent containers. Also suppress sluggish-boundary warnings for epoch jumps larger than the current leader's tracking age. When a target is excluded and reintegrated on a running engine (for example an evicted SSD), its container shards start with no EC aggregation boundary and wait for the leader to resend it. The rank's reporting ULT however skipped a zero minimum once it had reported a non-zero epoch before, so the leader never learned that a shard was waiting, kept the rank's stale epoch and pinned the pool-wide boundary. Report a zero minimum even after earlier non-zero reports. The leader ignores the value itself but marks the container for a boundary resend, which validates the boundary on the reintegrated target and lets it aggregate and report again. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Liu Xuezhao <xuezhao.liu@intel.com>
Reintegrated or extended targets can miss the last EC aggregation boundary and wait indefinitely for a refresh. Their zero epochs may be suppressed by the rank's previous report, or they may have no local VOS container and therefore never report. Refreshing only when the minimum epoch changes can leave the container's boundary and space reclamation stuck.
Track such waiting ranks with ea_need_refresh, set when a rank's zero report is suppressed by its earlier report or when some non-failed rank has reported nothing yet. While it is set, and rebuild is inactive and reclaim is enabled, resend the unchanged boundary every 30 seconds; the flag is cleared before each push so steady-state traffic drops back to zero. Allow leader pushes to create missing VOS containers, using deep-stack thread collectives and tolerating concurrent creation. Mark pushes using the otherwise unused boundary payload rank; clear that marker on fetch copies so cached boundaries cannot recreate containers. The IV wire layout remains unchanged.
When destroying a container, wait up to 10 seconds for an in-flight boundary push and return DER_BUSY if it has not finished, rather than proceeding after a timeout. Stop tracking before the destroy broadcast, including orphan cleanup, and check retired tracking entries as well. Skip refreshes after RDB errors or deletion during an RDB operation, and retire entries for nonexistent containers.
Also suppress sluggish-boundary warnings for epoch jumps larger than the current leader's tracking age.
Features: container rebuild
Steps for the author:
After all prior steps are complete: