Skip to content

DAOS-19396 pool: mind pool_discard() dss_rpc_send() errors (#18901) - #19254

Open
kccain wants to merge 1 commit into
release/2.6from
kccain/daos_19396_rel2p6_try2
Open

kccain wants to merge 1 commit into
release/2.6from
kccain/daos_19396_rel2p6_try2

Conversation

@kccain

@kccain kccain commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Before this change, ds_pool_update_handler() for reintegration pool_discard() swallowed a nonzero rc returned by dss_rpc_send() for the POOL_TGT_DISCARD corpc. The code execution continued forward to perform pool map update (transition reintegrating engine targets to UP state).

With this change, the code does not proceed to pool map update when the dss_rpc_send() call returns an error (e.g., -DER_TIMEDOUT).

This gives retries either by the MS host daos_engine dsc_pool_client or by the test program itself (re-issuing the dmg pool reintegrate command) a chance to wait out and possibly recover from whatever communication failures are occurring among some engines and the (recently-restarted) reintegrating engine.

Features: pool

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

Before this change, ds_pool_update_handler() for reintegration
pool_discard() swallowed a nonzero rc returned by dss_rpc_send()
for the POOL_TGT_DISCARD corpc. The code execution continued
forward to perform pool map update (transition reintegrating
engine targets to UP state).

With this change, the code does *not* proceed to pool map update
when the dss_rpc_send() call returns an error (e.g., -DER_TIMEDOUT).

This gives retries either by the MS host daos_engine dsc_pool_client
or by the test program itself (re-issuing the dmg pool reintegrate
command) a chance to wait out and possibly recover from whatever
communication failures are occurring among some engines and the
(recently-restarted) reintegrating engine.

Features: pool

Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
@github-actions

Copy link
Copy Markdown

Ticket title is 'daos_test/suite.py:DaosCoreTest.test_daos_rebuild_ec - pool reintegrate timeout'
Status is 'Awaiting backport'
Labels: '2.6.5.p2rc1,pr_test,request_for_2.6.6,request_for_2.8.2,testp1'
https://daosio.atlassian.net/browse/DAOS-19396

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/1/display/redirect

1 similar comment
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/1/display/redirect

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/1/display/redirect

1 similar comment
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/1/display/redirect

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/2/display/redirect

1 similar comment
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/2/display/redirect

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/2/display/redirect

1 similar comment
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19254/2/display/redirect

@daosbuild3

daosbuild3 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19254/3/execution/node/502/log

container/boundary.py test failures are unrelated, and an instance of issue DAOS-18610 (fixed on master and release/2.8 and decided not to be fixed on release/2.6 branch, affecting this PR/test run).

@kccain

kccain commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Build #1’s NLT instability appears unrelated to this patch: both affected commands succeeded, but Valgrind reported Go-runtime Syscall6 / read(buf) warnings. These are consistent with the intermittent syscall-suppression issue discussed in DAOS-19035 (comment 179329). Subsequent builds restarted at Test Hardware, so they did not provide an NLT retest.

@kccain kccain added clean-cherry-pick Cherry-pick from another branch that did not require additional edits forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. labels Oct 27, 2026
@kccain
kccain marked this pull request as ready for review October 27, 2026 13:46
@kccain
kccain requested review from a team as code owners October 27, 2026 13:46
@kccain
kccain requested a review from a team October 29, 2026 00:01
@kccain

kccain commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

@daos-stack/daos-gatekeeper the associated ticket has request_for_2.6.6 label, and so that needs to be granted first.

@kccain kccain added the waiting-for-merge-approval Waiting for merge approval label Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clean-cherry-pick Cherry-pick from another branch that did not require additional edits forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. waiting-for-merge-approval Waiting for merge approval

Development

Successfully merging this pull request may close these issues.

3 participants