Skip to content

DAOS-19396 pool: mind pool_discard() dss_rpc_send() errors - #18894

Draft
kccain wants to merge 4 commits into
release/2.6from
kccain/daos_19396_rel2p6
Draft

DAOS-19396 pool: mind pool_discard() dss_rpc_send() errors#18894
kccain wants to merge 4 commits into
release/2.6from
kccain/daos_19396_rel2p6

Conversation

@kccain

@kccain kccain commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Experimental product code change on release/2.6 branch to not proceed to pool map update step of ds_pool_update_handler() for reintegration operations when the previous pool_discard() step got an error in dss_rpc_send() (e.g., a -DER_TIMEDOUT failure).

This gives retries either by the MS host daos_engine dsc_pool_client or by the test program itself (re-issuing the dmg pool reintegrate command) a chance to wait out and possibly recover from whatever communication failures are occurring among some engines and the reintegrating engine.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@kccain
DAOS-19396 pool: mind pool_discard() dss_rpc_send() errors
b7fa5b5 
Experimental product code change on release/2.6 branch to
*not* proceed to pool map update step of ds_pool_update_handler()
for reintegration operations when the previous pool_discard()
step got an error in dss_rpc_send() (e.g., a -DER_TIMEDOUT failure).

This gives retries either by the MS host daos_engine dsc_pool_client
or by the test program itself (re-issuing the dmg pool reintegrate
command) a chance to wait out and possibly recover from whatever
communication failures are occurring among some engines and the
reintegrating engine.

Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18894/1/execution/node/308/log

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18894/1/execution/node/394/log

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18894/1/execution/node/429/log

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18894/1/display/redirect

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18894/1/display/redirect

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18894/1/display/redirect

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18894/1/display/redirect

@daosbuild3

daosbuild3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18894/1/display/redirect

@github-actions

github-actions Bot commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Ticket title is 'daos_test/suite.py:DaosCoreTest.test_daos_rebuild_ec - pool reintegrate timeout'
Status is 'Awaiting backport'
Labels: '2.6.5.p2rc1,pr_test,request_for_2.8.2,testp1'
https://daosio.atlassian.net/browse/DAOS-19396

@kccain
add a rebuild_ec subtest 9 only test
27052dd 
for CI repeat testing with/without a fix patch

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_daos_rebuild_ec_subtest9
Test-repeat: 10

Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
@daosbuild3

daosbuild3 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18894/2/execution/node/476/log

@daosbuild3

daosbuild3 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18894/2/execution/node/361/log

@daosbuild3

daosbuild3 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Test stage Test RPMs on Leap 15.4 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18894/2/execution/node/770/log

@daosbuild3

daosbuild3 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18894/2/display/redirect

@kccain kccain added the clean-cherry-pick Cherry-pick from another branch that did not require additional edits label Sep 29, 2026
@kccain
kccain requested review from liuxuezhao and wangshilong and removed request for liuxuezhao and wangshilong September 29, 2026 12:03
@daosbuild3

daosbuild3 commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18894/5/testReport/

@daosbuild3

daosbuild3 commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18894/6/testReport/

@wangshilong

wangshilong commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Since this was reverted test change, it should not be a clean-cherry-pick i supposed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wangshilong wangshilong wangshilong approved these changes

Assignees

No one assigned

Labels

clean-cherry-pick Cherry-pick from another branch that did not require additional edits

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kccain @daosbuild3 @wangshilong