Skip to content

DAOS-17519 test: Automate dlck testing (basic/fault_injection) - #17307

Open
rpadma2 wants to merge 93 commits into
masterfrom
rpadma2/daos_17519
Open

DAOS-17519 test: Automate dlck testing (basic/fault_injection)#17307
rpadma2 wants to merge 93 commits into
masterfrom
rpadma2/daos_17519

Conversation

@rpadma2

@rpadma2 rpadma2 commented Dec 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@rpadma2
DAOS-17519 test: Automate dlck testing (basic/fault_injection)
e44ee36 
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@github-actions

github-actions Bot commented Dec 23, 2025 •
edited
Loading

Copy link
Copy Markdown

Ticket title is 'Automation - Basic dlck test: scan the DAOS system by running the dlck tool.'
Status is 'In Progress'
Labels: 'test_2.8,testp2'
https://daosio.atlassian.net/browse/DAOS-17519

@rpadma2
DAOS-17519 test: Fix linting issues.
1dfe0ef 
Test-tag: DlckBasicTest DlckBasicFaultTest
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
janekmi
janekmi requested changes Dec 29, 2025
View reviewed changes
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic.yaml Outdated
Comment thread src/tests/ftest/recovery/dlck_basic.py Outdated
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
a376c98 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest
@daosbuild3

daosbuild3 commented Jan 8, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/3/execution/node/932/log

@daosbuild3

daosbuild3 commented Jan 8, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17307/3/testReport/

rpadma2 added 2 commits March 12, 2026 11:26
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
1f16a51
@rpadma2
DAOS-17519 test: Some of the review updates.
aa8ede9 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/4/execution/node/966/log

rpadma2 added 3 commits March 31, 2026 10:49
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
5213ba6
@rpadma2
DAOS-17519 test: Fix Linting issues.
0cea354 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Fix python/flake8 lint and copyright issues.
585528a 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/5/execution/node/988/log

rpadma2 added 2 commits May 22, 2026 13:18
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
593f473
@rpadma2
DAOS-17519 test: Update the control_meta_dir.
e4508aa 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/6/execution/node/867/log

@rpadma2
DAOS-17519 test: Fix typo on control_metadata_dir path.
10920ac 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/7/execution/node/866/log

rpadma2 added 2 commits August 12, 2026 14:18
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
2b4bc8a
@rpadma2
DAOS-17519 test: Fix some of the pylint errors.
4103769 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
rpadma2 requested a review from janekmi August 12, 2026 18:40
@daosbuild3

daosbuild3 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/8/execution/node/1073/log

@daosbuild3

daosbuild3 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/8/execution/node/1083/log

rpadma2 added 2 commits August 12, 2026 15:25
@rpadma2
DAOS-17519 test: Fix more pylint issues.
2191472 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Update test timeouts.
fb11310 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/9/execution/node/1072/log

@daosbuild3

daosbuild3 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/9/execution/node/1082/log

rpadma2 added 2 commits August 12, 2026 21:55
@rpadma2
DAOS-17519 test: add_pool needed parameter.
7284649 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Fix flake8 newline issue.
3b870dc 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/63/execution/node/1231/log

@daosbuild3

daosbuild3 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/63/execution/node/1284/log

@rpadma2
DAOS-17519 test: Archive the dlck log directories.
6c8b6a4 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
f7befc0 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest
janekmi
janekmi requested changes Sep 10, 2026
View reviewed changes
Comment thread src/tests/ftest/util/collection_utils.py
Comment thread src/tests/ftest/util/collection_utils.py Outdated
Comment thread src/tests/ftest/util/collection_utils.py Outdated
@daosbuild3

daosbuild3 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/65/execution/node/1231/log

@daosbuild3

daosbuild3 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/65/execution/node/1327/log

@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
d30d476 
Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Remove DLCK_FAULT_ENGINE_START/STOP fault flag.
6892c0a 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Minor pylint issue.
003343d 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Missed some review updates.
b9d170d 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17307/68/testReport/

@daosbuild3

daosbuild3 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/68/execution/node/1325/log

@rpadma2
DAOS-17519 test: Increasing the pool size to check whether MDonSSD wo…
d80c44b 
…rks.

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/69/execution/node/1273/log

@rpadma2
DAOS-17519 test: Add debug commands: mount, df
3aa10ef 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/70/execution/node/1230/log

janekmi
janekmi previously approved these changes Sep 18, 2026
View reviewed changes
rpadma2 added 2 commits October 1, 2026 08:45
@rpadma2
DAOS-17519 test:Merge branch 'master' into rpadma2/daos_17519
4110fc4
@rpadma2
DAOS-17519 test: Update the dlck_basic.yaml.
0c49e51 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/71/execution/node/1412/log

@rpadma2
DAOS-17519 test: Re-create tmpfs mount for MDonSSD environment.
8bcc6c2 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest
Test-repeat: 3

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/72/execution/node/1417/log

janekmi
janekmi requested changes Oct 5, 2026
View reviewed changes
Comment thread src/tests/ftest/recovery/dlck_basic.yaml Outdated
- DLCK_FAULT_CREATE_LOG_DIR
- DLCK_FAULT_ENGINE_START
- DAOS_FAULT_POOL_NVME_HEALTH
# - DAOS_FAULT_POOL_OPEN_BIO

@janekmi janekmi Oct 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, this particular fault works perfectly fine in the MD-on-SSD mode. Can we run it in this mode only?

I would like to keep it. Unless you plan to do it in the next stage. I am fine with it as long as there is a ticket for it.

@rpadma2 rpadma2 Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I noticed - DAOS_FAULT_POOL_OPEN_BIO was not working under PMEM environment [ https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/view/change-requests/job/PR-17307/68/artifact/Functional%20Hardware%20Medium/recovery/dlck_basic.py/job.log ]. If it works on MDonSSD, I will check how to validate it only under MDonSSD environment.

Comment thread src/tests/ftest/recovery/dlck_basic.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic.py
rpadma2 added 2 commits October 7, 2026 09:16
@rpadma2
DAOS-17519 test: Update the storage mount to dlck_test.
3badfd2 
Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
4027995 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest
Test-repeat: 2
janekmi
janekmi requested changes Oct 8, 2026
View reviewed changes
Comment thread src/tests/ftest/recovery/dlck_basic.py Outdated
Comment on lines +108 to +109
self.log_step("Perform dmg system stop to run dlck command")
dmg.system_stop()

@janekmi janekmi Oct 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending on the server configuration you may run out of RAM when you mount your tmpfs before you call dmg.system_stop(). It may be not the case right now but the risk is there. My recommended order of operations is:

  1. dmg.system_stop().
  2. mount -t tmpfs ...

@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
b3e195e
@rpadma2
DAOS-17519 test: Enable DAOS_FAULT_POOL_OPEN_BIO for MDonSSD.
6534a44 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest
Test-repeat: 2

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@phender phender phender requested changes
@janekmi janekmi janekmi requested changes
@shimizukko shimizukko shimizukko left review comments
@jamesanunez jamesanunez Awaiting requested review from jamesanunez
@daltonbohning daltonbohning Awaiting requested review from daltonbohning

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@rpadma2 @daosbuild3 @janekmi @shimizukko @daltonbohning @phender