Repository navigation
cleanup of tarball bucket to fix automated ingestion #185
Description
We recently seem to have hit some kind of limit with the bucket: the automated ingestion would only pick up the first 500 tarballs it would find in the S3 bucket, meaning that new skylake tarballs would not be found (as they are picked up alphabetically, and skylake is at the bottom of the list). We can probably work around it by doing some kind of pagination, and we were considering reimplementing this workflow anyway, but for now I've done a quick workaround by setting up a new bucket (software.eessi.io-archive), syncing all files from the existing to the archive bucket, and removing them from the existing one.
We can do that more often by running the following:
export AWS_SECRET_ACCESS_KEY=XXX AWS_ACCESS_KEY_ID=YYY
aws s3 sync s3://software.eessi.io-2023.06 s3://software.eessi.io-archive
aws s3 rm s3://software.eessi.io-2023.06 --recursive
This should only be run if there are no open PRs in the staging repo and if no tarballs were just uploaded to the bucket (i.e. no open PRs with bot: deploy label), otherwise they will be lost.
Activity
bedroge commentedon Jun 7, 2024
As the number of tarballs was getting quite large again, I've just run this cleanup operation again.
bedroge commentedon Sep 17, 2024
We were getting close to 1000 files in the bucket, so I've just moved all files to the archive bucket again.
bedroge commentedon Jan 21, 2025
I've done another cleanup today.
bedroge commentedon Mar 3, 2025
I've done another clean up today, mostly to get rid of the large number of Sapphire Rapids tarballs.
bedroge commentedon Jun 6, 2025
The automated ingestion was getting quite slow again, so I did another cleanup, also for the
riscv.eessi.io-20240402bucket:bedroge commentedon Jun 23, 2025
The ingestion now requires all tarballs to be signed. In order to get rid of any unsigned tarballs, I've done another cleanup, also for the dev repo this time:
bedroge commentedon Jul 18, 2025
And again, also for 2025.06 this time:
bedroge commentedon Aug 8, 2025
Cleaned the four buckets today to prepare for a switch to the new bundle ingestion method.
Neves-P commentedon Aug 13, 2025
Today I ran:
To clean a WIP deployment of EESSI/software-layer-scripts#59
bedroge commentedon Aug 26, 2025
Ran this again today, mostly to get rid of the large CUDA tarballs.
bedroge commentedon Sep 23, 2025
I've done this again today, the ingestion process was getting slow.
bedroge commentedon Oct 3, 2025
And again today, to get rid of the tarballs for the GPU builds that were recently ingested.
bedroge commentedon Oct 24, 2025
Just ran it again to speed up the ingestion.
bedroge commentedon Oct 30, 2025
Done again today.
bedroge commentedon Nov 18, 2025
Ran it again today.
bedroge commentedon Nov 21, 2025
Cleaned it again today.
bedroge commentedon Nov 22, 2025
And again today to get rid of some really large R-bundle-Bioconductor tarballs.
bedroge commentedon Dec 22, 2025
Again today.
Neves-P commentedon Jul 1, 2026
Today I did:
This addressed the incorrectly deployed tarballs due to EESSI/eessi-bot-software-layer#381 while working on EESSI/compatibility-layer#235 which got mixed up with tarballs from EESSI/software-layer-scripts#235.
Realizing archiving the tarballs is not necessary, I stopped after doing so for the first one.