From a26cb76610927595510ce4c1f7e07a36d9854725 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Thu, 17 Sep 2026 18:53:51 -0500 Subject: [PATCH 1/7] Added a PowerShell version of the WarmPerformnaceTier script. --- .../warm_performance_tier/README.md | 38 ++- .../WarmPerformanceTier.ps1 | 233 ++++++++++++++++++ 2 files changed, 264 insertions(+), 7 deletions(-) create mode 100755 Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index 2605ea20..4d8fcbbe 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -52,19 +52,24 @@ script twice does, typically, get more data into the performance tier so if you are trying to get as much data as possible into the performance tier, it is recommended to run the script twice. +There are two versions of the script. One that designed to run on a Linux operating system name `warm_performance_tier` +and one that is meant to run on a Windows based one name `WarmPerformanceTier.ps1`. + ## Set Up The first step is to ensure the volume's tiering policy is set to something other than "all" or "snapshot-only". You should also ensure that the cloud-retrieval-policy is set to "on-read". You can make both of these changes with the following commands: ``` -set advanced -c off +set advanced volume modify -vserver -volume -tiering-policy auto -cloud-retrieval-policy on-read ``` Where `` is the name of the SVM and `` is the name of the volume. -The next step is to copy the script to a Linux based host that is able to NFS -mount the volume to be warmed. If the volume is already mounted, then +The next step is to copy the appropriate script, either the Linux based one or the Windows +based one, to the system that has access to the volume to be warmed. + +On a Linux based system, if the volume is already mounted, then any user that has read access to all the files in the volume can run it. Otherwise, the script needs to be run as 'root' so it can mount the volume before reading the files. @@ -73,7 +78,9 @@ If the 'root' user can't read the all files in the volume, then you should use t to mount the volume and then run the script from a user ID that can read the contents of all the files in the volume. -# Running The Script +## Running The Script + +### Runing the script on a Linux based operating system There are two main ways to run the script. The first is to just provide the script with a directory to start from using the -d option. The script will then read every file in that directory and all its subdirectories. The second way @@ -115,7 +122,7 @@ Where: -d directory - Is the root directory to start the process from. -t max_directory_threads - Is the maximum number of threads to use to process directories. The default is 2. -x max_read_threads - Is the maximum number of threads to use to read files. The default is 5. - -V - Enable verbose output. Displays the thread ID, date (in epoch seconds), then the directory or file being processed. + -V - Enable verbose output. Displays progress information on the files being read. -h - Prints this help information. Notes: @@ -126,11 +133,28 @@ Notes: reading files. ``` +### Runing the script on a Windows based operating system + +To run on a Windows based operating system, you will need to run the script from a +PowerShell prompt. You can run the script with the following command: +``` +.\WarmPerformanceTier.ps1 -Path \\fsx-server\myshare +``` +Where `\\fsx-server\myshare` is the UNC path or mapped drive letter to the share/volume root. + +The script supports the following options: +o Path - UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare +o ThreadCount - Number of files to read in parallel. Default 8. +o BlockSizeMB - Block size in MB used for the out-of-order reads. Default 2 +o ReadMethod - 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). Default is 'Reverse'. +o BatchSize - How many files to queue at once before waiting for that batch to finish. + Bounds memory usage on volumes with very large file counts. Default is ThreadCount * 4. + ## Finishing Step After running the script, you should set the cloud-retrieval-policy back to "default" by running the following commands: ``` -set advanced -c off +set advanced volume modify -vserver -volume -cloud-retrieval-policy default ``` Where `` is the name of the SVM and `` is the name of the volume. @@ -149,4 +173,4 @@ Unless required by applicable law or agreed to in writing, software distributed See the License for the specific language governing permissions and limitations under the License. -© 2024 NetApp, Inc. All Rights Reserved. +© 2026 NetApp, Inc. All Rights Reserved. diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 new file mode 100755 index 00000000..ca02e494 --- /dev/null +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -0,0 +1,233 @@ +<# +.SYNOPSIS + Reads every file in a CIFS/SMB share to force promotion of capacity-tier + (cold/tiered) data back to the performance tier on an FSx for ONTAP volume. + +.DESCRIPTION + ONTAP's tiering scanner intentionally avoids promoting data when it detects + a sequential, backup-like read pattern (to avoid backup/AV jobs needlessly + re-filling the performance tier). This script defeats that detection by + reading each file's blocks in reverse order (or random order) instead of + front-to-back, so the read pattern looks like real client access rather + than a scan. + + This is a native Windows/PowerShell port of NetApp's warm_performance_tier + bash script (NFS-only) for environments with no Linux available. + +.PARAMETER Path + UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare + +.PARAMETER ThreadCount + Number of files to read in parallel. Default 8. + +.PARAMETER BlockSizeMB + Block size in MB used for the out-of-order reads. Default 2 (matches the + original script's default of 2MB). + +.PARAMETER ReadMethod + 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). + Default 'Reverse'. + +.PARAMETER BatchSize + How many files to queue at once before waiting for that batch to finish. + Bounds memory usage on volumes with very large file counts. Default is + ThreadCount * 4. + +.PARAMETER Help + Print invocation usage and exit. + +.NOTES + Promotion observed on FSx for ONTAP has been gradual rather than + immediate - a single pass over a volume may only promote a small + fraction of the read data back to the performance tier, with the rest + following over subsequent passes. Expect to run this multiple times + (or in a loop until volume footprint stabilizes) rather than a single + one-shot sweep for full promotion. + +.EXAMPLE + .\WarmPerformanceTier.ps1 -Path \\fsx-server\myshare -ThreadCount 12 + +.EXAMPLE + .\WarmPerformanceTier.ps1 -Path Z:\ -ReadMethod Random -BlockSizeMB 4 + +.EXAMPLE + .\WarmPerformanceTier.ps1 -Help +#> + +[CmdletBinding(DefaultParameterSetName = 'Run')] +param( + [Parameter(Mandatory = $true, ParameterSetName = 'Run', Position = 0)] + [string]$Path, + + [Parameter(ParameterSetName = 'Run')] + [int]$ThreadCount = 8, + + [Parameter(ParameterSetName = 'Run')] + [int]$BlockSizeMB = 2, + + [Parameter(ParameterSetName = 'Run')] + [ValidateSet('Reverse', 'Random')] + [string]$ReadMethod = 'Reverse', + + [Parameter(ParameterSetName = 'Run')] + [int]$BatchSize = 0, + + [Parameter(ParameterSetName = 'Run')] + [switch]$VerboseProgress, + + [Parameter(ParameterSetName = 'Help')] + [switch]$Help +) + +$ErrorActionPreference = 'Stop' + +if ($Help) { + Write-Host @" +Usage: .\WarmPerformanceTier.ps1 -Path [-ThreadCount n] [-BlockSizeMB n] [-ReadMethod Reverse|Random] [-BatchSize n] [-VerboseProgress] [-Help] + +Where: + -Path UNC path or mapped drive to the share/volume root (required). + Example: \\fsx-server\myshare or Z:\ + -ThreadCount Number of files to read in parallel. Default: 8 + -BlockSizeMB Block size in MB for out-of-order reads. Default: 2 + -ReadMethod Reverse (back-to-front) or Random. Default: Reverse + -BatchSize Files to queue before waiting. Default: ThreadCount * 4 + -VerboseProgress Print extra progress lines in addition to the progress bar + -Help Print this help and exit + +Examples: + .\WarmPerformanceTier.ps1 -Path \\fsx-server\myshare -ThreadCount 12 + .\WarmPerformanceTier.ps1 -Path Z:\ -ReadMethod Random -BlockSizeMB 4 +"@ + return +} + +if ($BatchSize -le 0) { + $BatchSize = $ThreadCount * 4 +} + +if (-not (Test-Path -LiteralPath $Path)) { + throw "Path '$Path' does not exist or is not accessible." +} + +Write-Host "Enumerating files under '$Path' ..." +$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue +$total = $files.Count + +if ($total -eq 0) { + Write-Host "No files found under '$Path'. Nothing to do." + return +} + +Write-Host "Found $total files. Reading with $ThreadCount parallel workers (method: $ReadMethod, block size: ${BlockSizeMB}MB, batch size: $BatchSize)..." + +$blockSize = [int64]$BlockSizeMB * 1MB + +# Script block executed inside each runspace. Reads the file's blocks out of +# sequential order and discards the contents - the goal is purely to trigger +# ONTAP's read-driven promotion, not to use the data. +$readScriptBlock = { + param($filePath, $blockSize, $readMethod) + + try { + $fs = [System.IO.File]::Open( + $filePath, + [System.IO.FileMode]::Open, + [System.IO.FileAccess]::Read, + [System.IO.FileShare]::ReadWrite + ) + try { + $length = $fs.Length + if ($length -eq 0) { + return [PSCustomObject]@{ Path = $filePath; Success = $true; Bytes = 0 } + } + + $buffer = New-Object byte[] $blockSize + $totalBlocks = [math]::Ceiling($length / $blockSize) + + if ($readMethod -eq 'Reverse') { + $blockOrder = ($totalBlocks - 1)..0 + } + else { + $blockOrder = 0..($totalBlocks - 1) | Get-Random -Count $totalBlocks + } + + foreach ($i in $blockOrder) { + $offset = [int64]$i * $blockSize + $toRead = [Math]::Min($blockSize, $length - $offset) + if ($toRead -le 0) { continue } + [void]$fs.Seek($offset, [System.IO.SeekOrigin]::Begin) + [void]$fs.Read($buffer, 0, [int]$toRead) + } + + return [PSCustomObject]@{ Path = $filePath; Success = $true; Bytes = $length } + } + finally { + $fs.Dispose() + } + } + catch { + return [PSCustomObject]@{ Path = $filePath; Success = $false; Error = $_.Exception.Message } + } +} + +$sessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault() +$pool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool(1, $ThreadCount, $sessionState, $Host) +$pool.Open() + +$startTime = Get-Date +$doneCount = 0 +$errorCount = 0 +$errorList = New-Object System.Collections.Generic.List[string] + +function Wait-AndCollect($jobs) { + foreach ($job in $jobs) { + $result = $job.Pipeline.EndInvoke($job.Handle) + $job.Pipeline.Dispose() + $script:doneCount++ + if ($result.Success -eq $false) { + $script:errorCount++ + $script:errorList.Add("$($result.Path): $($result.Error)") + } + } +} + +$batch = New-Object System.Collections.Generic.List[object] + +foreach ($file in $files) { + $ps = [System.Management.Automation.PowerShell]::Create() + $ps.RunspacePool = $pool + [void]$ps.AddScript($readScriptBlock).AddArgument($file.FullName).AddArgument($blockSize).AddArgument($ReadMethod) + $handle = $ps.BeginInvoke() + $batch.Add([PSCustomObject]@{ Pipeline = $ps; Handle = $handle; File = $file.FullName }) + + if ($batch.Count -ge $BatchSize) { + Wait-AndCollect $batch + $batch.Clear() + + $elapsed = (Get-Date) - $startTime + $pct = [math]::Round(($doneCount / $total) * 100, 1) + Write-Progress -Activity "Warming performance tier" -Status "$doneCount / $total files ($errorCount errors)" -PercentComplete $pct + if ($VerboseProgress) { + Write-Host ("[{0:hh\:mm\:ss}] {1}/{2} files done ({3}%), {4} errors" -f $elapsed, $doneCount, $total, $pct, $errorCount) + } + } +} + +# Drain the final partial batch. +if ($batch.Count -gt 0) { + Wait-AndCollect $batch +} + +$pool.Close() +$pool.Dispose() + +$elapsed = (Get-Date) - $startTime +Write-Progress -Activity "Warming performance tier" -Completed + +Write-Host "" +Write-Host "Done in $($elapsed.ToString('hh\:mm\:ss')). Processed $doneCount files, $errorCount errors." +if ($errorCount -gt 0) { + Write-Host "Errors:" + $errorList | ForEach-Object { Write-Host " $_" } +} From 12f9814d6f45b6cf092bce3015a652a8dee2d879 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Thu, 17 Sep 2026 19:15:26 -0500 Subject: [PATCH 2/7] Added a PowerShell version of the WarmPerformnaceTier script. --- .../list_fsxn_filesystems | 17 ++++++++++++----- .../fsx-ontap-aws-cli-scripts/list_fsxn_svms | 16 +++++++++++++--- .../fsx-ontap-aws-cli-scripts/list_fsxn_volumes | 17 +++++++++++++---- .../warm_performance_tier/README.md | 11 ++++++----- 4 files changed, 44 insertions(+), 17 deletions(-) diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems index 49bb98e7..c5d9077b 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems @@ -44,7 +44,7 @@ usage () { cat 1>&2 < /dev/null; then + i=1 + while [ $i -lt $((maxTime * 2)) ]; do + if ! ps -p $pid > /dev/null 2>&1; then + return 0 + fi + sleep .5 + let i+=1 + done + + if ps -p $pid > /dev/null 2>&1; then printf "\nWarning, timed out while getting the list of FSxNs from the $region region.\nConsider excluding it with the -X option or raising the timeout with the -t option.\n" 1>&2 pkill -P $pid kill $pid @@ -265,7 +273,7 @@ maxTime=20 # Process command line arguments. while getopts "bhcxsaX:nr:i:f:e:t:" option; do case "$option" in - r) region="$OPTARG" + r) region="$(echo $OPTARG | sed -e 's/,/ /g')" ;; a) allRegions=true ;; @@ -322,7 +330,6 @@ if [ "$allRegions" == "true" ]; then # support FSxN allEndabledRegions=$(aws ec2 describe-regions --query "Regions[].RegionName" --output=json | jq -r '.[]') allFsxnRegions=$(curl -s https://api.regional-table.region-services.aws.a2z.com/index.json | jq -r '.prices[] | select(.attributes."aws:serviceName" == "Amazon FSx for NetApp ONTAP") .attributes."aws:region"' | egrep -v "$excludeRegions") -# allFsxnRegions=$(curl -s https://api.regional-table.region-services.aws.a2z.com/index.json | jq -r '.prices[] | select(.attributes."aws:serviceName" == "Amazon FSx for NetApp ONTAP") .attributes."aws:region"') for reg in $allEndabledRegions; do for fsxnReg in $allFsxnRegions; do if [ $reg == $fsxnReg ]; then diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms index fc8d5104..ae25d299 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms @@ -30,6 +30,7 @@ usage () { cat 1>&2 < /dev/null; then + + i=1 + while [ $i -lt $((maxTime*2)) ]; do + if ! ps -p $pid > /dev/null 2>&1; then + return + fi + sleep .5 + let i+=1 + done + + if ps -p $pid > /dev/null 2>&1; then printf "\nWarning, timed out while getting the list of SVMs from the $region region.\nConsider excluding it with the -X option or raising the timeout with the -t option.\n" 1>&2 pkill -P $pid kill $pid @@ -129,7 +139,7 @@ excludeRegions="none" maxTime=20 while getopts "haX:nr:i:f:pe:t:" option; do case "$option" in - r) region="$OPTARG" + r) region="$(echo "$OPTARG" | sed -e 's/,/ /g')" ;; a) allRegions=true ;; diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes index 4d61d667..840487a0 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes @@ -36,7 +36,7 @@ usage () { Usage $(basename $0) [-r region] [-a] [-X excludeRegions] [-o] [-f fileSystemName] [-i fileSystemId] [-n] [-s svmID] [-x] [-t seconds] [-e endpoint_url] [volume_pattern] Where: volume_pattern - Only volumes that match the egrep pattern will be included. An assumed '^' and '$' is added to the beginning and end of the pattern respectively. - -r region allows you to specify the region you want the list from. + -r means to only include the specified region(s) (comma seperated list). -a means all regions. -X excludeRegions - Comma separated list of regions to exclude from the search. Only valid when -a is specified. -n means include the file system and SVM names. Not compatible with the -x option. @@ -61,8 +61,17 @@ start_watchdog () { pid=$1 maxTime=$2 region=$3 - sleep $maxTime - if ps -p $pid > /dev/null; then + + i=1 + while [ $i -lt $((maxTime * 2)) ]; do + if ! ps -p $pid > /dev/null 2>&1; then + return + fi + sleep .5 + let i+=1 + done + + if ps -p $pid > /dev/null 2>&1; then printf "\nWarning, timed out while getting the list of SVMs from the $region region.\nConsider excluding it with the -X option or raising the timeout with the -t option.\n" 1>&2 pkill -P $pid kill $pid @@ -214,7 +223,7 @@ volumePattern="*" maxTime=30 while getopts "hr:aX:f:i:nos:xe:t:" option; do case "$option" in - r) region="$OPTARG" + r) region="$(echo $OPTARG | sed -e 's/,/ /g')" ;; a) allRegions=true ;; diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index 4d8fcbbe..2f9eec9c 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -143,11 +143,12 @@ PowerShell prompt. You can run the script with the following command: Where `\\fsx-server\myshare` is the UNC path or mapped drive letter to the share/volume root. The script supports the following options: -o Path - UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare -o ThreadCount - Number of files to read in parallel. Default 8. -o BlockSizeMB - Block size in MB used for the out-of-order reads. Default 2 -o ReadMethod - 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). Default is 'Reverse'. -o BatchSize - How many files to queue at once before waiting for that batch to finish. + +* Path - UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare +* ThreadCount - Number of files to read in parallel. Default 8. +* BlockSizeMB - Block size in MB used for the out-of-order reads. Default 2 +* ReadMethod - 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). Default is 'Reverse'. +* BatchSize - How many files to queue at once before waiting for that batch to finish. Bounds memory usage on volumes with very large file counts. Default is ThreadCount * 4. ## Finishing Step From 6bef5d1f2c16ab65f53ed9d7246d93ad30b8080d Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 08:33:27 -0500 Subject: [PATCH 3/7] Ensure valid parameters were provided. --- .../WarmPerformanceTier.ps1 | 20 ++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 index ca02e494..3c658042 100755 --- a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -60,9 +60,11 @@ param( [string]$Path, [Parameter(ParameterSetName = 'Run')] + [ValidateRange(1, 1024)] [int]$ThreadCount = 8, [Parameter(ParameterSetName = 'Run')] + [ValidateScript({ $_ -gt 0 -and ([int64]$_ * 1MB) -le [int]::MaxValue })] [int]$BlockSizeMB = 2, [Parameter(ParameterSetName = 'Run')] @@ -111,9 +113,14 @@ if (-not (Test-Path -LiteralPath $Path)) { } Write-Host "Enumerating files under '$Path' ..." -$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue +$enumerationErrors = @() +$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue -ErrorVariable $enumerationErrors $total = $files.Count +if ($enumerationErrors.Count -gt 0) { + Write-Warning "Encountered $($enumerationErrors.Count) errors while enumerating files. Some files may be skipped." +} + if ($total -eq 0) { Write-Host "No files found under '$Path'. Nothing to do." return @@ -157,9 +164,15 @@ $readScriptBlock = { $toRead = [Math]::Min($blockSize, $length - $offset) if ($toRead -le 0) { continue } [void]$fs.Seek($offset, [System.IO.SeekOrigin]::Begin) - [void]$fs.Read($buffer, 0, [int]$toRead) + $remaining = [int]$toRead + while ($remaining -gt 0) { + $bytesRead = $fs.Read($buffer, 0, $remaining) + if ($bytesRead -eq 0) { + throw "Unexpected end of file while reading '$filePath'." + } + $remaining -= $bytesRead + } } - return [PSCustomObject]@{ Path = $filePath; Success = $true; Bytes = $length } } finally { @@ -230,4 +243,5 @@ Write-Host "Done in $($elapsed.ToString('hh\:mm\:ss')). Processed $doneCount fil if ($errorCount -gt 0) { Write-Host "Errors:" $errorList | ForEach-Object { Write-Host " $_" } + exit 1 } From 587cfb896c90bf8fda07a3d60ec36848b9afb36c Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 12:43:46 -0500 Subject: [PATCH 4/7] Change it so it would stream the file objects to loop that processes the files so that it could scale to volumes with a high number of files. --- .../WarmPerformanceTier.ps1 | 68 ++++++++++--------- 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 index 3c658042..6cef5f5d 100755 --- a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -112,21 +112,7 @@ if (-not (Test-Path -LiteralPath $Path)) { throw "Path '$Path' does not exist or is not accessible." } -Write-Host "Enumerating files under '$Path' ..." -$enumerationErrors = @() -$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue -ErrorVariable $enumerationErrors -$total = $files.Count - -if ($enumerationErrors.Count -gt 0) { - Write-Warning "Encountered $($enumerationErrors.Count) errors while enumerating files. Some files may be skipped." -} - -if ($total -eq 0) { - Write-Host "No files found under '$Path'. Nothing to do." - return -} - -Write-Host "Found $total files. Reading with $ThreadCount parallel workers (method: $ReadMethod, block size: ${BlockSizeMB}MB, batch size: $BatchSize)..." +Write-Host "Streaming files under '$Path' and reading with $ThreadCount parallel workers (method: $ReadMethod, block size: ${BlockSizeMB}MB, batch size: $BatchSize)..." $blockSize = [int64]$BlockSizeMB * 1MB @@ -206,26 +192,35 @@ function Wait-AndCollect($jobs) { } $batch = New-Object System.Collections.Generic.List[object] +$queuedCount = 0 +$enumerationErrors = @() -foreach ($file in $files) { - $ps = [System.Management.Automation.PowerShell]::Create() - $ps.RunspacePool = $pool - [void]$ps.AddScript($readScriptBlock).AddArgument($file.FullName).AddArgument($blockSize).AddArgument($ReadMethod) - $handle = $ps.BeginInvoke() - $batch.Add([PSCustomObject]@{ Pipeline = $ps; Handle = $handle; File = $file.FullName }) - - if ($batch.Count -ge $BatchSize) { - Wait-AndCollect $batch - $batch.Clear() - - $elapsed = (Get-Date) - $startTime - $pct = [math]::Round(($doneCount / $total) * 100, 1) - Write-Progress -Activity "Warming performance tier" -Status "$doneCount / $total files ($errorCount errors)" -PercentComplete $pct - if ($VerboseProgress) { - Write-Host ("[{0:hh\:mm\:ss}] {1}/{2} files done ({3}%), {4} errors" -f $elapsed, $doneCount, $total, $pct, $errorCount) +# Get-ChildItem streams FileInfo objects one at a time as it walks the tree, +# and piping straight into ForEach-Object dispatches each file to the worker +# queue as soon as it is discovered - the full listing is never held in +# memory, so this scales to directories with very large file counts. +Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue -ErrorVariable enumerationErrors | + ForEach-Object { + $file = $_ + $queuedCount++ + + $ps = [System.Management.Automation.PowerShell]::Create() + $ps.RunspacePool = $pool + [void]$ps.AddScript($readScriptBlock).AddArgument($file.FullName).AddArgument($blockSize).AddArgument($ReadMethod) + $handle = $ps.BeginInvoke() + $batch.Add([PSCustomObject]@{ Pipeline = $ps; Handle = $handle; File = $file.FullName }) + + if ($batch.Count -ge $BatchSize) { + Wait-AndCollect $batch + $batch.Clear() + + $elapsed = (Get-Date) - $startTime + Write-Progress -Activity "Warming performance tier" -Status "$doneCount processed, $queuedCount queued so far ($errorCount errors)" + if ($VerboseProgress) { + Write-Host ("[{0:hh\:mm\:ss}] {1} processed, {2} queued so far, {3} errors" -f $elapsed, $doneCount, $queuedCount, $errorCount) + } } } -} # Drain the final partial batch. if ($batch.Count -gt 0) { @@ -238,6 +233,15 @@ $pool.Dispose() $elapsed = (Get-Date) - $startTime Write-Progress -Activity "Warming performance tier" -Completed +if ($enumerationErrors.Count -gt 0) { + Write-Warning "Encountered $($enumerationErrors.Count) errors while enumerating files. Some files may be skipped." +} + +if ($queuedCount -eq 0) { + Write-Host "No files found under '$Path'. Nothing to do." + return +} + Write-Host "" Write-Host "Done in $($elapsed.ToString('hh\:mm\:ss')). Processed $doneCount files, $errorCount errors." if ($errorCount -gt 0) { From b4bfb51d818f0f9daab8d1d75a595c96059779f2 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 12:44:36 -0500 Subject: [PATCH 5/7] Made it clearer. --- Management-Utilities/warm_performance_tier/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index 2f9eec9c..a66549fc 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -52,8 +52,9 @@ script twice does, typically, get more data into the performance tier so if you are trying to get as much data as possible into the performance tier, it is recommended to run the script twice. -There are two versions of the script. One that designed to run on a Linux operating system name `warm_performance_tier` -and one that is meant to run on a Windows based one name `WarmPerformanceTier.ps1`. +There are two versions of the script: +o `warm_performance_tier` - Disigned for Linux. +o `WarmPerformanceTier.ps1` - Designed for Windows. ## Set Up The first step is to ensure the volume's tiering policy is set @@ -80,7 +81,7 @@ of all the files in the volume. ## Running The Script -### Runing the script on a Linux based operating system +### Running the script on a Linux-based operating system There are two main ways to run the script. The first is to just provide the script with a directory to start from using the -d option. The script will then read every file in that directory and all its subdirectories. The second way From 10f7641a698d833f1ed6eb15bd62524de0c8563e Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 12:49:07 -0500 Subject: [PATCH 6/7] Added multi-threading for the -a option. --- .../fsx-ontap-aws-cli-scripts/list_fsxn_filesystems | 2 +- .../fsx-ontap-aws-cli-scripts/list_fsxn_svms | 10 +++++----- .../fsx-ontap-aws-cli-scripts/list_fsxn_volumes | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems index c5d9077b..a314c97e 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems @@ -72,7 +72,7 @@ start_watchdog () { region=$3 i=1 - while [ $i -lt $((maxTime * 2)) ]; do + while [ $i -le $((maxTime * 2)) ]; do if ! ps -p $pid > /dev/null 2>&1; then return 0 fi diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms index ae25d299..3583f259 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms @@ -18,7 +18,7 @@ # o Region # o File system ID # o File System Name - optional -# o The managment and NAS IP address of the SVM - optional +# o The management and NAS IP address of the SVM - optional # o SVM ID # o SVM Name ################################################################################ @@ -30,7 +30,7 @@ usage () { cat 1>&2 < /dev/null 2>&1; then return fi diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes index 840487a0..b5af05ee 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes @@ -36,7 +36,7 @@ usage () { Usage $(basename $0) [-r region] [-a] [-X excludeRegions] [-o] [-f fileSystemName] [-i fileSystemId] [-n] [-s svmID] [-x] [-t seconds] [-e endpoint_url] [volume_pattern] Where: volume_pattern - Only volumes that match the egrep pattern will be included. An assumed '^' and '$' is added to the beginning and end of the pattern respectively. - -r means to only include the specified region(s) (comma seperated list). + -r means to only include the specified region(s) (comma separated list). -a means all regions. -X excludeRegions - Comma separated list of regions to exclude from the search. Only valid when -a is specified. -n means include the file system and SVM names. Not compatible with the -x option. @@ -53,8 +53,8 @@ EOF ################################################################################ # This function is used to kill a process, who's ID is passed as the first -# argument, that is running longger than the time specified as the second -# argurment. +# argument, that is running longer than the time specified as the second +# argument. ################################################################################ start_watchdog () { @@ -63,7 +63,7 @@ start_watchdog () { region=$3 i=1 - while [ $i -lt $((maxTime * 2)) ]; do + while [ $i -le $((maxTime * 2)) ]; do if ! ps -p $pid > /dev/null 2>&1; then return fi From eb44b89cb208c4ae65dd13dcad23ce9799d1f7a5 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 13:16:38 -0500 Subject: [PATCH 7/7] Adding a PowerShell version of the warm_performance_tier script. --- .../warm_performance_tier/WarmPerformanceTier.ps1 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 index 6cef5f5d..a8f624ea 100755 --- a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -54,6 +54,9 @@ .\WarmPerformanceTier.ps1 -Help #> +[Diagnostics.CodeAnalysis.SuppressMessageAttribute("PSAvoidUsingWriteHost", "")] +[Diagnostics.CodeAnalysis.SuppressMessageAttribute("PSReviewUnusedParameter", "")] # Since the GitHub PS Lint thinks $VerboseProgress is unused. + [CmdletBinding(DefaultParameterSetName = 'Run')] param( [Parameter(Mandatory = $true, ParameterSetName = 'Run', Position = 0)]