Tested how many files were processed when using --exclude with various prefix/suffix combinations

Unequal8761 · May 17, 2025, 6:01am

Tested how many files were scanned with various prefix/suffix compinations.

Setup

Ran using powershell on Windows 11.

pointed toward repo, it’s a --dry-run
created a partition called T:\
ran [[#File Structure Generator]]
ran [[#Tester]]

The command

Tested with powershell ps7 [[#Tester]]

command tested --exclude=`“$prefixexcluded$suffix”

`$command = "restic -r $repository backup T:\ --exclude=`"$($pattern.FullPattern)`" --password-file $passwordFile --dry-run --verbose"`

The Prefixes and Suffixes

$backslashPrefixList = @("" , "*", "**", "*.*", "\", "\*", "\**", "*\", "*\*", "*\**", "**\", "**\*", "**\**", "T:\", "T:\*", "T:\**", "T:\*\", "T:\**\", "T:\*\*", "T:\**\*", "T:\*\**", "T:\**\**")
$forwardSlashPrefixList = @("" , "*", "**", "*.*", "/", "/*", "/**", "*/", "*/*", "*/**", "**/", "**/*", "**/**", "T:/", "T:/*", "T:/**", "T:/*/", "T:/**/", "T:/*/*", "T:/**/*", "T:/*/**", "T:/**/**")

# Split suffix list by slash type
$backslashSuffixList = @("", "*", "**", "*.*", "\", "*\", "\*", "\**", "*\*", "*\**", "**\", "**\*", "**\**")
$forwardSlashSuffixList = @("", "*", "**", "*.*", "/", "*/", "/*", "/**", "*/*", "*/**", "**/", "**/*", "**/**")

Folder Structure

Generated with powershell ps7 [[#File Structure Generator]]

# T:.
# │   excluded.txt x 5
# │   file.txt x 1
# ├───excluded
# │   |    excluded.txt x 5
# │   |    file.txt x 100
# |   └───excluded
# |       |    excluded.txt x 5
# |       |    file.txt x 100
# |       └───excluded
# |               excluded.txt x 5
# |               file.txt x 100 
# ├───folder
# │   |    excluded.txt x 5
# │   |    file.txt x 1
# |   └───excluded
# |   |        excluded.txt x 5
# |   |        file.txt x 100
# |   └───folder
# |       |    excluded.txt x 5
# |       |    file.txt x 1
# |       └───excluded
# |       |       excluded.txt x 5
# |       |       file.txt x 100 
# |       └───folder
# |               excluded.txt x 5
# |               file.txt x 1

Results

Backslashes

Forward Slashes

Appendix

File Structure Generator

# Function to create multiple files with the same name

# Function to create multiple files with numbered suffixes
function Create-MultipleFiles {
    param (
        [string]$Path,
        [string]$FileName,
        [int]$Count
    )
   
    # Split filename into name and extension
    $name = [System.IO.Path]::GetFileNameWithoutExtension($FileName)
    $extension = [System.IO.Path]::GetExtension($FileName)
   
    for ($i = 1; $i -le $Count; $i++) {
        # Create filename with number suffix
        $numberedFileName = "$name$i$extension"
        $filePath = Join-Path -Path $Path -ChildPath $numberedFileName
        New-Item -Path $filePath -ItemType File -Force | Out-Null
        Set-Content -Path $filePath -Value "This is file $numberedFileName"
    }
}

# Create base directory structure
$baseDir = "T:"

# Create directory structure
New-Item -Path $baseDir -ItemType Directory -Force | Out-Null

# Base level folders
$excludedDir = Join-Path -Path $baseDir -ChildPath "excluded"
$folderDir = Join-Path -Path $baseDir -ChildPath "folder"
New-Item -Path $excludedDir -ItemType Directory -Force | Out-Null
New-Item -Path $folderDir -ItemType Directory -Force | Out-Null

# Second level in excluded branch
$excludedSubDir = Join-Path -Path $excludedDir -ChildPath "excluded"
New-Item -Path $excludedSubDir -ItemType Directory -Force | Out-Null

# Third level in excluded branch
$excludedSubSubDir = Join-Path -Path $excludedSubDir -ChildPath "excluded"
New-Item -Path $excludedSubSubDir -ItemType Directory -Force | Out-Null

# Second level in folder branch
$folderExcludedDir = Join-Path -Path $folderDir -ChildPath "excluded"
$folderSubDir = Join-Path -Path $folderDir -ChildPath "folder"
New-Item -Path $folderExcludedDir -ItemType Directory -Force | Out-Null
New-Item -Path $folderSubDir -ItemType Directory -Force | Out-Null

# Third level in folder branch
$folderSubExcludedDir = Join-Path -Path $folderSubDir -ChildPath "excluded"
$folderSubSubDir = Join-Path -Path $folderSubDir -ChildPath "folder"
New-Item -Path $folderSubExcludedDir -ItemType Directory -Force | Out-Null
New-Item -Path $folderSubSubDir -ItemType Directory -Force | Out-Null

# Create files in base directory
Create-MultipleFiles -Path $baseDir -FileName "excluded.txt" -Count 5
# Create single file.txt in root
New-Item -Path (Join-Path -Path $baseDir -ChildPath "file1.txt") -ItemType File -Force | Out-Null
Set-Content -Path (Join-Path -Path $baseDir -ChildPath "file1.txt") -Value "This is file1.txt in root"

# Create files in excluded directory
Create-MultipleFiles -Path $excludedDir -FileName "excluded.txt" -Count 5
Create-MultipleFiles -Path $excludedDir -FileName "file.txt" -Count 100

# Create files in excluded/excluded directory
Create-MultipleFiles -Path $excludedSubDir -FileName "excluded.txt" -Count 5
Create-MultipleFiles -Path $excludedSubDir -FileName "file.txt" -Count 100

# Create files in excluded/excluded/excluded directory
Create-MultipleFiles -Path $excludedSubSubDir -FileName "excluded.txt" -Count 5
Create-MultipleFiles -Path $excludedSubSubDir -FileName "file.txt" -Count 100

# Create files in folder directory
Create-MultipleFiles -Path $folderDir -FileName "excluded.txt" -Count 5
# Create single file.txt in folder
New-Item -Path (Join-Path -Path $folderDir -ChildPath "file1.txt") -ItemType File -Force | Out-Null
Set-Content -Path (Join-Path -Path $folderDir -ChildPath "file1.txt") -Value "This is file1.txt in folder"

# Create files in folder/excluded directory
Create-MultipleFiles -Path $folderExcludedDir -FileName "excluded.txt" -Count 5
Create-MultipleFiles -Path $folderExcludedDir -FileName "file.txt" -Count 100

# Create files in folder/folder directory
Create-MultipleFiles -Path $folderSubDir -FileName "excluded.txt" -Count 5
# Create single file.txt in folder/folder
New-Item -Path (Join-Path -Path $folderSubDir -ChildPath "file1.txt") -ItemType File -Force | Out-Null
Set-Content -Path (Join-Path -Path $folderSubDir -ChildPath "file1.txt") -Value "This is file1.txt in folder/folder"

# Create files in folder/folder/excluded directory
Create-MultipleFiles -Path $folderSubExcludedDir -FileName "excluded.txt" -Count 5
Create-MultipleFiles -Path $folderSubExcludedDir -FileName "file.txt" -Count 100

# Create files in folder/folder/folder directory
Create-MultipleFiles -Path $folderSubSubDir -FileName "excluded.txt" -Count 5
# Create single file.txt in folder/folder/folder
New-Item -Path (Join-Path -Path $folderSubSubDir -ChildPath "file1.txt") -ItemType File -Force | Out-Null
Set-Content -Path (Join-Path -Path $folderSubSubDir -ChildPath "file1.txt") -Value "This is file1.txt in folder/folder/folder"

Write-Host "Folder structure created successfully!" -ForegroundColor Green
Write-Host "Structure summary:"
Write-Host "T:."
Write-Host "│   excluded1.txt - excluded5.txt"
Write-Host "│   file1.txt"
Write-Host "├───excluded"
Write-Host "│   |    excluded1.txt - excluded5.txt"
Write-Host "│   |    file1.txt - file100.txt"
Write-Host "|   └───excluded"
Write-Host "|       |    excluded1.txt - excluded5.txt"
Write-Host "|       |    file1.txt - file100.txt"
Write-Host "|       └───excluded"
Write-Host "|               excluded1.txt - excluded5.txt"
Write-Host "|               file1.txt - file100.txt"
Write-Host "├───folder"
Write-Host "│   |    excluded1.txt - excluded5.txt"
Write-Host "│   |    file1.txt"
Write-Host "|   └───excluded"
Write-Host "|   |        excluded1.txt - excluded5.txt"
Write-Host "|   |        file1.txt - file100.txt"
Write-Host "|   └───folder"
Write-Host "|       |    excluded1.txt - excluded5.txt"
Write-Host "|       |    file1.txt"
Write-Host "|       └───excluded"
Write-Host "|       |       excluded1.txt - excluded5.txt"
Write-Host "|       |       file1.txt - file100.txt"
Write-Host "|       └───folder"
Write-Host "|               excluded1.txt - excluded5.txt"
Write-Host "|               file1.txt"

Tester

# Define your repository and password file variables here
$repository = 
$passwordFile = 
$logFile = "restic_backup_log.txt"
$csvSummaryFile = "restic_exclude_summary.csv"

Write-Host "All backup tests completed. Results saved to $logFile"# Script to run restic backup with multiple exclude patterns and log results

# Generate timestamp for filenames (format: YYYYMMDD_HHMMSS)
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$logFile = "restic_backup_log_$timestamp.txt"
$csvSummaryFile = "restic_exclude_summary_$timestamp.csv"

# Define prefixes and suffixes only once
# Split prefix list by slash type
$backslashPrefixList = @("" , "*", "**", "*.*", "\", "\*", "\**", "*\", "*\*", "*\**", "**\", "**\*", "**\**", "T:\", "T:\*", "T:\**", "T:\*\", "T:\**\", "T:\*\*", "T:\**\*", "T:\*\**", "T:\**\**")
$forwardSlashPrefixList = @("" , "*", "**", "*.*", "/", "/*", "/**", "*/", "*/*", "*/**", "**/", "**/*", "**/**", "T:/", "T:/*", "T:/**", "T:/*/", "T:/**/", "T:/*/*", "T:/**/*", "T:/*/**", "T:/**/**")

# Split suffix list by slash type
$backslashSuffixList = @("", "*", "**", "*.*", "\", "*\", "\*", "\**", "*\*", "*\**", "**\", "**\*", "**\**")
$forwardSlashSuffixList = @("", "*", "**", "*.*", "/", "*/", "/*", "/**", "*/*", "*/**", "**/", "**/*", "**/**")

# Create a new log file or clear existing one
"# Restic Exclude Pattern Test - $timestamp" | Out-File -FilePath $logFile
"# Repository: $repository" | Out-File -FilePath $logFile -Append
"" | Out-File -FilePath $logFile -Append

# Function to generate exclude patterns based on prefixes and suffixes
function New-ExcludePatterns {
    param (
        [string]$baseName = "excluded",
        [array]$prefixes,
        [array]$suffixes
    )
   
    $patterns = @()
   
    foreach ($prefix in $prefixes) {
        foreach ($suffix in $suffixes) {
            # Create a custom object with prefix, base, suffix, and the full pattern
            $patternObj = [PSCustomObject]@{
                Prefix      = $prefix
                Base        = $baseName
                Suffix      = $suffix
                FullPattern = "$prefix$baseName$suffix"
            }
            $patterns += $patternObj
        }
    }
   
    return $patterns
}

# Fixed CSV generation method
function New-CSVSummary {
    param (
        [string]$logFilePath,
        [string]$outputCSVPath,
        [array]$prefixes,
        [array]$suffixes
    )
    
    Write-Host "Generating CSV summary from log file..."
    
    # Parse the log file and extract data directly into a structure
    $content = Get-Content -Path $logFilePath -Raw
    
    # Create a 2D array to hold the results
    $resultData = @{}
    
    # Split the content into blocks based on the separator line
    $patternRegex = [regex]'(?s)={80}\r?\nEXCLUDE PATTERN: --exclude="(.*?)"\r?\nPREFIX: "(.*?)"\r?\nBASE: "(.*?)"\r?\nSUFFIX: "(.*?)"\r?\n={80}(.*?)(?=={80}|\z)'
    $regexMatches = $patternRegex.Matches($content)
    
    # Process each matched pattern block
    foreach ($match in $regexMatches) {
        $fullPattern = $match.Groups[1].Value
        $prefix = $match.Groups[2].Value
        $baseName = $match.Groups[3].Value
        $suffix = $match.Groups[4].Value
        $blockContent = $match.Groups[5].Value
        
        # Extract the file count from the block
        $filesProcessedMatch = [regex]::Match($blockContent, 'processed\s+(\d+)\s+files')
        
        if ($filesProcessedMatch.Success) {
            $filesProcessed = [int]$filesProcessedMatch.Groups[1].Value
            
            # Create entries for this prefix-suffix combination
            if (-not $resultData.ContainsKey($prefix)) {
                $resultData[$prefix] = @{}
            }
            
            # Store the file count
            $resultData[$prefix][$suffix] = $filesProcessed
            Write-Host "Found data: PREFIX='$prefix', SUFFIX='$suffix', FILES=$filesProcessed"
        }
    }
    
    # Create CSV data structure
    $csvRows = @()
    
    # Process each prefix to create rows
    foreach ($prefix in $prefixes) {
        $rowData = [ordered]@{ 
            "Prefix" = if ($prefix -eq "") { "(empty)" } else { $prefix } 
        }
        
        # Add columns for each suffix
        foreach ($suffix in $suffixes) {
            $columnName = if ($suffix -eq "") { "(empty)" } else { $suffix }
            
            # Get data if it exists, otherwise use 0
            if ($resultData.ContainsKey($prefix) -and $resultData[$prefix].ContainsKey($suffix)) {
                $rowData[$columnName] = $resultData[$prefix][$suffix]
            }
            else {
                $rowData[$columnName] = "NA"
            }
        }
        
        # Add this row to our collection
        $csvRows += New-Object PSObject -Property $rowData
    }
    
    # Export the data to CSV
    $csvRows | Export-Csv -Path $outputCSVPath -NoTypeInformation
    
    # Add header comments to a separate info file
    @"
# Restic Exclude Pattern Summary - Generated on $(Get-Date)
# Source log file: $logFilePath
"@ | Out-File -FilePath "$outputCSVPath.info.txt"
    
    Write-Host "CSV summary generated at: $outputCSVPath"
    Write-Host "Found data for $($resultData.Count) different prefixes"
}

# Generate the exclude patterns
$basename = "excluded"
$backSlashExcludePatterns = New-ExcludePatterns -baseName $basename -prefixes $backslashPrefixList -suffixes $backslashSuffixList
$forwardSlashExcludePatterns = New-ExcludePatterns -baseName $basename -prefixes $forwardSlashPrefixList -suffixes $forwardSlashSuffixList
$excludePatterns = $backSlashExcludePatterns + $forwardSlashExcludePatterns

# Display how many patterns we'll be testing
Write-Host "Generated $($excludePatterns.Count) exclude patterns to test"
Write-Host "Test started at $(Get-Date)"
"Test started at $(Get-Date)" | Out-File -FilePath $logFile -Append

foreach ($pattern in $excludePatterns) {
    # Add header for current exclude pattern with prefix and suffix on separate lines
    $header = "=" * 80
    $header | Out-File -FilePath $logFile -Append
    "EXCLUDE PATTERN: --exclude=`"$($pattern.FullPattern)`"" | Out-File -FilePath $logFile -Append
    "PREFIX: `"$($pattern.Prefix)`"" | Out-File -FilePath $logFile -Append
    "BASE: `"$($pattern.Base)`"" | Out-File -FilePath $logFile -Append
    "SUFFIX: `"$($pattern.Suffix)`"" | Out-File -FilePath $logFile -Append
    $header | Out-File -FilePath $logFile -Append
    "" | Out-File -FilePath $logFile -Append
   
    # Run the restic command with current exclude pattern and append output to log file
    Write-Host "Running backup with exclude pattern: $($pattern.FullPattern)"
    $command = "restic -r $repository backup T:\ --exclude=`"$($pattern.FullPattern)`" --password-file $passwordFile --dry-run --verbose"
   
    # Execute command and capture output
    $output = Invoke-Expression $command 2>&1 | Out-String
   
    # Append command output to log file
    $output | Out-File -FilePath $logFile -Append
   
    # Add separator after each run
    "" | Out-File -FilePath $logFile -Append
    "" | Out-File -FilePath $logFile -Append
}

"Test completed at $(Get-Date)" | Out-File -FilePath $logFile -Append
Write-Host "All backup tests completed at $(Get-Date)"
Write-Host "Results saved to $logFile"

# Generate CSV summary from the log file
$allPrefixes = $backslashPrefixList + $forwardSlashPrefixList | Select-Object -Unique
$allSuffixes = $backslashSuffixList + $forwardSlashSuffixList | Select-Object -Unique
New-CSVSummary -logFilePath $logFile -outputCSVPath $csvSummaryFile -prefixes $allPrefixes -suffixes $allSuffixes

Unequal8761 · May 25, 2025, 9:24pm

My initial concern was how long it would take to run a backup. My perspective is that backup time is directly correlated with how many files that have to be processed when the files are small. When the files are large that may be the limiting factor. I seem to have a lot of small files in directories like node_module directories and wanted to limit that. The green ones are what I wanted to see. I overwhelmed the excluded folders because I wanted to see how many of those directories were getting scanned completely.

The surprising one to me was T:\** as a prefix. T:\**excluded* seems to process all the files in the following, even though I would expect it completely safe to skip them because ** is any path:

folder/excluded
folder/folder/excluded

Limitations:
My provided folder structure might not be complete enough as I made it for one purpose.

MichaelEischer · May 26, 2025, 5:38pm

Does that answer rather belong to Tested how many files were processed when using --exclude with various prefix/suffix combinations - #2 by Unequal8761 ? The current thread contains too little context and background on what you’re trying to achieve, which is probably why there hasn’t been any feedback.

Unequal8761 · May 28, 2025, 2:54pm

That link is to this thread.