In a nutshell, I have had my initial cloud backup (on Backblaze B2) of a 130+GB directory run into problems after some time, but before completion of a full backup. I have two problems around this:
a) I don’t understand the error message or how to avoid the error:
Fatal: unable to save snapshot: Copy: write /root/.cache/restic/4e40fc3312bbd62e4f0bb04331adbc1f6ed5d64afc1721a992af709fb951fa35/data/f5/f5484474f94a003da23d707926e6ce0ec031ba3125de4aba1c0bbba8f7898a91: no space left on deviceEV1_DATA/tomprimary/Family/Libraries/Photos/2016/01 MexicoCubaUSA/RominiBatch1a/CSC_012
b) the information I get from restic about repo size, number of packs, etc. is contradictory and confounds troubleshooting i.e. restic command “restic … stats --mode raw-data” gives a repo size of 735B, the rebuild-index command indicates 16689 packs, and the backup command (when I was able to get it running) indicates >81GB data uploaded.
I’d like help to understand the error, how to avoid it and how to accurately interpret the numbers regarding data size from the restic commands above.
THE INFO ABOVE SUMMARISES MY ISSUE AND THE HELP WANTED. BELOW IS A FULL LOG OF THE ISSUE
20190430
I had started backup of approx. 140GB directory about 10 days before. Backup was conducted over several sessions with sessions finishing for different reasons (terminated by me (Ctl+c), terminated by cloud service, terminated by power failure). Backup process was resumed each time, sometimes after unlocking repo.
Today I returned to the screen/process on the NAS where ongoing restic backup was being conducted over several days. Previously, tens of GBs of data had been transferred to the cloud repo (Backblaze website currently shows my repo holds 85GB) and I had checked daily that the process continued. However, today the message in the terminal was:
[~] # restic -r b2:TomBucket1 backup /share/CE_CACHEDEV1_DATA/tomprimary
open repository
repository 4e40fc33 opened successfully, password is correct
Fatal: unable to save snapshot: Copy: write /root/.cache/restic/4e40fc3312bbd62e4f0bb04331adbc1f6ed5d64afc1721a992af709fb951fa35/data/f5/f5484474f94a003da23d707926e6ce0ec031ba3125de4aba1c0bbba8f7898a91: no space left on deviceEV1_DATA/tomprimary/Family/Libraries/Photos/2016/01 MexicoCubaUSA/RominiBatch1a/CSC_012
When I repeated the same backup command, I got the same message, albeit with a different filename at the end. I queried the total size of blobs in the repo:
[~] # restic -r b2:TomBucket1 stats
repository 4e40fc33 opened successfully, password is correct
scanning…
Stats for all snapshots in restore-size mode:
Total File Count: 4
Total Size: 0 B
[~] # restic -r b2:TomBucket1 stats --mode raw-data
repository 4e40fc33 opened successfully, password is correct
scanning…
Stats for all snapshots in raw-data mode:
Total Blob Count: 2
Total Size: 735 B
[~] # restic -r b2:TomBucket1 list snapshots
repository 4e40fc33 opened successfully, password is correct
14a4ae0a096d7ee6f809cc04927c3a93ea57f8eba9c2df51994ee7966f240079
7b6f26095cc0e06c8fa03d97e4d093a621f0a950373a3b68b6e7f3c103e5c6bc
I then tried unlocking the repo (in case locking was an issue) but the status was the same:
[~] # restic -r b2:TomBucket1 unlock
repository 4e40fc33 opened successfully, password is correct
successfully removed locks
[~] # restic -r b2:TomBucket1 stats --mode raw-data
repository 4e40fc33 opened successfully, password is correct
scanning…
Stats for all snapshots in raw-data mode:
Total Blob Count: 2
Total Size: 735 B
I then rebuilt the indexes, which removed hundreds of old indexes and created a new one. It also showed more than 16,000 packs but total size of “735 B”- this seems contradictory. (The process took about an hour and I noticed that Backblazes “Daily Class B Transactions” increased one-for-one as the packs got processed.)
[~] # restic -r b2:TomBucket1 rebuild-index
repository 4e40fc33 opened successfully, password is correct
counting files in repo
[1:01:13] 100.00% 16689 / 16689 packs
finding old index files
saved new indexes as [9bb6e5b9 397bef28 30be3c12 46fdc525 685b268c 73dca96e]
remove 520 old index files
[~] # restic -r b2:TomBucket1 stats --mode raw-data
repository 4e40fc33 opened successfully, password is correct
scanning…
Stats for all snapshots in raw-data mode:
Total Blob Count: 2
Total Size: 735 B
I then conducted another backup. The preliminary scan appeared to complete after about 24 minutes, showing that approx. 81 of 136 GiB had been uploaded, and backup then seemed to resume:
…60.01% 30266 files 81.888GiB, total 74226 files 136.466 GiB…
The backup process is ongoing. It would appear that rebuilding the indexes was necessary for the backup process to resume. I am still perplexed by the contradictory reports - repo size of “735 B” and 81+GiB uploaded.