Yes, I still have the repository. I can run some stuff against if you like. The log I have from a run that took 40 mins instead of 1 minutes is at bottom. Initially back-ups were so quick that two backups seldom if ever ran at once, but as they took longer to complete, up to around six would be running at once.
It did have one stale lock, because I had to restic unlock
it in order to run a restic prune
on it and it remove one stale lock (which did look like it was there for a while).
I was going to test forget
on it, but failed because all the paths were different, so nothing was forgotten. I realize now I needed --group-by tags
. So this prune actually had nothing to remove. Though it looks like it consolidated 17854 indices into about 14 new indices? Is that right and does consolidation significantly help performance of other tasks?
The prune took almost 8 hours, but the majority of that time was removing old index files. That stage is something else that I think ought to be multi-threaded, since I assume you know all the names that need deleting and the order you delete them probably doesn’t matter?
I often try setting b2.connections
but it is not clear which operations that actually impacts? Would it help with prunes?
$ time restic prune
repository 88b2b1b2 opened successfully, password is correct
counting files in repo
building new index for repo
[1:45:29] 100.00% 39377 / 39377 packs
repository contains 39377 packs (67853 blobs) with 54.278 GiB
processed 67853 blobs: 0 duplicate blobs, 0B duplicate
load all snapshots
find data that is still in use for 17854 snapshots
[4:03] 100.00% 17854 / 17854 snapshots
found 67853 of 67853 data blobs still in use, removing 0 blobs
will remove 0 invalid files
will delete 0 packs and rewrite 0 packs, this frees 0B
counting files in repo
[52:29] 100.00% 39377 / 39377 packs
finding old index files
saved new indexes as [a6fcda4b 785b9556 82079593 0476dd31 23124853 cd0a2e43 513b77f1 7ce9435c e9618549 dd7a34d0 d2ba118f f90a4bb8 fd2c591a 97705eb2]
remove 17854 old index files
done
real 461m19.283s
user 4m34.314s
sys 0m44.614s
2018-08-30 18:56:29+12:00: Backing up MySQL database 'foo'...
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 282.124449ms: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 921.950869ms: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 1.103703713s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 987.812036ms: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 1.368557177s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 5.506539331s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 5.404895106s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 4.799484004s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 13.589122457s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<lock/f7eeb40b91>, 0, 0) returned error, retrying after 9.765553474s: b2_download_file_by_name: 404: bucket foo-backup does not have file: restic/databases/locks/f7eeb40b91fe8b164d9a39abfa5e35018cc5b1bbb06d3ed1fc2d56889693c8b3
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 453.301987ms: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 601.704321ms: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 992.190357ms: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 2.514316474s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 2.744409146s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 4.187550861s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 6.196049677s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 10.743109693s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 11.605431782s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Load(<index/54c8ede2cd>, 0, 0) returned error, retrying after 18.87147923s: Create: open /var/cache/restic/restic/88b2b1b29d8ee557e09ce3af66910944e8956ffe6163a745465cb156b503d00a/index/54/54c8ede2cd9d1b38c2238582f95f1276a9c40a2cb41207f933d575c435d9d978: file exists
Files: 1 new, 0 changed, 0 unmodified
Dirs: 0 new, 0 changed, 0 unmodified
Added: 11.599 MiB
processed 1 files, 0 B in 39:17
snapshot 75e9b71b saved