Restic shows huge amount of data being backed up while there is none

aerbil313 · July 22, 2023, 2:58pm

I’m running restic on NixOS for the first time to backup / (filesystem root). I’m using it via a security wrapper, so I can run it without root. I gave cap_dac_read_search=+ep capability to it. It was fine, progress showing about 50 GiB of files (my machine has about 70 GiB) until it stumled upon symlinks in nix store. FYI AFAIK there are tons of symlinks in NixOS, sometimes taking up gigabytes of space themselves. You can see from the log that:

It could not read /proc despite the capability, but probably this is by design on the side of linux kernel, and I don’t care about /proc being backed up, since it’s the directory of running processes. Still curious though.
Restic shows 128 TiB of files in existence since when the scan is finished. My machine only has a 256 GB SSD. Why? This seems like a bug.

user@host ~> /run/wrappers/bin/restic --repo 'rclone:user@gmail.com:My Computer Backup' --verbose backup / --dry-run
open repository
enter password for repository: 
repository e1a1a228 opened (version 2, compression level auto)
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/]
start backup on [/]
<I trimmed lines here because it didn''t fit in the forum size limit>
[4:52] 135688 files 10.080 GiB, total 663033 files 44.386 GiB, 0 errors
scan: open /proc/1/fdinfo: permission denied
scan: Readdirnames /proc/1/map_files failed: readdirent /proc/1/map_files: permission denied
scan: open /proc/1/task/1/fdinfo: permission denied
scan: open /proc/10/fdinfo: permission denied
scan: Readdirnames /proc/10/map_files failed: readdirent /proc/10/map_files: permission denied
scan: open /proc/10/task/10/fdinfo: permission denied
scan: open /proc/1041/fdinfo: permission denied
scan: Readdirnames /proc/1041/map_files failed: readdirent /proc/1041/map_files: permission denied
scan: open /proc/1041/task/1041/fdinfo: permission denied
scan: open /proc/1043/fdinfo: permission denied
scan: Readdirnames /proc/1043/map_files failed: readdirent /proc/1043/map_files: permission denied
scan: open /proc/1043/task/1043/fdinfo: permission denied
scan: open /proc/1076/fdinfo: permission denied
scan: Readdirnames /proc/1076/map_files failed: readdirent /proc/1076/map_files: permission denied
scan: open /proc/1076/task/1076/fdinfo: permission denied
scan: open /proc/1076/task/1077/fdinfo: permission denied
scan: open /proc/1076/task/1078/fdinfo: permission denied
< again trimmed >
scan: Readdirnames /proc/959/map_files failed: readdirent /proc/959/map_files: permission denied
scan: open /proc/959/task/959/fdinfo: permission denied
scan: open /proc/971/fdinfo: permission denied
scan: Readdirnames /proc/971/map_files failed: readdirent /proc/971/map_files: permission denied
scan: open /proc/971/task/971/fdinfo: permission denied
scan finished in 1401.260s: 2126825 files, 128.113 TiB
[24:32] 0.03%  503596 files 39.940 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1342:37:03
[24:49] 0.03%  517975 files 40.701 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1332:44:29
[26:46] 0.03%  620457 files 43.562 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1343:01:19
[27:14] 0.03%  651610 files 44.144 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1348:24:12
[27:32] 0.03%  678366 files 44.866 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1341:19:16
[27:32] 0.03%  678393 files 44.866 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1341:18:53
[27:32] 0.03%  678583 files 44.868 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1341:16:10
[27:32] 0.03%  678713 files 44.869 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1341:13:50
[28:31] 0.03%  738353 files 45.784 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1361:21:14
[28:32] 0.03%  738971 files 45.837 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1360:35:24
[28:32] 0.03%  739085 files 45.839 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1360:31:00
[28:34] 0.04%  741525 files 45.958 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1358:34:26
[28:34] 0.04%  741731 files 45.963 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1358:26:15
[28:35] 0.04%  742518 files 45.998 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1358:12:14
[28:36] 0.04%  744583 files 46.068 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1356:55:45
[28:38] 0.04%  746336 files 46.145 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1356:14:29
[28:38] 0.04%  746371 files 46.148 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1356:09:44
[29:06] 0.04%  780799 files 47.293 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1344:53:02
[29:06] 0.04%  780812 files 47.295 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1344:48:59
[29:06] 0.04%  780874 files 47.296 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1344:47:16
[29:26] 0.04%  787665 files 48.020 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1339:40:41
[29:26] 0.04%  787966 files 48.029 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1339:26:01
[29:27] 0.04%  788235 files 48.062 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1339:15:07
[29:27] 0.04%  788265 files 48.063 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1339:14:15
[29:27] 0.04%  788446 files 48.067 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1339:07:30
[29:27] 0.04%  788484 files 48.068 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1339:06:16
[29:30] 0.04%  790967 files 48.188 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1338:01:48
[29:30] 0.04%  791751 files 48.223 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:04:07
[29:30] 0.04%  791759 files 48.223 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:03:37
[29:30] 0.04%  791776 files 48.223 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:03:14
[29:30] 0.04%  791788 files 48.224 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:01:43
[29:31] 0.04%  792458 files 48.237 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:24:34
[29:34] 0.04%  794480 files 48.299 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:57:57
[29:34] 0.04%  794555 files 48.300 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:57:12
[29:36] 0.04%  795758 files 48.378 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1337:17:04
[29:56] 0.04%  814329 files 49.330 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1326:13:36
[29:56] 0.04%  814377 files 49.332 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1326:11:56
[30:17] 0.04%  829514 files 49.920 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1325:52:57
[30:30] 0.04%  833929 files 50.185 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1328:19:27
[31:24] 0.04%  893141 files 51.881 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1322:47:44
[31:45] 0.04%  923021 files 52.074 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1332:34:19
[32:12] 0.04%  932756 files 52.362 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1344:01:47
[34:08] 0.04%  999656 files 54.550 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1367:33:50
[34:17] 0.04%  1002885 files 54.623 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1371:43:15
[36:39] 0.04%  1108849 files 59.020 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1357:08:15
[38:06] 0.05%  1164781 files 61.513 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1353:37:19
[38:06] 0.05%  1164785 files 61.513 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1353:36:47
[38:06] 0.05%  1164795 files 61.514 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1353:35:34
[38:06] 0.05%  1164810 files 61.514 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1353:35:22
[38:33] 0.05%  1174300 files 62.659 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1344:31:59
[38:51] 0.05%  1189261 files 62.728 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1353:30:34
[39:19] 0.05%  1220041 files 62.856 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1366:59:32
[39:28] 0.05%  1234820 files 63.198 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1364:46:22
[39:33] 0.05%  1238040 files 63.533 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1360:26:30
[39:33] 0.05%  1238190 files 63.536 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1360:21:56
[39:33] 0.05%  1239087 files 63.574 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1359:33:17
[39:43] 0.05%  1246829 files 63.997 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1356:15:29
[39:49] 0.05%  1255626 files 64.193 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1355:31:28
[40:00] 0.05%  1266562 files 64.913 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1346:38:33
[40:00] 0.05%  1266676 files 64.922 GiB, total 2126825 files 128.113 TiB, 0 errors ETA 1346:28:24

aerbil313 · July 22, 2023, 3:14pm

Update: It’s currently in the uploading the backup stage of the dry run and says it has uploaded 990 GiB of data and ~2 million files. The process went on in the same way as it was scanning, slow climb to around 70 GiB, and then accelerating fast to TiBs. It also shows 3000+ errors, all again caused by trying to read /proc.

MichaelEischer · July 22, 2023, 3:55pm

Please exclude /proc, /run, /sys and /dev from you backups. It make no sense at all to backup these folder as they can’t be restored anyways. I wonder whether some of the pseudofiles in /proc or the other pseudofilesystems cause the multi-TB size estimation.

To debug that, you’ll have to run restic backup -vv and comb through the output, which will include the size of each file.

aerbil313 · July 22, 2023, 7:47pm

Thanks!

Also, does it make sense to exclude ~/.cache, other cache directories, and /tmp? The OS and programs will start slower after a restore, but it’s fine to exclude them, isn’t it?

martinleben · July 23, 2023, 10:30am

/tmp will often be cleared by the OS after restart. You need to check and decide for yourself if you want to back it up.

Cache directories are safe to exclude, but it requires that programs using them are well designed. Once again, the decision is yours.

I exclude both of these categories.