Panic: offset or length does not fit in uint32. You have packs > 4GB!

Getting Help
1

I backup a folder of about 90TiB and got the below error:

“panic: offset or length does not fit in uint32. You have packs > 4GB!”

And there is no snapshot produced. I back up even large folder in the same system and is successful. Please advise how could it be fixed.

Many thanks.

2

And also below was recorded:
goroutine 124 [running]:
github.com/restic/restic/internal/repository.(*Index).store(0xc873142f00, 0x1da, 0xdf76817678ce93fb, 0xdbdf2adc6135cb3a, 0x3fc474dfa0b7c84b, 0xe893a60946777f41, 0x2, 0x12c88041c, 0x20bbde)
github.com/restic/restic/internal/repository/index.go:78 +0xc9
github.com/restic/restic/internal/repository.(*Index).StorePack(0xc873142f00, 0xc4e59fb2dc4c10ae, 0xce6f64d8b6a1648d, 0x77bcdfcc51e7c589, 0x5a4aeddb84755265, 0xc75f9ac000, 0x57a, 0x5b6)
github.com/restic/restic/internal/repository/index.go:161 +0x1f7
github.com/restic/restic/internal/repository.(*MasterIndex).StorePack(0xc0004e8a80, 0xc4e59fb2dc4c10ae, 0xce6f64d8b6a1648d, 0x77bcdfcc51e7c589, 0x5a4aeddb84755265, 0xc75f9ac000, 0x57a, 0x5b6)
github.com/restic/restic/internal/repository/master_index.go:176 +0x2ab
github.com/restic/restic/internal/repository.(*Repository).savePacker(0xc0004fbc00, 0x5559054e2080, 0xc00022a480, 0xc474dfa0b7c84b02, 0xc731192be0, 0xe8, 0xcd68902000)
github.com/restic/restic/internal/repository/packer_manager.go:144 +0x332
github.com/restic/restic/internal/repository.(*Repository).SaveAndEncrypt(0xc0004fbc00, 0x5559054e2080, 0xc00022a480, 0x3fc474dfa0b7c802, 0xcc3c080000, 0x12c8803fc, 0x12c882000, 0xdf76817678ce93fb, 0xdbdf2adc6135cb3a, 0x3fc474dfa0b7c84b, …)
github.com/restic/restic/internal/repository/repository.go:287 +0x305
github.com/restic/restic/internal/repository.(*Repository).SaveBlob(0xc0004fbc00, 0x5559054e2080, 0xc00022a480, 0x100c03e1aad02, 0xcc3c080000, 0x12c8803fc, 0x12c882000, 0x0, 0x0, 0x0, …)
github.com/restic/restic/internal/repository/repository.go:718 +0x225
github.com/restic/restic/internal/archiver.(*BlobSaver).saveBlob(0xc0003063c0, 0x5559054e2080, 0xc00022a480, 0x2, 0xcc3c080000, 0x12c8803fc, 0x12c882000, 0x0, 0x0, 0x0, …)
github.com/restic/restic/internal/archiver/blob_saver.go:102 +0xc8
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xc0003063c0, 0x5559054e2080, 0xc00022a480, 0xc00003dec0, 0xc00022a480, 0x699977de0587777a)
github.com/restic/restic/internal/archiver/blob_saver.go:123 +0x185
github.com/restic/restic/internal/archiver.NewBlobSaver.func1(0x829238941dfbf822, 0x563925a518d489ac)
github.com/restic/restic/internal/archiver/blob_saver.go:34 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000180050, 0xc06ce7eba0)
gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xbc

3

Until we can fix memory usage when backing up directories containing many empty files · Issue #2446 · restic/restic · GitHub there’s no real workaround for this issue. Using restic 0.9.6 might work, although it probably causes other problems. In particular that restic version won’t be able to prune such a large repository.

Essentially the only way forward for now is to somehow split that folder into separate backup runs.