Errors in following some rclone wasabi errors

deanroker123 · July 30, 2019, 8:34am

Hi Everyone,

I have a repo that is in Wasabi and is accessed via rclone. The I am backing up about 750GB of data.

I have had a few occasions recently where i think there were network problems with wasabi and had a few failed backups.

I tried to run a prune and got a stack trace out (that I forgot to copy, but it was complaining about a nil pointer on this line.

github.com

restic/restic/blob/v0.9.5/internal/repository/index.go#L377


	defer idx.m.Unlock()


	idx.final = true


	return idx.encode(w)
}


// ID returns the ID of the index, if available. If the index is not yet
// finalized, an error is returned.
func (idx *Index) ID() (restic.ID, error) {
	idx.m.Lock()
	defer idx.m.Unlock()


	if !idx.final {
		return restic.ID{}, errors.New("index not finalized")
	}


	return idx.id, nil
}


// SetID sets the ID the index has been written to. This requires that

I ran a rebuilt index and a check and there are issues with the repo. The files it is reporting problems with are new and I still have them. How do I upload the missing data?

Here is the output of check.

repository 4d8102dd opened successfully, password is correct
created new cache in /tmp/restic-check-cache-284961103
create exclusive lock for repository
load indexes
check all packs
pack 36a9eab0: not referenced in any index
pack 9da30dbb: not referenced in any index
2 additional files were found in the repo, which likely contain duplicate data.
You can run restic prune to correct this.
check snapshots, trees and blobs
error for tree 965eb013:
tree 965eb013: file “DRP_4225.NEF” blob 24 size could not be found
tree 965eb013: file “DRP_4225.NEF” blob 25 size could not be found
tree 965eb013: file “DRP_4225.NEF” blob 26 size could not be found
tree 965eb013: file “DRP_4225.NEF” blob 27 size could not be found
tree 965eb013, blob 06e35f41: not found in index
tree 965eb013, blob a5e1096e: not found in index
tree 965eb013, blob 8990cbfd: not found in index
tree 965eb013, blob 5a57d7b3: not found in index
error for tree 4b9e4553:
tree 4b9e4553: file “DRP_4225.NEF” blob 24 size could not be found
tree 4b9e4553: file “DRP_4225.NEF” blob 25 size could not be found
tree 4b9e4553: file “DRP_4225.NEF” blob 26 size could not be found
tree 4b9e4553: file “DRP_4225.NEF” blob 27 size could not be found
tree 4b9e4553, blob 06e35f41: not found in index
tree 4b9e4553, blob a5e1096e: not found in index
tree 4b9e4553, blob 8990cbfd: not found in index
tree 4b9e4553, blob 5a57d7b3: not found in index
error for tree c1861850:
tree c1861850: file “DRP_4225.NEF” blob 24 size could not be found
tree c1861850: file “DRP_4225.NEF” blob 25 size could not be found
tree c1861850: file “DRP_4225.NEF” blob 26 size could not be found
tree c1861850: file “DRP_4225.NEF” blob 27 size could not be found
tree c1861850, blob 06e35f41: not found in index
tree c1861850, blob a5e1096e: not found in index
tree c1861850, blob 8990cbfd: not found in index
tree c1861850, blob 5a57d7b3: not found in index
Fatal: repository contains errors

cdhowie · July 30, 2019, 3:31pm

This is very strange; if you did rebuild the index then check should not be indicating that there are packs that aren’t referenced in an index. Can you try rebuilding the index again?

deanroker123 · July 30, 2019, 5:16pm

Will do, I will start now and report back.

I was more worried about the blob not being found. How can I get the blobs re-uploaded? The files are still on disk so the blobs should be able to be re-uploaded.

cdhowie · July 30, 2019, 6:36pm

The blobs that can’t be found are probably in that pack.

You can always perform another backup. If the blobs aren’t in the index then the backup process should upload them.

deanroker123 · July 30, 2019, 6:46pm

I have done that and check still reports missing blobs

deanroker123 · July 30, 2019, 6:56pm

rebuild-index is running now. I have these errors so far.

counting files in repo
rclone: 2019/07/30 18:51:53 ERROR : data/3a/3a1e61a6f3080b171f7d4fb1fec22ae59d793bd8218082766b69ee063375ec66: Didn’t finish writing GET request (wrote 15875/23568 bytes): http2: stream closed
Load(<data/36a9eab0d3>, 591, 4351271) returned error, retrying after 720.254544ms: <data/36a9eab0d3> does not exist
Load(<data/36a9eab0d3>, 591, 4351271) returned error, retrying after 873.42004ms: <data/36a9eab0d3> does not exist
Load(<data/36a9eab0d3>, 591, 4351271) returned error, retrying after 1.054928461s: <data/36a9eab0d3> does not exist
Load(<data/36a9eab0d3>, 591, 4351271) returned error, retrying after 1.560325776s: <data/36a9eab0d3> does not exist
Load(<data/36a9eab0d3>, 591, 4351271) returned error, retrying after 3.004145903s: <data/36a9eab0d3> does not exist
pack file cannot be listed 36a9eab0d34802f9ffa98b803bda14b8b903cdf241e16a9c40c31a6a7f0de679: <data/36a9eab0d3> does not exist
Load(<data/7567d2435c>, 591, 4740437) returned error, retrying after 282.818509ms: <data/7567d2435c> does not exist
Load(<data/7573d70953>, 591, 5929857) returned error, retrying after 328.259627ms: <data/7573d70953> does not exist
Load(<data/755f1ee894>, 591, 4862607) returned error, retrying after 298.484759ms: <data/755f1ee894> does not exist
Load(<data/7570293fd8>, 591, 5247113) returned error, retrying after 400.45593ms: <data/7570293fd8> does not exist
Load(<data/758ab7763b>, 591, 5744917) returned error, retrying after 507.606314ms: <data/758ab7763b> does not exist
Load(<data/7573d70953>, 591, 5929857) returned error, retrying after 985.229971ms: <data/7573d70953> does not exist
Load(<data/7570293fd8>, 591, 5247113) returned error, retrying after 535.697904ms: <data/7570293fd8> does not exist
Load(<data/7567d2435c>, 591, 4740437) returned error, retrying after 660.492892ms: <data/7567d2435c> does not exist
Load(<data/755f1ee894>, 591, 4862607) returned error, retrying after 613.543631ms: <data/755f1ee894> does not exist
Load(<data/755f1ee894>, 591, 4862607) returned error, retrying after 1.090001075s: <data/755f1ee894> does not exist
Load(<data/7573d70953>, 591, 5929857) returned error, retrying after 880.91342ms: <data/7573d70953> does not exist
Load(<data/755f1ee894>, 591, 4862607) returned error, retrying after 1.338359384s: <data/755f1ee894> does not exist
Load(<data/7567d2435c>, 591, 4740437) returned error, retrying after 1.326470261s: <data/7567d2435c> does not exist
Load(<data/7570293fd8>, 591, 5247113) returned error, retrying after 808.372184ms: <data/7570293fd8> does not exist
Load(<data/755f1ee894>, 591, 4862607) returned error, retrying after 1.779941781s: <data/755f1ee894> does not exist
Load(<data/758ab7763b>, 591, 5744917) returned error, retrying after 645.653563ms: <data/758ab7763b> does not exist
Load(<data/759a1df963>, 591, 6053031) returned error, retrying after 535.336638ms: <data/759a1df963> does not exist

I am not sure if I got those when i ran it last time?

I have been seeing the rclone one since I upgraded to 0.9.5, although i think i read a post about that not being reported in previous versions even though it was happening?

deanroker123 · July 30, 2019, 9:53pm

I have been doing some poking about and I think this is a wasabi problem.

If i mount the repo using rclone I can see the files but cant download them. If I use the web gui on wasabi, I can see them on initial opening of the folder, but if i search they are not there. and if i try and download them I get a 404.

I have opened a case with wasabi to work out what is going on. I hope their marketing claim about losing one file every 600,000 years is true

Thanks

Dean

deanroker123 · July 31, 2019, 8:54pm

I believe this was a Wasabi error, they say they fixed the issue and now the rebuild index worked without problem. Im just running a check.

Could errors where getting some data times out rather than gives a 404 be reported as a problem with the back end rather than saying there is a problem with the repo during check, or abort the action saying the storage isn’t reliable enough?

I think that could save people a lot of time and heart ache when they think their backup might be faulty?

What do people think?