Restic sometimes storing user and/or group names in tree, depending on version

1

As my backup after upgrade to 0.8.0 was much bigger than anticipated, I investigated a bit. The whole tree metadata was stored again, even though nothing changed. The problem was gone when I build the current master from source with debug flags.
I seems that restic, depending on version, stores the plaintext username and/or group in all the tree nodes. I cannot exactly say which version I was using, but in older backups e.g. my root node looks like this (with username):

{
  "nodes": [
    {
      "name": "disk1",
      "type": "dir",
      "mode": 2147484159,
      "mtime": "2017-11-15T09:46:13.703046402+01:00",
      "atime": "2017-07-27T00:07:21.919194755+02:00",
      "ctime": "2017-11-15T09:46:13.703046402+01:00",
      "uid": 0,
      "gid": 0,
      "user": "root",
      "inode": 2,
      "device_id": 2064,
      "content": null,
      "subtree": "db2408e44675a1db87cee6ecce27770b895194bef062b493dc84bd6836b6406f"
    }
  ]
}

Somwhere inbetween (a betaversion with cache) the username vanished and with 0.8.0 it now came back and the groupname was added:

{
  "nodes": [
    {
      "name": "disk1",
      "type": "dir",
      "mode": 2147484159,
      "mtime": "2017-11-19T16:21:42.971778943+01:00",
      "atime": "2017-07-27T00:07:21.919194755+02:00",
      "ctime": "2017-11-19T16:21:42.971778943+01:00",
      "uid": 0,
      "gid": 0,
      "user": "root",
      "group": "root",
      "inode": 2,
      "device_id": 2064,
      "content": null,
      "subtree": "b4f2ed8f7fda339986c15cb19432a5ff687daeb79448a4e9c2c715981d837014"
    }
  ]
}

With the current master (restic 0.8.0 (v0.8.0-15-gbb448550)), none of the fields turn up:

{
  "nodes": [
    {
      "name": "disk1",
      "type": "dir",
      "mode": 2147484159,
      "mtime": "2017-11-19T16:21:42.971778943+01:00",
      "atime": "2017-07-27T00:07:21.919194755+02:00",
      "ctime": "2017-11-19T16:21:42.971778943+01:00",
      "uid": 0,
      "gid": 0,
      "inode": 2,
      "device_id": 2064,
      "content": null,
      "subtree": "57866eafbcca0d4c0c95f23710936758091e295a4fe3f8e43de09d6b9805662c"
    }
  ]
}

This is memorywise for the backup not that bad, but for the cache it creates a big increase of memory consumption, I now have the whole tree three times in there.
Are these wanted changes or am I doing something wrong? I didn’t find anything related in the commits …

2

Thanks for bringing this up. I’ve slightly corrected the title, initially the title nearly gave me a heart attack because I thought restic messed up the crypto and saved unencrypted data to the repo :slight_smile:

The issue here lies within the Go version you used to build restic. In Go < 1.9, the functions LookupId() and LookupGroupId(), which return the user and group name for a given user ID, internally called libc when CGO (linking Go and C code) was available. When restic is built with go run build.go, CGO is explicitely deactivated, so the function was not available and restic did not save the user name to the repo.

In Go 1.9, the functions were modified to run without any dependencies to C libraries or code, so from then on restic started to save the user name again.

I hope that this answers your questions and satisfies your curiosity :wink:

3

Ah, I see, thanks for clearing this up (and sorry about the misleading title :slight_smile:).
I was a bit confused because I didn’t see anything in the commits which could have changed the behavior.
The difference then occurred because I switched between beta and stable versions, and these are built with go1.8.1 and go1.9 respectively.
Maybe the build environments could be aligned, so a change between stable and beta doesn’t store all the metadata again in the repository?

4

The difference then occurred because I switched between beta and stable versions, and these are built with go1.8.1 and go1.9 respectively.

Oh, indeed! I’ll update the Go version on the server which builds the betas. Thanks!