Greetings Restic community, I need some guidance with the observation I made.
The problem is that I made a backup of an XML-like file and while making a dump of this same file, I can see the files are different. Every Line Feed (\n hex 0A) now also has a Carriage Return (\r hex 0D). This causes the file to not be handled in the way I expect it to be.
The file contains sensitive data, but if needed I can see if a smaller file also works. The following steps were taken.
Restoring that same file: restic dump --quiet --no-cache --tag test-backup latest "/stdin" > restored.bck
Checking the diff between those files: cmp --print-bytes backupfile.bck restored.bck
Note the difference: differ: byte 88, line 1 is 12 ^J 15 ^M
I can see in a hexdump that every \n now also has a \r. An XML file like below could be used to test this, if more is needed I can see if I can construct something better:
I suspect the shell is doing fishy things. Are you on Windows? Cygwin?
Can you please try backing up “backupfile.bck” as an argument to restic (and get rid of --stdin) and do the corresponding on restore?
I am actually running this from the Dockerfile, which is the main reason why I am using stdin/stdout.
I did some further testing (with your input) and came to the following conclusions:
Backing up without --stdin and using targeting directly to the file inside of the container causes this same issue. This is when restoring through stdout (restic dump)
Restoring this file through restic restore and using the container filesystem seems to not have this issue
So I assume this problem is either in the restic dump logic or in the conversion of stdout from the container to my main system.
So as additional info:
Running Restic in Docker (Alpine)
Main OS: MacOS
Note that I have been using this same flow for a couple of months every day now for other data. So it might be specific to the data that is being backed up.
dump just writes the file to the output without any extra carriage return bytes. Do you get a carriage return in the output when you just run printf 'abc' in the container? Why do you run restic in the container and not natively on macOS?
Copy the original backup file inside of the container (docker cp)
The cmp command does not show any difference
Now when I copy the dump backup back to my host (through stdout) and run the cmp command, there seems to be no difference. Which fits into the scenario of the first test.
I am thinking this is something of Docker… I will do some more investigation. Thanks for the comment at least!
The MacOS is merely used for development purpose on my local machine. We are using Docker to containerize the environment the Restic tool running in. I am using this locally to test a representative environment that will be running in a production context. Hopefully, to reduce the chance of finding any surprises when we actually deploy it.