Backup dump results in extra Carriage Return bytes

Getting Help
#1

Greetings Restic community, I need some guidance with the observation I made.

The problem is that I made a backup of an XML-like file and while making a dump of this same file, I can see the files are different. Every Line Feed (\n hex 0A) now also has a Carriage Return (\r hex 0D). This causes the file to not be handled in the way I expect it to be.

The file contains sensitive data, but if needed I can see if a smaller file also works. The following steps were taken.

  1. Backup the file: restic backup --host "$(hostname)" --verbose --tag test-backup --stdin < backupfile.bck
  2. Restoring that same file: restic dump --quiet --no-cache --tag test-backup latest "/stdin" > restored.bck
  3. Checking the diff between those files: cmp --print-bytes backupfile.bck restored.bck
  4. Note the difference: differ: byte 88, line 1 is 12 ^J 15 ^M

I can see in a hexdump that every \n now also has a \r. An XML file like below could be used to test this, if more is needed I can see if I can construct something better:

<?xml version="1.0" encoding="UTF-8"?>

<properties>
</properties>

Are there any problems related to this known? Would this be overcome by first compressing the file (ie tar)? Any help is appreciated.

**Restic version: ** 0.9.6

Note that forcing this conversion with dos2unix --force -b restored.bck does seem to work.

#2

I suspect the shell is doing fishy things. Are you on Windows? Cygwin?
Can you please try backing up “backupfile.bck” as an argument to restic (and get rid of --stdin) and do the corresponding on restore?

#3

I am actually running this from the Dockerfile, which is the main reason why I am using stdin/stdout.

I did some further testing (with your input) and came to the following conclusions:

  • Backing up without --stdin and using targeting directly to the file inside of the container causes this same issue. This is when restoring through stdout (restic dump)
  • Restoring this file through restic restore and using the container filesystem seems to not have this issue

So I assume this problem is either in the restic dump logic or in the conversion of stdout from the container to my main system.

So as additional info:

  • Running Restic in Docker (Alpine)
  • Main OS: MacOS

Note that I have been using this same flow for a couple of months every day now for other data. So it might be specific to the data that is being backed up.

#4

I only know how to spell Docker… Someone else might be able to help you more. Good luck!

#5

dump just writes the file to the output without any extra carriage return bytes. Do you get a carriage return in the output when you just run printf 'abc' in the container? Why do you run restic in the container and not natively on macOS?

1 Like
#6

I found the following interesting things.

Test 1: Using stdout for a file does not extra carriage bytes

  1. Inside the container printf "abc" > container.txt
  2. Copied the content to my host with stdout (docker exec <container> cat container.txt > container.txt)
  3. On my host printf "abc" > host.txt
  4. Compared cmp --print-bytes container.txt host.txt and they seem to be exactly the same

Test 2: Restoring the process inside of my container does not add extra carriage bytes

  1. Inside container: RESTIC_REPOSITORY=<repo> restic dump --quiet --no-cache --tag <tag> latest "/stdin" > container.bck
  2. Copy the original backup file inside of the container (docker cp)
  3. The cmp command does not show any difference

Now when I copy the dump backup back to my host (through stdout) and run the cmp command, there seems to be no difference. Which fits into the scenario of the first test.

I am thinking this is something of Docker… I will do some more investigation. Thanks for the comment at least!

The MacOS is merely used for development purpose on my local machine. We are using Docker to containerize the environment the Restic tool running in. I am using this locally to test a representative environment that will be running in a production context. Hopefully, to reduce the chance of finding any surprises when we actually deploy it.

#7

I found the issue. It has to do with TTY allocation, which allocates CRLF by default (defined by the OS’ kernel). See docker outputs to standard out with added carriage return character · Issue #8513 · moby/moby · GitHub. With that docker-compose run allocates a TTY by default… This can disabled by passing the -T option:

Disable pseudo-tty allocation. By default docker-compose run allocates a TTY. - docker-compose run | Docker Documentation

After running this with the -T option the files seem to be the same in the host :partying_face: Thanks for all the help, this was a real mystery to me.

3 Likes