Restic Backup

Restic is the current standard in encrypted remote backups. Having encrypted remote backups is a TNO Trust-No-One strategy where backups can make use of storage on an untrusted remote host. This would include cloud hosting providers. Restic is very fast. Storage uses a de-duplicated pool structure such that incremental backups use only storage that has changed since any previous backup snapshot.

I am going to walk through using restic with backups and restores as a non-root user. This is convenient to be able to restore as myself and not need root for it. Restic may also be used by the root superuser to backup the entire system. I am going to cover a non-root setup and use here. Non-root except for installation of restic, keychain, and openssh-client at least.

Installation

Restic is packaged for most operating systems. Simply install it. This only needs to be installed on the client. The remote backup storage server does not need it installed.

The setup I am going to show also requires openssh-client and the keychain utility. It is almost inconceivable that you don't already have ssh installed. Install the restic and keychain packages.

root# apt-get install restic
root# apt-get install keychain

Both of these install a single self-contained executable program each. These files could be copied as non-root from another system. In theory root is not even needed.

Restic is written in Go-lang and therefore the executable program it produces has very few dependencies as everything is compiled in. This has pros and cons and the public debate on the merits and demerits is ongoing.

rwp@ennui:~$ restic version
restic 0.14.0 compiled with go1.19.8 on linux/amd64

rwp@ennui:~$ type restic
restic is /usr/bin/restic

rwp@ennui:~$ file /usr/bin/restic
/usr/bin/restic: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=0f4e9855978400dd3c35d79150e0829996680ad8, for GNU/Linux 3.2.0, stripped

rwp@ennui:~$ ldd -d -r /usr/bin/restic
        linux-vdso.so.1 (0x00007ffe043cd000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f23b00aa000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f23b02a4000)

The keychain program is a single POSIX portable shell script.

rwp@ennui:~$ file /usr/bin/keychain
/usr/bin/keychain: POSIX shell script, ASCII text executable, with escape sequences

With installation done and the programs ready let's get ssh set up.

SSH SFTP Setup

The most typical transport used with Restic is sftp a component of SSH. The best way to use ssh is with ssh public-private keys. Everyone reading this document will already know all about using ssh with ssh keys but here is a quickstart overview.

Create an ed25519 key which becomes the default in OpenSSH 9.5/9.5p1 (2023-10-04) which is very new at this writing. For previous versions the type must be manually specified. If doing this as you a non-root user be sure to specify a passphrase to protect the key. If doing this as root for use by root then file permissions are sufficient and I do not specify a passphrase.

ssh-keygen -t ed25519

Add this ssh key to your ssh-agent which should already be running. If this says that it cannot connect to the agent then set that up first as it is needed. Setting up the ssh-agent is not difficult but beyond the scope of this document and there are many other HOWTOs and tutorials on setting up the ssh-agent.

ssh-add

Copy the public key to the remote server that is going to be used. Here the name "havoc" is the host name of my server machine.

ssh-copy-id havoc

That's most of the ssh setup. This gets things set up so that we can ssh from the local client to the remote host without needing to interactively type in a password. Normally ssh using ssh-agent is limited to interactive command line use. And I am going to run this by cron so that it happens automatically on a periodic schedule. Which means that my cron job needs access to the ssh key too. This will be done by using the keychain framework.

The man page for keychain describes it this way.

keychain is a manager for ssh-agent, typically run from
~/.bash_profile.  It allows your shells and cron jobs to easily
share a single ssh-agent process.  By default, the ssh-agent
started by keychain is long-running and will continue to run, even
after you have logged out from the system.  If you want to change
this behavior, take a look at the --clear and --timeout options,
described below.

It's perfect for our needs. I put this into my ~/.bash_profile file. Or my ~/.profile if using it. This will check that keychain is available and if available then ensure it is started and running and then import ("source" using the "." command) the shell syntax environment variable settings such that newly created login shells will share the single running ssh-agent.

if (keychain --version) >/dev/null 2>&1; then
    keychain -q
    if [ -f $HOME/.keychain/$(hostname)-sh ]; then
        . $HOME/.keychain/$(hostname)-sh
    fi
fi

This leaves the ssh-agent running between logins. If that is a concern such as on an untrusted host, well if you are running on an untrusted host then things cannot be completely secure no matter what you do anyway. But you can delete keys from the agent when you are not using them to limit the exposure. Deleting keys will prevent the cron job from being able to back up the system.

ssh-agent -D
All identities removed.

With the preliminaries of ssh done and ready to go we can now use it as a restic transport.

Running Restic

Backup Encrypt Key

Restic is an encrypted backup. Encryption requires an encryption key. This should be long and strong. Since we are not going to be typing this in let's make it full random and quite long. I am going to use the "pwgen" utility to create a long password and to store it in a read protected password file.

mkdir -p ~/etc/restic
touch ~/etc/restic/password
chmod go-rw ~/etc/restic/password
pwgen -s 32 1 > ~/etc/restic/password
ls -ld ~/etc/restic/password
-rw------- 1 rwp rwp 33 Nov 14 11:27 /home/rwp/etc/restic/password

Here I ensured that the directory I chose to hold these files existed. I created the file and ensured that it could not be read by anyone other than my account id. Then I stored one 32-bit random character password in the file. Even I the creator of it don't know what the password is that is stored in that file. I can look. Feel free to peek.

DO NOT LOSE THE PASSWORD! Let me repeat this again. DO NOT LOSE THE PASSWORD! If you lose the password you lose access to the encrypted backups. Copy that password off to a secure but safe location. Do it NOW. This is of ultimate importance to have available in order to recover files from the backup storage.

I am going to backup to another one of my servers. This uses ssh's sftp protocol. I am going to put it in my home directory under a directory called "backup".

cat etc/restic/repository
sftp:rwp@havoc:backup

This is too simple of an example. Here is another with a little more full paths.

cat etc/restic/repository
sftp:rwp@havoc.proulx.com:/srv/backup/myrwpbackupdir

Exclude List

Every backup almost always needs a list of exclusions to avoid backing up. In my case I have several. I have a directory "isos" containing ISO installation image files that are very large and not needed to back up since I downloaded them from the net and could do so again. I have a "tmp" directory with junk and scratch files that I expect to be ephemeral and I don't want to be in a backup. I have compiled object files and desktop cache files that again need to be excluded. Exclude emacs backup files, vim backup files, whatever is needed. Create a file ~/etc/restic/excludelist with these types of content exclusion patterns. Adjust as needed for you.

#*
*.[ao]
*.pyc
*.swp
.cache
.mozilla
isos

Remote Backup Repository

Decide where the backup is going to be stored. I am going to store my laptop backup in my home directory on my main Internet server. My server is named "havoc". Let's initialize the repository.

RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
  RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
    restic init

I am showing this on multiple lines with backslashes to continue the lines. But logically that is one command line. But one line would be very long in this documentation. Set those two environment variables for the command, then call the restic command, give the command the "init" action.

If you are using csh or tcsh then you will already know that you need to use the "env" command to pass those environment variables. This is why you see "env" used so often in documentation since it works with all shells.

...for use with csh/tcsh command shell users...
rwp@ennui:~% env RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
  RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
    restic init
...or anyone really...it works with all shells...

This passes those files to the restic command through the environment. Since backup may be running for a long time any command line arguments would show up in the "ps" process status listing. But by passing it through the environment that is avoided. None of this is particularly secret information, I am telling you to do it in this documentation, but it feels better to not put it on display.

Running the initialization will produce this output.

rwp@ennui:~$ RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
  RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
    restic init
created restic repository 9f28613e83 at 

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

It worked. All good. Let's poke at it.

rwp@ennui:~$ RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
  RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
    restic snapshots
repository 9f28613e opened (repository version 2) successfully, password is correct
created new cache in /home/rwp/.cache/restic

We listed the backed up snapshots but there were none to list. The output is empty. It also notes that it created a new cache file. Restic saves local cache information in ~/.cache/restic/* and that directory might get very large. It's a cache and can be removed at any time to empty out the space it is using. Restic will start a new cache on the next invocation.

Let's create a backup. Let's use "time" to see how long it will take from here to there over the WiFi network and off to the remote site. Here I add the --exclude option argument to include the exclude list. And also the -x, --one-file-system option to exclude other file systems. The -x option is just a good habit to always use in case there are mount points, such as when root is backing up "/" and then we would not want to be backing up /proc, /sys, /dev, and so forth.

rwp@ennui:~$ time RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
  RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
    restic backup --exclude=$HOME/etc/restic/excludelist -x $HOME
repository 9f28613e opened (repository version 2) successfully, password is correct
no parent snapshot found, will read all files

Files:        1080 new,     0 changed,     0 unmodified
Dirs:          343 new,     0 changed,     0 unmodified
Added to the repository: 111.499 MiB (82.401 MiB stored)

processed 1080 files, 122.715 MiB in 0:23
snapshot aca4f79c saved

real    0m24.050s
user    0m5.145s
sys     0m0.847s

This was the first backup. It took 24 seconds to copy 122MB of files. It created the remote backup repository. We can look at it now and see what is stored there and how.

rwp@havoc:~$ du -sh backup
84M     backup

rwp@havoc:~$ find backup -type f -ls
   135056      4 -rw-------   1 rwp      rwp           199 Nov 14 14:57 backup/snapshots/aca4f79cb96e73e0a239931eccdd3db54d5a2a16b301e9fd15e11d17698f6693
   135047      4 -rw-------   1 rwp      rwp           155 Nov 14 14:52 backup/config
   135046      4 -rw-------   1 rwp      rwp           446 Nov 14 14:52 backup/keys/43358cd08e96f8c6bc68e5ba39bff7629e26981bb556533477094df0d516a840
   135055     60 -rw-------   1 rwp      rwp         60161 Nov 14 14:57 backup/index/acee3c779d1e7850a6fe103d1eb9bb74ccfc3a8d459e0f53d20c5b020cb779f1
   135049  17052 -rw-------   1 rwp      rwp      17454492 Nov 14 14:57 backup/data/ca/ca0288c61116297c6fa1e764bcd49a73492b8a9543e231e14615dd7ee0098df8
   135052  17100 -rw-------   1 rwp      rwp      17503764 Nov 14 14:57 backup/data/a2/a2b23e697947406d4ed2bec00d197303f99bdf704fb167ad11c3e40f5218550b
   135050  17108 -rw-------   1 rwp      rwp      17514402 Nov 14 14:57 backup/data/59/591f48ccaa81a730d832a6cadb963b9bce1428f18469a9c820ac9c6e69b3409d
   135054  16260 -rw-------   1 rwp      rwp      16645414 Nov 14 14:57 backup/data/c8/c893a8321c63fb7422260ab09352b616851cec09a4016aac32fb42e502815d67
   135053    164 -rw-------   1 rwp      rwp        163895 Nov 14 14:57 backup/data/8b/8bdcce0cca7da0f3f886dad0ca1b94a3966ab01d3bd53f63d3c3d4ba54c9113b
   135051  16728 -rw-------   1 rwp      rwp      17122133 Nov 14 14:57 backup/data/41/41010f4c4e68bafaece94868f2bf0275e017c7c5f00fcacfdbc5b085e9e85ee2

rwp@havoc:~$ file backup/data/41/41010f4c4e68bafaece94868f2bf0275e017c7c5f00fcacfdbc5b085e9e85ee2
backup/data/41/41010f4c4e68bafaece94868f2bf0275e017c7c5f00fcacfdbc5b085e9e85ee2: data

It's just binary blob data files. No hint at all about what is in them. Compressed and de-duplicated. It's smaller than the original source files.

We can use restic to list out the snapshots.

rwp@ennui:~$ RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
  RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
    restic snapshots
repository 9f28613e opened (repository version 2) successfully, password is correct
ID        Time                 Host        Tags        Paths
----------------------------------------------------------------
aca4f79c  2023-11-14 14:57:27  ennui                   /home/rwp
----------------------------------------------------------------
1 snapshots

There is one snapshot. The one we just created. The backup repository lists the host it came from because we can store backups from multiple different client hosts in the same backup repository and it will compress and de-duplicate across all of the backups. This allows a similar home directory and multiple machines to be backed up but to take up almost no additional space.

Helper Scripts

Is anyone tired of tediously needing to supply those environment variables each and every time? I know I am. Therefore I create a small helper script wrapper restic-wrap to supply them for me. Using the wrapper means I don't need to remember those options.

I use this restic-wrap script to make this much easier. I then just call restic-wrap whenever I want to call the restic program.

#!/bin/sh

# This restic program wrapper supplies the password and the repository
# location which is always needed every time this program is called.

# Copyright 2023 Bob Proulx <bob@proulx.com>
#
# Copying and distribution of this file, with or without modification,
# are permitted in any medium without royalty provided the copyright
# notice and this notice are preserved.  This file is offered as-is,
# without any warranty.
#
# Written by Bob Proulx <bob@proulx.com>

# This method of passing these through the environment was chosen to
# avoid ever having them listed on the command line and therefore
# possibly showing up in a ps process status listing.  Just by
# themselves these are not security risks but following the need to
# know policy there is no need for anyone to know this from a ps
# listing and it is easy enough to avoid that by using the environment
# to pass this information.

RESTIC_PASSWORD_FILE=$HOME/etc/restic/password \
    RESTIC_REPOSITORY_FILE=$HOME/etc/restic/repository \
        exec restic "$@"

And the same thing using a restic-backup-wrap script for restic backup which requires the also long --exclude=$HOME/etc/restic/excludelist -x options.

#!/bin/sh

# This restic-backup program wrapper supplies the options that are
# always needed for the backup operation and then calls the main
# restic via its wrapper which suppliest the needed options there.

# Copyright 2023 Bob Proulx <bob@proulx.com>
#
# Copying and distribution of this file, with or without modification,
# are permitted in any medium without royalty provided the copyright
# notice and this notice are preserved.  This file is offered as-is,
# without any warranty.
#
# Written by Bob Proulx <bob@proulx.com>

exec restic-wrap backup --exclude-file=$HOME/etc/restic/excludelist -x "$@"

This restic-backup-wrap script calls restic-wrap too so that options are only ever written down in one place. That keeps things "DRY" as in "Don't Repeat Yourself".

With those helper wrappers in place it is less tedious to perform a backup.

rwp@ennui:~$ time restic-backup-wrap $HOME
repository 9f28613e opened (repository version 2) successfully, password is correct
using parent snapshot aca4f79c

Files:           0 new,     1 changed,  1079 unmodified
Dirs:            0 new,     2 changed,   341 unmodified
Added to the repository: 16.242 KiB (4.057 KiB stored)

processed 1080 files, 122.715 MiB in 0:00
snapshot 46278ec8 saved

real    0m1.750s
user    0m0.667s
sys     0m0.114s

Since there is now a parent future backups are incremental backups and much faster. This fresh incremental backup took only 1.75 seconds. If I want it to be more quiet than this add the -q quiet option. This makes backup fast and quiet. If there is an error that will interrupt the silence and produce an error message.

rwp@ennui:~$ restic-backup-wrap -q $HOME
rwp@ennui:~$ echo $?
0

rwp@ennui:~$ restic-wrap -q snapshots
ID        Time                 Host        Tags        Paths
----------------------------------------------------------------
aca4f79c  2023-11-14 14:57:27  ennui                   /home/rwp
46278ec8  2023-11-14 15:40:06  ennui                   /home/rwp
f516f3e7  2023-11-14 15:41:04  ennui                   /home/rwp
----------------------------------------------------------------
3 snapshots

After doing this a few times I have several snapshots. The backup is compressed and de-duplicated and takes up only incremental space. Let's look at what is there.

rwp@ennui:~$ restic-wrap -q ls f516f3e7 | head
/home
/home/rwp
/home/rwp/.ICEauthority
/home/rwp/.Xauthority
/home/rwp/.Xresources
/home/rwp/.bash_history
/home/rwp/.bashrc
/home/rwp/.config

Needing to always find the latest id would be tedious. The "latest" tag is always available. I normally use it.

rwp@ennui:~$ restic-wrap -q ls latest | grep /home/rwp/tmp/
...
/home/rwp/tmp/base.txz
...

rwp@ennui:~$ restic-wrap -q ls -l latest | grep /tmp/ |grep base
-rw-r--r--  1000  1000 38371328 1969-12-31 17:00:00 /home/rwp/tmp/base.txz

Oh! I forgot to include tmp in the exclude list. (Intentionally for this document I will add.) Let's add it to the exclude list. Back up again. And see about the changes. I'll exclude any directory named "tmp". Which might be more than you want. For this doc I will exclude just the top level "tmp" directory.

rwp@ennui:~$ echo /home/rwp/tmp >> ~/etc/restic/excludelist

rwp@ennui:~$ restic-backup-wrap $HOME
repository 9f28613e opened (repository version 2) successfully, password is correct
using parent snapshot b1625bef

Files:           0 new,     1 changed,  1079 unmodified
Dirs:            0 new,    11 changed,   332 unmodified
Added to the repository: 18.432 KiB (5.201 KiB stored)

processed 1080 files, 122.715 MiB in 0:00
snapshot c4405daa saved

restic-wrap diff b1625bef c4405daa | less
...
-    /home/rwp/tmp/base.txz
...

And now we can see that it has been removed from the most recent backup snapshot. But it is still in the previous backup snapshots and taking up disk space. Which is small in this case but what if it were large? We would tell restic to "forget" that snapshot. I have several and am going to delete several of them in the same action.

rwp@ennui:~$ restic-wrap forget aca4f79c 46278ec8 f516f3e7 b1625bef
repository 9f28613e opened (repository version 2) successfully, password is correct
[0:00] 100.00%  4 / 4 files deleted

rwp@ennui:~$ restic-wrap snapshots
repository 9f28613e opened (repository version 2) successfully, password is correct
ID        Time                 Host        Tags        Paths
----------------------------------------------------------------
c4405daa  2023-11-14 15:49:16  ennui                   /home/rwp
----------------------------------------------------------------
1 snapshots

Pruning Snapshots

Initially I suggest just getting backups going and not worry about pruning. Incremental snapshots are very light weight. You can probably backup many snapshots before needing to prune. And having backup is so important that good just to have them. But eventually you will need to start pruning. Just demonstrated was the "forget" snapshot action. This action has many useful pruning options.

I might suggest an expiration schedule like this. The "keep" options keep the number specified within that duration. I might have 20 in the last hour but --keep-hourly=4 would keep 4 of them and prune the rest.

restic-wrap forget -n --keep-last=5 --keep-hourly=4 --keep-daily=7 --keep-weekly=4 --keep-monthly=6

Always test using the "-n" "not-really" option first in order to see what it would do before it would do it.

rwp@ennui:~$ restic-wrap forget -n --keep-last=2
repository 9f28613e opened (repository version 2) successfully, password is correct
Applying Policy: keep 2 latest snapshots
keep 2 snapshots:
ID        Time                 Host        Tags        Reasons        Paths
-------------------------------------------------------------------------------
f489ae5e  2023-11-14 16:07:00  ennui                   last snapshot  /home/rwp
183d33c9  2023-11-14 16:07:04  ennui                   last snapshot  /home/rwp
-------------------------------------------------------------------------------
2 snapshots

remove 2 snapshots:
ID        Time                 Host        Tags        Paths
----------------------------------------------------------------
c4405daa  2023-11-14 15:49:16  ennui                   /home/rwp
9df17f47  2023-11-14 16:06:57  ennui                   /home/rwp
----------------------------------------------------------------
2 snapshots

Would have removed the following snapshots:
{9df17f47 c4405daa}

Running Periodically With Cron

I use this restic-backup-cron script to run the backup periodically from cron.

#!/bin/sh

# This restic-backup-cron script is called from cron on a regular
# basis to backup files from here to there.

# Copyright 2023 Bob Proulx <bob@proulx.com>
#
# Copying and distribution of this file, with or without modification,
# are permitted in any medium without royalty provided the copyright
# notice and this notice are preserved.  This file is offered as-is,
# without any warranty.
#
# Written by Bob Proulx <bob@proulx.com>

whoami=$(whoami)
lock=/tmp/restic-backup-cron-$whoami.lock
mailto=$whoami

# Check for stale locks.
if [ -e "$lock" ]; then
    if [ -n "$(find "$lock" -mtime +1 -print 2>/dev/null)" ]; then
        : Older lock file found.  Must be stale.  Complain.
        echo "Old lock found.  Stale?" 1>&2
        ls -ld "$lock" 1>&2
        exit 1
    fi
fi

# Atomic test-and-set operation 'mkdir' used as a semaphore.
if ! mkdir "$lock" 2>/dev/null; then
    : Another process got the semaphore and is now running.  They run, we exit.
    exit 0
fi
: "debug: Semaphore $lock created."

# On exit remove the flag saying we are running.

unset tmpfile
cleanup() {
    rmdir "$lock"
    test -n "$tmpfile" && rm -f "$tmpfile" && unset tmpfile
}
sighandler() {
    echo "$progname: signal $1 received, cleaning up." 1>&2
    cleanup
    trap - $1                 # reset to default
    kill -$1 $$               # exit due to signal
}
trap 'cleanup' EXIT
trap 'sighandler HUP' HUP
trap 'sighandler INT' INT
trap 'sighandler QUIT' QUIT
trap 'sighandler TERM' TERM

tmpfile=$(mktemp) || exit 1

exec </dev/null >"$tmpfile" 2>&1

# Source in the ssh keys using keychain.  Only the user has
# permission to read these files.
hostname=$(hostname)            # Possible FQDN here.
host=${hostname%%.*}            # Just the short name.
if [ -f $HOME/.keychain/$host-sh ]; then
    . $HOME/.keychain/$host-sh
fi

# Test that we have ssh keys available.
if ! ssh-add -ql >/dev/null 2>&1; then
    # If we do not then do not try to back up.  Exit.  Try again next time.
    exit 0
fi

# This backs up each of the named directories.  The files to be backed
# up must all be readable by the user running this.

restic-backup-wrap -q $HOME

exec >/dev/null 2>&1            # close previous output file

if [ -s "$tmpfile" ]; then
    # There was output.  Mail it.
    mailx -s "$(hostname): restic backup output" "$mailto" < "$tmpfile"
fi

# Cron scripts should always exit success even if there was a failure
# because cron is not equipped to deal with it other than email the
# caller and we already emailed the responsible party in the above.
exit 0

This script has several features that are useful and important for periodically invoked processes. Let's take a look at a few of them.

First this script makes a semaphore and exits if another version of the program is still running. Why is that important? It's important because sometimes programs in general can take much longer than expected. Sometimes programs get "stuck". If that ever happens and the program is being repeatedly invoked then each invocation will stack up an additional program. I have seen systems pile up literally tens and hundreds of stuck processes! Fortunately when the program is actually stuck it is usually not consuming heavy resources and the system will mostly continue.

But if it is simply a heavy program such as a backup running for hours and hours it would be very bad if another one were to start before the previous one finished! Therefore it is important to make sure there is a singleton semaphore to prevent multiple copies from running. In the case of cron invocation it is usually best if the duplicate invocations simply exit and let the original one complete.

I prefer not to print to stdout and then have cron email the result. The subject of the email from cron is simply that it is cron and this makes the email less obvious than if it were a direct email. Therefore I gather up all of the output into a temporary file and then mail it if there is something to mail.

Using a temporary file means that signals should be trapped and handled such that the temporary file will be cleaned up. Let's not leave scraps behind. Also the temporary file and the semaphore is placed in /tmp which on most systems is cleaned to empty on a reboot. Therefore if the system is rebooted this cleans up everything starting things off in a clean state.

Most importantly for the way I am using restic as the non-root user id the program must have access to the user's ssh keys while running from cron. This is not possible by default. This is where "keychain" comes into use. The script sources the user's keychain file to get connected up to the user's running ssh-agent with unlocked ssh keys available for use. If there are no keys available for use then the script exits silently no error. But if the keys are available then it will proceed with the backup.

If this is a desktop system there is no reason not to run this incremental backup quite often. Daily certainly. Hourly would be reasonable. Even every half hour would be little resource use. On a mobile laptop though perhaps something somewhat less often due to the lower bandwidth nature of WiFi connectivity. Perhaps hourly?

Here is a possible crontab entry to run this command every half hour. Adjust this as per your own sensibilities.

# The default vixie-cron PATH is "/usr/bin:/bin", overriding the environment.
PATH = "/usr/local/sbin:/sbin:/usr/sbin:/usr/local/bin:/bin:/usr/bin"
# Setting PATH is a vixie-cron extension.
# Variables are not expanded.  Quotes are optional.

# field descriptions
# minute (0-59),
#   hour (0-23),
#     day of the month (1-31),),
#       month of year (1-12),
#         day of the week (0-6 with 0=Sunday).

# This runs every hour, as the byrnes user.
7,37 * * * *      test ! -x /home/$USER/bin/restic-backup-cron || /home/$USER/bin/restic-backup-cron

Restoring From Backup

Some useful commands to look at the backup.

rwp@ennui:~$ restic-wrap -q ls latest | less

rwp@ennui:~$ restic-wrap -q ls -l latest | less

I find "diff" to be quite interesting. See what's changed. Lots of potential for good use from it.

rwp@ennui:~$ restic-wrap diff c4405daa 183d33c9
comparing snapshot c4405daa to 183d33c9:

M    /home/rwp/.bash_history
M    /home/rwp/.lesshst

Files:           0 new,     0 removed,     2 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:      2 new,     2 removed
Tree Blobs:      3 new,     3 removed
  Added:   44.800 KiB
  Removed: 40.770 KiB

The official way to restore is the "restore" action. It's a little tedious. It requires a path upon which to restore, because most often we restore to a new location and then manually check differences and then merge things in. That's done very often. But it can also be restored in place by giving a target location of "/". And then if you don't want a complete restoration of EVERYTHING one limits it with an "--include" filter. Let's restore "scratch" into /tmp first.

rwp@ennui:~$ restic-wrap restore latest -t /tmp --include /home/rwp/scratch
repository 9f28613e opened (repository version 2) successfully, password is correct
restoring <Snapshot 183d33c9 of [/home/rwp] at 2023-11-14 16:07:04.018696527 -0700 MST by rwp@ennui> to /tmp

This restores scratch with the entire hierarchy to /tmp.

rwp@ennui:~$ find /tmp/home -type f -ls
   227159     32 -rw-rw-r--   1 rwp      rwp         28706 Nov 10 00:10 /tmp/home/rwp/scratch

This restores scratch on top of the original location.

restic-wrap restore latest -t / --include /home/rwp/scratch

Sometimes it is just easier to dump out a single file though.

restic-wrap dump latest /home/rwp/.bashrc | less
restic-wrap dump latest /home/rwp/.bashrc > /tmp/foo
restic-wrap dump latest /home/rwp/.bashrc > ~/.bashrc

If "dump" is asked to dump a directory tree then that tree will be dumped as a "tar" archive.

Integrity Checking

Backups that are corrupted are a problem. Backups that are corrupted silently without anyone knowing are a tragedy! In order to ensure that the backups are working okay it is good to run a check on them periodically.

rwp@ennui:~$ restic-wrap check
using temporary cache in /tmp/restic-check-cache-642804176
repository 9f28613e opened (repository version 2) successfully, password is correct
created new cache in /tmp/restic-check-cache-642804176
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  4 / 4 snapshots
no errors were found

This should also be run periodically by cron. I'll put an example here at some point but at the moment I am simply trying to finish this article before presenting it at the local user group meet-up.

Additionally backups that are not running are another tragedy waiting to happen. One goes to restore from backup and find out that the most recent backup was six months ago. That's bad. On the server side of things if possible it is good to put a check in place to verify that something is happening over there.

#!/bin/sh
...
if [ -n "$(find $HOME/backup/snapshots -maxdepth 0 -mtime +3 -print)" ]; then

This is a simple brute force check to see that the snapshots directory is getting update. This only works if there is only one client storing backup snapshots there. But if so then something like this is reasonable. If not then the snapshots will give a false indication.

Better might be to run restic snapshots and verify that the latest snapshot is recent, at least within a threshold of days.

restic-wrap snapshots | check-for-latest

Due to time constraints this too is left as an exercise for the reader.

Conclusion

I hope you found this quick guide to restic useful. I wrote it quickly in order to be able to talk about it at the local user group meet-up which is happening now withing the next hour. I hope to see you there!

License

Copyright (C) 2023 Bob Proulx

Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.