How to Implement your own DirectoryStorage Backup Procedure

Taking a backup of a DirectoryStorage consists of forcing the storage into snapshot mode, then taking a copy of the files in the database directory. The essential directories are HOME/config, and HOME/A.

While in snapshot mode the storage still has write access to the database; writes go into the directories HOME/journal and HOME/B. You do not need to take copes of these in your backup.

Note that DirectoryStorage now has a standard backup tool (see doc/backup.txt).

To implement your own full backup procedure:

  1. Take a copy of the HOME/config directory. This directory is small, and is not modified by a running DirectoryStorage.
  2. If you know the storage is running, force it into snapshot mode. Lock the file HOME/misc/sublock to ensure that noone else will modify the directory during your backup, then check for the existense of a file HOME/misc/snapshot to prove that the database really is a snapshot. For more details on this process, see doc/snapshot.txt.
  3. Take a full copy of the HOME/A directory. This directory contains the full database content, complete as of the time where it entered snapshot mode.
  4. DO NOT backup the directories HOME/B or HOME/journal. These directories contain data from transactions that were committed after entering snapshot mode.
  5. Unlock HOME/misc/sublock and leave snapshot mode.

To implement your own incremental live backup:

  1. If your incremental backup system uses timestamps, record the time now. You will need it to perform your next incremental backup.
  2. Force it into snapshot mode, as described above.
  3. Take an incremental copy of HOME/A using the last-modified time, or other means. An incremental backup need not record which files have been deleted.
  4. Leave snapshot mode

Shutdown Storages

It is possible for a storage to be shut down and the file misc/snapshot does not exist. In this state the main directory is not a coherent snapshot. This can happen if:

  • The storage process terminates without calling .close()
  • The storage process calls .close() but the storage can not enter snapshot mode promptly. This may happen if it is closed immediately after leaving snapshot mode.
  • The quick_shutdown configuration option has been turned on.

When the DirectoryStorage is shut down in this state there are two options for taking a backup:

  • Start the storage. The storage will roll forwards any complete transactions, and you can then proceed as normal. This is the easiest approach.
  • Take a copy of everything, including HOME/B and HOME/journal. Note that this option is incompatible with incremental backups. This option is appropriate if something has crashed and your want a copy of the wreckage. You can only do this if DirectoryStorage is shut down, or you are using a system like LVM to give a filesystem-level snapshot.