Metadata-Version: 1.1
Name: backy
Version: 2.2
Summary: Block-based backup and restore utility for virtual machine images
Home-page: https://bitbucket.org/flyingcircus/backy
Author: Christian Theune <ct@flyingcircus.io>, Christian Kauhaus <kc@flyingcircus.io>, Daniel Kraft <daniel.kraft@d9t.de>
Author-email: mail@flyingcircus.io
License: GPL-3
Description: ========
        Overview
        ========
        
        Backy is a block-based backup and restore utility for virtual machine images.
        
        Backy is intended to be:
        
        * space-, time-, and network-efficient
        * trivial to restore
        * reliable.
        
        To achieve this, we rely on:
        
        * space-efficient storages (CoW filesystems, content-hashed chunking)
        * using a snapshot-capable source for our volumes (i.e. Ceph RBD)
          that allows easy extraction of changes between snapshots,
        * leverage proven, existing low-level tools,
        * keep the code-base small, simple, and well-tested.
        
        We also have a few ground rules for the implementation:
        
        * VM data is stored self-contained on the filesystem and can be
          moved between servers using regular FS tools like copy, rsync, or such.
        
        * No third party daemons are required to interact with backy: no database
          server. The scheduler daemon is only responsible for scheduling and simply
          calls regular CLI commands to perform a backup. Backy may interact with
          external daemons like Ceph or Consul, depending on the source storage
          implementation.
        
        
        Operations
        ==========
        
        Full restore
        ------------
        
        The most important question is: I screwed up -- how do I get my data back?
        
        Here's the fast answer to make a full restore of the most recent backup::
        
           $ cd /srv/backy/my-virtual-machine
           $ dd if=last of=/srv/kvm/my-virtual-machine.img bs=4096k
        
        In case Ceph is used for image storage, just import the image directly::
        
           $ rbd import last rbd/my-volume
        
        If you like to pick a specific version, it's only a little more effort::
        
           $ backy status
           +---------------------+------------+------------+---------+--------------+
           | Date                | ID         |       Size |   Durat | Tags         |
           +---------------------+------------+------------+---------+--------------+
           | 2015-11-04 11:09:26 | UT7PkENubw |  60.00 GiB | 845.0 s | weekly,daily |
           | 2015-11-05 10:32:03 | fPnbSvEHHy | 264.85 MiB |  88.1 s | daily        |
           | 2015-11-06 10:32:03 | cErS5GJ5sL | 172.34 MiB |  84.5 s | daily        |
           +---------------------+------------+------------+---------+--------------+
           3 revisions containing 60.43 GiB data (estimated)
           $ dd if=fPnbSvEHHymfztN9FuegLQ of=/srv/kvm/my-virtual-machine bs=4096k
        
        Or try `backy find` to locate the latest backup for a specific tag::
        
           $ rbd import $(backy find -r weekly) rbd/my-volume
        
        
        Restoring individual files
        --------------------------
        
        The image files are exact copies of the data from the virtual disks. You can use
        regular Linux tools to interact with them::
        
            $ kpartx -av last
            add map loop0p1 (253:9): 0 41934815 linear /dev/loop0 8192
            $ mkdir /mnt/restore
            $ mount -o ro /dev/mapper/loop0p1 /mnt/restore
            $ cd /mnt/restore
            $ ls
            bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run
            sbin  srv  sys  tmp  usr  var
        
        To clean up::
        
            $ umount /mnt/restore
            $ kpartx -d last
        
        
        Setting up backy
        ----------------
        
        #. Create a sufficiently large backup partition using a COW-capable filesystem
           like btrfs and mount it under `/srv/backy`.
        
        #. Create a configuration file at `/etc/backy.conf`. See man page for details.
        
        #. Start the scheduler with your favourite init system::
        
              backy -l /var/log/backy.log scheduler -c /path/to/backy.conf
        
           The scheduler runs in the foreground until it is shot by SIGTERM.
        
        #. Set up monitoring using `backy check`.
        
        #. Set up log rotation for `/var/log/backy.conf` and `/srv/backy/*/backy.log`.
        
        The file paths given above match the built-in defaults, but paths are fully
        configurable.
        
        
        Features
        ========
        
        Telnet shell
        ------------
        
        Telnet into localhost port 6023 to get an interactive console. The console can
        currently be used to inspect the scheduler's live status.
        
        
        Self-check
        ----------
        
        Backy includes a self-checking facility. Invoke `backy check` to see if there is
        a recent revision present for all configured backup jobs::
        
           $ backy check
           OK: 9 jobs within SLA
        
        Both output and exit code are suited for processing with Nagios-compatible
        monitoring systems.
        
        
        Pluggable backup sources
        ------------------------
        
        Backy comes with a number of plug-ins which define block-file like sources:
        
        - **file** extracts data from simple image files living on a regular file
          system.
        - **ceph-rbd** pulls data from RBD images using Ceph features like snapshots.
        - **flyingcircus** is an extension to the `ceph-rbd` source which we use
          internally on the `Flying Circus`_ hosting platform. It uses advanced features
          like Consul integration.
        
        .. _Flying Circus: http://flyingcircus.io/
        
        It should be easy to write plug-ins for additional sources.
        
        
        Adaptive verification
        ---------------------
        
        Backy always verifies freshly created backups. Verification scale depends on
        the source type: file-based sources get fully verified. Ceph-based sources are
        verified based on random samples for runtime reasons.
        
        Zero-configuration scheduling
        -----------------------------
        
        The backy scheduler is intended to run continuously. It will spread jobs
        according to the configured run intervals over the day. After resuming from an
        interruption, it will reschedule missed jobs so that SLAs are still kept if
        possible.
        
        Backup jobs can be triggered at specific times as well: just invoke `backy
        backup` manually.
        
        
        Authors
        =======
        
        * Christian Theune <ct@flyingcircus.io>
        * Christian Kauhaus <kc@flyingcircus.io>
        * Daniel Kraft <daniel.kraft@d9t.de>
        
        
        License
        =======
        
        GPLv3
        
        
        Links
        =====
        
        * `Bitbucket repository <https://bitbucket.org/flyingcircus/backy>`_
        * `PyPI page <https://pypi.python.org/pypi/backy>`_
        * `Online docs <http://pythonhosted.org/backy/>`_
        * `Build server <https://builds.flyingcircus.io/job/backy/>`_
        
        .. vim: set ft=rst spell spelllang=en sw=3:
        
        =========
        Changelog
        =========
        
        2.2 (2017-08-28)
        ================
        
        - Introduce a new backend storage mechanism, independent of BTRFS: instead of
          using COW, use a directory with content-hashed 4MiB chunks of the file.
          Deduplication happens automatically based on the 4MiB chunks.
        
        - Made the use of fadvise features more opportunistic: don't fail if they are
          not supported by the backend as they are only optimizations anyway.
        
        - Introduce an exponential backoff after a failed backup: instead of retrying
          quickly and thus hogging the queue (in the case that timeouts are involved)
          we now backoff exponentially starting with 2 minutes, then 4, then 8, ...
          until we reach 6 hours as the biggest backoff time.
        
          You can still trigger an explicit run using the telnet "run" command for
          the affected backup. This will put the backup in the run queue immediately
          but will not reset the error counter or backoff interval in case it
          fails again.
        
        - Performance improvement on restore: don't read the restore target. We don't
          have to optimize CoW in this case. (#28268)
        
        2.1.5 (2016-07-01)
        ==================
        
        - Bugfix release: fix data corruption bug in the new full-always mode. (FC
          #21963)
        
        
        2.1.4 (2016-06-20)
        ==================
        
        - Add "full-always" flag to Ceph and Flyingcircus sources. (FC #21960)
        
        - Rewrite full backup code to make use of shallow copies to conserve disk space.
          (FC #21960)
        
        
        2.1.3 (2016-06-09)
        ==================
        
        - Fix new timeout to be 5 minutes by default, not 5 days.
        
        - Do not sort blocks any longer: we do not win much from seeking
          over volumes with random blocks anyway and this helps for a more
          even distribution with the new timeout over multiple runs.
        
        
        2.1.2 (2016-06-09)
        ==================
        
        - Fix backup of images containing holes (#33).
        
        - Introduce a (short) timeout for partial image verification. Especially
          very large images and images that are backed up frequently do not profit
          from running for hours to verify them, blocking further backups.
          (FC #21879)
        
        2.1.1 (2016-01-15)
        ==================
        
        - Fix logging bugs.
        
        - Shut down daemon loop cleanly on signal reception.
        
        
        2.1 (2016-01-08)
        ================
        
        - Add optional regex filter to the `jobs` command in the telnet shell.
        
        - Provide list of failed jobs in check output, not only the total number.
        
        - Add `status-interval`, `telnet-addrs`, and `telnet-port` configuration
          options.
        
        - Automatically recover from missing/damaged last or last.rev symlinks (#19532).
        
        - Use `{BASE_DIR}/.lock` as daemon lock file instead of the status file.
        
        - Usability improvements: count jobs, more informative log output.
        
        - Support restoring to block special files like LVM volumes (#31).
        
        
        2.0 (2015-11-06)
        ================
        
        - backy now accepts a `-l` option to specify a log file. If no such option is
          given, it logs to stdout.
        
        - Add `backy find -r REVISION` subcommand to query image paths from shell scripts.
        
        - Fix monitoring bug where partially written images made the check go green
          (#30).
        
        - Greatly improve error handling and detection of failed jobs.
        
        - Performance improvement: turn off line buffering in bulk file operations
          (#20).
        
        - The scheduler reports child failures (exit status > 0) now in the main log.
        
        - Fix fallocate() behaviour on 32 bit systems.
        
        - The `flyingcircus` source type now requires 3 arguments: vm, pool, image.
        
        
        2.0b3 (2015-10-02)
        ==================
        
        - Improve telnet console.
        
        - Provide Nix build script.
        
        - Generate `requirements.txt` automatically from buildout's `versions.cfg`.
        
        
        2.0b2 (2015-09-15)
        ==================
        
        - Introduce scheduler and rework the main backup command. The backy
          command is now only responsible for dealing with individual backups.
        
          It does no longer care about scheduling.
        
          A new daemon and a central configuration file is responsible for that
          now. However, it simply calls out to the existing backy command
          so we can still manually interact with the system even if we do not
          use the daemon.
        
        - Add consul integration for backing up Flying Circus root disk images
          with clean snapshots (by asking fc.qemu to use fs-freeze before preparing
          a Ceph snapshot).
        
        - Switch to shorter UUIDs. Existing files with old UUIDs are compatible.
        
        - Turn the configuration format into YAML. Old files are still compatible.
          New configs will be generated as YAML.
        
        - Performance: defrag all new files automatically to avoid btrfs degrading
          extent performance. It appears this doesn't completely duplicate all CoW
          data. Will have to monitor this in the future.
        
        2.0b1 (2014-07-07)
        ==================
        
        - Clean up docs.
        
        - Add classifiers in setup.py.
        
        - More or less complete rewrite expecting a copy-on-write filesystem as the
          target.
        
        - Flexible backup scheduling using free-form tags.
        
        - Compatible with Python 3.2-3.4.
        
        - Initial open source import as provided by Daniel Kraft (D9T).
        
        .. vim: set ft=rst spell spelllang=en:
        
Keywords: backup
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Topic :: System :: Archiving :: Backup
