github-backup

backs up everything github knows about a repository, to the repository https://github-backup.branchable.com/

Version on this page:1.20160522
LTS Haskell 6.27:1.20160522
Stackage Nightly 2016-05-25:1.20160522
Latest on Hackage:1.20170301
GPL-3 licensed by Joey Hess
Maintained by Joey Hess

Module documentation for 1.20160522

There are no documented modules for this package.

github-backup is a simple tool you run in a git repository you cloned from GitHub. It backs up everything GitHub publishes about the repository, including branches, tags, other forks, issues, comments, wikis, milestones, pull requests, watchers, and stars.

Installation

First install Haskell's stack tool. For example, on a Debian system:

sudo apt-get install haskell-stack

Then to build and install github-backup:

git clone https://git.joeyh.name/git/github-backup.git
cd github-backup
stack install

There is also a Makefile, which uses cabal to build, and installs a man page, bash completion file, etc. This is recommended for use when making packages of github-backup.

Use

Run github-backup with no parameters, inside a git repository cloned from GitHub to back up that repository.

Or, if you have a GitHub account, run github-backup username to clone and back up your account's repositories, as well as the repositories you're watching and have starred.

Why backup GitHub repositories

There are a couple of reasons to want to back this stuff up:

  • In case something happens to GitHub. More generally because keeping your data in the cloud and relying on the cloud to back it up is foolish.

  • In case someone takes down a repository that you were interested in. If you run github-backup with your username, it will back up all the repositories you have watched and starred.

  • So you can keep working on your repository while on a plane, or on a remote beach or mountaintop. Just like Linus intended.

What to expect

Each time you run github-backup, it will find any new forks on GitHub. It will add remotes to your repository for the forks, using names like github_torvalds_subsurface. It will fetch from every fork.

It downloads metadata from each fork. This is stored into a branch named "github". Each fork gets a directory in there, like torvalds_subsurface. Inside the directory there will be some files, like torvalds_subsurface/watchers. There may be further directories, like for comments: torvalds_subsurface/comments/1.

You can follow the commits to the github branch to see what information changed on GitHub over time.

The format of the files in the github branch is currently Haskell serialized data types. This is plain text, and readable, if you squint.

Limitations

github-backup is repository-focused. It does not try to back up other information from GitHub. In particular, social network stuff, like users who are following you, is not backed up.

github-backup does not log into GitHub, so it cannot backup private repositories.

Notes added to commits and lines of code don't get backed up yet. There is only recently API support for this.

The labels that can be added to issues and milestones are not backed up. Neither are the hooks. They could be, but don't seem important enough for the extra work involved. Yell if you need them.

github-backup re-downloads all issues, comments, and so on each time it's run. This may be slow if your repo has a lot of them, or even if it just has a lot of forks.

Bear in mind that this uses the GitHub API; don't run it every 5 minutes. GitHub rate limits the API to some small number of requests per hour when used without authentication. To avoid this limit, you can set GITHUB_USER and GITHUB_PASSWORD (or GITHUB_OAUTH_TOKEN obtained from https://github.com/settings/tokens) in the environment and it will log in when making (most) API requests.

Anyway, github-backup does do an incremental backup, picking up where it left off, so will complete the backup eventually even if it's rate limited.

Author

github-backup was written by Joey Hess

It is made possible thanks to:

Changes

github-backup (1.20170301) unstable; urgency=medium

* No longer hosted on Github due to their horrible new TOS.
New site: https://github-backup.branchable.com/

-- Joey Hess <id@joeyh.name> Wed, 01 Mar 2017 17:28:50 -0400

github-backup (1.20170221) unstable; urgency=medium

* Updated to use github-0.15.
* Due to a regression in github-0.15, the pull request list is retrieved
without authentication.
* Various updates to internal git and utility libraries shared
with git-annex.

-- Joey Hess <id@joeyh.name> Tue, 21 Feb 2017 15:33:56 -0400

github-backup (1.20161118) unstable; urgency=medium

* Fix build with recent versions of cabal.

-- Joey Hess <id@joeyh.name> Fri, 18 Nov 2016 12:58:19 -0400

github-backup (1.20161110) unstable; urgency=medium

* Fix build with recent versions of the directory package.
Thanks, James McCoy.
* Various updates to internal git and utility libraries shared
with git-annex.

-- Joey Hess <id@joeyh.name> Thu, 10 Nov 2016 11:55:44 -0400

github-backup (1.20160922) unstable; urgency=medium

* Increase base bounds to 4.8 to get mapM_ etc that works on Vector.
* Recommended build method is now to use stack, for more reliable builds.
* Explicitly list modules and other files in github-backup.cabal.
* The hackage .tar.gz now omits the debian directory and may omit other
source files in the future; the purpose of that tarball is to make
stack/cabal install work, not to be a complete source distribution.
Use git repository to get complete source.
* Various updates to internal git and utility libraries shared
with git-annex.

-- Joey Hess <id@joeyh.name> Mon, 22 Aug 2016 23:57:16 -0400

github-backup (1.20160522) unstable; urgency=medium

* Work around git weirdness in handling of relative path to GIT_INDEX_FILE
when in a subdirectory of the repository.
* Various updates to internal git and utility libraries shared
with git-annex.
* debian: Add lintian overrides for rpath.
* debian: Bump standards-version.
* Makefile: Pass LDFLAGS, CFLAGS, and CPPFLAGS through ghc and on to
ld, cc, and cpp. This lets the Debian package build with various
hardening options. Although their benefit to a largely haskell program
is unknown.

-- Joey Hess <id@joeyh.name> Sun, 22 May 2016 15:24:04 -0400

github-backup (1.20160511) unstable; urgency=medium

* Fix build with directory-1.2.6.2.
* github-backup.cabal: Add Setup-Depends.

-- Joey Hess <id@joeyh.name> Wed, 11 May 2016 13:12:12 -0400

github-backup (1.20160319) unstable; urgency=medium

* Update to github-0.14.1.
Building with old versions is not supported due to the massive API
changes in this version of the github library.
* Added support for oath tokens.

-- Joey Hess <id@joeyh.name> Sat, 19 Mar 2016 14:11:55 -0400

github-backup (1.20160207) unstable; urgency=medium

* Switch to using https instead of git:// for a secure transport.
Note that existing git:// remotes will continue to be used.
Thanks, Robin Schneider.
* Add upper bound on github dependency, as version 0.14 has large API
changes and can't be used yet.
* Various updates to internal git and utility libraries shared
with git-annex.
* Silence build warnings with ghc 7.10.

-- Joey Hess <id@joeyh.name> Sun, 07 Feb 2016 23:29:59 -0400

github-backup (1.20150807) unstable; urgency=medium

* Added bash completion.
* Add --no-forks flag that turns off backing up forks.
Thanks, Phil Ruffwind.
* Avoid nonzero exit due to temporary failures that can be retried next
time github-backup runs, so that it can be used in a cron job with eg,
chronic.

-- Joey Hess <id@joeyh.name> Fri, 07 Aug 2015 12:50:02 -0400

github-backup (1.20150618) unstable; urgency=medium

* Fix broken configure script.

-- Joey Hess <id@joeyh.name> Thu, 18 Jun 2015 16:56:41 -0400

github-backup (1.20150617) unstable; urgency=medium

* Add missing build deps for windows to cabal file.
Thanks, Jeff Segal.
* Various updates to internal git and utility libraries shared
with git-annex.

-- Joey Hess <id@joeyh.name> Wed, 17 Jun 2015 14:44:14 -0400

github-backup (1.20150106) unstable; urgency=medium

* Fix build with process 1.2.1.0.
* Various updates to internal git and utility libraries shared
with git-annex.

-- Joey Hess <id@joeyh.name> Tue, 06 Jan 2015 19:08:08 -0400

github-backup (1.20141222) unstable; urgency=medium

* Added gitriddance(1), a utility to close all issues and pull requests,
for repos that don't want to be bothered with GitHub's proprietary
issue tracker.
* gitriddance depends on github 0.13.1, which has bug fixes
for posting comments.
* Various updates to internal git and utility libraries shared
with git-annex.

-- Joey Hess <id@joeyh.name> Mon, 22 Dec 2014 15:30:08 -0400

github-backup (1.20141204.1) unstable; urgency=medium

* Set myself as maintainer.

-- James McCoy <jamessan@debian.org> Fri, 19 Jun 2015 21:32:18 -0400

github-backup (1.20141204) unstable; urgency=high

* Fix broken argument parser for the username|organization parameter.
Closes: #772043

-- Joey Hess <id@joeyh.name> Thu, 04 Dec 2014 12:29:06 -0400

github-backup (1.20141110) unstable; urgency=medium

* Orphaned the Debian package.
(I continue to maintain github-backup upstream.)

-- Joey Hess <joeyh@debian.org> Mon, 10 Nov 2014 12:19:59 -0400

github-backup (1.20141031) unstable; urgency=medium

* Adjust cabal file for network-uri split.
* Avoid using optparse-applicate's argument combinator, so it will build
with 0.11 and older too.
* Various updates to internal git and utility libraries shared with
git-annex.

-- Joey Hess <joeyh@debian.org> Fri, 31 Oct 2014 11:17:49 -0400

github-backup (1.20140831) unstable; urgency=medium

* Fix build with github 0.10.

-- Joey Hess <joeyh@debian.org> Sun, 31 Aug 2014 15:31:26 -0700

github-backup (1.20140807) unstable; urgency=medium

* Fix build with github 0.9.
* Fix creation of github branch.

-- Joey Hess <joeyh@debian.org> Thu, 07 Aug 2014 22:17:22 -0400

github-backup (1.20140721) unstable; urgency=medium

* Fix typo in fix for url parsing. Closes: #755261

-- Joey Hess <joeyh@debian.org> Mon, 21 Jul 2014 13:10:49 -0400

github-backup (1.20140720) unstable; urgency=medium

* Deal with trailing slashes on github repo urls. Closes: #755261
* Fix bug introduced by change to embedded git libraries
in last release. Closes: #755262

-- Joey Hess <joeyh@debian.org> Sun, 20 Jul 2014 17:20:26 -0400

github-backup (1.20140707) unstable; urgency=medium

* Add --exclude to skip backing up a specific repository
when backing up a user or organization's repositories.
Closes: #754072
* Converted to using optparse-applicative.
* Multiple usernames can now be specified to back up at once.

-- Joey Hess <joeyh@debian.org> Mon, 07 Jul 2014 23:01:03 -0400

github-backup (1.20140704) unstable; urgency=medium

* Avoid making signed commits when committing to the github-backup branch
and the user has commit.gpgsign=true.
Closes: #753720
* Various updates to internal git and utility libraries shared with git-annex.

-- Joey Hess <joeyh@debian.org> Fri, 04 Jul 2014 12:01:11 -0400

github-backup (1.20131203) unstable; urgency=low

* Now also backs up the repos a user has starred, when run with a user's
name.
* Now finds and backs up the parent repository that a repository got forked
from.
* Uses authentication for all API calls.
* Fairer ordering of requests when backing up many repositories at once.
* Avoid making requests for data that has already been backed up until
after new data has been backed up. Handles API rate limiting much better.
Closes: #723859

-- Joey Hess <joeyh@debian.org> Tue, 03 Dec 2013 12:45:18 -0400

github-backup (1.20131101) unstable; urgency=low

* Now also backs up the repos a user is watching, when run with a user's
name. Useful if you want to back up repositories that you have not forked;
just watch them and run github-backup.
* Can now log in to github, to avoid increasingly small API rate limits.
Set GITHUB_USER and GITHUB_PASSWORD environment to enable.
Note that a few api calls don't use authentication; see
https://github.com/fpco/github/issues/40
* Build-Depend on git. Closes: #728481
* Don't include tmp directory in files stored in the github branch.

-- Joey Hess <joeyh@debian.org> Fri, 01 Nov 2013 18:00:16 -0400

github-backup (1.20131006) unstable; urgency=low

* Ported to Windows.
* Improve error message when it fails to query github for repositories
belonging to a user. Closes: #705084
* Various updates to internal git and utility libraries shared with git-annex.
* Makefile now uses cabal to build.

-- Joey Hess <joeyh@debian.org> Sun, 06 Oct 2013 18:04:56 -0400

github-backup (1.20130622) unstable; urgency=low

* Add missing unix-compat build dependency. Closes: #713279

-- Joey Hess <joeyh@debian.org> Sat, 22 Jun 2013 13:08:57 -0400

github-backup (1.20130618) unstable; urgency=low

* Much better creation and committing to the github branch.

-- Joey Hess <joeyh@debian.org> Mon, 17 Jun 2013 17:40:02 -0400

github-backup (1.20130617) unstable; urgency=low

* Build-Depend on libghc-extensible-exceptions-dev. Closes: #712549
* Various updates to internal git and utility libraries shared with
git-annex, including some Windows portability.
* Fixed to never touch the git work tree or index file, instead using
its own to commit to the github branch.

-- Joey Hess <joeyh@debian.org> Mon, 17 Jun 2013 12:28:30 -0400

github-backup (1.20130614) unstable; urgency=low

* Pass --ignore-removal to git-add, in preparation for a future change
to its default behavior. Requires git 1.8.3. Closes: #711287

-- Joey Hess <joeyh@debian.org> Fri, 14 Jun 2013 15:50:49 -0400

github-backup (1.20130414) experimental; urgency=low

* Updated to use haskell-github 0.6.0, which supports pagination of queries
Thanks to John Wiegley for making those changes.
* Also backup closed issues. Thanks, John Wiegley.
* cabal file no longer tries to list every source file, as that was
error-prone, and I left some out.

-- Joey Hess <joeyh@debian.org> Fri, 12 Apr 2013 18:33:11 -0400

github-backup (1.20120627) unstable; urgency=low

* Rebuilt with new haskell-github, that works with the new version
of http-conduit in Debian. Closes: #678787
* Various updates to internal git and utility libraries shared with git-annex.

-- Joey Hess <joeyh@debian.org> Wed, 27 Jun 2012 22:21:01 -0400

github-backup (1.20120314) unstable; urgency=low

* First release.

-- Joey Hess <joeyh@debian.org> Tue, 13 Mar 2012 20:22:43 -0400
comments powered byDisqus