Open-source, not open-contribution


Litestream is a standalone streaming replication instrument for SQLite. It runs as a
background task and safely replicates changes incrementally to 1 other file
or S3. Litestream top communicates with SQLite through the SQLite API so it
just isn’t going to execrable your database.

If you desire reinforce or have ideas for bettering Litestream, please seek recommendation from the
GitHub Discussions to
chat.

If you glimpse this project gripping, please possess in mind starring the project on
GitHub.

Set up

Mac OS (Homebrew)

To install from homebrew, speed the following repeat:

$ brew install benbjohnson/litestream/litestream

Linux (Debian)

You seemingly also can download the .deb file from the Releases net page net page and
then speed the following:

$ sudo dpkg -i litestream-v0.3.0-linux-amd64.deb

As soon as build apart in, you are going to establish on to enable & delivery the service:

$ sudo systemctl enable litestream
$ sudo systemctl delivery litestream

Open binaries

You seemingly can download the unlock binary for your system from the
releases net page and speed it as a standalone application.

Building from offer

Download and install the Dash toolchain and then speed:

$ trail install ./cmd/litestream

The litestream binary may perchance well also fair peaceable be for your $GOPATH/bin folder.

Fleet Open

Litestream provides a configuration file that can even be passe for production
deployments nonetheless it’s seemingly you’ll specify a single database and replica on the
repeat line when making an strive it out.

First, you are going to establish on to crash an S3 bucket that we’ll call "mybkt" in this
instance. You are going to also identify on to dwelling your AWS credentials:

$ export AWS_ACCESS_KEY_ID=AKIAxxxxxxxxxxxxxxxx
$ export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

Subsequent, you furthermore may perchance can speed the litestream replicate repeat with the path to the
database you identify on to backup and the URL of your replica destination:

$ litestream replicate /route/to/db s3://mybkt/db

If you procure changes to your native database, these changes will seemingly be replicated
to S3 every 10 seconds. From one other terminal window, you furthermore may perchance can restore your
database from your S3 replica:

$ litestream restore -o /route/to/restored/db s3://mybkt/db

Voila! 🎉

Your database may perchance well also fair peaceable be restored to the closing replicated enlighten that
turned into as soon as sent to S3. You seemingly also can alter your replication frequency and varied alternatives by
using a configuration-based totally skill specified below.

Configuration

A configuration-based totally install provides you more replication alternatives. By default,
the config file lives at /and so forth/litestream.yml nonetheless you furthermore may perchance can pass in a sure
path to any litestream repeat using the -config PATH flag. You seemingly can
dwelling the LITESTREAM_CONFIG environment variable to specify a unique route.

The configuration specifies quite a lot of dbs and a checklist of quite a lot of replica
locations for every and each db. Below are some traditional configurations:

Replicate to S3

This can even fair replicate the database at /route/to/db to the "/db" route internal
the S3 bucket named "mybkt".

procure entry to-key-id:     AKIAxxxxxxxxxxxxxxxx
secret-procure entry to-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

dbs:
  - route: /route/to/db
    replicas:
      - url: s3://mybkt/db

Replicate to 1 other file route

This can even fair replicate the database at /route/to/db to a checklist named
/route/to/replica.

dbs:
  - route: /route/to/db
    replicas:
      - route: /route/to/replica

Retention period

By default, replicas may perchance lend a hand a snapshot & subsequent WAL changes for 24
hours. When the snapshot age exceeds the retention threshold, a unique snapshot
is taken and uploaded and the earlier snapshot and WAL info are removed.

You seemingly also can configure this environment per-replica. Cases are parsed using Dash’s
period
so time objects of hours
(h), minutes (m), and seconds (s) are allowed nonetheless days, weeks, months, and
years will not be.

db:
  - route: /route/to/db
    replicas:
      - url: s3://mybkt/db
        retention: 1h          # 1 hour retention

Monitoring replication

You seemingly can enable a Prometheus metrics endpoint to computer screen replication by
specifying a bind take care of with the addr discipline:

This can even fair procure metrics available at: http://localhost: 9090/metrics

Various configuration alternatives

These are some extra configuration alternatives available on replicas:

  • kind—Specify the form of replica ("file" or "s3"). Derived from "route".
  • name—Specify an optionally available name for the replica whereas you’re using a pair of replicas.
  • route—File path to the replica space.
  • url—URL to the replica space.
  • retention-take a look at-interval—Time between retention enforcement tests. Defaults to 1h.
  • validation-interval—Interval between periodic tests to make certain restored backup matches most up-to-the-minute database. Disabled by default.

These replica alternatives are top available for S3 replicas:

  • bucket—S3 bucket name. Derived from "route".
  • space—S3 bucket space. Seemed up on startup if unspecified.
  • sync-interval—Replication sync frequency.

Usage

Replication

As soon as your configuration is saved, you are going to establish on to delivery up replication. If you
build apart within the .deb file then speed:

$ sudo systemctl restart litestream

To speed litestream by itself, speed:

# Replicate using the /and so forth/litestream.yml configuration.
$ litestream replicate

# Replicate using a sure configuration route.
$ litestream replicate -config /route/to/litestream.yml

The litestream repeat will initialize and then wait indefinitely for changes.
You seemingly also can fair peaceable look for your destination replica route is now populated with a
generations checklist. Internal there may perchance well also fair peaceable be a 16-persona hex era
checklist and internal there may perchance well also fair peaceable be snapshots & WAL info. As you procure changes
to your offer database, changes will seemingly be copied over to your replica incrementally.

Restoring a backup

Litestream can restore a earlier snapshot and replay all replicated WAL info.
By default, it will restore as a lot as the most up-to-the-minute WAL file nonetheless it’s seemingly you’ll crash
level-in-time restores.

A database can top be restored to a route that doesn’t exist so you don’t desire
to anguish about unintentionally overwriting your most up-to-the-minute database.

# Restore database to long-established route.
$ litestream restore /route/to/db

# Restore database to a unique space.
$ litestream restore -o /route/to/restored/db /route/to/db

# Restore from a copy URL.
$ litestream restore -o /route/to/restored/db s3://mybkt/db

# Restore database to a explicit level-in-time.
$ litestream restore -timestamp 2020-01-01T00: 00: 00Z /route/to/db

Level-in-time restores top have the decision of the timestamp of the WAL file
itself. By default, Litestream will delivery a unique WAL file every minute so
level-in-time restores are top moral to the minute.

The arrangement it if truth be told works

SQLite provides a WAL (write-forward log) journaling mode which writes pages to
a -wal file sooner than within the crash being copied over to the genuine database file.
This copying task is identified as checkpointing. The WAL file works as a spherical
buffer so when the WAL reaches a sure size then it restarts from the starting.

Litestream works by taking on the checkpointing task and controlling when
it’s restarted to make certain it copies every unique net page. Checkpointing is top
allowed when there are no learn transactions so Litestream maintains a
long-operating learn transaction against each and each database till it’s in a position to
checkpoint.

The SQLite WAL file is copied to a separate space known as the shadow WAL which
ensures that it may perchance well probably well also not be overwritten by SQLite. This shadow WAL acts as a
non eternal buffer in narrate that replicas can replicate to their destination (e.g.
one other file route or to S3). The shadow WAL info are removed as soon as they have
been totally replicated. You seemingly can get the shadow checklist as a hidden checklist
next to your database file. If you database file is named /var/lib/my.db then
the shadow checklist will seemingly be /var/lib/.my.db-litestream.

Litestream groups a snapshot and all subsequent WAL changes into “generations”.
A era is started on preliminary replication of a database and a unique
era will seemingly be started if litestream detects that the WAL replication is
now not contiguous. This can even fair occur if the litestream task is stopped and
one other task is allowed to checkpoint the WAL.

Open-offer, not delivery-contribution

A lot like SQLite, Litestream is delivery
offer nonetheless closed to contributions. This retains the code improper freed from proprietary
or licensed code nonetheless it completely also helps me continue to lend a hand and blueprint Litestream.

As the creator of BoltDB, I came across that
accepting and declaring third birthday party patches contributed to my burn out and
I within the crash archived the project. Writing databases & low-stage replication
instruments entails nuance and clear-reduce one line changes can have profound and
sudden changes in correctness and efficiency. Runt contributions
usually required hours of my time to successfully take a look at and validate them.

I am grateful for group involvement, malicious program reports, & characteristic requests. I form
not like to reach reduction off as the relaxation nonetheless welcoming, on the opposite hand, I’ve
made the decision to preserve this project closed to contributions for my dangle
psychological successfully being and long speed viability of the project.

Read More

Recent Content