SpliceFS GitHub

How SpliceFS works

SpliceFS mounts the files of a cluster of servers as one read-only folder. It reads each file from several servers at once.

Status: alpha, step one. Read-only. The first 0.x versions are published as packages. It has not run in production.

Today

This section describes what the code does now, as written in the design documents of the server and the client.

The pieces

A client looks up the cluster host name, which points at three servers. Each server holds plain files, and your own sync tool keeps them equal. node-1.example.com splicefsd, plain files node-2.example.com splicefsd, plain files node-3.example.com splicefsd, plain files your sync tool, not SpliceFS Client splicefs DNS name files.example.com (cluster host) one name
One name leads to the servers. The sync tool is yours; SpliceFS does not copy data between servers.

The servers are called nodes. Each one runs the server program splicefsd and holds the files as plain files in a folder. You keep those folders equal with any sync tool you like, because SpliceFS does not copy data between servers today. One host name, the cluster host, points at one, several or all of the servers. The client program splicefs is told only that name.

Connecting

Three steps. One: the client logs in at the cluster host. Two: the cluster host sends back the list of servers. Three: the client talks to each server directly. Client splicefs files.example.com cluster host Servers node-1, node-2, node-3 1 Log in over HTTPS 2 Receive the list of servers 3 Talk to each server directly
Only the cluster host is asked for the list. After that the client uses each server's own address.

The client logs in at the cluster host over HTTPS and asks it for the list of servers. Every server publishes its own direct address, and any server can answer with the whole list. The small files that hold users and server addresses travel between servers with the same sync tool. After that the client talks to each server directly. It asks the cluster host for the list again every minute, so servers can join and leave.

Reading a file

One file shown as nine numbered blocks. Each block is colored by the server that sent it, node-1, node-2 or node-3. All blocks are kept in the local cache on disk. node-1 node-2 node-3 One file, read as blocks. Each block has the color of the server that sent it. 1 node-1 2 node-2 3 node-3 4 node-1 5 node-2 6 node-3 7 node-2 8 node-1 9 node-3 Local cache on disk A second read of these blocks comes from here
Three servers send different blocks at the same time. The client puts them in order and keeps them in its cache.

The client reads a file in blocks, 5 MiB each by default. It asks several servers for different blocks at the same time and puts them together in order, so the speeds add up. Blocks that were read are kept in a cache on local disk, and a second read comes from the cache. Small files and small isolated reads are read straight from the servers and are not cached.

When a server fails

The same file as before, but node-2 is down and crossed out. Blocks 2, 5 and 7 now come from node-1 and node-3, and the read continues. node-1 node-2 is down node-3 Blocks 2, 5 and 7 (thick outline) come from node-1 and node-3 instead. 1 node-1 2 node-1 3 node-3 4 node-1 5 node-3 6 node-3 7 node-1 8 node-1 9 node-3 Local cache on disk The read goes on and the mount stays up
A block that node-2 did not deliver is asked from another server. The reader does not notice.

If a server is slow or does not answer, the client asks another server for that block and uses the first complete answer. The read goes on and the mount stays up. A server whose copy of a file has changed is dropped for that file. If no server can deliver a block within a time limit, the read fails with an I/O error and does not hang.

What it does not do today

Linux and FreeBSD mount with FUSE. macOS mounts through a small local NFS server that is part of the client.

  • It is read-only. Nothing can be written through the mount.
  • It does not copy data between servers. Your sync tool does that.
  • A removed or disabled user keeps access until the token expires, 30 minutes by default.
  • A token from one server is rejected by another until the sync tool has delivered the first server's key file.
  • A client that is restarted while no server can be reached cannot list folders, although cached file contents are still on its disk.
  • macOS limits: a read that waits longer than about 9 seconds fails, requests are served one at a time per connection so speed stays below Linux, and an outage can make a new name look missing for up to 30 seconds.

For the full comparison with other systems, see How SpliceFS compares.

The plan

Plan, not built. Read-only is step one. Everything below is the direction the project is headed. None of it exists yet, and details will change.

Solid outline = exists today. Dashed outline = planned.

Writing, with transactions

A client sends a change to a file. On the left, one outcome: every server that holds the file commits the change. On the right, the other outcome: no server commits it. Client sends a change All servers commit node-1 changed node-2 changed node-3 changed The new content is everywhere or No server commits node-1 unchanged node-2 unchanged node-3 unchanged The old content stays everywhere
Plan, not built. One change has one outcome: every server that holds the file commits it, or none does.

Clients will write through the mount. A change to a file is a transaction: it happens completely or not at all, on every server that holds the file.

Cohorts

Six servers grouped into two cohorts of three. In cohort 1 the blocks of one file are spread over the members. One member, node-2, is lost and crossed out, and the file is still complete. Cohort 1 node-1 member node-2 lost node-3 member One file, in blocks 1 2 3 4 5 6 The file is still complete Cohort 2 node-4 member node-5 member node-6 member Not involved in this file
Plan, not built. Six servers in two cohorts of three. In cohort 1 one member is lost and the file is still complete; blocks 2 and 5 (thick outline) come from the other members.

A cohort is a group of servers that hold data together, the way the disks of a RAID set do. The data survives the loss of a member. Reads are spread over the members.

Locks and full POSIX behavior

Two clients at different places and one file. Client 1 holds the lock on the file. Client 2 waits for it. Client 1 one location A file on the mount Client 2 another location takes lock waits Lock held by Client 1
Plan, not built. Client 1 holds the lock on a file. Client 2, at another place, waits.

A lock that a program takes on a file through one client is honored by every other client, wherever it is. The goal is full POSIX behavior, so ordinary programs work on a SpliceFS mount without changes.

Near and far, and where data lives

Three clients, on the local network, over a VPN and over the public internet, all drawn the same and all reaching one cluster. The cluster has three locations, A, B and C, each with two servers. The folder reports is held at A and B only. The folder tools is held everywhere. Client local network Client VPN Client public internet Same treatment for every client Location A node-1 node-2 reports tools Location B node-3 node-4 reports tools Location C node-5 node-6 tools reports: held at A and B only tools: held everywhere
Plan, not built. All three clients are treated the same way. One folder is held at locations A and B only, another is held everywhere.

SpliceFS aims to treat a client on the local network, a client over a VPN and a client over the public internet the same way. Geographic tenancy lets a rule on one file, one directory or a whole cluster say at which locations that data is held.

The plan by version

A line with three stops. 0.x is marked now: the core of the SpliceFS protocol and access controls. 1.x is planned: enterprise tooling and visibility. 2.x is planned: only time will tell. now 0.x 1.x 2.x Core of the SpliceFS protocol and access controls Enterprise tooling and visibility Only time will tell
Plan, not built. The project is in 0.x now. 1.x and 2.x come later.

0.x is the core of the SpliceFS protocol, and access controls. 1.x is enterprise tooling and visibility. 2.x: only time will tell.

For where SpliceFS stands today next to mature systems, see How SpliceFS compares.