IPFS: The Permanent Web, by Juan Benet (Talks at Sourcegraph 003)
Juan Benet (@juanbenet) spoke at last week’s Talks at Sourcegraph about his project, “IPFS: The Permanent Web” (slides and video). IPFS is a bold attempt at evolving the Internet’s infrastructure. Here’s how Juan describes it:
IPFS is a global, versioned, peer-to-peer file system. It combines good ideas from Git, BitTorrent, Kademlia, and SFS. You can think of it like a single BitTorrent swarm, exchanging Git objects, making up the web. IPFS provides an interface much simpler than HTTP, but has permanence built in. You can also use it to mount the world at /ipfs.
Let’s dig a bit deeper into IPFS and how it could be used.
The Permanent Web
IPFS envisions a world where any resource is available via a locally mounted filesystem at paths like:
# a mutable path /ipns/my.host.com/some/file.txt # or a permanent path /ipfs/QmaKtFK3wiq9z2LmhMKgoXvuB4XDeY5Xrac8vVBLPjc9CX/some/file.txt
But this file doesn’t necessarily reside on the local disk. Instead, it exists in IPFS, a global distributed storage system. IPFS makes files in this namespace:
- highly available, so they can be fetched from any host that stores and is willing to provide the data, not just their originator or owner
- fast to access, so you can treat the /ipns and /ipfs filesystems as local storage instead of remote servers, as in HTTP. In fact, nodes chose which files they store locally
- trusted, like in git where you can trust files in a commit if you trust the commit ID
- signed, so you can attest that you published a certain file at a certain path
- Merkle-linked, so you can version and back up everything, and so you can create a link structure like the web
To achieve these goals, IPFS synthesizes many successful techniques from the last 15 years of distributed systems research. Central to the IPFS design is the Merkle DAG, a data structure that represents all the files. It’s like Git’s blob, tree, and commit, except IPFS has a more flexible model: you can define what your link structure is and how it works. This means you can implement Git on top, or a Blockchain like Bitcoin, or linked web pages.
Juan also discussed a list of use cases for IPFS:
- Mounted global filesystem, under /ipfs (permanent) and /ipns (mutable)
- Mounted personal sync folder that automatically versions, publishes, and backs up any writes (like a versioned, distributed Dropbox)
- Encrypted file or data sharing system
- Distributing mutable content (BitTorrent + Git)
- Blockchain data structures
- Versioned package manager for all software (see this discussion)
- Booting a virtual machine from the network
- Booting a VM from a hash (trust your system is at that particular state)
- Databases (applications write directly to the Merkle DAG data model, getting all the versioning, caching, and distribution that IPFS provides)
- Linked (and encrypted) communications platform
- Integrity-checked CDN for large static files (without SSL)
- Encrypted CDN
- The Permanent Web, where links do not die
Getting involved with IPFS
IPFS is an open-source, MIT-licensed project. You can use the ideas and the implementations for whatever you want. It’s still early, so the implementations are not complete. You should follow @juanbenet and check out the IPFS spec and open-source implementations. Watch or star these repositories on GitHub to follow along:
Also, you can join #ipfs on irc.freenode.org.
Juan has a compelling vision for a future where the Internet is more efficient, robust, and safe. He has been thinking about these things for at least the last 5 years I’ve known him, and he’s now turning all that thought into action. Watch what he does, and if you’re interested in joining Juan, he told me he’s hiring engineers who:
- understand distributed systems well,
- know either node or Go,
- write open source,
- and want to do their part in evolving the Internet.
Reach out to him at firstname.lastname@example.org.
Upcoming Talks at Sourcegraph
We bring in speakers to the Sourcegraph office in San Francisco every month to talk about the tech behind the most ambitious and interesting open source projects. Join the Sourcegraph meetup group to hear about the next one.
(Also, at the end of the talk, Juan previewed Filecoin, the sister protocol of IPFS. Filecoin was announced the day after the IPFS talk, but our attendees got a sneak peek.)