IPFS: The Permanent Web, by Juan Benet (Talks at Sourcegraph 003)



image

Juan Benet (@juanbenet) spoke at last week’s Talks at Sourcegraph about his project, “IPFS: The Permanent Web” (slides and video). IPFS is a bold attempt at evolving the Internet’s infrastructure. Here’s how Juan describes it:

IPFS is a global, versioned, peer-to-peer file system. It combines good ideas from Git, BitTorrent, Kademlia, and SFS. You can think of it like a single BitTorrent swarm, exchanging Git objects, making up the web. IPFS provides an interface much simpler than HTTP, but has permanence built in. You can also use it to mount the world at /ipfs.

So far, IPFS has 2 implementations: go-ipfs and node-ipfs.

Let’s dig a bit deeper into IPFS and how it could be used.

The Permanent Web

IPFS envisions a world where any resource is available via a locally mounted filesystem at paths like:

# a mutable path
/ipns/my.host.com/some/file.txt

# or a permanent path
/ipfs/QmaKtFK3wiq9z2LmhMKgoXvuB4XDeY5Xrac8vVBLPjc9CX/some/file.txt

But this file doesn’t necessarily reside on the local disk. Instead, it exists in IPFS, a global distributed storage system. IPFS makes files in this namespace:

  • highly available, so they can be fetched from any host that stores and is willing to provide the data, not just their originator or owner
  • fast to access, so you can treat the /ipns and /ipfs filesystems as local storage instead of remote servers, as in HTTP. In fact, nodes chose which files they store locally
  • trusted, like in git where you can trust files in a commit if you trust the commit ID
  • signed, so you can attest that you published a certain file at a certain path
  • Merkle-linked, so you can version and back up everything, and so you can create a link structure like the web

To achieve these goals, IPFS synthesizes many successful techniques from the last 15 years of distributed systems research. Central to the IPFS design is the Merkle DAG, a data structure that represents all the files. It’s like Git’s blob, tree, and commit, except IPFS has a more flexible model: you can define what your link structure is and how it works. This means you can implement Git on top, or a Blockchain like Bitcoin, or linked web pages.

Juan also discussed a list of use cases for IPFS:

  1. Mounted global filesystem, under /ipfs (permanent) and /ipns (mutable)
  2. Mounted personal sync folder that automatically versions, publishes, and backs up any writes (like a versioned, distributed Dropbox)
  3. Encrypted file or data sharing system
  4. Distributing mutable content (BitTorrent + Git)
  5. Blockchain data structures
  6. Versioned package manager for all software (see this discussion)
  7. Booting a virtual machine from the network
  8. Booting a VM from a hash (trust your system is at that particular state)
  9. Databases (applications write directly to the Merkle DAG data model, getting all the versioning, caching, and distribution that IPFS provides)
  10. Linked (and encrypted) communications platform
  11. Integrity-checked CDN for large static files (without SSL)
  12. Encrypted CDN
  13. The Permanent Web, where links do not die

Watch the talk, and check out the IPFS paper for more details.

Getting involved with IPFS

IPFS is an open-source, MIT-licensed project. You can use the ideas and the implementations for whatever you want. It’s still early, so the implementations are not complete. You should follow @juanbenet and check out the IPFS spec and open-source implementations. Watch or star these repositories on GitHub to follow along:

Also, you can join #ipfs on irc.freenode.org.

image

Juan has a compelling vision for a future where the Internet is more efficient, robust, and safe. He has been thinking about these things for at least the last 5 years I’ve known him, and he’s now turning all that thought into action. Watch what he does, and if you’re interested in joining Juan, he told me he’s hiring engineers who:

  • understand distributed systems well,
  • know either node or Go,
  • write open source,
  • and want to do their part in evolving the Internet.

Reach out to him at juan@ipfs.io.

Upcoming Talks at Sourcegraph

We bring in speakers to the Sourcegraph office in San Francisco every month to talk about the tech behind the most ambitious and interesting open source projects. Join the Sourcegraph meetup group to hear about the next one.

(Also, at the end of the talk, Juan previewed Filecoin, the sister protocol of IPFS. Filecoin was announced the day after the IPFS talk, but our attendees got a sneak peek.)