GopherCon 2018 - Go in Debian

Presenter: Michael Stapelberg, [email protected] Creator of the i3 Linux window manager

Using Go in and with Debian

Summary

Debian services and tools written in Go. Developing and packaging Go software for use in Debian. Upstream best practices.

This is a broad overview, not tons of detail. Happy to chat about any of it.

Intro to Debian

Debian is an OS; most popular is Debian GNU/Linux, first announced in 1993

Many derivatives, most popular is Ubuntu

Debian services written in Go

codesearch.debian.net

regexp search on all software in Debian. Find examples, usages, definitions, etc.

Based on Russ Cox's "Regular Expression Matching with a Trigram Index"

manpages.debian.org

All of Debian's man pages. Launched 2017, using mandoc. Good internationalization thanks to golang.org/x/text.

Fairly advanced cross-referencing. So if a link from a Norwegian manpage points to another page not yet translated to Norwegian, they can link to the Swedish version instead. (Norwegians can also read Swedish.)

Completeness graph

Debian is green all the way across. Most other man page archives on the web are red.

Converts 500k manpages in the archive in about 30 minutes.

Good for quick disaster recovery; increases developer velocity. They've found it's faster to regenerate from scratch than restore from backup.
Easy to eliminate bottlenecks in Go
- pprof invaluable
- scaled from machines from 2008 to modern laptops (2018)

Debian tools in Go

ratt

Rebuild All The Things!

Departure from the usual "Debian way". Find issues before you upload your changes. ratt identifies and builds reverse-dependencies.

pk4

Get source for a given package, e.g. given the path of a binary that comes in the package. Fairly painless way to build Debian packages & inspect source.

Programmatically working with Debian in Go

These are Go packages to work with Debian packages.

https://pault.ag/go/debian

read .deb files
parse/compare version numbers
read changelog/control metadata
calculate dependencies

https://pault.ag/go/archive

Debian apt archives (e.g. ftp.debian.org)

Go software in Debian

The big hitters, of course: Docker, Influx, k8s, Prometheus, etc. Lots of other stuff, though.

Debian Go packaging team — 64 volunteers. Anyone can contribute; no formal status needed. Don't need to be a Debian project member, even.

Packaging workflow

Create tarball from a source code management system (e.g. git). Add Debian metadata (copyright, dependencies, etc).
Optional: Patches.
Build the package: dpkg-buildpackage

2 types of Debian packages

Programs
- compile source, add manpages, etc
Libraries
- more Go-specific stuff here
- source in /usr/share/gocode
- only used for building Debian programs. Different from e.g. Python
- don't use these libs for your own development

Tooling

ch-make-golang — largely automates package creation
edit-build-test cycle
- upload once satisfied
- lintian identifies issues
package / update transitive dependency tree first! this is the real bulk of the work

Scaling issues

Updates to one lib may break the libs or programs Continuous Integration

pgt-gopath; build/test; about 30s for changes to small packages. Go's test & compilation speed really pays off here.
automate updates to new versions? maybe. CI coverage needs to be high enough; sometimes it's not

Upstream best practices

How can you help make life easier for Debian folks?

Few dependencies

Like Go, in Debian, a little copying is better than a little dependency. Take it to heart. Even more so in Debian than in Go.
Dependencies are way more costly in Debian
dh-make-golang can tell you if your dependency is already packaged

Use the go tool

Do not require custom build systems
Debian's tooling uses the go tool. If you don't, that gives them extra work
Don't use custom build tags

Test w/out vendoring

Debian always throws away vendor/. So you should test w/out vendor, too.
Corollary: Never modify vendor/ contents

Challenges

Portability

Debian builds for many architectures, some fairly obscure (IBM s/390 anyone?). Debian usually finds portability issues before upstream does

Diamond problem

package A requires a newer B,
but pkg C is broken by newer B
Semantic Import Versioning should help

Not enough volunteers! 🙂

Links and feedback

Send feedback (Google form) https://go-team.pages.debian.net/ Talk to Michael for questions! He's friendly.