Description

Package journal reads and writes sequences of journals. Each journal is a stream of bytes that completes before the next journal starts.

When reading, call Next to obtain an io.Reader for the next journal. Next will return io.EOF when there are no more journals. It is valid to call Next without reading the current journal to exhaustion.

When writing, call Next to obtain an io.Writer for the next journal. Calling Next finishes the current journal. Call Close to finish the final journal.

Optionally, call Flush to finish the current journal and flush the underlying writer without starting a new journal. To start a new journal after flushing, call Next.

Neither Readers or Writers are safe to use concurrently.

Example code:

func read(r io.Reader) ([]string, error) {
	var ss []string
	journals := journal.NewReader(r, nil, true, true)
	for {
		j, err := journals.Next()
		if err == io.EOF {
			break
		}
		if err != nil {
			return nil, err
		}
		s, err := ioutil.ReadAll(j)
		if err != nil {
			return nil, err
		}
		ss = append(ss, string(s))
	}
	return ss, nil
}

func write(w io.Writer, ss []string) error {
	journals := journal.NewWriter(w)
	for _, s := range ss {
		j, err := journals.Next()
		if err != nil {
			return err
		}
		if _, err := j.Write([]byte(s)), err != nil {
			return err
		}
	}
	return journals.Close()
}

The wire format is that the stream is divided into 32KiB blocks, and each block contains a number of tightly packed chunks. Chunks cannot cross block boundaries. The last block may be shorter than 32 KiB. Any unused bytes in a block must be zero.

A journal maps to one or more chunks. Each chunk has a 7 byte header (a 4 byte checksum, a 2 byte little-endian uint16 length, and a 1 byte chunk type) followed by a payload. The checksum is over the chunk type and the payload.

There are four chunk types: whether the chunk is the full journal, or the first, middle or last chunk of a multi-chunk journal. A multi-chunk journal has one first chunk, zero or more middle chunks, and one last chunk.

The wire format allows for limited recovery in the face of data corruption: on a format error (such as a checksum mismatch), the reader moves to the next block and looks for the next full or first chunk.

journal is referenced in 0 repositories