View definition

package sarama

Defined in github.com/Shopify/sarama/.

Description

Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). It includes a high-level API for easily producing and consuming messages, and a low-level API for controlling bytes on the wire when the high-level API is insufficient. Usage examples for the high-level APIs are provided inline with their full documentation.

To produce messages, use either the AsyncProducer or the SyncProducer. The AsyncProducer accepts messages on a channel and produces them asynchronously in the background as efficiently as possible; it is preferred in most cases. The SyncProducer provides a method which will block until Kafka acknowledges the message as produced. This can be useful but comes with two caveats: it will generally be less efficient, and the actual durability guarantees depend on the configured value of `Producer.RequiredAcks`. There are configurations where a message acknowledged by the SyncProducer can still sometimes be lost.

To consume messages, use the Consumer. Note that Sarama's Consumer implementation does not currently support automatic consumer-group rebalancing and offset tracking. For Zookeeper-based tracking (Kafka 0.8.2 and earlier), the https://github.com/wvanbergen/kafka library builds on Sarama to add this support. For Kafka-based tracking (Kafka 0.9 and later), the https://github.com/bsm/sarama-cluster library builds on Sarama to add this support.

For lower-level needs, the Broker and Request/Response objects permit precise control over each connection and message sent on the wire; the Client provides higher-level metadata management that is shared between the producers and the consumer. The Request/Response objects and properties are mostly undocumented, as they line up exactly with the protocol fields documented by Kafka at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

Metrics are exposed through https://github.com/rcrowley/go-metrics library.

Broker related metrics:

+------------------------------------------------+------------+---------------------------------------------------------------+
| Name                                           | Type       | Description                                                   |
+------------------------------------------------+------------+---------------------------------------------------------------+
| incoming-byte-rate                             | meter      | Bytes/second read off all brokers                             |
| incoming-byte-rate-for-broker-<broker-id>      | meter      | Bytes/second read off a given broker                          |
| outgoing-byte-rate                             | meter      | Bytes/second written off all brokers                          |
| outgoing-byte-rate-for-broker-<broker-id>      | meter      | Bytes/second written off a given broker                       |
| request-rate                                   | meter      | Requests/second sent to all brokers                           |
| request-rate-for-broker-<broker-id>            | meter      | Requests/second sent to a given broker                        |
| histogram request-size                         | histogram  | Distribution of the request size in bytes for all brokers     |
| histogram request-size-for-broker-<broker-id>  | histogram  | Distribution of the request size in bytes for a given broker  |
| response-rate                                  | meter      | Responses/second received from all brokers                    |
| response-rate-for-broker-<broker-id>           | meter      | Responses/second received from a given broker                 |
| histogram response-size                        | histogram  | Distribution of the response size in bytes for all brokers    |
| histogram response-size-for-broker-<broker-id> | histogram  | Distribution of the response size in bytes for a given broker |
+------------------------------------------------+------------+---------------------------------------------------------------+

Note that we do not gather specific metrics for seed brokers but they are part of the "all brokers" metrics.