Kafka command line tools

https://stackoverflow.com/questions/31061781/what-command-shows-all-of-the-topics-and-offsets-of-partitions-in-kafka

up vote13down voteaccepted

Kafka ships with some tools you can use to accomplish this. List topics: # ./bin/kafka-topics.sh --list --zookeeper localhost:2181 test_topic_1 test_topic_2 ... List partitions and offsets: # ./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group test_group --topic test_topic --zookeeper localhost:2181 Group Topic Pid Offset logSize Lag Owner test_group test_topic 0 698020 698021 1 test_group-0 test_group test_topic 1 235699 235699 0 test_group-1 test_group test_topic 2 117189 117189 0

https://www.cloudera.com/documentation/kafka/latest/topics/kafka_command_line.html

Using Kafka Command-line Tools

Kafka command-line tools are located in /usr/bin:

• kafka-topics
  Create, alter, list, and describe topics. For example:
  

  $ /usr/bin/kafka-topics --zookeeper zk01.example.com:2181 --list
  sink1
  t1
  t2
  $ /usr/bin/kafka-topics --create --zookeeper hostname:2181/kafka --replication-factor 2 
    --partitions 4 --topic topicname 
                          

• kafka-console-consumer
  Read data from a Kafka topic and write it to standard output. For example:
  

  $ /usr/bin/kafka-console-consumer --zookeeper zk01.example.com:2181 --topic t1

• kafka-console-producer
  Read data from standard output and write it to a Kafka topic. For example:
  

  $ /usr/bin/kafka-console-producer --broker-list kafka02.example.com:9092,kafka03.example.com:9092 --topic t1

• kafka-consumer-offset-checker (deprecated)
  
  
  

  Note: kafka-consumer-offset-checker is not supported in the new Consumer API. Use the ConsumerGroupCommand tool, below.

  Check the number of messages read and written, as well as the lag for each consumer in a specific consumer group. For example:
  

  $ /usr/bin/kafka-consumer-offset-checker --group flume --topic t1 --zookeeper zk01.example.com:2181

• kafka-consumer-groups
  To view offsets as in the previous example with the ConsumerOffsetChecker, you describe the consumer group using the following command:

  $ /usr/bin/kafka-consumer-groups --zookeeper zk01.example.com:2181 --describe --group flume
  
  GROUP   TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     OWNER
  flume   t1     0          1               3               2       test-consumer-group_postamac.local-1456198719410-29ccd54f-0