kafka 命令行命令大全

kafka 脚本

connect-distributed.sh
connect-mirror-maker.sh
connect-standalone.sh
kafka-acls.sh
kafka-broker-api-versions.sh
kafka-configs.sh
kafka-console-consumer.sh
kafka-console-producer.sh
kafka-consumer-groups.sh
kafka-consumer-perf-test.sh
kafka-delegation-tokens.sh
kafka-delete-records.sh
kafka-dump-log.sh
kafka-leader-election.sh
kafka-log-dirs.sh
kafka-mirror-maker.sh
kafka-preferred-replica-election.sh
kafka-producer-perf-test.sh
kafka-reassign-partitions.sh
kafka-replica-verification.sh
kafka-run-class.sh
kafka-server-start.sh
kafka-server-stop.sh
kafka-streams-application-reset.sh
kafka-topics.sh
kafka-verifiable-consumer.sh
kafka-verifiable-producer.sh
trogdor.sh
windows
zookeeper-security-migration.sh
zookeeper-server-start.sh
zookeeper-server-stop.sh
zookeeper-shell.sh

集群管理

# 启动zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties &# 停止zookeeper
bin/zookeeper-server-stop.sh# 前台启动broker Ctrl + C 关闭
bin/kafka-server-start.sh <path>/server.properties# 后台启动broker
bin/kafka-server-start.sh -daemon <path>/server.properties# 关闭broker
bin/kafka-server-stop.sh

topic相关

# 创建topic(4个分区,2个副本)
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 4 --topic test
# kafka版本 >= 2.2
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test--create:指定创建topic动作--topic:指定新建topic的名称--zookeeper: 指定kafka连接zk的连接url,该值和server.properties文件中的配置项{zookeeper.connect}一样--partitions:指定当前创建的kafka分区数量,默认为1个--replication-factor:指定每个分区的复制因子个数,默认1个# 分区扩容,注意:分区数量只能增加,不能减少
# kafka版本 < 2.2
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 --partitions 2
# kafka版本 >= 2.2
bin/kafka-topics.sh --bootstrap-server broker_host:port --alter --topic topic1 --partitions 2# 删除topic
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test# 查询topic列表
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
# 查询topic列表(支持0.9版本+)
bin/kafka-topics.sh --list --bootstrap-server localhost:9092# 查看所有topic的详细信息
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe # 查询topic详情
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic topicname

消费者组

查看consumer group列表有新、旧两种命令,分别查看新版(信息保存在broker中)consumer列表和老版(信息保存在zookeeper中)consumer列表,因而需要区分指定bootstrap–server和zookeeper参数

# 新消费者列表查询(支持0.9版本+)
bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list
# 消费者列表查询(支持0.10版本+)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list# 显示某个消费组的消费详情(仅支持offset存储在zookeeper上的)
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group test
# 显示某个新消费组的消费详情(0.9版本 - 0.10.1.0 之前)
bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --describe --group my-group
## 显示某个消费组的消费详情(0.10.1.0版本+)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group# 重设消费者组位移
# 最早处
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupname --reset-offsets --all-topics --to-earliest --execute
# 最新处
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupname --reset-offsets --all-topics --to-latest --execute
# 某个位置
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupname --reset-offsets --all-topics --to-offset 2000 --execute
# 调整到某个时间之后的最早位移
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupname --reset-offsets --all-topics --to-datetime 2019-09-15T00:00:00.000# 删除消费者组
bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --delete --group groupname

显示某个消费组的消费详情
显示某个消费组的消费详情
各字段含义如下

TOPICPARTITIONCURRENT-OFFSETLOG-END-OFFSETLAGCONSUMER-IDHOSTCLIENT-ID
topic名字分区id当前已消费的条数总条数未消费的条数消费id主机ip客户端id

发送和消费

# 生产者
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test# 消费者,其中"--from-beginning"为可选参数,表示要从头消费消息
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topicname --from-beginning
# 指定groupid
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topicname --from-beginning --consumer-property group.id=old-consumer-group
# 指定分区
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topicname --from-beginning --partition 0# 新生产者(支持0.9版本+)
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties# 新消费者(支持0.9版本+)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --new-consumer --from-beginning --consumer.config config/consumer.properties# kafka-verifiable-consumer.sh(消费者事件,例如:offset提交等)
bin/kafka-verifiable-consumer.sh --broker-list localhost:9092 --topic test --group-id groupName# 高级点的用法
bin/kafka-simple-consumer-shell.sh --brist localhost:9092 --topic test --partition 0 --offset 1234  --max-messages 10

切换leader

# kafka版本 <= 2.4
bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot# kafka新版本
bin/kafka-preferred-replica-election.sh --bootstrap-server broker_host:port

kafka自带压测命令

bin/kafka-producer-perf-test.sh --topic test --num-records 100 --record-size 1 --throughput 100  --producer-props bootstrap.servers=localhost:9092

kafka持续发送消息

持续发送消息到指定的topic中,且每条发送的消息都会有响应信息:

kafka-verifiable-producer.sh --broker-list $(hostname -i):9092 --topic test --max-messages 100000

zookeeper-shell.sh

如果kafka集群的zk配置了chroot路径,那么需要加上/path

bin/zookeeper-shell.sh localhost:2181[/path]
ls /brokers/ids
get /brokers/ids/0

迁移分区

  1. 创建规则json
cat > increase-replication-factor.json <<EOF
{"version":1, "partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":1,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":2,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":3,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":4,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":5,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":6,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":7,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":8,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":9,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":10,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":11,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":12,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":13,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":14,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":15,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":16,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":17,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":18,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":19,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":20,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":21,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":22,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":23,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":24,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":25,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":26,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":27,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":28,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":29,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":30,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":31,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":32,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":33,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":34,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":35,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":36,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":37,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":38,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":39,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":40,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":41,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":42,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":43,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":44,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":45,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":46,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":47,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":48,"replicas":[0,1]},
{"topic":"__consumer_offsets","partition":49,"replicas":[0,1]}]
}
EOF
  1. 执行
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
  1. 验证
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify

MirrorMaker 跨机房灾备工具

bin/kafka-mirror-maker.sh --consumer.config consumer.properties --producer.config producer.properties --whitelist topicA|topicB

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/news/453825.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

python 第三方模块之 APScheduler - 定时任务

介绍 APScheduler的全称是Advanced Python Scheduler。它是一个轻量级的 Python 定时任务调度框架。APScheduler 支持三种调度任务&#xff1a;固定时间间隔&#xff0c;固定时间点&#xff08;日期&#xff09;&#xff0c;Linux 下的 Crontab 命令。同时&#xff0c;它还支持…

hadoop分布式搭建

一&#xff0c;前提&#xff1a;下载好虚拟机和安装完毕Ubuntu系统。因为我们配置的是hadoop分布式&#xff0c;所以需要两台虚拟机&#xff0c;一台主机&#xff08;master&#xff09;&#xff0c;一台从机&#xff08;slave&#xff09; 选定一台机器作为 Master 在 Master …

xvid 详解 代码分析 编译等

1. Xvid参数详解 众所周知&#xff0c;Mencoder以其极高的压缩速率和不错的画质赢得了很多朋友的认同&#xff01; 原来用Mencoder压缩Xvid的AVI都是使用Xvid编码器的默认设置&#xff0c;现在我来给大家冲冲电&#xff0c;讲解一下怎样使用Mencoder命令行高级参数制作Xvid编…

很多人喜欢露脚踝你觉得时尚吗?

当然是 时尚时尚最时尚的 露&#xff01;****脚&#xff01;脖&#xff01;子&#xff01;image.png人生就是这么奇怪 美容整形可以让你拥有想要的五官 做个手术健个身能让你拥有梦寐的线条 唯独身高这事很难改变&#xff08;说多了都是泪&#xff09; 氮素呢 再难也难不倒众位…

深度学习之生成式对抗网络 GAN(Generative Adversarial Networks)

一、GAN介绍 生成式对抗网络GAN&#xff08;Generative Adversarial Networks&#xff09;是一种深度学习模型&#xff0c;是近年来复杂分布上无监督学习最具前景的方法之一。它源于2014年发表的论文&#xff1a;《Generative Adversarial Nets》&#xff0c;论文地址&#xf…

深度学习之目标检测:R-CNN、Fast R-CNN、Faster R-CNN

object detection 就是在给定的图片中精确找到物体所在位置&#xff0c;并标注出物体的类别。object detection 要解决的问题就是物体在哪里&#xff0c;是什么这整个流程的问题。然而&#xff0c;这个问题不是容易解决的&#xff0c;物体的尺寸变化范围很大&#xff0c;摆放物…

深度学习之边框回归(Bounding Box Regression)

从rcnn&#xff0c; fast rcnn, faster rcnn, yolo, r-fcn, ssd&#xff0c;到cvpr的yolo9000。这些paper中损失函数都包含了边框回归&#xff0c;除了rcnn详细介绍了&#xff0c;其他的paper都是一笔带过&#xff0c;或者直接引用rcnn就把损失函数写出来了。前三条网上解释比较…

2018 年,React 将独占web前端框架鳌头?

相比 Angular 和 Vue&#xff0c; React 是 2017 年的主要 JS 框架&#xff0c;尤其是 React Native 以前所未有的速度提升自己。 Stateofjs 2017前端框架调查结果 相比较 2016 年的调查结果 所以 &#xff0c;1 年过去了&#xff0c;Vue.js 显然在前端框架中占据了领导地位&am…

python 第三方模块之 pandas 操作 excel

python 解析 excel 对比 包版本xls读xlsx读xls写xlsx写备注xlrd1.1.0&#xff08;2017年8月22日&#xff09;√√2.0 之后不支持xlsxxlwt1.3.0&#xff08;2017年8月22日&#xff09;√openpyxl2.6.2&#xff08;2019年3月29日&#xff09;√√XlsxWriter1.2.1&#xff08;201…

YUV / RGB 格式及快速转换

YUV是指亮度参量和色度参量分开表示的像素格式&#xff0c;而这样分开的好处就是不但可以避免相互干扰&#xff0c;还可以降低色度的采样率而不会对图像质量影响太大。 YUV是一个比较笼统地说法&#xff0c;针对它的具体排列方式&#xff0c;可以分为很多种具体的格式。转载一篇…

深度学习之 SSD(Single Shot MultiBox Detector)

目标检测近年来已经取得了很重要的进展&#xff0c;主流的算法主要分为两个类型&#xff1a; &#xff08;1&#xff09;two-stage方法&#xff0c;如R-CNN系算法&#xff0c;其主要思路是先通过启发式方法&#xff08;selective search&#xff09;或者CNN网络&#xff08;RP…

短时程突触可塑性(short-term synaptic plasticity)

介绍神经元的突触可塑性一般被认为是大脑学习与记忆的分子生物学机制&#xff0c;它是指突触传递效率增强或减弱的变化现象。若这种变化只持续数十毫秒到几分&#xff0c;便称之为短时程突触可塑性&#xff0c;其中效率增强与减弱分别叫做短时程增强&#xff08;short-term enh…

windows平台下vlc编译

转自:http://jeremiah.blog.51cto.com/539865/114190Jeremiah刚刚工作几个月&#xff0c;参与的第一个项目是与视频监控有关&#xff0c;分配给我的任务就是用开源的vlc做一个自己的播放器。对于开源项目来说&#xff0c;搭建起编译环境是第一步也是最重要的一步。Jeremiah在历…

深度学习之卷积神经网络 AlexNet

AlexNet 是 2012年ILSVRC 比赛冠军&#xff0c;远超第二名的CNN&#xff0c;比LeNet更深&#xff0c;用多层小卷积叠加来替换单个的大卷积&#xff0c;结构如下图所示。 ​​ 结构 预处理 原始图片&#xff1a;256∗256∗3256*256*3256∗256∗3 图像处理&#xff1a; 1.随机…

jstl处理栏目与子栏目_芬顿氧化法废水处理工程技术规范(征求意见稿)

日前&#xff0c;生态环境部印发《芬顿氧化法废水处理工程技术规范(征求意见稿)》&#xff0c;详情如下&#xff1a;各有关单位&#xff1a;为贯彻《中华人民共和国环境保护法》和《中华人民共和国水污染防治法》等法律法规&#xff0c;防治环境污染&#xff0c;改善环境质量&a…

深度学习之卷积神经网络 ZF Net

ZFNet出自论文《 Visualizing and Understanding Convolutional Networks》&#xff0c;作者Matthew D. Zeiler和Rob Fergus——显然ZFNet是以两位作者名字的首字母命名的。ZFNet通常被认为是ILSVRC 2013的冠军方法&#xff0c;但实际上ZFNet排在第3名&#xff0c;前两名分别是…

vb整合多个excel表格到一张_[Excel]同一工作簿中多个工作表保存成独立的表格

一个工作簿中有多个表格&#xff0c;如何将其表格单独保存成一个独立的文档呢&#xff1f;如果表格少&#xff0c;操作如下&#xff1a;选中要导出表格的标签名--鼠标邮件--移动或复制表格--新建工作簿。当如果表格太多呢&#xff0c;以上方法就太罗嗦了。简单方法用VBA,步骤如…

OpenCore 的代码结构

OpenCore的代码结构 以开源Android 的代码为例&#xff0c;Open Core 的代码在Android 代码的External/Opencore 目录 中。这个目录是OpenCore 的根目录&#xff0c;其中包含的子目录如下所示&#xff1a; android&#xff1a;这里面是一个上层的库&#xff0c;它基于PVPlaye…

深度学习之卷积神经网络 GoogleNet

GoogLeNet Incepetion V1 这是GoogLeNet的最早版本&#xff0c;出现在2014年的《Going deeper with convolutions》。之所以名为“GoogLeNet”而非“GoogleNet”,文章说是为了向早期的LeNet致敬。 深度学习以及神经网络快速发展&#xff0c;人们不再只关注更给力的硬件、更大…

Jzoj4348 打击目标

又是被水题坑了。。。 一直想不出来看题解说要什么主席树&#xff0c;于是开始打离线算法 结果打到一半发现要强制在线。。No!!! 发现直接AC自动机似乎可做&#xff1f;树剖之后在AC自动机上跑的时候判断一下不就好了吗!连线段树都不要 让后快乐切掉&#xff0c;速度还可以&…