JStorm的搭建文档
1、下载jstorm的jar包
https://github.com/alibaba/jstorm/releases
2、解压jstorm的包
tar -xvf jstorm-2.4.0.tgz
mv jstorm-2.4.0 jstorm
修改环境变量：
vi /etc/profile
export JSTORM_HOME=/usr/local/jstorm
export PATH=$PATH:$JSTORM_HOME/bin
生效 source /etc/profile
2、上传并修改storm.yaml
mv storm.yaml storm.yaml.bak
vi storm.yaml
nimbus.host: 表示nimbus的地址， 填写ip
storm.zookeeper.servers: - "zk-datanode-01" - "zk-datanode-02" - "zk-datanode-03" nimbus.host: "192.168.46.160" storm.zookeeper.root: "/jstorm" storm.local.dir: "/usr/local/jstorm/data" supervisor.slots.ports: - 6800 - 6801 - 6802 - 6803
3、部署到其他节点
scp -r jstorm zk-datanode-02:/usr/local/
scp -r jstorm zk-datanode-03:/usr/local/
4、启动storm
在nimbus 节点上执行
nohup jstorm nimbus & 查看$JSTORM_HOME/logs/nimbus.log检查有无错误
在supervisor节点上执行
nohup jstorm supervisor & 查看$JSTORM_HOME/logs/supervisor.log检查有无错误
若没有配置环境变量
nohup bin/jstorm supervisor &
5、webUI的部署
必须使用tomcat 7.x版本， 注意不要忘记拷贝 ~/.jstorm/storm.yaml
Web UI 可以和Nimbus不在同一个节点
mkdir ~/.jstorm cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm cd /usr/local/tomcat cd webapps cp $JSTORM_HOME/jstorm-ui-2.4.0.war ./ mv ROOT ROOT.old ln -s jstorm-ui-2.4.0 ROOT cd ../bin ./startup.sh
访问http://192.168.46.160:8080/ 即可查看到jstorm的界面
6、WebUI 增加新集群
注意： web ui使用的版本必须和集群中JStorm最高的版本一致
在运行Web UI的机器上， 修改~/.jstorm/storm.yaml
把默认的ui.clusters注释给去掉， 补充
# UI MultiCluster # Following is an example of multicluster UI configuration ui.clusters: - { name: "jstorm.share", zkRoot: "/jstorm", zkServers: [ "zk-datanode-01", "zk-datanode-02", "zk-datanode-03"], zkPort: 2181, } - { name: "jstorm.bu1", zkRoot: "/jstorm.dw", zkServers: [ "10.125.100.101", "10.125.100.101", "10.125.100.101"], zkPort: 2181, } 解释一下 - { name: "jstorm.bu1", --- 这个集群的名字， 每个集群的名字必须不一样 zkRoot: "/jstorm.dw", --- 这个集群 zk的根节点，可以参考$JSTORM_HOME/con/storm.yaml 中“storm.zookeeper.root” 字段 zkServers: [ "10.125.100.101", "10.125.100.101", "10.125.100.101"], -- zk 机器列表 zkPort: 2181, -- zk 的客户端端口 }
提交jstorm的任务
jstorm jar /usr/local/jstormpack/JstormTest-0.0.1-SNAPSHOT.jar com.dinpay.bdp.rcp.mywordcount.WordCountTopologMain WordCountTopolog
停止jstorm的任务
jstorm kill KTTAndNormalTopology
nimbus.childopts: "-Xms1g -Xmx1g -Xmn512m -XX:PermSize=256m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000" worker.childopts: "-Xms512m -Xmx512m -Xmn256m -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=65" ui.childopts: "-Xms768m -Xmx768m -Xmn256m -XX:PermSize=96m -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000" supervisor.childopts: "-Xms2g -Xmx2g -Xmn1g -XX:PermSize=256m -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000"