一、配置azkaban-web-server配置文件
1. 记得指定executor.port,不然当executor重启时,会重新分配一个port,这样会导致之前的再跑任务,怎么也取消不了,会生成“幽灵任务”
2. 如果并发跑的任务过多,建议开启Multiple Executor Mode
3. 为了任务能够正常运行,建议开启队列
配置如下:
# Azkaban Executor settings
executor.port=39753
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
#set Hhould queue
azkaban.queueprocessing.enabled=true
azkaban.webserver.queue.size=100000
azkaban.activeexecutor.refresh.milisecinterval=50000
azkaban.activeexecutor.refresh.flowinterval=5
azkaban.executorinfo.refresh.maxThreads=5
二、配置azkaban-exec-server配置文件
1. 指定executor.port, executor.maxThreads,executor.flow.threads
2. 指定azkaban.webserver.url
配置如下:
azkaban.webserver.url=http://shucang-10.szanba.ren:7282
# Azkaban Executor settings
executor.port=39753
executor.maxThreads=150
executor.flow.threads=150
三、启动azkaban-exec-server
先把azkaban的executor启动,如果是Multiple Executor Mode,就把部署了executor的每台机器上的executor启动
启动脚本如下
./bin/start-exec.sh
四、激活executor
启动好azkaban的executor后,需要手动把每个executor进行激活;如下,host是每个executor所在主机名,port是executor指定的端口
curl http://host:port/executor?action=activate
五、启动azkaban-web-server
前面四个步骤完成后,web server直接启动就可以
启动脚本
./bin/start-web.sh