mxnet分布式1
可能的阻塞原因
启动分布式的时候一开始经常程序阻塞住，自以为一切都按照官方的操作了，从表面现象看发射机在启动了launcher.py进程后，shell停住，这个是最让人头疼的，这种情况首先确保：
- 每一台机器上的环境一样，包括代码路径，python环境
- 要启动的进程是否已经存在，如果已经存在，先杀死它们
- 防火墙是否已经关闭
- 两台机器是否能免密ssh了
启动方式
-
通过官方提供的launcher.py启动
参考：https://github.com/apache/incubator-mxnet/tree/master/example/image-classification
为了看明白其中的过程，看一种别的启动方式
首先启动scheduler，scheduler进程会阻塞等待，再启动两个server，每个server都指定了PS的IP地址，最后启动两个worker，整个分布式程序开始启动运行，worker的shell动起来
针对mnist
sch---export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_ROLE=scheduler; export DMLC_PS_ROOT_PORT=9001; export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=2;
cd /path/to;
python train_mnist.py --kv-store dist_sync
ps1---export DMLC_SERVER_ID=0; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_ROLE=server; export DMLC_PS_ROOT_PORT=9001; export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=2;
cd /path/to;
python train_mnist.py --kv-store dist_sync
ps2---export DMLC_SERVER_ID=1; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_ROLE=server; export DMLC_PS_ROOT_PORT=9001; export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=2
cd /path/to;
python train_mnist.py --kv-store dist_sync
wk1---export DMLC_WORKER_ID=0; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_ROLE=worker; export DMLC_PS_ROOT_PORT=9001; export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=2
cd /path/to;
python train_mnist.py --kv-store dist_sync
wk2---export DMLC_WORKER_ID=2; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_ROLE=worker; export DMLC_PS_ROOT_PORT=9001; export DMLC_NUM_WORKER=2; export DMLC_NUM_SERVER=2
cd /path/to;
python train_mnist.py --kv-store dist_sync
启动过程分析
在x.x.x.x/x两台机器上做实验
启动脚本：
python ../../tools/launch.py -n 2 --launcher ssh -H hosts `which python` train_mnist.py --kv-store=dist_sync
启动后两台机器上的启动的进程分析
- 发射机
/home/xxx/anaconda2/envs/ps_lite/bin/python train_mnist.py --kv-store=dist_sync这条命令执行了3次，第一次是parameter server启动的scheduler进程，由tracker的pserver = PSTracker(hostIP=hostIP, cmd=pscmd, envs=envs)代码启动，scheduler进程由PSTtacker的构造函数启动，另外两个是由发射机ssh启动的server和worker进程，以上所有进程启动都是用异步线程启动
ssh -o StrictHostKeyChecking=no x.x.x.x -p 22 export LD_LIBRARY_PATH=.::/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/xxx/xxx-workspace/cuda-8.0-cudnn-6.0/lib64; export DMLC_ROLE=server; export DMLC_PS_ROOT_PORT=9091; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_NUM_SERVER=2; export DMLC_NUM_WORKER=2; cd /path/to/example/image-classification/; `which python` train_mnist.py --kv-store=dist_sync
这个进程起了四次，分别是向两个worker和两个server发送ssh进程，IP从hosts文件读取，PS都是x.x.x.x这台机器
bash -c export LD_LIBRARY_PATH=.::/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/xxx/xxx-workspace/cuda-8.0-cudnn-6.0/lib64; export DMLC_ROLE=server; export DMLC_PS_ROOT_PORT=9092; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_NUM_SERVER=2; export DMLC_NUM_WORKER=2; cd /path/to/example/image-classification/; `which python` train_mnist.py --kv-store=dist_sync
这个进程起了2次，接收到发射机发送的两次请求，分别启动server进程和worker进程
- worker节点
/home/xxx/anaconda2/envs/ps_lite/bin/python train_mnist.py --kv-store=dist_sync这条命令执行了2次
bash -c export LD_LIBRARY_PATH=.::/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/xxx/xxx-workspace/cuda-8.0-cudnn-6.0/lib64; export DMLC_ROLE=worker; export DMLC_PS_ROOT_PORT=9092; export DMLC_PS_ROOT_URI=x.x.x.x; export DMLC_NUM_SERVER=2; export DMLC_NUM_WORKER=2; cd /path/to/example/image-classification/; `which python` train_mnist.py --kv-store=dist_sync
这个进程启动了2次， 接收到发射机发送的两次请求，分别启动server进程和worker进程
启动后的参数
Namespace(archives=[], auto_file_cache=True, cluster='ssh', command=['`which', 'python`', 'train_mnist.py', '--kv-store=dist_sync'], env=[], files=[], hdfs_tempdir='/tmp', host_file='hosts', host_ip=None, jobname=None, kube_namespace='default', kube_server_image='mxnet/python', kube_server_template=None, kube_worker_image='mxnet/python', kube_worker_template=None, log_file=None, log_level='INFO', mesos_master=None, num_servers=2, num_workers=2, queue='default', server_cores=1, server_memory='1g', server_memory_mb=1024, sge_log_dir=None, ship_libcxx=None, slurm_server_nodes=None, slurm_worker_nodes=None, sync_dst_dir='None', worker_cores=1, worker_memory='1g', worker_memory_mb=1024, yarn_app_classpath=None, yarn_app_dir='/path/to/tools/../dmlc-core/tracker/dmlc_tracker/../yarn')
上面一堆参数中只有num_workers, num_servers,cluseter,host_file,sync_dst_dir,command是从外部给出，其他的参数从
try:
from dmlc_tracker import opts
except ImportError:
print("Can't load dmlc_tracker package. Perhaps you need to run")
print(" git submodule update --init --recursive")
raise
dmlc_opts = opts.get_opts(args)
中最后一行加载进来, opt.py中定义了很多参数parser
ssh.py->submit(args)->tracker.py:submit()->fun_submit->ssh.py:submit():ssh_submit()
hosts对象包装了hosts文件的IP地址和对应的端口
在ssh.py的方法ssh_submit()方法中，for语句中依次从hosts文件中拿到IP启动server和worker
代码分析
只试过ssh的启动方式，目前只看到python层的代码，看到启动了各自的进程，主要就四个类，launcher调用ssh.py，ssh.py调用tracker.py依次启动scheduler,server,worker进程
