问题描述
在热更新的时候，经常发现日志里面出现"release_handler: cannot find top supervisor for application amqp_client"这个日志。
原因分析
erlang 热更新过程涉及到release_handler这个模块，release_handler又调用了release_handler_1这个模块，最终调用到code模块。观察每次生成的relup文件,比如这次的relup文件
<pre>
{"1.3.6",
[{"1.3.5",[],
[{load_object_code,{be3_logic,"1.3.6",
[mall_manager,mod_hero,mongo_mall,
statistics]}},
point_of_no_return,
{load,{mall_manager,brutal_purge,brutal_purge}},
{load,{mod_hero,brutal_purge,brutal_purge}},
{load,{mongo_mall,brutal_purge,brutal_purge}},
{suspend,[statistics]},
{load,{statistics,brutal_purge,brutal_purge}},
{code_change,up,[{statistics,[]}]},
{resume,[statistics]}]}],
[{"1.3.5",[],[point_of_no_return]}]}.
</pre>
热更新的过程，就是release_handler_1.erl解析这个文件，根据指令调用code:load(),code:purge()和erlang:suspend和resume相关进程的过程。
<pre>
eval({load_object_code,)...
eval({point_of_no_return,)...
eval({load,)...
eval({suspend,)...
....
</pre>
问题主要出在suspend这个原子的匹配：即需要suspend进程的时候。
suspend会调用get_supervised_procs()方法
<pre>
get_supervised_procs() ->
lists:foldl(
fun(Application, Procs) ->
get_master_procs(Application,
Procs,
application_controller:get_master(Application))
end,
[],
get_application_names()).
get_supervised_procs(_, Root, Procs, {ok, SupMod}) ->
get_procs(maybe_supervisor_which_children(Root, SupMod, Root), Root) ++
[{undefined, undefined, Root, [SupMod]} | Procs];
get_supervised_procs(Application, Root, Procs, {error, _}) ->
error_logger:error_msg("release_handler: cannot find top supervisor for "
"application wn", [Application]),
get_procs(maybe_supervisor_which_children(Root, Application, Root), Root) ++ Procs.
</pre>
get_supervised_procs(Application, Root, Procs, {error, _})这个分支，已经会打印错误信息了。
get_master_procs/1方法如下
<pre>
get_master_procs(Application, Procs, Pid) when is_pid(Pid) ->
{Root, AppMod} = application_master:get_child(Pid),
get_supervised_procs(Application, Root, Procs, get_supervisor_module(Root));
get_master_procs(, Procs, _) ->
Procs.
</pre>
get_supervisor_module/1方法如下
<pre>
get_supervisor_module(SupPid) ->
case catch supervisor:get_callback_module(SupPid) of
Module when is_atom(Module) ->
{ok, Module};
_Other ->
io:format("~w: reason: wn", [SupPid, _Other]),
{error, undefined}
end.
</pre>
如果supervisor:get_callback_module()走到了_Other分支，就会返回{error, undefined}，从而
get_master_procs()方法打印出我们看见的错误日志。
为什么supervisor:get_callback_module()会走到_Other分支呢？我在节点终端里面直接按照步骤执行如下：
<pre>
(s0@10.10.14.118)1> application_controller:get_master(amqp_client).
<0.1693.0>
(s0@10.10.14.118)2> {Root, _AppMod} = application_master:get_child(pid(0,1693,0)).
{<0.1695.0>,amqp_client}
(s0@10.10.14.118)4> supervisor:get_callback_module(pid(0,1695,0)).
** exception error: {badrecord,state}
</pre>
这里就出错了，被catch住，所以跑到了_Other分支里面。让我们看看<0.1695.0>这个进程的真容吧
<pre>
process_info(pid(0,1695,0)).
[{registered_name,amqp_sup},
{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.1724.0>,<0.1780.0>,<0.1808.0>,<0.1822.0>,
<0.1794.0>,<0.1752.0>,<0.1766.0>,<0.1738.0>,<0.1696.0>,
<0.1710.0>,<0.1694.0>]},
{dictionary,[{'$ancestors',[<0.1694.0>]},
{'$initial_call',{supervisor2,init,1}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.1693.0>},
{total_heap_size,4184},
{heap_size,1598},
{stack_size,9},
{reductions,11292},
{garbage_collection,[{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,1}]},
{suspending,[]}]
</pre>
注意{'$initial_call',{supervisor2,init,1}}]}。看了看amqp_sup，发现的类型是supervisor2，而不是supervisor，而supervisor2的state变量和supervisor的state变量的record定义是不是一样的，后者没有dynamic_restarts。
<pre>
supervisor的state变量
-record(state, {name,
strategy :: strategy(),
children = [] :: [child_rec()],
dynamics :: ?DICT(pid(), list()) | ?SET(pid()),
intensity :: non_neg_integer(),
period :: pos_integer(),
restarts = [],
dynamic_restarts = 0 :: non_neg_integer(),
module,
args}).
supervisor2的state变量
-record(state, {name,
strategy,
children = [],
dynamics = ?DICT:new(),
intensity,
period,
restarts = [],
module,
args}).
</pre>
而get_callback_module方法刚好用了模式匹配，这里向supervisor2调用，所以一定会报错
<pre>
get_callback_module(Pid) ->
{status, _Pid, {module, _Mod},
[_PDict, _SysState, _Parent, _Dbg, Misc]} = sys:get_status(Pid),
[_Header, _Data, {data, [{"State", State}]}] = Misc,
State#state.module.
</pre>
问题处理
这个问题不致命，在logstash加上对于"cannot find top supervisor for application amqp_client"的屏蔽
