Zipkin+Sleuth调用链监控集成和使用

图片

背景与需求

  • 跨微服务的API调用发生异常,要求快速定位出问题出在哪里。
  • 跨微服务的API调用发生性能瓶颈,要求迅速定位出性能瓶颈。

集成

整体结构

整体机构为C/S模式,客户端(Sleuth)来监控采集调用链信息,汇报给服务端(Zipkin),通过Zipkin提供的web页面来展示链路调用和异常信息,统计链路图等功能。如下图:
图片

操作步骤

1. 引入依赖

<dependency>  <groupId>org.springframework.cloud</groupId>  <artifactId>spring-cloud-starter-zipkin</artifactId></dependency>

其中,zipkin中已经包含了 spring-cloud-starter-sleuth 的依赖,不需要再次引入。

2. 配置

日志输出级别

logging:  level:    com.alibaba.nacos: error    org.springframework.cloud.sleuth: debug

端口等配置

<pre class="code-snippet__js" data-lang="cs" style="margin: 0px; padding: 1em 1em 1em 0px; max-width: 1000%; box-sizing: border-box !important; overflow-wrap: break-word !important; overflow-x: auto; white-space: normal; -webkit-box-flex: 1; flex: 1 1 0%;">spring: zipkin: base-url: http://localhost:9411/</pre>

如果不配置端口等信息,默认会使用 localhost:9411 来访问服务,更多的Zipkin和Sleuth配置可以查看官方文档。

3. 启动Zipkin服务端

官网 https://zipkin.io/

有两种启动方式,第一种是使用Docker来启动:

docker run -d -p 9411:9411 openzipkin/zipkin

第二种是使用jar包的方式来启动:

curl -sSL https://zipkin.io/quickstart.sh | bash -sjava -jar zipkin.jar

4. 启动所有服务

如果不报错,日志输出中包含sleuth的信息,说明集成成功,然后访问Zipkin的服务来查看链路监控信息吧!

踩坑记录

报错情况一

2020-12-30 14:55:33.914 ERROR [log-manager,,,] 97245 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'queueMessageListener' defined in file [/Users/peiel/WorkSpace/youlu/myspringcloud/log-manager/target/classes/com/youlu/logmanager/queue/QueueMessageListener.class]: Initialization of bean failed; nested exception is java.lang.IllegalStateException: Need to invoke method 'listListener' found on proxy for target class 'QueueMessageListener' but cannot be delegated to target bean. Switch its visibility to package or protected.
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:603)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895)
  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
  at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
  at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
  at com.youlu.logmanager.LogManagerApplication.main(LogManagerApplication.java:23)
Caused by: java.lang.IllegalStateException: Need to invoke method 'listListener' found on proxy for target class 'QueueMessageListener' but cannot be delegated to target bean. Switch its visibility to package or protected.
  at org.springframework.aop.support.AopUtils.selectInvocableMethod(AopUtils.java:138)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.createRunnable(ScheduledAnnotationBeanPostProcessor.java:514)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.processScheduled(ScheduledAnnotationBeanPostProcessor.java:381)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.lambda$null$1(ScheduledAnnotationBeanPostProcessor.java:362)
  at java.lang.Iterable.forEach(Iterable.java:75)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.lambda$postProcessAfterInitialization$2(ScheduledAnnotationBeanPostProcessor.java:362)
  at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.postProcessAfterInitialization(ScheduledAnnotationBeanPostProcessor.java:361)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:431)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
  ... 16 common frames omitted

解决方式,根据日志查看,是日志服务中, @Scheduled 定时任务中方法报错,导致无法实例化该方法所属的类,仔细查看日志报错,其中说明了产生了访问可见性问题,修改该方法为public描述即可解决。

报错情况二

日志如下:

2021-03-15 15:01:11.023 ERROR [mdm,,,] 97454 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'resourceRolesMapInitRunner': Invocation of init method failed; nested exception is org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at com.youlu.mdm.MdmApplication.main(MdmApplication.java:17)
Caused by: org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:273)
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.convertLettuceAccessException(LettuceKeyCommands.java:809)
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.del(LettuceKeyCommands.java:128)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.del(DefaultedRedisConnection.java:82)
    at org.springframework.data.redis.core.RedisTemplate.lambda$delete$2(RedisTemplate.java:713)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:228)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
    at org.springframework.data.redis.core.RedisTemplate.delete(RedisTemplate.java:713)
    at com.youlu.common.service.impl.RedisServiceImpl.del(RedisServiceImpl.java:36)
    at com.youlu.mdm.service.impl.MdmResourceServiceImpl.initResourceRolesMap(MdmResourceServiceImpl.java:105)
    at com.youlu.mdm.component.ResourceRolesMapInitRunner.initResourceRolesMap(ResourceRolesMapInitRunner.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
    ... 19 common frames omitted
Caused by: io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:132)
    at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69)
    at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
    at com.sun.proxy.$Proxy161.del(Unknown Source)
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.del(LettuceKeyCommands.java:126)
    ... 34 common frames omitted
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
    at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:167)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

产生原因:仔细查看日志,发现其中 resourceRolesMapInitRunner 类执行报错,查看源代码,发现使用了 @PostConstruct 注解来执行方法,该注解的生命周期的触发是所属类Bean实例化后,此时我们 Sleuth 并没有实例化完成,导致执行该方法报错。解决方式:修改该类,让该方法在SpringBoot启动完成后执行(ApplicationRunner等),即可解决。

使用方式

打开我们线上的Zipkin(http://47.114.174.96:9411/zipkin),可以看到如下页面。

图片
图片

举个例子

比如说,我访问了接口,这个接口报错,返回500,这个时候,我们如何快速查询出这个接口具体的失败原因?第一步:筛选出最近的带有报错的链路。
图片

图片
如图,我们已经看到异常的原因和请求的path。
©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 212,686评论 6 492
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 90,668评论 3 385
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 158,160评论 0 348
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 56,736评论 1 284
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 65,847评论 6 386
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 50,043评论 1 291
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 39,129评论 3 410
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 37,872评论 0 268
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 44,318评论 1 303
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 36,645评论 2 327
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 38,777评论 1 341
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 34,470评论 4 333
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 40,126评论 3 317
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,861评论 0 21
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,095评论 1 267
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 46,589评论 2 362
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 43,687评论 2 351

推荐阅读更多精彩内容