背景与需求
- 跨微服务的API调用发生异常,要求快速定位出问题出在哪里。
- 跨微服务的API调用发生性能瓶颈,要求迅速定位出性能瓶颈。
集成
整体结构
整体机构为C/S模式,客户端(Sleuth)来监控采集调用链信息,汇报给服务端(Zipkin),通过Zipkin提供的web页面来展示链路调用和异常信息,统计链路图等功能。如下图:操作步骤
1. 引入依赖
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-zipkin</artifactId></dependency>
其中,zipkin中已经包含了 spring-cloud-starter-sleuth
的依赖,不需要再次引入。
2. 配置
日志输出级别
logging: level: com.alibaba.nacos: error org.springframework.cloud.sleuth: debug
端口等配置
<pre class="code-snippet__js" data-lang="cs" style="margin: 0px; padding: 1em 1em 1em 0px; max-width: 1000%; box-sizing: border-box !important; overflow-wrap: break-word !important; overflow-x: auto; white-space: normal; -webkit-box-flex: 1; flex: 1 1 0%;">spring:
zipkin:
base-url: http://localhost:9411/
</pre>
如果不配置端口等信息,默认会使用 localhost:9411
来访问服务,更多的Zipkin和Sleuth配置可以查看官方文档。
3. 启动Zipkin服务端
有两种启动方式,第一种是使用Docker来启动:
docker run -d -p 9411:9411 openzipkin/zipkin
第二种是使用jar包的方式来启动:
curl -sSL https://zipkin.io/quickstart.sh | bash -sjava -jar zipkin.jar
4. 启动所有服务
如果不报错,日志输出中包含sleuth的信息,说明集成成功,然后访问Zipkin的服务来查看链路监控信息吧!
踩坑记录
报错情况一
2020-12-30 14:55:33.914 ERROR [log-manager,,,] 97245 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'queueMessageListener' defined in file [/Users/peiel/WorkSpace/youlu/myspringcloud/log-manager/target/classes/com/youlu/logmanager/queue/QueueMessageListener.class]: Initialization of bean failed; nested exception is java.lang.IllegalStateException: Need to invoke method 'listListener' found on proxy for target class 'QueueMessageListener' but cannot be delegated to target bean. Switch its visibility to package or protected.
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:603)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at com.youlu.logmanager.LogManagerApplication.main(LogManagerApplication.java:23)
Caused by: java.lang.IllegalStateException: Need to invoke method 'listListener' found on proxy for target class 'QueueMessageListener' but cannot be delegated to target bean. Switch its visibility to package or protected.
at org.springframework.aop.support.AopUtils.selectInvocableMethod(AopUtils.java:138)
at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.createRunnable(ScheduledAnnotationBeanPostProcessor.java:514)
at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.processScheduled(ScheduledAnnotationBeanPostProcessor.java:381)
at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.lambda$null$1(ScheduledAnnotationBeanPostProcessor.java:362)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.lambda$postProcessAfterInitialization$2(ScheduledAnnotationBeanPostProcessor.java:362)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.postProcessAfterInitialization(ScheduledAnnotationBeanPostProcessor.java:361)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:431)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
... 16 common frames omitted
解决方式,根据日志查看,是日志服务中, @Scheduled
定时任务中方法报错,导致无法实例化该方法所属的类,仔细查看日志报错,其中说明了产生了访问可见性问题,修改该方法为public描述即可解决。
报错情况二
日志如下:
2021-03-15 15:01:11.023 ERROR [mdm,,,] 97454 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'resourceRolesMapInitRunner': Invocation of init method failed; nested exception is org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at com.youlu.mdm.MdmApplication.main(MdmApplication.java:17)
Caused by: org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:273)
at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.convertLettuceAccessException(LettuceKeyCommands.java:809)
at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.del(LettuceKeyCommands.java:128)
at org.springframework.data.redis.connection.DefaultedRedisConnection.del(DefaultedRedisConnection.java:82)
at org.springframework.data.redis.core.RedisTemplate.lambda$delete$2(RedisTemplate.java:713)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:228)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
at org.springframework.data.redis.core.RedisTemplate.delete(RedisTemplate.java:713)
at com.youlu.common.service.impl.RedisServiceImpl.del(RedisServiceImpl.java:36)
at com.youlu.mdm.service.impl.MdmResourceServiceImpl.initResourceRolesMap(MdmResourceServiceImpl.java:105)
at com.youlu.mdm.component.ResourceRolesMapInitRunner.initResourceRolesMap(ResourceRolesMapInitRunner.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
... 19 common frames omitted
Caused by: io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:132)
at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy161.del(Unknown Source)
at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.del(LettuceKeyCommands.java:126)
... 34 common frames omitted
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:167)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
产生原因:仔细查看日志,发现其中 resourceRolesMapInitRunner
类执行报错,查看源代码,发现使用了 @PostConstruct
注解来执行方法,该注解的生命周期的触发是所属类Bean实例化后,此时我们 Sleuth 并没有实例化完成,导致执行该方法报错。解决方式:修改该类,让该方法在SpringBoot启动完成后执行(ApplicationRunner等),即可解决。
使用方式
打开我们线上的Zipkin(http://47.114.174.96:9411/zipkin),可以看到如下页面。