springboot 集成impala连接池问题

HikariDataSource连接池

配置

datasource:
   type: com.zaxxer.hikari.HikariDataSource
   driver-class-name: com.cloudera.impala.jdbc41.Driver
   url: jdbc:impala://118.31.173.39:25001/bill_gypoc_0802
   hikari:
     minimumIdle: 5
     maximumPoolSize: 5

问题

2020-06-23 19:43:48,404 DEBUG (PoolBase.java:134)[HikariPool-1 housekeeper]- HikariPool-1 - Closing connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@34b395a2: (connection
 has passed maxLifetime)
2020-06-23 19:43:49,069 DEBUG (HikariPool.java:728)[HikariPool-1 connection adder]- HikariPool-1 - Added connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@273fe02c
2020-06-23 19:43:49,569 DEBUG (HikariPool.java:728)[HikariPool-1 connection adder]- HikariPool-1 - Added connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@6602ea17
2020-06-23 19:47:50,578 DEBUG (PoolBase.java:147)[HikariPool-1 connection closer]- HikariPool-1 - Closing connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@166d2158 failed
java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10060) Connection has been closed.
    at com.cloudera.impala.exceptions.ExceptionConverter.toSQLException(Unknown Source)
    at com.cloudera.impala.jdbc.common.SConnection.closeConnection(Unknown Source)
    at com.cloudera.impala.jdbc.common.SConnection.abortInternal(Unknown Source)
    at com.cloudera.impala.jdbc.common.SConnection.close(Unknown Source)
    at com.zaxxer.hikari.pool.PoolBase.quietlyCloseConnection(PoolBase.java:143)
    at com.zaxxer.hikari.pool.HikariPool.lambda$closeConnection$1(HikariPool.java:451)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2020-06-23 19:47:50,579 DEBUG (HikariPool.java:517)[HikariPool-1 connection closer]- HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2020-06-23 19:47:50,579 DEBUG (PoolBase.java:134)[HikariPool-1 connection closer]- HikariPool-1 - Closing connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@7af13fae: (conn
ection has passed maxLifetime)
2020-06-23 19:59:36,114 DEBUG (PoolBase.java:147)[HikariPool-1 housekeeper]- HikariPool-1 - Closing connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@34b395a2 failed
java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10060) Connection has been closed.
    at com.cloudera.impala.exceptions.ExceptionConverter.toSQLException(Unknown Source)
    at com.cloudera.impala.jdbc.common.SConnection.closeConnection(Unknown Source)
    at com.cloudera.impala.jdbc.common.SConnection.abortInternal(Unknown Source)
    at com.cloudera.impala.jdbc.common.SConnection.close(Unknown Source)
    at com.zaxxer.hikari.pool.PoolBase.quietlyCloseConnection(PoolBase.java:143)
    at com.zaxxer.hikari.pool.HikariPool.lambda$closeConnection$1(HikariPool.java:451)
    at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
    at com.zaxxer.hikari.pool.HikariPool.closeConnection(HikariPool.java:450)
    at com.zaxxer.hikari.pool.HikariPool.softEvictConnection(HikariPool.java:615)
    at com.zaxxer.hikari.pool.HikariPool.lambda$createPoolEntry$2(HikariPool.java:486)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2020-06-23 19:59:36,115 DEBUG (HikariPool.java:517)[HikariPool-1 housekeeper]- HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2020-06-23 19:59:36,115 WARN (HikariPool.java:787)[HikariPool-1 housekeeper]- HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=15m47s712ms92µs886ns).
2020-06-23 19:59:36,116 DEBUG (HikariPool.java:421)[HikariPool-1 housekeeper]- HikariPool-1 - Pool stats (total=4, active=0, idle=4, waiting=0)
2020-06-23 19:59:36,116 DEBUG (HikariPool.java:517)[HikariPool-1 housekeeper]- HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2020-06-23 19:59:36,116 DEBUG (PoolBase.java:134)[HikariPool-1 housekeeper]- HikariPool-1 - Closing connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@1db905ff: (connection
 has passed maxLifetime)
2020-06-23 19:59:36,446 DEBUG (HikariPool.java:728)[HikariPool-1 connection adder]- HikariPool-1 - Added connection com.cloudera.impala.impala.jdbc41.ImpalaJDBC41Connection@4bcd29fe

连接达到最大生命周期时间，关闭连接超时导致线程池阻塞。此时访问数据库也会hold不返回。

解决方法

添加参数：

maxLifetime: 0

代表每个连接的生命周期无限长，目前测试没有发现问题。

druid连接池

配置

  datasource:
    type: com.alibaba.druid.pool.DruidDataSource
    driver-class-name: com.cloudera.impala.jdbc41.Driver
    url: jdbc:impala://118.31.173.39:25001/bill_gypoc_0802

问题

java.util.concurrent.ExecutionException: org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL [SELECT cust_code FROM `bill_gypoc_0802`.`m_diag_tag_cust` WHERE busi_date=? AND tag_code=? AND tag_val='4']; [Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Broken pipe (Write failed).; nested exception is java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Broken pipe (Write failed).
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
    at com.dengtacj.personasapi.manager.CustomTagManager.listCustomCode(CustomTagManager.java:75)
    at com.dengtacj.personasapi.service.CustomTagServiceImpl.listCustomWithTagQry(CustomTagServiceImpl.java:68)
    at com.dengtacj.personasapi.controller.CustomTagController.listCustomWithTag(CustomTagController.java:31)
    at com.dengtacj.personasapi.controller.CustomTagController$$FastClassBySpringCGLIB$$cff60dc1.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
    at com.dengtacj.personasapi.aop.WebLogAspect.doAround(WebLogAspect.java:75)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
    at com.dengtacj.personasapi.controller.CustomTagController$$EnhancerBySpringCGLIB$$56c3335a.listCustomWithTag(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

com.cloudera.impala.jdbc41.DataSource

配置

    type: com.cloudera.impala.jdbc41.DataSource
    driver-class-name: com.cloudera.impala.jdbc41.Driver
    url: jdbc:impala://118.31.173.39:25001/bill_gypoc_0802

使用cloudera驱动一起提供的官方datasource，这个其实并没有使用连接池，每次访问时候建立连接，完成后立即关闭连接。

结论

com.cloudera.impala.jdbc41.Driver 可能是存在一些问题，例如连接关闭长时间不释放。导致主流的连接池使用这个驱动会遇到问题。目前从有限测试看HikariDataSource保持连接一直不断，暂时比较稳定。使用cloudera提供的Datasource，没有使用连接池，功能暂时是没有问题，性能和保持长连接的访问要差一些。

©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 213,417评论 6赞 492
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 90,921评论 3赞 387
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 158,850评论 0赞 349
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 56,945评论 1赞 285
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 66,069评论 6赞 385
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 50,188评论 1赞 291
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 39,239评论 3赞 412
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 37,994评论 0赞 268
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 44,409评论 1赞 304
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 36,735评论 2赞 327
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 38,898评论 1赞 341
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 34,578评论 4赞 336
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 40,205评论 3赞 317
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,916评论 0赞 21
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,156评论 1赞 267
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 46,722评论 2赞 363
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 43,781评论 2赞 351