Druid 连接池 连接TiDB 引发异常

系统版本

CentOS Linux release 7.6.1810 (Core)

TiDB 版本

TiDB-v3.0.3

集群节点分布

HostName IP DES ---- 内存
dev10 192.168.180.32 主控机 master 16G
dev11 192.168.180.33 监控机 TiDB, PD 16G
dev12 192.168.180.34 工作节点 TiDB, PD 16G
dev17 192.168.180.52 工作节点 TiKV1-1 TiKV1-2 TiKV1-3 32G
dev18 192.168.180.53 工作节点 TiDB, PD 32G

负载

HostName IP DES ---- 内存 端口
dev18 192.168.180.53 负载 HAProxy 32G 4600

haproxy.cfg

global
  log     127.0.0.1 local0
  maxconn 300000                # 最大同时300000连接
  daemon                        # 以daemon方式在后台运行

defaults
  # mode http                   # 默认的模式mode { tcp|http|health },tcp是4层,http是7层,health只会返回OK
  retries         3             # 连接后端服务器失败重试次数,超过3次后会将失败的后端服务器标记为不可用。
  timeout client  30s           # 客户端响应超时             30秒
  timeout server  30s           # server端响应超时           30秒
  timeout connect 5s            # 连接server端超时           5秒
  timeout check   10s           # 对后端服务器的检测超时时间 10秒

listen stats                    # 定义监控页面
  mode  http
  bind  *:1080                  # 绑定端口1080
  stats refresh 1s              # 每1秒更新监控数据
  stats uri /stats              # 访问监控页面的uri
  stats realm HAProxy Stats    # 监控页面的认证提示
  stats auth admin:654321       # 监控页面的用户名和密码

frontend tidb_front
  mode  tcp
  bind  *:4000                  # 监听4000端口
  default_backend tidb_back

backend tidb_back

  mode    tcp
  option  tcp-check
  balance static-rr

  server TiDB-Server-dev18 192.168.180.53:4000 check inter 10s rise 3 fall 3 weight 3
  server TiDB-Server-dev11 192.168.180.33:4000 check inter 10s rise 3 fall 3 weight 1
  server TiDB-Server-dev12 192.168.180.34:4000 check inter 10s rise 3 fall 3 weight 1

程序

程序框架 spring-boot 数据库连接池 Druid

我做了什么

正常向数据库发起查询请求 192.168.180.53:4600

异常

2019-10-17 16:45:35.666 ERROR 6832 --- [o-20602-exec-10] c.a.druid.pool.DruidPooledStatement      : CommunicationsException, druid version 1.1.12, jdbcUrl : jdbc:mysql://192.168.180.53:4600/dev2_hengrui?characterEncoding=utf8&useSSL=false, testWhileIdle true, idle millis 47703, minIdle 5, poolingCount 4, timeBetweenEvictionRunsMillis 60000, lastValidIdleMillis 47703, driver com.mysql.jdbc.Driver, exceptionSorter com.alibaba.druid.pool.vendor.MySqlExceptionSorter
2019-10-17 16:45:35.669 ERROR 6832 --- [o-20602-exec-10] com.alibaba.druid.pool.DruidDataSource   : discard connection

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 47,699 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2483)
	at com.mysql.jdbc.StatementImpl.executeUpdateInternal(StatementImpl.java:1552)
	at com.mysql.jdbc.StatementImpl.executeLargeUpdate(StatementImpl.java:2607)
	at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1480)
	at com.alibaba.druid.pool.DruidPooledStatement.executeUpdate(DruidPooledStatement.java:319)
	at org.springframework.jdbc.core.JdbcTemplate$1UpdateStatementCallback.doInStatement(JdbcTemplate.java:532)
	at org.springframework.jdbc.core.JdbcTemplate$1UpdateStatementCallback.doInStatement(JdbcTemplate.java:529)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408)
	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:544)
	at com.sinoeyes.web.rpc.dailyrawflow.DailySalRawPreStorageInterceptor.process(DailySalRawPreStorageInterceptor.java:44)
	at com.sinoeyes.web.rpc.dailyrawflow.DailySalRawPreStorageInterceptor$$FastClassBySpringCGLIB$$8cb5d550.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.sinoeyes.log.annotation.TuningMonitorLogger.aroundAdvisor(TuningMonitorLogger.java:79)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.sinoeyes.web.rpc.dailyrawflow.DailySalRawPreStorageInterceptor$$EnhancerBySpringCGLIB$$c81229a2.process(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.sinoeyes.web.serivces.utils.DymaicRpcUtils.rpcMethod(DymaicRpcUtils.java:27)
	at com.sinoeyes.common.excelimport.FileImportService.doStorageFile(FileImportService.java:248)
	at com.sinoeyes.common.excelimport.FileImportService$$FastClassBySpringCGLIB$$89cbeda7.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.sinoeyes.log.annotation.TuningMonitorLogger.aroundAdvisor(TuningMonitorLogger.java:79)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.sinoeyes.common.excelimport.FileImportService$$EnhancerBySpringCGLIB$$dbbee107.doStorageFile(<generated>)
	at com.sinoeyes.common.excelimport.FileImportController.doStorage(FileImportController.java:245)
	at com.sinoeyes.common.excelimport.FileImportController$$FastClassBySpringCGLIB$$172a7f6a.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.sinoeyes.log.annotation.TuningMonitorLogger.aroundAdvisor(TuningMonitorLogger.java:79)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.sinoeyes.common.excelimport.FileImportController$$EnhancerBySpringCGLIB$$c65b3ec6.doStorage(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:167)
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3011)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3469)
	... 146 common frames omitted

2019-10-17 16:45:35.717  INFO 6832 --- [o-20602-exec-10] o.s.b.f.xml.XmlBeanDefinitionReader      : Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
2019-10-17 16:45:35.896  INFO 6832 --- [o-20602-exec-10] o.s.jdbc.support.SQLErrorCodesFactory    : SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase, Hana]

结果

查询的结果偶尔会因为异常查不到数据,偶尔可以查询到数据

你好: 查看报错,看起来是无法访问服务. 192.168.180.53:4600是您配置的HA端口。他已经配置了可以连接33,32,53上的3个tidb对吧? 根据您反馈的,看起来有时可以成功,有时失败,我怀疑是否是在连接到某些节点时可以成功,有些失败,建议先直连一下每个tidb节点,看是否成功。 如果可以,能否先用HAproxy连接一个tidb,测试如果单独的tidb是否通过HAproxy能够成功,依次类推,看看是某个具体的tidb有问题?

你好: 如果多次测试,直连都是正常的,那么tidb应该是不存在问题的,问题应该在HAproxy.
为了排除tidb问题,请检查下tidb的日志,检查下是否tidb日志中有报错

没有错误 , 我也认为 是 HAproxy 但是不知道如何来解决

帮忙反馈下tidb的报错日志,谢谢

[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Unavailable desc = transport is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20173] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]
[2019/10/17 19:32:01.636 +08:00] [ERROR] [client_batch.go:278] ["batchRecvLoop error when receive"] [target=192.168.180.52:20171] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [stack="github.com/pingcap/tidb/store/tikv.(*batchCommandsClient).batchRecvLoop
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:278"]

就是这个,我查了咱们的tug资料,怎么好像说这个是允许的

原因找到了 是因为 HAProxy 超时时间 设置的太短,导致应用程序连接断开

你好,我也出现过这个问题,请问HAProxy的配置在哪修改?

配置文件存放在哪里是无所谓的,启动时指定即可。详情请参考 haproxy 官方文档 https://cbonte.github.io/haproxy-dconv/2.1/management.html#3

global
  maxconn 32                    # 最大同时32连接
  daemon                        # 以daemon方式在后台运行

defaults
  log     127.0.0.1 local0 debug       # [emerg, alert, crit, err, warning, notice, info, debug]
  # mode http                   # 默认的模式mode { tcp|http|health },tcp是4层,http是7层,health只会返回OK
  retries         3             # 连接后端服务器失败重试次数,超过3次后会将失败的后端服务器标记为不可用。
  timeout client  1h            # 客户端响应超时             1小时
  timeout server  1h            # server端响应超时           1小时
  timeout connect 1h            # 连接server端超时           1小时
  timeout check   10s           # 对后端服务器的检测超时时间 10秒

listen stats                    # 定义监控页面
  mode  http
  bind  *:1080                  # 绑定容器内的1080端口
  stats refresh 1s              # 每1秒更新监控数据
  stats uri /stats              # 访问监控页面的uri
  stats realm HAProxy\ Stats    # 监控页面的认证提示
  stats auth admin:654321       # 监控页面的用户名和密码

frontend tidb_front
  mode  tcp
  bind  *:4000                  # 监听容器内的4000端口
  default_backend tidb_back

backend tidb_back

  mode    tcp
  option  tcp-check
  balance roundrobin

  server TiDB-Server-test2 192.168.180.47:4000 check inter 10s rise 3 fall 3 weight 1
  server TiDB-Server-test3 192.168.180.48:4000 check inter 10s rise 3 fall 3 weight 2
  server TiDB-Server-test4 192.168.181.18:4000 check inter 10s rise 3 fall 3 weight 3

:+1: