haproxy负载tidb 报8052错误

🪐 TiDB 技术问题
1

【 TiDB 使用环境】测试
【 TiDB 版本】5.4.0

【遇到的问题：问题现象及影响】
连接haproxy 代理会报8052错误，但连后端的tidb 4000是不报错的。以前也负载过没报错。现在报错了
【资源配置】
【附件：截图/日志/监控】

image
image695×823 20.5 KB

haproxy.cfg

listen tidb_cluster
    bind    0.0.0.0:3161
    mode    tcp
    option  tcplog
    balance roundrobin
    timeout client  3h
    timeout server  3h
    server  tidb1ser  10.34.89.30:4000 send-proxy  check  inter  5000  rise  2  fall  3
    server  tidb2ser  10.34.89.31:4000 send-proxy  check  inter  5000  rise  2  fall  3
    server  tidb3ser  10.34.89.32:4000 send-proxy  check  inter  5000  rise  2  fall  3

image
image1017×854 48.8 KB

2

这个报错 :thinking:，sequence？ 用了 sequence？资管感觉是什么 sequence 有问题？

3

配置文件里搜一下有没有88

4

搜那个配置文件啊？我们是从5.2.1 升级上来的5.4.0

find /tidb/app -type f -name “*.toml” | xargs grep “88”
没找到88

5

发下tidb的log看看呢

6

image
image940×364 13.6 KB

需要那个log

7

tidb.log

8

[2022/11/24 09:40:03.177 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55248: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:08.178 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55284: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:13.177 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55326: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:18.178 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55372: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:23.179 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55412: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:28.181 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55454: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:33.181 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55490: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:38.182 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55524: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:43.183 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55564: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:48.184 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55600: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]

9

这个日志看起来都是心跳的，没影响。试着模拟一下，再抓取日志

10

[2022/11/24 09:52:03.405 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32786: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:08.407 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32826: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:08.735 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[server:8052]invalid sequence 88 != 1”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:13.409 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32872: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:13.624 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[server:8052]invalid sequence 88 != 1”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:18.410 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32928: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]

image

11

看起来应该还是haproxy配置的问题，参考此处再检查下： HAProxy 在 TiDB 中的最佳实践 | PingCAP Docs
另外检查下防火墙、权限之类的。

12

proxy-protocol.networks: 这个参数在升级以后不见了。配上就好了

13

这是透传参数。