11:19 和15:23 都是这个大sql跑挂了重新启动 在此之间没有执行过这个大sql。
[tidb@prod-tidb-tidb001 log]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 9.5G 28G 26% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 580K 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/vdb1 197G 11G 177G 6% /alidata
tmpfs 1.6G 0 1.6G 0% /run/user/1002
[tidb@prod-tidb-tidb001 log]$ free -h
total used free shared buff/cache available
Mem: 15G 1.4G 12G 1.2M 1.7G 13G
Swap: 0B 0B 0B
[tidb@prod-tidb-tidb001 log]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
Stepping: 4
CPU MHz: 2499.984
BogoMIPS: 4999.96
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 33792K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 spec_ctrl intel_stibp
top - 17:14:34 up 53 days, 49 min, 2 users, load average: 2.18, 2.07, 2.14
Tasks: 169 total, 1 running, 168 sleeping, 0 stopped, 0 zombie
%Cpu(s): 25.7 us, 2.0 sy, 0.0 ni, 72.1 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 16265764 total, 13034988 free, 1477740 used, 1753036 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 14465576 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15073 tidb 20 0 1726584 557100 21412 S 216.7 3.4 167:13.62 tidb-server
1015 tidb 20 0 7210364 305392 7276 S 2.0 1.9 1144:12 prometheus
1398 root 20 0 951640 215048 2680 S 0.7 1.3 24:37.84 rsyslogd
642 root 20 0 240424 52916 52436 S 0.0 0.3 29:58.61 systemd-journal
12050 tidb 20 0 2449656 42092 8376 S 0.0 0.3 259:02.82 grafana-server
1170 root 20 0 1950296 30564 4988 S 0.0 0.2 222:34.38 exe
1394 root 20 0 659796 15508 2060 S 0.7 0.1 87:25.51 node_exporter
7403 tidb 20 0 124168 13592 3056 S 0.0 0.1 84:23.04 alertmanager
1563 root 20 0 245280 12384 1488 S 0.3 0.1 124:21.74 ilogtail
1392 root 20 0 573920 11244 180 S 0.0 0.1 7:12.92 tuned
10407 root 20 0 687156 9396 1916 S 0.0 0.1 31:06.23 falcon-agent
1086 root 20 0 309724 9112 0 S 0.0 0.1 0:02.09 CmsGoAgent.linu
7605 tidb 20 0 114380 8592 2072 S 0.0 0.1 19:41.75 pushgateway
15204 root 10 -10 129272 8576 3464 S 3.0 0.1 329:19.02 AliYunDun
1080 polkitd 20 0 612352 8040 364 S 0.0 0.0 0:18.36 polkitd
6961 tidb 20 0 22184 7544 692 S 0.3 0.0 296:10.99 blackbox_export
18546 root 20 0 154656 5448 4148 S 0.0 0.0 0:00.00 sshd
24031 root 10 -10 2176704 5172 936 S 0.0 0.0 5:42.58 AliHips
19408 root 20 0 241172 4616 3452 S 0.0 0.0 0:00.00 sudo
18576 root 20 0 241172 4612 3452 S 0.0 0.0 0:00.00 sudo
18672 tidb 20 0 116628 3392 1820 S 0.0 0.0 0:00.05 bash