tidb通过sqoop 导出数据到hadoop大表 OOM

🪐 TiDB 技术问题 TiFlash
1
  • 【TiDB 版本】：4.0
  • 【问题描述】：
    tidb中的单表数据通过sqoop 同步到hadoop，表数据量3千多w条，
    同步命令为：sqoop import
    –connect jdbc:mysql://1.xxxxx:4000/dbname1?tinyInt1isBit=false
    –username “$username”
    –password “$password”
    –table tablename1
    –delete-target-dir
    –fields-terminated-by ‘\001’
    –target-dir hdfs://haddoopserver/user/test/warehouse/ods.db/tablename1/sdt=19990101
    –columns “${columns}”
    –hive-drop-import-delims
    –null-string ‘\N’
    –null-non-string ‘\N’
    –hive-import
    –hive-database test_ods
    –hive-table tablename1
    –hive-partition-key sdt
    –hive-partition-value “19990101”
    .text-only,.text-card-text{white-space: pre;}.rich-text-paragraph{min-height: 15px;}

20/07/08 17:59:32 INFO mapreduce.Job: Task Id : attempt_1594105654926_43897_m_000002_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:170) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:223) at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:168) … 10 more

2

你好，

请详细描述你的环境和你的问题。

3

已重新说明

4

收到，在 tidb 服务器执行，dmesg | grep -i oom，看下返回结果

确认 tidb 服务器与 hadoop 服务器网络是否通畅

检查下 hadoop 的状态，是否正常访问和写入。

5

有部分表是可以的，表大小是1.5g写入成功 ，大表会报上面的错误。
使用dmesg|grep -i oom
发现是 有内存溢出的情况了、内容如下：
[691172.525928] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691172.993848] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691172.993879] [] oom_kill_process+0x254/0x3d0
[691172.993881] [] ? oom_unkillable_task+0xcd/0x120
[691172.993986] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691173.018832] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691173.018862] [] oom_kill_process+0x254/0x3d0
[691173.018864] [] ? oom_unkillable_task+0xcd/0x120
[691173.018951] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691173.089930] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691173.089968] [] oom_kill_process+0x254/0x3d0
[691173.089971] [] ? oom_unkillable_task+0xcd/0x120
[691173.090101] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691173.191010] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691173.191039] [] oom_kill_process+0x254/0x3d0
[691173.191041] [] ? oom_unkillable_task+0xcd/0x120
[691173.191129] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691173.227173] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691173.227200] [] oom_kill_process+0x254/0x3d0
[691173.227202] [] ? oom_unkillable_task+0xcd/0x120
[691173.227287] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691173.248439] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691173.248466] [] oom_kill_process+0x254/0x3d0
[691173.248468] [] ? oom_unkillable_task+0xcd/0x120
[691173.248553] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691173.811809] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691173.811842] [] oom_kill_process+0x254/0x3d0
[691173.811844] [] ? oom_unkillable_task+0xcd/0x120
[691173.811936] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.025821] oom_kill_process: 14 callbacks suppressed
[691177.025825] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.025853] [] oom_kill_process+0x254/0x3d0
[691177.025855] [] ? oom_unkillable_task+0xcd/0x120
[691177.025947] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.050817] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.050844] [] oom_kill_process+0x254/0x3d0
[691177.050846] [] ? oom_unkillable_task+0xcd/0x120
[691177.050934] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.182779] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.182809] [] oom_kill_process+0x254/0x3d0
[691177.182811] [] ? oom_unkillable_task+0xcd/0x120
[691177.182899] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.280794] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.280824] [] oom_kill_process+0x254/0x3d0
[691177.280826] [] ? oom_unkillable_task+0xcd/0x120
[691177.280940] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.295911] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.295935] [] oom_kill_process+0x254/0x3d0
[691177.295936] [] ? oom_unkillable_task+0xcd/0x120
[691177.296019] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.328786] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.328809] [] oom_kill_process+0x254/0x3d0
[691177.328811] [] ? oom_unkillable_task+0xcd/0x120
[691177.328891] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[691177.465830] tuned invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[691177.465862] [] oom_kill_process+0x254/0x3d0
[691177.465864] [] ? oom_unkillable_task+0xcd/0x120
[691177.465963] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
是不是可以理解为tidb导出数据大小依赖tidb服务器内存大小。导出数据大小一定要小于tidb服务器内存大小吗?

6

看下是否可以通过分页的形式，想数据导出到 hadoop ，tidb 这边会将 select 出来的结果集都放在内存中，譬如出现 3kw 的数据，select * ，可能就会出现 oom 问题。

7

好的，谢谢

8

ok，