sync-diff-inspector 使用range分割数据时报错panic: runtime error: index out of range [0] with length 0

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:tidbv3.0.9
  • 【问题描述】:
[2020/07/14 20:45:49.954 +08:00] [INFO] [diff.go:199] ["find matched source tables"] ["source tables"="[{\"instance-id\":\"source-1\",\"Schema\":\"id-mapping-2\",\"Table\":\"t_uid_relation\"}]"] ["target schema"=id-mapping-2] [table=t_uid_relation]
[2020/07/14 20:45:50.902 +08:00] [INFO] [main.go:71] ["check data finished"] [cost=977.193302ms]
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/pingcap/tidb-tools/pkg/diff.splitRangeByRandom(0xc0003d4000, 0xc000096d20, 0x10, 0xc000353a70, 0xc, 0xc000378c20, 0xe, 0xc0004d6030, 0x2, 0x2, ...)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:268 +0x16c1
github.com/pingcap/tidb-tools/pkg/diff.(*bucketSpliter).getChunksByBuckets(0xc000268ec0, 0xc000042088, 0xc0003d4000, 0xc000353a70, 0xc, 0xc000378c20)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:362 +0x11f5
github.com/pingcap/tidb-tools/pkg/diff.(*bucketSpliter).split(0xc000268ec0, 0xc0000ceb40, 0xc000439040, 0x2, 0x7, 0x3e8, 0xc0003af184, 0x4b, 0x0, 0x0, ...)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:303 +0x116
github.com/pingcap/tidb-tools/pkg/diff.getChunksForTable(0xc0000ceb40, 0xc000439040, 0x2, 0x7, 0x3e8, 0xc0003af184, 0x4b, 0x0, 0x0, 0xc000040101, ...)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:386 +0x1e1
github.com/pingcap/tidb-tools/pkg/diff.SplitChunks(0x1a6d4a0, 0xc000042088, 0xc0000ceb40, 0x0, 0x0, 0xc0003af184, 0x4b, 0x3e8, 0x0, 0x0, ...)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:449 +0x1d8
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).CheckTableData(0xc000164000, 0x1a6d4a0, 0xc000042088, 0xc0003c2300, 0x0, 0x0)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:230 +0x7eb
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).Equal(0xc000164000, 0x1a6d4a0, 0xc000042088, 0xc0000844b0, 0x0, 0x0, 0xc000353a70)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:145 +0x17a
main.(*Diff).Equal(0xc0003c0000, 0x0, 0x0)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/diff.go:416 +0x7ec
main.checkSyncState(0x1a6d4a0, 0xc000042088, 0xc0000e98c0, 0x0)
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:79 +0x279
main.main()
	/home/jenkins/agent/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:62 +0x4eb
# Diff Configuration.

######################### Global config #########################

# The log level. You can set it to "info" or "debug".
log-level = "info"

# sync-diff-inspector divides the data into multiple chunks based on the primary key,
# unique key, or the index, and then compares the data of each chunk.
# Uses "chunk-size" to set the size of a chunk.
chunk-size = 1000

# The number of goroutines created to check data
check-thread-count = 4

# The proportion of sampling check. If you set it to 100, all the data is checked.
sample-percent = 100

# If enabled, the chunk's checksum is calculated and data is compared by checksum.
# If disabled, data is compared line by line.
use-checksum = true

# If it is set to true, data is checked only by calculating checksum. Data is not checked after inspection, even if the upstream and downstream checksums are inconsistent.
only-use-checksum = false

# Whether to use the checkpoint of the last check. If it is enabled, the inspector only checks the last unverified chunks and chunks that failed the verification.
use-checkpoint = true

# If it is set to true, data check is ignored.
# If it is set to false, data is checked.
ignore-data-check = false

# If it is set to true, the table struct comparison is ignored.
# If set to false, the table struct is compared.
ignore-struct-check = false

# The name of the file which saves the SQL statements used to repair data
fix-sql-file = "fix.sql"

######################### Tables config #########################

# If you need to compare the data of a large number of tables with different schema names or table names, use the table-rule to configure the mapping relationship. You can configure the mapping rule only for the schema or table, or you can also configure the mapping rules for both the schema and table.
#[[table-rules]]
    # schema-pattern and table-pattern support the wildcard *?
    # schema-pattern = "test_*"
    # table-pattern = "t_*"
    # target-schema = "test"
    # target-table = "t"

# Configures the tables of the target database that need to be compared.
[[check-tables]]
    # The name of the schema in the target database
    schema = "id-mapping-2"

    # The list of tables that need to be checked in the target database
    tables = ["t_uid_relation"]

    # Supports using regular expressions to configure tables to be checked.
    # You need to start with '~'. For example, the following configuration checks
    # all the tables with the prefix 'test' in the table name.
    # tables = ["~^test.*"]
    # The following configuration checks all the tables in the database.
    # tables = ["~^"]

# Special configuration for some tables
# The configured table must be included in "check-tables'.
#[[table-config]]
#    # The name of the schema in the target database
#    schema = "id-mapping"
#
#    # The table name
#    table = "t_uid_relation"
#
#    # Specifies the column used to divide data into chunks. If you do not configure it,
#    # sync-diff-inspector chooses an appropriate column (primary key, unique key, or a field with index).
#    index-field = "id"
#
#    # Specifies the range of the data to be checked
#    # It needs to comply with the syntax of the WHERE clause in SQL.
#    range = "update_time < 2020-07-10 14:44:11 and update_time < 2020-07-07 14:44:11"
#
#    # Sets it to "true" when comparing the data of multiple sharded tables
#    # with the data of the combined table.
#    is-sharding = false

    # The collation of the string type of data might be inconsistent in some conditions.
    # You can specify "collation" to guarantee the order consistency.
    # You need to keep it corresponding to the "charset" setting in the database.
    # collation = "latin1_bin"

    # Ignores checking some columns such as some types (json, bit, blob, etc.)
    # that sync-diff-inspector does not currently support.
    # The floating-point data type behaves differently in TiDB and MySQL. You can use
    # `ignore-columns` to skip checking these columns.
    # ignore-columns = ["name"]

# Configuration example of comparing two tables with different schema names and table names.
[[table-config]]
    # The name of the target schema
    schema = "id-mapping-2"

    # The name of the target table
    table = "t_uid_relation"

    # Sets it to "false" in non-sharding scenarios.
    is-sharding = false

    #range = "update_time < '2020-07-10 14:44:11' AND update_time > '2020-07-09 14:44:11'"
    range = 'update_time < "2020-07-10 14:44:11" and update_time > "2020-07-07 14:44:11"'
    index-field = "id"
    # Configuration of the source data
    [[table-config.source-tables]]
        # The instance ID of the source schema
        instance-id = "source-1"
        # The name of the source schema
        schema = "id-mapping"
        # The name of the source table
        table  = "t_uid_relation"

######################### Databases config #########################

# Configuration of the source database instance
[[source-db]]
    host = "1.1.1.1"
    port = 4000
    user = "root"
    #password = 
    # The instance ID of the source database, the unique identifier of a database instance
    instance-id = "source-1"
    # Uses the snapshot function of TiDB.
    # If enabled, the history data is used for comparison.
    # snapshot = "2016-10-08 16:45:26"
    # Sets the `sql-mode` of the database to parse table structures.
    # sql-mode = ""

# Configuration of the target database instance
[target-db]
    host = "1.1.1.1"
    port = 4000
    user = "root"
    #password = 
    # Uses the snapshot function of TiDB.
    # If enabled, the history data is used for comparison.
    # snapshot = "2016-10-08 16:45:26"
    # Sets the `sql-mode` of the database to parse table structures.
    # sql-mode = ""

tidb的两个表中查询关于update_time < “2020-07-10 14:44:11” and update_time > "2020-07-07 14:44:11的数据都有700多条

求解~~~~~

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

你好可以提供下sync-diff-inspector 版本

这个问题已经修复了,可以使用 3.0.16 版本的 sync-diff-inspector 试下
https://download.pingcap.org/tidb-toolkit-v3.0.16-linux-amd64.tar.gz

好的 我来验证下

用的3.0.16版本 对比完成后有如下warnning

bin/sync_diff_inspector --config=sync-diff.conf
[2020/07/15 11:29:57.214 +08:00] [INFO] [printer.go:45] ["Welcome to sync_diff_inspector"] ["Release Version"=1.0.0~rc2+git.763078d] ["Git Commit Hash"=763078d527f5deb7c816a0fa6d0521f90ba27c41] ["Git Branch"=HEAD] ["UTC Build Time"="2020-07-03 11:15:33"] ["Go Version"=go1.13]
[2020/07/15 11:29:57.214 +08:00] [INFO] [main.go:62] [config="{\"log-level\":\"info\",\"source-db\":[{\"host\":\"10.11.34.251\",\"port\":4000,\"user\":\"root\",\"schema\":\"\",\"snapshot\":\"\",\"sql-mode\":\"\",\"instance-id\":\"source-1\",\"Conn\":null}],\"target-db\":{\"host\":\"10.11.34.251\",\"port\":4000,\"user\":\"root\",\"schema\":\"\",\"snapshot\":\"\",\"sql-mode\":\"\",\"instance-id\":\"target\",\"Conn\":null},\"chunk-size\":1000,\"sample-percent\":100,\"check-thread-count\":4,\"use-checksum\":true,\"only-use-checksum\":false,\"fix-sql-file\":\"fix.sql\",\"check-tables\":[{\"schema\":\"id-mapping-2\",\"tables\":[\"t_id_mapping\"],\"exclude-tables\":null}],\"table-rules\":null,\"table-config\":[{\"instance-id\":\"\",\"Schema\":\"id-mapping-2\",\"Table\":\"t_id_mapping\",\"IgnoreColumns\":null,\"Fields\":\"\",\"Range\":\" id \\u003c 203108858 and id \\u003e 203105858 \",\"IsSharding\":false,\"SourceTables\":null,\"TargetTableInfo\":null,\"Collation\":\"\"},{\"instance-id\":\"\",\"Schema\":\"id-mapping-2\",\"Table\":\"t_id_mapping\",\"IgnoreColumns\":null,\"Fields\":\"\",\"Range\":\"update_time \\u003c '2020-07-10 14:44:11' AND update_time \\u003e '2020-07-09 14:44:11'\",\"IsSharding\":false,\"SourceTables\":[{\"instance-id\":\"source-1\",\"Schema\":\"id-mapping\",\"Table\":\"t_id_mapping\"}],\"TargetTableInfo\":null,\"Collation\":\"\"}],\"ignore-struct-check\":false,\"ignore-data-check\":false,\"use-checkpoint\":true,\"ConfigFile\":\"sync-diff.conf\",\"PrintVersion\":false}"]
[2020/07/15 11:29:57.247 +08:00] [INFO] [diff.go:214] ["find matched source tables"] ["source tables"="[{\"instance-id\":\"source-1\",\"Schema\":\"id-mapping-2\",\"Table\":\"t_id_mapping\"}]"] ["target schema"=id-mapping-2] [table=t_id_mapping]
[2020/07/15 11:30:32.219 +08:00] [WARN] [chunk.go:394] ["use tidb bucket information to get chunks failed, will split chunk by random again"] ["get chunk"=0] [error="analyze value 203108858 failed"] [errorVerbose="analyze value 203108858 failed\
github.com/pingcap/tidb-tools/pkg/dbutil.AnalyzeValuesFromBuckets\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/dbutil/common.go:497\
github.com/pingcap/tidb-tools/pkg/diff.(*bucketSpliter).getChunksByBuckets\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:330\
github.com/pingcap/tidb-tools/pkg/diff.(*bucketSpliter).split\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:306\
github.com/pingcap/tidb-tools/pkg/diff.getChunksForTable\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:389\
github.com/pingcap/tidb-tools/pkg/diff.SplitChunks\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/diff/chunk.go:452\
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).CheckTableData\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:242\
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).Equal\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:149\
main.(*Diff).Equal\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/diff.go:431\
main.checkSyncState\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:83\
main.main\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:65\
runtime.main\
\t/usr/local/go/src/runtime/proc.go:203\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/07/15 11:30:35.115 +08:00] [INFO] [diff.go:463] ["checksum is equal"] [table=`id-mapping-2`.`t_id_mapping`] [where="(TRUE AND update_time < '2020-07-10 14:44:11' AND update_time > '2020-07-09 14:44:11')"] [args=null] [checksum=738028086]
[2020/07/15 11:30:35.218 +08:00] [INFO] [checkpoint.go:237] ["summary info"] [instance_id=target] [schema=id-mapping-2] [table=t_id_mapping] ["chunk num"=1] ["success num"=1] ["failed num"=0] ["ignore num"=0]
[2020/07/15 11:30:35.246 +08:00] [INFO] [report.go:80] ["check result summary"] ["check passed num"=1] ["check failed num"=0]
[2020/07/15 11:30:35.246 +08:00] [INFO] [report.go:87] ["table check result"] [schema=id-mapping-2] [table=t_id_mapping] ["struct equal"=true] ["data equal"=true]
[2020/07/15 11:30:35.246 +08:00] [INFO] [main.go:75] ["check data finished"] [cost=38.031499854s]
[2020/07/15 11:30:35.246 +08:00] [INFO] [main.go:69] ["check pass!!!"]

warning可忽略?

可以忽略

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。