DataX 使用过程常见问题汇总

作者 csdn代下载 2023年12月01日 14:21 阅读 170

一、启动报错

datax 连接mysql报错,datax  No suitable driver found for ["jdbc:mysql:

报的驱动错误,如果已经安装了mysql相关的插件,一般是配置文件错误造成的。

如:(注意 reader 和 writer 的 jdbcUrl ,前者是个数组,后者是个字符串。本人把后者配置也成[] 报了上面的错误

{
    "job": {
        "setting": {
            "speed": {
                "channel": 1
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "root",
                        "password": "qwer1234",
                        "column": ["*"],
                        "connection": [
                            {                           
                             "jdbcUrl": ["jdbc:mysql://192.168.88.114:3306/dpcsys_test"],
                             "table": ["dpc_his_notice_detailed_tab"]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "mysqlwriter", 
                    "parameter": {
                     "write": {  
                        "check": false  # 设置忽略脏数据检查  
                          },  
			"writeMode": "update",
                        "column": ["*"],
                        "session": [
                        	"set session sql_mode='ANSI'"
                        ],
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/dpcsys_test",
                                "table": ["dpc_his_notice_detailed_testmonth"]
                            }
                         ],
		        "username": "root",
                        "password": "qwer1234"
                    }
                }
            }
        ]
    }
}

二、同步过程中报错

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-14], Description:[DataX传输脏数据超过用户预期,该错误通常是由于源端数据存在较多业务脏数据导致,请仔细检查DataX汇报的脏数据日志信息, 或者您可以适当调大脏数据阈值 .].  - 脏数据条数检查不通过,限制是[0]条,但实际上捕获了[14]条.

 

ERROR StdoutPluginCollector - 脏数据: 


{"exception":"Table has no partition for value 740559","record":[{"byteSize":19,"index":0,"rawData":1172967129725534208,"type":"LONG"},{"byteSize":10,"index":1,"rawData":"1817127676","type":"STRING"},{"byteSize":2,"index":2,"rawData":"AI","type":"STRING"},{"byteSize":1,"index":3,"rawData":"0","type":"STRING"},{"byteSize":4,"index":4,"rawData":3897,"type":"LONG"},{"byteSize":1,"index":5,"rawData":8,"type":"LONG"},{"byteSize":1,"index":6,"rawData":"1","type":"STRING"},{"byteSize":5,"index":7,"rawData":18031,"type":"LONG"},{"byteSize":4,"index":8,"rawData":"3958","type":"STRING"},{"byteSize":5,"index":9,"rawData":26477,"type":"LONG"},{"byteSize":1,"index":10,"rawData":"1","type":"STRING"},{"byteSize":7,"index":11,"rawData":1869542,"type":"LONG"},{"byteSize":1,"index":12,"rawData":"0","type":"STRING"},{"byteSize":3,"index":13,"rawData":"369","type":"STRING"},{"byteSize":4,"index":14,"rawData":"环境湿度","type":"STRING"},{"byteSize":21,"index":15,"rawData":"DHS_13652259383777545","type":"STRING"},{"byteSize":1,"index":16,"rawData":"2","type":"STRING"},{"byteSize":3,"index":17,"rawData":"157","type":"STRING"},{"byteSize":5,"index":18,"rawData":"81.79","type":"STRING"},{"byteSize":0,"index":19,"rawData":"","type":"STRING"},{"byteSize":0,"index":20,"rawData":"","type":"STRING"},{"byteSize":8,"index":21,"rawData":1817127675000,"type":"DATE"},{"byteSize":0,"index":22,"type":"DATE"},{"byteSize":0,"index":23,"type":"STRING"},{"byteSize":8,"index":24,"rawData":1690898404000,"type":"DATE"},{"byteSize":8,"index":25,"rawData":1701303827000,"type":"DATE"},{"byteSize":0,"index":26,"type":"STRING"},{"byteSize":0,"index":27,"type":"STRING"},{"byteSize":5,"index":28,"rawData":"81.99","type":"STRING"},{"byteSize":0,"index":29,"type":"LONG"},{"byteSize":0,"index":30,"type":"STRING"},{"byteSize":12,"index":31,"rawData":"QY1050000003","type":"STRING"},{"byteSize":1,"index":32,"rawData":"0","type":"STRING"},{"byteSize":0,"index":33,"type":"STRING"},{"byteSize":8,"index":34,"rawData":1699698760000,"type":"DATE"},{"byteSize":11,"index":35,"rawData":"BengalAdmin","type":"STRING"},{"byteSize":8,"index":36,"rawData":1701303828000,"type":"DATE"},{"byteSize":0,"index":37,"type":"STRING"},{"byteSize":0,"index":38,"type":"STRING"},{"byteSize":0,"index":39,"type":"STRING"},{"byteSize":0,"index":40,"type":"STRING"},{"byteSize":0,"index":41,"type":"STRING"},{"byteSize":0,"index":42,"type":"LONG"}],"type":"writer"}

看日志发现 "type":"LONG"  可能数据类型Long有关?

 

发表评论