在使用Scala将DataFrame的数据放入ES中时
1.建立连接时发生错误
Caused by: java.lang.IllegalStateException: availableProcessors is already set to [8], rejecting [8]
解决方法
配置中加入
System.setProperty("es.set.netty.runtime.available.processors", "false")
// 新建es配置
val settings: Settings = Settings.builder().put("cluster.name", eSConfig.clusterName).build()
// 新建一个es客户端
System.setProperty("es.set.netty.runtime.available.processors", "false")
//解决netty冲突 --- es的和spring的netty版本冲突
val esClient = new PreBuiltTransportClient(settings)
2.写入数据时发生错误
Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: resource must
查看日志
Caused by: java.lang.IllegalArgumentException: Invalid format: "null"
at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[joda-time-2.9.5.jar:2.9.5]
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[joda-time-2.9.5.jar:2.9.5]
at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:240) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:465) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.6.2.jar:5.6.2]
... 36 more
原因：在数据预处理时对数据空值采用null来填补，es在缓存过程中Java识别到null关键字发生错误，解决方法为将null字段修改为别的填充数据
3.org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [XXX:9200] returned Bad Request(400) - failed to parse [XXX]; Bailing out..
常见错误为es中的字段值错误，我在开发过程中的错误为 failed to parse [release_date]
格式为2000-01-01
2000-01-01：String类型存储到es中会自动转换为date类型，但是在数据预处理时对于没有 release_date的字段设置为空，因此转换出错
解决方法：设置一个默认值 比如 2000-01-01
