在使用Scala将DataFrame的数据放入ES中时

1.建立连接时发生错误

Caused by: java.lang.IllegalStateException: availableProcessors is already set to [8], rejecting [8]

解决方法

配置中加入
System.setProperty("es.set.netty.runtime.available.processors", "false")

    // 新建es配置

    val settings: Settings = Settings.builder().put("cluster.name", eSConfig.clusterName).build()

    // 新建一个es客户端
    System.setProperty("es.set.netty.runtime.available.processors", "false")

    //解决netty冲突 ---  es的和spring的netty版本冲突
    val esClient = new PreBuiltTransportClient(settings)

2.写入数据时发生错误

Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: resource must

查看日志

Caused by: java.lang.IllegalArgumentException: Invalid format: "null"
    at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[joda-time-2.9.5.jar:2.9.5]
    at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[joda-time-2.9.5.jar:2.9.5]
    at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:240) ~[elasticsearch-5.6.2.jar:5.6.2]
    at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:465) ~[elasticsearch-5.6.2.jar:5.6.2]
    at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.6.2.jar:5.6.2]
    ... 36 more

原因：在数据预处理时对数据空值采用null来填补，es在缓存过程中Java识别到null关键字发生错误，解决方法为将null字段修改为别的填充数据

3.org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [XXX:9200] returned Bad Request(400) - failed to parse [XXX]; Bailing out..

常见错误为es中的字段值错误，我在开发过程中的错误为 failed to parse [release_date]

格式为2000-01-01
2000-01-01：String类型存储到es中会自动转换为date类型，但是在数据预处理时对于没有 release_date的字段设置为空，因此转换出错
解决方法：设置一个默认值比如 2000-01-01

elasticSearch踩的坑