在使用python版本的bigdl时,运行demo:报如下错误:
Py4JJavaError: An error occurred while calling o255.json.: java.lang.IllegalArgumentException: Illegal pattern component: XXX
at org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)at org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)at org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)at org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)at org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82)at org.apache.commons.lang3.time.FastDateFormat.getInstance(FastDateFormat.java:165)at org.apache.spark.sql.catalyst.json.JSONOptions.(JSONOptions.scala:83)at org.apache.spark.sql.catalyst.json.JSONOptions.(JSONOptions.scala:43)at
org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:439)at
org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:425)at
org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:411)at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at
java.lang.reflect.Method.invoke(Method.java:498)at
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)at
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)at
py4j.Gateway.invoke(Gateway.java:282)at
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)at
py4j.commands.CallCommand.execute(CallCommand.java:79)at
py4j.GatewayConnection.run(GatewayConnection.java:238)at
java.lang.Thread.run(Thread.java:748)
在网上查得原因是
1.此问题经常出现在spark2.1.x升级到spark2.2.x的时候出现。比如通过maven构建spark环境的时候,依赖maven进行版本升级。
2.此问题出现在调用spark.read.json或者csv的时候出现。
解决方法:
spark.read.option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ").json(peopleRDD)
其他的方法(还未验证)