【1】 Expected only partition pruning predicates
解决方案:设置spark.sql.hive.metastorePartitionPruning=false
【2】 Error in query: Detected cartesian product for INNER join between logical plans
Project-Join condition is missing or trivial.
Use the CROSS JOIN syntax to allow cartesian products between these relations
解决方案:设置spark.sql.crossJoin.enabled=true
【3】 ERROR ApplicationMaster: User class threw exception: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
- image.png
- image.png
解决方案:设置spark.sql.autoBroadcastJoinThreshold为-1,尝试关闭BroadCast Join
【4】org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: No suitable driver found for
bigquery打包时,生成了spark-1.0.3的包,用它起thriftserver,里面逻辑涉及到访问mysql时,报No suitable driver found for错误,看错误是没拿到mysql的url。检查jar包,common模块的resource中的配置文件没有打进去,看了下是spark模块pom.xml打包时没有加上common模块的resource包。