断点调试
参考：http://danosipov.com/?p=779
测试代码
object SparkTest {
def main(args: Array[String]): Unit = {
// Local
val spark = new SparkContext(new SparkConf()
.setMaster("local").setAppName("Test")
)
//spark.setCheckpointDir("/user/mai/input")
println("-------------Attach debugger now!--------------")
Thread.sleep(8000)
// Your job code here, with breakpoints set on the lines you want to pause
val s = "abc1"
print(s)
}
}
IntelliJ Idea 生成jar
-
设置artifacts
image.png - build artifacts
提交命令
export SPARK_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
./bin/spark-submit.sh --class com.huaan.bd.spark.SparkTest --master local[4] --driver-memory 1G --executor-memory 1G ./testjars/mix-bdbasic.jar
连接断点
代码中有8秒钟时间等待连接
