构建spark docker 镜像
下载spark-2.4.8-bin-hadoop2.7.tgz
Note: 这里下载spark包一定不是能是without hadoop 的。不然构建完以后运行，会报一些包找不到。比如log4j
tar -xvf spark-2.4.8-bin-hadoop2.7.tgz
cd spark-2.4.8-bin-hadoop2.7
编辑spark dockerfile
vim kubernetes/dockerfiles/spark/Dockerfile
将18行的 FROM openjdk:8-jdk-slim 替换成 FROM openjdk:8-jdk-slim-buster
因为默认openjdk基础形象是debian 11 ，后面的spark-py镜像会依赖此镜像。debian11 安装的python3 是python3.8 以上版本。spark2.4 不支持python3.7以上版本，所以会报”TypeError:an integer is required（got type bytes）” 这样的错误。
所以将基础镜像更换成debian 10。安装的python3 的版本是3.7
构建镜像
bin/docker-image-tool.sh -t v2.4.8 build
由于apt-get 源是国外的构建会比较慢，最好是开代理。
没有代理的也可以跟换国内源 （更换国内源会有包依赖的问题导致spark-py镜像是构建失败）
vim kubernetes/dockerfiles/spark/Dockerfile
在 29，31行之间 加入
ADD sources.list /etc/apt/sources.list
然后把sources.list文件放在spark-2.4.8-bin-hadoop2.7目录下
最后镜像构建好以后，会有3个镜像
spark、spark-py、spark-r
把这3个镜像push到镜像仓库。
spark 支持 OSS
spark 支持oss 可以再修改spark dockerfile
vim kubernetes/dockerfiles/spark/Dockerfile
将下面的几行jar文件放在COPY data /opt/spark/data 下面
ADD https://repo1.maven.org/maven2/com/aliyun/odps/hadoop-fs-oss/3.3.8-public/hadoop-fs-oss-3.3.8-public.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.8.1/aliyun-sdk-oss-3.8.1.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/org/aspectj/aspectjweaver/1.9.5/aspectjweaver-1.9.5.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/org/jdom/jdom/1.1.3/jdom-1.1.3.jar $SPARK_HOME/jars
也可以重新重新再写一个dockerfile 重新构建一个新的镜像。保持原有镜像
FROM acr-test01-registry.cn-beijing.cr.aliyuncs.com/netops/spark-py:v2.4.8
RUN mkdir -p /opt/spark/jars
# 如果需要使用OSS（读取OSS数据或者离线Event到OSS)，可以添加以下JAR包到镜像中
ADD https://repo1.maven.org/maven2/com/aliyun/odps/hadoop-fs-oss/3.3.8-public/hadoop-fs-oss-3.3.8-public.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.8.1/aliyun-sdk-oss-3.8.1.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/org/aspectj/aspectjweaver/1.9.5/aspectjweaver-1.9.5.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/org/jdom/jdom/1.1.3/jdom-1.1.3.jar $SPARK_HOME/jars
docker build -t ack-spark-oss:v2.4.8 .
docker tag
docker push
spark on ack yaml
这个yaml是用来提交spark任务。
scala 、java 和 python 不太一样。
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: spark-pi
namespace: default
spec:
type: Scala
mode: cluster
image: "acr-test01-registry.cn-beijing.cr.aliyuncs.com/netops/ack-spark-2.4.5:v9"
imagePullPolicy: Always
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: "oss://qa-oss/spark-examples_2.11-2.4.8.jar"
sparkConf:
"spark.eventLog.enabled": "true"
"spark.eventLog.dir": "oss://qa-oss/spark-events"
"spark.hadoop.fs.oss.impl": "org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem"
"spark.hadoop.fs.oss.endpoint": "oss-cn-beijing-internal.aliyuncs.com"
"spark.hadoop.fs.oss.accessKeySecret": "OSd0RVN"
"spark.hadoop.fs.oss.accessKeyId": "LTADXrW"
sparkVersion: "2.4.5"
imagePullSecrets: [spark]
restartPolicy:
type: Never
driver:
cores: 2
coreLimit: "2"
memory: "3g"
memoryOverhead: "1g"
labels:
version: 2.4.5
serviceAccount: spark
annotations:
k8s.aliyun.com/eci-kube-proxy-enabled: 'true'
k8s.aliyun.com/eci-image-cache: "true"
executor:
cores: 2
instances: 5
memory: "3g"
memoryOverhead: "1g"
labels:
version: 2.4.5
annotations:
k8s.aliyun.com/eci-kube-proxy-enabled: 'true'
k8s.aliyun.com/eci-image-cache: "true"
如果你的镜像仓库是public的就不需要imagePullSecrets 这个参数。
如果你的镜像仓库是带验证的。那么就要使用imagePullSecrets 在验证，后面[spark] 是一个configmap 里面是镜像仓库用户名，密码
另外mainApplicationFile 这个是任务jar 包位置。可以是 oss,或是 hdfs 如果用local就需要jar 在镜像内。
sparkconf 部分是配置 spark-history ，如果不需要删除掉。
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: spark-pi
namespace: default
spec:
type: Python
mode: cluster
image: "acr-test01-registry.cn-beijing.cr.aliyuncs.com/netops/ack-spark-2.4.5:v9"
imagePullPolicy: Always
mainApplicationFile: "local:///opt/spark/examples/src/main/python/pi.py"
sparkVersion: "2.4.5"
pythonVersion: "3"
imagePullSecrets: [spark]
restartPolicy:
type: Never
driver:
cores: 2
coreLimit: "2"
memory: "3g"
memoryOverhead: "1g"
labels:
version: 2.4.5
serviceAccount: spark
annotations:
k8s.aliyun.com/eci-kube-proxy-enabled: 'true'
k8s.aliyun.com/eci-image-cache: "true"
executor:
cores: 2
instances: 5
memory: "3g"
memoryOverhead: "1g"
labels:
