本人最近在jupyter notebook中学习spark,但是总是提示一下错误;
Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
查了一下本系统python默认的是2.7 但是jupyter notebook中配置的是python3
此时查看系统环境变量设置为,路径为~/.bash_profile,为用户级环境变量。
export PYSPARK_PYTHON=/usr/local/bin/python3
解决办法为在程序中重新设置一下环境变量,即可:
import os
os.environ["PYSPARK_PYTHON"] = "/usr/local/bin/python3"
同样在cmd中端里,如果报错,也需要重新执行一下,才可生效。
. ~/.bash_profile
如有不明白的同学,可去了解下linux的环境变量和配置。