Entry point and basic abstraction
For Spark base
main entry point: SparkContext
basic abstraction: RDD
For Spark SQL
main entry point: SparkSession
basic abstraction: DataFrame
For Spark Streaming
Main entry point:
basic abstraction: DStream
For Spark ML
Main entry point:
Core Classes
Spark base
pyspark.SparkContext
Main entry point for Spark functionality.pyspark.RDD
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.Spark Streaming
pyspark.streaming.StreamingContext
Main entry point for Spark Streaming functionality.pyspark.streaming.DStream
A Discretized Stream (DStream), the basic abstraction in Spark Streaming.Spark SQL and DataFrame
pyspark.sql.SQLContext
Main entry point for DataFrame and SQL functionality.pyspark.sql.DataFrame
A distributed collection of data grouped into named columns.
Spark running mode
Locally
Cluster
Setup and run/submit job
Locally
Setup
Spark shell and submit job
./bin/spark-shell --master local[2]
OR
./bin/pyspark --master local[2]
Submit job
./bin/spark-submit examples/src/main/python/pi.py 10
OR
./bin/spark-submit examples/src/main/r/dataframe.R
Spark Stand alone cluster
Spark YARN cluster
What ?
:paste
:help
Spark context available as sc.
SQL context available as sqlContext.
Read csv files as Dataframe in Apache Spark with spark-csv package. after loading data to Dataframe save dataframe to parquetfile.
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true")
.option("inferSchema", "true")
.option("mode", "DROPMALFORMED")
.load("/home/myuser/data/log/*.csv")
df.saveAsParquetFile("/home/myuser/data.parquet")
val df_1 = sqlContext.read.parquet("/Users/user_name/Work/tmp/sample.parquet")
df.dtypes
df.show()
