Dataproc Summary
How to load data?
a managed Spark and Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning.
Dataproc connects to BigQuery
Option 1:
BigQuery does not natively know how to work with a Hadoop file system.
Cloud storage can act as an intermediary between BigQuery and data proc.
You would export the data from BigQuery into cloud storage as sharded data.
Then the worker notes in data proc would read the sharded data.
Symmetrically, if the data proc job is producing output it can be stored in a format in cloud storage that can be input to BigQuery.
Appropriate for periodic or infrequent transfers
Option 2:
Another option is to setup a BigQuery connector on the Dataproc cluster. The connector is a Java library that enables read write access from Spark and Hadoop directly into BigQuery.
Need to save BigQuery result as table first.
Option 3:
When you want to process data in memory for speed - Pandas Dataframe
In memory, fast but limited in size
Creating a Dataproc cluster
Ways:
Deployment manager template, which is an infrastructure automation service in Google Cloud.
CLI commands
Google cloud console
Keys:
0 Create a cluster specifically for one job
1 Match your data location to the compute location
-> better performance
-> also able to shut down cluster when not processing jobs
2 use Cloud Storage instead of HDFS, shutdown the cluster when it’s not actually processing data
-> It reduces the complexity of disk provisioning and enables you to shut down your cluster when it's not processing a job.
3 Use custom machine types to closely manage the resources that the job requires
4 On non-critical jobs requiring huge clusters, use preemptible VMs to hasten results and cut costs at the same time
