Apache Hadoop - open source software projects{core Hadoop package and its accessories licensed by Apache}that can be used to efficiently process large datasets, Haddop alllows clustering commodity hardware together to analyze massive data sets in parallel. Hadoop commonly refers to the actual Apache Hadoop project, which includes
core Hadoop set
MapReduce - execution framework/engine
YARN (Yet Another Resource Negotiator) - resource manager
HDFS (Hadoop distributed storage system) - distributed storage file system accessories or complementary tools
Hive - data analysis tool
Spark - general engine for processing big data/another framework
Pig - data flow language
HBase - data base tool
Ambari - Hadoop ecosystem manager
Amazon EMR(Elastic MapReduce Management) - used to create and manage fully configured, elastic clusters of Amazon EC2 instances running Hadoop and other applications in the Hadoop ecosystem.
MapReduce/Tez - frameworks that break down jobs that can be distributed across nodes EMR cluster
Hive/Pig use Apache Tez(NEW) instead of Hadoop MapReduce as an execution engine starting with EMR 5.x
designed with fault tolerance Hadoop reruns a failed task on another machine
write MapReduce and Tez programs in Java,
use Hadoop Streaming to execute custom scripts in a parallel fashion
utilize Hive and Pig for higher level abstractions over MapReduce and Tez,
YARN - keeps track of all the resources across your cluster, and it ensures that these resources are dynamically allocated to accomplish the tasks in your processing job.
HDFS - On-cluster storage/stores data across local disks of your cluster in large blocks.
configurable replication factor (with a default of 3x)
monitors replication and balances your data across your nodes as nodes fail and new nodes are added
stores intermediate data created during Hadoop MapReduce jobs
EMRFS - file System/connector allowing Hadoop to use Amazon S3 as a storage layer
Spark - an open-source, distributed processing system
does not come with its own file management system
generally a lot faster than MapReduce, ideal for streaming data
data objects can be stored in memory or on disks, and RDD(resilient distributed dataset) provides full recovery from faults or failures
