Distributed programming
Hadoop
Storage unit -> HDFS
Data is stored across nodes sliced to blocks (128 Mb each)
Blocks are replicated on diff nodes with factor of 3 (two copies of each block)
Processing -> MapReduce
YARN (large-scale distributed operating system used for Big Data processing)
Apache Spark vs MapReduce
Spark was designed to overcome limitations of MapReduce.
RDD Resilient Distributed Dataset (foundation)
readonly objects distributed across cluster
Dataset can be built from: files, SQL dbs, noSQL dbs, HDFS
⚠️Processing of RDD is in RAM
This is a core of Spark
Last updated