🟢App aspects
Databases
System design
🟢Programming languages
OS components
- Network
  - TCP/IP model
    IP address in action
  - OSI model
🟢Specifications
- JAX-RS
- REST
  - Multi part
🟢Protocols
🟢Testing
- Selenium anatomy
- Testcafe
🟢Tools
🟢CI
- CircleCi
🟢Platforms
🟢Frameworks
🟢Awesome
🟢Electronics
- Arduino
- IoT
Artificial intelligence
- Artificial Intelligence (AI)
🚀Performance
- BE
📘Computer science
- Data structures
- Algorithms
  - HowTo algorithms for interview
🕸️Web dev (Frontend)
- Trends
- Web (to change)
📈Data science
- Time series

Powered by GitBook

On this page

Hadoop
Apache Spark vs MapReduce

Was this helpful?

🟢Programming languages

Java

Distributed programming

PreviousJava 8 NextNetwork

Last updated 2 years ago

Hadoop

Storage unit -> HDFS
- Data is stored across nodes sliced to blocks (128 Mb each)
- Blocks are replicated on diff nodes with factor of 3 (two copies of each block)
Processing -> MapReduce
YARN (large-scale distributed operating system used for Big Data processing)

Apache Spark vs MapReduce

Spark was designed to overcome limitations of MapReduce.

RDD Resilient Distributed Dataset (foundation)
- readonly objects distributed across cluster
- Dataset can be built from: files, SQL dbs, noSQL dbs, HDFS
⚠️Processing of RDD is in RAM
This is a core of Spark