> For the complete documentation index, see [llms.txt](https://amartyushov.gitbook.io/tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://amartyushov.gitbook.io/tech/programming-languages/java/distributed-programming.md).

# Distributed programming

## Hadoop

* Storage unit -> **HDFS**
  * Data is stored across nodes sliced to blocks (128 Mb each)
  * Blocks are replicated on diff nodes with factor of 3 (two copies of each block)
* Processing -> **MapReduce**
  \*

  ```
  <figure><img src="/files/mBIyuNq9Lnt9nAz921RW" alt=""><figcaption></figcaption></figure>
  ```
* YARN (large-scale distributed operating system used for Big Data processing)

## Apache Spark vs MapReduce

Spark was designed to overcome limitations of MapReduce.

* RDD Resilient Distributed Dataset (foundation)
  * readonly objects distributed across cluster
  * Dataset can be built from: files, SQL dbs, noSQL dbs, HDFS
* :warning:Processing of RDD is in RAM
* This is a core of Spark

  <figure><img src="https://415484505-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LxtoAXZwwOc4XGto8vb%2Fuploads%2FRFyFJxoFuR00WV9NLebK%2FScreenshot%202023-01-05%20at%2009.09.23.png?alt=media&amp;token=44e18c52-9a70-493e-9121-ead24a5aba3a" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://415484505-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LxtoAXZwwOc4XGto8vb%2Fuploads%2FSy6FwAxPzCmLRPHeCM8F%2FScreenshot%202023-01-05%20at%2009.14.18.png?alt=media&amp;token=9477d1c7-a8c7-482e-be2b-08aa4d1686a2" alt=""><figcaption></figcaption></figure>