🔏
Tech
  • 🟢App aspects
    • Software architecture
      • Caching
      • Anti-patterns
      • System X-ability
      • Coupling
      • Event driven architecture
        • Command Query Responsibility Segregation (CQRS)
        • Change Data Capture (CDC)
      • Distributed transactions
      • App dev notes
      • TEMP. Check list
      • Hexagonal arch
      • Communication
        • REST vs messaging
        • gRPC
        • WebSocket
      • Load balancers
      • Storage limits
      • Event storming
    • Authentication
    • Deployment strategy
  • Databases
    • Classification
    • DB migration tools
    • PostreSQL
    • Decision guidance
    • Index
      • Hash indexes
      • SSTable, LSM-Trees
      • B-Tree
      • Engines, internals
    • Performance
  • System design
    • Interview preparation
      • Plan
        • Instagram
        • Tinder
        • Digital wallet
        • Dropbox
        • Live video streaming
        • Uber
        • Whatsup
        • Tiktok
        • Twitter
        • Proximity service
    • Algorithms
    • Acronyms
  • 🟢Programming languages
    • Java
      • Features
        • Field hiding
        • HashCode() and Equals()
        • Reference types
        • Pass by value
        • Atomic variables
      • Types
      • IO / NIO
        • Java NIO
          • Buffer
          • Channel
        • Java IO: Streams
          • Input streams
            • BufferedInputStream
            • DataInputStream
            • ObjectInputStream
            • FilterInputStream
            • ByteArrayInputStream
        • Java IO: Pipes
        • Java IO: Byte & Char Arrays
        • Java IO: Input Parsing
          • PushbackReader
          • StreamTokenizer
          • LineNumberReader
          • PushbackInputStream
        • System.in, System.out, System.error
        • Java IO: Files
          • FileReader
          • FileWriter
          • FileOutputStream
          • FileInputStream
      • Multithreading
        • Thread liveness
        • False sharing
        • Actor model
        • Singleton
        • Future, CompletableFuture
        • Semaphore
      • Coursera: parallel programming
      • Coursera: concurrent programming
      • Serialization
      • JVM internals
      • Features track
        • Java 8
      • Distributed programming
      • Network
      • Patterns
        • Command
      • Garbage Collectors
        • GC Types
        • How GC works
        • Tools for GC
    • Kotlin
      • Scope functions
      • Inline value classes
      • Coroutines
      • Effective Kotlin
    • Javascript
      • Javascript vs Java
      • TypeScript
    • SQL
      • select for update
  • OS components
    • Network
      • TCP/IP model
        • IP address in action
      • OSI model
  • 🟢Specifications
    • JAX-RS
    • REST
      • Multi part
  • 🟢Protocols
    • HTTP
    • OAuth 2.0
    • LDAP
    • SAML
  • 🟢Testing
    • Selenium anatomy
    • Testcafe
  • 🟢Tools
    • JDBC
      • Connection pool
    • Gradle
    • vim
    • git
    • IntelliJ Idea
    • Elastic search
    • Docker
    • Terraform
    • CDK
    • Argo CD
      • app-of-app setup
    • OpenTelemetry
    • Prometheus
    • Kafka
      • Consumer lag
  • 🟢CI
    • CircleCi
  • 🟢Platforms
    • AWS
      • VPC
      • EC2
      • RDS
      • S3
      • IAM
      • CloudWatch
      • CloudTrail
      • ELB
      • SNS
      • Route 53
      • CloudFront
      • Athena
      • EKS
    • Kubernetes
      • Networking
      • RBAC
      • Architecture
      • Pod
        • Resources
      • How to try
      • Kubectl
      • Service
      • Tooling
        • ArgoCD
        • Helm
        • Istio
    • GraalVM
    • Node.js
    • Camunda
      • Service tasks
      • Transactions
      • Performance
      • How it executes
  • 🟢Frameworks
    • Hibernate
      • JPA vs Spring Data
    • Micronaut
    • Spring
      • Security
      • JDBC, JPA, Hibernate
      • Transactions
      • Servlet containers
  • 🟢Awesome
    • Нейробиология
    • Backend
      • System design
    • DevOps
    • Data
    • AI
    • Frontend
    • Mobile
    • Testing
    • Mac
    • Books & courses
      • Path: Java Concurrency
    • Algorithms
      • Competitive programming
    • Processes
    • Finance
    • Electronics
  • 🟢Electronics
    • Arduino
    • IoT
  • Artificial intelligence
    • Artificial Intelligence (AI)
  • 🚀Performance
    • BE
  • 📘Computer science
    • Data structures
      • Array
      • String
      • LinkedList
      • Tree
    • Algorithms
      • HowTo algorithms for interview
  • 🕸️Web dev (Frontend)
    • Trends
    • Web (to change)
  • 📈Data science
    • Time series
Powered by GitBook
On this page
  • Hadoop
  • Apache Spark vs MapReduce

Was this helpful?

  1. Programming languages
  2. Java

Distributed programming

PreviousJava 8NextNetwork

Last updated 2 years ago

Was this helpful?

Hadoop

  • Storage unit -> HDFS

    • Data is stored across nodes sliced to blocks (128 Mb each)

    • Blocks are replicated on diff nodes with factor of 3 (two copies of each block)

  • Processing -> MapReduce

  • YARN (large-scale distributed operating system used for Big Data processing)

Apache Spark vs MapReduce

Spark was designed to overcome limitations of MapReduce.

  • RDD Resilient Distributed Dataset (foundation)

    • readonly objects distributed across cluster

    • Dataset can be built from: files, SQL dbs, noSQL dbs, HDFS

  • This is a core of Spark

Processing of RDD is in RAM

🟢
⚠️