Distributed Computing Basics

How to process data across multiple machines

From One Machine to Many

In Lesson 1 we saw why single machines can't handle Big Data. The solution is distributed computing: spreading both data and computation across many regular computers (nodes) that work together as a team. This lesson focuses on the fundamental ideas that make distributed processing possible, the same concepts that power Hadoop, Spark, Kafka, and modern cloud data platforms.

1. Horizontal Scaling (Scale-Out)

Instead of buying a bigger computer (vertical scaling), we add more normal computers.

Vertical Scaling (Traditional)
  • Buy bigger server
  • More CPU / RAM / Disk
  • Expensive , costs grow exponentially
  • Single point of failure
  • Hard limit exists
Horizontal Scaling (Distributed)
  • Add more normal servers
  • Commodity hardware → cheap
  • Linear cost increase
  • More capacity = more fault tolerance
  • Practically unlimited scale
Key Insight: In 2025–2026, the most cost-effective way to get 10× more compute power is almost always to buy 10× more small machines rather than 1× very large machine.

2. Parallel Processing – Divide & Conquer

Break big problem into many small independent pieces → solve simultaneously → combine results.

Classic Example: Counting Words in 12 Books
Single Machine Approach

Read all books sequentially → count words → 12 hours

Distributed Parallel Approach

Give each of 12 machines one book → each counts words independently (parallel) → combine 12 partial counts → ~1 hour (plus small combining overhead)

Ideal speedup ≈ number of machines (when problem is perfectly parallelizable)

3. Data Locality – Move Code, Not Data

Moving large data over network is very slow and expensive. Modern distributed systems try to send small computation to where large data already lives.

Bad: Traditional approach
  • Move 100 GB data to central server
  • Very slow network transfer
  • Network becomes bottleneck
Good: Data Locality
  • Data already split across 10 nodes
  • Send ~100 KB program to each node
  • Each node processes its local 10 GB
  • Network traffic reduced 1000×

4. Fault Tolerance & Data Replication

With hundreds or thousands of machines, failure is normal, not exceptional.

How Distributed Systems Survive Failure
Replication

Keep 2–3 copies of every piece of data on different machines

Automatic Recovery

When a machine dies → system automatically re-assigns work to other nodes

Speculative Execution

If one task is slow → run it on another machine too → take fastest result

Idempotent Operations

Repeating the same operation multiple times gives same result

Realistic expectation: In large clusters, some machine fails almost every day. Good distributed systems make these failures invisible to users.
✨ The Magic of Distribution: By distributing work across many machines, Big Data systems can process petabytes of data in minutes or hours instead of weeks or months. A 1,000-node Spark cluster can analyze data at speeds that would require decades on a single machine.

Classic Pattern: Map → Shuffle → Reduce (Conceptual)

Most distributed batch systems follow this simple two-phase pattern (used by Hadoop MapReduce, Spark, etc.)

Input text:
"cat dog cat bird dog cat"

# Map phase (parallel on many machines):
→ cat  → (cat,1)
→ dog  → (dog,1)
→ cat  → (cat,1)
→ bird → (bird,1)
→ dog  → (dog,1)
→ cat  → (cat,1)

# Shuffle & Sort (automatic – groups by key):
bird: [1]
cat:  [1,1,1]
dog:  [1,1]

# Reduce phase (parallel – one task per key):
bird → 1
cat  → 3
dog  → 2

This innocent-looking pattern is powerful enough to process petabytes of data every day at companies worldwide.

Key Takeaways

  • Horizontal scaling (add more machines) beats vertical scaling for Big Data
  • Parallel processing = divide work → do simultaneously → combine results
  • Data locality principle: move small code to data, not big data to code
  • Fault tolerance achieved mainly through replication + automatic recovery
  • The famous Map → Shuffle → Reduce pattern is the foundation of most distributed batch systems
Big Data FundamentalsLesson 2