Hadoop Ecosystem

HDFS, MapReduce, and the Hadoop framework

The Foundation of Big Data Processing

Apache Hadoop is an open-source framework that revolutionized Big Data by enabling reliable and scalable distributed computing. Inspired by Google's GFS and MapReduce papers, it provides the core infrastructure for storing and processing massive datasets across clusters of commodity hardware. The ecosystem extends beyond core components to include tools for data ingestion, querying, workflow management, and more, forming a complete platform for Big Data applications.

In this lesson, we'll dive deep into the core: HDFS for storage, YARN for resource management, and MapReduce for processing. We'll also cover key ecosystem components that build upon these foundations.

1. HDFS: Hadoop Distributed File System

HDFS is a distributed file system designed for high-throughput access to large datasets. It stores data across multiple machines, providing fault tolerance through replication and enabling data locality for efficient processing.

Architecture Overview

HDFS follows a master-slave architecture:

NameNode (Master)

Manages file system namespace, metadata, and block locations. Single point but with high-availability options.

DataNodes (Slaves)

Store actual data blocks (default 128MB each) and report to NameNode. Handle replication and data access.

Key Features

Replication

Default 3 copies for fault tolerance. Configurable per file.

Rack Awareness

Places replicas across racks to survive rack failures.

Write-Once, Read-Many

Optimized for streaming large files, not random writes.

Federation

Multiple NameNodes for scalability in large clusters.

Hands-On Examples (Shell Commands)

Basic: Create Directory
hdfs dfs -mkdir /user/data
Result: Creates a directory in HDFS. No output if successful.
Intermediate: Upload File with Replication
hdfs dfs -Ddfs.replication=2 -put localfile.txt /user/data/file.txt
Result: Uploads file with 2 replicas. Verifiable with: hdfs dfs -ls /user/data (shows file).
Advanced: Check File Status & Blocks
hdfs fsck /user/data/file.txt -files -blocks -locations
Example Result:
/user/data/file.txt 123456 bytes, 1 block(s): OK
0. BP-123:blk_1073741825_1001 len=123456 Live_repl=3 [DatanodeInfo1, DatanodeInfo2, DatanodeInfo3]
⚠️ Note: HDFS is not for small files (millions cause NameNode overload) or low-latency access. Use for large, sequential reads/writes.

2. YARN: Yet Another Resource Negotiator

YARN is Hadoop's resource management layer, separating resource allocation from processing frameworks. It allows multiple engines (MapReduce, Spark) to run on the same cluster.

Architecture Overview

YARN uses a master-slave model for cluster resource management:

ResourceManager (Master)

Global scheduler, allocates containers, handles applications.

NodeManagers (Slaves)

Per-node agents, launch/monitor containers, report resources.

ApplicationMaster

Per-app negotiator, requests resources, coordinates tasks.

Containers

Resource bundles (CPU, memory) for tasks.

Key Features

Multi-Tenancy

Multiple apps/engines share cluster securely.

Schedulers

Fair, Capacity, FIFO for resource allocation.

High Availability

Active/Standby ResourceManagers.

Dynamic Allocation

Apps can request/release resources on-demand.

Hands-On Examples (Commands)

Basic: Check Cluster Status
yarn node -list
Example Result:
Total Nodes:3
Node-Id: host1:8041 State:RUNNING
Node-Id: host2:8041 State:RUNNING
Intermediate: Submit Simple App
yarn jar hadoop-examples.jar pi 2 10
Result: Submits Pi estimator job. Output: Estimated value of Pi is 3.20000000000000000000
Advanced: Monitor Application Logs
yarn logs -applicationId application_1234567890_0001
Example Result: Dumps container logs, showing task attempts, errors, progress.
Key Insight: YARN evolved Hadoop from MapReduce-only to a general-purpose cluster manager, enabling diverse workloads.

3. MapReduce: Distributed Processing Framework

MapReduce is a programming model for processing large datasets in parallel. It abstracts distributed execution into map (transform) and reduce (aggregate) functions.

Core Phases

Map Phase

Processes input key-value pairs, emits intermediate key-value pairs.

Shuffle & Sort

Groups and sorts intermediate data by key, sends to reducers.

Reduce Phase

Aggregates values for each key, emits final output.

Combiners/Partitioners

Optional: Mini-reducers and custom partitioning for optimization.

Hands-On Examples (Java Code Snippets)

Basic: Mapper for Word Split
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
    StringTokenizer itr = new StringTokenizer(value.toString());
    while (itr.hasMoreTokens()) {
        context.write(new Text(itr.nextToken()), new IntWritable(1));
    }
}
Input: "hello world"
Result (Emitted): (hello,1), (world,1)
Intermediate: Reducer for Sum
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
        sum += val.get();
    }
    context.write(key, new IntWritable(sum));
}
Input: hello: [1,1,1]
Result (Emitted): (hello,3)
Advanced: Driver with Combiner
job.setMapperClass(WordCountMapper.class);
job.setCombinerClass(WordCountReducer.class);  // Mini-reducer for optimization
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
Result: Runs word count job with combiner to reduce network traffic (e.g., partial sums per mapper).

Hands-On Examples (Python + mrjob)

Basic: Word Split (Mapper logic)
def mapper(self, _, line):
    for word in line.strip().split():
        yield word.lower(), 1
Input line: "Hello World hello"
Emitted pairs: ("hello", 1), ("world", 1), ("hello", 1)
Intermediate: Sum per word (Reducer)
def reducer(self, word, counts):
    yield word, sum(counts)
Input: "hello" → [1, 1, 1]
Result (Emitted): ("hello", 3)
Advanced: Complete WordCount job with Combiner
from mrjob.job import MRJob

class MRWordCount(MRJob):

    def mapper(self, _, line):
        for word in line.strip().split():
            yield word.lower(), 1

    def combiner(self, word, counts):    # ← optimization!
        yield word, sum(counts)

    def reducer(self, word, counts):
        yield word, sum(counts)

if __name__ == '__main__':
    MRWordCount.run()
How to run:
python wordcount.py input.txt > output.txt

On Hadoop cluster:
python wordcount.py -r hadoop hdfs:///input/*.txt -o hdfs:///output/wordcount

Benefit of combiner: Reduces network traffic by doing partial sums on each mapper node
⚠️ Limitations: MapReduce is batch-oriented and disk-heavy; often replaced by faster in-memory alternatives like Spark for iterative workloads.

4. Key Ecosystem Components

Hadoop's power comes from its rich ecosystem of integrated tools building on HDFS and YARN.

Hive

Data warehouse with SQL-like queries (HiveQL) on HDFS. Compiles to MapReduce/Spark jobs.

Pig

Scripting language (Pig Latin) for data flows. Good for ETL, compiles to MapReduce.

HBase

NoSQL database on HDFS. Column-oriented, for random read/write access.

Sqoop

Tool for transferring data between Hadoop and relational databases.

Flume

Service for collecting and aggregating streaming data (e.g., logs) into HDFS.

Oozie

Workflow scheduler for managing Hadoop jobs (e.g., chain MapReduce, Hive).

Zookeeper

Distributed coordination service for configuration, naming, synchronization.

Ambari / Cloudera Manager

Management tools for deployment, monitoring, and administration.

Key Takeaways

  • HDFS provides scalable, fault-tolerant storage with replication and data locality.
  • YARN manages resources efficiently, supporting multiple processing frameworks.
  • MapReduce enables parallel processing via map-shuffle-reduce paradigm, ideal for batch jobs.
  • The ecosystem extends Hadoop with tools for querying (Hive), NoSQL (HBase), ingestion (Sqoop/Flume), and orchestration (Oozie).
  • Hadoop powers petabyte-scale systems but is often augmented with faster alternatives like Spark for modern workloads.
What's Next?

With Hadoop fundamentals in place, we're ready for faster processing! Next: Apache Spark for in-memory analytics.

Big Data FundamentalsLesson 3