Apache Spark Fundamentals

Fast, in-memory data processing at scale

The Evolution of Big Data Processing

Apache Spark is an open-source, unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python (PySpark), and R, along with an optimized engine that supports general execution graphs. Spark extends the MapReduce model to efficiently support more types of computations, including interactive queries and stream processing. Its in-memory caching makes it 100x faster than Hadoop MapReduce for iterative algorithms and interactive data mining.

Born at UC Berkeley's AMPLab in 2009, Spark addresses Hadoop's limitations by keeping data in memory across operations. It runs on Hadoop YARN, Kubernetes, Mesos, or standalone, and integrates with HDFS, S3, Cassandra, and more. In this lesson, we'll explore Spark's core components, with a focus on PySpark for practical examples.

1. Spark Core: Resilient Distributed Datasets (RDDs)

Spark Core is the foundation, providing distributed task dispatching, scheduling, and basic I/O. At its heart are RDDs: immutable, partitioned collections of objects that can be computed in parallel. RDDs support fault-tolerant, in-memory processing with lineage for recomputation on failure.

Architecture Overview

Spark follows a driver-executor model:

Driver Program

Main program, creates SparkContext, defines transformations/actions, manages job submission.

Executors

Worker processes on cluster nodes, run tasks, store data in memory/disk.

Cluster Manager

External service (YARN, Kubernetes) for resource allocation.

DAG Scheduler

Builds directed acyclic graph of stages for optimization.

Key Features

Lazy Evaluation

Transformations build DAG; actions trigger computation.

In-Memory Caching

Persist RDDs in memory for reuse, reducing I/O.

Fault Tolerance

Lineage allows recomputation of lost partitions.

Partitioning

Data divided into partitions for parallel processing.

Hands-On Examples (PySpark RDDs)

Basic: Create RDD and Simple Transformation
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
rdd = sc.parallelize([1, 2, 3, 4, 5])
doubled = rdd.map(lambda x: x * 2)
print(doubled.collect())
Result: [2, 4, 6, 8, 10]
Intermediate: Filter and Reduce
rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
evens = rdd.filter(lambda x: x % 2 == 0)
sum_evens = evens.reduce(lambda a, b: a + b)
print(sum_evens)
Result: 12
Advanced: Cache and Join RDDs
rdd1 = sc.parallelize([("a", 1), ("b", 2)]).cache()
rdd2 = sc.parallelize([("a", 3), ("b", 4)])
joined = rdd1.join(rdd2)
print(joined.collect())
Result: [('a', (1, 3)), ('b', (2, 4))]
⚠️ Note: While RDDs are low-level and flexible, modern Spark workflows prefer higher-level DataFrames for better optimization.

2. Spark SQL: Structured Data Processing

Spark SQL is a module for structured data processing, providing DataFrames (distributed collections with schema) and SQL querying. It integrates with Hive, supports multiple formats (Parquet, ORC, JSON), and uses Catalyst optimizer for efficient execution plans.

Architecture Overview

Built on Spark Core, with key components:

DataFrames/Datasets

Typed, distributed datasets with schema; optimized for queries.

Catalyst Optimizer

Rule-based and cost-based optimization for query plans.

Tungsten Engine

Whole-stage code generation for CPU efficiency.

Hive Integration

Reads Hive metastore for tables and UDFs.

Key Features

Unified Data Access

SQL, DataFrame API, and Dataset API interoperable.

Window Functions

Advanced analytics like ranking, aggregates over windows.

UDFs/UDAFs

User-defined functions and aggregations.

Adaptive Query Execution

Runtime optimizations like dynamic partitioning.

Hands-On Examples (PySpark SQL)

Basic: Create DataFrame and Show
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [("Alice", 25), ("Bob", 30)]
df = spark.createDataFrame(data, ["name", "age"])
df.show()
OUTPUT2 rows
┌───────┬─────┐
│ name  │ age │
├───────┼─────┤
│ Alice │ 25  │
│ Bob   │ 30  │
└───────┴─────┘
Intermediate: SQL Query on DataFrame
df.createOrReplaceTempView("people")
result = spark.sql("SELECT name FROM people WHERE age > 25")
result.show()
OUTPUT1 rows
┌──────┐
│ name │
├──────┤
│ Bob  │
└──────┘
Advanced: GroupBy with Window
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
data = [("Alice", "Sales", 3000), ("Bob", "Sales", 4600), ("Carol", "HR", 5000)]
df = spark.createDataFrame(data, ["name", "dept", "salary"])
window = Window.partitionBy("dept").orderBy("salary")
df.withColumn("rank", row_number().over(window)).show()
OUTPUT3 rows
┌───────┬───────┬────────┬──────┐
│ name  │ dept  │ salary │ rank │
├───────┼───────┼────────┼──────┤
│ Carol │ HR    │ 5000   │ 1    │
│ Alice │ Sales │ 3000   │ 1    │
│ Bob   │ Sales │ 4600   │ 2    │
└───────┴───────┴────────┴──────┘
Key Insight: Spark SQL's Catalyst optimizer can push down predicates and projections, making it highly efficient for big data queries.

3. Spark Streaming: Real-Time Data Processing

Spark Streaming enables scalable, high-throughput, fault-tolerant stream processing. Structured Streaming (introduced in Spark 2.0) treats streams as unbounded tables, allowing SQL-like queries on streaming data with exactly-once semantics.

Architecture Overview

Key elements in Structured Streaming:

Sources

Kafka, Flume, files, sockets; support rate limiting.

Processing Engine

Incremental execution on streaming DataFrames.

Sinks

HDFS, databases, console; support idempotent writes.

Checkpointing

State recovery for fault tolerance.

Key Features

Exactly-Once Semantics

Guaranteed with idempotent sinks and offsets.

Event-Time Processing

Handles late data with watermarks.

Continuous Mode

Millisecond latency for supported operations.

Integration with Batch

Same code for batch and streaming.

Hands-On Examples (PySpark Structured Streaming)

Basic: Read Stream from Socket
from pyspark.sql.functions import explode, split
lines = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()
words = lines.select(explode(split(lines.value, " ")).alias("word"))
query = words.writeStream.outputMode("append").format("console").start()
Result: Prints incoming words to console as they arrive.
Intermediate: Word Count Aggregation
wordCounts = words.groupBy("word").count()
query = wordCounts.writeStream.outputMode("complete").format("console").start()
Result: Updates console with running word counts.
Advanced: Windowed Count with Watermark
from pyspark.sql.functions import window, current_timestamp
lines = lines.withColumn("timestamp", current_timestamp())
windowedCounts = lines.groupBy(
    window("timestamp", "10 minutes", "5 minutes"),
    "word"
).count().withWatermark("timestamp", "10 minutes")
query = windowedCounts.writeStream.outputMode("update").format("console").start()
Result: Computes sliding window word counts, handling late data.
⚠️ Note: For production, use reliable sources like Kafka and sinks like Parquet for fault tolerance.

4. MLlib: Machine Learning Library

MLlib is Spark's scalable machine learning library, providing algorithms for classification, regression, clustering, and more. It leverages Spark's distributed nature for training on massive datasets, with pipelines for workflow management.

Architecture Overview

Integrated with DataFrames:

Algorithms

LogisticRegression, DecisionTrees, KMeans, etc.

Pipelines

Stages for feature extraction, transformation, modeling.

Model Tuning

CrossValidator, TrainValidationSplit.

Persistence

Save/load models for deployment.

Key Features

Scalability

Distributed training on clusters.

Feature Engineering

VectorAssembler, StringIndexer, etc.

Deep Learning

Integration with TensorFlow, PyTorch via sparkdl.

Model Serving

Export to PMML or use Spark ML serving.

Hands-On Examples (PySpark MLlib)

Basic: Linear Regression
from pyspark.ml.regression import LinearRegression
from pyspark.ml.linalg import Vectors
data = [(Vectors.dense([1.0]), 2.0), (Vectors.dense([2.0]), 4.0)]
df = spark.createDataFrame(data, ["features", "label"])
lr = LinearRegression().fit(df)
print(lr.coefficients)
Result: [2.0]
Intermediate: KMeans Clustering
from pyspark.ml.clustering import KMeans
data = [(Vectors.dense([0.0, 0.0]),), (Vectors.dense([1.0, 1.0]),), (Vectors.dense([9.0, 9.0]),)]
df = spark.createDataFrame(data, ["features"])
kmeans = KMeans(k=2).fit(df)
print(kmeans.clusterCenters())
Result: [array([0.5, 0.5]), array([9., 9.])]
Advanced: Pipeline with CrossValidation
from pyspark.ml import Pipeline
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import RegressionEvaluator
lr = LinearRegression()
pipeline = Pipeline(stages=[lr])
paramGrid = ParamGridBuilder().addGrid(lr.maxIter, [10, 20]).build()
cv = CrossValidator(estimator=pipeline, estimatorParamMaps=paramGrid, evaluator=RegressionEvaluator(), numFolds=2)
cvModel = cv.fit(df)
print(cvModel.bestModel.stages[0].extractParamMap())
Result: Best model parameters from cross-validation.
Key Insight: MLlib's distributed nature allows training on datasets too large for single-machine libraries like scikit-learn.

5. GraphX: Graph Computation

GraphX is Spark's API for graphs and graph-parallel computation, unifying graph processing with general data processing. It extends RDDs to Property Graphs with vertices and edges having properties.

Architecture Overview

Built on RDDs with:

Graph

RDD of vertices and edges with properties.

Operators

Subgraph, joinVertices, aggregateMessages.

Algorithms

PageRank, ConnectedComponents, TriangleCount.

Pregel API

Bulk-synchronous parallel message passing.

Key Features

Graph Views

EdgeTriplets, VertexRDD, EdgeRDD.

Partitioning Strategies

VertexCut, EdgePartition2D for efficiency.

Dynamic Graphs

Support for evolving graphs over time.

Integration

Seamless with other Spark components.

Hands-On Examples (PySpark GraphFrames - Modern Alternative)

Note: GraphX is Scala-only; in PySpark, use GraphFrames for similar functionality.

Basic: Create Graph
from graphframes import GraphFrame
vertices = spark.createDataFrame([("a", "Alice"), ("b", "Bob")], ["id", "name"])
edges = spark.createDataFrame([("a", "b", "friend")], ["src", "dst", "relationship"])
g = GraphFrame(vertices, edges)
g.vertices.show()
RESULT SET
┌────┬───────┐
│ id │ name  │
├────┼───────┤
│ a  │ Alice │
│ b  │ Bob   │
└────┴───────┘
Intermediate: Find Degrees
g.degrees.show()
RESULT SET
┌────┬────────┐
│ id │ degree │
├────┼────────┤
│ a  │ 1      │
│ b  │ 1      │
└────┴────────┘
Advanced: PageRank
results = g.pageRank(resetProbability=0.15, maxIter=10)
results.vertices.select("id", "pagerank").show()
RESULT SET
┌────┬──────────┐
│ id │ pagerank │
├────┼──────────┤
│ a  │ 0.5      │
│ b  │ 0.5      │
└────┴──────────┘
⚠️ Note: For large graphs, optimize partitioning and use GraphFrames for Python compatibility.

6. Broader Spark Ecosystem & Integrations

Spark's ecosystem includes tools and integrations that extend its capabilities.

Delta Lake

ACID transactions on data lakes, time travel, schema enforcement.

Spark on Kubernetes

Native K8s scheduler for dynamic scaling.

Koalas/Pandas API

Pandas-like API on Spark for familiarity.

Hadoop Integration

Runs on YARN, reads/writes HDFS.

Kafka Connector

Structured Streaming with Kafka for reliable ingestion.

MLflow

Tracking, packaging, deploying ML models.

Bonus: Real-Time Fraud Detection with PySpark Structured Streaming

Every streaming concept from this lesson: Structured Streaming, Kafka source, tumbling windows, sliding windows, watermarks, late-data handling, and exactly-once checkpointing, is demonstrated in a production-grade banking fraud detection project. Two concurrent PySpark jobs consume from Kafka, enrich each micro-batch with reference data via stream-batch broadcast joins, score transactions against weighted fraud rules, and write results to Delta Lake(local filesystem or MinIO S3-compatible). A Streamlit dashboard reads the Delta tables and auto-refreshes every few seconds.

Streaming features demonstrated:

  • Job 1: fraud_detection: 100 msg/s Kafka source, 10-min watermark, stream-batch broadcast join against 10k-row customer and merchant-blacklist tables, weighted fraud scoring (6 rules, score threshold 0.35), Delta append to enriched_transactions and fraud_alerts, Delta upsert to window_aggregates
  • Job 2: account_risk: 30-min / 5-min sliding window of failed logins, 15-min watermark, tier-weighted risk scores, Delta upsert with merge keys for idempotent restarts; uses outputMode("update") to emit only changed windows
  • Late-data handling in practice: the transaction producer deliberately injects 10% of events 8 minutes late; both watermarks are tuned to process them while dropping events that arrive beyond the tolerance window
# Prerequisites: Docker + Compose, Python 3.12+, Java 21 LTS
# (Java 24+ breaks PySpark 4.x – use Java 21)
git clone https://gitlab.com/bytecode-solutions/examples/pyspark-streaming-processing
cd pyspark-streaming-processing

pip install -e ".[dev]"   # pyspark, delta-spark, confluent-kafka, streamlit, plotly …

make up          # Kafka, Postgres, MinIO, Spark cluster, Kafka UI
make init-data   # generates 10k customers + blacklist, seeds Postgres

# Terminal A: transaction producer (100 msg/s, 5% fraud, 10% late events)
make producer-transactions

# Terminal B: account event producer (30 msg/s, 3% credential-stuffing bursts)
make producer-accounts

# Terminal C: Job 1 – fraud detection pipeline
make job-fraud

# Terminal D: Job 2 – account risk scorer (optional, runs independently)
make job-accounts

# Streamlit dashboard → http://localhost:8501
make dashboard

# Query Delta Lake output from an interactive Spark shell
make spark-shell
# >>> spark.read.format("delta").load("data/delta/fraud_alerts") \
# ...     .select("transaction_id","account_id","amount","fraud_score") \
# ...     .show(10, truncate=False)

# Tests (no Kafka or Docker required)
make test-unit          # fast pytest, no JVM
make test-integration   # local[2] Spark + file fixtures
Explore the project:gitlab.com/bytecode-solutions/examples/pyspark-streaming-processing

Key Takeaways

  • Spark Core provides RDDs for low-level, resilient distributed processing.
  • Spark SQL offers optimized structured data handling with DataFrames and Catalyst.
  • Structured Streaming unifies batch and stream processing with exactly-once guarantees.
  • MLlib enables scalable ML with pipelines and distributed algorithms.
  • GraphX/GraphFrames handle graph computations efficiently.
  • Spark's ecosystem integrates with data lakes, clusters, and tools for end-to-end analytics.
What's Next?

With Spark fundamentals mastered, explore advanced topics like data storage architectures in the next lesson.

Big Data FundamentalsLesson 4