NoSQL Databases
Document stores, key-value, and all other database types
Beyond Relational Databases
NoSQL databases emerged to handle the demands of modern applications: massive scale, flexible schemas, distributed architectures, and specific use cases like caching, real-time analytics, and graph traversals. "NoSQL" doesn't mean "no SQL" but rather "Not Only SQL". These databases sacrifice some ACID guarantees and relational features for horizontal scalability, performance, and flexibility. Each type is optimized for specific use cases.
Database Types Covered
1. Document Stores
MongoDB, CouchDB, Firestore
2. Key-Value Stores
Redis, DynamoDB, Memcached
3. Column-Family Stores
Cassandra, HBase, ScyllaDB
4. Graph Databases
Neo4j, Amazon Neptune, ArangoDB
5. Time-Series Databases
InfluxDB, TimescaleDB, Prometheus
6. Search Engines
Elasticsearch, Solr, Meilisearch
7. Vector Databases
Pinecone, Weaviate, Qdrant
8. Multi-Model Databases
ArangoDB, OrientDB, Couchbase
1. Document Stores
Store data as JSON-like documents
Document databases store data in flexible, schema-less documents (typically JSON or BSON). Each document can have a different structure, making them ideal for evolving data models and hierarchical data.
Popular Engines
MongoDB
Most popular, rich query language, ACID transactions
CouchDB
HTTP API, multi-master replication, offline-first
Firestore
Google's serverless, real-time syncing, mobile-optimized
When to Use
- Content management systems
- User profiles and personalization
- Catalogs and product information
- Mobile and web applications
- Real-time analytics and logging
MongoDB Examples
Insert a Document
db.users.insertOne({
name: "Alice Johnson",
email: "alice@example.com",
age: 28,
interests: ["coding", "hiking", "photography"],
address: {
city: "San Francisco",
state: "CA",
zip: "94102"
},
created_at: new Date()
});Inserted 1 document with _id: ObjectId("507f1f77bcf86cd799439011")
Query Documents
db.users.find({
age: { $gte: 25 },
"address.city": "San Francisco"
});{ _id: ..., name: "Alice Johnson", age: 28, ... }
{ _id: ..., name: "Bob Smith", age: 32, ... }
Update with Operators
db.users.updateOne(
{ email: "alice@example.com" },
{
$set: { age: 29 },
$push: { interests: "cooking" }
}
);Modified 1 document (age updated, "cooking" added to interests array)
Aggregation Pipeline
db.orders.aggregate([
{ $match: { status: "completed" } },
{ $group: {
_id: "$customer_id",
total_spent: { $sum: "$amount" },
order_count: { $sum: 1 }
}},
{ $sort: { total_spent: -1 } },
{ $limit: 10 }
]);{ _id: "user123", total_spent: 2450, order_count: 12 }
{ _id: "user456", total_spent: 1890, order_count: 8 }
Pros & Cons
✅ Pros
- Flexible schema, no migrations needed
- Natural data representation (JSON)
- Easy to scale horizontally
- Rich query capabilities
- Good for hierarchical data
❌ Cons
- Data duplication common
- Complex joins are difficult
- No enforced schema can cause inconsistencies
- Memory usage can be high
- Less mature transaction support
2. Key-Value Stores
Simplest NoSQL model, maps keys to values
Key-value stores are the simplest NoSQL databases. They store data as a collection of key-value pairs, like a giant distributed hash table. Extremely fast for lookups by key, perfect for caching and session management.
Popular Engines
Redis
In-memory, pub/sub, data structures, persistence options
DynamoDB
AWS managed, auto-scaling, single-digit millisecond latency
Memcached
Simple, fast, distributed caching, volatile storage
When to Use
- Caching and session storage
- Real-time leaderboards and counters
- Shopping carts
- User preferences and settings
- Rate limiting and throttling
- Pub/sub messaging systems
Redis Examples
Basic Operations
SET user:1000:name "Alice Johnson" SET user:1000:email "alice@example.com" SET session:xyz123 "user_data" EX 3600
OK (session expires in 3600 seconds)
Get Values
GET user:1000:name MGET user:1000:name user:1000:email
"Alice Johnson"
["Alice Johnson", "alice@example.com"]
Lists (for queues)
LPUSH queue:tasks "process_payment" LPUSH queue:tasks "send_email" RPOP queue:tasks
"process_payment" (first in, first out)
Sorted Sets (for leaderboards)
ZADD leaderboard 1500 "player1" ZADD leaderboard 2300 "player2" ZADD leaderboard 1800 "player3" ZREVRANGE leaderboard 0 2 WITHSCORES
1. player2 (2300)
2. player3 (1800)
3. player1 (1500)
Hashes (for objects)
HSET user:1000 name "Alice" email "alice@example.com" age 28 HGETALL user:1000 HINCRBY user:1000 age 1
{ name: "Alice", email: "alice@example.com", age: "29" }
Pros & Cons
✅ Pros
- Extremely fast (in-memory)
- Simple data model
- Easy to scale horizontally
- Low latency
- Perfect for caching
❌ Cons
- No complex queries
- No relationships between data
- Limited data modeling
- Memory constraints (for in-memory stores)
- Not suitable as primary database for complex apps
3. Column-Family Stores
Wide-column stores for massive scale
Column-family stores organize data into rows with dynamic columns. Unlike relational databases that store data row-by-row, these databases store data column-by-column, optimized for reading and writing large amounts of data quickly. Designed for massive scale and high throughput.
Popular Engines
Cassandra
Highly scalable, no single point of failure, tunable consistency
HBase
Hadoop-based, strong consistency, billions of rows/columns
ScyllaDB
Cassandra-compatible, C++ rewrite, 10x faster performance
When to Use
- Time-series data (IoT, sensors)
- Event logging and analytics
- Product catalogs with many attributes
- Messaging and social media feeds
- High-write throughput applications
- Multi-datacenter deployments
Cassandra Examples
Create Keyspace (Database)
CREATE KEYSPACE ecommerce
WITH replication = {
'class': 'SimpleStrategy',
'replication_factor': 3
};Keyspace created with 3 replicas
Create Table
CREATE TABLE user_activity ( user_id UUID, activity_date DATE, activity_time TIMESTAMP, action TEXT, details MAP<TEXT, TEXT>, PRIMARY KEY ((user_id), activity_date, activity_time) ) WITH CLUSTERING ORDER BY (activity_date DESC, activity_time DESC);
Table created with partition key (user_id) and clustering keys
Insert Data
INSERT INTO user_activity (user_id, activity_date, activity_time, action, details)
VALUES (
uuid(),
'2024-01-15',
toTimestamp(now()),
'page_view',
{'page': '/products', 'duration': '45s'}
);1 row inserted
Query by Partition Key
SELECT * FROM user_activity WHERE user_id = '550e8400-e29b-41d4-a716-446655440000' AND activity_date >= '2024-01-01' LIMIT 10;
Returns up to 10 activities for specific user, sorted by date (descending)
Pros & Cons
✅ Pros
- Handles petabytes of data
- High write throughput
- Linear scalability
- No single point of failure
- Multi-datacenter support
- Tunable consistency levels
❌ Cons
- Limited joins
- Must design around queries (query-first modeling)
- Data duplication common
- Eventually consistent by default
- Steep learning curve
4. Graph Databases
Relationships are first-class citizens
Graph databases store data as nodes (entities) and edges (relationships). They excel at traversing relationships and finding patterns in connected data. Perfect for social networks, recommendation engines, and fraud detection.
Popular Engines
Neo4j
Most popular, Cypher query language, ACID compliant
Amazon Neptune
Fully managed, supports Gremlin and SPARQL
ArangoDB
Multi-model, graph + document, AQL query language
When to Use
- Social networks (friends, followers)
- Recommendation engines
- Fraud detection and pattern analysis
- Network and IT operations
- Knowledge graphs
- Supply chain and logistics
Neo4j (Cypher) Examples
Create Nodes
CREATE (alice:Person {name: 'Alice', age: 28})
CREATE (bob:Person {name: 'Bob', age: 32})
CREATE (coding:Skill {name: 'Coding', level: 'Advanced'})Created 3 nodes (2 Person, 1 Skill)
Create Relationships
MATCH (a:Person {name: 'Alice'}), (b:Person {name: 'Bob'})
CREATE (a)-[:KNOWS {since: 2020}]->(b)
MATCH (a:Person {name: 'Alice'}), (s:Skill {name: 'Coding'})
CREATE (a)-[:HAS_SKILL {years: 5}]->(s)Alice KNOWS Bob (since 2020)
Alice HAS_SKILL Coding (5 years)
Find Friends of Friends
MATCH (person:Person {name: 'Alice'})-[:KNOWS]->(friend)-[:KNOWS]->(fof)
WHERE NOT (person)-[:KNOWS]->(fof) AND person <> fof
RETURN fof.name, COUNT(*) AS mutual_friends
ORDER BY mutual_friends DESCCarol | 3 mutual friends
David | 2 mutual friends
(People Alice doesn't know but her friends do)
Shortest Path
MATCH path = shortestPath(
(alice:Person {name: 'Alice'})-[:KNOWS*]-(target:Person {name: 'Eve'})
)
RETURN [node in nodes(path) | node.name] AS connection_path,
length(path) AS degrees_of_separationPath: [Alice, Bob, Carol, Eve]
Degrees: 3
Recommendation Query
MATCH (person:Person {name: 'Alice'})-[:LIKES]->(item:Product)
<-[:LIKES]-(other:Person)-[:LIKES]->(recommendation:Product)
WHERE NOT (person)-[:LIKES]->(recommendation)
RETURN recommendation.name, COUNT(*) AS score
ORDER BY score DESC
LIMIT 5Wireless Headphones | 12 (12 similar users liked it)
Standing Desk | 8
Mechanical Keyboard | 6
Pros & Cons
✅ Pros
- Excellent for relationship queries
- Intuitive data modeling
- Fast traversals (constant time)
- Flexible schema
- Pattern matching capabilities
❌ Cons
- Limited for non-graph queries
- Harder to scale horizontally
- Not ideal for simple CRUD operations
- Smaller ecosystem than relational
- Query performance varies with graph size
5. Time-Series Databases
Optimized for timestamped data
Time-series databases are optimized for data points indexed by time. They handle massive write loads and provide efficient storage through compression and downsampling. Perfect for IoT, monitoring, and financial data.
Popular Engines
InfluxDB
Purpose-built, high compression, built-in downsampling
TimescaleDB
PostgreSQL extension, SQL interface, relational features
Prometheus
Monitoring-focused, pull-based, PromQL query language
When to Use
- IoT sensor data
- Application and infrastructure monitoring
- Financial tick data and trading
- Industrial telemetry
- Real-time analytics
- Event logs and clickstreams
InfluxDB Examples
Write Data Points
temperature,location=office,sensor=sensor1 value=22.5 1642464000000000000 temperature,location=office,sensor=sensor1 value=23.1 1642464060000000000 temperature,location=warehouse,sensor=sensor2 value=18.3 1642464000000000000
3 data points written (measurement: temperature, tags: location/sensor, field: value)
Query Recent Data
from(bucket: "sensors") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "temperature") |> filter(fn: (r) => r.location == "office") |> mean()
Average office temperature: 22.8°C (last hour)
Downsampling (Aggregation)
from(bucket: "sensors") |> range(start: -24h) |> filter(fn: (r) => r._measurement == "cpu_usage") |> aggregateWindow(every: 5m, fn: mean) |> yield(name: "mean_5m")
CPU usage averaged into 5-minute buckets (reduces 1,440 points to 288)
Detect Anomalies
from(bucket: "sensors") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "temperature") |> filter(fn: (r) => r._value > 30.0 or r._value < 10.0) |> group(columns: ["location"])
warehouse | 35.2°C at 14:23 (alert: too hot!)
freezer | 8.1°C at 14:45 (alert: too warm!)
Pros & Cons
✅ Pros
- Extremely high write throughput
- Excellent compression ratios
- Built-in time-based operations
- Automatic data retention policies
- Optimized for time-range queries
❌ Cons
- Not for general-purpose use
- Updates/deletes are inefficient
- Limited JOIN capabilities
- Requires different query mindset
- Storage grows quickly without retention policies
6. Search Engines
Full-text search and analytics
Search engine databases are optimized for full-text search, faceted search, and text analysis. They index documents for fast searching and support features like fuzzy matching, relevance scoring, and highlighting.
Popular Engines
Elasticsearch
Most popular, distributed, RESTful API, rich aggregations
Apache Solr
Enterprise search, faceting, highlighting, spell-check
Meilisearch
Typo-tolerant, fast, easy to set up, instant search
When to Use
- E-commerce product search
- Log and event search (ELK stack)
- Document repositories
- Autocomplete and suggestions
- Content discovery
- Site search functionality
Elasticsearch Examples
Index a Document
POST /products/_doc/1
{
"name": "Wireless Headphones",
"description": "Premium noise-canceling headphones with Bluetooth 5.0",
"price": 199.99,
"category": "Electronics",
"tags": ["audio", "wireless", "bluetooth"]
}Document indexed with ID 1, ready for searching
Full-Text Search
GET /products/_search
{
"query": {
"multi_match": {
"query": "bluetooth headphones",
"fields": ["name^2", "description"]
}
}
}Wireless Headphones (score: 3.45)
Bluetooth Speaker (score: 2.12)
(Ranked by relevance, name field weighted 2x)
Fuzzy Search (Typo Tolerance)
GET /products/_search
{
"query": {
"match": {
"name": {
"query": "headphnes",
"fuzziness": "AUTO"
}
}
}
}Wireless Headphones (matched despite typo)
Gaming Headphones
Faceted Search (Filters + Aggregations)
GET /products/_search
{
"query": {
"bool": {
"must": [{"match": {"category": "Electronics"}}],
"filter": [{"range": {"price": {"gte": 100, "lte": 300}}}]
}
},
"aggs": {
"price_ranges": {
"range": {
"field": "price",
"ranges": [
{"to": 100}, {"from": 100, "to": 200}, {"from": 200}
]
}
}
}
}Facets: $0-100 (45 items) | $100-200 (12 items) | $200+ (8 items)
Autocomplete
GET /products/_search
{
"query": {
"match_phrase_prefix": {
"name": {
"query": "wirele"
}
}
},
"size": 5
}Wireless Headphones
Wireless Mouse
Wireless Keyboard
Pros & Cons
✅ Pros
- Extremely fast text search
- Typo tolerance and fuzzy matching
- Relevance scoring
- Rich aggregations and analytics
- Horizontal scalability
- Near real-time indexing
❌ Cons
- Not a primary database (no ACID)
- Memory intensive
- Complex to tune and optimize
- Updates require reindexing
- Can be expensive to operate
7. Vector Databases
Similarity search for AI/ML applications
Vector databases store and query high-dimensional vectors (embeddings) from machine learning models. They enable semantic search, recommendation systems, and RAG (Retrieval-Augmented Generation) for LLMs by finding similar items based on vector similarity.
Popular Engines
Pinecone
Fully managed, auto-scaling, optimized for production
Weaviate
Open-source, GraphQL API, built-in vectorization
Qdrant
Rust-based, fast, filtering capabilities, payload support
When to Use
- Semantic search (find similar content)
- RAG for LLMs (ChatGPT-style apps)
- Recommendation engines
- Image similarity search
- Anomaly detection
- Duplicate detection
Pinecone Examples
Insert Vectors (Embeddings)
import pinecone
index.upsert([
{
"id": "doc1",
"values": [0.1, 0.2, 0.3, ..., 0.8], # 1536-dim embedding
"metadata": {
"text": "Machine learning is a subset of AI",
"category": "AI"
}
},
{
"id": "doc2",
"values": [0.2, 0.1, 0.4, ..., 0.7],
"metadata": {
"text": "Deep learning uses neural networks",
"category": "AI"
}
}
])2 vectors inserted (typically from OpenAI, Cohere, or custom models)
Similarity Search
query_vector = [0.15, 0.18, 0.35, ..., 0.75] # Query embedding results = index.query( vector=query_vector, top_k=3, include_metadata=True )
doc1 | score: 0.95 | "Machine learning is..."
doc2 | score: 0.89 | "Deep learning uses..."
doc5 | score: 0.82 | "Neural networks are..."
Filtered Search
results = index.query(
vector=query_vector,
top_k=5,
filter={"category": {"$eq": "AI"}},
include_metadata=True
)Returns top 5 similar vectors, but only from "AI" category
RAG Pattern (for LLMs)
# 1. Convert user question to embedding
question = "What is machine learning?"
question_embedding = openai.Embedding.create(input=question)
# 2. Find relevant documents
results = index.query(vector=question_embedding, top_k=3)
# 3. Create context from results
context = "\n".join([r.metadata['text'] for r in results])
# 4. Send to LLM with context
response = openai.ChatCompletion.create(
messages=[
{"role": "system", "content": f"Context: {context}"},
{"role": "user", "content": question}
]
)LLM answers using retrieved relevant documents as context
Pros & Cons
✅ Pros
- Semantic similarity search
- Fast approximate nearest neighbor (ANN)
- Handles high-dimensional data
- Perfect for AI/ML workflows
- Enables RAG for LLMs
❌ Cons
- Requires ML models for embeddings
- Results are approximate, not exact
- High memory usage
- Embedding quality affects results
- Not suitable for traditional queries
8. Multi-Model Databases
One database, multiple data models
Multi-model databases support multiple data models (document, graph, key-value) in a single database engine. They eliminate the need for multiple specialized databases and reduce operational complexity.
Popular Engines
ArangoDB
Document, graph, key-value in one, AQL query language
OrientDB
Graph and document, SQL-like syntax, ACID compliant
Couchbase
Document and key-value, SQL++ queries, mobile sync
When to Use
- Applications with diverse data needs
- Reduce database sprawl
- When you need both documents and relationships
- Microservices with varied data patterns
- Prototyping and experimentation
ArangoDB Examples
Document Operations
// Insert document
db._create("users");
db.users.save({
name: "Alice",
email: "alice@example.com",
age: 28
});Document inserted with _key automatically generated
Graph Operations
// Create edge collection for relationships
db._createEdgeCollection("knows");
// Create relationships
db.knows.save({
_from: "users/alice",
_to: "users/bob",
since: 2020
});Edge created: Alice KNOWS Bob (since 2020)
Unified Query (AQL)
// Query combining documents and graph traversal
FOR user IN users
FILTER user.age >= 25
FOR friend IN 1..2 OUTBOUND user knows
RETURN {
user: user.name,
friend: friend.name,
friendAge: friend.age
}Alice → Bob (32)
Alice → Carol (29) (friend of friend)
(Combines document filtering with graph traversal)
Key-Value Access
// Fast key-value lookup
db.users.document("alice")
// Or by generated key
db.users.document("users/12345"){ name: "Alice", email: "alice@example.com", age: 28 }
Pros & Cons
✅ Pros
- Single database for multiple models
- Reduced operational complexity
- Unified query language
- Flexible data modeling
- Lower infrastructure costs
❌ Cons
- Jack of all trades, master of none
- May not excel at any single model
- Smaller community than specialized DBs
- Complex query optimization
- Potential performance trade-offs
Understanding CAP Theorem
The CAP theorem states that a distributed database can only guarantee two of three properties: Consistency, Availability, and Partition Tolerance. Understanding this helps you choose the right database.
Consistency
All nodes see the same data at the same time
Availability
Every request receives a response (success or failure)
Partition Tolerance
System continues to work despite network failures
Database Trade-offs
| Choice | Guarantees | Examples | Trade-off |
|---|---|---|---|
| CP (Consistency + Partition) | Strong consistency, tolerates partitions | MongoDB, HBase, Redis | May become unavailable during partition |
| AP (Availability + Partition) | Always available, tolerates partitions | Cassandra, DynamoDB, CouchDB | Eventual consistency (temporary stale reads) |
| CA (Consistency + Availability) | Strong consistency, always available | PostgreSQL, MySQL (single node) | Cannot tolerate network partitions |
Quick Comparison
| Type | Best For | Flagship | Scale |
|---|---|---|---|
| Document | Flexible schemas, nested data | MongoDB | Excellent |
| Key-Value | Caching, sessions, simple lookups | Redis | Excellent |
| Column-Family | Time-series, high write volume | Cassandra | Outstanding |
| Graph | Relationships, social networks | Neo4j | Good |
| Time-Series | IoT, monitoring, metrics | InfluxDB | Excellent |
| Search | Full-text search, analytics | Elasticsearch | Excellent |
| Vector | AI/ML, semantic search, RAG | Pinecone | Excellent |
| Multi-Model | Diverse data needs, flexibility | ArangoDB | Good |
Choosing the Right Database
Decision Framework
Hierarchical → Document | Flat key-value → Key-Value | Connected → Graph | Time-stamped → Time-Series
Simple lookups → Key-Value | Complex queries → Document/SQL | Traversals → Graph | Aggregations → Column-Family
Massive writes → Cassandra | High reads → Redis | Petabyte-scale → Cassandra/HBase
Strong ACID → PostgreSQL | Eventual consistency OK → Cassandra | Session consistency → MongoDB
Common Combinations
E-Commerce
PostgreSQL (transactions) + Elasticsearch (search) + Redis (cart/session) + Neo4j (recommendations)
Social Media
MongoDB (posts/profiles) + Neo4j (social graph) + Cassandra (feeds) + Redis (real-time notifications)
IoT Platform
InfluxDB (sensor data) + MongoDB (device metadata) + PostgreSQL (users/billing) + Redis (device state)
AI Application
Pinecone (embeddings) + PostgreSQL (structured data) + Redis (caching) + MongoDB (unstructured content)
Analytics Platform
Cassandra (event storage) + Elasticsearch (log search) + InfluxDB (metrics) + PostgreSQL (reporting)
Content Platform
MongoDB (CMS content) + Elasticsearch (content search) + Redis (page caching) + Neo4j (content relationships)
Migration & Integration
Moving from SQL to NoSQL
✅ Good Reasons
- Need horizontal scalability
- Schema is constantly changing
- Handling massive write volume
- Specific use case (graphs, time-series)
- Geographic distribution requirements
❌ Bad Reasons
- "NoSQL is faster" (not always true)
- "It's more modern" (hype-driven)
- Avoiding learning SQL
- Without understanding trade-offs
- Your data is truly relational
Polyglot Persistence Pattern
Use different databases for different parts of your application, each optimized for its purpose.
Polyglot Persistence Pattern
Each database handles what it does best
Data Synchronization
Common patterns for keeping multiple databases in sync:
Application-Level
Write to multiple databases in your code
✗ Can fail inconsistently
Change Data Capture
Monitor primary DB, replicate changes
✓ Eventually consistent
Event Streaming
Use Kafka/message queue as source of truth
✓ Scalable
NoSQL Best Practices
✅ Design for Your Queries
In NoSQL, you denormalize and design your schema around how you'll query the data, not around normalized entities like in SQL.
✅ Monitor Performance
NoSQL databases behave differently under load. Monitor query performance, memory usage, and replication lag continuously.
✅ Plan for Consistency
Understand your consistency requirements. Use stronger consistency for critical operations, eventual consistency for others.
✅ Backup and Recovery
NoSQL doesn't mean "no backups". Implement regular backups and test your recovery procedures.
✅ Index Strategically
Indexes speed up reads but slow writes and use memory. Only index fields you frequently query.
✅ Handle Growth
Plan for data growth. Implement data retention policies, archiving strategies, and sharding before you need them.
Key Takeaways
- NoSQL isn't better than SQL, it's different, with specific trade-offs
- Document stores offer flexibility with JSON-like documents
- Key-value stores provide blazing speed for simple lookups
- Column-family databases handle massive scale and writes
- Graph databases excel at relationship queries
- Time-series databases optimize for timestamped data
- Search engines enable full-text search and analytics
- Vector databases power AI/ML similarity search