Vector Databases & AI Integration

Storing and searching high-dimensional embeddings for machine learning

The AI Database Revolution

Traditional databases excel at exact matches: "find user with ID 42" or "products under $100." But modern AI applications need to answer questions like "find images similar to this one," "recommend products based on user behavior," or "retrieve documents semantically related to a query." These tasks require comparing high-dimensional vectors (embeddings) where each data point is represented as a list of 384 to 1536+ floating-point numbers. Storing and searching billions of these vectors in PostgreSQL is impossibly slow, a single similarity search across 10 million vectors would take minutes. Vector databases solve this with specialized indexing algorithms (HNSW, IVF-PQ) that reduce search time from O(n) to O(log n), enabling millisecond queries across billions of embeddings. This lesson covers Pinecone (managed, production-ready), Milvus (open-source, horizontally scalable), and Weaviate (schema-based with GraphQL). You'll learn how embeddings work, similarity search algorithms, RAG (Retrieval Augmented Generation) patterns, and integrating vector search with traditional databases for hybrid search systems.

Real-World Scale: OpenAI's ChatGPT uses vector databases to search millions of documents in real-time for RAG. Spotify stores 100M+ song embeddings for music recommendations. Google Images compares your search image against billions of image embeddings in milliseconds. Vector databases power the AI applications you use daily.

Understanding Embeddings

Embeddings are numerical representations of data (text, images, audio) in high-dimensional space where semantically similar items are close together. "Dog" and "puppy" have similar embeddings; "dog" and "spaceship" are far apart.

From Text to Vectors

Traditional keyword search:

Query: "king" → Exact match only
Misses: "monarch," "ruler," "sovereign" (synonyms)
No semantic understanding

Vector embeddings:

"king" → [0.24, -0.15, 0.89, ..., 0.42] (384 dimensions)
"monarch" → [0.22, -0.14, 0.91, ..., 0.40] (very similar!)
"pizza" → [-0.62, 0.73, -0.11, ..., -0.88] (far apart)

Key Insight: Embeddings capture semantic meaning. The distance between vectors represents similarity, not exact matches.

Generating Embeddings with OpenAI

# Install: pip install openai

import openai

openai.api_key = "sk-..."

# Generate text embedding
response = openai.embeddings.create(
    model="text-embedding-3-small",  # 1536 dimensions
    input="The quick brown fox jumps over the lazy dog"
)

embedding = response.data[0].embedding
print(f"Embedding dimensions: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")

Result:

Embedding dimensions: 1536
First 5 values: [0.0234, -0.0156, 0.0891, -0.0423, 0.0067]

Why PostgreSQL Can't Handle This

Brute-force search: Must compare query vector to every stored vector (O(n) complexity)
No specialized indexes: B-trees don't work for high-dimensional data
Performance collapse: 10M vectors × 1536 dimensions = billions of comparisons per query
No approximate search: Can't trade accuracy for speed

Pinecone: Managed Vector Database

Pinecone is a fully managed vector database optimized for production AI applications. No infrastructure setup, automatic scaling, and sub-50ms queries at billion-vector scale.

Step 1: Create Index

# Install: pip install pinecone-client

import pinecone

# Initialize
pinecone.init(
    api_key="YOUR_API_KEY",
    environment="us-west1-gcp"
)

# Create index
pinecone.create_index(
    name="semantic-search",
    dimension=1536,  # Must match embedding model
    metric="cosine"  # cosine, euclidean, or dotproduct
)

index = pinecone.Index("semantic-search")
print(f"Index created: {index.describe_index_stats()}")

Result: Index 'semantic-search' ready with 1536 dimensions, cosine similarity

Step 2: Insert Vectors (Upsert)

# Insert vectors with metadata
vectors = [
    {
        "id": "doc1",
        "values": embedding_1,  # 1536-dim vector from OpenAI
        "metadata": {
            "text": "Python is a programming language",
            "category": "tech"
        }
    },
    {
        "id": "doc2",
        "values": embedding_2,
        "metadata": {
            "text": "Dogs are loyal companions",
            "category": "animals"
        }
    },
    {
        "id": "doc3",
        "values": embedding_3,
        "metadata": {
            "text": "JavaScript powers web applications",
            "category": "tech"
        }
    }
]

# Upsert (insert or update)
index.upsert(vectors=vectors)
print(f"Inserted {len(vectors)} vectors")

Result: Inserted 3 vectors, Pinecone automatically indexes them for fast retrieval.

Step 3: Similarity Search

# Search for similar vectors
query_text = "What are the best coding languages?"

# Get query embedding
query_response = openai.embeddings.create(
    model="text-embedding-3-small",
    input=query_text
)
query_embedding = query_response.data[0].embedding

# Search in Pinecone
results = index.query(
    vector=query_embedding,
    top_k=3,
    include_metadata=True
)

# Display results
for match in results['matches']:
    print(f"Score: {match['score']:.4f}")
    print(f"Text: {match['metadata']['text']}")
    print(f"Category: {match['metadata']['category']}")
    print()

Result:

Score: 0.8942
Text: Python is a programming language
Category: tech

Score: 0.8735
Text: JavaScript powers web applications
Category: tech

Score: 0.2134
Text: Dogs are loyal companions
Category: animals

Note: Higher scores = more similar. Semantic search found programming languages, not keyword matches!

Step 4: Filtered Search (Hybrid)

# Search with metadata filtering
results = index.query(
    vector=query_embedding,
    top_k=10,
    filter={
        "category": {"$eq": "tech"}  # Only tech documents
    },
    include_metadata=True
)

for match in results['matches']:
    print(f"{match['score']:.4f}: {match['metadata']['text']}")

Result: Returns only tech documents, ranked by semantic similarity. Combines vector search with traditional filtering!

Milvus: Open-Source Vector Database

Milvus is an open-source vector database that supports horizontal scaling, multiple index types, and GPU acceleration. Ideal for self-hosted, high-scale deployments.

Step 1: Connect and Create Collection

# Install: pip install pymilvus

from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType

# Connect to Milvus
connections.connect(
    alias="default",
    host="localhost",
    port="19530"
)

# Define schema
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=1536),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=1000),
    FieldSchema(name="category", dtype=DataType.VARCHAR, max_length=50)
]

schema = CollectionSchema(fields=fields, description="Document embeddings")

# Create collection
collection = Collection(name="documents", schema=schema)
print(f"Collection created: {collection.name}")

Result: Collection created: documents

Step 2: Create Index (HNSW for Speed)

# Create HNSW index for fast approximate search
index_params = {
    "index_type": "HNSW",  # Hierarchical Navigable Small World
    "metric_type": "IP",   # Inner Product (similar to cosine)
    "params": {
        "M": 16,           # Number of connections per layer
        "efConstruction": 256  # Search depth during construction
    }
}

collection.create_index(
    field_name="embedding",
    index_params=index_params
)

collection.load()  # Load into memory for searching
print("Index created and loaded")

Result: HNSW index created. Enables O(log n) search instead of O(n) brute force!

Step 3: Insert Embeddings

# Prepare data
data = [
    {
        "embedding": embedding_1.tolist(),
        "text": "Machine learning models require training data",
        "category": "AI"
    },
    {
        "embedding": embedding_2.tolist(),
        "text": "Neural networks use backpropagation",
        "category": "AI"
    },
    {
        "embedding": embedding_3.tolist(),
        "text": "Databases store structured information",
        "category": "databases"
    }
]

# Convert to column format
entities = [
    [item["embedding"] for item in data],
    [item["text"] for item in data],
    [item["category"] for item in data]
]

# Insert
collection.insert(entities)
collection.flush()  # Ensure data is persisted
print(f"Inserted {len(data)} documents")

Result: Inserted 3 documents

Step 4: Vector Search with Filtering

# Search parameters
search_params = {
    "metric_type": "IP",
    "params": {"ef": 64}  # Search depth (higher = more accurate, slower)
}

# Perform search with filter
results = collection.search(
    data=[query_embedding],
    anns_field="embedding",
    param=search_params,
    limit=3,
    expr='category == "AI"',  # SQL-like filtering
    output_fields=["text", "category"]
)

# Display results
for hits in results:
    for hit in hits:
        print(f"Distance: {hit.distance:.4f}")
        print(f"Text: {hit.entity.get('text')}")
        print(f"Category: {hit.entity.get('category')}")
        print()

Result:

Distance: 0.9234
Text: Machine learning models require training data
Category: AI

Distance: 0.8876
Text: Neural networks use backpropagation
Category: AI

Weaviate: Schema-Based Vector Database

Weaviate combines vector search with traditional database features: schemas, relationships, and GraphQL queries. Great for complex data models with semantic search.

Step 1: Create Schema

# Install: pip install weaviate-client

import weaviate

# Connect
client = weaviate.Client("http://localhost:8080")

# Define schema
schema = {
    "classes": [
        {
            "class": "Article",
            "description": "A blog article",
            "vectorizer": "text2vec-openai",  # Auto-vectorize text
            "moduleConfig": {
                "text2vec-openai": {
                    "model": "text-embedding-3-small"
                }
            },
            "properties": [
                {
                    "name": "title",
                    "dataType": ["text"],
                    "description": "Article title"
                },
                {
                    "name": "content",
                    "dataType": ["text"],
                    "description": "Article content"
                },
                {
                    "name": "category",
                    "dataType": ["string"],
                    "description": "Article category"
                },
                {
                    "name": "publishDate",
                    "dataType": ["date"]
                }
            ]
        }
    ]
}

client.schema.create(schema)
print("Schema created")

Result: Schema created, Weaviate will automatically vectorize text fields!

Step 2: Insert Objects (Auto-Vectorization)

# Insert articles (vectorization happens automatically)
articles = [
    {
        "title": "Introduction to Python",
        "content": "Python is a versatile programming language...",
        "category": "Programming",
        "publishDate": "2025-01-15T00:00:00Z"
    },
    {
        "title": "Deep Learning Basics",
        "content": "Neural networks are the foundation of deep learning...",
        "category": "AI",
        "publishDate": "2025-02-01T00:00:00Z"
    },
    {
        "title": "Database Design Patterns",
        "content": "Normalization and indexing are key to performance...",
        "category": "Databases",
        "publishDate": "2025-01-20T00:00:00Z"
    }
]

# Batch insert
with client.batch as batch:
    for article in articles:
        batch.add_data_object(
            data_object=article,
            class_name="Article"
        )

print(f"Inserted {len(articles)} articles")

Result: Inserted 3 articles, Embeddings generated automatically via OpenAI API.

Step 3: Semantic Search with GraphQL

# Semantic search using natural language
result = (
    client.query
    .get("Article", ["title", "content", "category"])
    .with_near_text({
        "concepts": ["machine learning and artificial intelligence"]
    })
    .with_limit(2)
    .do()
)

# Display results
for article in result['data']['Get']['Article']:
    print(f"Title: {article['title']}")
    print(f"Category: {article['category']}")
    print(f"Content: {article['content'][:80]}...")
    print()

Result:

Title: Deep Learning Basics
Category: AI
Content: Neural networks are the foundation of deep learning...

Title: Introduction to Python
Category: Programming
Content: Python is a versatile programming language...

Step 4: Hybrid Search (Vector + Keyword)

# Combine semantic search with keyword filtering
result = (
    client.query
    .get("Article", ["title", "content", "category"])
    .with_hybrid(
        query="neural networks",
        alpha=0.5  # 0.5 = balanced (0=keyword only, 1=vector only)
    )
    .with_where({
        "path": ["category"],
        "operator": "Equal",
        "valueString": "AI"
    })
    .with_limit(3)
    .do()
)

for article in result['data']['Get']['Article']:
    print(f"- {article['title']} ({article['category']})")

Result: - Deep Learning Basics (AI), Found via both semantic and keyword matching!

Similarity Search Algorithms

Vector databases use distance metrics to measure similarity. Understanding these is crucial for choosing the right metric for your use case.

Cosine Similarity

Measures angle between vectors. Range: -1 to 1 (1 = identical direction).

(A·B) / (||A|| ||B||)

Best for: Text embeddings, normalized vectors

Euclidean (L2)

Straight-line distance between points. Sensitive to magnitude.

√(Σ(A[i] - B[i])²)

Best for: Image embeddings, spatial data

Dot Product

Sum of element-wise products. Fast, considers magnitude.

Σ(A[i] × B[i])

Best for: Recommendation systems, ranking

Comparing Distance Metrics

import numpy as np
from numpy.linalg import norm

# Sample vectors
vec1 = np.array([1.0, 2.0, 3.0])
vec2 = np.array([1.5, 2.2, 2.8])

# Cosine similarity
cosine = np.dot(vec1, vec2) / (norm(vec1) * norm(vec2))
print(f"Cosine similarity: {cosine:.4f}")

# Euclidean distance
euclidean = norm(vec1 - vec2)
print(f"Euclidean distance: {euclidean:.4f}")

# Dot product
dot_prod = np.dot(vec1, vec2)
print(f"Dot product: {dot_prod:.4f}")

Result:

Cosine similarity: 0.9892
Euclidean distance: 0.5745
Dot product: 14.3000

Indexing Strategies for Speed

Algorithm	Type	Speed	Accuracy	Best For
HNSW	Graph-based	Very Fast	High	< 10M vectors, low latency
IVF-PQ	Quantization	Fast	Medium	> 10M vectors, memory constrained
Flat	Brute force	Slow	Perfect	< 1M vectors, exact search
ANNOY	Tree-based	Medium	Medium	Static datasets, read-heavy

RAG: Retrieval Augmented Generation

RAG combines vector search with Large Language Models (LLMs) to provide accurate, contextual responses grounded in your data. This powers ChatGPT-like experiences over private documents.

How RAG Works

Step 1: Index Documents

Split documents into chunks (500-1000 tokens)
Generate embeddings for each chunk
Store in vector database with metadata

Step 2: User Query

User asks: "What are our refund policies?"
Generate embedding for query
Search vector DB for top 5 similar chunks

Step 3: LLM Generation

Inject retrieved chunks into LLM prompt
LLM generates answer using retrieved context
Answer is grounded in your documents, not hallucinated!

Implementing RAG with Pinecone + OpenAI

import openai
import pinecone

# Step 1: Retrieve relevant context
def retrieve_context(query, top_k=3):
    # Get query embedding
    query_emb = openai.embeddings.create(
        model="text-embedding-3-small",
        input=query
    ).data[0].embedding

    # Search vector DB
    results = index.query(
        vector=query_emb,
        top_k=top_k,
        include_metadata=True
    )

    # Extract text chunks
    contexts = [match['metadata']['text'] for match in results['matches']]
    return "\n\n".join(contexts)

# Step 2: Generate answer with LLM
def answer_question(question):
    # Retrieve relevant context
    context = retrieve_context(question)

    # Build prompt with context
    prompt = f"""Answer the question based on the context below.

Context:
{context}

Question: {question}

Answer:"""

    # Generate answer
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )

    return response.choices[0].message.content

# Usage
question = "What are the benefits of vector databases?"
answer = answer_question(question)
print(answer)

Result:

Vector databases enable fast similarity search across billions of
high-dimensional embeddings. They use specialized indexes like HNSW
to achieve sub-50ms query times, making them essential for AI applications
like semantic search, recommendation systems, and RAG pipelines.

Answer generated from retrieved context, not LLM's training data!

Integration with Big Data Pipelines

Vector databases integrate with data pipelines to keep embeddings in sync with source data (PostgreSQL, Kafka, S3, etc.).

Real-Time Ingestion with Kafka

from kafka import KafkaConsumer
import openai
import pinecone
import json

# Consume from Kafka topic
consumer = KafkaConsumer(
    'documents',
    bootstrap_servers=['localhost:9092'],
    value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)

# Process messages and index in vector DB
for message in consumer:
    doc = message.value

    # Generate embedding
    embedding = openai.embeddings.create(
        model="text-embedding-3-small",
        input=doc['content']
    ).data[0].embedding

    # Upsert to Pinecone
    index.upsert(vectors=[{
        "id": doc['id'],
        "values": embedding,
        "metadata": {
            "title": doc['title'],
            "source": doc['source']
        }
    }])

    print(f"Indexed document: {doc['id']}")

Result: Documents are automatically indexed as they arrive in Kafka. Real-time semantic search!

Batch Processing with Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import ArrayType, FloatType

spark = SparkSession.builder.appName("VectorIndexing").getOrCreate()

# Load data from S3
df = spark.read.parquet("s3://bucket/documents/")

# UDF to generate embeddings
@udf(returnType=ArrayType(FloatType()))
def generate_embedding(text):
    response = openai.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

# Apply UDF
df_with_embeddings = df.withColumn("embedding", generate_embedding(df.content))

# Batch upsert to Milvus (or Pinecone)
def upsert_to_milvus(partition_iter):
    from pymilvus import Collection
    collection = Collection("documents")

    for row in partition_iter:
        collection.insert([
            [row.embedding],
            [row.title],
            [row.category]
        ])

    collection.flush()

df_with_embeddings.foreachPartition(upsert_to_milvus)
print("Batch indexing complete")

Result: Millions of documents indexed in parallel using Spark's distributed processing.

Choosing the Right Vector Database

Choose Pinecone

Production-ready out of the box
No infrastructure management
Automatic scaling and backups
Startup/MVP speed
Budget for managed service

Choose Milvus

Billion+ vector scale
Self-hosted requirements
GPU acceleration needed
Advanced index control
Cost optimization via self-hosting

Choose Weaviate

Complex schemas with relationships
GraphQL query flexibility
Hybrid search (keyword + vector)
Multi-tenancy requirements
Auto-vectorization preferred

Key Takeaways

Vector databases solve AI-scale search: Traditional databases can't handle billion-vector similarity search at millisecond latency
Embeddings capture semantic meaning: Similar concepts have similar vectors, enabling "Google-like" search over any data type
Choose based on scale and ops: Pinecone (managed, easy), Milvus (scale, self-hosted), Weaviate (complex schemas)
RAG is the killer app: Combine vector search with LLMs for accurate, grounded AI responses
Hybrid search wins: Combine vector similarity with traditional filters (metadata, dates, categories) for production systems

Back to "Full-Text Search"

Databases AdvancedLesson 20

Next: "Database Security"