Advanced Kubernetes

Helm, operators, service mesh, monitoring, and production-grade patterns

From Fundamentals to Production

You know pods, deployments, and services. Now it's time to level up. This lesson covers the tools and patterns that separate hobby clusters from production-grade infrastructure: package management with Helm, custom automation with Operators, zero-trust networking with Service Mesh, comprehensive observability, and battle-tested patterns from companies running Kubernetes at scale.

Helm, The Kubernetes Package Manager

Deploying complex applications means managing dozens of YAML files. Helm packages these into reusable, versioned "charts", think of it as apt/yum for Kubernetes.

Why Helm?

❌ Without Helm

Manage 20+ YAML files manually
Copy-paste with search-replace for values
No versioning or rollback
Difficult to share configurations
Environment differences = separate files
Hard to track what's deployed

✅ With Helm

Single command installs entire app
Template-based, parameterized configs
Version control and rollback built-in
Share via Helm repositories
One chart, multiple environments
Track releases and history

Core Concepts

Concept	Description
Chart	Package of Kubernetes resources. Contains templates, default values, and metadata.
Release	Instance of a chart running in your cluster. You can have multiple releases of same chart.
Repository	Place where charts are stored and shared (like Docker Hub for Helm charts).
Values	Configuration parameters that customize chart behavior for different environments.

Installing and Using Helm

# Install Helm
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Verify installation
$ helm version
version.BuildInfo{Version:"v3.14.0"}

# Add popular charts repositories
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# Update repo index
$ helm repo update

# Search for charts
$ helm search repo nginx
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
bitnami/nginx                   15.4.4          1.25.3          NGINX Open Source is a web server
ingress-nginx/ingress-nginx     4.9.0           1.9.5           Ingress controller for Kubernetes

Installing a Chart

# Install chart
$ helm install my-nginx bitnami/nginx
NAME: my-nginx
LAST DEPLOYED: Mon Jan 19 14:30:00 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1

# List releases
$ helm list
NAME        NAMESPACE   REVISION    STATUS      CHART           APP VERSION
my-nginx    default     1           deployed    nginx-15.4.4    1.25.3

# Get release status
$ helm status my-nginx

# Uninstall release
$ helm uninstall my-nginx

Customizing with Values

# values.yaml

replicaCount: 3

image:
  repository: nginx
  tag: "1.25-alpine"
  pullPolicy: IfNotPresent

service:
  type: LoadBalancer
  port: 80

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

ingress:
  enabled: true
  hosts:
    - host: myapp.example.com
      paths:
        - path: /
          pathType: Prefix

# Install with custom values
$ helm install my-app bitnami/nginx -f values.yaml

# Or set values via command line
$ helm install my-app bitnami/nginx \
  --set replicaCount=3 \
  --set service.type=LoadBalancer

# Upgrade release with new values
$ helm upgrade my-app bitnami/nginx -f values.yaml

# Rollback to previous version
$ helm rollback my-app 1

# View history
$ helm history my-app
REVISION    UPDATED                     STATUS          DESCRIPTION
1           Mon Jan 19 14:30:00 2026    superseded      Install complete
2           Mon Jan 19 15:00:00 2026    deployed        Upgrade complete

Creating Your Own Chart

# Create new chart
$ helm create my-app

$ tree my-app/
my-app/
├── Chart.yaml          # Chart metadata
├── values.yaml         # Default values
├── charts/             # Chart dependencies
├── templates/          # Kubernetes manifest templates
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── _helpers.tpl   # Template helpers
│   └── NOTES.txt      # Post-install notes
└── .helmignore

# templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "my-app.fullname" . }}
  labels:
    {{- include "my-app.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "my-app.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "my-app.selectorLabels" . | nindent 8 }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - containerPort: {{ .Values.service.port }}
        resources:
          {{- toYaml .Values.resources | nindent 12 }}

# Test rendering templates (dry-run)
$ helm template my-app ./my-app

# Validate chart
$ helm lint ./my-app

# Package chart
$ helm package ./my-app
Successfully packaged chart and saved it to: my-app-0.1.0.tgz

# Install local chart
$ helm install my-release ./my-app

Real-world usage:
• Install databases: helm install postgres bitnami/postgresql
• Install monitoring: helm install prometheus prometheus-community/prometheus
• Install ingress: helm install nginx ingress-nginx/ingress-nginx
Most popular open-source tools have official Helm charts!

Operators, Automating Complex Applications

Some applications need more than just deploying pods. They need domain-specific operational knowledge: backups, upgrades, scaling strategies, failover. Operators encode this knowledge as software.

What is an Operator?

An Operator encodes human operational knowledge into software. Instead of manually managing complex stateful workloads (databases, message queues, caches), an Operator automates the entire lifecycle: provisioning, configuration, scaling, backup, failover, and upgrades.

Operator = Custom Controller + Custom Resource Definitions (CRDs)

Traditional Approach:
1. Deploy database pods
2. Manually configure replication
3. Write scripts for backup
4. Handle failover manually
5. Manage upgrades carefully

Operator Approach:
1. Define: kubectl apply -f postgres-cluster.yaml
2. Operator handles everything automatically:
   - Sets up replication
   - Schedules backups
   - Monitors health
   - Handles failover
   - Manages upgrades

Example Flow:
┌─────────────────────┐
│  You: Create CRD    │ → PostgreSQL cluster with 3 replicas
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Operator watches    │ → Detects new resource
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Operator creates:   │ → 3 pods, services, persistent volumes
│ - Primary pod       │ → Configures replication
│ - 2 Replica pods    │ → Sets up backups
│ - PVCs, Services    │ → Monitors continuously
└─────────────────────┘

Popular Operators in Production

Operator	What It Manages	Key Features
Prometheus Operator	Prometheus monitoring clusters	Auto-discovery, configuration, alerting
PostgreSQL Operator	PostgreSQL databases	HA, backups, point-in-time recovery
Elasticsearch Operator	Elasticsearch clusters	Rolling upgrades, scaling, snapshots
Cert Manager	TLS certificates	Let's Encrypt integration, auto-renewal
Istio Operator	Service mesh	Installation, upgrades, configuration

Using an Operator (Example: Cert Manager)

# Install Cert Manager operator
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

# Verify installation
$ kubectl get pods -n cert-manager
NAME                                      READY   STATUS    AGE
cert-manager-5d7f97b46d-xp8rj            1/1     Running   1m
cert-manager-cainjector-69d885bf55-m4j2k 1/1     Running   1m
cert-manager-webhook-54754dcdfd-8kp9w    1/1     Running   1m

# Create ClusterIssuer (uses Let's Encrypt)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

# Request a certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-tls
  namespace: default
spec:
  secretName: myapp-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - myapp.example.com
  - www.myapp.example.com

$ kubectl apply -f certificate.yaml

# Cert Manager automatically:
# 1. Creates ACME challenge
# 2. Proves domain ownership
# 3. Gets certificate from Let's Encrypt
# 4. Stores in secret "myapp-tls-secret"
# 5. Auto-renews before expiration

$ kubectl get certificate
NAME        READY   SECRET              AGE
myapp-tls   True    myapp-tls-secret    2m

Magic: Cert Manager watches Certificate resources, handles the entire ACME challenge process, and automatically renews certificates. You just declare what you want!

Custom Resource Definitions (CRDs)

# Example: PostgreSQL CRD

apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: production-db
spec:
  teamId: "database-team"
  volume:
    size: 100Gi
  numberOfInstances: 3
  users:
    appuser:
    - superuser
    - createdb
  databases:
    myapp: appuser
  postgresql:
    version: "16"
  resources:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      cpu: 2000m
      memory: 4Gi
  patroni:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10

Result: Operator creates 3-node PostgreSQL cluster with automatic failover, creates users and databases, configures replication, handles backups. All from one YAML!

Service Mesh, Advanced Networking & Security

As microservices grow, inter-service communication becomes complex. Service Mesh adds a dedicated infrastructure layer for service-to-service communication with built-in observability, security, and traffic management.

Problems Service Mesh Solves

In microservices, every service must independently handle cross-cutting concerns like retries, TLS, and observability, leading to duplicated code and inconsistent behavior. A service mesh moves these concerns to a dedicated infrastructure layer (sidecar proxies), so application code stays focused on business logic.

Without Service Mesh:
- Each service implements retry logic
- Each service handles circuit breaking
- Mutual TLS requires code changes
- Observability scattered across services
- Traffic routing in application code
- Rate limiting per service

With Service Mesh:
- Transparent retries and timeouts
- Automatic circuit breaking
- mTLS between all services (zero-trust)
- Unified observability (traces, metrics)
- Declarative traffic routing
- Centralized rate limiting

Architecture:
┌────────────────────────────────────────────────────┐
│              Control Plane (Istiod)                │
│  (Configuration, Certificate Authority, Telemetry) │
└──────────────┬─────────────────────────────────────┘
               │ Pushes config
               ▼
    ┌──────────────────────────────────────┐
    │           Data Plane                 │
    │  (Envoy sidecars in each pod)        │
    └──────────────────────────────────────┘

Every pod gets a sidecar proxy (Envoy):
┌─────────────────────┐
│      Your Pod       │
│  ┌───────────────┐  │
│  │ App Container │  │
│  └───────┬───────┘  │
│          │          │
│  ┌───────▼───────┐  │
│  │ Envoy Sidecar │  │  ← Intercepts all traffic
│  └───────────────┘  │
└─────────────────────┘

Popular Service Meshes

Service Mesh	Strengths	Best For
Istio	Most features, powerful traffic management, enterprise-grade	Large orgs, complex requirements, multi-cluster
Linkerd	Lightweight, simple, fast, Rust-based	Smaller teams, performance-critical, simplicity
Consul	Multi-platform (K8s + VMs), service discovery, HashiCorp ecosystem	Hybrid environments, existing Consul users

Istio Example: Traffic Management

# Install Istio
$ curl -L https://istio.io/downloadIstio | sh -
$ cd istio-1.20.2
$ ./bin/istioctl install --set profile=default -y

# Enable sidecar injection for namespace
$ kubectl label namespace default istio-injection=enabled

# Deploy app - sidecars auto-injected
$ kubectl apply -f app.yaml

# Verify sidecars
$ kubectl get pods
NAME                    READY   STATUS    AGE
myapp-5d7f97b46d-xp8rj  2/2     Running   1m
                        ↑
                        2 containers: app + envoy

# Canary Deployment (90% v1, 10% v2)

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp.default.svc.cluster.local
  http:
  - match:
    - headers:
        user-type:
          exact: beta-tester
    route:
    - destination:
        host: myapp
        subset: v2
      weight: 100
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10

# Circuit Breaking

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-circuit-breaker
spec:
  host: backend-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

# Mutual TLS (mTLS) - Enforce encrypted traffic

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT  # All traffic must be encrypted

Zero code changes! Istio handles:
• Automatic certificate rotation
• Traffic splitting for canary deployments
• Circuit breaking to prevent cascade failures
• End-to-end encryption between all services

Monitoring & Observability, The Three Pillars

Production Kubernetes needs comprehensive observability: Metrics (what's happening),Logs (detailed events), and Traces (request flows).

Three Pillars of Observability:

1. METRICS (Numbers over time)
   - CPU, memory, request rate, error rate
   - "What is happening right now?"
   - Tools: Prometheus, Grafana

2. LOGS (Discrete events)
   - Application logs, error messages, events
   - "What happened at this specific time?"
   - Tools: ELK/EFK stack, Loki

3. TRACES (Request journey)
   - Follow a request across services
   - "Where did this request go and how long did each step take?"
   - Tools: Jaeger, Zipkin, Tempo

Together: Complete visibility into system behavior

Prometheus + Grafana Stack

# Install Prometheus + Grafana using Helm
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

# This installs:
# - Prometheus (metrics collection)
# - Grafana (visualization)
# - Alertmanager (alerting)
# - Node Exporter (node metrics)
# - Kube State Metrics (K8s resource metrics)

# Port forward to Grafana
$ kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

# Access Grafana: http://localhost:3000
# Default login: admin / prom-operator

Exposing Custom Metrics

# app with metrics endpoint

# Python Flask example
from flask import Flask
from prometheus_client import Counter, Histogram, generate_latest

app = Flask(__name__)

# Define metrics
REQUEST_COUNT = Counter('app_requests_total', 'Total requests', ['method', 'endpoint'])
REQUEST_DURATION = Histogram('app_request_duration_seconds', 'Request duration')

@app.route('/api/users')
@REQUEST_DURATION.time()
def users():
    REQUEST_COUNT.labels(method='GET', endpoint='/api/users').inc()
    return {"users": []}

@app.route('/metrics')
def metrics():
    return generate_latest()  # Prometheus scrapes this

if __name__ == '__main__':
    app.run(port=8080)

# ServiceMonitor - tells Prometheus to scrape

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-metrics
  namespace: default
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: http
    path: /metrics
    interval: 30s

Useful Prometheus Queries (PromQL)

# CPU usage by pod
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

# Memory usage by namespace
sum(container_memory_usage_bytes) by (namespace)

# Request rate (requests per second)
rate(http_requests_total[1m])

# Error rate percentage
rate(http_requests_total{status=~"5.."}[5m]) / 
rate(http_requests_total[5m]) * 100

# 95th percentile request duration
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Pods restarting frequently
rate(kube_pod_container_status_restarts_total[15m]) > 0

Alerting with PrometheusRule

# prometheus-alerts.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
  namespace: monitoring
spec:
  groups:
  - name: app-alerts
    interval: 30s
    rules:
    # High error rate
    - alert: HighErrorRate
      expr: |
        rate(http_requests_total{status=~"5.."}[5m]) / 
        rate(http_requests_total[5m]) > 0.05
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High error rate on {{ $labels.instance }}"
        description: "Error rate is {{ $value | humanizePercentage }}"
    
    # Pod memory usage high
    - alert: PodMemoryUsageHigh
      expr: |
        container_memory_usage_bytes / 
        container_spec_memory_limit_bytes > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} memory usage > 90%"
    
    # Pod restarting
    - alert: PodRestarting
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} is restarting"

Logging: EFK Stack (Elasticsearch, Fluentd, Kibana)

# Install EFK using Helm
$ helm repo add elastic https://helm.elastic.co
$ helm repo add fluent https://fluent.github.io/helm-charts

# Elasticsearch
$ helm install elasticsearch elastic/elasticsearch \
  --namespace logging --create-namespace

# Kibana
$ helm install kibana elastic/kibana \
  --namespace logging

# Fluentd (log collector)
$ helm install fluentd fluent/fluentd \
  --namespace logging

# Fluentd runs as DaemonSet (one per node)
# Collects logs from: /var/log/containers/*.log
# Forwards to: Elasticsearch
# Query in: Kibana UI

Alternative: Loki (Grafana's logging solution)
Lighter weight than Elasticsearch, integrates seamlessly with Grafana. Good for most use cases unless you need advanced Elasticsearch features.

Production Patterns, Battle-Tested Strategies

Real-world Kubernetes requires more than just deploying pods. Here are patterns from companies running production clusters at scale.

1. Blue-Green Deployment

Run two identical environments (blue and green). Switch traffic instantly, easy rollback.

Step 1: Blue (v1) is live, receiving traffic
┌──────────────┐
│   Service    │ → Blue deployment (v1)
└──────────────┘

Step 2: Deploy Green (v2), test internally
┌──────────────┐
│   Service    │ → Blue deployment (v1)  ← Live traffic
└──────────────┘
                  Green deployment (v2) ← Testing

Step 3: Switch service to Green
┌──────────────┐
│   Service    │ → Green deployment (v2) ← Live traffic
└──────────────┘
                  Blue deployment (v1)  ← Standby

Step 4: If issues, instant rollback
┌──────────────┐
│   Service    │ → Blue deployment (v1)  ← Rolled back
└──────────────┘

# Deploy green (v2)
$ kubectl apply -f deployment-green.yaml

# Test green internally
$ kubectl port-forward deployment/myapp-green 8080:8080

# Switch traffic (update service selector)
$ kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'

# Rollback if needed
$ kubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'

Pros: Instant rollback, no downtime
Cons: 2x resources during deployment, database migrations tricky

2. Canary Deployment

Gradually shift traffic from old to new version. Monitor metrics at each step.

Phase 1: 95% v1, 5% v2    → Monitor error rate, latency
Phase 2: 90% v1, 10% v2   → Check for issues
Phase 3: 75% v1, 25% v2   → Confidence building
Phase 4: 50% v1, 50% v2   → Half traffic
Phase 5: 0% v1, 100% v2   → Full rollout

If any phase shows issues → rollback immediately

# Using Argo Rollouts (GitOps tool)

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10     # 10% to canary
      - pause: {duration: 5m}
      - setWeight: 25     # 25% to canary
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 75
      - pause: {duration: 5m}
  template:
    spec:
      containers:
      - name: myapp
        image: myapp:v2

3. GitOps Pattern

Git as single source of truth. Cluster state automatically syncs with Git repository.

Traditional:
Developer → kubectl apply → Cluster
Problems: No audit trail, hard to rollback, manual

GitOps:
Developer → Git commit → GitOps tool watches → Cluster syncs
Benefits: Git history = deployment history, easy rollback, automated

┌──────────┐   commit   ┌──────────┐   watches   ┌────────────┐
│Developer │ ────────▶  │   Git    │  ◀────────  │ ArgoCD/Flux│
└──────────┘            │Repository│             └──────┬─────┘
                        └──────────┘                    │
                                                        │ syncs
                                                        ▼
                                                  ┌───────────┐
                                                  │  Cluster  │
                                                  └───────────┘

# Install ArgoCD
$ kubectl create namespace argocd
$ kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Access UI
$ kubectl port-forward svc/argocd-server -n argocd 8080:443

# Add Git repository
$ argocd repo add https://github.com/yourorg/k8s-manifests

# Create application
$ argocd app create myapp \
  --repo https://github.com/yourorg/k8s-manifests \
  --path k8s/production \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default

# Enable auto-sync
$ argocd app set myapp --sync-policy automated

Now: Every Git commit triggers automatic deployment. Rollback = git revert. Full audit trail in Git history.

4. Resource Management Strategy

# QoS Classes in Kubernetes

# GUARANTEED (highest priority)
# - requests == limits for all containers
# - Won't be evicted unless exceeds limits
resources:
  requests:
    memory: "1Gi"
    cpu: "1"
  limits:
    memory: "1Gi"
    cpu: "1"

# BURSTABLE (medium priority)
# - requests < limits
# - Can use extra resources when available
# - May be evicted under pressure
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "2Gi"
    cpu: "2"

# BEST EFFORT (lowest priority)
# - No requests or limits
# - First to be evicted
# - Use for non-critical workloads only
# (No resources specified)

5. Pod Disruption Budgets (High Availability)

# Ensure minimum availability during disruptions

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
spec:
  minAvailable: 2  # Always keep at least 2 pods running
  # OR: maxUnavailable: 1  (max 1 pod can be down)
  selector:
    matchLabels:
      app: myapp

# Protects against:
# - Node draining
# - Cluster upgrades
# - Voluntary evictions
# Kubernetes won't evict pods if it violates PDB

6. Pod Affinity & Anti-Affinity

# Spread pods across nodes for HA

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      # ANTI-AFFINITY: Don't schedule pods on same node
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - myapp
            topologyKey: kubernetes.io/hostname

# Result: 3 replicas spread across 3 different nodes
# If node fails, only 1 pod affected (not all 3)

# AFFINITY: Schedule related pods together

# Example: Web app should be on same node as cache for low latency
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  template:
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - redis-cache
            topologyKey: kubernetes.io/hostname

# Result: Webapp pods scheduled on same nodes as Redis pods

7. Init Containers Pattern

# Run setup tasks before main container starts

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  # Init containers run sequentially, must succeed
  initContainers:
  # Wait for database to be ready
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 
      'until nc -z postgres 5432; do echo waiting; sleep 2; done']
  
  # Run database migrations
  - name: run-migrations
    image: myapp:1.0
    command: ['python', 'manage.py', 'migrate']
    env:
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: db-creds
          key: url
  
  # Download config from S3
  - name: fetch-config
    image: amazon/aws-cli
    command: ['aws', 's3', 'cp', 's3://configs/app.json', '/config/']
    volumeMounts:
    - name: config
      mountPath: /config
  
  # Main application container (starts after init containers succeed)
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: config
      mountPath: /config
  
  volumes:
  - name: config
    emptyDir: {}

Use cases:
• Wait for dependencies
• Run database migrations
• Fetch configurations
• Clone Git repos
• Warm up caches

Security Best Practices

Security in Kubernetes is multi-layered. Here are essential practices for production clusters.

Role-Based Access Control (RBAC)

# Create read-only role for developers

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer-role
  namespace: development
rules:
# Can read pods, services, deployments
- apiGroups: ["", "apps"]
  resources: ["pods", "services", "deployments"]
  verbs: ["get", "list", "watch"]
# Can view logs
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
  namespace: development
subjects:
- kind: User
  name: alice@company.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer-role
  apiGroup: rbac.authorization.k8s.io

Pod Security Standards

# Secure pod configuration

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  # Run as non-root user
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  
  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      # Don't allow privilege escalation
      allowPrivilegeEscalation: false
      # Drop all capabilities, add only what's needed
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE
      # Read-only root filesystem
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache
  
  volumes:
  # Writable directories via tmpfs
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

Network Policies (Micro-segmentation)

# Deny all traffic by default, allow specific

# Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress

---
# Allow frontend to talk to backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

---
# Allow backend to access database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 5432

Result: Zero-trust networking. By default, nothing can talk to anything. Only explicitly allowed connections work. Limits lateral movement if compromised.

Cost Optimization Strategies

Cloud bills can spiral. Here's how to optimize Kubernetes costs without sacrificing reliability.

Right-Sizing Resources

Most pods are over-provisioned

# Use VPA (Vertical Pod Autoscaler)
# Recommends optimal requests/limits
$ kubectl top pods
# Compare usage vs requests

Spot/Preemptible Instances

70-90% cheaper for fault-tolerant workloads

# Use node taints/tolerations
# Schedule batch jobs on spot
# Keep stateful apps on on-demand

Cluster Autoscaler

Scale down unused nodes automatically

# Removes nodes <50% utilized
# for 10+ minutes
# Can save 30-50% on compute

Namespace Quotas

Prevent runaway resource consumption

# Set hard limits per namespace
# Force teams to optimize
# Track costs by team/project

# Monitor costs with Kubecost

$ helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost --create-namespace

# Provides:
# - Cost per namespace, deployment, pod
# - Right-sizing recommendations
# - Idle resource detection
# - Cost allocation and chargeback

Quick wins:
1. Delete unused resources (kubectl get all --all-namespaces)
2. Right-size over-provisioned pods (use VPA)
3. Enable cluster autoscaler
4. Use spot instances for non-critical workloads
5. Implement PodDisruptionBudgets to allow safe scale-down

Key Takeaways

Helm: Package manager for K8s, templates, versioning, reusable charts
Operators: Encode operational knowledge as software, CRDs + controllers
Service Mesh: Istio/Linkerd, mTLS, traffic management, observability without code changes
Monitoring: Prometheus (metrics), Grafana (dashboards), EFK/Loki (logs), Jaeger (traces)
Deployment Patterns: Blue-green (instant switch), canary (gradual), GitOps (Git as truth)
High Availability: PodDisruptionBudgets, anti-affinity, multi-zone deployments
Security: RBAC, pod security standards, network policies, non-root containers
Cost Optimization: Right-sizing, cluster autoscaler, spot instances, namespace quotas
Init Containers: Setup tasks before main container, migrations, config fetching
Production Readiness: Combine all these patterns for reliable, secure, cost-effective clusters

Back to "Kubernetes Fundamentals"

Infrastructure for EngineersLesson 6

Next: "CI/CD Pipelines"