managedsecret-operator-public/examples/EXAMPLES-SUMMARY.md

ManagedSecret Operator - Examples Summary

This directory contains comprehensive examples for the ManagedSecret Operator, demonstrating various rotation strategies and use cases.

Files Overview

Documentation

• README.md - Complete installation and usage guide with new features
  • Added keepPreviousVersion and previousVersionTTL documentation
  • Added Reloader integration guide
  • Updated best practices and architecture diagrams

Example Files

Example 1: No Rotation (Simple API Key)

File: example-1-no-rotation.yaml

Use Case: Long-lived API keys that don't need rotation

Key Features:

• Generated once and kept indefinitely
• No Reloader needed (secret never changes)
• Perfect for external service API keys

When to Use:

• Third-party API keys (Stripe, SendGrid, etc.)
• Service identifiers
• Static configuration values

---

Example 2: Simple Rotation with Reloader (Container Registry)

File: example-2-registry-rotation-simple.yaml

Use Case: Docker/Container registry with HTTP basic auth

Key Features:

• Password rotates every 90 days
• InitContainer regenerates htpasswd on pod restart
• Reloader automatically restarts pods
• No keepPreviousVersion needed - htpasswd is regenerated fresh

Rotation Flow:

1. ManagedSecret generates new password → Updates Vault → Updates K8s Secret
2. Reloader detects Secret change → Triggers pod restart
3. InitContainer runs → Reads new password → Generates fresh htpasswd
4. Registry starts with new htpasswd

When to Use:

• Container registries (Docker Registry, Harbor)
• Web servers with htpasswd (Apache, Nginx)
• Any service where credentials are regenerated at startup

---

Example 3: Complex Rotation with Previous Password (PostgreSQL)

File: example-3-postgres-rotation-previous.yaml

Use Case: PostgreSQL user password that requires authentication to change

Key Features:

• Password rotates every 90 days
• keepPreviousVersion: true creates -previous secret
• CronJob uses old password to authenticate, sets new password
• previousVersionTTL: 1h cleans up after grace period

Rotation Flow:

1. ManagedSecret creates both postgres-secret and postgres-secret-previous
2. CronJob (every 5 min) detects rotation
3. Authenticates with OLD password: PGPASSWORD=$OLD_PASSWORD psql
4. Changes password: ALTER USER app_user PASSWORD '$NEW_PASSWORD'
5. Verifies new password works
6. After 1 hour, -previous secret is deleted

When to Use:

• PostgreSQL, MySQL, MariaDB
• Any database requiring authentication to change passwords
• Services with ALTER USER or similar commands

---

Example 4: MinIO Admin Password Rotation

File: example-4-minio-rotation-previous.yaml

Use Case: MinIO admin credentials with rotation

Key Features:

• Password rotates every 60 days
• Uses MinIO Client (mc) to change password
• keepPreviousVersion: true with 2h TTL
• CronJob handles rotation via mc admin user password

Special Considerations:

• MinIO does NOT reload credentials from environment variables
• Manual pod restart required after rotation
• Consider using a sidecar to monitor and restart

Rotation Flow:

1. ManagedSecret creates both secrets
2. CronJob authenticates with old credentials: mc alias set oldminio
3. Changes password: mc admin user password myminio admin $NEW_PASSWORD
4. Verifies new credentials
5. Important: MinIO pods must be restarted manually

When to Use:

• MinIO object storage
• Any service that doesn't auto-reload credentials
• Services requiring separate admin tools for password changes

---

Example 5: MySQL User Password Rotation

File: example-5-mysql-rotation-previous.yaml

Use Case: MySQL application user with automated rotation

Key Features:

• Password rotates every 90 days
• Multiple password change methods (ALTER USER, SET PASSWORD)
• Root credentials as fallback
• keepPreviousVersion: true with 1h TTL

Rotation Flow:

1. ManagedSecret creates both secrets
2. CronJob tries multiple approaches:
   • Method 1: ALTER USER as root (preferred)
   • Method 2: SET PASSWORD as root
   • Method 3: SET PASSWORD as user with old password
3. Flushes privileges
4. Verifies new password

When to Use:

• MySQL 5.7+, MySQL 8.0
• MariaDB
• Any MySQL-compatible database

---

Example 6: Comprehensive Multi-Service Deployment

File: example-6-comprehensive-multi-service.yaml

Use Case: Complete application stack showing all strategies together

Components:

1. PostgreSQL - Rotates with previous version (90 days)
2. Redis - Rotates without previous version (30 days)
3. API Keys - No rotation (managed externally)
4. TLS Certificates - No rotation (cert-manager)
5. JWT Secret - Simple rotation with Reloader (180 days)

Demonstrates:

• Multiple secrets with different rotation schedules
• Mixed rotation strategies in one application
• How Reloader restarts pods for all credential changes
• Real-world production setup

When to Use:

• As a template for complex applications
• To understand how different strategies work together
• For learning all features in one example

---

Quick Decision Tree

Does your service need the OLD password to set the NEW password?

YES (PostgreSQL, MySQL, MinIO, LDAP):

• Use keepPreviousVersion: true
• Set appropriate previousVersionTTL
• Create CronJob to handle rotation
• See Examples 3, 4, or 5

NO (Registry, Redis, Web servers):

• Use simple rotation with Reloader
• InitContainer or startup script regenerates credentials
• No CronJob needed
• See Example 2

NOT ROTATING (API keys, Certificates):

• Set rotation.enabled: false
• No Reloader annotation needed
• See Example 1

---

Rotation Schedule Recommendations

• High Security (User passwords, admin credentials): 30-90 days
• Medium Security (Application credentials, service accounts): 90-180 days
• Low Security (Read-only API keys, development): 180+ days or disabled
• Previous Version TTL:
  • Automated rotation: 1h
  • Manual intervention: 2-4h
  • Complex workflows: 24h

---

Prerequisites for All Examples

1. OpenBao/Vault deployed and initialized
2. ManagedSecret Operator installed
3. Reloader installed (for rotation examples)
4. cert-manager (optional, for TLS examples)

Installation Order

# 1. Install OpenBao
kubectl create namespace openbao
helm install openbao openbao/openbao -n openbao

# 2. Install ManagedSecret Operator
kubectl apply -f managedsecret-operator/

# 3. Install Reloader
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

# 4. Deploy your application with examples
kubectl apply -f example-X-your-usecase.yaml

---

Testing Rotation

Force Immediate Rotation

# Trigger rotation for any ManagedSecret
kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite

Watch the Flow

# Terminal 1: Watch secrets
kubectl get secrets -n <namespace> -w

# Terminal 2: Watch pods
kubectl get pods -n <namespace> -w

# Terminal 3: Watch jobs (for CronJobs)
kubectl get jobs -n <namespace> -w

# Terminal 4: Reloader logs
kubectl logs -n reloader -l app=reloader -f

Verify Rotation Completed

# Check if -previous secret exists (should only exist during grace period)
kubectl get secret <name>-previous -n <namespace>

# View rotation job logs
kubectl logs -n <namespace> -l job-name=<rotation-job>

# Test connection with new credentials
kubectl exec -it -n <namespace> <pod> -- <test-command>

---

Common Issues and Solutions

Issue: Pod not restarting after rotation

Solution: Check Reloader is installed and annotation is correct

kubectl get pods -n reloader
kubectl get deployment <app> -o yaml | grep reloader

Issue: Rotation job failing

Solution: Check if previous secret exists and credentials work

kubectl get secret <name>-previous -n <namespace>
kubectl logs -n <namespace> -l job-name=<rotation-job>

Issue: Application still using old credentials

Solution: Verify pod actually restarted after secret change

kubectl get pods -n <namespace> -o wide
# Check pod start time vs secret update time

Issue: Previous secret not cleaned up

Solution: Check operator logs for cleanup job

kubectl logs -n managedsecret-operator-system -l control-plane=controller-manager | grep cleanup

---

Best Practices Summary

1. ✅ Always use Reloader with rotating secrets
2. ✅ Use keepPreviousVersion for services requiring old password
3. ✅ Set realistic previousVersionTTL based on your workflow
4. ✅ Test rotation before production deployment
5. ✅ Monitor rotation jobs for failures
6. ✅ Document rotation procedures for your team
7. ✅ Use appropriate rotation schedules based on security requirements
8. ✅ Backup Vault regularly - it's your source of truth
9. ✅ Never modify K8s Secrets manually - always use ManagedSecret or Vault
10. ✅ Set up alerts for failed rotations

---

Support and Contributing

For issues or questions about these examples:

• Check the main README.md for troubleshooting
• Review operator logs for detailed error messages
• Test in development environment before production

Remember: Vault/OpenBao is ALWAYS the source of truth. All changes should go through ManagedSecret resources or directly in Vault, never by manually editing Kubernetes Secrets.