Initial public release v1.0.7

- Streamlined README focused on quick start
- Complete examples for all major use cases
- Decision tree for choosing right pattern
- Comprehensive troubleshooting guide
This commit is contained in:
2025-10-26 14:53:01 +01:00
commit 1bc8aadb85
9 changed files with 3485 additions and 0 deletions

View File

@@ -0,0 +1,316 @@
# ManagedSecret Operator - Examples Summary
This directory contains comprehensive examples for the ManagedSecret Operator, demonstrating various rotation strategies and use cases.
## Files Overview
### Documentation
- **README.md** - Complete installation and usage guide with new features
- Added `keepPreviousVersion` and `previousVersionTTL` documentation
- Added Reloader integration guide
- Updated best practices and architecture diagrams
### Example Files
#### Example 1: No Rotation (Simple API Key)
**File:** `example-1-no-rotation.yaml`
**Use Case:** Long-lived API keys that don't need rotation
**Key Features:**
- Generated once and kept indefinitely
- No Reloader needed (secret never changes)
- Perfect for external service API keys
**When to Use:**
- Third-party API keys (Stripe, SendGrid, etc.)
- Service identifiers
- Static configuration values
---
#### Example 2: Simple Rotation with Reloader (Container Registry)
**File:** `example-2-registry-rotation-simple.yaml`
**Use Case:** Docker/Container registry with HTTP basic auth
**Key Features:**
- Password rotates every 90 days
- InitContainer regenerates htpasswd on pod restart
- Reloader automatically restarts pods
- **No `keepPreviousVersion` needed** - htpasswd is regenerated fresh
**Rotation Flow:**
1. ManagedSecret generates new password → Updates Vault → Updates K8s Secret
2. Reloader detects Secret change → Triggers pod restart
3. InitContainer runs → Reads new password → Generates fresh htpasswd
4. Registry starts with new htpasswd
**When to Use:**
- Container registries (Docker Registry, Harbor)
- Web servers with htpasswd (Apache, Nginx)
- Any service where credentials are regenerated at startup
---
#### Example 3: Complex Rotation with Previous Password (PostgreSQL)
**File:** `example-3-postgres-rotation-previous.yaml`
**Use Case:** PostgreSQL user password that requires authentication to change
**Key Features:**
- Password rotates every 90 days
- `keepPreviousVersion: true` creates `-previous` secret
- CronJob uses old password to authenticate, sets new password
- `previousVersionTTL: 1h` cleans up after grace period
**Rotation Flow:**
1. ManagedSecret creates both `postgres-secret` and `postgres-secret-previous`
2. CronJob (every 5 min) detects rotation
3. Authenticates with OLD password: `PGPASSWORD=$OLD_PASSWORD psql`
4. Changes password: `ALTER USER app_user PASSWORD '$NEW_PASSWORD'`
5. Verifies new password works
6. After 1 hour, `-previous` secret is deleted
**When to Use:**
- PostgreSQL, MySQL, MariaDB
- Any database requiring authentication to change passwords
- Services with `ALTER USER` or similar commands
---
#### Example 4: MinIO Admin Password Rotation
**File:** `example-4-minio-rotation-previous.yaml`
**Use Case:** MinIO admin credentials with rotation
**Key Features:**
- Password rotates every 60 days
- Uses MinIO Client (mc) to change password
- `keepPreviousVersion: true` with 2h TTL
- CronJob handles rotation via `mc admin user password`
**Special Considerations:**
- MinIO does NOT reload credentials from environment variables
- Manual pod restart required after rotation
- Consider using a sidecar to monitor and restart
**Rotation Flow:**
1. ManagedSecret creates both secrets
2. CronJob authenticates with old credentials: `mc alias set oldminio`
3. Changes password: `mc admin user password myminio admin $NEW_PASSWORD`
4. Verifies new credentials
5. **Important:** MinIO pods must be restarted manually
**When to Use:**
- MinIO object storage
- Any service that doesn't auto-reload credentials
- Services requiring separate admin tools for password changes
---
#### Example 5: MySQL User Password Rotation
**File:** `example-5-mysql-rotation-previous.yaml`
**Use Case:** MySQL application user with automated rotation
**Key Features:**
- Password rotates every 90 days
- Multiple password change methods (ALTER USER, SET PASSWORD)
- Root credentials as fallback
- `keepPreviousVersion: true` with 1h TTL
**Rotation Flow:**
1. ManagedSecret creates both secrets
2. CronJob tries multiple approaches:
- Method 1: `ALTER USER` as root (preferred)
- Method 2: `SET PASSWORD` as root
- Method 3: `SET PASSWORD` as user with old password
3. Flushes privileges
4. Verifies new password
**When to Use:**
- MySQL 5.7+, MySQL 8.0
- MariaDB
- Any MySQL-compatible database
---
#### Example 6: Comprehensive Multi-Service Deployment
**File:** `example-6-comprehensive-multi-service.yaml`
**Use Case:** Complete application stack showing all strategies together
**Components:**
1. **PostgreSQL** - Rotates with previous version (90 days)
2. **Redis** - Rotates without previous version (30 days)
3. **API Keys** - No rotation (managed externally)
4. **TLS Certificates** - No rotation (cert-manager)
5. **JWT Secret** - Simple rotation with Reloader (180 days)
**Demonstrates:**
- Multiple secrets with different rotation schedules
- Mixed rotation strategies in one application
- How Reloader restarts pods for all credential changes
- Real-world production setup
**When to Use:**
- As a template for complex applications
- To understand how different strategies work together
- For learning all features in one example
---
## Quick Decision Tree
### Does your service need the OLD password to set the NEW password?
**YES** (PostgreSQL, MySQL, MinIO, LDAP):
- Use `keepPreviousVersion: true`
- Set appropriate `previousVersionTTL`
- Create CronJob to handle rotation
- See Examples 3, 4, or 5
**NO** (Registry, Redis, Web servers):
- Use simple rotation with Reloader
- InitContainer or startup script regenerates credentials
- No CronJob needed
- See Example 2
**NOT ROTATING** (API keys, Certificates):
- Set `rotation.enabled: false`
- No Reloader annotation needed
- See Example 1
---
## Rotation Schedule Recommendations
- **High Security** (User passwords, admin credentials): 30-90 days
- **Medium Security** (Application credentials, service accounts): 90-180 days
- **Low Security** (Read-only API keys, development): 180+ days or disabled
- **Previous Version TTL**:
- Automated rotation: 1h
- Manual intervention: 2-4h
- Complex workflows: 24h
---
## Prerequisites for All Examples
1. **OpenBao/Vault** deployed and initialized
2. **ManagedSecret Operator** installed
3. **Reloader** installed (for rotation examples)
4. **cert-manager** (optional, for TLS examples)
## Installation Order
```bash
# 1. Install OpenBao
kubectl create namespace openbao
helm install openbao openbao/openbao -n openbao
# 2. Install ManagedSecret Operator
kubectl apply -f managedsecret-operator/
# 3. Install Reloader
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
# 4. Deploy your application with examples
kubectl apply -f example-X-your-usecase.yaml
```
---
## Testing Rotation
### Force Immediate Rotation
```bash
# Trigger rotation for any ManagedSecret
kubectl annotate managedsecret <name> -n <namespace> \
reconcile="$(date +%s)" --overwrite
```
### Watch the Flow
```bash
# Terminal 1: Watch secrets
kubectl get secrets -n <namespace> -w
# Terminal 2: Watch pods
kubectl get pods -n <namespace> -w
# Terminal 3: Watch jobs (for CronJobs)
kubectl get jobs -n <namespace> -w
# Terminal 4: Reloader logs
kubectl logs -n reloader -l app=reloader -f
```
### Verify Rotation Completed
```bash
# Check if -previous secret exists (should only exist during grace period)
kubectl get secret <name>-previous -n <namespace>
# View rotation job logs
kubectl logs -n <namespace> -l job-name=<rotation-job>
# Test connection with new credentials
kubectl exec -it -n <namespace> <pod> -- <test-command>
```
---
## Common Issues and Solutions
### Issue: Pod not restarting after rotation
**Solution:** Check Reloader is installed and annotation is correct
```bash
kubectl get pods -n reloader
kubectl get deployment <app> -o yaml | grep reloader
```
### Issue: Rotation job failing
**Solution:** Check if previous secret exists and credentials work
```bash
kubectl get secret <name>-previous -n <namespace>
kubectl logs -n <namespace> -l job-name=<rotation-job>
```
### Issue: Application still using old credentials
**Solution:** Verify pod actually restarted after secret change
```bash
kubectl get pods -n <namespace> -o wide
# Check pod start time vs secret update time
```
### Issue: Previous secret not cleaned up
**Solution:** Check operator logs for cleanup job
```bash
kubectl logs -n managedsecret-operator-system -l control-plane=controller-manager | grep cleanup
```
---
## Best Practices Summary
1.**Always use Reloader** with rotating secrets
2.**Use `keepPreviousVersion`** for services requiring old password
3.**Set realistic `previousVersionTTL`** based on your workflow
4.**Test rotation** before production deployment
5.**Monitor rotation jobs** for failures
6.**Document rotation procedures** for your team
7.**Use appropriate rotation schedules** based on security requirements
8.**Backup Vault regularly** - it's your source of truth
9.**Never modify K8s Secrets manually** - always use ManagedSecret or Vault
10.**Set up alerts** for failed rotations
---
## Support and Contributing
For issues or questions about these examples:
- Check the main README.md for troubleshooting
- Review operator logs for detailed error messages
- Test in development environment before production
Remember: Vault/OpenBao is ALWAYS the source of truth. All changes should go through ManagedSecret resources or directly in Vault, never by manually editing Kubernetes Secrets.

View File

@@ -0,0 +1,122 @@
# Example 1: Simple API Key (No Rotation)
#
# Use case: Long-lived API key that doesn't need rotation
#
# Characteristics:
# - Generated once
# - No rotation schedule
# - No Reloader needed (static secret)
---
apiVersion: v1
kind: Namespace
metadata:
name: api-service
---
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: api-keys
namespace: api-service
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: api/service/keys
fields:
# Static service identifier
- name: service-id
type: static
value: "api-service-prod"
# Generated API key (long, alphanumeric)
- name: api-key
type: generated
generator:
type: password
length: 64
minDigits: 10
minSymbols: 0
minLowercase: 20
minUppercase: 20
symbolCharacters: "" # No symbols, just alphanumeric
allowRepeat: false
# Static API endpoint
- name: api-endpoint
type: static
value: "https://api.external-service.com/v1"
destination:
name: api-secret
type: Opaque
# No rotation - generate once and keep
rotation:
enabled: false
---
# Example Deployment using the API key
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-consumer
namespace: api-service
spec:
replicas: 2
selector:
matchLabels:
app: api-consumer
template:
metadata:
labels:
app: api-consumer
# No Reloader annotation needed - secret doesn't change
spec:
containers:
- name: app
image: your-app:latest
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: api-secret
key: api-key
- name: API_ENDPOINT
valueFrom:
secretKeyRef:
name: api-secret
key: api-endpoint
- name: SERVICE_ID
valueFrom:
secretKeyRef:
name: api-secret
key: service-id
---
# Example: Using the secret in a ConfigMap template
apiVersion: v1
kind: ConfigMap
metadata:
name: api-config
namespace: api-service
data:
config.yaml: |
service:
name: api-consumer
# API credentials loaded from secret via environment variables
# See deployment above for how to inject API_KEY
endpoints:
external:
timeout: 30s
retry: 3
---
# How to retrieve the API key for external use
# kubectl get secret api-secret -n api-service -o jsonpath='{.data.api-key}' | base64 -d

View File

@@ -0,0 +1,287 @@
# Example 2: Container Registry with Rotation + Reloader
#
# Use case: Docker registry with basic auth that rotates every 90 days
#
# Characteristics:
# - Password rotates every 90 days
# - Registry doesn't need old password to accept new one
# - InitContainer regenerates htpasswd on pod restart
# - Reloader automatically restarts pod when secret changes
# - No keepPreviousVersion needed
---
apiVersion: v1
kind: Namespace
metadata:
name: registry
---
# ManagedSecret generates and rotates registry credentials
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: registry-auth
namespace: registry
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: registry/auth
fields:
- name: username
type: static
value: "admin"
- name: password
type: generated
generator:
type: password
length: 32
minDigits: 5
minSymbols: 5
minLowercase: 5
minUppercase: 5
symbolCharacters: "!@#$%^&*"
allowRepeat: false
destination:
name: registry-credentials
type: Opaque
# No need for previous version - htpasswd is regenerated fresh
rotation:
enabled: true
schedule: 2160h # 90 days
rotateGeneratedOnly: true # Only rotate password, keep username
---
# PersistentVolumeClaim for registry storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
namespace: registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn
---
# Registry Deployment with Reloader integration
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: registry
labels:
app: registry
spec:
replicas: 1
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
annotations:
# Reloader watches registry-credentials and restarts pod on change
reloader.stakater.com/auto: "true"
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
fsGroup: 65534
seccompProfile:
type: RuntimeDefault
initContainers:
# Generate htpasswd file from current credentials at pod startup
- name: generate-htpasswd
image: httpd:2.4-alpine
command:
- sh
- -c
- |
USERNAME=$(cat /credentials/username)
PASSWORD=$(cat /credentials/password)
htpasswd -Bbn "$USERNAME" "$PASSWORD" > /auth/htpasswd
chmod 644 /auth/htpasswd
echo "htpasswd generated with user: $USERNAME"
volumeMounts:
- name: credentials
mountPath: /credentials
readOnly: true
- name: auth
mountPath: /auth
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 65534
capabilities:
drop:
- ALL
containers:
- name: registry
image: registry:2
ports:
- containerPort: 5000
name: http
env:
- name: REGISTRY_STORAGE_DELETE_ENABLED
value: "true"
- name: REGISTRY_HTTP_ADDR
value: "0.0.0.0:5000"
# Enable HTTP basic auth
- name: REGISTRY_AUTH
value: "htpasswd"
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: "Registry Realm"
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: "/auth/htpasswd"
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 65534
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: registry-storage
mountPath: /var/lib/registry
- name: auth
mountPath: /auth
readOnly: true
livenessProbe:
httpGet:
path: /
port: 5000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 5000
initialDelaySeconds: 5
periodSeconds: 5
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
volumes:
- name: registry-storage
persistentVolumeClaim:
claimName: registry-pvc
- name: credentials
secret:
secretName: registry-credentials
- name: auth
emptyDir: {}
---
# Service for internal access
apiVersion: v1
kind: Service
metadata:
name: registry
namespace: registry
spec:
type: ClusterIP
selector:
app: registry
ports:
- port: 5000
targetPort: 5000
protocol: TCP
name: http
---
# Ingress for external access with TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: registry
namespace: registry
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
ingressClassName: nginx
tls:
- hosts:
- registry.c5ai.ch
secretName: registry-ingress-tls
rules:
- host: registry.c5ai.ch
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: registry
port:
number: 5000
---
# How the rotation flow works:
#
# 1. After 90 days, ManagedSecret operator:
# - Generates new password
# - Updates Vault
# - Updates registry-credentials Secret
#
# 2. Reloader detects Secret change:
# - Sees resourceVersion changed
# - Triggers rolling restart of registry Deployment
#
# 3. Pod restart with new credentials:
# - InitContainer runs
# - Reads NEW password from registry-credentials
# - Generates fresh htpasswd file
# - Registry container starts with new htpasswd
#
# 4. Result:
# - Old clients with old password: FAIL (401 Unauthorized)
# - New clients with new password: SUCCESS
# - Downtime: ~1-2 minutes during rolling restart
---
# Testing commands:
#
# # Get current credentials
# kubectl get secret registry-credentials -n registry -o jsonpath='{.data.username}' | base64 -d
# kubectl get secret registry-credentials -n registry -o jsonpath='{.data.password}' | base64 -d
#
# # Test without auth (should fail)
# curl -i https://registry.c5ai.ch/v2/
#
# # Test with auth (should succeed)
# REGISTRY_USER=$(kubectl get secret registry-credentials -n registry -o jsonpath='{.data.username}' | base64 -d)
# REGISTRY_PASS=$(kubectl get secret registry-credentials -n registry -o jsonpath='{.data.password}' | base64 -d)
# curl -u $REGISTRY_USER:$REGISTRY_PASS https://registry.c5ai.ch/v2/_catalog
#
# # Docker login
# docker login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS
#
# # Force rotation test
# kubectl annotate managedsecret registry-auth -n registry reconcile="$(date +%s)" --overwrite
# kubectl get pods -n registry -w # Watch pod restart

View File

@@ -0,0 +1,369 @@
# Example 3: PostgreSQL Password Rotation with Previous Version
#
# Use case: PostgreSQL user password that rotates every 90 days
#
# Characteristics:
# - Password rotates every 90 days
# - PostgreSQL REQUIRES old password to authenticate before changing
# - keepPreviousVersion creates postgres-secret-previous
# - CronJob uses old password to authenticate, sets new password
# - After 1 hour grace period, old secret is cleaned up
---
apiVersion: v1
kind: Namespace
metadata:
name: database
---
# ManagedSecret with previous version support
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: postgres-app-user
namespace: database
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: postgres/app-user
fields:
- name: username
type: static
value: "app_user"
- name: password
type: generated
generator:
type: password
length: 40
minDigits: 8
minSymbols: 8
minLowercase: 8
minUppercase: 8
symbolCharacters: "!@#$%^&*()"
allowRepeat: false
- name: host
type: static
value: "postgres.database.svc.cluster.local"
- name: port
type: static
value: "5432"
- name: database
type: static
value: "app_db"
destination:
name: postgres-secret
type: Opaque
# Keep previous version for authentication during rotation
keepPreviousVersion: true
previousVersionTTL: 1h # Clean up after 1 hour
rotation:
enabled: true
schedule: 2160h # 90 days
rotateGeneratedOnly: true
---
# ServiceAccount for password rotation job
apiVersion: v1
kind: ServiceAccount
metadata:
name: postgres-rotator
namespace: database
---
# Role to allow reading secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: postgres-rotator
namespace: database
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
resourceNames: ["postgres-secret", "postgres-secret-previous"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: postgres-rotator
namespace: database
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: postgres-rotator
subjects:
- kind: ServiceAccount
name: postgres-rotator
namespace: database
---
# CronJob to detect rotation and update PostgreSQL password
apiVersion: batch/v1
kind: CronJob
metadata:
name: postgres-password-rotator
namespace: database
spec:
# Run every 5 minutes to check for rotation
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 3
template:
metadata:
annotations:
# Reloader can also trigger this job when secret changes
reloader.stakater.com/auto: "true"
spec:
serviceAccountName: postgres-rotator
restartPolicy: OnFailure
containers:
- name: rotate-password
image: postgres:15-alpine
env:
# Current credentials
- name: PGUSER
valueFrom:
secretKeyRef:
name: postgres-secret
key: username
- name: NEW_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
- name: PGHOST
valueFrom:
secretKeyRef:
name: postgres-secret
key: host
- name: PGPORT
valueFrom:
secretKeyRef:
name: postgres-secret
key: port
- name: PGDATABASE
valueFrom:
secretKeyRef:
name: postgres-secret
key: database
# Previous credentials (for authentication)
- name: OLD_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret-previous
key: password
optional: true # Might not exist if no rotation yet
command:
- sh
- -c
- |
set -e
echo "=========================================="
echo "PostgreSQL Password Rotation Check"
echo "=========================================="
echo "User: $PGUSER"
echo "Host: $PGHOST:$PGPORT"
echo "Database: $PGDATABASE"
echo ""
# Check if previous password exists (rotation happened)
if [ -z "$OLD_PASSWORD" ]; then
echo "No previous password found - no rotation needed"
exit 0
fi
# Check if passwords are different
if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then
echo "Passwords are identical - no rotation needed"
exit 0
fi
echo "Rotation detected! Old and new passwords differ."
echo ""
# Test if current password in PostgreSQL is the old one
echo "Testing authentication with OLD password..."
if PGPASSWORD=$OLD_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then
echo "✓ Old password still active in PostgreSQL"
echo ""
echo "Updating to NEW password..."
# Authenticate with OLD password, set NEW password
if PGPASSWORD=$OLD_PASSWORD psql -c "ALTER USER $PGUSER PASSWORD '$NEW_PASSWORD';"; then
echo "✓ Password updated successfully!"
echo ""
# Verify new password works
echo "Verifying NEW password..."
if PGPASSWORD=$NEW_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then
echo "✓ New password verified and working!"
echo ""
echo "=========================================="
echo "Rotation completed successfully"
echo "=========================================="
exit 0
else
echo "✗ ERROR: New password doesn't work!"
exit 1
fi
else
echo "✗ ERROR: Failed to update password"
exit 1
fi
else
echo "Old password doesn't work - checking if new password already active..."
if PGPASSWORD=$NEW_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then
echo "✓ New password is already active in PostgreSQL"
echo "Rotation was already completed"
exit 0
else
echo "✗ ERROR: Neither old nor new password works!"
echo "Manual intervention required"
exit 1
fi
fi
---
# Example application Deployment using the credentials
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
namespace: database
spec:
replicas: 2
selector:
matchLabels:
app: postgres-client
template:
metadata:
labels:
app: postgres-client
annotations:
# Reloader restarts app when credentials change
reloader.stakater.com/auto: "true"
spec:
containers:
- name: app
image: postgres:15-alpine
env:
- name: PGUSER
valueFrom:
secretKeyRef:
name: postgres-secret
key: username
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
- name: PGHOST
valueFrom:
secretKeyRef:
name: postgres-secret
key: host
- name: PGPORT
valueFrom:
secretKeyRef:
name: postgres-secret
key: port
- name: PGDATABASE
valueFrom:
secretKeyRef:
name: postgres-secret
key: database
command:
- sh
- -c
- |
# Your application code here
while true; do
psql -c "SELECT NOW();" || echo "Connection failed"
sleep 30
done
---
# How the rotation flow works:
#
# Day 0: Initial setup
# - ManagedSecret creates postgres-secret with generated password
# - No postgres-secret-previous exists yet
# - CronJob runs every 5 minutes, finds no previous password, does nothing
#
# Day 90: Rotation triggered
# 1. ManagedSecret operator:
# - Generates NEW password
# - Updates Vault with new password
# - Creates postgres-secret-previous with OLD password
# - Updates postgres-secret with NEW password
#
# 2. Within 5 minutes, CronJob runs:
# - Detects postgres-secret-previous exists
# - Compares OLD_PASSWORD vs NEW_PASSWORD (different!)
# - Authenticates to PostgreSQL with OLD_PASSWORD
# - Runs: ALTER USER app_user PASSWORD 'new_password'
# - Verifies new password works
# - Job succeeds
#
# 3. After 1 hour (previousVersionTTL):
# - ManagedSecret operator deletes postgres-secret-previous
# - CronJob future runs find no previous password, do nothing
#
# 4. Application pods:
# - Reloader detects postgres-secret change
# - Triggers rolling restart
# - Pods reconnect with NEW password
# - ~1 minute downtime per pod during restart
---
# Manual rotation testing:
#
# # Check current state
# kubectl get secret postgres-secret -n database -o yaml
# kubectl get secret postgres-secret-previous -n database -o yaml # Might not exist
#
# # Force rotation
# kubectl annotate managedsecret postgres-app-user -n database reconcile="$(date +%s)" --overwrite
#
# # Watch the rotation
# kubectl get secrets -n database -w
#
# # Check CronJob executes
# kubectl get jobs -n database -w
#
# # View rotation logs
# kubectl logs -n database -l job-name=postgres-password-rotator-XXXXX
#
# # Verify password in PostgreSQL matches new secret
# NEW_PASS=$(kubectl get secret postgres-secret -n database -o jsonpath='{.data.password}' | base64 -d)
# kubectl exec -it -n database deployment/app -- psql -c "SELECT 1;"
---
# Troubleshooting:
#
# If rotation fails:
# 1. Check CronJob logs for errors
# 2. Verify PostgreSQL is accessible from CronJob pod
# 3. Check if old password still works in PostgreSQL
# 4. Manually update if needed:
# kubectl exec -it postgres-0 -n database -- psql -U postgres
# ALTER USER app_user PASSWORD 'password_from_secret';

View File

@@ -0,0 +1,430 @@
# Example 4: MinIO Admin Password Rotation with Previous Version
#
# Use case: MinIO admin credentials that rotate every 60 days
#
# Characteristics:
# - Password rotates every 60 days
# - MinIO Client (mc) needs old credentials to change password
# - keepPreviousVersion creates minio-admin-secret-previous
# - CronJob uses old credentials to authenticate, sets new password
# - After 2 hour grace period, old secret is cleaned up
---
apiVersion: v1
kind: Namespace
metadata:
name: minio
---
# ManagedSecret for MinIO admin credentials
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: minio-admin
namespace: minio
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: minio/admin
fields:
- name: username
type: static
value: "admin"
- name: password
type: generated
generator:
type: password
length: 32
minDigits: 5
minSymbols: 5
minLowercase: 5
minUppercase: 5
symbolCharacters: "!@#$%^&*"
allowRepeat: false
- name: endpoint
type: static
value: "http://minio.minio.svc.cluster.local:9000"
destination:
name: minio-admin-secret
type: Opaque
# Keep previous version for authentication during rotation
keepPreviousVersion: true
previousVersionTTL: 2h # Longer grace period for MinIO
rotation:
enabled: true
schedule: 1440h # 60 days
rotateGeneratedOnly: true
---
# ServiceAccount for password rotation job
apiVersion: v1
kind: ServiceAccount
metadata:
name: minio-rotator
namespace: minio
---
# Role to allow reading secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: minio-rotator
namespace: minio
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
resourceNames: ["minio-admin-secret", "minio-admin-secret-previous"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: minio-rotator
namespace: minio
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: minio-rotator
subjects:
- kind: ServiceAccount
name: minio-rotator
namespace: minio
---
# CronJob to detect rotation and update MinIO password
apiVersion: batch/v1
kind: CronJob
metadata:
name: minio-password-rotator
namespace: minio
spec:
# Run every 10 minutes to check for rotation
schedule: "*/10 * * * *"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 3
template:
metadata:
annotations:
reloader.stakater.com/auto: "true"
spec:
serviceAccountName: minio-rotator
restartPolicy: OnFailure
containers:
- name: rotate-password
image: minio/mc:latest
env:
# Current credentials
- name: MC_HOST_myminio
value: "" # Will be set in script
- name: USERNAME
valueFrom:
secretKeyRef:
name: minio-admin-secret
key: username
- name: NEW_PASSWORD
valueFrom:
secretKeyRef:
name: minio-admin-secret
key: password
- name: ENDPOINT
valueFrom:
secretKeyRef:
name: minio-admin-secret
key: endpoint
# Previous credentials (for authentication)
- name: OLD_USERNAME
valueFrom:
secretKeyRef:
name: minio-admin-secret-previous
key: username
optional: true
- name: OLD_PASSWORD
valueFrom:
secretKeyRef:
name: minio-admin-secret-previous
key: password
optional: true
command:
- sh
- -c
- |
set -e
echo "=========================================="
echo "MinIO Password Rotation Check"
echo "=========================================="
echo "User: $USERNAME"
echo "Endpoint: $ENDPOINT"
echo ""
# Check if previous password exists (rotation happened)
if [ -z "$OLD_PASSWORD" ]; then
echo "No previous password found - no rotation needed"
exit 0
fi
# Check if passwords are different
if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then
echo "Passwords are identical - no rotation needed"
exit 0
fi
echo "Rotation detected! Old and new passwords differ."
echo ""
# Test if current password in MinIO is the old one
echo "Testing authentication with OLD credentials..."
if mc alias set oldminio $ENDPOINT $OLD_USERNAME $OLD_PASSWORD > /dev/null 2>&1; then
echo "✓ Old credentials still active in MinIO"
echo ""
# Test admin access
if mc admin info oldminio > /dev/null 2>&1; then
echo "✓ Admin access confirmed with old credentials"
echo ""
echo "Updating to NEW password..."
# Change password using old credentials
if mc admin user password oldminio $USERNAME $NEW_PASSWORD; then
echo "✓ Password updated successfully!"
echo ""
# Verify new password works
echo "Verifying NEW password..."
if mc alias set newminio $ENDPOINT $USERNAME $NEW_PASSWORD > /dev/null 2>&1; then
if mc admin info newminio > /dev/null 2>&1; then
echo "✓ New password verified and working!"
echo ""
echo "=========================================="
echo "Rotation completed successfully"
echo "=========================================="
exit 0
else
echo "✗ ERROR: New credentials don't have admin access!"
exit 1
fi
else
echo "✗ ERROR: New password doesn't work!"
exit 1
fi
else
echo "✗ ERROR: Failed to update password"
exit 1
fi
else
echo "✗ ERROR: No admin access with old credentials"
exit 1
fi
else
echo "Old credentials don't work - checking if new credentials already active..."
if mc alias set newminio $ENDPOINT $USERNAME $NEW_PASSWORD > /dev/null 2>&1; then
if mc admin info newminio > /dev/null 2>&1; then
echo "✓ New credentials are already active in MinIO"
echo "Rotation was already completed"
exit 0
else
echo "✗ ERROR: New credentials exist but no admin access"
exit 1
fi
else
echo "✗ ERROR: Neither old nor new credentials work!"
echo "Manual intervention required"
exit 1
fi
fi
---
# Example: MinIO StatefulSet (simplified)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: minio
namespace: minio
spec:
serviceName: minio
replicas: 1
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
annotations:
# Note: MinIO doesn't auto-reload credentials from env vars
# You may need to restart pods manually or use a sidecar
reloader.stakater.com/auto: "false"
spec:
containers:
- name: minio
image: minio/minio:latest
args:
- server
- /data
- --console-address
- ":9001"
env:
- name: MINIO_ROOT_USER
valueFrom:
secretKeyRef:
name: minio-admin-secret
key: username
- name: MINIO_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: minio-admin-secret
key: password
ports:
- containerPort: 9000
name: api
- containerPort: 9001
name: console
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
---
# Service for MinIO
apiVersion: v1
kind: Service
metadata:
name: minio
namespace: minio
spec:
selector:
app: minio
ports:
- port: 9000
targetPort: 9000
name: api
- port: 9001
targetPort: 9001
name: console
---
# How the rotation flow works:
#
# Day 0: Initial setup
# - ManagedSecret creates minio-admin-secret with generated password
# - MinIO starts with these credentials
# - No minio-admin-secret-previous exists yet
# - CronJob runs every 10 minutes, finds no previous password, does nothing
#
# Day 60: Rotation triggered
# 1. ManagedSecret operator:
# - Generates NEW password
# - Updates Vault with new password
# - Creates minio-admin-secret-previous with OLD password
# - Updates minio-admin-secret with NEW password
#
# 2. Within 10 minutes, CronJob runs:
# - Detects minio-admin-secret-previous exists
# - Compares OLD_PASSWORD vs NEW_PASSWORD (different!)
# - Authenticates to MinIO with OLD credentials
# - Runs: mc admin user password myminio admin new_password
# - Verifies new password works
# - Job succeeds
#
# 3. After 2 hours (previousVersionTTL):
# - ManagedSecret operator deletes minio-admin-secret-previous
# - CronJob future runs find no previous password, do nothing
#
# 4. MinIO pods:
# - IMPORTANT: MinIO does NOT reload credentials from env vars
# - You need to manually restart MinIO pods after rotation
# - Or use a sidecar to monitor and trigger restart
# - Consider NOT using reloader.stakater.com/auto for MinIO
---
# Alternative: Sidecar to handle MinIO restart
# Add this to the MinIO StatefulSet if you want automatic restarts
---
# sidecar:
# - name: credential-watcher
# image: bitnami/kubectl:latest
# command:
# - sh
# - -c
# - |
# LAST_VERSION=""
# while true; do
# CURRENT_VERSION=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.metadata.resourceVersion}')
# if [ -n "$LAST_VERSION" ] && [ "$LAST_VERSION" != "$CURRENT_VERSION" ]; then
# echo "Credentials changed! Signaling MinIO to restart..."
# # Send SIGTERM to MinIO process to trigger graceful shutdown
# killall minio
# fi
# LAST_VERSION=$CURRENT_VERSION
# sleep 30
# done
---
# Manual rotation testing:
#
# # Check current state
# kubectl get secret minio-admin-secret -n minio -o yaml
# kubectl get secret minio-admin-secret-previous -n minio -o yaml
#
# # Force rotation
# kubectl annotate managedsecret minio-admin -n minio reconcile="$(date +%s)" --overwrite
#
# # Watch the rotation
# kubectl get secrets -n minio -w
#
# # Check CronJob executes
# kubectl get jobs -n minio -w
#
# # View rotation logs
# kubectl logs -n minio -l job-name=minio-password-rotator-XXXXX
#
# # Test MinIO access with new credentials
# NEW_USER=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.data.username}' | base64 -d)
# NEW_PASS=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.data.password}' | base64 -d)
# mc alias set testminio http://minio.minio.svc.cluster.local:9000 $NEW_USER $NEW_PASS
# mc admin info testminio
---
# Important Notes:
#
# 1. MinIO credential reload:
# - Unlike PostgreSQL, MinIO does NOT automatically reload credentials
# - After rotation, you must restart MinIO pods
# - Consider using a sidecar or manual restart workflow
#
# 2. High Availability:
# - For HA MinIO clusters, all nodes share the same root credentials
# - Rotation updates credentials cluster-wide
# - Rolling restart required for all nodes
#
# 3. Grace Period:
# - 2 hour previousVersionTTL allows time for manual intervention
# - Adjust based on your operational procedures
#
# 4. Monitoring:
# - Monitor CronJob success/failure
# - Alert on rotation failures
# - Track when MinIO pods were last restarted

View File

@@ -0,0 +1,566 @@
# Example 5: MySQL User Password Rotation with Previous Version
#
# Use case: MySQL application user password that rotates every 90 days
#
# Characteristics:
# - Password rotates every 90 days
# - MySQL requires old password to authenticate before changing (when using SET PASSWORD)
# - keepPreviousVersion creates mysql-app-user-previous
# - CronJob uses old password to authenticate, sets new password
# - Application pods restart automatically via Reloader
---
apiVersion: v1
kind: Namespace
metadata:
name: mysql
---
# ManagedSecret for MySQL application user
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: mysql-app-user
namespace: mysql
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: mysql/app-user
fields:
- name: username
type: static
value: "app_user"
- name: password
type: generated
generator:
type: password
length: 32
minDigits: 6
minSymbols: 6
minLowercase: 8
minUppercase: 8
# MySQL doesn't allow some special chars in passwords
symbolCharacters: "!@#$%^&*"
allowRepeat: false
- name: host
type: static
value: "mysql.mysql.svc.cluster.local"
- name: port
type: static
value: "3306"
- name: database
type: static
value: "app_db"
- name: connection-string
type: static
value: "mysql://app_user@mysql.mysql.svc.cluster.local:3306/app_db"
destination:
name: mysql-app-secret
type: Opaque
# Keep previous version for authentication during rotation
keepPreviousVersion: true
previousVersionTTL: 1h
rotation:
enabled: true
schedule: 2160h # 90 days
rotateGeneratedOnly: true
---
# Separate ManagedSecret for MySQL root credentials (no rotation)
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: mysql-root
namespace: mysql
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: mysql/root
fields:
- name: username
type: static
value: "root"
- name: password
type: generated
generator:
type: password
length: 40
minDigits: 8
minSymbols: 8
destination:
name: mysql-root-secret
type: Opaque
# Root password doesn't rotate automatically
rotation:
enabled: false
---
# ServiceAccount for password rotation job
apiVersion: v1
kind: ServiceAccount
metadata:
name: mysql-rotator
namespace: mysql
---
# Role to allow reading secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: mysql-rotator
namespace: mysql
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
resourceNames:
- "mysql-app-secret"
- "mysql-app-secret-previous"
- "mysql-root-secret"
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: mysql-rotator
namespace: mysql
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: mysql-rotator
subjects:
- kind: ServiceAccount
name: mysql-rotator
namespace: mysql
---
# CronJob to detect rotation and update MySQL password
apiVersion: batch/v1
kind: CronJob
metadata:
name: mysql-password-rotator
namespace: mysql
spec:
# Run every 5 minutes to check for rotation
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 3
template:
metadata:
annotations:
reloader.stakater.com/auto: "true"
spec:
serviceAccountName: mysql-rotator
restartPolicy: OnFailure
containers:
- name: rotate-password
image: mysql:8.0
env:
# Current credentials
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: username
- name: NEW_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: password
- name: MYSQL_HOST
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: host
- name: MYSQL_PORT
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: port
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: database
# Previous credentials (for authentication)
- name: OLD_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-app-secret-previous
key: password
optional: true
# Root credentials for fallback
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-root-secret
key: password
command:
- bash
- -c
- |
set -e
echo "=========================================="
echo "MySQL Password Rotation Check"
echo "=========================================="
echo "User: $MYSQL_USER"
echo "Host: $MYSQL_HOST:$MYSQL_PORT"
echo "Database: $MYSQL_DATABASE"
echo ""
# Check if previous password exists (rotation happened)
if [ -z "$OLD_PASSWORD" ]; then
echo "No previous password found - no rotation needed"
exit 0
fi
# Check if passwords are different
if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then
echo "Passwords are identical - no rotation needed"
exit 0
fi
echo "Rotation detected! Old and new passwords differ."
echo ""
# Test if current password in MySQL is the old one
echo "Testing authentication with OLD password..."
if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$OLD_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then
echo "✓ Old password still active in MySQL"
echo ""
echo "Updating to NEW password..."
# Method 1: Use ALTER USER (MySQL 5.7.6+, doesn't require old password)
# Connect as root to change password
if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \
-e "ALTER USER '$MYSQL_USER'@'%' IDENTIFIED BY '$NEW_PASSWORD';" 2>/dev/null; then
echo "✓ Password updated using ALTER USER (as root)"
elif mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \
-e "SET PASSWORD FOR '$MYSQL_USER'@'%' = PASSWORD('$NEW_PASSWORD');" 2>/dev/null; then
echo "✓ Password updated using SET PASSWORD (as root)"
else
# Method 2: Use SET PASSWORD as the user (requires old password connection)
if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$OLD_PASSWORD" \
-e "SET PASSWORD = PASSWORD('$NEW_PASSWORD');" 2>/dev/null; then
echo "✓ Password updated using SET PASSWORD (as user)"
else
echo "✗ ERROR: Failed to update password with any method"
exit 1
fi
fi
echo ""
# Verify new password works
echo "Verifying NEW password..."
if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$NEW_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then
echo "✓ New password verified and working!"
echo ""
echo "Flushing privileges..."
mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" -e "FLUSH PRIVILEGES;" || echo "Warning: Could not flush privileges"
echo ""
echo "=========================================="
echo "Rotation completed successfully"
echo "=========================================="
exit 0
else
echo "✗ ERROR: New password doesn't work!"
exit 1
fi
else
echo "Old password doesn't work - checking if new password already active..."
if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$NEW_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then
echo "✓ New password is already active in MySQL"
echo "Rotation was already completed"
exit 0
else
echo "✗ ERROR: Neither old nor new password works!"
echo "Manual intervention required"
echo ""
echo "Attempting to reset using root credentials..."
if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \
-e "ALTER USER '$MYSQL_USER'@'%' IDENTIFIED BY '$NEW_PASSWORD';"; then
echo "✓ Password reset successfully using root"
exit 0
else
echo "✗ Failed to reset password"
exit 1
fi
fi
fi
---
# Example MySQL StatefulSet (simplified)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: mysql
spec:
serviceName: mysql
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:8.0
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-root-secret
key: password
- name: MYSQL_DATABASE
value: "app_db"
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
livenessProbe:
exec:
command:
- mysqladmin
- ping
- -h
- localhost
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command:
- mysql
- -h
- localhost
- -e
- SELECT 1
initialDelaySeconds: 5
periodSeconds: 2
# InitContainer to create app user on first run
initContainers:
- name: init-db-user
image: mysql:8.0
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-root-secret
key: password
- name: APP_USER
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: username
- name: APP_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: password
command:
- sh
- -c
- |
# Wait for MySQL to be ready
until mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SELECT 1" > /dev/null 2>&1; do
echo "Waiting for MySQL..."
sleep 2
done
# Create user if not exists
mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" <<EOF
CREATE USER IF NOT EXISTS '$APP_USER'@'%' IDENTIFIED BY '$APP_PASSWORD';
GRANT ALL PRIVILEGES ON app_db.* TO '$APP_USER'@'%';
FLUSH PRIVILEGES;
EOF
echo "User $APP_USER created/updated"
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
---
# Service for MySQL
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: mysql
spec:
selector:
app: mysql
ports:
- port: 3306
targetPort: 3306
name: mysql
clusterIP: None # Headless service
---
# Example application using MySQL
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
namespace: mysql
spec:
replicas: 2
selector:
matchLabels:
app: mysql-client
template:
metadata:
labels:
app: mysql-client
annotations:
# Reloader restarts app when credentials change
reloader.stakater.com/auto: "true"
spec:
containers:
- name: app
image: mysql:8.0
env:
- name: MYSQL_HOST
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: host
- name: MYSQL_PORT
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: port
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: password
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: mysql-app-secret
key: database
command:
- sh
- -c
- |
# Your application code here
while true; do
mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$MYSQL_PASSWORD" \
-D"$MYSQL_DATABASE" -e "SELECT NOW();" || echo "Connection failed"
sleep 30
done
---
# How the rotation flow works:
#
# Initial Setup:
# - mysql-app-secret created with initial password
# - InitContainer creates MySQL user with this password
# - Application connects successfully
#
# Day 90: Rotation
# 1. ManagedSecret operator:
# - Generates NEW password
# - Creates mysql-app-secret-previous (OLD password)
# - Updates mysql-app-secret (NEW password)
#
# 2. CronJob (within 5 minutes):
# - Detects previous secret exists
# - Authenticates to MySQL with OLD password (or as root)
# - Runs ALTER USER to set NEW password
# - Verifies NEW password works
#
# 3. Reloader:
# - Detects mysql-app-secret change
# - Restarts application pods
# - Pods reconnect with NEW password
#
# 4. Cleanup (after 1 hour):
# - mysql-app-secret-previous deleted automatically
---
# Testing commands:
#
# # Check current credentials
# kubectl get secret mysql-app-secret -n mysql -o yaml
#
# # Test connection with current credentials
# MYSQL_PASS=$(kubectl get secret mysql-app-secret -n mysql -o jsonpath='{.data.password}' | base64 -d)
# kubectl run -it --rm mysql-test --image=mysql:8.0 --restart=Never -- \
# mysql -h mysql.mysql.svc.cluster.local -u app_user -p"$MYSQL_PASS" -e "SELECT NOW();"
#
# # Force rotation
# kubectl annotate managedsecret mysql-app-user -n mysql reconcile="$(date +%s)" --overwrite
#
# # Watch rotation
# kubectl get secrets -n mysql -w
# kubectl get jobs -n mysql -w
#
# # View rotation logs
# kubectl logs -n mysql -l job-name=mysql-password-rotator-XXXXX
---
# Important MySQL-specific notes:
#
# 1. User Host Specification:
# - '%' allows connection from any host
# - Consider restricting to specific pods/namespaces
# - Example: 'app_user'@'%.mysql.svc.cluster.local'
#
# 2. Authentication Methods:
# - MySQL 8.0+ uses caching_sha2_password by default
# - Older apps may need mysql_native_password
# - Specify during user creation if needed
#
# 3. Privilege Management:
# - Rotation only changes password, not privileges
# - Manage privileges separately via root access
#
# 4. Connection Pooling:
# - Application connection pools need to be refreshed
# - Reloader restart handles this automatically
# - Consider graceful shutdown for long-running transactions

View File

@@ -0,0 +1,572 @@
# Example 6: Comprehensive Multi-Service Deployment
#
# This example demonstrates a complete application stack with:
# - PostgreSQL (rotating with previous version)
# - Redis (rotating without previous version)
# - External API key (no rotation)
# - TLS certificates (no rotation)
# - Application credentials (rotating with Reloader)
#
# Shows different rotation strategies working together
---
apiVersion: v1
kind: Namespace
metadata:
name: myapp
---
# ============================================================================
# 1. PostgreSQL Credentials - Rotates with previous version support
# ============================================================================
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: postgres-credentials
namespace: myapp
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: myapp/postgres
fields:
- name: username
type: static
value: "myapp_user"
- name: password
type: generated
generator:
type: password
length: 32
minDigits: 6
minSymbols: 6
- name: host
type: static
value: "postgres.database.svc.cluster.local"
- name: port
type: static
value: "5432"
- name: database
type: static
value: "myapp_db"
destination:
name: postgres-creds
type: Opaque
keepPreviousVersion: true # Needs old password for rotation
previousVersionTTL: 1h
rotation:
enabled: true
schedule: 2160h # 90 days
---
# ============================================================================
# 2. Redis Password - Rotates without previous version (CONFIG SET)
# ============================================================================
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: redis-credentials
namespace: myapp
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: myapp/redis
fields:
- name: password
type: generated
generator:
type: password
length: 40
minDigits: 8
minSymbols: 0 # Redis doesn't like special chars
symbolCharacters: ""
- name: host
type: static
value: "redis.myapp.svc.cluster.local"
- name: port
type: static
value: "6379"
destination:
name: redis-creds
type: Opaque
# No previous version needed - Redis CONFIG SET doesn't require auth
rotation:
enabled: true
schedule: 720h # 30 days
---
# ============================================================================
# 3. External API Keys - No rotation
# ============================================================================
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: external-api-keys
namespace: myapp
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: myapp/api-keys
fields:
- name: stripe-api-key
type: generated
generator:
type: password
length: 64
minDigits: 0
minSymbols: 0
- name: sendgrid-api-key
type: generated
generator:
type: password
length: 64
minDigits: 0
minSymbols: 0
destination:
name: api-keys
type: Opaque
rotation:
enabled: false # Managed externally
---
# ============================================================================
# 4. TLS Certificate - No rotation (managed by cert-manager)
# ============================================================================
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: app-tls-cert
namespace: myapp
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: myapp/tls
fields:
- name: tls.crt
type: static
value: |
-----BEGIN CERTIFICATE-----
# Your certificate here
-----END CERTIFICATE-----
- name: tls.key
type: static
value: |
-----BEGIN PRIVATE KEY-----
# Your private key here
-----END PRIVATE KEY-----
destination:
name: app-tls
type: kubernetes.io/tls
rotation:
enabled: false
---
# ============================================================================
# 5. Application JWT Secret - Rotates with Reloader
# ============================================================================
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
name: app-jwt-secret
namespace: myapp
spec:
vault:
address: "http://openbao.openbao.svc.cluster.local:8200"
authMethod: kubernetes
role: managedsecret-operator
kvVersion: v2
mount: secret
path: myapp/jwt
fields:
- name: jwt-secret
type: generated
generator:
type: password
length: 64
minDigits: 16
minSymbols: 0
symbolCharacters: ""
destination:
name: jwt-secret
type: Opaque
rotation:
enabled: true
schedule: 4320h # 180 days
---
# ============================================================================
# PostgreSQL Password Rotation Job
# ============================================================================
apiVersion: batch/v1
kind: CronJob
metadata:
name: postgres-password-rotator
namespace: myapp
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: rotate
image: postgres:15-alpine
env:
- name: PGUSER
valueFrom:
secretKeyRef:
name: postgres-creds
key: username
- name: NEW_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-creds
key: password
- name: OLD_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-creds-previous
key: password
optional: true
- name: PGHOST
valueFrom:
secretKeyRef:
name: postgres-creds
key: host
command:
- sh
- -c
- |
if [ -n "$OLD_PASSWORD" ] && [ "$OLD_PASSWORD" != "$NEW_PASSWORD" ]; then
echo "Rotating PostgreSQL password..."
PGPASSWORD=$OLD_PASSWORD psql -c "ALTER USER $PGUSER PASSWORD '$NEW_PASSWORD';"
echo "Rotation complete!"
fi
---
# ============================================================================
# Redis Password Rotation Job
# ============================================================================
apiVersion: batch/v1
kind: CronJob
metadata:
name: redis-password-rotator
namespace: myapp
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: rotate
image: redis:7-alpine
env:
- name: REDIS_HOST
valueFrom:
secretKeyRef:
name: redis-creds
key: host
- name: REDIS_PORT
valueFrom:
secretKeyRef:
name: redis-creds
key: port
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-creds
key: password
command:
- sh
- -c
- |
# Redis CONFIG SET doesn't require current password
redis-cli -h $REDIS_HOST -p $REDIS_PORT CONFIG SET requirepass "$REDIS_PASSWORD"
echo "Redis password updated"
---
# ============================================================================
# Main Application Deployment
# ============================================================================
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
namespace: myapp
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
annotations:
# Reloader restarts pods when ANY of these secrets change
reloader.stakater.com/auto: "true"
spec:
containers:
- name: app
image: myapp:latest
ports:
- containerPort: 8080
name: http
env:
# PostgreSQL connection
- name: POSTGRES_HOST
valueFrom:
secretKeyRef:
name: postgres-creds
key: host
- name: POSTGRES_PORT
valueFrom:
secretKeyRef:
name: postgres-creds
key: port
- name: POSTGRES_DB
valueFrom:
secretKeyRef:
name: postgres-creds
key: database
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-creds
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-creds
key: password
# Redis connection
- name: REDIS_HOST
valueFrom:
secretKeyRef:
name: redis-creds
key: host
- name: REDIS_PORT
valueFrom:
secretKeyRef:
name: redis-creds
key: port
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-creds
key: password
# API Keys
- name: STRIPE_API_KEY
valueFrom:
secretKeyRef:
name: api-keys
key: stripe-api-key
- name: SENDGRID_API_KEY
valueFrom:
secretKeyRef:
name: api-keys
key: sendgrid-api-key
# JWT Secret
- name: JWT_SECRET
valueFrom:
secretKeyRef:
name: jwt-secret
key: jwt-secret
volumeMounts:
- name: tls
mountPath: /etc/tls
readOnly: true
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
volumes:
- name: tls
secret:
secretName: app-tls
---
# ============================================================================
# Service
# ============================================================================
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: myapp
spec:
selector:
app: myapp
ports:
- port: 80
targetPort: 8080
name: http
---
# ============================================================================
# Ingress with TLS
# ============================================================================
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
namespace: myapp
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
spec:
ingressClassName: nginx
tls:
- hosts:
- myapp.c5ai.ch
secretName: myapp-tls-ingress
rules:
- host: myapp.c5ai.ch
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp
port:
number: 80
---
# ============================================================================
# Summary of Rotation Strategies
# ============================================================================
#
# 1. PostgreSQL (postgres-creds):
# - Rotates every 90 days
# - keepPreviousVersion: true
# - CronJob uses old password to authenticate and change
# - Reloader restarts app pods after rotation
#
# 2. Redis (redis-creds):
# - Rotates every 30 days
# - No previous version needed (CONFIG SET doesn't require auth)
# - CronJob updates password directly
# - Reloader restarts app pods after rotation
#
# 3. API Keys (api-keys):
# - No automatic rotation
# - Generated once, managed externally
# - Update via kubectl or Vault UI when provider rotates keys
#
# 4. TLS Certificates (app-tls):
# - No rotation
# - Managed by cert-manager or external process
# - Static content stored in Vault
#
# 5. JWT Secret (jwt-secret):
# - Rotates every 180 days
# - Simple rotation - just restart with new secret
# - Reloader handles pod restart
#
# ============================================================================
# Timeline Example: What Happens on Day 90
# ============================================================================
#
# T+0: ManagedSecret operator triggers rotation
# - Generates new passwords for postgres-creds and redis-creds
# - Updates Vault
# - Creates postgres-creds-previous
# - Updates postgres-creds
# - Updates redis-creds (no previous needed)
#
# T+1min: CronJobs detect changes
# - Postgres rotator: Uses old password to set new password in DB
# - Redis rotator: Sets new password via CONFIG SET
#
# T+2min: Reloader detects secret changes
# - Triggers rolling restart of myapp Deployment
# - Pods start with new credentials
#
# T+1hour: Cleanup
# - postgres-creds-previous automatically deleted
# - Old credentials no longer accessible
#
# ============================================================================
# Monitoring Commands
# ============================================================================
#
# # Watch all secrets
# kubectl get secrets -n myapp -w
#
# # Check rotation status
# kubectl get managedsecrets -n myapp
#
# # View rotation logs
# kubectl logs -n myapp -l job-name=postgres-password-rotator-XXXXX
# kubectl logs -n myapp -l job-name=redis-password-rotator-XXXXX
#
# # Check Reloader activity
# kubectl logs -n reloader -l app=reloader | grep myapp
#
# # Verify app pods restarted
# kubectl get pods -n myapp -o wide
#
# # Test connections with new credentials
# kubectl exec -it -n myapp deployment/myapp -- sh
# # psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB
# # redis-cli -h $REDIS_HOST -a $REDIS_PASSWORD ping
#
# ============================================================================
# Force Rotation for Testing
# ============================================================================
#
# # Force PostgreSQL rotation
# kubectl annotate managedsecret postgres-credentials -n myapp \
# reconcile="$(date +%s)" --overwrite
#
# # Force Redis rotation
# kubectl annotate managedsecret redis-credentials -n myapp \
# reconcile="$(date +%s)" --overwrite
#
# # Force all rotations
# kubectl annotate managedsecrets -n myapp --all \
# reconcile="$(date +%s)" --overwrite
#
# # Watch the cascade of events
# kubectl get events -n myapp --sort-by='.lastTimestamp' -w