Initial public release v1.0.7

- Streamlined README focused on quick start - Complete examples for all major use cases - Decision tree for choosing right pattern - Comprehensive troubleshooting guide
2025-10-26 14:53:01 +01:00
commit 1bc8aadb85
9 changed files with 3485 additions and 0 deletions
--- a/INDEX.md
+++ b/INDEX.md
@@ -0,0 +1,112 @@
 # ManagedSecret Operator - Documentation Index
 Quick navigation for the ManagedSecret Operator documentation and examples.
 ## 📖 Main Documentation
 **[README.md](./README.md)** - Start here!
 - Installation guide
 - Quick start
 - Basic usage examples
 - Troubleshooting
 - Best practices
 ## 📚 Examples
 **[examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md)** - Overview of all examples
 - Use case descriptions
 - Decision tree for choosing the right pattern
 - Quick comparison table
 ### Individual Examples
 1. **[example-1-no-rotation.yaml](./examples/example-1-no-rotation.yaml)**
   - Simple API key without rotation
   - Perfect for: Third-party API keys, static service identifiers
 2. **[example-2-registry-rotation-simple.yaml](./examples/example-2-registry-rotation-simple.yaml)**
   - Container registry with basic rotation
   - Perfect for: Docker Registry, Harbor, web servers with htpasswd
 3. **[example-3-postgres-rotation-previous.yaml](./examples/example-3-postgres-rotation-previous.yaml)**
   - PostgreSQL with zero-downtime rotation
   - Perfect for: PostgreSQL, any service needing old password to authenticate
 4. **[example-4-minio-rotation-previous.yaml](./examples/example-4-minio-rotation-previous.yaml)**
   - MinIO object storage with rotation
   - Perfect for: MinIO, services not auto-reloading credentials
 5. **[example-5-mysql-rotation-previous.yaml](./examples/example-5-mysql-rotation-previous.yaml)**
   - MySQL with multiple rotation strategies
   - Perfect for: MySQL, MariaDB, MySQL-compatible databases
 6. **[example-6-comprehensive-multi-service.yaml](./examples/example-6-comprehensive-multi-service.yaml)**
   - Complete application stack
   - Perfect for: Understanding how different strategies work together
 ## 🎯 Quick Links by Use Case
 ### I need to store a secret that never changes
 → [Example 1: No Rotation](./examples/example-1-no-rotation.yaml)
 ### I need to rotate credentials automatically
 → [Example 2: Simple Rotation](./examples/example-2-registry-rotation-simple.yaml)
 ### My service needs the old password to change passwords
 → [Example 3: PostgreSQL](./examples/example-3-postgres-rotation-previous.yaml)  
 → [Example 4: MinIO](./examples/example-4-minio-rotation-previous.yaml)  
 → [Example 5: MySQL](./examples/example-5-mysql-rotation-previous.yaml)
 ### I have a complex multi-service application
 → [Example 6: Multi-Service](./examples/example-6-comprehensive-multi-service.yaml)
 ## 🚀 Getting Started Path
 1. Read [README.md](./README.md) - Installation & Quick Start
 2. Review [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md) - Understand patterns
 3. Pick the example closest to your use case
 4. Adapt and deploy!
 ## 🔧 Common Tasks
 ### Install the operator
 → [README.md - Installation Steps](./README.md#installation-steps)
 ### Create your first secret
 → [README.md - Usage](./README.md#usage)
 ### Understand rotation
 → [README.md - Secret Rotation](./README.md#secret-rotation)
 ### Troubleshoot issues
 → [README.md - Troubleshooting](./README.md#troubleshooting)
 ### Set up for production
 → [README.md - Best Practices](./README.md#best-practices)
 ## 📦 Repository Structure
 ```
 managedsecret-operator/
 ├── README.md                    # Main documentation
 ├── INDEX.md                     # This file
 └── examples/
    ├── EXAMPLES-SUMMARY.md      # Examples overview
    ├── example-1-no-rotation.yaml
    ├── example-2-registry-rotation-simple.yaml
    ├── example-3-postgres-rotation-previous.yaml
    ├── example-4-minio-rotation-previous.yaml
    ├── example-5-mysql-rotation-previous.yaml
    └── example-6-comprehensive-multi-service.yaml
 ```
 ## 📞 Need Help?
 1. Check [README.md - Troubleshooting](./README.md#troubleshooting)
 2. Review the examples for similar use cases
 3. Contact your administrator for registry/Vault setup issues
 ---
 **Version:** 1.0.7  
 **Registry:** registry.c5ai.ch/charts/managedsecret-operator
--- a/README-PUBLIC.md
+++ b/README-PUBLIC.md
@@ -0,0 +1,711 @@
 # ManagedSecret Operator
 A Kubernetes operator for declarative secret management with automated rotation, drift detection, and zero-downtime password changes.
 ## Overview
 The ManagedSecret operator provides:
 - 🔐 **Generates secrets** with configurable password policies
 - 🏦 **Stores in Vault/OpenBao** as the authoritative source
 - 🔄 **Syncs to Kubernetes Secrets** for application consumption
 - 🔍 **Detects and fixes drift** automatically (Vault → K8s)
 - 🔁 **Rotates secrets** on a configurable schedule
 - ⏱️ **Preserves previous versions** during rotation for zero-downtime password changes
 ### Key Principle: Vault/OpenBao is ALWAYS the source of truth
 - Changes in Vault → Synced to K8s automatically
 - Changes in K8s → Overwritten by Vault values
 - Drift detection runs every 1 minute
 ---
 ## 🚀 Quick Start
 > **Prerequisites:** Contact your administrator to receive:
 > - Registry username and password for `registry.c5ai.ch`
 > - Confirmation that OpenBao/Vault is configured for your cluster
 ```bash
 # 1. Set your credentials (provided by administrator)
 export REGISTRY_USER="<provided-by-admin>"
 export REGISTRY_PASS="<provided-by-admin>"
 # 2. Install Reloader (for automatic pod restarts during rotation)
 kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
 # 3. Login to Helm registry
 helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS
 # 4. Create imagePullSecret
 kubectl create namespace managedsecret-operator-system
 kubectl create secret docker-registry registry-pull-secret \
  --docker-server=registry.c5ai.ch \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --namespace managedsecret-operator-system
 # 5. Install operator via Helm
 helm install managedsecret-operator oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  --set imagePullSecrets[0].name=registry-pull-secret
 # 6. Verify installation
 kubectl get pods -n managedsecret-operator-system
 ```
 That's it! Now check the [examples](./examples/) folder for comprehensive usage scenarios.
 ---
 ## 📦 What You Get
 ### Helm Chart from registry.c5ai.ch
 The operator is distributed as a Helm chart:
 - ✅ **No building required** - Just `helm install`
 - ✅ **Versioned releases** - Use specific versions or upgrade easily
 - ✅ **Pre-configured** - Sensible defaults, customize via `values.yaml`
 - ✅ **Private & secure** - Hosted on your infrastructure
 ### What's Deployed
 ```
 managedsecret-operator-system/
 ├── CustomResourceDefinition (ManagedSecret)
 ├── ServiceAccount (controller-manager)
 ├── Role & RoleBinding (RBAC)
 ├── ClusterRole & ClusterRoleBinding
 ├── Deployment (controller-manager)
 └── ConfigMap (operator configuration)
 ```
 ---
 ## Prerequisites
 ### On Your Side (User)
 - ✅ Kubernetes cluster (1.24+)
 - ✅ kubectl configured and working
 - ✅ Helm 3.x installed
 - ✅ Cluster admin permissions
 ### Provided by Administrator
 - ✅ **Registry credentials** for `registry.c5ai.ch`
 - ✅ **OpenBao/Vault configuration** for your cluster
  - Vault address (e.g., `http://openbao.openbao.svc.cluster.local:8200`)
  - Kubernetes auth configured
  - Vault role name (typically `managedsecret-operator`)
 ### Optional (Recommended)
 - ✅ **Reloader by Stakater** (for automatic pod restarts on rotation)
 ---
 ## Installation Steps
 ### Step 1: Install Reloader (Recommended)
 Reloader automatically restarts pods when secrets change:
 ```bash
 # Direct install (simplest)
 kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
 # Via Helm
 helm repo add stakater https://stakater.github.io/stakater-charts
 helm install reloader stakater/reloader \
  --namespace reloader \
  --create-namespace
 ```
 ### Step 2: Authenticate to Registry
 ```bash
 # Set credentials from administrator
 export REGISTRY_USER="<your-username>"
 export REGISTRY_PASS="<your-password>"
 # Login to Helm registry
 helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS
 # Create namespace
 kubectl create namespace managedsecret-operator-system
 # Create imagePullSecret
 kubectl create secret docker-registry registry-pull-secret \
  --docker-server=registry.c5ai.ch \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --namespace managedsecret-operator-system
 ```
 ### Step 3: Install the Operator
 ```bash
 # Install with default configuration
 helm install managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  --set imagePullSecrets[0].name=registry-pull-secret
 # Or with custom values
 cat > values.yaml <<EOF
 imagePullSecrets:
  - name: registry-pull-secret
 replicaCount: 1
 resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi
 EOF
 helm install managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  -f values.yaml
 ```
 ### Step 4: Verify Installation
 ```bash
 # Check operator is running
 kubectl get pods -n managedsecret-operator-system
 # Check CRD is installed
 kubectl get crd managedsecrets.secrets.c5ai.ch
 # View operator logs
 kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager --tail=50
 ```
 ---
 ## 📖 Usage
 ### Basic Example: Simple API Key (No Rotation)
 ```yaml
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: api-keys
  namespace: my-app
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: my-app/api-keys
  fields:
  - name: api-key
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 10
      symbolCharacters: ""  # Alphanumeric only
  - name: api-endpoint
    type: static
    value: "https://api.example.com/v1"
  destination:
    name: api-secret
    type: Opaque
  rotation:
    enabled: false
 ```
 Apply it:
 ```bash
 kubectl apply -f managedsecret.yaml
 ```
 The operator will:
 1. Generate a 64-character API key
 2. Store it in Vault at `secret/data/my-app/api-keys`
 3. Create Kubernetes Secret `api-secret` in namespace `my-app`
 Use it in your pod:
 ```yaml
 env:
 - name: API_KEY
  valueFrom:
    secretKeyRef:
      name: api-secret
      key: api-key
 ```
 ### Rotation Example: PostgreSQL with Previous Version
 For services that need the old password to authenticate before changing to the new password:
 ```yaml
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: postgres-credentials
  namespace: database
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: database/postgres
  fields:
  - name: username
    type: static
    value: "app_user"
  - name: password
    type: generated
    generator:
      type: password
      length: 40
      minDigits: 8
      minSymbols: 8
  destination:
    name: postgres-secret
    type: Opaque
    keepPreviousVersion: true  # Creates postgres-secret-previous
    previousVersionTTL: 1h     # Cleanup after 1 hour
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
    rotateGeneratedOnly: true
 ```
 When rotation happens:
 1. New password is generated
 2. `postgres-secret` is updated with new password
 3. `postgres-secret-previous` is created with old password
 4. Your CronJob uses old password to authenticate and set new password
 5. After 1 hour, `-previous` secret is automatically cleaned up
 **Important:** You'll need a CronJob to handle the password change. See [examples/example-3-postgres-rotation-previous.yaml](./examples/example-3-postgres-rotation-previous.yaml) for a complete implementation.
 ---
 ## 📚 Examples
 The `examples/` folder contains comprehensive scenarios:
 | Example | Use Case | Description |
 |---------|----------|-------------|
 | **[Example 1](./examples/example-1-no-rotation.yaml)** | API Keys | Long-lived credentials without rotation |
 | **[Example 2](./examples/example-2-registry-rotation-simple.yaml)** | Container Registry | Simple rotation with Reloader |
 | **[Example 3](./examples/example-3-postgres-rotation-previous.yaml)** | PostgreSQL | Rotation with previous version (needs old password) |
 | **[Example 4](./examples/example-4-minio-rotation-previous.yaml)** | MinIO | Object storage with admin credentials rotation |
 | **[Example 5](./examples/example-5-mysql-rotation-previous.yaml)** | MySQL | Database rotation with multiple strategies |
 | **[Example 6](./examples/example-6-comprehensive-multi-service.yaml)** | Full Stack | Complete application with mixed strategies |
 **See [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md) for detailed explanations and decision trees.**
 ---
 ## 🔄 Secret Rotation
 ### How Rotation Works
 ```
 Day 0: Initial setup
  - ManagedSecret creates secret with generated password
  - Stored in Vault and synced to K8s
 Day 90: Rotation triggered (schedule: 2160h)
  1. Operator generates new password
  2. Updates Vault
  3. If keepPreviousVersion=true:
     - Creates <secret-name>-previous with old values
  4. Updates K8s Secret with new values
  5. If Reloader is enabled:
     - Reloader detects change → Restarts pods
  6. After previousVersionTTL:
     - Operator deletes -previous secret
 ```
 ### When to Use `keepPreviousVersion`
 Use `keepPreviousVersion: true` when your service needs the **old password to authenticate** before changing to the new password:
 - ✅ PostgreSQL, MySQL, MariaDB (`ALTER USER` requires authentication)
 - ✅ MinIO (`mc admin` needs old credentials)
 - ✅ LDAP, Active Directory
 - ❌ Container registries (htpasswd regenerated fresh)
 - ❌ Redis (CONFIG SET doesn't need old password)
 - ❌ Web services with startup scripts
 ### Forcing Rotation
 ```bash
 # Trigger immediate rotation
 kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite
 ```
 ---
 ## 🔍 Drift Detection
 The operator automatically detects and fixes drift every 60 seconds:
 **Scenario 1: Secret deleted in Kubernetes**
 ```bash
 kubectl delete secret my-secret -n my-app
 # Operator recreates it from Vault within 60 seconds
 ```
 **Scenario 2: Secret modified in Vault**
 ```bash
 # Update value in Vault
 vault kv put secret/my-app/keys password=newvalue
 # Operator syncs to K8s within 60 seconds
 ```
 **Scenario 3: Secret manually edited in K8s**
 ```bash
 kubectl edit secret my-secret -n my-app
 # Changes are overwritten by Vault values within 60 seconds
 ```
 ---
 ## 🔧 Troubleshooting
 ### Operator Not Starting
 ```bash
 # Check pod status
 kubectl get pods -n managedsecret-operator-system
 # View logs
 kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager --tail=100
 # Common issues:
 # - ImagePullBackOff: Check imagePullSecret is created correctly
 # - CrashLoopBackOff: Check Vault connectivity
 ```
 ### Secret Not Created
 ```bash
 # Check ManagedSecret status
 kubectl get managedsecret <name> -n <namespace> -o yaml
 # Look for conditions/events
 kubectl describe managedsecret <name> -n <namespace>
 # Check operator logs
 kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager | grep <name>
 ```
 ### Vault Authentication Failing
 ```bash
 # Verify ServiceAccount exists
 kubectl get sa controller-manager -n managedsecret-operator-system
 # Check Vault role configuration
 vault read auth/kubernetes/role/managedsecret-operator
 # Test authentication manually
 kubectl exec -it -n managedsecret-operator-system \
  deployment/managedsecret-operator-controller-manager -- sh
 # Inside pod: Check /var/run/secrets/kubernetes.io/serviceaccount/token exists
 ```
 ### Rotation Not Working
 ```bash
 # Check ManagedSecret schedule
 kubectl get managedsecret <name> -n <namespace> -o yaml | grep schedule
 # Force rotation to test
 kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite
 # Check if Reloader is installed (if using rotation)
 kubectl get pods -n reloader
 ```
 ### Reloader Not Restarting Pods
 ```bash
 # Verify Reloader is running
 kubectl get pods -n reloader
 # Check pod has correct annotation
 kubectl get deployment <name> -n <namespace> -o yaml | grep reloader
 # View Reloader logs
 kubectl logs -n reloader -l app=reloader
 # Required annotation on pod template:
 # annotations:
 #   reloader.stakater.com/auto: "true"
 ```
 ---
 ## 🎯 Best Practices
 ### 1. Always Use Reloader for Rotating Secrets
 Without Reloader, pods won't pick up new credentials:
 ```yaml
 template:
  metadata:
    annotations:
      reloader.stakater.com/auto: "true"
 ```
 ### 2. Set Appropriate Rotation Schedules
 - **High Security** (passwords, admin credentials): 30-90 days
 - **Medium Security** (application credentials): 90-180 days
 - **Low Security** (read-only API keys): 180+ days or disabled
 ### 3. Use `keepPreviousVersion` Correctly
 Only use it when the service needs the old password to authenticate:
 ```yaml
 destination:
  keepPreviousVersion: true  # Only if old password is needed
  previousVersionTTL: 1h     # Adjust based on your workflow
 ```
 ### 4. Test Rotation Before Production
 ```bash
 # Create test ManagedSecret with short rotation
 spec:
  rotation:
    enabled: true
    schedule: 5m  # Test rotation every 5 minutes
 # Watch the rotation
 kubectl get secrets -n <namespace> -w
 # Verify application handles rotation
 kubectl logs -n <namespace> <pod-name>
 ```
 ### 5. Monitor Rotation Jobs
 For services using `keepPreviousVersion`, monitor your CronJobs:
 ```bash
 # Check job status
 kubectl get jobs -n <namespace>
 # View job logs
 kubectl logs -n <namespace> -l job-name=<rotation-job>
 # Set up alerts for failed jobs
 ```
 ### 6. Backup Vault Regularly
 Vault is your source of truth:
 ```bash
 # Example: Backup with Velero
 velero backup create vault-backup \
  --include-namespaces openbao \
  --snapshot-volumes
 ```
 ### 7. Version Control Your ManagedSecrets
 Store ManagedSecret manifests in Git:
 ```yaml
 # managedsecrets/
 # ├── production/
 # │   ├── postgres-credentials.yaml
 # │   └── api-keys.yaml
 # └── staging/
 #     ├── postgres-credentials.yaml
 #     └── api-keys.yaml
 ```
 ### 8. Use GitOps for Deployment
 Deploy with ArgoCD or Flux:
 ```yaml
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: managedsecrets
 spec:
  source:
    path: managedsecrets/production
  destination:
    namespace: default
 ```
 ---
 ## 🔒 Security Considerations
 ### Production Checklist
 - [ ] OpenBao/Vault is properly secured (TLS, audit logging, backups)
 - [ ] RBAC configured with least-privilege
 - [ ] Network policies in place
 - [ ] Monitoring and alerting configured
 - [ ] Disaster recovery plan documented
 - [ ] Registry credentials stored securely (not in Git)
 ### Credentials Management
 The operator needs:
 1. Access to Vault (via Kubernetes ServiceAccount auth)
 2. Access to pull its image (via imagePullSecret)
 Both use Kubernetes-native methods. **Never hardcode credentials in manifests!**
 ---
 ## 📞 Support
 ### Getting Help
 1. **Check documentation**:
   - This README
   - [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md)
   - Individual example files
 2. **Check operator logs**:
   ```bash
   kubectl logs -n managedsecret-operator-system \
     -l control-plane=controller-manager --tail=100
   ```
 3. **Contact your administrator** if:
   - You need registry credentials
   - Vault is not configured for your cluster
   - Operator authentication issues
 ### For Administrators
 #### Information to Provide Users
 **Registry Credentials:**
 ```
 Registry: registry.c5ai.ch
 Username: <username>
 Password: <password>
 Chart: oci://registry.c5ai.ch/charts/managedsecret-operator
 Version: 1.0.7
 ```
 **Vault Configuration:**
 ```
 Address: http://openbao.openbao.svc.cluster.local:8200
 Auth Method: kubernetes
 Role: managedsecret-operator
 KV Mount: secret
 KV Version: v2
 ```
 #### Prerequisites for New Cluster
 ```bash
 # 1. Enable Kubernetes auth
 bao auth enable kubernetes
 bao write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"
 # 2. Create policy
 bao policy write managedsecret-operator - <<'EOF'
 path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
 }
 path "secret/metadata/*" {
  capabilities = ["list", "read", "delete"]
 }
 EOF
 # 3. Create role
 bao write auth/kubernetes/role/managedsecret-operator \
    bound_service_account_names=controller-manager \
    bound_service_account_namespaces=managedsecret-operator-system \
    policies=managedsecret-operator \
    ttl=1h
 ```
 ---
 ## 🔄 Upgrading
 ### Upgrade Operator Version
 ```bash
 # Check current version
 helm list -n managedsecret-operator-system
 # Upgrade to new version
 helm upgrade managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.8 \
  --namespace managedsecret-operator-system \
  --reuse-values
 ```
 ### Migration Notes
 When upgrading from earlier versions:
 - CRD updates are handled automatically
 - Existing ManagedSecrets continue to work
 - No manual migration required
 - Check changelog for breaking changes
 ---
 ## 📝 Changelog
 ### Version 1.0.7
 - ✨ Added `keepPreviousVersion` for zero-downtime rotation
 - ✨ Added `previousVersionTTL` for automatic cleanup
 - 📦 Helm chart deployment support
 - 📚 Comprehensive examples and documentation
 - 🐛 Bug fixes and improvements
 ---
 ## License
 Proprietary Use License
 Permission is granted to use this software for any purpose, including commercial use, without charge.
 Modification, adaptation, or creation of derivative works is prohibited.
 Redistribution of modified versions is prohibited.
 ---
 **Version:** 1.0.7  
 **Registry:** registry.c5ai.ch/charts/managedsecret-operator  
 **Minimum Kubernetes:** 1.24+  
 **Compatible with:** OpenBao, HashiCorp Vault
--- a/examples/EXAMPLES-SUMMARY.md
+++ b/examples/EXAMPLES-SUMMARY.md
@@ -0,0 +1,316 @@
 # ManagedSecret Operator - Examples Summary
 This directory contains comprehensive examples for the ManagedSecret Operator, demonstrating various rotation strategies and use cases.
 ## Files Overview
 ### Documentation
 - **README.md** - Complete installation and usage guide with new features
  - Added `keepPreviousVersion` and `previousVersionTTL` documentation
  - Added Reloader integration guide
  - Updated best practices and architecture diagrams
 ### Example Files
 #### Example 1: No Rotation (Simple API Key)
 **File:** `example-1-no-rotation.yaml`
 **Use Case:** Long-lived API keys that don't need rotation
 **Key Features:**
 - Generated once and kept indefinitely
 - No Reloader needed (secret never changes)
 - Perfect for external service API keys
 **When to Use:**
 - Third-party API keys (Stripe, SendGrid, etc.)
 - Service identifiers
 - Static configuration values
 ---
 #### Example 2: Simple Rotation with Reloader (Container Registry)
 **File:** `example-2-registry-rotation-simple.yaml`
 **Use Case:** Docker/Container registry with HTTP basic auth
 **Key Features:**
 - Password rotates every 90 days
 - InitContainer regenerates htpasswd on pod restart
 - Reloader automatically restarts pods
 - **No `keepPreviousVersion` needed** - htpasswd is regenerated fresh
 **Rotation Flow:**
 1. ManagedSecret generates new password → Updates Vault → Updates K8s Secret
 2. Reloader detects Secret change → Triggers pod restart
 3. InitContainer runs → Reads new password → Generates fresh htpasswd
 4. Registry starts with new htpasswd
 **When to Use:**
 - Container registries (Docker Registry, Harbor)
 - Web servers with htpasswd (Apache, Nginx)
 - Any service where credentials are regenerated at startup
 ---
 #### Example 3: Complex Rotation with Previous Password (PostgreSQL)
 **File:** `example-3-postgres-rotation-previous.yaml`
 **Use Case:** PostgreSQL user password that requires authentication to change
 **Key Features:**
 - Password rotates every 90 days
 - `keepPreviousVersion: true` creates `-previous` secret
 - CronJob uses old password to authenticate, sets new password
 - `previousVersionTTL: 1h` cleans up after grace period
 **Rotation Flow:**
 1. ManagedSecret creates both `postgres-secret` and `postgres-secret-previous`
 2. CronJob (every 5 min) detects rotation
 3. Authenticates with OLD password: `PGPASSWORD=$OLD_PASSWORD psql`
 4. Changes password: `ALTER USER app_user PASSWORD '$NEW_PASSWORD'`
 5. Verifies new password works
 6. After 1 hour, `-previous` secret is deleted
 **When to Use:**
 - PostgreSQL, MySQL, MariaDB
 - Any database requiring authentication to change passwords
 - Services with `ALTER USER` or similar commands
 ---
 #### Example 4: MinIO Admin Password Rotation
 **File:** `example-4-minio-rotation-previous.yaml`
 **Use Case:** MinIO admin credentials with rotation
 **Key Features:**
 - Password rotates every 60 days
 - Uses MinIO Client (mc) to change password
 - `keepPreviousVersion: true` with 2h TTL
 - CronJob handles rotation via `mc admin user password`
 **Special Considerations:**
 - MinIO does NOT reload credentials from environment variables
 - Manual pod restart required after rotation
 - Consider using a sidecar to monitor and restart
 **Rotation Flow:**
 1. ManagedSecret creates both secrets
 2. CronJob authenticates with old credentials: `mc alias set oldminio`
 3. Changes password: `mc admin user password myminio admin $NEW_PASSWORD`
 4. Verifies new credentials
 5. **Important:** MinIO pods must be restarted manually
 **When to Use:**
 - MinIO object storage
 - Any service that doesn't auto-reload credentials
 - Services requiring separate admin tools for password changes
 ---
 #### Example 5: MySQL User Password Rotation
 **File:** `example-5-mysql-rotation-previous.yaml`
 **Use Case:** MySQL application user with automated rotation
 **Key Features:**
 - Password rotates every 90 days
 - Multiple password change methods (ALTER USER, SET PASSWORD)
 - Root credentials as fallback
 - `keepPreviousVersion: true` with 1h TTL
 **Rotation Flow:**
 1. ManagedSecret creates both secrets
 2. CronJob tries multiple approaches:
   - Method 1: `ALTER USER` as root (preferred)
   - Method 2: `SET PASSWORD` as root
   - Method 3: `SET PASSWORD` as user with old password
 3. Flushes privileges
 4. Verifies new password
 **When to Use:**
 - MySQL 5.7+, MySQL 8.0
 - MariaDB
 - Any MySQL-compatible database
 ---
 #### Example 6: Comprehensive Multi-Service Deployment
 **File:** `example-6-comprehensive-multi-service.yaml`
 **Use Case:** Complete application stack showing all strategies together
 **Components:**
 1. **PostgreSQL** - Rotates with previous version (90 days)
 2. **Redis** - Rotates without previous version (30 days)
 3. **API Keys** - No rotation (managed externally)
 4. **TLS Certificates** - No rotation (cert-manager)
 5. **JWT Secret** - Simple rotation with Reloader (180 days)
 **Demonstrates:**
 - Multiple secrets with different rotation schedules
 - Mixed rotation strategies in one application
 - How Reloader restarts pods for all credential changes
 - Real-world production setup
 **When to Use:**
 - As a template for complex applications
 - To understand how different strategies work together
 - For learning all features in one example
 ---
 ## Quick Decision Tree
 ### Does your service need the OLD password to set the NEW password?
 **YES** (PostgreSQL, MySQL, MinIO, LDAP):
 - Use `keepPreviousVersion: true`
 - Set appropriate `previousVersionTTL`
 - Create CronJob to handle rotation
 - See Examples 3, 4, or 5
 **NO** (Registry, Redis, Web servers):
 - Use simple rotation with Reloader
 - InitContainer or startup script regenerates credentials
 - No CronJob needed
 - See Example 2
 **NOT ROTATING** (API keys, Certificates):
 - Set `rotation.enabled: false`
 - No Reloader annotation needed
 - See Example 1
 ---
 ## Rotation Schedule Recommendations
 - **High Security** (User passwords, admin credentials): 30-90 days
 - **Medium Security** (Application credentials, service accounts): 90-180 days
 - **Low Security** (Read-only API keys, development): 180+ days or disabled
 - **Previous Version TTL**: 
  - Automated rotation: 1h
  - Manual intervention: 2-4h
  - Complex workflows: 24h
 ---
 ## Prerequisites for All Examples
 1. **OpenBao/Vault** deployed and initialized
 2. **ManagedSecret Operator** installed
 3. **Reloader** installed (for rotation examples)
 4. **cert-manager** (optional, for TLS examples)
 ## Installation Order
 ```bash
 # 1. Install OpenBao
 kubectl create namespace openbao
 helm install openbao openbao/openbao -n openbao
 # 2. Install ManagedSecret Operator
 kubectl apply -f managedsecret-operator/
 # 3. Install Reloader
 kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
 # 4. Deploy your application with examples
 kubectl apply -f example-X-your-usecase.yaml
 ```
 ---
 ## Testing Rotation
 ### Force Immediate Rotation
 ```bash
 # Trigger rotation for any ManagedSecret
 kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite
 ```
 ### Watch the Flow
 ```bash
 # Terminal 1: Watch secrets
 kubectl get secrets -n <namespace> -w
 # Terminal 2: Watch pods
 kubectl get pods -n <namespace> -w
 # Terminal 3: Watch jobs (for CronJobs)
 kubectl get jobs -n <namespace> -w
 # Terminal 4: Reloader logs
 kubectl logs -n reloader -l app=reloader -f
 ```
 ### Verify Rotation Completed
 ```bash
 # Check if -previous secret exists (should only exist during grace period)
 kubectl get secret <name>-previous -n <namespace>
 # View rotation job logs
 kubectl logs -n <namespace> -l job-name=<rotation-job>
 # Test connection with new credentials
 kubectl exec -it -n <namespace> <pod> -- <test-command>
 ```
 ---
 ## Common Issues and Solutions
 ### Issue: Pod not restarting after rotation
 **Solution:** Check Reloader is installed and annotation is correct
 ```bash
 kubectl get pods -n reloader
 kubectl get deployment <app> -o yaml | grep reloader
 ```
 ### Issue: Rotation job failing
 **Solution:** Check if previous secret exists and credentials work
 ```bash
 kubectl get secret <name>-previous -n <namespace>
 kubectl logs -n <namespace> -l job-name=<rotation-job>
 ```
 ### Issue: Application still using old credentials
 **Solution:** Verify pod actually restarted after secret change
 ```bash
 kubectl get pods -n <namespace> -o wide
 # Check pod start time vs secret update time
 ```
 ### Issue: Previous secret not cleaned up
 **Solution:** Check operator logs for cleanup job
 ```bash
 kubectl logs -n managedsecret-operator-system -l control-plane=controller-manager | grep cleanup
 ```
 ---
 ## Best Practices Summary
 1. ✅ **Always use Reloader** with rotating secrets
 2. ✅ **Use `keepPreviousVersion`** for services requiring old password
 3. ✅ **Set realistic `previousVersionTTL`** based on your workflow
 4. ✅ **Test rotation** before production deployment
 5. ✅ **Monitor rotation jobs** for failures
 6. ✅ **Document rotation procedures** for your team
 7. ✅ **Use appropriate rotation schedules** based on security requirements
 8. ✅ **Backup Vault regularly** - it's your source of truth
 9. ✅ **Never modify K8s Secrets manually** - always use ManagedSecret or Vault
 10. ✅ **Set up alerts** for failed rotations
 ---
 ## Support and Contributing
 For issues or questions about these examples:
 - Check the main README.md for troubleshooting
 - Review operator logs for detailed error messages
 - Test in development environment before production
 Remember: Vault/OpenBao is ALWAYS the source of truth. All changes should go through ManagedSecret resources or directly in Vault, never by manually editing Kubernetes Secrets.
--- a/examples/example-1-no-rotation.yaml
+++ b/examples/example-1-no-rotation.yaml
@@ -0,0 +1,122 @@
 # Example 1: Simple API Key (No Rotation)
 #
 # Use case: Long-lived API key that doesn't need rotation
 # 
 # Characteristics:
 # - Generated once
 # - No rotation schedule
 # - No Reloader needed (static secret)
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: api-service
 ---
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: api-keys
  namespace: api-service
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: api/service/keys
  fields:
  # Static service identifier
  - name: service-id
    type: static
    value: "api-service-prod"
  # Generated API key (long, alphanumeric)
  - name: api-key
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 10
      minSymbols: 0
      minLowercase: 20
      minUppercase: 20
      symbolCharacters: ""  # No symbols, just alphanumeric
      allowRepeat: false
  # Static API endpoint
  - name: api-endpoint
    type: static
    value: "https://api.external-service.com/v1"
  destination:
    name: api-secret
    type: Opaque
  # No rotation - generate once and keep
  rotation:
    enabled: false
 ---
 # Example Deployment using the API key
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: api-consumer
  namespace: api-service
 spec:
  replicas: 2
  selector:
    matchLabels:
      app: api-consumer
  template:
    metadata:
      labels:
        app: api-consumer
      # No Reloader annotation needed - secret doesn't change
    spec:
      containers:
      - name: app
        image: your-app:latest
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: api-secret
              key: api-key
        - name: API_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: api-secret
              key: api-endpoint
        - name: SERVICE_ID
          valueFrom:
            secretKeyRef:
              name: api-secret
              key: service-id
 ---
 # Example: Using the secret in a ConfigMap template
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: api-config
  namespace: api-service
 data:
  config.yaml: |
    service:
      name: api-consumer
      # API credentials loaded from secret via environment variables
      # See deployment above for how to inject API_KEY
    endpoints:
      external:
        timeout: 30s
        retry: 3
 ---
 # How to retrieve the API key for external use
 # kubectl get secret api-secret -n api-service -o jsonpath='{.data.api-key}' | base64 -d
--- a/examples/example-2-registry-rotation-simple.yaml
+++ b/examples/example-2-registry-rotation-simple.yaml
@@ -0,0 +1,287 @@
 # Example 2: Container Registry with Rotation + Reloader
 #
 # Use case: Docker registry with basic auth that rotates every 90 days
 # 
 # Characteristics:
 # - Password rotates every 90 days
 # - Registry doesn't need old password to accept new one
 # - InitContainer regenerates htpasswd on pod restart
 # - Reloader automatically restarts pod when secret changes
 # - No keepPreviousVersion needed
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: registry
 ---
 # ManagedSecret generates and rotates registry credentials
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: registry-auth
  namespace: registry
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: registry/auth
  fields:
  - name: username
    type: static
    value: "admin"
  - name: password
    type: generated
    generator:
      type: password
      length: 32
      minDigits: 5
      minSymbols: 5
      minLowercase: 5
      minUppercase: 5
      symbolCharacters: "!@#$%^&*"
      allowRepeat: false
  destination:
    name: registry-credentials
    type: Opaque
    # No need for previous version - htpasswd is regenerated fresh
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
    rotateGeneratedOnly: true  # Only rotate password, keep username
 ---
 # PersistentVolumeClaim for registry storage
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: registry-pvc
  namespace: registry
 spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: longhorn
 ---
 # Registry Deployment with Reloader integration
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: registry
  namespace: registry
  labels:
    app: registry
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
      annotations:
        # Reloader watches registry-credentials and restarts pod on change
        reloader.stakater.com/auto: "true"
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 65534
        seccompProfile:
          type: RuntimeDefault
      initContainers:
      # Generate htpasswd file from current credentials at pod startup
      - name: generate-htpasswd
        image: httpd:2.4-alpine
        command:
        - sh
        - -c
        - |
          USERNAME=$(cat /credentials/username)
          PASSWORD=$(cat /credentials/password)
          htpasswd -Bbn "$USERNAME" "$PASSWORD" > /auth/htpasswd
          chmod 644 /auth/htpasswd
          echo "htpasswd generated with user: $USERNAME"
        volumeMounts:
        - name: credentials
          mountPath: /credentials
          readOnly: true
        - name: auth
          mountPath: /auth
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 65534
          capabilities:
            drop:
            - ALL
      containers:
      - name: registry
        image: registry:2
        ports:
        - containerPort: 5000
          name: http
        env:
        - name: REGISTRY_STORAGE_DELETE_ENABLED
          value: "true"
        - name: REGISTRY_HTTP_ADDR
          value: "0.0.0.0:5000"
        # Enable HTTP basic auth
        - name: REGISTRY_AUTH
          value: "htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: "Registry Realm"
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: "/auth/htpasswd"
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 65534
          capabilities:
            drop:
            - ALL
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - name: registry-storage
          mountPath: /var/lib/registry
        - name: auth
          mountPath: /auth
          readOnly: true
        livenessProbe:
          httpGet:
            path: /
            port: 5000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 5000
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
      volumes:
      - name: registry-storage
        persistentVolumeClaim:
          claimName: registry-pvc
      - name: credentials
        secret:
          secretName: registry-credentials
      - name: auth
        emptyDir: {}
 ---
 # Service for internal access
 apiVersion: v1
 kind: Service
 metadata:
  name: registry
  namespace: registry
 spec:
  type: ClusterIP
  selector:
    app: registry
  ports:
  - port: 5000
    targetPort: 5000
    protocol: TCP
    name: http
 ---
 # Ingress for external access with TLS
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: registry
  namespace: registry
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
 spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - registry.c5ai.ch
    secretName: registry-ingress-tls
  rules:
  - host: registry.c5ai.ch
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: registry
            port:
              number: 5000
 ---
 # How the rotation flow works:
 #
 # 1. After 90 days, ManagedSecret operator:
 #    - Generates new password
 #    - Updates Vault
 #    - Updates registry-credentials Secret
 #
 # 2. Reloader detects Secret change:
 #    - Sees resourceVersion changed
 #    - Triggers rolling restart of registry Deployment
 #
 # 3. Pod restart with new credentials:
 #    - InitContainer runs
 #    - Reads NEW password from registry-credentials
 #    - Generates fresh htpasswd file
 #    - Registry container starts with new htpasswd
 #
 # 4. Result:
 #    - Old clients with old password: FAIL (401 Unauthorized)
 #    - New clients with new password: SUCCESS
 #    - Downtime: ~1-2 minutes during rolling restart
 ---
 # Testing commands:
 #
 # # Get current credentials
 # kubectl get secret registry-credentials -n registry -o jsonpath='{.data.username}' | base64 -d
 # kubectl get secret registry-credentials -n registry -o jsonpath='{.data.password}' | base64 -d
 #
 # # Test without auth (should fail)
 # curl -i https://registry.c5ai.ch/v2/
 #
 # # Test with auth (should succeed)
 # REGISTRY_USER=$(kubectl get secret registry-credentials -n registry -o jsonpath='{.data.username}' | base64 -d)
 # REGISTRY_PASS=$(kubectl get secret registry-credentials -n registry -o jsonpath='{.data.password}' | base64 -d)
 # curl -u $REGISTRY_USER:$REGISTRY_PASS https://registry.c5ai.ch/v2/_catalog
 #
 # # Docker login
 # docker login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS
 #
 # # Force rotation test
 # kubectl annotate managedsecret registry-auth -n registry reconcile="$(date +%s)" --overwrite
 # kubectl get pods -n registry -w  # Watch pod restart
--- a/examples/example-3-postgres-rotation-previous.yaml
+++ b/examples/example-3-postgres-rotation-previous.yaml
@@ -0,0 +1,369 @@
 # Example 3: PostgreSQL Password Rotation with Previous Version
 #
 # Use case: PostgreSQL user password that rotates every 90 days
 # 
 # Characteristics:
 # - Password rotates every 90 days
 # - PostgreSQL REQUIRES old password to authenticate before changing
 # - keepPreviousVersion creates postgres-secret-previous
 # - CronJob uses old password to authenticate, sets new password
 # - After 1 hour grace period, old secret is cleaned up
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: database
 ---
 # ManagedSecret with previous version support
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: postgres-app-user
  namespace: database
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: postgres/app-user
  fields:
  - name: username
    type: static
    value: "app_user"
  - name: password
    type: generated
    generator:
      type: password
      length: 40
      minDigits: 8
      minSymbols: 8
      minLowercase: 8
      minUppercase: 8
      symbolCharacters: "!@#$%^&*()"
      allowRepeat: false
  - name: host
    type: static
    value: "postgres.database.svc.cluster.local"
  - name: port
    type: static
    value: "5432"
  - name: database
    type: static
    value: "app_db"
  destination:
    name: postgres-secret
    type: Opaque
    # Keep previous version for authentication during rotation
    keepPreviousVersion: true
    previousVersionTTL: 1h  # Clean up after 1 hour
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
    rotateGeneratedOnly: true
 ---
 # ServiceAccount for password rotation job
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: postgres-rotator
  namespace: database
 ---
 # Role to allow reading secrets
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: postgres-rotator
  namespace: database
 rules:
 - apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
  resourceNames: ["postgres-secret", "postgres-secret-previous"]
 ---
 # RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: postgres-rotator
  namespace: database
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: postgres-rotator
 subjects:
 - kind: ServiceAccount
  name: postgres-rotator
  namespace: database
 ---
 # CronJob to detect rotation and update PostgreSQL password
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: postgres-password-rotator
  namespace: database
 spec:
  # Run every 5 minutes to check for rotation
  schedule: "*/5 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        metadata:
          annotations:
            # Reloader can also trigger this job when secret changes
            reloader.stakater.com/auto: "true"
        spec:
          serviceAccountName: postgres-rotator
          restartPolicy: OnFailure
          containers:
          - name: rotate-password
            image: postgres:15-alpine
            env:
            # Current credentials
            - name: PGUSER
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: username
            - name: NEW_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: PGHOST
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: host
            - name: PGPORT
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: port
            - name: PGDATABASE
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: database
            # Previous credentials (for authentication)
            - name: OLD_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret-previous
                  key: password
                  optional: true  # Might not exist if no rotation yet
            command:
            - sh
            - -c
            - |
              set -e
              echo "=========================================="
              echo "PostgreSQL Password Rotation Check"
              echo "=========================================="
              echo "User: $PGUSER"
              echo "Host: $PGHOST:$PGPORT"
              echo "Database: $PGDATABASE"
              echo ""
              # Check if previous password exists (rotation happened)
              if [ -z "$OLD_PASSWORD" ]; then
                echo "No previous password found - no rotation needed"
                exit 0
              fi
              # Check if passwords are different
              if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then
                echo "Passwords are identical - no rotation needed"
                exit 0
              fi
              echo "Rotation detected! Old and new passwords differ."
              echo ""
              # Test if current password in PostgreSQL is the old one
              echo "Testing authentication with OLD password..."
              if PGPASSWORD=$OLD_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then
                echo "✓ Old password still active in PostgreSQL"
                echo ""
                echo "Updating to NEW password..."
                # Authenticate with OLD password, set NEW password
                if PGPASSWORD=$OLD_PASSWORD psql -c "ALTER USER $PGUSER PASSWORD '$NEW_PASSWORD';"; then
                  echo "✓ Password updated successfully!"
                  echo ""
                  # Verify new password works
                  echo "Verifying NEW password..."
                  if PGPASSWORD=$NEW_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then
                    echo "✓ New password verified and working!"
                    echo ""
                    echo "=========================================="
                    echo "Rotation completed successfully"
                    echo "=========================================="
                    exit 0
                  else
                    echo "✗ ERROR: New password doesn't work!"
                    exit 1
                  fi
                else
                  echo "✗ ERROR: Failed to update password"
                  exit 1
                fi
              else
                echo "Old password doesn't work - checking if new password already active..."
                if PGPASSWORD=$NEW_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then
                  echo "✓ New password is already active in PostgreSQL"
                  echo "Rotation was already completed"
                  exit 0
                else
                  echo "✗ ERROR: Neither old nor new password works!"
                  echo "Manual intervention required"
                  exit 1
                fi
              fi
 ---
 # Example application Deployment using the credentials
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: app
  namespace: database
 spec:
  replicas: 2
  selector:
    matchLabels:
      app: postgres-client
  template:
    metadata:
      labels:
        app: postgres-client
      annotations:
        # Reloader restarts app when credentials change
        reloader.stakater.com/auto: "true"
    spec:
      containers:
      - name: app
        image: postgres:15-alpine
        env:
        - name: PGUSER
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: username
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGHOST
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: host
        - name: PGPORT
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: port
        - name: PGDATABASE
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: database
        command:
        - sh
        - -c
        - |
          # Your application code here
          while true; do
            psql -c "SELECT NOW();" || echo "Connection failed"
            sleep 30
          done
 ---
 # How the rotation flow works:
 #
 # Day 0: Initial setup
 #   - ManagedSecret creates postgres-secret with generated password
 #   - No postgres-secret-previous exists yet
 #   - CronJob runs every 5 minutes, finds no previous password, does nothing
 #
 # Day 90: Rotation triggered
 #   1. ManagedSecret operator:
 #      - Generates NEW password
 #      - Updates Vault with new password
 #      - Creates postgres-secret-previous with OLD password
 #      - Updates postgres-secret with NEW password
 #
 #   2. Within 5 minutes, CronJob runs:
 #      - Detects postgres-secret-previous exists
 #      - Compares OLD_PASSWORD vs NEW_PASSWORD (different!)
 #      - Authenticates to PostgreSQL with OLD_PASSWORD
 #      - Runs: ALTER USER app_user PASSWORD 'new_password'
 #      - Verifies new password works
 #      - Job succeeds
 #
 #   3. After 1 hour (previousVersionTTL):
 #      - ManagedSecret operator deletes postgres-secret-previous
 #      - CronJob future runs find no previous password, do nothing
 #
 #   4. Application pods:
 #      - Reloader detects postgres-secret change
 #      - Triggers rolling restart
 #      - Pods reconnect with NEW password
 #      - ~1 minute downtime per pod during restart
 ---
 # Manual rotation testing:
 #
 # # Check current state
 # kubectl get secret postgres-secret -n database -o yaml
 # kubectl get secret postgres-secret-previous -n database -o yaml  # Might not exist
 #
 # # Force rotation
 # kubectl annotate managedsecret postgres-app-user -n database reconcile="$(date +%s)" --overwrite
 #
 # # Watch the rotation
 # kubectl get secrets -n database -w
 #
 # # Check CronJob executes
 # kubectl get jobs -n database -w
 #
 # # View rotation logs
 # kubectl logs -n database -l job-name=postgres-password-rotator-XXXXX
 #
 # # Verify password in PostgreSQL matches new secret
 # NEW_PASS=$(kubectl get secret postgres-secret -n database -o jsonpath='{.data.password}' | base64 -d)
 # kubectl exec -it -n database deployment/app -- psql -c "SELECT 1;"
 ---
 # Troubleshooting:
 #
 # If rotation fails:
 # 1. Check CronJob logs for errors
 # 2. Verify PostgreSQL is accessible from CronJob pod
 # 3. Check if old password still works in PostgreSQL
 # 4. Manually update if needed:
 #    kubectl exec -it postgres-0 -n database -- psql -U postgres
 #    ALTER USER app_user PASSWORD 'password_from_secret';
--- a/examples/example-4-minio-rotation-previous.yaml
+++ b/examples/example-4-minio-rotation-previous.yaml
@@ -0,0 +1,430 @@
 # Example 4: MinIO Admin Password Rotation with Previous Version
 #
 # Use case: MinIO admin credentials that rotate every 60 days
 # 
 # Characteristics:
 # - Password rotates every 60 days
 # - MinIO Client (mc) needs old credentials to change password
 # - keepPreviousVersion creates minio-admin-secret-previous
 # - CronJob uses old credentials to authenticate, sets new password
 # - After 2 hour grace period, old secret is cleaned up
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: minio
 ---
 # ManagedSecret for MinIO admin credentials
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: minio-admin
  namespace: minio
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: minio/admin
  fields:
  - name: username
    type: static
    value: "admin"
  - name: password
    type: generated
    generator:
      type: password
      length: 32
      minDigits: 5
      minSymbols: 5
      minLowercase: 5
      minUppercase: 5
      symbolCharacters: "!@#$%^&*"
      allowRepeat: false
  - name: endpoint
    type: static
    value: "http://minio.minio.svc.cluster.local:9000"
  destination:
    name: minio-admin-secret
    type: Opaque
    # Keep previous version for authentication during rotation
    keepPreviousVersion: true
    previousVersionTTL: 2h  # Longer grace period for MinIO
  rotation:
    enabled: true
    schedule: 1440h  # 60 days
    rotateGeneratedOnly: true
 ---
 # ServiceAccount for password rotation job
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: minio-rotator
  namespace: minio
 ---
 # Role to allow reading secrets
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: minio-rotator
  namespace: minio
 rules:
 - apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
  resourceNames: ["minio-admin-secret", "minio-admin-secret-previous"]
 ---
 # RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: minio-rotator
  namespace: minio
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: minio-rotator
 subjects:
 - kind: ServiceAccount
  name: minio-rotator
  namespace: minio
 ---
 # CronJob to detect rotation and update MinIO password
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: minio-password-rotator
  namespace: minio
 spec:
  # Run every 10 minutes to check for rotation
  schedule: "*/10 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        metadata:
          annotations:
            reloader.stakater.com/auto: "true"
        spec:
          serviceAccountName: minio-rotator
          restartPolicy: OnFailure
          containers:
          - name: rotate-password
            image: minio/mc:latest
            env:
            # Current credentials
            - name: MC_HOST_myminio
              value: ""  # Will be set in script
            - name: USERNAME
              valueFrom:
                secretKeyRef:
                  name: minio-admin-secret
                  key: username
            - name: NEW_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: minio-admin-secret
                  key: password
            - name: ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: minio-admin-secret
                  key: endpoint
            # Previous credentials (for authentication)
            - name: OLD_USERNAME
              valueFrom:
                secretKeyRef:
                  name: minio-admin-secret-previous
                  key: username
                  optional: true
            - name: OLD_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: minio-admin-secret-previous
                  key: password
                  optional: true
            command:
            - sh
            - -c
            - |
              set -e
              echo "=========================================="
              echo "MinIO Password Rotation Check"
              echo "=========================================="
              echo "User: $USERNAME"
              echo "Endpoint: $ENDPOINT"
              echo ""
              # Check if previous password exists (rotation happened)
              if [ -z "$OLD_PASSWORD" ]; then
                echo "No previous password found - no rotation needed"
                exit 0
              fi
              # Check if passwords are different
              if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then
                echo "Passwords are identical - no rotation needed"
                exit 0
              fi
              echo "Rotation detected! Old and new passwords differ."
              echo ""
              # Test if current password in MinIO is the old one
              echo "Testing authentication with OLD credentials..."
              if mc alias set oldminio $ENDPOINT $OLD_USERNAME $OLD_PASSWORD > /dev/null 2>&1; then
                echo "✓ Old credentials still active in MinIO"
                echo ""
                # Test admin access
                if mc admin info oldminio > /dev/null 2>&1; then
                  echo "✓ Admin access confirmed with old credentials"
                  echo ""
                  echo "Updating to NEW password..."
                  # Change password using old credentials
                  if mc admin user password oldminio $USERNAME $NEW_PASSWORD; then
                    echo "✓ Password updated successfully!"
                    echo ""
                    # Verify new password works
                    echo "Verifying NEW password..."
                    if mc alias set newminio $ENDPOINT $USERNAME $NEW_PASSWORD > /dev/null 2>&1; then
                      if mc admin info newminio > /dev/null 2>&1; then
                        echo "✓ New password verified and working!"
                        echo ""
                        echo "=========================================="
                        echo "Rotation completed successfully"
                        echo "=========================================="
                        exit 0
                      else
                        echo "✗ ERROR: New credentials don't have admin access!"
                        exit 1
                      fi
                    else
                      echo "✗ ERROR: New password doesn't work!"
                      exit 1
                    fi
                  else
                    echo "✗ ERROR: Failed to update password"
                    exit 1
                  fi
                else
                  echo "✗ ERROR: No admin access with old credentials"
                  exit 1
                fi
              else
                echo "Old credentials don't work - checking if new credentials already active..."
                if mc alias set newminio $ENDPOINT $USERNAME $NEW_PASSWORD > /dev/null 2>&1; then
                  if mc admin info newminio > /dev/null 2>&1; then
                    echo "✓ New credentials are already active in MinIO"
                    echo "Rotation was already completed"
                    exit 0
                  else
                    echo "✗ ERROR: New credentials exist but no admin access"
                    exit 1
                  fi
                else
                  echo "✗ ERROR: Neither old nor new credentials work!"
                  echo "Manual intervention required"
                  exit 1
                fi
              fi
 ---
 # Example: MinIO StatefulSet (simplified)
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
  name: minio
  namespace: minio
 spec:
  serviceName: minio
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
      annotations:
        # Note: MinIO doesn't auto-reload credentials from env vars
        # You may need to restart pods manually or use a sidecar
        reloader.stakater.com/auto: "false"
    spec:
      containers:
      - name: minio
        image: minio/minio:latest
        args:
        - server
        - /data
        - --console-address
        - ":9001"
        env:
        - name: MINIO_ROOT_USER
          valueFrom:
            secretKeyRef:
              name: minio-admin-secret
              key: username
        - name: MINIO_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: minio-admin-secret
              key: password
        ports:
        - containerPort: 9000
          name: api
        - containerPort: 9001
          name: console
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
 ---
 # Service for MinIO
 apiVersion: v1
 kind: Service
 metadata:
  name: minio
  namespace: minio
 spec:
  selector:
    app: minio
  ports:
  - port: 9000
    targetPort: 9000
    name: api
  - port: 9001
    targetPort: 9001
    name: console
 ---
 # How the rotation flow works:
 #
 # Day 0: Initial setup
 #   - ManagedSecret creates minio-admin-secret with generated password
 #   - MinIO starts with these credentials
 #   - No minio-admin-secret-previous exists yet
 #   - CronJob runs every 10 minutes, finds no previous password, does nothing
 #
 # Day 60: Rotation triggered
 #   1. ManagedSecret operator:
 #      - Generates NEW password
 #      - Updates Vault with new password
 #      - Creates minio-admin-secret-previous with OLD password
 #      - Updates minio-admin-secret with NEW password
 #
 #   2. Within 10 minutes, CronJob runs:
 #      - Detects minio-admin-secret-previous exists
 #      - Compares OLD_PASSWORD vs NEW_PASSWORD (different!)
 #      - Authenticates to MinIO with OLD credentials
 #      - Runs: mc admin user password myminio admin new_password
 #      - Verifies new password works
 #      - Job succeeds
 #
 #   3. After 2 hours (previousVersionTTL):
 #      - ManagedSecret operator deletes minio-admin-secret-previous
 #      - CronJob future runs find no previous password, do nothing
 #
 #   4. MinIO pods:
 #      - IMPORTANT: MinIO does NOT reload credentials from env vars
 #      - You need to manually restart MinIO pods after rotation
 #      - Or use a sidecar to monitor and trigger restart
 #      - Consider NOT using reloader.stakater.com/auto for MinIO
 ---
 # Alternative: Sidecar to handle MinIO restart
 # Add this to the MinIO StatefulSet if you want automatic restarts
 ---
 # sidecar:
 # - name: credential-watcher
 #   image: bitnami/kubectl:latest
 #   command:
 #   - sh
 #   - -c
 #   - |
 #     LAST_VERSION=""
 #     while true; do
 #       CURRENT_VERSION=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.metadata.resourceVersion}')
 #       if [ -n "$LAST_VERSION" ] && [ "$LAST_VERSION" != "$CURRENT_VERSION" ]; then
 #         echo "Credentials changed! Signaling MinIO to restart..."
 #         # Send SIGTERM to MinIO process to trigger graceful shutdown
 #         killall minio
 #       fi
 #       LAST_VERSION=$CURRENT_VERSION
 #       sleep 30
 #     done
 ---
 # Manual rotation testing:
 #
 # # Check current state
 # kubectl get secret minio-admin-secret -n minio -o yaml
 # kubectl get secret minio-admin-secret-previous -n minio -o yaml
 #
 # # Force rotation
 # kubectl annotate managedsecret minio-admin -n minio reconcile="$(date +%s)" --overwrite
 #
 # # Watch the rotation
 # kubectl get secrets -n minio -w
 #
 # # Check CronJob executes
 # kubectl get jobs -n minio -w
 #
 # # View rotation logs
 # kubectl logs -n minio -l job-name=minio-password-rotator-XXXXX
 #
 # # Test MinIO access with new credentials
 # NEW_USER=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.data.username}' | base64 -d)
 # NEW_PASS=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.data.password}' | base64 -d)
 # mc alias set testminio http://minio.minio.svc.cluster.local:9000 $NEW_USER $NEW_PASS
 # mc admin info testminio
 ---
 # Important Notes:
 #
 # 1. MinIO credential reload:
 #    - Unlike PostgreSQL, MinIO does NOT automatically reload credentials
 #    - After rotation, you must restart MinIO pods
 #    - Consider using a sidecar or manual restart workflow
 #
 # 2. High Availability:
 #    - For HA MinIO clusters, all nodes share the same root credentials
 #    - Rotation updates credentials cluster-wide
 #    - Rolling restart required for all nodes
 #
 # 3. Grace Period:
 #    - 2 hour previousVersionTTL allows time for manual intervention
 #    - Adjust based on your operational procedures
 #
 # 4. Monitoring:
 #    - Monitor CronJob success/failure
 #    - Alert on rotation failures
 #    - Track when MinIO pods were last restarted
--- a/examples/example-5-mysql-rotation-previous.yaml
+++ b/examples/example-5-mysql-rotation-previous.yaml
@@ -0,0 +1,566 @@
 # Example 5: MySQL User Password Rotation with Previous Version
 #
 # Use case: MySQL application user password that rotates every 90 days
 # 
 # Characteristics:
 # - Password rotates every 90 days
 # - MySQL requires old password to authenticate before changing (when using SET PASSWORD)
 # - keepPreviousVersion creates mysql-app-user-previous
 # - CronJob uses old password to authenticate, sets new password
 # - Application pods restart automatically via Reloader
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: mysql
 ---
 # ManagedSecret for MySQL application user
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: mysql-app-user
  namespace: mysql
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: mysql/app-user
  fields:
  - name: username
    type: static
    value: "app_user"
  - name: password
    type: generated
    generator:
      type: password
      length: 32
      minDigits: 6
      minSymbols: 6
      minLowercase: 8
      minUppercase: 8
      # MySQL doesn't allow some special chars in passwords
      symbolCharacters: "!@#$%^&*"
      allowRepeat: false
  - name: host
    type: static
    value: "mysql.mysql.svc.cluster.local"
  - name: port
    type: static
    value: "3306"
  - name: database
    type: static
    value: "app_db"
  - name: connection-string
    type: static
    value: "mysql://app_user@mysql.mysql.svc.cluster.local:3306/app_db"
  destination:
    name: mysql-app-secret
    type: Opaque
    # Keep previous version for authentication during rotation
    keepPreviousVersion: true
    previousVersionTTL: 1h
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
    rotateGeneratedOnly: true
 ---
 # Separate ManagedSecret for MySQL root credentials (no rotation)
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: mysql-root
  namespace: mysql
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: mysql/root
  fields:
  - name: username
    type: static
    value: "root"
  - name: password
    type: generated
    generator:
      type: password
      length: 40
      minDigits: 8
      minSymbols: 8
  destination:
    name: mysql-root-secret
    type: Opaque
  # Root password doesn't rotate automatically
  rotation:
    enabled: false
 ---
 # ServiceAccount for password rotation job
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: mysql-rotator
  namespace: mysql
 ---
 # Role to allow reading secrets
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: mysql-rotator
  namespace: mysql
 rules:
 - apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
  resourceNames: 
  - "mysql-app-secret"
  - "mysql-app-secret-previous"
  - "mysql-root-secret"
 ---
 # RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: mysql-rotator
  namespace: mysql
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: mysql-rotator
 subjects:
 - kind: ServiceAccount
  name: mysql-rotator
  namespace: mysql
 ---
 # CronJob to detect rotation and update MySQL password
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: mysql-password-rotator
  namespace: mysql
 spec:
  # Run every 5 minutes to check for rotation
  schedule: "*/5 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        metadata:
          annotations:
            reloader.stakater.com/auto: "true"
        spec:
          serviceAccountName: mysql-rotator
          restartPolicy: OnFailure
          containers:
          - name: rotate-password
            image: mysql:8.0
            env:
            # Current credentials
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: mysql-app-secret
                  key: username
            - name: NEW_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-app-secret
                  key: password
            - name: MYSQL_HOST
              valueFrom:
                secretKeyRef:
                  name: mysql-app-secret
                  key: host
            - name: MYSQL_PORT
              valueFrom:
                secretKeyRef:
                  name: mysql-app-secret
                  key: port
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: mysql-app-secret
                  key: database
            # Previous credentials (for authentication)
            - name: OLD_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-app-secret-previous
                  key: password
                  optional: true
            # Root credentials for fallback
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-root-secret
                  key: password
            command:
            - bash
            - -c
            - |
              set -e
              echo "=========================================="
              echo "MySQL Password Rotation Check"
              echo "=========================================="
              echo "User: $MYSQL_USER"
              echo "Host: $MYSQL_HOST:$MYSQL_PORT"
              echo "Database: $MYSQL_DATABASE"
              echo ""
              # Check if previous password exists (rotation happened)
              if [ -z "$OLD_PASSWORD" ]; then
                echo "No previous password found - no rotation needed"
                exit 0
              fi
              # Check if passwords are different
              if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then
                echo "Passwords are identical - no rotation needed"
                exit 0
              fi
              echo "Rotation detected! Old and new passwords differ."
              echo ""
              # Test if current password in MySQL is the old one
              echo "Testing authentication with OLD password..."
              if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$OLD_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then
                echo "✓ Old password still active in MySQL"
                echo ""
                echo "Updating to NEW password..."
                # Method 1: Use ALTER USER (MySQL 5.7.6+, doesn't require old password)
                # Connect as root to change password
                if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \
                  -e "ALTER USER '$MYSQL_USER'@'%' IDENTIFIED BY '$NEW_PASSWORD';" 2>/dev/null; then
                  echo "✓ Password updated using ALTER USER (as root)"
                elif mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \
                  -e "SET PASSWORD FOR '$MYSQL_USER'@'%' = PASSWORD('$NEW_PASSWORD');" 2>/dev/null; then
                  echo "✓ Password updated using SET PASSWORD (as root)"
                else
                  # Method 2: Use SET PASSWORD as the user (requires old password connection)
                  if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$OLD_PASSWORD" \
                    -e "SET PASSWORD = PASSWORD('$NEW_PASSWORD');" 2>/dev/null; then
                    echo "✓ Password updated using SET PASSWORD (as user)"
                  else
                    echo "✗ ERROR: Failed to update password with any method"
                    exit 1
                  fi
                fi
                echo ""
                # Verify new password works
                echo "Verifying NEW password..."
                if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$NEW_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then
                  echo "✓ New password verified and working!"
                  echo ""
                  echo "Flushing privileges..."
                  mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" -e "FLUSH PRIVILEGES;" || echo "Warning: Could not flush privileges"
                  echo ""
                  echo "=========================================="
                  echo "Rotation completed successfully"
                  echo "=========================================="
                  exit 0
                else
                  echo "✗ ERROR: New password doesn't work!"
                  exit 1
                fi
              else
                echo "Old password doesn't work - checking if new password already active..."
                if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$NEW_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then
                  echo "✓ New password is already active in MySQL"
                  echo "Rotation was already completed"
                  exit 0
                else
                  echo "✗ ERROR: Neither old nor new password works!"
                  echo "Manual intervention required"
                  echo ""
                  echo "Attempting to reset using root credentials..."
                  if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \
                    -e "ALTER USER '$MYSQL_USER'@'%' IDENTIFIED BY '$NEW_PASSWORD';"; then
                    echo "✓ Password reset successfully using root"
                    exit 0
                  else
                    echo "✗ Failed to reset password"
                    exit 1
                  fi
                fi
              fi
 ---
 # Example MySQL StatefulSet (simplified)
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
  name: mysql
  namespace: mysql
 spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-root-secret
              key: password
        - name: MYSQL_DATABASE
          value: "app_db"
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        livenessProbe:
          exec:
            command:
            - mysqladmin
            - ping
            - -h
            - localhost
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - mysql
            - -h
            - localhost
            - -e
            - SELECT 1
          initialDelaySeconds: 5
          periodSeconds: 2
      # InitContainer to create app user on first run
      initContainers:
      - name: init-db-user
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-root-secret
              key: password
        - name: APP_USER
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: username
        - name: APP_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: password
        command:
        - sh
        - -c
        - |
          # Wait for MySQL to be ready
          until mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SELECT 1" > /dev/null 2>&1; do
            echo "Waiting for MySQL..."
            sleep 2
          done
          # Create user if not exists
          mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" <<EOF
          CREATE USER IF NOT EXISTS '$APP_USER'@'%' IDENTIFIED BY '$APP_PASSWORD';
          GRANT ALL PRIVILEGES ON app_db.* TO '$APP_USER'@'%';
          FLUSH PRIVILEGES;
          EOF
          echo "User $APP_USER created/updated"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
 ---
 # Service for MySQL
 apiVersion: v1
 kind: Service
 metadata:
  name: mysql
  namespace: mysql
 spec:
  selector:
    app: mysql
  ports:
  - port: 3306
    targetPort: 3306
    name: mysql
  clusterIP: None  # Headless service
 ---
 # Example application using MySQL
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: app
  namespace: mysql
 spec:
  replicas: 2
  selector:
    matchLabels:
      app: mysql-client
  template:
    metadata:
      labels:
        app: mysql-client
      annotations:
        # Reloader restarts app when credentials change
        reloader.stakater.com/auto: "true"
    spec:
      containers:
      - name: app
        image: mysql:8.0
        env:
        - name: MYSQL_HOST
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: host
        - name: MYSQL_PORT
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: port
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: username
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: password
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: mysql-app-secret
              key: database
        command:
        - sh
        - -c
        - |
          # Your application code here
          while true; do
            mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$MYSQL_PASSWORD" \
              -D"$MYSQL_DATABASE" -e "SELECT NOW();" || echo "Connection failed"
            sleep 30
          done
 ---
 # How the rotation flow works:
 #
 # Initial Setup:
 #   - mysql-app-secret created with initial password
 #   - InitContainer creates MySQL user with this password
 #   - Application connects successfully
 #
 # Day 90: Rotation
 #   1. ManagedSecret operator:
 #      - Generates NEW password
 #      - Creates mysql-app-secret-previous (OLD password)
 #      - Updates mysql-app-secret (NEW password)
 #
 #   2. CronJob (within 5 minutes):
 #      - Detects previous secret exists
 #      - Authenticates to MySQL with OLD password (or as root)
 #      - Runs ALTER USER to set NEW password
 #      - Verifies NEW password works
 #
 #   3. Reloader:
 #      - Detects mysql-app-secret change
 #      - Restarts application pods
 #      - Pods reconnect with NEW password
 #
 #   4. Cleanup (after 1 hour):
 #      - mysql-app-secret-previous deleted automatically
 ---
 # Testing commands:
 #
 # # Check current credentials
 # kubectl get secret mysql-app-secret -n mysql -o yaml
 #
 # # Test connection with current credentials
 # MYSQL_PASS=$(kubectl get secret mysql-app-secret -n mysql -o jsonpath='{.data.password}' | base64 -d)
 # kubectl run -it --rm mysql-test --image=mysql:8.0 --restart=Never -- \
 #   mysql -h mysql.mysql.svc.cluster.local -u app_user -p"$MYSQL_PASS" -e "SELECT NOW();"
 #
 # # Force rotation
 # kubectl annotate managedsecret mysql-app-user -n mysql reconcile="$(date +%s)" --overwrite
 #
 # # Watch rotation
 # kubectl get secrets -n mysql -w
 # kubectl get jobs -n mysql -w
 #
 # # View rotation logs
 # kubectl logs -n mysql -l job-name=mysql-password-rotator-XXXXX
 ---
 # Important MySQL-specific notes:
 #
 # 1. User Host Specification:
 #    - '%' allows connection from any host
 #    - Consider restricting to specific pods/namespaces
 #    - Example: 'app_user'@'%.mysql.svc.cluster.local'
 #
 # 2. Authentication Methods:
 #    - MySQL 8.0+ uses caching_sha2_password by default
 #    - Older apps may need mysql_native_password
 #    - Specify during user creation if needed
 #
 # 3. Privilege Management:
 #    - Rotation only changes password, not privileges
 #    - Manage privileges separately via root access
 #
 # 4. Connection Pooling:
 #    - Application connection pools need to be refreshed
 #    - Reloader restart handles this automatically
 #    - Consider graceful shutdown for long-running transactions
--- a/examples/example-6-comprehensive-multi-service.yaml
+++ b/examples/example-6-comprehensive-multi-service.yaml
@@ -0,0 +1,572 @@
 # Example 6: Comprehensive Multi-Service Deployment
 #
 # This example demonstrates a complete application stack with:
 # - PostgreSQL (rotating with previous version)
 # - Redis (rotating without previous version)
 # - External API key (no rotation)
 # - TLS certificates (no rotation)
 # - Application credentials (rotating with Reloader)
 #
 # Shows different rotation strategies working together
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: myapp
 ---
 # ============================================================================
 # 1. PostgreSQL Credentials - Rotates with previous version support
 # ============================================================================
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: postgres-credentials
  namespace: myapp
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: myapp/postgres
  fields:
  - name: username
    type: static
    value: "myapp_user"
  - name: password
    type: generated
    generator:
      type: password
      length: 32
      minDigits: 6
      minSymbols: 6
  - name: host
    type: static
    value: "postgres.database.svc.cluster.local"
  - name: port
    type: static
    value: "5432"
  - name: database
    type: static
    value: "myapp_db"
  destination:
    name: postgres-creds
    type: Opaque
    keepPreviousVersion: true  # Needs old password for rotation
    previousVersionTTL: 1h
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
 ---
 # ============================================================================
 # 2. Redis Password - Rotates without previous version (CONFIG SET)
 # ============================================================================
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: redis-credentials
  namespace: myapp
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: myapp/redis
  fields:
  - name: password
    type: generated
    generator:
      type: password
      length: 40
      minDigits: 8
      minSymbols: 0  # Redis doesn't like special chars
      symbolCharacters: ""
  - name: host
    type: static
    value: "redis.myapp.svc.cluster.local"
  - name: port
    type: static
    value: "6379"
  destination:
    name: redis-creds
    type: Opaque
    # No previous version needed - Redis CONFIG SET doesn't require auth
  rotation:
    enabled: true
    schedule: 720h  # 30 days
 ---
 # ============================================================================
 # 3. External API Keys - No rotation
 # ============================================================================
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: external-api-keys
  namespace: myapp
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: myapp/api-keys
  fields:
  - name: stripe-api-key
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 0
      minSymbols: 0
  - name: sendgrid-api-key
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 0
      minSymbols: 0
  destination:
    name: api-keys
    type: Opaque
  rotation:
    enabled: false  # Managed externally
 ---
 # ============================================================================
 # 4. TLS Certificate - No rotation (managed by cert-manager)
 # ============================================================================
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: app-tls-cert
  namespace: myapp
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: myapp/tls
  fields:
  - name: tls.crt
    type: static
    value: |
      -----BEGIN CERTIFICATE-----
      # Your certificate here
      -----END CERTIFICATE-----
  - name: tls.key
    type: static
    value: |
      -----BEGIN PRIVATE KEY-----
      # Your private key here
      -----END PRIVATE KEY-----
  destination:
    name: app-tls
    type: kubernetes.io/tls
  rotation:
    enabled: false
 ---
 # ============================================================================
 # 5. Application JWT Secret - Rotates with Reloader
 # ============================================================================
 apiVersion: secrets.c5ai.ch/v1alpha1
 kind: ManagedSecret
 metadata:
  name: app-jwt-secret
  namespace: myapp
 spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: myapp/jwt
  fields:
  - name: jwt-secret
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 16
      minSymbols: 0
      symbolCharacters: ""
  destination:
    name: jwt-secret
    type: Opaque
  rotation:
    enabled: true
    schedule: 4320h  # 180 days
 ---
 # ============================================================================
 # PostgreSQL Password Rotation Job
 # ============================================================================
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: postgres-password-rotator
  namespace: myapp
 spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: rotate
            image: postgres:15-alpine
            env:
            - name: PGUSER
              valueFrom:
                secretKeyRef:
                  name: postgres-creds
                  key: username
            - name: NEW_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-creds
                  key: password
            - name: OLD_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-creds-previous
                  key: password
                  optional: true
            - name: PGHOST
              valueFrom:
                secretKeyRef:
                  name: postgres-creds
                  key: host
            command:
            - sh
            - -c
            - |
              if [ -n "$OLD_PASSWORD" ] && [ "$OLD_PASSWORD" != "$NEW_PASSWORD" ]; then
                echo "Rotating PostgreSQL password..."
                PGPASSWORD=$OLD_PASSWORD psql -c "ALTER USER $PGUSER PASSWORD '$NEW_PASSWORD';"
                echo "Rotation complete!"
              fi
 ---
 # ============================================================================
 # Redis Password Rotation Job
 # ============================================================================
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: redis-password-rotator
  namespace: myapp
 spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: rotate
            image: redis:7-alpine
            env:
            - name: REDIS_HOST
              valueFrom:
                secretKeyRef:
                  name: redis-creds
                  key: host
            - name: REDIS_PORT
              valueFrom:
                secretKeyRef:
                  name: redis-creds
                  key: port
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: redis-creds
                  key: password
            command:
            - sh
            - -c
            - |
              # Redis CONFIG SET doesn't require current password
              redis-cli -h $REDIS_HOST -p $REDIS_PORT CONFIG SET requirepass "$REDIS_PASSWORD"
              echo "Redis password updated"
 ---
 # ============================================================================
 # Main Application Deployment
 # ============================================================================
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: myapp
  namespace: myapp
 spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        # Reloader restarts pods when ANY of these secrets change
        reloader.stakater.com/auto: "true"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080
          name: http
        env:
        # PostgreSQL connection
        - name: POSTGRES_HOST
          valueFrom:
            secretKeyRef:
              name: postgres-creds
              key: host
        - name: POSTGRES_PORT
          valueFrom:
            secretKeyRef:
              name: postgres-creds
              key: port
        - name: POSTGRES_DB
          valueFrom:
            secretKeyRef:
              name: postgres-creds
              key: database
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: postgres-creds
              key: username
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-creds
              key: password
        # Redis connection
        - name: REDIS_HOST
          valueFrom:
            secretKeyRef:
              name: redis-creds
              key: host
        - name: REDIS_PORT
          valueFrom:
            secretKeyRef:
              name: redis-creds
              key: port
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-creds
              key: password
        # API Keys
        - name: STRIPE_API_KEY
          valueFrom:
            secretKeyRef:
              name: api-keys
              key: stripe-api-key
        - name: SENDGRID_API_KEY
          valueFrom:
            secretKeyRef:
              name: api-keys
              key: sendgrid-api-key
        # JWT Secret
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: jwt-secret
              key: jwt-secret
        volumeMounts:
        - name: tls
          mountPath: /etc/tls
          readOnly: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
      volumes:
      - name: tls
        secret:
          secretName: app-tls
 ---
 # ============================================================================
 # Service
 # ============================================================================
 apiVersion: v1
 kind: Service
 metadata:
  name: myapp
  namespace: myapp
 spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
    name: http
 ---
 # ============================================================================
 # Ingress with TLS
 # ============================================================================
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: myapp
  namespace: myapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
 spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.c5ai.ch
    secretName: myapp-tls-ingress
  rules:
  - host: myapp.c5ai.ch
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
 ---
 # ============================================================================
 # Summary of Rotation Strategies
 # ============================================================================
 #
 # 1. PostgreSQL (postgres-creds):
 #    - Rotates every 90 days
 #    - keepPreviousVersion: true
 #    - CronJob uses old password to authenticate and change
 #    - Reloader restarts app pods after rotation
 #
 # 2. Redis (redis-creds):
 #    - Rotates every 30 days
 #    - No previous version needed (CONFIG SET doesn't require auth)
 #    - CronJob updates password directly
 #    - Reloader restarts app pods after rotation
 #
 # 3. API Keys (api-keys):
 #    - No automatic rotation
 #    - Generated once, managed externally
 #    - Update via kubectl or Vault UI when provider rotates keys
 #
 # 4. TLS Certificates (app-tls):
 #    - No rotation
 #    - Managed by cert-manager or external process
 #    - Static content stored in Vault
 #
 # 5. JWT Secret (jwt-secret):
 #    - Rotates every 180 days
 #    - Simple rotation - just restart with new secret
 #    - Reloader handles pod restart
 #
 # ============================================================================
 # Timeline Example: What Happens on Day 90
 # ============================================================================
 #
 # T+0: ManagedSecret operator triggers rotation
 #   - Generates new passwords for postgres-creds and redis-creds
 #   - Updates Vault
 #   - Creates postgres-creds-previous
 #   - Updates postgres-creds
 #   - Updates redis-creds (no previous needed)
 #
 # T+1min: CronJobs detect changes
 #   - Postgres rotator: Uses old password to set new password in DB
 #   - Redis rotator: Sets new password via CONFIG SET
 #
 # T+2min: Reloader detects secret changes
 #   - Triggers rolling restart of myapp Deployment
 #   - Pods start with new credentials
 #
 # T+1hour: Cleanup
 #   - postgres-creds-previous automatically deleted
 #   - Old credentials no longer accessible
 #
 # ============================================================================
 # Monitoring Commands
 # ============================================================================
 #
 # # Watch all secrets
 # kubectl get secrets -n myapp -w
 #
 # # Check rotation status
 # kubectl get managedsecrets -n myapp
 #
 # # View rotation logs
 # kubectl logs -n myapp -l job-name=postgres-password-rotator-XXXXX
 # kubectl logs -n myapp -l job-name=redis-password-rotator-XXXXX
 #
 # # Check Reloader activity
 # kubectl logs -n reloader -l app=reloader | grep myapp
 #
 # # Verify app pods restarted
 # kubectl get pods -n myapp -o wide
 #
 # # Test connections with new credentials
 # kubectl exec -it -n myapp deployment/myapp -- sh
 #   # psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB
 #   # redis-cli -h $REDIS_HOST -a $REDIS_PASSWORD ping
 #
 # ============================================================================
 # Force Rotation for Testing
 # ============================================================================
 #
 # # Force PostgreSQL rotation
 # kubectl annotate managedsecret postgres-credentials -n myapp \
 #   reconcile="$(date +%s)" --overwrite
 #
 # # Force Redis rotation
 # kubectl annotate managedsecret redis-credentials -n myapp \
 #   reconcile="$(date +%s)" --overwrite
 #
 # # Force all rotations
 # kubectl annotate managedsecrets -n myapp --all \
 #   reconcile="$(date +%s)" --overwrite
 #
 # # Watch the cascade of events
 # kubectl get events -n myapp --sort-by='.lastTimestamp' -w