# ManagedSecret Operator - Examples Summary This directory contains comprehensive examples for the ManagedSecret Operator, demonstrating various rotation strategies and use cases. ## Files Overview ### Documentation - **README.md** - Complete installation and usage guide with new features - Added `keepPreviousVersion` and `previousVersionTTL` documentation - Added Reloader integration guide - Updated best practices and architecture diagrams ### Example Files #### Example 1: No Rotation (Simple API Key) **File:** `example-1-no-rotation.yaml` **Use Case:** Long-lived API keys that don't need rotation **Key Features:** - Generated once and kept indefinitely - No Reloader needed (secret never changes) - Perfect for external service API keys **When to Use:** - Third-party API keys (Stripe, SendGrid, etc.) - Service identifiers - Static configuration values --- #### Example 2: Simple Rotation with Reloader (Container Registry) **File:** `example-2-registry-rotation-simple.yaml` **Use Case:** Docker/Container registry with HTTP basic auth **Key Features:** - Password rotates every 90 days - InitContainer regenerates htpasswd on pod restart - Reloader automatically restarts pods - **No `keepPreviousVersion` needed** - htpasswd is regenerated fresh **Rotation Flow:** 1. ManagedSecret generates new password → Updates Vault → Updates K8s Secret 2. Reloader detects Secret change → Triggers pod restart 3. InitContainer runs → Reads new password → Generates fresh htpasswd 4. Registry starts with new htpasswd **When to Use:** - Container registries (Docker Registry, Harbor) - Web servers with htpasswd (Apache, Nginx) - Any service where credentials are regenerated at startup --- #### Example 3: Complex Rotation with Previous Password (PostgreSQL) **File:** `example-3-postgres-rotation-previous.yaml` **Use Case:** PostgreSQL user password that requires authentication to change **Key Features:** - Password rotates every 90 days - `keepPreviousVersion: true` creates `-previous` secret - CronJob uses old password to authenticate, sets new password - `previousVersionTTL: 1h` cleans up after grace period **Rotation Flow:** 1. ManagedSecret creates both `postgres-secret` and `postgres-secret-previous` 2. CronJob (every 5 min) detects rotation 3. Authenticates with OLD password: `PGPASSWORD=$OLD_PASSWORD psql` 4. Changes password: `ALTER USER app_user PASSWORD '$NEW_PASSWORD'` 5. Verifies new password works 6. After 1 hour, `-previous` secret is deleted **When to Use:** - PostgreSQL, MySQL, MariaDB - Any database requiring authentication to change passwords - Services with `ALTER USER` or similar commands --- #### Example 4: MinIO Admin Password Rotation **File:** `example-4-minio-rotation-previous.yaml` **Use Case:** MinIO admin credentials with rotation **Key Features:** - Password rotates every 60 days - Uses MinIO Client (mc) to change password - `keepPreviousVersion: true` with 2h TTL - CronJob handles rotation via `mc admin user password` **Special Considerations:** - MinIO does NOT reload credentials from environment variables - Manual pod restart required after rotation - Consider using a sidecar to monitor and restart **Rotation Flow:** 1. ManagedSecret creates both secrets 2. CronJob authenticates with old credentials: `mc alias set oldminio` 3. Changes password: `mc admin user password myminio admin $NEW_PASSWORD` 4. Verifies new credentials 5. **Important:** MinIO pods must be restarted manually **When to Use:** - MinIO object storage - Any service that doesn't auto-reload credentials - Services requiring separate admin tools for password changes --- #### Example 5: MySQL User Password Rotation **File:** `example-5-mysql-rotation-previous.yaml` **Use Case:** MySQL application user with automated rotation **Key Features:** - Password rotates every 90 days - Multiple password change methods (ALTER USER, SET PASSWORD) - Root credentials as fallback - `keepPreviousVersion: true` with 1h TTL **Rotation Flow:** 1. ManagedSecret creates both secrets 2. CronJob tries multiple approaches: - Method 1: `ALTER USER` as root (preferred) - Method 2: `SET PASSWORD` as root - Method 3: `SET PASSWORD` as user with old password 3. Flushes privileges 4. Verifies new password **When to Use:** - MySQL 5.7+, MySQL 8.0 - MariaDB - Any MySQL-compatible database --- #### Example 6: Comprehensive Multi-Service Deployment **File:** `example-6-comprehensive-multi-service.yaml` **Use Case:** Complete application stack showing all strategies together **Components:** 1. **PostgreSQL** - Rotates with previous version (90 days) 2. **Redis** - Rotates without previous version (30 days) 3. **API Keys** - No rotation (managed externally) 4. **TLS Certificates** - No rotation (cert-manager) 5. **JWT Secret** - Simple rotation with Reloader (180 days) **Demonstrates:** - Multiple secrets with different rotation schedules - Mixed rotation strategies in one application - How Reloader restarts pods for all credential changes - Real-world production setup **When to Use:** - As a template for complex applications - To understand how different strategies work together - For learning all features in one example --- ## Quick Decision Tree ### Does your service need the OLD password to set the NEW password? **YES** (PostgreSQL, MySQL, MinIO, LDAP): - Use `keepPreviousVersion: true` - Set appropriate `previousVersionTTL` - Create CronJob to handle rotation - See Examples 3, 4, or 5 **NO** (Registry, Redis, Web servers): - Use simple rotation with Reloader - InitContainer or startup script regenerates credentials - No CronJob needed - See Example 2 **NOT ROTATING** (API keys, Certificates): - Set `rotation.enabled: false` - No Reloader annotation needed - See Example 1 --- ## Rotation Schedule Recommendations - **High Security** (User passwords, admin credentials): 30-90 days - **Medium Security** (Application credentials, service accounts): 90-180 days - **Low Security** (Read-only API keys, development): 180+ days or disabled - **Previous Version TTL**: - Automated rotation: 1h - Manual intervention: 2-4h - Complex workflows: 24h --- ## Prerequisites for All Examples 1. **OpenBao/Vault** deployed and initialized 2. **ManagedSecret Operator** installed 3. **Reloader** installed (for rotation examples) 4. **cert-manager** (optional, for TLS examples) ## Installation Order ```bash # 1. Install OpenBao kubectl create namespace openbao helm install openbao openbao/openbao -n openbao # 2. Install ManagedSecret Operator kubectl apply -f managedsecret-operator/ # 3. Install Reloader kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml # 4. Deploy your application with examples kubectl apply -f example-X-your-usecase.yaml ``` --- ## Testing Rotation ### Force Immediate Rotation ```bash # Trigger rotation for any ManagedSecret kubectl annotate managedsecret -n \ reconcile="$(date +%s)" --overwrite ``` ### Watch the Flow ```bash # Terminal 1: Watch secrets kubectl get secrets -n -w # Terminal 2: Watch pods kubectl get pods -n -w # Terminal 3: Watch jobs (for CronJobs) kubectl get jobs -n -w # Terminal 4: Reloader logs kubectl logs -n reloader -l app=reloader -f ``` ### Verify Rotation Completed ```bash # Check if -previous secret exists (should only exist during grace period) kubectl get secret -previous -n # View rotation job logs kubectl logs -n -l job-name= # Test connection with new credentials kubectl exec -it -n -- ``` --- ## Common Issues and Solutions ### Issue: Pod not restarting after rotation **Solution:** Check Reloader is installed and annotation is correct ```bash kubectl get pods -n reloader kubectl get deployment -o yaml | grep reloader ``` ### Issue: Rotation job failing **Solution:** Check if previous secret exists and credentials work ```bash kubectl get secret -previous -n kubectl logs -n -l job-name= ``` ### Issue: Application still using old credentials **Solution:** Verify pod actually restarted after secret change ```bash kubectl get pods -n -o wide # Check pod start time vs secret update time ``` ### Issue: Previous secret not cleaned up **Solution:** Check operator logs for cleanup job ```bash kubectl logs -n managedsecret-operator-system -l control-plane=controller-manager | grep cleanup ``` --- ## Best Practices Summary 1. ✅ **Always use Reloader** with rotating secrets 2. ✅ **Use `keepPreviousVersion`** for services requiring old password 3. ✅ **Set realistic `previousVersionTTL`** based on your workflow 4. ✅ **Test rotation** before production deployment 5. ✅ **Monitor rotation jobs** for failures 6. ✅ **Document rotation procedures** for your team 7. ✅ **Use appropriate rotation schedules** based on security requirements 8. ✅ **Backup Vault regularly** - it's your source of truth 9. ✅ **Never modify K8s Secrets manually** - always use ManagedSecret or Vault 10. ✅ **Set up alerts** for failed rotations --- ## Support and Contributing For issues or questions about these examples: - Check the main README.md for troubleshooting - Review operator logs for detailed error messages - Test in development environment before production Remember: Vault/OpenBao is ALWAYS the source of truth. All changes should go through ManagedSecret resources or directly in Vault, never by manually editing Kubernetes Secrets.