commit 1bc8aadb851dbc8b180c47cfc5f316f24ef532dd Author: admin Date: Sun Oct 26 14:53:01 2025 +0100 Initial public release v1.0.7 - Streamlined README focused on quick start - Complete examples for all major use cases - Decision tree for choosing right pattern - Comprehensive troubleshooting guide diff --git a/INDEX.md b/INDEX.md new file mode 100644 index 0000000..3a1d61f --- /dev/null +++ b/INDEX.md @@ -0,0 +1,112 @@ +# ManagedSecret Operator - Documentation Index + +Quick navigation for the ManagedSecret Operator documentation and examples. + +## 📖 Main Documentation + +**[README.md](./README.md)** - Start here! +- Installation guide +- Quick start +- Basic usage examples +- Troubleshooting +- Best practices + +## 📚 Examples + +**[examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md)** - Overview of all examples +- Use case descriptions +- Decision tree for choosing the right pattern +- Quick comparison table + +### Individual Examples + +1. **[example-1-no-rotation.yaml](./examples/example-1-no-rotation.yaml)** + - Simple API key without rotation + - Perfect for: Third-party API keys, static service identifiers + +2. **[example-2-registry-rotation-simple.yaml](./examples/example-2-registry-rotation-simple.yaml)** + - Container registry with basic rotation + - Perfect for: Docker Registry, Harbor, web servers with htpasswd + +3. **[example-3-postgres-rotation-previous.yaml](./examples/example-3-postgres-rotation-previous.yaml)** + - PostgreSQL with zero-downtime rotation + - Perfect for: PostgreSQL, any service needing old password to authenticate + +4. **[example-4-minio-rotation-previous.yaml](./examples/example-4-minio-rotation-previous.yaml)** + - MinIO object storage with rotation + - Perfect for: MinIO, services not auto-reloading credentials + +5. **[example-5-mysql-rotation-previous.yaml](./examples/example-5-mysql-rotation-previous.yaml)** + - MySQL with multiple rotation strategies + - Perfect for: MySQL, MariaDB, MySQL-compatible databases + +6. **[example-6-comprehensive-multi-service.yaml](./examples/example-6-comprehensive-multi-service.yaml)** + - Complete application stack + - Perfect for: Understanding how different strategies work together + +## 🎯 Quick Links by Use Case + +### I need to store a secret that never changes +→ [Example 1: No Rotation](./examples/example-1-no-rotation.yaml) + +### I need to rotate credentials automatically +→ [Example 2: Simple Rotation](./examples/example-2-registry-rotation-simple.yaml) + +### My service needs the old password to change passwords +→ [Example 3: PostgreSQL](./examples/example-3-postgres-rotation-previous.yaml) +→ [Example 4: MinIO](./examples/example-4-minio-rotation-previous.yaml) +→ [Example 5: MySQL](./examples/example-5-mysql-rotation-previous.yaml) + +### I have a complex multi-service application +→ [Example 6: Multi-Service](./examples/example-6-comprehensive-multi-service.yaml) + +## 🚀 Getting Started Path + +1. Read [README.md](./README.md) - Installation & Quick Start +2. Review [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md) - Understand patterns +3. Pick the example closest to your use case +4. Adapt and deploy! + +## 🔧 Common Tasks + +### Install the operator +→ [README.md - Installation Steps](./README.md#installation-steps) + +### Create your first secret +→ [README.md - Usage](./README.md#usage) + +### Understand rotation +→ [README.md - Secret Rotation](./README.md#secret-rotation) + +### Troubleshoot issues +→ [README.md - Troubleshooting](./README.md#troubleshooting) + +### Set up for production +→ [README.md - Best Practices](./README.md#best-practices) + +## 📦 Repository Structure + +``` +managedsecret-operator/ +├── README.md # Main documentation +├── INDEX.md # This file +└── examples/ + ├── EXAMPLES-SUMMARY.md # Examples overview + ├── example-1-no-rotation.yaml + ├── example-2-registry-rotation-simple.yaml + ├── example-3-postgres-rotation-previous.yaml + ├── example-4-minio-rotation-previous.yaml + ├── example-5-mysql-rotation-previous.yaml + └── example-6-comprehensive-multi-service.yaml +``` + +## 📞 Need Help? + +1. Check [README.md - Troubleshooting](./README.md#troubleshooting) +2. Review the examples for similar use cases +3. Contact your administrator for registry/Vault setup issues + +--- + +**Version:** 1.0.7 +**Registry:** registry.c5ai.ch/charts/managedsecret-operator \ No newline at end of file diff --git a/README-PUBLIC.md b/README-PUBLIC.md new file mode 100644 index 0000000..127f8db --- /dev/null +++ b/README-PUBLIC.md @@ -0,0 +1,711 @@ +# ManagedSecret Operator + +A Kubernetes operator for declarative secret management with automated rotation, drift detection, and zero-downtime password changes. + +## Overview + +The ManagedSecret operator provides: + +- 🔐 **Generates secrets** with configurable password policies +- 🏦 **Stores in Vault/OpenBao** as the authoritative source +- 🔄 **Syncs to Kubernetes Secrets** for application consumption +- 🔍 **Detects and fixes drift** automatically (Vault → K8s) +- 🔁 **Rotates secrets** on a configurable schedule +- ⏱️ **Preserves previous versions** during rotation for zero-downtime password changes + +### Key Principle: Vault/OpenBao is ALWAYS the source of truth + +- Changes in Vault → Synced to K8s automatically +- Changes in K8s → Overwritten by Vault values +- Drift detection runs every 1 minute + +--- + +## 🚀 Quick Start + +> **Prerequisites:** Contact your administrator to receive: +> - Registry username and password for `registry.c5ai.ch` +> - Confirmation that OpenBao/Vault is configured for your cluster + +```bash +# 1. Set your credentials (provided by administrator) +export REGISTRY_USER="" +export REGISTRY_PASS="" + +# 2. Install Reloader (for automatic pod restarts during rotation) +kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml + +# 3. Login to Helm registry +helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS + +# 4. Create imagePullSecret +kubectl create namespace managedsecret-operator-system +kubectl create secret docker-registry registry-pull-secret \ + --docker-server=registry.c5ai.ch \ + --docker-username=$REGISTRY_USER \ + --docker-password=$REGISTRY_PASS \ + --namespace managedsecret-operator-system + +# 5. Install operator via Helm +helm install managedsecret-operator oci://registry.c5ai.ch/charts/managedsecret-operator \ + --version 1.0.7 \ + --namespace managedsecret-operator-system \ + --set imagePullSecrets[0].name=registry-pull-secret + +# 6. Verify installation +kubectl get pods -n managedsecret-operator-system +``` + +That's it! Now check the [examples](./examples/) folder for comprehensive usage scenarios. + +--- + +## 📦 What You Get + +### Helm Chart from registry.c5ai.ch + +The operator is distributed as a Helm chart: + +- ✅ **No building required** - Just `helm install` +- ✅ **Versioned releases** - Use specific versions or upgrade easily +- ✅ **Pre-configured** - Sensible defaults, customize via `values.yaml` +- ✅ **Private & secure** - Hosted on your infrastructure + +### What's Deployed + +``` +managedsecret-operator-system/ +├── CustomResourceDefinition (ManagedSecret) +├── ServiceAccount (controller-manager) +├── Role & RoleBinding (RBAC) +├── ClusterRole & ClusterRoleBinding +├── Deployment (controller-manager) +└── ConfigMap (operator configuration) +``` + +--- + +## Prerequisites + +### On Your Side (User) +- ✅ Kubernetes cluster (1.24+) +- ✅ kubectl configured and working +- ✅ Helm 3.x installed +- ✅ Cluster admin permissions + +### Provided by Administrator +- ✅ **Registry credentials** for `registry.c5ai.ch` +- ✅ **OpenBao/Vault configuration** for your cluster + - Vault address (e.g., `http://openbao.openbao.svc.cluster.local:8200`) + - Kubernetes auth configured + - Vault role name (typically `managedsecret-operator`) + +### Optional (Recommended) +- ✅ **Reloader by Stakater** (for automatic pod restarts on rotation) + +--- + +## Installation Steps + +### Step 1: Install Reloader (Recommended) + +Reloader automatically restarts pods when secrets change: + +```bash +# Direct install (simplest) +kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml + +# Via Helm +helm repo add stakater https://stakater.github.io/stakater-charts +helm install reloader stakater/reloader \ + --namespace reloader \ + --create-namespace +``` + +### Step 2: Authenticate to Registry + +```bash +# Set credentials from administrator +export REGISTRY_USER="" +export REGISTRY_PASS="" + +# Login to Helm registry +helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS + +# Create namespace +kubectl create namespace managedsecret-operator-system + +# Create imagePullSecret +kubectl create secret docker-registry registry-pull-secret \ + --docker-server=registry.c5ai.ch \ + --docker-username=$REGISTRY_USER \ + --docker-password=$REGISTRY_PASS \ + --namespace managedsecret-operator-system +``` + +### Step 3: Install the Operator + +```bash +# Install with default configuration +helm install managedsecret-operator \ + oci://registry.c5ai.ch/charts/managedsecret-operator \ + --version 1.0.7 \ + --namespace managedsecret-operator-system \ + --set imagePullSecrets[0].name=registry-pull-secret + +# Or with custom values +cat > values.yaml <-previous with old values + 4. Updates K8s Secret with new values + 5. If Reloader is enabled: + - Reloader detects change → Restarts pods + 6. After previousVersionTTL: + - Operator deletes -previous secret +``` + +### When to Use `keepPreviousVersion` + +Use `keepPreviousVersion: true` when your service needs the **old password to authenticate** before changing to the new password: + +- ✅ PostgreSQL, MySQL, MariaDB (`ALTER USER` requires authentication) +- ✅ MinIO (`mc admin` needs old credentials) +- ✅ LDAP, Active Directory +- ❌ Container registries (htpasswd regenerated fresh) +- ❌ Redis (CONFIG SET doesn't need old password) +- ❌ Web services with startup scripts + +### Forcing Rotation + +```bash +# Trigger immediate rotation +kubectl annotate managedsecret -n \ + reconcile="$(date +%s)" --overwrite +``` + +--- + +## 🔍 Drift Detection + +The operator automatically detects and fixes drift every 60 seconds: + +**Scenario 1: Secret deleted in Kubernetes** +```bash +kubectl delete secret my-secret -n my-app +# Operator recreates it from Vault within 60 seconds +``` + +**Scenario 2: Secret modified in Vault** +```bash +# Update value in Vault +vault kv put secret/my-app/keys password=newvalue +# Operator syncs to K8s within 60 seconds +``` + +**Scenario 3: Secret manually edited in K8s** +```bash +kubectl edit secret my-secret -n my-app +# Changes are overwritten by Vault values within 60 seconds +``` + +--- + +## 🔧 Troubleshooting + +### Operator Not Starting + +```bash +# Check pod status +kubectl get pods -n managedsecret-operator-system + +# View logs +kubectl logs -n managedsecret-operator-system \ + -l control-plane=controller-manager --tail=100 + +# Common issues: +# - ImagePullBackOff: Check imagePullSecret is created correctly +# - CrashLoopBackOff: Check Vault connectivity +``` + +### Secret Not Created + +```bash +# Check ManagedSecret status +kubectl get managedsecret -n -o yaml + +# Look for conditions/events +kubectl describe managedsecret -n + +# Check operator logs +kubectl logs -n managedsecret-operator-system \ + -l control-plane=controller-manager | grep +``` + +### Vault Authentication Failing + +```bash +# Verify ServiceAccount exists +kubectl get sa controller-manager -n managedsecret-operator-system + +# Check Vault role configuration +vault read auth/kubernetes/role/managedsecret-operator + +# Test authentication manually +kubectl exec -it -n managedsecret-operator-system \ + deployment/managedsecret-operator-controller-manager -- sh +# Inside pod: Check /var/run/secrets/kubernetes.io/serviceaccount/token exists +``` + +### Rotation Not Working + +```bash +# Check ManagedSecret schedule +kubectl get managedsecret -n -o yaml | grep schedule + +# Force rotation to test +kubectl annotate managedsecret -n \ + reconcile="$(date +%s)" --overwrite + +# Check if Reloader is installed (if using rotation) +kubectl get pods -n reloader +``` + +### Reloader Not Restarting Pods + +```bash +# Verify Reloader is running +kubectl get pods -n reloader + +# Check pod has correct annotation +kubectl get deployment -n -o yaml | grep reloader + +# View Reloader logs +kubectl logs -n reloader -l app=reloader + +# Required annotation on pod template: +# annotations: +# reloader.stakater.com/auto: "true" +``` + +--- + +## 🎯 Best Practices + +### 1. Always Use Reloader for Rotating Secrets + +Without Reloader, pods won't pick up new credentials: +```yaml +template: + metadata: + annotations: + reloader.stakater.com/auto: "true" +``` + +### 2. Set Appropriate Rotation Schedules + +- **High Security** (passwords, admin credentials): 30-90 days +- **Medium Security** (application credentials): 90-180 days +- **Low Security** (read-only API keys): 180+ days or disabled + +### 3. Use `keepPreviousVersion` Correctly + +Only use it when the service needs the old password to authenticate: +```yaml +destination: + keepPreviousVersion: true # Only if old password is needed + previousVersionTTL: 1h # Adjust based on your workflow +``` + +### 4. Test Rotation Before Production + +```bash +# Create test ManagedSecret with short rotation +spec: + rotation: + enabled: true + schedule: 5m # Test rotation every 5 minutes + +# Watch the rotation +kubectl get secrets -n -w + +# Verify application handles rotation +kubectl logs -n +``` + +### 5. Monitor Rotation Jobs + +For services using `keepPreviousVersion`, monitor your CronJobs: +```bash +# Check job status +kubectl get jobs -n + +# View job logs +kubectl logs -n -l job-name= + +# Set up alerts for failed jobs +``` + +### 6. Backup Vault Regularly + +Vault is your source of truth: +```bash +# Example: Backup with Velero +velero backup create vault-backup \ + --include-namespaces openbao \ + --snapshot-volumes +``` + +### 7. Version Control Your ManagedSecrets + +Store ManagedSecret manifests in Git: +```yaml +# managedsecrets/ +# ├── production/ +# │ ├── postgres-credentials.yaml +# │ └── api-keys.yaml +# └── staging/ +# ├── postgres-credentials.yaml +# └── api-keys.yaml +``` + +### 8. Use GitOps for Deployment + +Deploy with ArgoCD or Flux: +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: managedsecrets +spec: + source: + path: managedsecrets/production + destination: + namespace: default +``` + +--- + +## 🔒 Security Considerations + +### Production Checklist + +- [ ] OpenBao/Vault is properly secured (TLS, audit logging, backups) +- [ ] RBAC configured with least-privilege +- [ ] Network policies in place +- [ ] Monitoring and alerting configured +- [ ] Disaster recovery plan documented +- [ ] Registry credentials stored securely (not in Git) + +### Credentials Management + +The operator needs: +1. Access to Vault (via Kubernetes ServiceAccount auth) +2. Access to pull its image (via imagePullSecret) + +Both use Kubernetes-native methods. **Never hardcode credentials in manifests!** + +--- + +## 📞 Support + +### Getting Help + +1. **Check documentation**: + - This README + - [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md) + - Individual example files + +2. **Check operator logs**: + ```bash + kubectl logs -n managedsecret-operator-system \ + -l control-plane=controller-manager --tail=100 + ``` + +3. **Contact your administrator** if: + - You need registry credentials + - Vault is not configured for your cluster + - Operator authentication issues + +### For Administrators + +#### Information to Provide Users + +**Registry Credentials:** +``` +Registry: registry.c5ai.ch +Username: +Password: +Chart: oci://registry.c5ai.ch/charts/managedsecret-operator +Version: 1.0.7 +``` + +**Vault Configuration:** +``` +Address: http://openbao.openbao.svc.cluster.local:8200 +Auth Method: kubernetes +Role: managedsecret-operator +KV Mount: secret +KV Version: v2 +``` + +#### Prerequisites for New Cluster + +```bash +# 1. Enable Kubernetes auth +bao auth enable kubernetes +bao write auth/kubernetes/config \ + kubernetes_host="https://kubernetes.default.svc:443" + +# 2. Create policy +bao policy write managedsecret-operator - <<'EOF' +path "secret/data/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} +path "secret/metadata/*" { + capabilities = ["list", "read", "delete"] +} +EOF + +# 3. Create role +bao write auth/kubernetes/role/managedsecret-operator \ + bound_service_account_names=controller-manager \ + bound_service_account_namespaces=managedsecret-operator-system \ + policies=managedsecret-operator \ + ttl=1h +``` + +--- + +## 🔄 Upgrading + +### Upgrade Operator Version + +```bash +# Check current version +helm list -n managedsecret-operator-system + +# Upgrade to new version +helm upgrade managedsecret-operator \ + oci://registry.c5ai.ch/charts/managedsecret-operator \ + --version 1.0.8 \ + --namespace managedsecret-operator-system \ + --reuse-values +``` + +### Migration Notes + +When upgrading from earlier versions: +- CRD updates are handled automatically +- Existing ManagedSecrets continue to work +- No manual migration required +- Check changelog for breaking changes + +--- + +## 📝 Changelog + +### Version 1.0.7 +- ✨ Added `keepPreviousVersion` for zero-downtime rotation +- ✨ Added `previousVersionTTL` for automatic cleanup +- 📦 Helm chart deployment support +- 📚 Comprehensive examples and documentation +- 🐛 Bug fixes and improvements + +--- + +## License + +Proprietary Use License + +Permission is granted to use this software for any purpose, including commercial use, without charge. + +Modification, adaptation, or creation of derivative works is prohibited. +Redistribution of modified versions is prohibited. + +--- + +**Version:** 1.0.7 +**Registry:** registry.c5ai.ch/charts/managedsecret-operator +**Minimum Kubernetes:** 1.24+ +**Compatible with:** OpenBao, HashiCorp Vault \ No newline at end of file diff --git a/examples/EXAMPLES-SUMMARY.md b/examples/EXAMPLES-SUMMARY.md new file mode 100644 index 0000000..8c2e615 --- /dev/null +++ b/examples/EXAMPLES-SUMMARY.md @@ -0,0 +1,316 @@ +# ManagedSecret Operator - Examples Summary + +This directory contains comprehensive examples for the ManagedSecret Operator, demonstrating various rotation strategies and use cases. + +## Files Overview + +### Documentation +- **README.md** - Complete installation and usage guide with new features + - Added `keepPreviousVersion` and `previousVersionTTL` documentation + - Added Reloader integration guide + - Updated best practices and architecture diagrams + +### Example Files + +#### Example 1: No Rotation (Simple API Key) +**File:** `example-1-no-rotation.yaml` + +**Use Case:** Long-lived API keys that don't need rotation + +**Key Features:** +- Generated once and kept indefinitely +- No Reloader needed (secret never changes) +- Perfect for external service API keys + +**When to Use:** +- Third-party API keys (Stripe, SendGrid, etc.) +- Service identifiers +- Static configuration values + +--- + +#### Example 2: Simple Rotation with Reloader (Container Registry) +**File:** `example-2-registry-rotation-simple.yaml` + +**Use Case:** Docker/Container registry with HTTP basic auth + +**Key Features:** +- Password rotates every 90 days +- InitContainer regenerates htpasswd on pod restart +- Reloader automatically restarts pods +- **No `keepPreviousVersion` needed** - htpasswd is regenerated fresh + +**Rotation Flow:** +1. ManagedSecret generates new password → Updates Vault → Updates K8s Secret +2. Reloader detects Secret change → Triggers pod restart +3. InitContainer runs → Reads new password → Generates fresh htpasswd +4. Registry starts with new htpasswd + +**When to Use:** +- Container registries (Docker Registry, Harbor) +- Web servers with htpasswd (Apache, Nginx) +- Any service where credentials are regenerated at startup + +--- + +#### Example 3: Complex Rotation with Previous Password (PostgreSQL) +**File:** `example-3-postgres-rotation-previous.yaml` + +**Use Case:** PostgreSQL user password that requires authentication to change + +**Key Features:** +- Password rotates every 90 days +- `keepPreviousVersion: true` creates `-previous` secret +- CronJob uses old password to authenticate, sets new password +- `previousVersionTTL: 1h` cleans up after grace period + +**Rotation Flow:** +1. ManagedSecret creates both `postgres-secret` and `postgres-secret-previous` +2. CronJob (every 5 min) detects rotation +3. Authenticates with OLD password: `PGPASSWORD=$OLD_PASSWORD psql` +4. Changes password: `ALTER USER app_user PASSWORD '$NEW_PASSWORD'` +5. Verifies new password works +6. After 1 hour, `-previous` secret is deleted + +**When to Use:** +- PostgreSQL, MySQL, MariaDB +- Any database requiring authentication to change passwords +- Services with `ALTER USER` or similar commands + +--- + +#### Example 4: MinIO Admin Password Rotation +**File:** `example-4-minio-rotation-previous.yaml` + +**Use Case:** MinIO admin credentials with rotation + +**Key Features:** +- Password rotates every 60 days +- Uses MinIO Client (mc) to change password +- `keepPreviousVersion: true` with 2h TTL +- CronJob handles rotation via `mc admin user password` + +**Special Considerations:** +- MinIO does NOT reload credentials from environment variables +- Manual pod restart required after rotation +- Consider using a sidecar to monitor and restart + +**Rotation Flow:** +1. ManagedSecret creates both secrets +2. CronJob authenticates with old credentials: `mc alias set oldminio` +3. Changes password: `mc admin user password myminio admin $NEW_PASSWORD` +4. Verifies new credentials +5. **Important:** MinIO pods must be restarted manually + +**When to Use:** +- MinIO object storage +- Any service that doesn't auto-reload credentials +- Services requiring separate admin tools for password changes + +--- + +#### Example 5: MySQL User Password Rotation +**File:** `example-5-mysql-rotation-previous.yaml` + +**Use Case:** MySQL application user with automated rotation + +**Key Features:** +- Password rotates every 90 days +- Multiple password change methods (ALTER USER, SET PASSWORD) +- Root credentials as fallback +- `keepPreviousVersion: true` with 1h TTL + +**Rotation Flow:** +1. ManagedSecret creates both secrets +2. CronJob tries multiple approaches: + - Method 1: `ALTER USER` as root (preferred) + - Method 2: `SET PASSWORD` as root + - Method 3: `SET PASSWORD` as user with old password +3. Flushes privileges +4. Verifies new password + +**When to Use:** +- MySQL 5.7+, MySQL 8.0 +- MariaDB +- Any MySQL-compatible database + +--- + +#### Example 6: Comprehensive Multi-Service Deployment +**File:** `example-6-comprehensive-multi-service.yaml` + +**Use Case:** Complete application stack showing all strategies together + +**Components:** +1. **PostgreSQL** - Rotates with previous version (90 days) +2. **Redis** - Rotates without previous version (30 days) +3. **API Keys** - No rotation (managed externally) +4. **TLS Certificates** - No rotation (cert-manager) +5. **JWT Secret** - Simple rotation with Reloader (180 days) + +**Demonstrates:** +- Multiple secrets with different rotation schedules +- Mixed rotation strategies in one application +- How Reloader restarts pods for all credential changes +- Real-world production setup + +**When to Use:** +- As a template for complex applications +- To understand how different strategies work together +- For learning all features in one example + +--- + +## Quick Decision Tree + +### Does your service need the OLD password to set the NEW password? + +**YES** (PostgreSQL, MySQL, MinIO, LDAP): +- Use `keepPreviousVersion: true` +- Set appropriate `previousVersionTTL` +- Create CronJob to handle rotation +- See Examples 3, 4, or 5 + +**NO** (Registry, Redis, Web servers): +- Use simple rotation with Reloader +- InitContainer or startup script regenerates credentials +- No CronJob needed +- See Example 2 + +**NOT ROTATING** (API keys, Certificates): +- Set `rotation.enabled: false` +- No Reloader annotation needed +- See Example 1 + +--- + +## Rotation Schedule Recommendations + +- **High Security** (User passwords, admin credentials): 30-90 days +- **Medium Security** (Application credentials, service accounts): 90-180 days +- **Low Security** (Read-only API keys, development): 180+ days or disabled +- **Previous Version TTL**: + - Automated rotation: 1h + - Manual intervention: 2-4h + - Complex workflows: 24h + +--- + +## Prerequisites for All Examples + +1. **OpenBao/Vault** deployed and initialized +2. **ManagedSecret Operator** installed +3. **Reloader** installed (for rotation examples) +4. **cert-manager** (optional, for TLS examples) + +## Installation Order + +```bash +# 1. Install OpenBao +kubectl create namespace openbao +helm install openbao openbao/openbao -n openbao + +# 2. Install ManagedSecret Operator +kubectl apply -f managedsecret-operator/ + +# 3. Install Reloader +kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml + +# 4. Deploy your application with examples +kubectl apply -f example-X-your-usecase.yaml +``` + +--- + +## Testing Rotation + +### Force Immediate Rotation +```bash +# Trigger rotation for any ManagedSecret +kubectl annotate managedsecret -n \ + reconcile="$(date +%s)" --overwrite +``` + +### Watch the Flow +```bash +# Terminal 1: Watch secrets +kubectl get secrets -n -w + +# Terminal 2: Watch pods +kubectl get pods -n -w + +# Terminal 3: Watch jobs (for CronJobs) +kubectl get jobs -n -w + +# Terminal 4: Reloader logs +kubectl logs -n reloader -l app=reloader -f +``` + +### Verify Rotation Completed +```bash +# Check if -previous secret exists (should only exist during grace period) +kubectl get secret -previous -n + +# View rotation job logs +kubectl logs -n -l job-name= + +# Test connection with new credentials +kubectl exec -it -n -- +``` + +--- + +## Common Issues and Solutions + +### Issue: Pod not restarting after rotation +**Solution:** Check Reloader is installed and annotation is correct +```bash +kubectl get pods -n reloader +kubectl get deployment -o yaml | grep reloader +``` + +### Issue: Rotation job failing +**Solution:** Check if previous secret exists and credentials work +```bash +kubectl get secret -previous -n +kubectl logs -n -l job-name= +``` + +### Issue: Application still using old credentials +**Solution:** Verify pod actually restarted after secret change +```bash +kubectl get pods -n -o wide +# Check pod start time vs secret update time +``` + +### Issue: Previous secret not cleaned up +**Solution:** Check operator logs for cleanup job +```bash +kubectl logs -n managedsecret-operator-system -l control-plane=controller-manager | grep cleanup +``` + +--- + +## Best Practices Summary + +1. ✅ **Always use Reloader** with rotating secrets +2. ✅ **Use `keepPreviousVersion`** for services requiring old password +3. ✅ **Set realistic `previousVersionTTL`** based on your workflow +4. ✅ **Test rotation** before production deployment +5. ✅ **Monitor rotation jobs** for failures +6. ✅ **Document rotation procedures** for your team +7. ✅ **Use appropriate rotation schedules** based on security requirements +8. ✅ **Backup Vault regularly** - it's your source of truth +9. ✅ **Never modify K8s Secrets manually** - always use ManagedSecret or Vault +10. ✅ **Set up alerts** for failed rotations + +--- + +## Support and Contributing + +For issues or questions about these examples: +- Check the main README.md for troubleshooting +- Review operator logs for detailed error messages +- Test in development environment before production + +Remember: Vault/OpenBao is ALWAYS the source of truth. All changes should go through ManagedSecret resources or directly in Vault, never by manually editing Kubernetes Secrets. \ No newline at end of file diff --git a/examples/example-1-no-rotation.yaml b/examples/example-1-no-rotation.yaml new file mode 100644 index 0000000..eecc7dc --- /dev/null +++ b/examples/example-1-no-rotation.yaml @@ -0,0 +1,122 @@ +# Example 1: Simple API Key (No Rotation) +# +# Use case: Long-lived API key that doesn't need rotation +# +# Characteristics: +# - Generated once +# - No rotation schedule +# - No Reloader needed (static secret) + +--- +apiVersion: v1 +kind: Namespace +metadata: + name: api-service + +--- +apiVersion: secrets.c5ai.ch/v1alpha1 +kind: ManagedSecret +metadata: + name: api-keys + namespace: api-service +spec: + vault: + address: "http://openbao.openbao.svc.cluster.local:8200" + authMethod: kubernetes + role: managedsecret-operator + kvVersion: v2 + mount: secret + path: api/service/keys + + fields: + # Static service identifier + - name: service-id + type: static + value: "api-service-prod" + + # Generated API key (long, alphanumeric) + - name: api-key + type: generated + generator: + type: password + length: 64 + minDigits: 10 + minSymbols: 0 + minLowercase: 20 + minUppercase: 20 + symbolCharacters: "" # No symbols, just alphanumeric + allowRepeat: false + + # Static API endpoint + - name: api-endpoint + type: static + value: "https://api.external-service.com/v1" + + destination: + name: api-secret + type: Opaque + + # No rotation - generate once and keep + rotation: + enabled: false + +--- +# Example Deployment using the API key +apiVersion: apps/v1 +kind: Deployment +metadata: + name: api-consumer + namespace: api-service +spec: + replicas: 2 + selector: + matchLabels: + app: api-consumer + template: + metadata: + labels: + app: api-consumer + # No Reloader annotation needed - secret doesn't change + spec: + containers: + - name: app + image: your-app:latest + env: + - name: API_KEY + valueFrom: + secretKeyRef: + name: api-secret + key: api-key + - name: API_ENDPOINT + valueFrom: + secretKeyRef: + name: api-secret + key: api-endpoint + - name: SERVICE_ID + valueFrom: + secretKeyRef: + name: api-secret + key: service-id + +--- +# Example: Using the secret in a ConfigMap template +apiVersion: v1 +kind: ConfigMap +metadata: + name: api-config + namespace: api-service +data: + config.yaml: | + service: + name: api-consumer + # API credentials loaded from secret via environment variables + # See deployment above for how to inject API_KEY + + endpoints: + external: + timeout: 30s + retry: 3 + +--- +# How to retrieve the API key for external use +# kubectl get secret api-secret -n api-service -o jsonpath='{.data.api-key}' | base64 -d \ No newline at end of file diff --git a/examples/example-2-registry-rotation-simple.yaml b/examples/example-2-registry-rotation-simple.yaml new file mode 100644 index 0000000..4c1b58c --- /dev/null +++ b/examples/example-2-registry-rotation-simple.yaml @@ -0,0 +1,287 @@ +# Example 2: Container Registry with Rotation + Reloader +# +# Use case: Docker registry with basic auth that rotates every 90 days +# +# Characteristics: +# - Password rotates every 90 days +# - Registry doesn't need old password to accept new one +# - InitContainer regenerates htpasswd on pod restart +# - Reloader automatically restarts pod when secret changes +# - No keepPreviousVersion needed + +--- +apiVersion: v1 +kind: Namespace +metadata: + name: registry + +--- +# ManagedSecret generates and rotates registry credentials +apiVersion: secrets.c5ai.ch/v1alpha1 +kind: ManagedSecret +metadata: + name: registry-auth + namespace: registry +spec: + vault: + address: "http://openbao.openbao.svc.cluster.local:8200" + authMethod: kubernetes + role: managedsecret-operator + kvVersion: v2 + mount: secret + path: registry/auth + + fields: + - name: username + type: static + value: "admin" + + - name: password + type: generated + generator: + type: password + length: 32 + minDigits: 5 + minSymbols: 5 + minLowercase: 5 + minUppercase: 5 + symbolCharacters: "!@#$%^&*" + allowRepeat: false + + destination: + name: registry-credentials + type: Opaque + # No need for previous version - htpasswd is regenerated fresh + + rotation: + enabled: true + schedule: 2160h # 90 days + rotateGeneratedOnly: true # Only rotate password, keep username + +--- +# PersistentVolumeClaim for registry storage +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: registry-pvc + namespace: registry +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: longhorn + +--- +# Registry Deployment with Reloader integration +apiVersion: apps/v1 +kind: Deployment +metadata: + name: registry + namespace: registry + labels: + app: registry +spec: + replicas: 1 + selector: + matchLabels: + app: registry + template: + metadata: + labels: + app: registry + annotations: + # Reloader watches registry-credentials and restarts pod on change + reloader.stakater.com/auto: "true" + spec: + securityContext: + runAsNonRoot: true + runAsUser: 65534 + fsGroup: 65534 + seccompProfile: + type: RuntimeDefault + + initContainers: + # Generate htpasswd file from current credentials at pod startup + - name: generate-htpasswd + image: httpd:2.4-alpine + command: + - sh + - -c + - | + USERNAME=$(cat /credentials/username) + PASSWORD=$(cat /credentials/password) + htpasswd -Bbn "$USERNAME" "$PASSWORD" > /auth/htpasswd + chmod 644 /auth/htpasswd + echo "htpasswd generated with user: $USERNAME" + volumeMounts: + - name: credentials + mountPath: /credentials + readOnly: true + - name: auth + mountPath: /auth + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 65534 + capabilities: + drop: + - ALL + + containers: + - name: registry + image: registry:2 + ports: + - containerPort: 5000 + name: http + env: + - name: REGISTRY_STORAGE_DELETE_ENABLED + value: "true" + - name: REGISTRY_HTTP_ADDR + value: "0.0.0.0:5000" + # Enable HTTP basic auth + - name: REGISTRY_AUTH + value: "htpasswd" + - name: REGISTRY_AUTH_HTPASSWD_REALM + value: "Registry Realm" + - name: REGISTRY_AUTH_HTPASSWD_PATH + value: "/auth/htpasswd" + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 65534 + capabilities: + drop: + - ALL + seccompProfile: + type: RuntimeDefault + volumeMounts: + - name: registry-storage + mountPath: /var/lib/registry + - name: auth + mountPath: /auth + readOnly: true + livenessProbe: + httpGet: + path: / + port: 5000 + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + httpGet: + path: / + port: 5000 + initialDelaySeconds: 5 + periodSeconds: 5 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + + volumes: + - name: registry-storage + persistentVolumeClaim: + claimName: registry-pvc + - name: credentials + secret: + secretName: registry-credentials + - name: auth + emptyDir: {} + +--- +# Service for internal access +apiVersion: v1 +kind: Service +metadata: + name: registry + namespace: registry +spec: + type: ClusterIP + selector: + app: registry + ports: + - port: 5000 + targetPort: 5000 + protocol: TCP + name: http + +--- +# Ingress for external access with TLS +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: registry + namespace: registry + annotations: + cert-manager.io/cluster-issuer: letsencrypt-production + nginx.ingress.kubernetes.io/proxy-body-size: "0" + nginx.ingress.kubernetes.io/proxy-read-timeout: "600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "600" + nginx.ingress.kubernetes.io/force-ssl-redirect: "true" + nginx.ingress.kubernetes.io/backend-protocol: "HTTP" +spec: + ingressClassName: nginx + tls: + - hosts: + - registry.c5ai.ch + secretName: registry-ingress-tls + rules: + - host: registry.c5ai.ch + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: registry + port: + number: 5000 + +--- +# How the rotation flow works: +# +# 1. After 90 days, ManagedSecret operator: +# - Generates new password +# - Updates Vault +# - Updates registry-credentials Secret +# +# 2. Reloader detects Secret change: +# - Sees resourceVersion changed +# - Triggers rolling restart of registry Deployment +# +# 3. Pod restart with new credentials: +# - InitContainer runs +# - Reads NEW password from registry-credentials +# - Generates fresh htpasswd file +# - Registry container starts with new htpasswd +# +# 4. Result: +# - Old clients with old password: FAIL (401 Unauthorized) +# - New clients with new password: SUCCESS +# - Downtime: ~1-2 minutes during rolling restart + +--- +# Testing commands: +# +# # Get current credentials +# kubectl get secret registry-credentials -n registry -o jsonpath='{.data.username}' | base64 -d +# kubectl get secret registry-credentials -n registry -o jsonpath='{.data.password}' | base64 -d +# +# # Test without auth (should fail) +# curl -i https://registry.c5ai.ch/v2/ +# +# # Test with auth (should succeed) +# REGISTRY_USER=$(kubectl get secret registry-credentials -n registry -o jsonpath='{.data.username}' | base64 -d) +# REGISTRY_PASS=$(kubectl get secret registry-credentials -n registry -o jsonpath='{.data.password}' | base64 -d) +# curl -u $REGISTRY_USER:$REGISTRY_PASS https://registry.c5ai.ch/v2/_catalog +# +# # Docker login +# docker login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS +# +# # Force rotation test +# kubectl annotate managedsecret registry-auth -n registry reconcile="$(date +%s)" --overwrite +# kubectl get pods -n registry -w # Watch pod restart \ No newline at end of file diff --git a/examples/example-3-postgres-rotation-previous.yaml b/examples/example-3-postgres-rotation-previous.yaml new file mode 100644 index 0000000..8ac39e0 --- /dev/null +++ b/examples/example-3-postgres-rotation-previous.yaml @@ -0,0 +1,369 @@ +# Example 3: PostgreSQL Password Rotation with Previous Version +# +# Use case: PostgreSQL user password that rotates every 90 days +# +# Characteristics: +# - Password rotates every 90 days +# - PostgreSQL REQUIRES old password to authenticate before changing +# - keepPreviousVersion creates postgres-secret-previous +# - CronJob uses old password to authenticate, sets new password +# - After 1 hour grace period, old secret is cleaned up + +--- +apiVersion: v1 +kind: Namespace +metadata: + name: database + +--- +# ManagedSecret with previous version support +apiVersion: secrets.c5ai.ch/v1alpha1 +kind: ManagedSecret +metadata: + name: postgres-app-user + namespace: database +spec: + vault: + address: "http://openbao.openbao.svc.cluster.local:8200" + authMethod: kubernetes + role: managedsecret-operator + kvVersion: v2 + mount: secret + path: postgres/app-user + + fields: + - name: username + type: static + value: "app_user" + + - name: password + type: generated + generator: + type: password + length: 40 + minDigits: 8 + minSymbols: 8 + minLowercase: 8 + minUppercase: 8 + symbolCharacters: "!@#$%^&*()" + allowRepeat: false + + - name: host + type: static + value: "postgres.database.svc.cluster.local" + + - name: port + type: static + value: "5432" + + - name: database + type: static + value: "app_db" + + destination: + name: postgres-secret + type: Opaque + # Keep previous version for authentication during rotation + keepPreviousVersion: true + previousVersionTTL: 1h # Clean up after 1 hour + + rotation: + enabled: true + schedule: 2160h # 90 days + rotateGeneratedOnly: true + +--- +# ServiceAccount for password rotation job +apiVersion: v1 +kind: ServiceAccount +metadata: + name: postgres-rotator + namespace: database + +--- +# Role to allow reading secrets +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: postgres-rotator + namespace: database +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + resourceNames: ["postgres-secret", "postgres-secret-previous"] + +--- +# RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: postgres-rotator + namespace: database +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: postgres-rotator +subjects: +- kind: ServiceAccount + name: postgres-rotator + namespace: database + +--- +# CronJob to detect rotation and update PostgreSQL password +apiVersion: batch/v1 +kind: CronJob +metadata: + name: postgres-password-rotator + namespace: database +spec: + # Run every 5 minutes to check for rotation + schedule: "*/5 * * * *" + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 3 + template: + metadata: + annotations: + # Reloader can also trigger this job when secret changes + reloader.stakater.com/auto: "true" + spec: + serviceAccountName: postgres-rotator + restartPolicy: OnFailure + containers: + - name: rotate-password + image: postgres:15-alpine + env: + # Current credentials + - name: PGUSER + valueFrom: + secretKeyRef: + name: postgres-secret + key: username + - name: NEW_PASSWORD + valueFrom: + secretKeyRef: + name: postgres-secret + key: password + - name: PGHOST + valueFrom: + secretKeyRef: + name: postgres-secret + key: host + - name: PGPORT + valueFrom: + secretKeyRef: + name: postgres-secret + key: port + - name: PGDATABASE + valueFrom: + secretKeyRef: + name: postgres-secret + key: database + + # Previous credentials (for authentication) + - name: OLD_PASSWORD + valueFrom: + secretKeyRef: + name: postgres-secret-previous + key: password + optional: true # Might not exist if no rotation yet + + command: + - sh + - -c + - | + set -e + + echo "==========================================" + echo "PostgreSQL Password Rotation Check" + echo "==========================================" + echo "User: $PGUSER" + echo "Host: $PGHOST:$PGPORT" + echo "Database: $PGDATABASE" + echo "" + + # Check if previous password exists (rotation happened) + if [ -z "$OLD_PASSWORD" ]; then + echo "No previous password found - no rotation needed" + exit 0 + fi + + # Check if passwords are different + if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then + echo "Passwords are identical - no rotation needed" + exit 0 + fi + + echo "Rotation detected! Old and new passwords differ." + echo "" + + # Test if current password in PostgreSQL is the old one + echo "Testing authentication with OLD password..." + if PGPASSWORD=$OLD_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then + echo "✓ Old password still active in PostgreSQL" + echo "" + echo "Updating to NEW password..." + + # Authenticate with OLD password, set NEW password + if PGPASSWORD=$OLD_PASSWORD psql -c "ALTER USER $PGUSER PASSWORD '$NEW_PASSWORD';"; then + echo "✓ Password updated successfully!" + echo "" + + # Verify new password works + echo "Verifying NEW password..." + if PGPASSWORD=$NEW_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then + echo "✓ New password verified and working!" + echo "" + echo "==========================================" + echo "Rotation completed successfully" + echo "==========================================" + exit 0 + else + echo "✗ ERROR: New password doesn't work!" + exit 1 + fi + else + echo "✗ ERROR: Failed to update password" + exit 1 + fi + else + echo "Old password doesn't work - checking if new password already active..." + if PGPASSWORD=$NEW_PASSWORD psql -c "SELECT 1;" > /dev/null 2>&1; then + echo "✓ New password is already active in PostgreSQL" + echo "Rotation was already completed" + exit 0 + else + echo "✗ ERROR: Neither old nor new password works!" + echo "Manual intervention required" + exit 1 + fi + fi + +--- +# Example application Deployment using the credentials +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app + namespace: database +spec: + replicas: 2 + selector: + matchLabels: + app: postgres-client + template: + metadata: + labels: + app: postgres-client + annotations: + # Reloader restarts app when credentials change + reloader.stakater.com/auto: "true" + spec: + containers: + - name: app + image: postgres:15-alpine + env: + - name: PGUSER + valueFrom: + secretKeyRef: + name: postgres-secret + key: username + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: postgres-secret + key: password + - name: PGHOST + valueFrom: + secretKeyRef: + name: postgres-secret + key: host + - name: PGPORT + valueFrom: + secretKeyRef: + name: postgres-secret + key: port + - name: PGDATABASE + valueFrom: + secretKeyRef: + name: postgres-secret + key: database + command: + - sh + - -c + - | + # Your application code here + while true; do + psql -c "SELECT NOW();" || echo "Connection failed" + sleep 30 + done + +--- +# How the rotation flow works: +# +# Day 0: Initial setup +# - ManagedSecret creates postgres-secret with generated password +# - No postgres-secret-previous exists yet +# - CronJob runs every 5 minutes, finds no previous password, does nothing +# +# Day 90: Rotation triggered +# 1. ManagedSecret operator: +# - Generates NEW password +# - Updates Vault with new password +# - Creates postgres-secret-previous with OLD password +# - Updates postgres-secret with NEW password +# +# 2. Within 5 minutes, CronJob runs: +# - Detects postgres-secret-previous exists +# - Compares OLD_PASSWORD vs NEW_PASSWORD (different!) +# - Authenticates to PostgreSQL with OLD_PASSWORD +# - Runs: ALTER USER app_user PASSWORD 'new_password' +# - Verifies new password works +# - Job succeeds +# +# 3. After 1 hour (previousVersionTTL): +# - ManagedSecret operator deletes postgres-secret-previous +# - CronJob future runs find no previous password, do nothing +# +# 4. Application pods: +# - Reloader detects postgres-secret change +# - Triggers rolling restart +# - Pods reconnect with NEW password +# - ~1 minute downtime per pod during restart + +--- +# Manual rotation testing: +# +# # Check current state +# kubectl get secret postgres-secret -n database -o yaml +# kubectl get secret postgres-secret-previous -n database -o yaml # Might not exist +# +# # Force rotation +# kubectl annotate managedsecret postgres-app-user -n database reconcile="$(date +%s)" --overwrite +# +# # Watch the rotation +# kubectl get secrets -n database -w +# +# # Check CronJob executes +# kubectl get jobs -n database -w +# +# # View rotation logs +# kubectl logs -n database -l job-name=postgres-password-rotator-XXXXX +# +# # Verify password in PostgreSQL matches new secret +# NEW_PASS=$(kubectl get secret postgres-secret -n database -o jsonpath='{.data.password}' | base64 -d) +# kubectl exec -it -n database deployment/app -- psql -c "SELECT 1;" + +--- +# Troubleshooting: +# +# If rotation fails: +# 1. Check CronJob logs for errors +# 2. Verify PostgreSQL is accessible from CronJob pod +# 3. Check if old password still works in PostgreSQL +# 4. Manually update if needed: +# kubectl exec -it postgres-0 -n database -- psql -U postgres +# ALTER USER app_user PASSWORD 'password_from_secret'; \ No newline at end of file diff --git a/examples/example-4-minio-rotation-previous.yaml b/examples/example-4-minio-rotation-previous.yaml new file mode 100644 index 0000000..81bad0e --- /dev/null +++ b/examples/example-4-minio-rotation-previous.yaml @@ -0,0 +1,430 @@ +# Example 4: MinIO Admin Password Rotation with Previous Version +# +# Use case: MinIO admin credentials that rotate every 60 days +# +# Characteristics: +# - Password rotates every 60 days +# - MinIO Client (mc) needs old credentials to change password +# - keepPreviousVersion creates minio-admin-secret-previous +# - CronJob uses old credentials to authenticate, sets new password +# - After 2 hour grace period, old secret is cleaned up + +--- +apiVersion: v1 +kind: Namespace +metadata: + name: minio + +--- +# ManagedSecret for MinIO admin credentials +apiVersion: secrets.c5ai.ch/v1alpha1 +kind: ManagedSecret +metadata: + name: minio-admin + namespace: minio +spec: + vault: + address: "http://openbao.openbao.svc.cluster.local:8200" + authMethod: kubernetes + role: managedsecret-operator + kvVersion: v2 + mount: secret + path: minio/admin + + fields: + - name: username + type: static + value: "admin" + + - name: password + type: generated + generator: + type: password + length: 32 + minDigits: 5 + minSymbols: 5 + minLowercase: 5 + minUppercase: 5 + symbolCharacters: "!@#$%^&*" + allowRepeat: false + + - name: endpoint + type: static + value: "http://minio.minio.svc.cluster.local:9000" + + destination: + name: minio-admin-secret + type: Opaque + # Keep previous version for authentication during rotation + keepPreviousVersion: true + previousVersionTTL: 2h # Longer grace period for MinIO + + rotation: + enabled: true + schedule: 1440h # 60 days + rotateGeneratedOnly: true + +--- +# ServiceAccount for password rotation job +apiVersion: v1 +kind: ServiceAccount +metadata: + name: minio-rotator + namespace: minio + +--- +# Role to allow reading secrets +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: minio-rotator + namespace: minio +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + resourceNames: ["minio-admin-secret", "minio-admin-secret-previous"] + +--- +# RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: minio-rotator + namespace: minio +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: minio-rotator +subjects: +- kind: ServiceAccount + name: minio-rotator + namespace: minio + +--- +# CronJob to detect rotation and update MinIO password +apiVersion: batch/v1 +kind: CronJob +metadata: + name: minio-password-rotator + namespace: minio +spec: + # Run every 10 minutes to check for rotation + schedule: "*/10 * * * *" + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 3 + template: + metadata: + annotations: + reloader.stakater.com/auto: "true" + spec: + serviceAccountName: minio-rotator + restartPolicy: OnFailure + containers: + - name: rotate-password + image: minio/mc:latest + env: + # Current credentials + - name: MC_HOST_myminio + value: "" # Will be set in script + - name: USERNAME + valueFrom: + secretKeyRef: + name: minio-admin-secret + key: username + - name: NEW_PASSWORD + valueFrom: + secretKeyRef: + name: minio-admin-secret + key: password + - name: ENDPOINT + valueFrom: + secretKeyRef: + name: minio-admin-secret + key: endpoint + + # Previous credentials (for authentication) + - name: OLD_USERNAME + valueFrom: + secretKeyRef: + name: minio-admin-secret-previous + key: username + optional: true + - name: OLD_PASSWORD + valueFrom: + secretKeyRef: + name: minio-admin-secret-previous + key: password + optional: true + + command: + - sh + - -c + - | + set -e + + echo "==========================================" + echo "MinIO Password Rotation Check" + echo "==========================================" + echo "User: $USERNAME" + echo "Endpoint: $ENDPOINT" + echo "" + + # Check if previous password exists (rotation happened) + if [ -z "$OLD_PASSWORD" ]; then + echo "No previous password found - no rotation needed" + exit 0 + fi + + # Check if passwords are different + if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then + echo "Passwords are identical - no rotation needed" + exit 0 + fi + + echo "Rotation detected! Old and new passwords differ." + echo "" + + # Test if current password in MinIO is the old one + echo "Testing authentication with OLD credentials..." + if mc alias set oldminio $ENDPOINT $OLD_USERNAME $OLD_PASSWORD > /dev/null 2>&1; then + echo "✓ Old credentials still active in MinIO" + echo "" + + # Test admin access + if mc admin info oldminio > /dev/null 2>&1; then + echo "✓ Admin access confirmed with old credentials" + echo "" + echo "Updating to NEW password..." + + # Change password using old credentials + if mc admin user password oldminio $USERNAME $NEW_PASSWORD; then + echo "✓ Password updated successfully!" + echo "" + + # Verify new password works + echo "Verifying NEW password..." + if mc alias set newminio $ENDPOINT $USERNAME $NEW_PASSWORD > /dev/null 2>&1; then + if mc admin info newminio > /dev/null 2>&1; then + echo "✓ New password verified and working!" + echo "" + echo "==========================================" + echo "Rotation completed successfully" + echo "==========================================" + exit 0 + else + echo "✗ ERROR: New credentials don't have admin access!" + exit 1 + fi + else + echo "✗ ERROR: New password doesn't work!" + exit 1 + fi + else + echo "✗ ERROR: Failed to update password" + exit 1 + fi + else + echo "✗ ERROR: No admin access with old credentials" + exit 1 + fi + else + echo "Old credentials don't work - checking if new credentials already active..." + if mc alias set newminio $ENDPOINT $USERNAME $NEW_PASSWORD > /dev/null 2>&1; then + if mc admin info newminio > /dev/null 2>&1; then + echo "✓ New credentials are already active in MinIO" + echo "Rotation was already completed" + exit 0 + else + echo "✗ ERROR: New credentials exist but no admin access" + exit 1 + fi + else + echo "✗ ERROR: Neither old nor new credentials work!" + echo "Manual intervention required" + exit 1 + fi + fi + +--- +# Example: MinIO StatefulSet (simplified) +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: minio + namespace: minio +spec: + serviceName: minio + replicas: 1 + selector: + matchLabels: + app: minio + template: + metadata: + labels: + app: minio + annotations: + # Note: MinIO doesn't auto-reload credentials from env vars + # You may need to restart pods manually or use a sidecar + reloader.stakater.com/auto: "false" + spec: + containers: + - name: minio + image: minio/minio:latest + args: + - server + - /data + - --console-address + - ":9001" + env: + - name: MINIO_ROOT_USER + valueFrom: + secretKeyRef: + name: minio-admin-secret + key: username + - name: MINIO_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: minio-admin-secret + key: password + ports: + - containerPort: 9000 + name: api + - containerPort: 9001 + name: console + volumeMounts: + - name: data + mountPath: /data + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 10Gi + +--- +# Service for MinIO +apiVersion: v1 +kind: Service +metadata: + name: minio + namespace: minio +spec: + selector: + app: minio + ports: + - port: 9000 + targetPort: 9000 + name: api + - port: 9001 + targetPort: 9001 + name: console + +--- +# How the rotation flow works: +# +# Day 0: Initial setup +# - ManagedSecret creates minio-admin-secret with generated password +# - MinIO starts with these credentials +# - No minio-admin-secret-previous exists yet +# - CronJob runs every 10 minutes, finds no previous password, does nothing +# +# Day 60: Rotation triggered +# 1. ManagedSecret operator: +# - Generates NEW password +# - Updates Vault with new password +# - Creates minio-admin-secret-previous with OLD password +# - Updates minio-admin-secret with NEW password +# +# 2. Within 10 minutes, CronJob runs: +# - Detects minio-admin-secret-previous exists +# - Compares OLD_PASSWORD vs NEW_PASSWORD (different!) +# - Authenticates to MinIO with OLD credentials +# - Runs: mc admin user password myminio admin new_password +# - Verifies new password works +# - Job succeeds +# +# 3. After 2 hours (previousVersionTTL): +# - ManagedSecret operator deletes minio-admin-secret-previous +# - CronJob future runs find no previous password, do nothing +# +# 4. MinIO pods: +# - IMPORTANT: MinIO does NOT reload credentials from env vars +# - You need to manually restart MinIO pods after rotation +# - Or use a sidecar to monitor and trigger restart +# - Consider NOT using reloader.stakater.com/auto for MinIO + +--- +# Alternative: Sidecar to handle MinIO restart +# Add this to the MinIO StatefulSet if you want automatic restarts +--- +# sidecar: +# - name: credential-watcher +# image: bitnami/kubectl:latest +# command: +# - sh +# - -c +# - | +# LAST_VERSION="" +# while true; do +# CURRENT_VERSION=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.metadata.resourceVersion}') +# if [ -n "$LAST_VERSION" ] && [ "$LAST_VERSION" != "$CURRENT_VERSION" ]; then +# echo "Credentials changed! Signaling MinIO to restart..." +# # Send SIGTERM to MinIO process to trigger graceful shutdown +# killall minio +# fi +# LAST_VERSION=$CURRENT_VERSION +# sleep 30 +# done + +--- +# Manual rotation testing: +# +# # Check current state +# kubectl get secret minio-admin-secret -n minio -o yaml +# kubectl get secret minio-admin-secret-previous -n minio -o yaml +# +# # Force rotation +# kubectl annotate managedsecret minio-admin -n minio reconcile="$(date +%s)" --overwrite +# +# # Watch the rotation +# kubectl get secrets -n minio -w +# +# # Check CronJob executes +# kubectl get jobs -n minio -w +# +# # View rotation logs +# kubectl logs -n minio -l job-name=minio-password-rotator-XXXXX +# +# # Test MinIO access with new credentials +# NEW_USER=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.data.username}' | base64 -d) +# NEW_PASS=$(kubectl get secret minio-admin-secret -n minio -o jsonpath='{.data.password}' | base64 -d) +# mc alias set testminio http://minio.minio.svc.cluster.local:9000 $NEW_USER $NEW_PASS +# mc admin info testminio + +--- +# Important Notes: +# +# 1. MinIO credential reload: +# - Unlike PostgreSQL, MinIO does NOT automatically reload credentials +# - After rotation, you must restart MinIO pods +# - Consider using a sidecar or manual restart workflow +# +# 2. High Availability: +# - For HA MinIO clusters, all nodes share the same root credentials +# - Rotation updates credentials cluster-wide +# - Rolling restart required for all nodes +# +# 3. Grace Period: +# - 2 hour previousVersionTTL allows time for manual intervention +# - Adjust based on your operational procedures +# +# 4. Monitoring: +# - Monitor CronJob success/failure +# - Alert on rotation failures +# - Track when MinIO pods were last restarted \ No newline at end of file diff --git a/examples/example-5-mysql-rotation-previous.yaml b/examples/example-5-mysql-rotation-previous.yaml new file mode 100644 index 0000000..a59f367 --- /dev/null +++ b/examples/example-5-mysql-rotation-previous.yaml @@ -0,0 +1,566 @@ +# Example 5: MySQL User Password Rotation with Previous Version +# +# Use case: MySQL application user password that rotates every 90 days +# +# Characteristics: +# - Password rotates every 90 days +# - MySQL requires old password to authenticate before changing (when using SET PASSWORD) +# - keepPreviousVersion creates mysql-app-user-previous +# - CronJob uses old password to authenticate, sets new password +# - Application pods restart automatically via Reloader + +--- +apiVersion: v1 +kind: Namespace +metadata: + name: mysql + +--- +# ManagedSecret for MySQL application user +apiVersion: secrets.c5ai.ch/v1alpha1 +kind: ManagedSecret +metadata: + name: mysql-app-user + namespace: mysql +spec: + vault: + address: "http://openbao.openbao.svc.cluster.local:8200" + authMethod: kubernetes + role: managedsecret-operator + kvVersion: v2 + mount: secret + path: mysql/app-user + + fields: + - name: username + type: static + value: "app_user" + + - name: password + type: generated + generator: + type: password + length: 32 + minDigits: 6 + minSymbols: 6 + minLowercase: 8 + minUppercase: 8 + # MySQL doesn't allow some special chars in passwords + symbolCharacters: "!@#$%^&*" + allowRepeat: false + + - name: host + type: static + value: "mysql.mysql.svc.cluster.local" + + - name: port + type: static + value: "3306" + + - name: database + type: static + value: "app_db" + + - name: connection-string + type: static + value: "mysql://app_user@mysql.mysql.svc.cluster.local:3306/app_db" + + destination: + name: mysql-app-secret + type: Opaque + # Keep previous version for authentication during rotation + keepPreviousVersion: true + previousVersionTTL: 1h + + rotation: + enabled: true + schedule: 2160h # 90 days + rotateGeneratedOnly: true + +--- +# Separate ManagedSecret for MySQL root credentials (no rotation) +apiVersion: secrets.c5ai.ch/v1alpha1 +kind: ManagedSecret +metadata: + name: mysql-root + namespace: mysql +spec: + vault: + address: "http://openbao.openbao.svc.cluster.local:8200" + authMethod: kubernetes + role: managedsecret-operator + kvVersion: v2 + mount: secret + path: mysql/root + + fields: + - name: username + type: static + value: "root" + + - name: password + type: generated + generator: + type: password + length: 40 + minDigits: 8 + minSymbols: 8 + + destination: + name: mysql-root-secret + type: Opaque + + # Root password doesn't rotate automatically + rotation: + enabled: false + +--- +# ServiceAccount for password rotation job +apiVersion: v1 +kind: ServiceAccount +metadata: + name: mysql-rotator + namespace: mysql + +--- +# Role to allow reading secrets +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: mysql-rotator + namespace: mysql +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + resourceNames: + - "mysql-app-secret" + - "mysql-app-secret-previous" + - "mysql-root-secret" + +--- +# RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: mysql-rotator + namespace: mysql +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: mysql-rotator +subjects: +- kind: ServiceAccount + name: mysql-rotator + namespace: mysql + +--- +# CronJob to detect rotation and update MySQL password +apiVersion: batch/v1 +kind: CronJob +metadata: + name: mysql-password-rotator + namespace: mysql +spec: + # Run every 5 minutes to check for rotation + schedule: "*/5 * * * *" + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 3 + template: + metadata: + annotations: + reloader.stakater.com/auto: "true" + spec: + serviceAccountName: mysql-rotator + restartPolicy: OnFailure + containers: + - name: rotate-password + image: mysql:8.0 + env: + # Current credentials + - name: MYSQL_USER + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: username + - name: NEW_PASSWORD + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: password + - name: MYSQL_HOST + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: host + - name: MYSQL_PORT + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: port + - name: MYSQL_DATABASE + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: database + + # Previous credentials (for authentication) + - name: OLD_PASSWORD + valueFrom: + secretKeyRef: + name: mysql-app-secret-previous + key: password + optional: true + + # Root credentials for fallback + - name: MYSQL_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: mysql-root-secret + key: password + + command: + - bash + - -c + - | + set -e + + echo "==========================================" + echo "MySQL Password Rotation Check" + echo "==========================================" + echo "User: $MYSQL_USER" + echo "Host: $MYSQL_HOST:$MYSQL_PORT" + echo "Database: $MYSQL_DATABASE" + echo "" + + # Check if previous password exists (rotation happened) + if [ -z "$OLD_PASSWORD" ]; then + echo "No previous password found - no rotation needed" + exit 0 + fi + + # Check if passwords are different + if [ "$OLD_PASSWORD" = "$NEW_PASSWORD" ]; then + echo "Passwords are identical - no rotation needed" + exit 0 + fi + + echo "Rotation detected! Old and new passwords differ." + echo "" + + # Test if current password in MySQL is the old one + echo "Testing authentication with OLD password..." + if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$OLD_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then + echo "✓ Old password still active in MySQL" + echo "" + echo "Updating to NEW password..." + + # Method 1: Use ALTER USER (MySQL 5.7.6+, doesn't require old password) + # Connect as root to change password + if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \ + -e "ALTER USER '$MYSQL_USER'@'%' IDENTIFIED BY '$NEW_PASSWORD';" 2>/dev/null; then + echo "✓ Password updated using ALTER USER (as root)" + elif mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \ + -e "SET PASSWORD FOR '$MYSQL_USER'@'%' = PASSWORD('$NEW_PASSWORD');" 2>/dev/null; then + echo "✓ Password updated using SET PASSWORD (as root)" + else + # Method 2: Use SET PASSWORD as the user (requires old password connection) + if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$OLD_PASSWORD" \ + -e "SET PASSWORD = PASSWORD('$NEW_PASSWORD');" 2>/dev/null; then + echo "✓ Password updated using SET PASSWORD (as user)" + else + echo "✗ ERROR: Failed to update password with any method" + exit 1 + fi + fi + + echo "" + + # Verify new password works + echo "Verifying NEW password..." + if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$NEW_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then + echo "✓ New password verified and working!" + echo "" + echo "Flushing privileges..." + mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" -e "FLUSH PRIVILEGES;" || echo "Warning: Could not flush privileges" + echo "" + echo "==========================================" + echo "Rotation completed successfully" + echo "==========================================" + exit 0 + else + echo "✗ ERROR: New password doesn't work!" + exit 1 + fi + else + echo "Old password doesn't work - checking if new password already active..." + if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" -p"$NEW_PASSWORD" -e "SELECT 1;" > /dev/null 2>&1; then + echo "✓ New password is already active in MySQL" + echo "Rotation was already completed" + exit 0 + else + echo "✗ ERROR: Neither old nor new password works!" + echo "Manual intervention required" + echo "" + echo "Attempting to reset using root credentials..." + if mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -uroot -p"$MYSQL_ROOT_PASSWORD" \ + -e "ALTER USER '$MYSQL_USER'@'%' IDENTIFIED BY '$NEW_PASSWORD';"; then + echo "✓ Password reset successfully using root" + exit 0 + else + echo "✗ Failed to reset password" + exit 1 + fi + fi + fi + +--- +# Example MySQL StatefulSet (simplified) +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: mysql + namespace: mysql +spec: + serviceName: mysql + replicas: 1 + selector: + matchLabels: + app: mysql + template: + metadata: + labels: + app: mysql + spec: + containers: + - name: mysql + image: mysql:8.0 + env: + - name: MYSQL_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: mysql-root-secret + key: password + - name: MYSQL_DATABASE + value: "app_db" + ports: + - containerPort: 3306 + name: mysql + volumeMounts: + - name: data + mountPath: /var/lib/mysql + livenessProbe: + exec: + command: + - mysqladmin + - ping + - -h + - localhost + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + exec: + command: + - mysql + - -h + - localhost + - -e + - SELECT 1 + initialDelaySeconds: 5 + periodSeconds: 2 + # InitContainer to create app user on first run + initContainers: + - name: init-db-user + image: mysql:8.0 + env: + - name: MYSQL_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: mysql-root-secret + key: password + - name: APP_USER + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: username + - name: APP_PASSWORD + valueFrom: + secretKeyRef: + name: mysql-app-secret + key: password + command: + - sh + - -c + - | + # Wait for MySQL to be ready + until mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SELECT 1" > /dev/null 2>&1; do + echo "Waiting for MySQL..." + sleep 2 + done + + # Create user if not exists + mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" <