managedsecret-operator-public/examples/EXAMPLES-SUMMARY.md

# ManagedSecret Operator - Examples Summary

This directory contains comprehensive examples for the ManagedSecret Operator, demonstrating various rotation strategies and use cases.

## Files Overview

### Documentation
- **README.md** - Complete installation and usage guide with new features
  - Added `keepPreviousVersion` and `previousVersionTTL` documentation
  - Added Reloader integration guide
  - Updated best practices and architecture diagrams

### Example Files

#### Example 1: No Rotation (Simple API Key)
**File:** `example-1-no-rotation.yaml`

**Use Case:** Long-lived API keys that don't need rotation

**Key Features:**
- Generated once and kept indefinitely
- No Reloader needed (secret never changes)
- Perfect for external service API keys

**When to Use:**
- Third-party API keys (Stripe, SendGrid, etc.)
- Service identifiers
- Static configuration values

---

#### Example 2: Simple Rotation with Reloader (Container Registry)
**File:** `example-2-registry-rotation-simple.yaml`

**Use Case:** Docker/Container registry with HTTP basic auth

**Key Features:**
- Password rotates every 90 days
- InitContainer regenerates htpasswd on pod restart
- Reloader automatically restarts pods
- **No `keepPreviousVersion` needed** - htpasswd is regenerated fresh

**Rotation Flow:**
1. ManagedSecret generates new password → Updates Vault → Updates K8s Secret
2. Reloader detects Secret change → Triggers pod restart
3. InitContainer runs → Reads new password → Generates fresh htpasswd
4. Registry starts with new htpasswd

**When to Use:**
- Container registries (Docker Registry, Harbor)
- Web servers with htpasswd (Apache, Nginx)
- Any service where credentials are regenerated at startup

---

#### Example 3: Complex Rotation with Previous Password (PostgreSQL)
**File:** `example-3-postgres-rotation-previous.yaml`

**Use Case:** PostgreSQL user password that requires authentication to change

**Key Features:**
- Password rotates every 90 days
- `keepPreviousVersion: true` creates `-previous` secret
- CronJob uses old password to authenticate, sets new password
- `previousVersionTTL: 1h` cleans up after grace period

**Rotation Flow:**
1. ManagedSecret creates both `postgres-secret` and `postgres-secret-previous`
2. CronJob (every 5 min) detects rotation
3. Authenticates with OLD password: `PGPASSWORD=$OLD_PASSWORD psql`
4. Changes password: `ALTER USER app_user PASSWORD '$NEW_PASSWORD'`
5. Verifies new password works
6. After 1 hour, `-previous` secret is deleted

**When to Use:**
- PostgreSQL, MySQL, MariaDB
- Any database requiring authentication to change passwords
- Services with `ALTER USER` or similar commands

---

#### Example 4: MinIO Admin Password Rotation
**File:** `example-4-minio-rotation-previous.yaml`

**Use Case:** MinIO admin credentials with rotation

**Key Features:**
- Password rotates every 60 days
- Uses MinIO Client (mc) to change password
- `keepPreviousVersion: true` with 2h TTL
- CronJob handles rotation via `mc admin user password`

**Special Considerations:**
- MinIO does NOT reload credentials from environment variables
- Manual pod restart required after rotation
- Consider using a sidecar to monitor and restart

**Rotation Flow:**
1. ManagedSecret creates both secrets
2. CronJob authenticates with old credentials: `mc alias set oldminio`
3. Changes password: `mc admin user password myminio admin $NEW_PASSWORD`
4. Verifies new credentials
5. **Important:** MinIO pods must be restarted manually

**When to Use:**
- MinIO object storage
- Any service that doesn't auto-reload credentials
- Services requiring separate admin tools for password changes

---

#### Example 5: MySQL User Password Rotation
**File:** `example-5-mysql-rotation-previous.yaml`

**Use Case:** MySQL application user with automated rotation

**Key Features:**
- Password rotates every 90 days
- Multiple password change methods (ALTER USER, SET PASSWORD)
- Root credentials as fallback
- `keepPreviousVersion: true` with 1h TTL

**Rotation Flow:**
1. ManagedSecret creates both secrets
2. CronJob tries multiple approaches:
   - Method 1: `ALTER USER` as root (preferred)
   - Method 2: `SET PASSWORD` as root
   - Method 3: `SET PASSWORD` as user with old password
3. Flushes privileges
4. Verifies new password

**When to Use:**
- MySQL 5.7+, MySQL 8.0
- MariaDB
- Any MySQL-compatible database

---

#### Example 6: Comprehensive Multi-Service Deployment
**File:** `example-6-comprehensive-multi-service.yaml`

**Use Case:** Complete application stack showing all strategies together

**Components:**
1. **PostgreSQL** - Rotates with previous version (90 days)
2. **Redis** - Rotates without previous version (30 days)
3. **API Keys** - No rotation (managed externally)
4. **TLS Certificates** - No rotation (cert-manager)
5. **JWT Secret** - Simple rotation with Reloader (180 days)

**Demonstrates:**
- Multiple secrets with different rotation schedules
- Mixed rotation strategies in one application
- How Reloader restarts pods for all credential changes
- Real-world production setup

**When to Use:**
- As a template for complex applications
- To understand how different strategies work together
- For learning all features in one example

---

## Quick Decision Tree

### Does your service need the OLD password to set the NEW password?

**YES** (PostgreSQL, MySQL, MinIO, LDAP):
- Use `keepPreviousVersion: true`
- Set appropriate `previousVersionTTL`
- Create CronJob to handle rotation
- See Examples 3, 4, or 5

**NO** (Registry, Redis, Web servers):
- Use simple rotation with Reloader
- InitContainer or startup script regenerates credentials
- No CronJob needed
- See Example 2

**NOT ROTATING** (API keys, Certificates):
- Set `rotation.enabled: false`
- No Reloader annotation needed
- See Example 1

---

## Rotation Schedule Recommendations

- **High Security** (User passwords, admin credentials): 30-90 days
- **Medium Security** (Application credentials, service accounts): 90-180 days
- **Low Security** (Read-only API keys, development): 180+ days or disabled
- **Previous Version TTL**: 
  - Automated rotation: 1h
  - Manual intervention: 2-4h
  - Complex workflows: 24h

---

## Prerequisites for All Examples

1. **OpenBao/Vault** deployed and initialized
2. **ManagedSecret Operator** installed
3. **Reloader** installed (for rotation examples)
4. **cert-manager** (optional, for TLS examples)

## Installation Order

```bash
# 1. Install OpenBao
kubectl create namespace openbao
helm install openbao openbao/openbao -n openbao

# 2. Install ManagedSecret Operator
kubectl apply -f managedsecret-operator/

# 3. Install Reloader
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

# 4. Deploy your application with examples
kubectl apply -f example-X-your-usecase.yaml
```

---

## Testing Rotation

### Force Immediate Rotation
```bash
# Trigger rotation for any ManagedSecret
kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite
```

### Watch the Flow
```bash
# Terminal 1: Watch secrets
kubectl get secrets -n <namespace> -w

# Terminal 2: Watch pods
kubectl get pods -n <namespace> -w

# Terminal 3: Watch jobs (for CronJobs)
kubectl get jobs -n <namespace> -w

# Terminal 4: Reloader logs
kubectl logs -n reloader -l app=reloader -f
```

### Verify Rotation Completed
```bash
# Check if -previous secret exists (should only exist during grace period)
kubectl get secret <name>-previous -n <namespace>

# View rotation job logs
kubectl logs -n <namespace> -l job-name=<rotation-job>

# Test connection with new credentials
kubectl exec -it -n <namespace> <pod> -- <test-command>
```

---

## Common Issues and Solutions

### Issue: Pod not restarting after rotation
**Solution:** Check Reloader is installed and annotation is correct
```bash
kubectl get pods -n reloader
kubectl get deployment <app> -o yaml | grep reloader
```

### Issue: Rotation job failing
**Solution:** Check if previous secret exists and credentials work
```bash
kubectl get secret <name>-previous -n <namespace>
kubectl logs -n <namespace> -l job-name=<rotation-job>
```

### Issue: Application still using old credentials
**Solution:** Verify pod actually restarted after secret change
```bash
kubectl get pods -n <namespace> -o wide
# Check pod start time vs secret update time
```

### Issue: Previous secret not cleaned up
**Solution:** Check operator logs for cleanup job
```bash
kubectl logs -n managedsecret-operator-system -l control-plane=controller-manager | grep cleanup
```

---

## Best Practices Summary

1. ✅ **Always use Reloader** with rotating secrets
2. ✅ **Use `keepPreviousVersion`** for services requiring old password
3. ✅ **Set realistic `previousVersionTTL`** based on your workflow
4. ✅ **Test rotation** before production deployment
5. ✅ **Monitor rotation jobs** for failures
6. ✅ **Document rotation procedures** for your team
7. ✅ **Use appropriate rotation schedules** based on security requirements
8. ✅ **Backup Vault regularly** - it's your source of truth
9. ✅ **Never modify K8s Secrets manually** - always use ManagedSecret or Vault
10. ✅ **Set up alerts** for failed rotations

---

## Support and Contributing

For issues or questions about these examples:
- Check the main README.md for troubleshooting
- Review operator logs for detailed error messages
- Test in development environment before production

Remember: Vault/OpenBao is ALWAYS the source of truth. All changes should go through ManagedSecret resources or directly in Vault, never by manually editing Kubernetes Secrets.