Initial public release v1.0.7

- Streamlined README focused on quick start - Complete examples for all major use cases - Decision tree for choosing right pattern - Comprehensive troubleshooting guide
2025-10-26 14:53:01 +01:00
commit 1bc8aadb85
9 changed files with 3485 additions and 0 deletions
--- a/README-PUBLIC.md
+++ b/README-PUBLIC.md
@@ -0,0 +1,711 @@
+# ManagedSecret Operator
+
+A Kubernetes operator for declarative secret management with automated rotation, drift detection, and zero-downtime password changes.
+
+## Overview
+
+The ManagedSecret operator provides:
+
+- 🔐 **Generates secrets** with configurable password policies
+- 🏦 **Stores in Vault/OpenBao** as the authoritative source
+- 🔄 **Syncs to Kubernetes Secrets** for application consumption
+- 🔍 **Detects and fixes drift** automatically (Vault → K8s)
+- 🔁 **Rotates secrets** on a configurable schedule
+- ⏱️ **Preserves previous versions** during rotation for zero-downtime password changes
+
+### Key Principle: Vault/OpenBao is ALWAYS the source of truth
+
+- Changes in Vault → Synced to K8s automatically
+- Changes in K8s → Overwritten by Vault values
+- Drift detection runs every 1 minute
+
+---
+
+## 🚀 Quick Start
+
+> **Prerequisites:** Contact your administrator to receive:
+> - Registry username and password for `registry.c5ai.ch`
+> - Confirmation that OpenBao/Vault is configured for your cluster
+
+```bash
+# 1. Set your credentials (provided by administrator)
+export REGISTRY_USER="<provided-by-admin>"
+export REGISTRY_PASS="<provided-by-admin>"
+
+# 2. Install Reloader (for automatic pod restarts during rotation)
+kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
+
+# 3. Login to Helm registry
+helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS
+
+# 4. Create imagePullSecret
+kubectl create namespace managedsecret-operator-system
+kubectl create secret docker-registry registry-pull-secret \
+  --docker-server=registry.c5ai.ch \
+  --docker-username=$REGISTRY_USER \
+  --docker-password=$REGISTRY_PASS \
+  --namespace managedsecret-operator-system
+
+# 5. Install operator via Helm
+helm install managedsecret-operator oci://registry.c5ai.ch/charts/managedsecret-operator \
+  --version 1.0.7 \
+  --namespace managedsecret-operator-system \
+  --set imagePullSecrets[0].name=registry-pull-secret
+
+# 6. Verify installation
+kubectl get pods -n managedsecret-operator-system
+```
+
+That's it! Now check the [examples](./examples/) folder for comprehensive usage scenarios.
+
+---
+
+## 📦 What You Get
+
+### Helm Chart from registry.c5ai.ch
+
+The operator is distributed as a Helm chart:
+
+- ✅ **No building required** - Just `helm install`
+- ✅ **Versioned releases** - Use specific versions or upgrade easily
+- ✅ **Pre-configured** - Sensible defaults, customize via `values.yaml`
+- ✅ **Private & secure** - Hosted on your infrastructure
+
+### What's Deployed
+
+```
+managedsecret-operator-system/
+├── CustomResourceDefinition (ManagedSecret)
+├── ServiceAccount (controller-manager)
+├── Role & RoleBinding (RBAC)
+├── ClusterRole & ClusterRoleBinding
+├── Deployment (controller-manager)
+└── ConfigMap (operator configuration)
+```
+
+---
+
+## Prerequisites
+
+### On Your Side (User)
+- ✅ Kubernetes cluster (1.24+)
+- ✅ kubectl configured and working
+- ✅ Helm 3.x installed
+- ✅ Cluster admin permissions
+
+### Provided by Administrator
+- ✅ **Registry credentials** for `registry.c5ai.ch`
+- ✅ **OpenBao/Vault configuration** for your cluster
+  - Vault address (e.g., `http://openbao.openbao.svc.cluster.local:8200`)
+  - Kubernetes auth configured
+  - Vault role name (typically `managedsecret-operator`)
+
+### Optional (Recommended)
+- ✅ **Reloader by Stakater** (for automatic pod restarts on rotation)
+
+---
+
+## Installation Steps
+
+### Step 1: Install Reloader (Recommended)
+
+Reloader automatically restarts pods when secrets change:
+
+```bash
+# Direct install (simplest)
+kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
+
+# Via Helm
+helm repo add stakater https://stakater.github.io/stakater-charts
+helm install reloader stakater/reloader \
+  --namespace reloader \
+  --create-namespace
+```
+
+### Step 2: Authenticate to Registry
+
+```bash
+# Set credentials from administrator
+export REGISTRY_USER="<your-username>"
+export REGISTRY_PASS="<your-password>"
+
+# Login to Helm registry
+helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS
+
+# Create namespace
+kubectl create namespace managedsecret-operator-system
+
+# Create imagePullSecret
+kubectl create secret docker-registry registry-pull-secret \
+  --docker-server=registry.c5ai.ch \
+  --docker-username=$REGISTRY_USER \
+  --docker-password=$REGISTRY_PASS \
+  --namespace managedsecret-operator-system
+```
+
+### Step 3: Install the Operator
+
+```bash
+# Install with default configuration
+helm install managedsecret-operator \
+  oci://registry.c5ai.ch/charts/managedsecret-operator \
+  --version 1.0.7 \
+  --namespace managedsecret-operator-system \
+  --set imagePullSecrets[0].name=registry-pull-secret
+
+# Or with custom values
+cat > values.yaml <<EOF
+imagePullSecrets:
+  - name: registry-pull-secret
+
+replicaCount: 1
+
+resources:
+  limits:
+    cpu: 200m
+    memory: 256Mi
+  requests:
+    cpu: 100m
+    memory: 128Mi
+EOF
+
+helm install managedsecret-operator \
+  oci://registry.c5ai.ch/charts/managedsecret-operator \
+  --version 1.0.7 \
+  --namespace managedsecret-operator-system \
+  -f values.yaml
+```
+
+### Step 4: Verify Installation
+
+```bash
+# Check operator is running
+kubectl get pods -n managedsecret-operator-system
+
+# Check CRD is installed
+kubectl get crd managedsecrets.secrets.c5ai.ch
+
+# View operator logs
+kubectl logs -n managedsecret-operator-system \
+  -l control-plane=controller-manager --tail=50
+```
+
+---
+
+## 📖 Usage
+
+### Basic Example: Simple API Key (No Rotation)
+
+```yaml
+apiVersion: secrets.c5ai.ch/v1alpha1
+kind: ManagedSecret
+metadata:
+  name: api-keys
+  namespace: my-app
+spec:
+  vault:
+    address: "http://openbao.openbao.svc.cluster.local:8200"
+    authMethod: kubernetes
+    role: managedsecret-operator
+    kvVersion: v2
+    mount: secret
+    path: my-app/api-keys
+  
+  fields:
+  - name: api-key
+    type: generated
+    generator:
+      type: password
+      length: 64
+      minDigits: 10
+      symbolCharacters: ""  # Alphanumeric only
+  
+  - name: api-endpoint
+    type: static
+    value: "https://api.example.com/v1"
+  
+  destination:
+    name: api-secret
+    type: Opaque
+  
+  rotation:
+    enabled: false
+```
+
+Apply it:
+```bash
+kubectl apply -f managedsecret.yaml
+```
+
+The operator will:
+1. Generate a 64-character API key
+2. Store it in Vault at `secret/data/my-app/api-keys`
+3. Create Kubernetes Secret `api-secret` in namespace `my-app`
+
+Use it in your pod:
+```yaml
+env:
+- name: API_KEY
+  valueFrom:
+    secretKeyRef:
+      name: api-secret
+      key: api-key
+```
+
+### Rotation Example: PostgreSQL with Previous Version
+
+For services that need the old password to authenticate before changing to the new password:
+
+```yaml
+apiVersion: secrets.c5ai.ch/v1alpha1
+kind: ManagedSecret
+metadata:
+  name: postgres-credentials
+  namespace: database
+spec:
+  vault:
+    address: "http://openbao.openbao.svc.cluster.local:8200"
+    authMethod: kubernetes
+    role: managedsecret-operator
+    kvVersion: v2
+    mount: secret
+    path: database/postgres
+  
+  fields:
+  - name: username
+    type: static
+    value: "app_user"
+  
+  - name: password
+    type: generated
+    generator:
+      type: password
+      length: 40
+      minDigits: 8
+      minSymbols: 8
+  
+  destination:
+    name: postgres-secret
+    type: Opaque
+    keepPreviousVersion: true  # Creates postgres-secret-previous
+    previousVersionTTL: 1h     # Cleanup after 1 hour
+  
+  rotation:
+    enabled: true
+    schedule: 2160h  # 90 days
+    rotateGeneratedOnly: true
+```
+
+When rotation happens:
+1. New password is generated
+2. `postgres-secret` is updated with new password
+3. `postgres-secret-previous` is created with old password
+4. Your CronJob uses old password to authenticate and set new password
+5. After 1 hour, `-previous` secret is automatically cleaned up
+
+**Important:** You'll need a CronJob to handle the password change. See [examples/example-3-postgres-rotation-previous.yaml](./examples/example-3-postgres-rotation-previous.yaml) for a complete implementation.
+
+---
+
+## 📚 Examples
+
+The `examples/` folder contains comprehensive scenarios:
+
+| Example | Use Case | Description |
+|---------|----------|-------------|
+| **[Example 1](./examples/example-1-no-rotation.yaml)** | API Keys | Long-lived credentials without rotation |
+| **[Example 2](./examples/example-2-registry-rotation-simple.yaml)** | Container Registry | Simple rotation with Reloader |
+| **[Example 3](./examples/example-3-postgres-rotation-previous.yaml)** | PostgreSQL | Rotation with previous version (needs old password) |
+| **[Example 4](./examples/example-4-minio-rotation-previous.yaml)** | MinIO | Object storage with admin credentials rotation |
+| **[Example 5](./examples/example-5-mysql-rotation-previous.yaml)** | MySQL | Database rotation with multiple strategies |
+| **[Example 6](./examples/example-6-comprehensive-multi-service.yaml)** | Full Stack | Complete application with mixed strategies |
+
+**See [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md) for detailed explanations and decision trees.**
+
+---
+
+## 🔄 Secret Rotation
+
+### How Rotation Works
+
+```
+Day 0: Initial setup
+  - ManagedSecret creates secret with generated password
+  - Stored in Vault and synced to K8s
+
+Day 90: Rotation triggered (schedule: 2160h)
+  1. Operator generates new password
+  2. Updates Vault
+  3. If keepPreviousVersion=true:
+     - Creates <secret-name>-previous with old values
+  4. Updates K8s Secret with new values
+  5. If Reloader is enabled:
+     - Reloader detects change → Restarts pods
+  6. After previousVersionTTL:
+     - Operator deletes -previous secret
+```
+
+### When to Use `keepPreviousVersion`
+
+Use `keepPreviousVersion: true` when your service needs the **old password to authenticate** before changing to the new password:
+
+- ✅ PostgreSQL, MySQL, MariaDB (`ALTER USER` requires authentication)
+- ✅ MinIO (`mc admin` needs old credentials)
+- ✅ LDAP, Active Directory
+- ❌ Container registries (htpasswd regenerated fresh)
+- ❌ Redis (CONFIG SET doesn't need old password)
+- ❌ Web services with startup scripts
+
+### Forcing Rotation
+
+```bash
+# Trigger immediate rotation
+kubectl annotate managedsecret <name> -n <namespace> \
+  reconcile="$(date +%s)" --overwrite
+```
+
+---
+
+## 🔍 Drift Detection
+
+The operator automatically detects and fixes drift every 60 seconds:
+
+**Scenario 1: Secret deleted in Kubernetes**
+```bash
+kubectl delete secret my-secret -n my-app
+# Operator recreates it from Vault within 60 seconds
+```
+
+**Scenario 2: Secret modified in Vault**
+```bash
+# Update value in Vault
+vault kv put secret/my-app/keys password=newvalue
+# Operator syncs to K8s within 60 seconds
+```
+
+**Scenario 3: Secret manually edited in K8s**
+```bash
+kubectl edit secret my-secret -n my-app
+# Changes are overwritten by Vault values within 60 seconds
+```
+
+---
+
+## 🔧 Troubleshooting
+
+### Operator Not Starting
+
+```bash
+# Check pod status
+kubectl get pods -n managedsecret-operator-system
+
+# View logs
+kubectl logs -n managedsecret-operator-system \
+  -l control-plane=controller-manager --tail=100
+
+# Common issues:
+# - ImagePullBackOff: Check imagePullSecret is created correctly
+# - CrashLoopBackOff: Check Vault connectivity
+```
+
+### Secret Not Created
+
+```bash
+# Check ManagedSecret status
+kubectl get managedsecret <name> -n <namespace> -o yaml
+
+# Look for conditions/events
+kubectl describe managedsecret <name> -n <namespace>
+
+# Check operator logs
+kubectl logs -n managedsecret-operator-system \
+  -l control-plane=controller-manager | grep <name>
+```
+
+### Vault Authentication Failing
+
+```bash
+# Verify ServiceAccount exists
+kubectl get sa controller-manager -n managedsecret-operator-system
+
+# Check Vault role configuration
+vault read auth/kubernetes/role/managedsecret-operator
+
+# Test authentication manually
+kubectl exec -it -n managedsecret-operator-system \
+  deployment/managedsecret-operator-controller-manager -- sh
+# Inside pod: Check /var/run/secrets/kubernetes.io/serviceaccount/token exists
+```
+
+### Rotation Not Working
+
+```bash
+# Check ManagedSecret schedule
+kubectl get managedsecret <name> -n <namespace> -o yaml | grep schedule
+
+# Force rotation to test
+kubectl annotate managedsecret <name> -n <namespace> \
+  reconcile="$(date +%s)" --overwrite
+
+# Check if Reloader is installed (if using rotation)
+kubectl get pods -n reloader
+```
+
+### Reloader Not Restarting Pods
+
+```bash
+# Verify Reloader is running
+kubectl get pods -n reloader
+
+# Check pod has correct annotation
+kubectl get deployment <name> -n <namespace> -o yaml | grep reloader
+
+# View Reloader logs
+kubectl logs -n reloader -l app=reloader
+
+# Required annotation on pod template:
+# annotations:
+#   reloader.stakater.com/auto: "true"
+```
+
+---
+
+## 🎯 Best Practices
+
+### 1. Always Use Reloader for Rotating Secrets
+
+Without Reloader, pods won't pick up new credentials:
+```yaml
+template:
+  metadata:
+    annotations:
+      reloader.stakater.com/auto: "true"
+```
+
+### 2. Set Appropriate Rotation Schedules
+
+- **High Security** (passwords, admin credentials): 30-90 days
+- **Medium Security** (application credentials): 90-180 days
+- **Low Security** (read-only API keys): 180+ days or disabled
+
+### 3. Use `keepPreviousVersion` Correctly
+
+Only use it when the service needs the old password to authenticate:
+```yaml
+destination:
+  keepPreviousVersion: true  # Only if old password is needed
+  previousVersionTTL: 1h     # Adjust based on your workflow
+```
+
+### 4. Test Rotation Before Production
+
+```bash
+# Create test ManagedSecret with short rotation
+spec:
+  rotation:
+    enabled: true
+    schedule: 5m  # Test rotation every 5 minutes
+
+# Watch the rotation
+kubectl get secrets -n <namespace> -w
+
+# Verify application handles rotation
+kubectl logs -n <namespace> <pod-name>
+```
+
+### 5. Monitor Rotation Jobs
+
+For services using `keepPreviousVersion`, monitor your CronJobs:
+```bash
+# Check job status
+kubectl get jobs -n <namespace>
+
+# View job logs
+kubectl logs -n <namespace> -l job-name=<rotation-job>
+
+# Set up alerts for failed jobs
+```
+
+### 6. Backup Vault Regularly
+
+Vault is your source of truth:
+```bash
+# Example: Backup with Velero
+velero backup create vault-backup \
+  --include-namespaces openbao \
+  --snapshot-volumes
+```
+
+### 7. Version Control Your ManagedSecrets
+
+Store ManagedSecret manifests in Git:
+```yaml
+# managedsecrets/
+# ├── production/
+# │   ├── postgres-credentials.yaml
+# │   └── api-keys.yaml
+# └── staging/
+#     ├── postgres-credentials.yaml
+#     └── api-keys.yaml
+```
+
+### 8. Use GitOps for Deployment
+
+Deploy with ArgoCD or Flux:
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: managedsecrets
+spec:
+  source:
+    path: managedsecrets/production
+  destination:
+    namespace: default
+```
+
+---
+
+## 🔒 Security Considerations
+
+### Production Checklist
+
+- [ ] OpenBao/Vault is properly secured (TLS, audit logging, backups)
+- [ ] RBAC configured with least-privilege
+- [ ] Network policies in place
+- [ ] Monitoring and alerting configured
+- [ ] Disaster recovery plan documented
+- [ ] Registry credentials stored securely (not in Git)
+
+### Credentials Management
+
+The operator needs:
+1. Access to Vault (via Kubernetes ServiceAccount auth)
+2. Access to pull its image (via imagePullSecret)
+
+Both use Kubernetes-native methods. **Never hardcode credentials in manifests!**
+
+---
+
+## 📞 Support
+
+### Getting Help
+
+1. **Check documentation**:
+   - This README
+   - [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md)
+   - Individual example files
+
+2. **Check operator logs**:
+   ```bash
+   kubectl logs -n managedsecret-operator-system \
+     -l control-plane=controller-manager --tail=100
+   ```
+
+3. **Contact your administrator** if:
+   - You need registry credentials
+   - Vault is not configured for your cluster
+   - Operator authentication issues
+
+### For Administrators
+
+#### Information to Provide Users
+
+**Registry Credentials:**
+```
+Registry: registry.c5ai.ch
+Username: <username>
+Password: <password>
+Chart: oci://registry.c5ai.ch/charts/managedsecret-operator
+Version: 1.0.7
+```
+
+**Vault Configuration:**
+```
+Address: http://openbao.openbao.svc.cluster.local:8200
+Auth Method: kubernetes
+Role: managedsecret-operator
+KV Mount: secret
+KV Version: v2
+```
+
+#### Prerequisites for New Cluster
+
+```bash
+# 1. Enable Kubernetes auth
+bao auth enable kubernetes
+bao write auth/kubernetes/config \
+    kubernetes_host="https://kubernetes.default.svc:443"
+
+# 2. Create policy
+bao policy write managedsecret-operator - <<'EOF'
+path "secret/data/*" {
+  capabilities = ["create", "read", "update", "delete", "list"]
+}
+path "secret/metadata/*" {
+  capabilities = ["list", "read", "delete"]
+}
+EOF
+
+# 3. Create role
+bao write auth/kubernetes/role/managedsecret-operator \
+    bound_service_account_names=controller-manager \
+    bound_service_account_namespaces=managedsecret-operator-system \
+    policies=managedsecret-operator \
+    ttl=1h
+```
+
+---
+
+## 🔄 Upgrading
+
+### Upgrade Operator Version
+
+```bash
+# Check current version
+helm list -n managedsecret-operator-system
+
+# Upgrade to new version
+helm upgrade managedsecret-operator \
+  oci://registry.c5ai.ch/charts/managedsecret-operator \
+  --version 1.0.8 \
+  --namespace managedsecret-operator-system \
+  --reuse-values
+```
+
+### Migration Notes
+
+When upgrading from earlier versions:
+- CRD updates are handled automatically
+- Existing ManagedSecrets continue to work
+- No manual migration required
+- Check changelog for breaking changes
+
+---
+
+## 📝 Changelog
+
+### Version 1.0.7
+- ✨ Added `keepPreviousVersion` for zero-downtime rotation
+- ✨ Added `previousVersionTTL` for automatic cleanup
+- 📦 Helm chart deployment support
+- 📚 Comprehensive examples and documentation
+- 🐛 Bug fixes and improvements
+
+---
+
+## License
+
+Proprietary Use License
+
+Permission is granted to use this software for any purpose, including commercial use, without charge.
+
+Modification, adaptation, or creation of derivative works is prohibited.
+Redistribution of modified versions is prohibited.
+
+---
+
+**Version:** 1.0.7  
+**Registry:** registry.c5ai.ch/charts/managedsecret-operator  
+**Minimum Kubernetes:** 1.24+  
+**Compatible with:** OpenBao, HashiCorp Vault