managedsecret-operator-public/README.md

# ManagedSecret Operator

A Kubernetes operator for declarative secret management with automated rotation, drift detection, and zero-downtime password changes.

## Overview

The ManagedSecret operator provides:

- 🔐 **Generates secrets** with configurable password policies
- 🏦 **Stores in Vault/OpenBao** as the authoritative source
- 🔄 **Syncs to Kubernetes Secrets** for application consumption
- 🔍 **Detects and fixes drift** automatically (Vault → K8s)
- 🔁 **Rotates secrets** on a configurable schedule
- ⏱️ **Preserves previous versions** during rotation for zero-downtime password changes

### Key Principle: Vault/OpenBao is ALWAYS the source of truth

- Changes in Vault → Synced to K8s automatically
- Changes in K8s → Overwritten by Vault values
- Drift detection runs every 1 minute

---

## 🚀 Quick Start

> **Prerequisites:** Contact your administrator to receive:
> - Registry username and password for `registry.c5ai.ch`
> - Confirmation that OpenBao/Vault is configured for your cluster

```bash
# 1. Set your credentials (provided by administrator)
export REGISTRY_USER="<provided-by-admin>"
export REGISTRY_PASS="<provided-by-admin>"

# 2. Install Reloader (for automatic pod restarts during rotation)
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

# 3. Login to Helm registry
helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS

# 4. Create imagePullSecret
kubectl create namespace managedsecret-operator-system
kubectl create secret docker-registry registry-pull-secret \
  --docker-server=registry.c5ai.ch \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --namespace managedsecret-operator-system

# 5. Install operator via Helm
helm install managedsecret-operator oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  --set imagePullSecrets[0].name=registry-pull-secret

# 6. Verify installation
kubectl get pods -n managedsecret-operator-system
```

That's it! Now check the [examples](./examples/) folder for comprehensive usage scenarios.

---

## 📦 What You Get

### Helm Chart from registry.c5ai.ch

The operator is distributed as a Helm chart:

- ✅ **No building required** - Just `helm install`
- ✅ **Versioned releases** - Use specific versions or upgrade easily
- ✅ **Pre-configured** - Sensible defaults, customize via `values.yaml`
- ✅ **Private & secure** - Hosted on your infrastructure

### What's Deployed

```
managedsecret-operator-system/
├── CustomResourceDefinition (ManagedSecret)
├── ServiceAccount (controller-manager)
├── Role & RoleBinding (RBAC)
├── ClusterRole & ClusterRoleBinding
├── Deployment (controller-manager)
└── ConfigMap (operator configuration)
```

---

## Prerequisites

### On Your Side (User)
- ✅ Kubernetes cluster (1.24+)
- ✅ kubectl configured and working
- ✅ Helm 3.x installed
- ✅ Cluster admin permissions

### Provided by Administrator
- ✅ **Registry credentials** for `registry.c5ai.ch`
- ✅ **OpenBao/Vault configuration** for your cluster
  - Vault address (e.g., `http://openbao.openbao.svc.cluster.local:8200`)
  - Kubernetes auth configured
  - Vault role name (typically `managedsecret-operator`)

### Optional (Recommended)
- ✅ **Reloader by Stakater** (for automatic pod restarts on rotation)

---

## Installation Steps

### Step 1: Install Reloader (Recommended)

Reloader automatically restarts pods when secrets change:

```bash
# Direct install (simplest)
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

# Via Helm
helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader \
  --namespace reloader \
  --create-namespace
```

### Step 2: Authenticate to Registry

```bash
# Set credentials from administrator
export REGISTRY_USER="<your-username>"
export REGISTRY_PASS="<your-password>"

# Login to Helm registry
helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS

# Create namespace
kubectl create namespace managedsecret-operator-system

# Create imagePullSecret
kubectl create secret docker-registry registry-pull-secret \
  --docker-server=registry.c5ai.ch \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --namespace managedsecret-operator-system
```

### Step 3: Install the Operator

```bash
# Install with default configuration
helm install managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  --set imagePullSecrets[0].name=registry-pull-secret

# Or with custom values
cat > values.yaml <<EOF
imagePullSecrets:
  - name: registry-pull-secret

replicaCount: 1

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi
EOF

helm install managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  -f values.yaml
```

### Step 4: Verify Installation

```bash
# Check operator is running
kubectl get pods -n managedsecret-operator-system

# Check CRD is installed
kubectl get crd managedsecrets.secrets.c5ai.ch

# View operator logs
kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager --tail=50
```

---

## 📖 Usage

### Basic Example: Simple API Key (No Rotation)

```yaml
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
  name: api-keys
  namespace: my-app
spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: my-app/api-keys
  
  fields:
  - name: api-key
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 10
      symbolCharacters: ""  # Alphanumeric only
  
  - name: api-endpoint
    type: static
    value: "https://api.example.com/v1"
  
  destination:
    name: api-secret
    type: Opaque
  
  rotation:
    enabled: false
```

Apply it:
```bash
kubectl apply -f managedsecret.yaml
```

The operator will:
1. Generate a 64-character API key
2. Store it in Vault at `secret/data/my-app/api-keys`
3. Create Kubernetes Secret `api-secret` in namespace `my-app`

Use it in your pod:
```yaml
env:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: api-secret
      key: api-key
```

### Rotation Example: PostgreSQL with Previous Version

For services that need the old password to authenticate before changing to the new password:

```yaml
apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
  name: postgres-credentials
  namespace: database
spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: database/postgres
  
  fields:
  - name: username
    type: static
    value: "app_user"
  
  - name: password
    type: generated
    generator:
      type: password
      length: 40
      minDigits: 8
      minSymbols: 8
  
  destination:
    name: postgres-secret
    type: Opaque
    keepPreviousVersion: true  # Creates postgres-secret-previous
    previousVersionTTL: 1h     # Cleanup after 1 hour
  
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
    rotateGeneratedOnly: true
```

When rotation happens:
1. New password is generated
2. `postgres-secret` is updated with new password
3. `postgres-secret-previous` is created with old password
4. Your CronJob uses old password to authenticate and set new password
5. After 1 hour, `-previous` secret is automatically cleaned up

**Important:** You'll need a CronJob to handle the password change. See [examples/example-3-postgres-rotation-previous.yaml](./examples/example-3-postgres-rotation-previous.yaml) for a complete implementation.

---

## 📚 Examples

The `examples/` folder contains comprehensive scenarios:

| Example | Use Case | Description |
|---------|----------|-------------|
| **[Example 1](./examples/example-1-no-rotation.yaml)** | API Keys | Long-lived credentials without rotation |
| **[Example 2](./examples/example-2-registry-rotation-simple.yaml)** | Container Registry | Simple rotation with Reloader |
| **[Example 3](./examples/example-3-postgres-rotation-previous.yaml)** | PostgreSQL | Rotation with previous version (needs old password) |
| **[Example 4](./examples/example-4-minio-rotation-previous.yaml)** | MinIO | Object storage with admin credentials rotation |
| **[Example 5](./examples/example-5-mysql-rotation-previous.yaml)** | MySQL | Database rotation with multiple strategies |
| **[Example 6](./examples/example-6-comprehensive-multi-service.yaml)** | Full Stack | Complete application with mixed strategies |

**See [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md) for detailed explanations and decision trees.**

---

## 🔄 Secret Rotation

### How Rotation Works

```
Day 0: Initial setup
  - ManagedSecret creates secret with generated password
  - Stored in Vault and synced to K8s

Day 90: Rotation triggered (schedule: 2160h)
  1. Operator generates new password
  2. Updates Vault
  3. If keepPreviousVersion=true:
     - Creates <secret-name>-previous with old values
  4. Updates K8s Secret with new values
  5. If Reloader is enabled:
     - Reloader detects change → Restarts pods
  6. After previousVersionTTL:
     - Operator deletes -previous secret
```

### When to Use `keepPreviousVersion`

Use `keepPreviousVersion: true` when your service needs the **old password to authenticate** before changing to the new password:

- ✅ PostgreSQL, MySQL, MariaDB (`ALTER USER` requires authentication)
- ✅ MinIO (`mc admin` needs old credentials)
- ✅ LDAP, Active Directory
- ❌ Container registries (htpasswd regenerated fresh)
- ❌ Redis (CONFIG SET doesn't need old password)
- ❌ Web services with startup scripts

### Forcing Rotation

```bash
# Trigger immediate rotation
kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite
```

---

## 🔍 Drift Detection

The operator automatically detects and fixes drift every 60 seconds:

**Scenario 1: Secret deleted in Kubernetes**
```bash
kubectl delete secret my-secret -n my-app
# Operator recreates it from Vault within 60 seconds
```

**Scenario 2: Secret modified in Vault**
```bash
# Update value in Vault
vault kv put secret/my-app/keys password=newvalue
# Operator syncs to K8s within 60 seconds
```

**Scenario 3: Secret manually edited in K8s**
```bash
kubectl edit secret my-secret -n my-app
# Changes are overwritten by Vault values within 60 seconds
```

---

## 🔧 Troubleshooting

### Operator Not Starting

```bash
# Check pod status
kubectl get pods -n managedsecret-operator-system

# View logs
kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager --tail=100

# Common issues:
# - ImagePullBackOff: Check imagePullSecret is created correctly
# - CrashLoopBackOff: Check Vault connectivity
```

### Secret Not Created

```bash
# Check ManagedSecret status
kubectl get managedsecret <name> -n <namespace> -o yaml

# Look for conditions/events
kubectl describe managedsecret <name> -n <namespace>

# Check operator logs
kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager | grep <name>
```

### Vault Authentication Failing

```bash
# Verify ServiceAccount exists
kubectl get sa controller-manager -n managedsecret-operator-system

# Check Vault role configuration
vault read auth/kubernetes/role/managedsecret-operator

# Test authentication manually
kubectl exec -it -n managedsecret-operator-system \
  deployment/managedsecret-operator-controller-manager -- sh
# Inside pod: Check /var/run/secrets/kubernetes.io/serviceaccount/token exists
```

### Rotation Not Working

```bash
# Check ManagedSecret schedule
kubectl get managedsecret <name> -n <namespace> -o yaml | grep schedule

# Force rotation to test
kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite

# Check if Reloader is installed (if using rotation)
kubectl get pods -n reloader
```

### Reloader Not Restarting Pods

```bash
# Verify Reloader is running
kubectl get pods -n reloader

# Check pod has correct annotation
kubectl get deployment <name> -n <namespace> -o yaml | grep reloader

# View Reloader logs
kubectl logs -n reloader -l app=reloader

# Required annotation on pod template:
# annotations:
#   reloader.stakater.com/auto: "true"
```

---

## 🎯 Best Practices

### 1. Always Use Reloader for Rotating Secrets

Without Reloader, pods won't pick up new credentials:
```yaml
template:
  metadata:
    annotations:
      reloader.stakater.com/auto: "true"
```

### 2. Set Appropriate Rotation Schedules

- **High Security** (passwords, admin credentials): 30-90 days
- **Medium Security** (application credentials): 90-180 days
- **Low Security** (read-only API keys): 180+ days or disabled

### 3. Use `keepPreviousVersion` Correctly

Only use it when the service needs the old password to authenticate:
```yaml
destination:
  keepPreviousVersion: true  # Only if old password is needed
  previousVersionTTL: 1h     # Adjust based on your workflow
```

### 4. Test Rotation Before Production

```bash
# Create test ManagedSecret with short rotation
spec:
  rotation:
    enabled: true
    schedule: 5m  # Test rotation every 5 minutes

# Watch the rotation
kubectl get secrets -n <namespace> -w

# Verify application handles rotation
kubectl logs -n <namespace> <pod-name>
```

### 5. Monitor Rotation Jobs

For services using `keepPreviousVersion`, monitor your CronJobs:
```bash
# Check job status
kubectl get jobs -n <namespace>

# View job logs
kubectl logs -n <namespace> -l job-name=<rotation-job>

# Set up alerts for failed jobs
```

### 6. Backup Vault Regularly

Vault is your source of truth:
```bash
# Example: Backup with Velero
velero backup create vault-backup \
  --include-namespaces openbao \
  --snapshot-volumes
```

### 7. Version Control Your ManagedSecrets

Store ManagedSecret manifests in Git:
```yaml
# managedsecrets/
# ├── production/
# │   ├── postgres-credentials.yaml
# │   └── api-keys.yaml
# └── staging/
#     ├── postgres-credentials.yaml
#     └── api-keys.yaml
```

### 8. Use GitOps for Deployment

Deploy with ArgoCD or Flux:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: managedsecrets
spec:
  source:
    path: managedsecrets/production
  destination:
    namespace: default
```

---

## 🔒 Security Considerations

### Production Checklist

- [ ] OpenBao/Vault is properly secured (TLS, audit logging, backups)
- [ ] RBAC configured with least-privilege
- [ ] Network policies in place
- [ ] Monitoring and alerting configured
- [ ] Disaster recovery plan documented
- [ ] Registry credentials stored securely (not in Git)

### Credentials Management

The operator needs:
1. Access to Vault (via Kubernetes ServiceAccount auth)
2. Access to pull its image (via imagePullSecret)

Both use Kubernetes-native methods. **Never hardcode credentials in manifests!**

---

## 📞 Support

### Getting Help

1. **Check documentation**:
   - This README
   - [examples/EXAMPLES-SUMMARY.md](./examples/EXAMPLES-SUMMARY.md)
   - Individual example files

2. **Check operator logs**:
   ```bash
   kubectl logs -n managedsecret-operator-system \
     -l control-plane=controller-manager --tail=100
   ```

3. **Contact your administrator** if:
   - You need registry credentials
   - Vault is not configured for your cluster
   - Operator authentication issues

### For Administrators

#### Information to Provide Users

**Registry Credentials:**
```
Registry: registry.c5ai.ch
Username: <username>
Password: <password>
Chart: oci://registry.c5ai.ch/charts/managedsecret-operator
Version: 1.0.7
```

**Vault Configuration:**
```
Address: http://openbao.openbao.svc.cluster.local:8200
Auth Method: kubernetes
Role: managedsecret-operator
KV Mount: secret
KV Version: v2
```

#### Prerequisites for New Cluster

```bash
# 1. Enable Kubernetes auth
bao auth enable kubernetes
bao write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"

# 2. Create policy
bao policy write managedsecret-operator - <<'EOF'
path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/metadata/*" {
  capabilities = ["list", "read", "delete"]
}
EOF

# 3. Create role
bao write auth/kubernetes/role/managedsecret-operator \
    bound_service_account_names=controller-manager \
    bound_service_account_namespaces=managedsecret-operator-system \
    policies=managedsecret-operator \
    ttl=1h
```

---

## 🔄 Upgrading

### Upgrade Operator Version

```bash
# Check current version
helm list -n managedsecret-operator-system

# Upgrade to new version
helm upgrade managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.8 \
  --namespace managedsecret-operator-system \
  --reuse-values
```

### Migration Notes

When upgrading from earlier versions:
- CRD updates are handled automatically
- Existing ManagedSecrets continue to work
- No manual migration required
- Check changelog for breaking changes

---

## 📝 Changelog

### Version 1.0.7
- ✨ Added `keepPreviousVersion` for zero-downtime rotation
- ✨ Added `previousVersionTTL` for automatic cleanup
- 📦 Helm chart deployment support
- 📚 Comprehensive examples and documentation
- 🐛 Bug fixes and improvements

---

## License

Proprietary Use License

Permission is granted to use this software for any purpose, including commercial use, without charge.

Modification, adaptation, or creation of derivative works is prohibited.
Redistribution of modified versions is prohibited.

---

**Version:** 1.0.7  
**Registry:** registry.c5ai.ch/charts/managedsecret-operator  
**Minimum Kubernetes:** 1.24+  
**Compatible with:** OpenBao, HashiCorp Vault