managedsecret-operator-public/README.md

ManagedSecret Operator

A Kubernetes operator for declarative secret management with automated rotation, drift detection, and zero-downtime password changes.

Overview

The ManagedSecret operator provides:

• 🔐 Generates secrets with configurable password policies
• 🏦 Stores in Vault/OpenBao as the authoritative source
• 🔄 Syncs to Kubernetes Secrets for application consumption
• 🔍 Detects and fixes drift automatically (Vault → K8s)
• 🔁 Rotates secrets on a configurable schedule
• ⏱️ Preserves previous versions during rotation for zero-downtime password changes

Key Principle: Vault/OpenBao is ALWAYS the source of truth

• Changes in Vault → Synced to K8s automatically
• Changes in K8s → Overwritten by Vault values
• Drift detection runs every 1 minute

---

🚀 Quick Start

Prerequisites: Contact your administrator to receive:

• Registry username and password for registry.c5ai.ch
• Confirmation that OpenBao/Vault is configured for your cluster

# 1. Set your credentials (provided by administrator)
export REGISTRY_USER="<provided-by-admin>"
export REGISTRY_PASS="<provided-by-admin>"

# 2. Install Reloader (for automatic pod restarts during rotation)
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

# 3. Login to Helm registry
helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS

# 4. Create imagePullSecret
kubectl create namespace managedsecret-operator-system
kubectl create secret docker-registry registry-pull-secret \
  --docker-server=registry.c5ai.ch \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --namespace managedsecret-operator-system

# 5. Install operator via Helm
helm install managedsecret-operator oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  --set imagePullSecrets[0].name=registry-pull-secret

# 6. Verify installation
kubectl get pods -n managedsecret-operator-system

That's it! Now check the examples folder for comprehensive usage scenarios.

---

📦 What You Get

Helm Chart from registry.c5ai.ch

The operator is distributed as a Helm chart:

• ✅ No building required - Just helm install
• ✅ Versioned releases - Use specific versions or upgrade easily
• ✅ Pre-configured - Sensible defaults, customize via values.yaml
• ✅ Private & secure - Hosted on your infrastructure

What's Deployed

managedsecret-operator-system/
├── CustomResourceDefinition (ManagedSecret)
├── ServiceAccount (controller-manager)
├── Role & RoleBinding (RBAC)
├── ClusterRole & ClusterRoleBinding
├── Deployment (controller-manager)
└── ConfigMap (operator configuration)

---

Prerequisites

On Your Side (User)

• ✅ Kubernetes cluster (1.24+)
• ✅ kubectl configured and working
• ✅ Helm 3.x installed
• ✅ Cluster admin permissions

Provided by Administrator

• ✅ Registry credentials for registry.c5ai.ch
• ✅ OpenBao/Vault configuration for your cluster
  • Vault address (e.g., http://openbao.openbao.svc.cluster.local:8200)
  • Kubernetes auth configured
  • Vault role name (typically managedsecret-operator)

Optional (Recommended)

• ✅ Reloader by Stakater (for automatic pod restarts on rotation)

---

Installation Steps

Step 1: Install Reloader (Recommended)

Reloader automatically restarts pods when secrets change:

# Direct install (simplest)
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

# Via Helm
helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader \
  --namespace reloader \
  --create-namespace

Step 2: Authenticate to Registry

# Set credentials from administrator
export REGISTRY_USER="<your-username>"
export REGISTRY_PASS="<your-password>"

# Login to Helm registry
helm registry login registry.c5ai.ch -u $REGISTRY_USER -p $REGISTRY_PASS

# Create namespace
kubectl create namespace managedsecret-operator-system

# Create imagePullSecret
kubectl create secret docker-registry registry-pull-secret \
  --docker-server=registry.c5ai.ch \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --namespace managedsecret-operator-system

Step 3: Install the Operator

# Install with default configuration
helm install managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  --set imagePullSecrets[0].name=registry-pull-secret

# Or with custom values
cat > values.yaml <<EOF
imagePullSecrets:
  - name: registry-pull-secret

replicaCount: 1

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi
EOF

helm install managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.7 \
  --namespace managedsecret-operator-system \
  -f values.yaml

Step 4: Verify Installation

# Check operator is running
kubectl get pods -n managedsecret-operator-system

# Check CRD is installed
kubectl get crd managedsecrets.secrets.c5ai.ch

# View operator logs
kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager --tail=50

---

📖 Usage

Basic Example: Simple API Key (No Rotation)

apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
  name: api-keys
  namespace: my-app
spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: my-app/api-keys
  
  fields:
  - name: api-key
    type: generated
    generator:
      type: password
      length: 64
      minDigits: 10
      symbolCharacters: ""  # Alphanumeric only
  
  - name: api-endpoint
    type: static
    value: "https://api.example.com/v1"
  
  destination:
    name: api-secret
    type: Opaque
  
  rotation:
    enabled: false

Apply it:

kubectl apply -f managedsecret.yaml

The operator will:

1. Generate a 64-character API key
2. Store it in Vault at secret/data/my-app/api-keys
3. Create Kubernetes Secret api-secret in namespace my-app

Use it in your pod:

env:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: api-secret
      key: api-key

Rotation Example: PostgreSQL with Previous Version

For services that need the old password to authenticate before changing to the new password:

apiVersion: secrets.c5ai.ch/v1alpha1
kind: ManagedSecret
metadata:
  name: postgres-credentials
  namespace: database
spec:
  vault:
    address: "http://openbao.openbao.svc.cluster.local:8200"
    authMethod: kubernetes
    role: managedsecret-operator
    kvVersion: v2
    mount: secret
    path: database/postgres
  
  fields:
  - name: username
    type: static
    value: "app_user"
  
  - name: password
    type: generated
    generator:
      type: password
      length: 40
      minDigits: 8
      minSymbols: 8
  
  destination:
    name: postgres-secret
    type: Opaque
    keepPreviousVersion: true  # Creates postgres-secret-previous
    previousVersionTTL: 1h     # Cleanup after 1 hour
  
  rotation:
    enabled: true
    schedule: 2160h  # 90 days
    rotateGeneratedOnly: true

When rotation happens:

1. New password is generated
2. postgres-secret is updated with new password
3. postgres-secret-previous is created with old password
4. Your CronJob uses old password to authenticate and set new password
5. After 1 hour, -previous secret is automatically cleaned up

Important: You'll need a CronJob to handle the password change. See examples/example-3-postgres-rotation-previous.yaml for a complete implementation.

---

📚 Examples

The examples/ folder contains comprehensive scenarios:

Example	Use Case	Description
Example 1	API Keys	Long-lived credentials without rotation
Example 2	Container Registry	Simple rotation with Reloader
Example 3	PostgreSQL	Rotation with previous version (needs old password)
Example 4	MinIO	Object storage with admin credentials rotation
Example 5	MySQL	Database rotation with multiple strategies
Example 6	Full Stack	Complete application with mixed strategies

See examples/EXAMPLES-SUMMARY.md for detailed explanations and decision trees.

---

🔄 Secret Rotation

How Rotation Works

Day 0: Initial setup
  - ManagedSecret creates secret with generated password
  - Stored in Vault and synced to K8s

Day 90: Rotation triggered (schedule: 2160h)
  1. Operator generates new password
  2. Updates Vault
  3. If keepPreviousVersion=true:
     - Creates <secret-name>-previous with old values
  4. Updates K8s Secret with new values
  5. If Reloader is enabled:
     - Reloader detects change → Restarts pods
  6. After previousVersionTTL:
     - Operator deletes -previous secret

When to Use keepPreviousVersion

Use keepPreviousVersion: true when your service needs the old password to authenticate before changing to the new password:

• ✅ PostgreSQL, MySQL, MariaDB (ALTER USER requires authentication)
• ✅ MinIO (mc admin needs old credentials)
• ✅ LDAP, Active Directory
• ❌ Container registries (htpasswd regenerated fresh)
• ❌ Redis (CONFIG SET doesn't need old password)
• ❌ Web services with startup scripts

Forcing Rotation

# Trigger immediate rotation
kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite

---

🔍 Drift Detection

The operator automatically detects and fixes drift every 60 seconds:

Scenario 1: Secret deleted in Kubernetes

kubectl delete secret my-secret -n my-app
# Operator recreates it from Vault within 60 seconds

Scenario 2: Secret modified in Vault

# Update value in Vault
vault kv put secret/my-app/keys password=newvalue
# Operator syncs to K8s within 60 seconds

Scenario 3: Secret manually edited in K8s

kubectl edit secret my-secret -n my-app
# Changes are overwritten by Vault values within 60 seconds

---

🔧 Troubleshooting

Operator Not Starting

# Check pod status
kubectl get pods -n managedsecret-operator-system

# View logs
kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager --tail=100

# Common issues:
# - ImagePullBackOff: Check imagePullSecret is created correctly
# - CrashLoopBackOff: Check Vault connectivity

Secret Not Created

# Check ManagedSecret status
kubectl get managedsecret <name> -n <namespace> -o yaml

# Look for conditions/events
kubectl describe managedsecret <name> -n <namespace>

# Check operator logs
kubectl logs -n managedsecret-operator-system \
  -l control-plane=controller-manager | grep <name>

Vault Authentication Failing

# Verify ServiceAccount exists
kubectl get sa controller-manager -n managedsecret-operator-system

# Check Vault role configuration
vault read auth/kubernetes/role/managedsecret-operator

# Test authentication manually
kubectl exec -it -n managedsecret-operator-system \
  deployment/managedsecret-operator-controller-manager -- sh
# Inside pod: Check /var/run/secrets/kubernetes.io/serviceaccount/token exists

Rotation Not Working

# Check ManagedSecret schedule
kubectl get managedsecret <name> -n <namespace> -o yaml | grep schedule

# Force rotation to test
kubectl annotate managedsecret <name> -n <namespace> \
  reconcile="$(date +%s)" --overwrite

# Check if Reloader is installed (if using rotation)
kubectl get pods -n reloader

Reloader Not Restarting Pods

# Verify Reloader is running
kubectl get pods -n reloader

# Check pod has correct annotation
kubectl get deployment <name> -n <namespace> -o yaml | grep reloader

# View Reloader logs
kubectl logs -n reloader -l app=reloader

# Required annotation on pod template:
# annotations:
#   reloader.stakater.com/auto: "true"

---

🎯 Best Practices

1. Always Use Reloader for Rotating Secrets

Without Reloader, pods won't pick up new credentials:

template:
  metadata:
    annotations:
      reloader.stakater.com/auto: "true"

2. Set Appropriate Rotation Schedules

• High Security (passwords, admin credentials): 30-90 days
• Medium Security (application credentials): 90-180 days
• Low Security (read-only API keys): 180+ days or disabled

3. Use keepPreviousVersion Correctly

Only use it when the service needs the old password to authenticate:

destination:
  keepPreviousVersion: true  # Only if old password is needed
  previousVersionTTL: 1h     # Adjust based on your workflow

4. Test Rotation Before Production

# Create test ManagedSecret with short rotation
spec:
  rotation:
    enabled: true
    schedule: 5m  # Test rotation every 5 minutes

# Watch the rotation
kubectl get secrets -n <namespace> -w

# Verify application handles rotation
kubectl logs -n <namespace> <pod-name>

5. Monitor Rotation Jobs

For services using keepPreviousVersion, monitor your CronJobs:

# Check job status
kubectl get jobs -n <namespace>

# View job logs
kubectl logs -n <namespace> -l job-name=<rotation-job>

# Set up alerts for failed jobs

6. Backup Vault Regularly

Vault is your source of truth:

# Example: Backup with Velero
velero backup create vault-backup \
  --include-namespaces openbao \
  --snapshot-volumes

7. Version Control Your ManagedSecrets

Store ManagedSecret manifests in Git:

# managedsecrets/
# ├── production/
# │   ├── postgres-credentials.yaml
# │   └── api-keys.yaml
# └── staging/
#     ├── postgres-credentials.yaml
#     └── api-keys.yaml

8. Use GitOps for Deployment

Deploy with ArgoCD or Flux:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: managedsecrets
spec:
  source:
    path: managedsecrets/production
  destination:
    namespace: default

---

🔒 Security Considerations

Production Checklist

• OpenBao/Vault is properly secured (TLS, audit logging, backups)
• RBAC configured with least-privilege
• Network policies in place
• Monitoring and alerting configured
• Disaster recovery plan documented
• Registry credentials stored securely (not in Git)

Credentials Management

The operator needs:

1. Access to Vault (via Kubernetes ServiceAccount auth)
2. Access to pull its image (via imagePullSecret)

Both use Kubernetes-native methods. Never hardcode credentials in manifests!

---

📞 Support

Getting Help

1. Check documentation:

   • This README
   • examples/EXAMPLES-SUMMARY.md
   • Individual example files

2. Check operator logs:

   kubectl logs -n managedsecret-operator-system \
     -l control-plane=controller-manager --tail=100
   

3. Contact your administrator if:

   • You need registry credentials
   • Vault is not configured for your cluster
   • Operator authentication issues

For Administrators

Information to Provide Users

Registry Credentials:

Registry: registry.c5ai.ch
Username: <username>
Password: <password>
Chart: oci://registry.c5ai.ch/charts/managedsecret-operator
Version: 1.0.7

Vault Configuration:

Address: http://openbao.openbao.svc.cluster.local:8200
Auth Method: kubernetes
Role: managedsecret-operator
KV Mount: secret
KV Version: v2

Prerequisites for New Cluster

# 1. Enable Kubernetes auth
bao auth enable kubernetes
bao write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"

# 2. Create policy
bao policy write managedsecret-operator - <<'EOF'
path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/metadata/*" {
  capabilities = ["list", "read", "delete"]
}
EOF

# 3. Create role
bao write auth/kubernetes/role/managedsecret-operator \
    bound_service_account_names=controller-manager \
    bound_service_account_namespaces=managedsecret-operator-system \
    policies=managedsecret-operator \
    ttl=1h

---

🔄 Upgrading

Upgrade Operator Version

# Check current version
helm list -n managedsecret-operator-system

# Upgrade to new version
helm upgrade managedsecret-operator \
  oci://registry.c5ai.ch/charts/managedsecret-operator \
  --version 1.0.8 \
  --namespace managedsecret-operator-system \
  --reuse-values

Migration Notes

When upgrading from earlier versions:

• CRD updates are handled automatically
• Existing ManagedSecrets continue to work
• No manual migration required
• Check changelog for breaking changes

---

📝 Changelog

Version 1.0.7

• ✨ Added keepPreviousVersion for zero-downtime rotation
• ✨ Added previousVersionTTL for automatic cleanup
• 📦 Helm chart deployment support
• 📚 Comprehensive examples and documentation
• 🐛 Bug fixes and improvements

---

License

Proprietary Use License

Permission is granted to use this software for any purpose, including commercial use, without charge.

Modification, adaptation, or creation of derivative works is prohibited. Redistribution of modified versions is prohibited.

---

Version: 1.0.7
Registry: registry.c5ai.ch/charts/managedsecret-operator
Minimum Kubernetes: 1.24+
Compatible with: OpenBao, HashiCorp Vault