Day 04: Configuration & Secrets Management | EmpowerNow Developer Academy

🎯 Learning Objectives

By the end of today's session, you will be able to:

Navigate the configuration file hierarchy
Understand the difference between env files and Docker secrets
Configure service-specific settings correctly
Use OpenBao (Vault) for sensitive secret management
Apply best practices for configuration management

🏗️ Configuration Hierarchy

EmpowerNow uses a layered configuration system where settings can come from multiple sources, each with different security implications:

Configuration Sources (Highest to Lowest Priority)

🔐

Vault Secrets

🔑 OpenBao/Vault

📜 Dynamic Secrets

🔄 Auto-rotation

▼

🔒

Docker Secrets

postgres-password

jwt-signing-key

api-keys

▼

📁

Service Env Files

env/services/idp.env

env/services/pdp.env

env/services/bff.env

▼

🌐

Root .env

.env (shared vars)

IMAGE_TAG

DOMAIN_SUFFIX

🔐 Security Levels

Different types of configuration data require different security levels:

🔐

Critical Secrets

High Security

Data that, if compromised, would cause severe security breaches. Must be stored in Vault or Docker secrets, never in plain text.

Examples: JWT signing keys, database passwords, API keys for external services, encryption keys, OAuth client secrets

🔒

Sensitive Config

Medium Security

Configuration that contains internal details but isn't directly exploitable. Should be in Docker secrets or protected env files.

Examples: Internal service URLs, database connection strings, Redis passwords, internal ports

📝

General Config

Low Security

Non-sensitive configuration that can be in version control. Feature flags, timeouts, image tags, and public URLs.

Examples: LOG_LEVEL, DOMAIN_SUFFIX, IMAGE_TAG, feature toggles, timeout values

📁 Environment Files

Root .env File

The root .env file contains shared variables used across multiple services. Keep this minimal - only truly shared values belong here.

Deployment/.env

# Root environment variables - shared across services
# Keep this minimal! Service-specific vars go in env/services/

# Image Configuration
IMAGE_TAG=latest
REGISTRY=empowernow.azurecr.io

# Domain Configuration  
DOMAIN_SUFFIX=self.empowernow.ai

# Network Configuration
TRAEFIK_NETWORK=traefik-public

# Logging
LOG_LEVEL=INFO
                

⚠️

Avoid Config Sprawl

Don't add service-specific settings to the root .env. This creates configuration sprawl that DevOps must manually manage. Use env/services/*.env instead.

Service-Specific Env Files

Each service has its own configuration file in env/services/. These files contain literal values only - no variable interpolation.

env/services/idp.env

# IdP Service Configuration
# All values must be literals - no ${VAR} interpolation

# Server Settings
IDP_HOST=0.0.0.0
IDP_PORT=8002
IDP_WORKERS=4

# OIDC Settings
IDP_ISSUER=https://idp.self.empowernow.ai
IDP_TOKEN_EXPIRY=3600
IDP_REFRESH_EXPIRY=86400

# Database
IDP_DATABASE_SCHEMA=idp
IDP_POOL_SIZE=10

# Features
IDP_ENABLE_CAEP=true
IDP_ENABLE_DPOP=true

# Logging
IDP_LOG_LEVEL=INFO
IDP_LOG_FORMAT=json
                

BFF Service Configuration

env/services/bff.env

# BFF Service Configuration
# Backend-for-Frontend - API gateway and auth proxy

# Server Settings
BFF_HOST=0.0.0.0
BFF_PORT=8000

# Session Configuration
BFF_COOKIE_DOMAIN=.self.empowernow.ai
SESSION_LIFETIME=3600

# DCR (Dynamic Client Registration)
DCR_ENABLED=true
DCR_CLIENT_PROFILE=code-flow-pkjwt

# IdP Connection
IDP_BASE_URL=http://idp-app:8002
OIDC_ISSUER=https://idp.self.empowernow.ai/api/oidc

# Logging
BFF_LOG_LEVEL=INFO
                

PDP Service Configuration

env/services/pdp.env

# PDP Service Configuration
# Policy Decision Point - AuthZEN 1.1 authorization

# Server Settings
PDP_HOST=0.0.0.0
PDP_PORT=8001

# Neo4j Connection (policy graph)
PDP_NEO4J_URI=bolt://neo4j:7687

# Redis (caching)
PDP_REDIS_URL=redis://shared-redis:6379/0

# Policy Settings
PDP_POLICY_CACHE_TTL=300
PDP_ENABLE_EXPLANATIONS=true

# Logging
PDP_LOG_LEVEL=INFO
                

Key Configuration Variables

Common variables you'll encounter across services:

Variable	Type	Description
*_HOST	Env	Bind address (usually 0.0.0.0 in containers)
*_PORT	Env	Internal service port
*_DATABASE_URL	Secret	Database connection string with credentials
*_JWT_SECRET	Secret	JWT signing key (critical security)
*_LOG_LEVEL	Env	Logging verbosity (DEBUG, INFO, WARNING, ERROR)
*_REDIS_URL	Env	Redis connection URL

Commonly Modified Variables

Variables you'll frequently adjust during development:

Variable	Service	When to Change
LOG_LEVEL	All	Set to `DEBUG` when troubleshooting issues
IDP_ISSUER	IdP	Different domain or environment (dev/staging/prod)
BFF_COOKIE_DOMAIN	BFF	Different domain for session cookies
DEFAULT_VAULT_PROVIDER	CrudService	Switch between `yaml` (dev) and `openbao` (prod-like)
SESSION_LIFETIME	BFF	Adjust session timeout (seconds)
*_POOL_SIZE	Various	Database connection pool tuning

🔒 Docker Secrets

🔧 IdP bootstrap secrets generation and idp-admin-jwt.txt creation are automated using your organization’s deployment procedures and approved automation.

Docker secrets provide a secure way to pass sensitive data to containers. The secret content is mounted as a file at /run/secrets/SECRET_NAME.

📁 config_secrets/

🔑 postgres-password.txt

🔑 db-conn-string.txt

🔑 jwt-signing-key.txt

🔑 redis-password.txt

🔑 neo4j-password.txt

your_secure_postgres_password_here

postgresql://postgres:password@postgres:5432/empowernow

HS256_secret_key_minimum_32_characters_long

your_redis_password

your_neo4j_password

Defining Secrets in Docker Compose

docker-compose.yml (secrets section)

secrets:
  postgres-password:
    file: ./config_secrets/postgres-password.txt
  db-conn-string:
    file: ./config_secrets/db-conn-string.txt
  jwt-signing-key:
    file: ./config_secrets/jwt-signing-key.txt

services:
  idp:
    image: empowernow.azurecr.io/idp:latest
    secrets:
      - postgres-password
      - jwt-signing-key
    environment:
      # Reference secret file path, not content
      DATABASE_PASSWORD_FILE: /run/secrets/postgres-password
      JWT_SECRET_FILE: /run/secrets/jwt-signing-key
                

💜

Secret File Pattern

Services are configured with *_FILE environment variables pointing to secret files, not * variables with inline values. The service reads the file content at startup.

🔐 OpenBao (Vault) Integration

For production environments and advanced secret management, EmpowerNow integrates with OpenBao (HashiCorp Vault fork) for:

Dynamic Secrets: Generate short-lived database credentials on demand
Secret Rotation: Automatically rotate secrets without service restarts
Audit Logging: Track who accessed what secrets and when
PKI: Certificate authority for service-to-service TLS

Vault Configuration Example

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="idp-role" \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/empowernow" \
    username="vault" \
    password="vault-password"

# Create role for IdP service
vault write database/roles/idp-role \
    db_name=postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA idp TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"
                

💡

Development vs Production

In local development, Docker secrets are sufficient. OpenBao is primarily used in staging/production for dynamic secrets and compliance requirements.

🔄 Vault Provider Patterns

CrudService supports multiple vault backends. Choose the right pattern for your development workflow:

Pattern 1: All-YAML (Fast Local Development)

For rapid iteration without vault infrastructure. Secrets are stored in a plain YAML file - ideal for prototyping and testing.

env/services/crud-service.env (YAML mode)

# Fast local development - no vault container needed
DEFAULT_VAULT_PROVIDER=yaml
YAML_VAULT_PATH=/app/config/data/dev_secrets.yaml

# Secrets stored in plain YAML file
# WARNING: Never use this pattern in production!
                

💜

When to Use YAML Provider

Use YAML provider when you need fast iteration cycles and don't need to test vault-specific features. No OpenBao container required - saves resources on your dev machine.

Pattern 2: OpenBao (Dev Mirrors Production)

When you need to test vault integration or develop features that interact with the secrets API. Uses containerized OpenBao to mimic production behavior.

env/services/crud-service.env (OpenBao mode)

# Testing vault integration locally
DEFAULT_VAULT_PROVIDER=openbao
VAULT_URL=http://openbao:8200
VAULT_TOKEN_FILE=/bootstrap/service-token

# Mimics production vault behavior
# Required for testing secrets API endpoints
                

💡

When to Use OpenBao Provider

Use OpenBao when testing secrets-related features, developing connector credentials, or validating your code will work in production. Requires the openbao container to be running.

Switching Between Providers

To switch providers:

Edit DEFAULT_VAULT_PROVIDER in env/services/crud-service.env
Restart CrudService: docker compose restart crud-service
Verify with health check: curl http://localhost:8000/health

✅ Best Practices

DO:

✅ Keep service-specific settings in env/services/*.env
✅ Use Docker secrets for passwords, keys, and tokens
✅ Use env_file: directive, not environment: blocks
✅ Keep all .env files within the Deployment/ folder
✅ Use literal values in service env files (no ${VAR})
✅ Follow existing patterns for consistency

DON'T:

❌ Add service settings to root .env
❌ Commit secrets to version control
❌ Use variable interpolation in service env files
❌ Reference external env files outside Deployment/
❌ Hardcode secrets in application code
❌ Share secrets across environments

🚨

Security Warning

Never commit config_secrets/ files to git. Ensure they're in .gitignore. Use different secrets for each environment (dev, staging, production).

🔧 Troubleshooting

Common configuration errors and how to fix them:

Symptom	Likely Cause	Fix
Service won't start, missing secret	Docker secret not mounted	Check `config_secrets/` file exists and docker-compose secrets section
`TOKEN_HASH_SALT required`	BFF missing required secret	Create `config_secrets/token-hash-salt.txt` with a random value
`Connection refused` to Redis/Postgres	Service dependency not ready	Wait for health checks or check network configuration
`Invalid vault provider`	Typo in provider name	Use exact names: `yaml`, `openbao`, `hashicorp`
Config change not taking effect	Container using cached config	Restart the service: `docker compose restart <service>`

💜

Quick Debug Commands

Use docker compose logs -f <service> to watch logs in real-time. Check docker compose ps to see which services are healthy vs unhealthy.

🛠️ Hands-on Exercises

Exercise 1: Explore Configuration Files

Navigate to ServiceConfigs/Deployment/
Open the root .env file and note what's configured
Browse env/services/ and examine service-specific configs
Check config_secrets/ for Docker secrets

Exercise 2: Modify IdP Configuration

Open env/services/idp.env
Change IDP_LOG_LEVEL to DEBUG
Restart the IdP service: docker compose restart idp
View logs: docker compose logs -f idp
Observe the more verbose logging

Exercise 3: Create a New Secret

Create a test secret file: echo "test-api-key" > config_secrets/test-api-key.txt
Reference it in docker-compose.yml (don't commit!)
Verify the service can read it
Clean up: remove the test secret

Exercise 4: Switch Vault Provider

Practice switching between vault backends:

Check current provider: open env/services/crud-service.env
If using YAML, switch to OpenBao:

bash

# Edit the env file DEFAULT_VAULT_PROVIDER=openbao VAULT_URL=http://openbao:8200
Restart CrudService: docker compose restart crud-service
Verify vault connectivity in logs: docker compose logs crud-service | grep -i vault
Revert to your original setting after testing

Exercise 5: Add a Custom Environment Variable

Learn the config change workflow:

Add a test variable to env/services/bff.env:

env

# Add at the end of bff.env MY_TEST_VAR=hello-from-config
Restart BFF: docker compose restart bff
Verify the variable is set:

bash

docker exec bff_app printenv | grep MY_TEST
Clean up: remove MY_TEST_VAR from bff.env and restart

🧪 Try It: Verify IdP Is Using New Config

After restarting services with new config, check the IdP is running correctly:

🧪 Try It: Verify BFF Connected to Services

The BFF proxies requests to IdP, PDP, and other services. Check its health:

📋 Day 4 Summary

✅

Configuration Mastery

You now understand the configuration hierarchy, security levels for different types of data, and how to properly manage secrets in the EmpowerNow platform.

Key Takeaways:

Root .env for shared, non-sensitive variables only
Service env files for service-specific configuration
Docker secrets for passwords and keys
OpenBao/Vault for production dynamic secrets
Never commit secrets to version control

Want to Go Deeper?

Advanced DCR, BFF token routing, and vault-backed flows are documented for administrators and support engineers in the official EmpowerNow documentation. This enablement track stays at the operations and verification level.

For detailed vault configuration including multi-instance setups, see the Vault Provider Configuration documentation.

Tomorrow's Preview:

Day 5 is about Exploring Running Services - checking health endpoints, viewing logs, accessing admin interfaces, and debugging common issues.