DAW/furt

Fork 0

Config change detection and validation for production deployments #84

New issue

Closed

opened 2025-06-24 19:48:32 +02:00 by Michael · 2 comments

Michael commented

2025-06-24 19:48:32 +02:00

(Migrated from gitea.dragons-at-work.de)

Problem

Production server configs exist only on servers (not in Git), leading to config-drift issues:

Local: config/server.lua (in Git, without secrets)
walter: /etc/furt/server.lua (production config, not versioned)
aitvaras: /etc/furt/server.lua (production config, not versioned)

When local config/server.lua adds new fields or changes structure, production configs become outdated but we have no detection mechanism.

Real-World Scenario

Developer adds new CORS configuration to config/server.lua
Deploys API code to walter
API fails to start because walter's server.lua lacks new required fields
Debug time: 30+ minutes to figure out config mismatch

Requirements

Config schema validation - API validates config on startup
Missing field detection - API warns about missing config keys
Config diff tool - Compare local template vs remote config
Pre-deploy validation - Check config compatibility before deployment
Config migration guide - Help for updating production configs

Proposed Implementation

1. Config Schema Validation (startup.lua)

-- Validate required config fields on API startup
local required_fields = {
    "host", "port", "mail.smtp_server", "cors.allowed_origins"
}

function validate_config(config)
    for _, field in ipairs(required_fields) do
        if not get_nested_field(config, field) then
            error("Missing required config field: " .. field)
        end
    end
end

2. Config Health Endpoint

GET /health/config
{
    "config_valid": true,
    "missing_fields": [],
    "deprecated_fields": ["old_setting"],
    "config_version": "1.2.0"
}

3. Pre-Deploy Config Check

# scripts/check_config_compatibility.sh walter
echo "Checking config compatibility with walter..."

# Get remote config structure
REMOTE_FIELDS=$(curl -s http://walter:8080/health/config | jq -r '.fields[]')

# Check local config requirements  
LOCAL_REQUIRED=$(lua -e 'print(required_fields)' config/server.lua)

# Show missing fields
if [ differences ]; then
    echo "⚠️  CONFIG COMPATIBILITY ISSUES:"
    echo "Missing on walter: field1, field2"
    echo "New in local: field3, field4"
    echo ""
    echo "Update walter config before deployment? (y/N)"
fi

4. Config Template Generator

# scripts/generate_config_template.sh walter
echo "Generating walter config template..."

# Take local config/server.lua
# Replace secrets with placeholders
# Add walter-specific defaults
# Output: config-walter-template.lua

cat > config-walter-template.lua << EOF
-- walter production config template
-- Copy to walter:/etc/furt/server.lua and adjust secrets

return {
    host = "127.0.0.1",
    port = 8080,
    
    cors = {
        allowed_origins = {
            "https://walter.dragons-at-work.de",  -- walter-specific
            -- ADD YOUR DOMAINS HERE
        }
    },
    
    mail = {
        smtp_server = "mail.dragons-at-work.de",
        username = "REPLACE_WITH_SMTP_USER",      -- ← EDIT THIS
        password = "REPLACE_WITH_SMTP_PASSWORD",  -- ← EDIT THIS
    }
}
EOF

Use Cases

Development: Validate local config changes before commit
Pre-Deploy: Check walter/aitvaras config compatibility
Production: Quick diagnosis of config-related startup failures
Migration: Guided config updates when structure changes

Priority

High - This prevents production deployment failures and reduces debug time.

Files to Create

src/config_validator.lua
scripts/check_config_compatibility.sh
scripts/generate_config_template.sh
docs/CONFIG_MIGRATION.md

API versioning (DAW/furt#83) - Config version should be part of API version info

## Problem Production server configs exist only on servers (not in Git), leading to config-drift issues: - **Local:** config/server.lua (in Git, without secrets) - **walter:** /etc/furt/server.lua (production config, not versioned) - **aitvaras:** /etc/furt/server.lua (production config, not versioned) When local config/server.lua adds new fields or changes structure, production configs become outdated but we have no detection mechanism. ## Real-World Scenario 1. Developer adds new CORS configuration to config/server.lua 2. Deploys API code to walter 3. API fails to start because walter's server.lua lacks new required fields 4. Debug time: 30+ minutes to figure out config mismatch ## Requirements - [ ] **Config schema validation** - API validates config on startup - [ ] **Missing field detection** - API warns about missing config keys - [ ] **Config diff tool** - Compare local template vs remote config - [ ] **Pre-deploy validation** - Check config compatibility before deployment - [ ] **Config migration guide** - Help for updating production configs ## Proposed Implementation ### 1. Config Schema Validation (startup.lua) ```lua -- Validate required config fields on API startup local required_fields = { "host", "port", "mail.smtp_server", "cors.allowed_origins" } function validate_config(config) for _, field in ipairs(required_fields) do if not get_nested_field(config, field) then error("Missing required config field: " .. field) end end end ``` ### 2. Config Health Endpoint ``` GET /health/config { "config_valid": true, "missing_fields": [], "deprecated_fields": ["old_setting"], "config_version": "1.2.0" } ``` ### 3. Pre-Deploy Config Check ```bash # scripts/check_config_compatibility.sh walter echo "Checking config compatibility with walter..." # Get remote config structure REMOTE_FIELDS=$(curl -s http://walter:8080/health/config | jq -r '.fields[]') # Check local config requirements LOCAL_REQUIRED=$(lua -e 'print(required_fields)' config/server.lua) # Show missing fields if [ differences ]; then echo "⚠️ CONFIG COMPATIBILITY ISSUES:" echo "Missing on walter: field1, field2" echo "New in local: field3, field4" echo "" echo "Update walter config before deployment? (y/N)" fi ``` ### 4. Config Template Generator ```bash # scripts/generate_config_template.sh walter echo "Generating walter config template..." # Take local config/server.lua # Replace secrets with placeholders # Add walter-specific defaults # Output: config-walter-template.lua cat > config-walter-template.lua << EOF -- walter production config template -- Copy to walter:/etc/furt/server.lua and adjust secrets return { host = "127.0.0.1", port = 8080, cors = { allowed_origins = { "https://walter.dragons-at-work.de", -- walter-specific -- ADD YOUR DOMAINS HERE } }, mail = { smtp_server = "mail.dragons-at-work.de", username = "REPLACE_WITH_SMTP_USER", -- ← EDIT THIS password = "REPLACE_WITH_SMTP_PASSWORD", -- ← EDIT THIS } } EOF ``` ## Use Cases - **Development:** Validate local config changes before commit - **Pre-Deploy:** Check walter/aitvaras config compatibility - **Production:** Quick diagnosis of config-related startup failures - **Migration:** Guided config updates when structure changes ## Priority **High** - This prevents production deployment failures and reduces debug time. ## Files to Create - src/config_validator.lua - scripts/check_config_compatibility.sh - scripts/generate_config_template.sh - docs/CONFIG_MIGRATION.md ## Related Issues - API versioning (DAW/furt#83) - Config version should be part of API version info

michael added this to the v0.1.2 - Gateway Basics milestone 2025-08-14 05:20:48 +02:00

michael referenced this issue

2025-08-14 06:27:03 +02:00

Repository Redesign: Clean structure for v0.1.0 Open Source release #86

michael commented

2025-08-15 08:48:46 +02:00

Owner

SCOPE-ÄNDERUNG - Teil der Deployment-Modernisierung

Config-Change-Detection ist eng verknüpft mit DAW/furt#87 (Wiki + Helper Scripts):

Pre-Deploy Config-Checks → Teil des Deploy-Workflows
Config-Migration → Teil der Helper-Scripts
Config-Validation → Teil des Setup-Prozesses

Neue Kategorisierung: Deployment-Infrastruktur (nicht standalone)

Integration in DAW/furt#87: Config-Management-Komponente

**SCOPE-ÄNDERUNG - Teil der Deployment-Modernisierung** Config-Change-Detection ist eng verknüpft mit DAW/furt#87 (Wiki + Helper Scripts): - Pre-Deploy Config-Checks → Teil des Deploy-Workflows - Config-Migration → Teil der Helper-Scripts - Config-Validation → Teil des Setup-Prozesses **Neue Kategorisierung:** Deployment-Infrastruktur (nicht standalone) **Integration in DAW/furt#87:** Config-Management-Komponente

michael referenced this issue

2025-08-15 08:52:55 +02:00

Implement API file versioning system for deployment tracking #83

michael referenced this issue

2025-08-15 13:29:14 +02:00

Git-based Deployment System: karl/erna → walter2 Pipeline #90

michael referenced this issue

2025-08-15 17:27:33 +02:00

Session Handover: Issue-Cleanup + Deployment-Pipeline Status #92

michael referenced this issue

2025-08-15 17:59:10 +02:00

Git-based Deployment System: karl/erna → walter2 Pipeline #90