Gateway: Service Discovery & Registration System #118

New issue

Open

opened 2025-11-14 07:23:38 +01:00 by michael · 0 comments

michael commented

2025-11-14 07:23:38 +01:00

Owner

Overview

Implement service discovery and registration system in furt to support microservice architecture with automatic health monitoring and route management.

Problem

Currently furt only supports static routes defined in furt.conf. Services like sagjan, obr, etc. need dynamic registration with their own configs and automatic health monitoring.

Requirements

1. Service Registration Endpoint

POST /internal/register-service (internal only, not exposed externally)
Request body:

{
  "service": "sagjan",
  "url": "http://127.0.0.1:7812",
  "health_endpoint": "/health",
  "routes": [
    {"method": "POST", "path": "/api/comments/*", "permission": "comments:write"},
    {"method": "GET", "path": "/api/comments/*", "permission": "comments:read"},
    {"method": "PUT", "path": "/api/comments/*", "permission": "comments:moderate"},
    {"method": "DELETE", "path": "/api/comments/*", "permission": "comments:delete"}
  ]
}

Response: 200 OK with registration confirmation

2. Service Registry Storage

services = {
  ["sagjan"] = {
    url = "http://127.0.0.1:7812",
    health_endpoint = "/health",
    status = "UP",  -- UP/DOWN/UNKNOWN
    last_ping = timestamp,
    consecutive_failures = 0,
    routes = {...}
  }
}

3. Health-Check System

Health-Check Loop (background thread/coroutine)
- Ping interval: 30 seconds
- Timeout: 5 seconds
- Max consecutive failures: 3
Auto-Deregistration
- After 3 failed health checks → mark service DOWN
- Remove routes from active routing table
- Continue pinging (reduced frequency: 120s)
- After 10 consecutive failures → stop pinging completely
Auto-Recovery
- When service responds again → mark UP
- Re-add routes to routing table
- Resume normal ping interval (30s)

4. Route Proxying

Pattern Matching for wildcard routes
- /api/comments/* matches /api/comments/test-post
- /api/comments/*/reply matches /api/comments/abc123/reply
Request Forwarding
- Auth happens in furt (API-Key validation)
- Validated request forwarded to service
- Service receives: original path, headers, body, api_key_name
Response Handling
- Forward service response to client
- Add furt headers (X-Gateway, timing)
- Handle service errors gracefully

5. Enhanced /health Endpoint

Extend furt's /health to include service status:

{
  "status": "healthy",
  "service": "furt",
  "version": "1.0.0",
  "services": {
    "sagjan": {
      "status": "UP",
      "url": "http://127.0.0.1:7812",
      "last_ping": "2025-11-14T20:30:00Z",
      "response_time_ms": 5
    },
    "obr": {
      "status": "DOWN",
      "url": "http://127.0.0.1:7820",
      "last_ping": "2025-11-14T20:29:45Z",
      "consecutive_failures": 5
    }
  }
}

6. Configuration

Add to furt.conf:

[gateway]
enabled = true
health_check_interval = 30
health_check_timeout = 5
max_consecutive_failures = 3
stop_ping_after_failures = 10
allow_registration_from = 127.0.0.1, 10.0.0.0/8

Implementation Notes

Security

Registration endpoint only accessible from localhost/trusted IPs
Services cannot override existing static routes
Services cannot register routes for permissions they don't have access to
Rate-limit registration attempts

Crash Recovery

furt crashes: Services re-register on furt restart
Service crashes: furt detects via health checks, deregisters routes

Monitoring

Log all registration/deregistration events
Log health check failures
Track service uptime metrics

Files to Create/Modify

New Files

src/service_registry.lua - Service registration logic
src/health_monitor.lua - Background health checking
src/route_proxy.lua - Request proxying to services
src/routes/internal.lua - Internal endpoints (/internal/register-service)

Modified Files

src/http_server.lua - Add pattern matching for wildcard routes
src/routes/health.lua - Include service status
config/furt.conf.example - Add gateway configuration section

Testing Requirements

Unit Tests

Service registration/deregistration
Health check logic
Pattern matching for routes
Crash recovery scenarios

Integration Tests

Register sagjan, send requests through furt
Simulate sagjan crash, verify deregistration
Simulate furt restart, verify re-registration
Multiple services registered simultaneously

Acceptance Criteria

Services can register dynamically via POST /internal/register-service
furt performs health checks every 30s
Services marked DOWN after 3 failed checks, routes deregistered
Services marked UP when healthy again, routes re-registered
Ping stops after 10 consecutive failures
/health endpoint shows all registered services with status
Pattern matching works for wildcard routes (/api/comments/*)
Requests forwarded correctly to registered services
Service responses proxied back to client
furt restart triggers re-registration from services
All scenarios tested and documented

Priority

High - Required for sagjan integration and future microservice architecture

Effort

Large - Core infrastructure change affecting routing, health monitoring, and request handling

DAW/sagjan#2 - Service-side gateway integration

## Overview Implement service discovery and registration system in furt to support microservice architecture with automatic health monitoring and route management. ## Problem Currently furt only supports static routes defined in furt.conf. Services like sagjan, obr, etc. need dynamic registration with their own configs and automatic health monitoring. ## Requirements ### 1. Service Registration Endpoint - **POST /internal/register-service** (internal only, not exposed externally) - Request body: ```json { "service": "sagjan", "url": "http://127.0.0.1:7812", "health_endpoint": "/health", "routes": [ {"method": "POST", "path": "/api/comments/*", "permission": "comments:write"}, {"method": "GET", "path": "/api/comments/*", "permission": "comments:read"}, {"method": "PUT", "path": "/api/comments/*", "permission": "comments:moderate"}, {"method": "DELETE", "path": "/api/comments/*", "permission": "comments:delete"} ] } ``` - Response: 200 OK with registration confirmation ### 2. Service Registry Storage ```lua services = { ["sagjan"] = { url = "http://127.0.0.1:7812", health_endpoint = "/health", status = "UP", -- UP/DOWN/UNKNOWN last_ping = timestamp, consecutive_failures = 0, routes = {...} } } ``` ### 3. Health-Check System - **Health-Check Loop** (background thread/coroutine) - Ping interval: 30 seconds - Timeout: 5 seconds - Max consecutive failures: 3 - **Auto-Deregistration** - After 3 failed health checks → mark service DOWN - Remove routes from active routing table - Continue pinging (reduced frequency: 120s) - After 10 consecutive failures → stop pinging completely - **Auto-Recovery** - When service responds again → mark UP - Re-add routes to routing table - Resume normal ping interval (30s) ### 4. Route Proxying - **Pattern Matching** for wildcard routes - `/api/comments/*` matches `/api/comments/test-post` - `/api/comments/*/reply` matches `/api/comments/abc123/reply` - **Request Forwarding** - Auth happens in furt (API-Key validation) - Validated request forwarded to service - Service receives: original path, headers, body, api_key_name - **Response Handling** - Forward service response to client - Add furt headers (X-Gateway, timing) - Handle service errors gracefully ### 5. Enhanced /health Endpoint Extend furt's /health to include service status: ```json { "status": "healthy", "service": "furt", "version": "1.0.0", "services": { "sagjan": { "status": "UP", "url": "http://127.0.0.1:7812", "last_ping": "2025-11-14T20:30:00Z", "response_time_ms": 5 }, "obr": { "status": "DOWN", "url": "http://127.0.0.1:7820", "last_ping": "2025-11-14T20:29:45Z", "consecutive_failures": 5 } } } ``` ### 6. Configuration Add to furt.conf: ```ini [gateway] enabled = true health_check_interval = 30 health_check_timeout = 5 max_consecutive_failures = 3 stop_ping_after_failures = 10 allow_registration_from = 127.0.0.1, 10.0.0.0/8 ``` ## Implementation Notes ### Security - Registration endpoint only accessible from localhost/trusted IPs - Services cannot override existing static routes - Services cannot register routes for permissions they don't have access to - Rate-limit registration attempts ### Crash Recovery - **furt crashes:** Services re-register on furt restart - **Service crashes:** furt detects via health checks, deregisters routes ### Monitoring - Log all registration/deregistration events - Log health check failures - Track service uptime metrics ## Files to Create/Modify ### New Files - `src/service_registry.lua` - Service registration logic - `src/health_monitor.lua` - Background health checking - `src/route_proxy.lua` - Request proxying to services - `src/routes/internal.lua` - Internal endpoints (/internal/register-service) ### Modified Files - `src/http_server.lua` - Add pattern matching for wildcard routes - `src/routes/health.lua` - Include service status - `config/furt.conf.example` - Add gateway configuration section ## Testing Requirements ### Unit Tests - Service registration/deregistration - Health check logic - Pattern matching for routes - Crash recovery scenarios ### Integration Tests - Register sagjan, send requests through furt - Simulate sagjan crash, verify deregistration - Simulate furt restart, verify re-registration - Multiple services registered simultaneously ## Acceptance Criteria - [ ] Services can register dynamically via POST /internal/register-service - [ ] furt performs health checks every 30s - [ ] Services marked DOWN after 3 failed checks, routes deregistered - [ ] Services marked UP when healthy again, routes re-registered - [ ] Ping stops after 10 consecutive failures - [ ] /health endpoint shows all registered services with status - [ ] Pattern matching works for wildcard routes (/api/comments/*) - [ ] Requests forwarded correctly to registered services - [ ] Service responses proxied back to client - [ ] furt restart triggers re-registration from services - [ ] All scenarios tested and documented ## Priority **High** - Required for sagjan integration and future microservice architecture ## Effort **Large** - Core infrastructure change affecting routing, health monitoring, and request handling ## Related Issues - DAW/sagjan#2 - Service-side gateway integration