GroveEngine/planTI/scenario_05_multimodule.md
StillHammer d8c5f93429 feat: Add comprehensive hot-reload test suite with 3 integration scenarios
This commit implements a complete test infrastructure for validating
hot-reload stability and robustness across multiple scenarios.

## New Test Infrastructure

### Test Helpers (tests/helpers/)
- TestMetrics: FPS, memory, reload time tracking with statistics
- TestReporter: Assertion tracking and formatted test reports
- SystemUtils: Memory usage monitoring via /proc/self/status
- TestAssertions: Macro-based assertion framework

### Test Modules
- TankModule: Realistic module with 50 tanks for production testing
- ChaosModule: Crash-injection module for robustness validation
- StressModule: Lightweight module for long-duration stability tests

## Integration Test Scenarios

### Scenario 1: Production Hot-Reload (test_01_production_hotreload.cpp)
 PASSED - End-to-end hot-reload validation
- 30 seconds simulation (1800 frames @ 60 FPS)
- TankModule with 50 tanks, realistic state
- Source modification (v1.0 → v2.0), recompilation, reload
- State preservation: positions, velocities, frameCount
- Metrics: ~163ms reload time, 0.88MB memory growth

### Scenario 2: Chaos Monkey (test_02_chaos_monkey.cpp)
 PASSED - Extreme robustness testing
- 150+ random crashes per run (5% crash probability per frame)
- 5 crash types: runtime_error, logic_error, out_of_range, domain_error, state corruption
- 100% recovery rate via automatic hot-reload
- Corrupted state detection and rejection
- Random seed for unpredictable crash patterns
- Proof of real reload: temporary files in /tmp/grove_module_*.so

### Scenario 3: Stress Test (test_03_stress_test.cpp)
 PASSED - Long-duration stability validation
- 10 minutes simulation (36000 frames @ 60 FPS)
- 120 hot-reloads (every 5 seconds)
- 100% reload success rate (120/120)
- Memory growth: 2 MB (threshold: 50 MB)
- Avg reload time: 160ms (threshold: 500ms)
- No memory leaks, no file descriptor leaks

## Core Engine Enhancements

### ModuleLoader (src/ModuleLoader.cpp)
- Temporary file copy to /tmp/ for Linux dlopen cache bypass
- Robust reload() method: getState() → unload() → load() → setState()
- Automatic cleanup of temporary files
- Comprehensive error handling and logging

### DebugEngine (src/DebugEngine.cpp)
- Automatic recovery in processModuleSystems()
- Exception catching → logging → module reload → continue
- Module state dump utilities for debugging

### SequentialModuleSystem (src/SequentialModuleSystem.cpp)
- extractModule() for safe module extraction
- registerModule() for module re-registration
- Enhanced processModules() with error handling

## Build System
- CMake configuration for test infrastructure
- Shared library compilation for test modules (.so)
- CTest integration for all scenarios
- PIC flag management for spdlog compatibility

## Documentation (planTI/)
- Complete test architecture documentation
- Detailed scenario specifications with success criteria
- Global test plan and validation thresholds

## Validation Results
All 3 integration scenarios pass successfully:
- Production hot-reload: State preservation validated
- Chaos Monkey: 100% recovery from 150+ crashes
- Stress Test: Stable over 120 reloads, minimal memory growth

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 22:13:07 +08:00

16 KiB

Scénario 5: Multi-Module Orchestration

Priorité: IMPORTANT Phase: 2 (SHOULD HAVE) Durée estimée: ~2 minutes Effort implémentation: ~4-6 heures


🎯 Objectif

Valider que le hot-reload d'un module spécifique n'affecte pas les autres modules:

  • Isolation complète entre modules
  • Ordre d'exécution préservé
  • State non corrompu dans modules non-reloadés
  • Communication inter-modules fonctionnelle (si applicable)

Critique pour systèmes multi-modules en production.


📋 Description

Setup

Charger 3 modules avec dépendances:

  1. MapModule: Grille 100x100, pas de dépendance
  2. TankModule: Dépend de MapModule (positions valides)
  3. ProductionModule: Spawne des tanks, dépend de TankModule

Scénario

  1. Exécuter pendant 30 secondes avec les 3 modules
  2. Hot-reload ProductionModule uniquement à t=15s
  3. Vérifier que:
    • MapModule non affecté (state identique)
    • TankModule non affecté (tanks toujours présents)
    • ProductionModule rechargé (version mise à jour)
    • Ordre d'exécution toujours: Map → Tank → Production

🏗️ Implémentation

Module Dependencies

// MapModule.h
class MapModule : public IModule {
public:
    bool isPositionValid(float x, float y) const {
        int ix = static_cast<int>(x);
        int iy = static_cast<int>(y);
        return ix >= 0 && ix < mapSize && iy >= 0 && iy < mapSize;
    }

    void process(float deltaTime) override {
        // Update fog of war, etc.
        frameCount++;
    }

private:
    int mapSize = 100;
    int frameCount = 0;
};

// TankModule.h
class TankModule : public IModule {
public:
    void setMapModule(MapModule* map) { mapModule = map; }

    void process(float deltaTime) override {
        for (auto& tank : tanks) {
            // Vérifier position valide via MapModule
            if (mapModule && !mapModule->isPositionValid(tank.x, tank.y)) {
                // Correction position
                tank.x = std::clamp(tank.x, 0.0f, 99.0f);
                tank.y = std::clamp(tank.y, 0.0f, 99.0f);
            }

            // Update position
            tank.x += tank.vx * deltaTime;
            tank.y += tank.vy * deltaTime;
        }
        frameCount++;
    }

    int getTankCount() const { return tanks.size(); }

private:
    std::vector<Tank> tanks;
    MapModule* mapModule = nullptr;
    int frameCount = 0;
};

// ProductionModule.h
class ProductionModule : public IModule {
public:
    void setTankModule(TankModule* tanks) { tankModule = tanks; }

    void process(float deltaTime) override {
        timeSinceLastSpawn += deltaTime;

        if (timeSinceLastSpawn >= 1.0f) {
            // Notifier TankModule de spawner un tank
            // (Simplification: on log juste ici)
            spawned++;
            timeSinceLastSpawn -= 1.0f;
            logger->debug("Spawned tank #{}", spawned);
        }
        frameCount++;
    }

private:
    TankModule* tankModule = nullptr;
    int spawned = 0;
    float timeSinceLastSpawn = 0.0f;
    int frameCount = 0;
    std::string version = "v1.0";
};

Test Principal

// test_05_multimodule.cpp
#include "helpers/TestMetrics.h"
#include "helpers/TestReporter.h"

int main() {
    TestReporter reporter("Multi-Module Orchestration");
    TestMetrics metrics;

    std::cout << "================================================================================\n";
    std::cout << "MULTI-MODULE ORCHESTRATION TEST\n";
    std::cout << "================================================================================\n\n";

    // === SETUP - Charger 3 modules ===
    DebugEngine engine;

    engine.loadModule("MapModule", "build/modules/libMapModule.so");
    engine.loadModule("TankModule", "build/modules/libTankModule.so");
    engine.loadModule("ProductionModule", "build/modules/libProductionModule.so");

    auto mapConfig = createJsonConfig({{"mapSize", 100}});
    auto tankConfig = createJsonConfig({{"tankCount", 50}});
    auto prodConfig = createJsonConfig({{"version", "v1.0"}});

    engine.initializeModule("MapModule", mapConfig);
    engine.initializeModule("TankModule", tankConfig);
    engine.initializeModule("ProductionModule", prodConfig);

    std::cout << "Loaded 3 modules: Map, Tank, Production\n";

    // === PHASE 1: Pre-Reload (15s) ===
    std::cout << "\nPhase 1: Running 15s before reload...\n";

    for (int frame = 0; frame < 900; frame++) { // 15s * 60 FPS
        engine.update(1.0f / 60.0f);

        if (frame % 300 == 0) { // Progress toutes les 5s
            std::cout << "  Frame " << frame << "/900\n";
        }
    }

    // Snapshot states AVANT reload
    auto mapStateBefore = engine.getModuleState("MapModule");
    auto tankStateBefore = engine.getModuleState("TankModule");
    auto prodStateBefore = engine.getModuleState("ProductionModule");

    auto* mapJsonBefore = dynamic_cast<JsonDataNode*>(mapStateBefore.get());
    auto* tankJsonBefore = dynamic_cast<JsonDataNode*>(tankStateBefore.get());
    auto* prodJsonBefore = dynamic_cast<JsonDataNode*>(prodStateBefore.get());

    int mapFramesBefore = mapJsonBefore->getJsonData()["frameCount"];
    int tankFramesBefore = tankJsonBefore->getJsonData()["frameCount"];
    int tankCountBefore = tankJsonBefore->getJsonData()["tanks"].size();
    std::string prodVersionBefore = prodJsonBefore->getJsonData()["version"];

    std::cout << "\nState snapshot BEFORE reload:\n";
    std::cout << "  MapModule frames:     " << mapFramesBefore << "\n";
    std::cout << "  TankModule frames:    " << tankFramesBefore << "\n";
    std::cout << "  TankModule tanks:     " << tankCountBefore << "\n";
    std::cout << "  ProductionModule ver: " << prodVersionBefore << "\n\n";

    // === HOT-RELOAD ProductionModule UNIQUEMENT ===
    std::cout << "Hot-reloading ProductionModule ONLY...\n";

    // Modifier version dans source
    modifySourceFile("tests/modules/ProductionModule.cpp", "v1.0", "v2.0 HOT-RELOADED");

    // Recompiler
    system("cmake --build build --target ProductionModule 2>&1 | grep -v '^\\['");

    // Reload
    auto reloadStart = std::chrono::high_resolution_clock::now();
    engine.reloadModule("ProductionModule");
    auto reloadEnd = std::chrono::high_resolution_clock::now();

    float reloadTime = std::chrono::duration<float, std::milli>(reloadEnd - reloadStart).count();
    metrics.recordReloadTime(reloadTime);

    std::cout << "  → Reload completed in " << reloadTime << "ms\n\n";

    // === VÉRIFICATIONS POST-RELOAD ===
    std::cout << "Verifying isolation...\n";

    auto mapStateAfter = engine.getModuleState("MapModule");
    auto tankStateAfter = engine.getModuleState("TankModule");
    auto prodStateAfter = engine.getModuleState("ProductionModule");

    auto* mapJsonAfter = dynamic_cast<JsonDataNode*>(mapStateAfter.get());
    auto* tankJsonAfter = dynamic_cast<JsonDataNode*>(tankStateAfter.get());
    auto* prodJsonAfter = dynamic_cast<JsonDataNode*>(prodStateAfter.get());

    int mapFramesAfter = mapJsonAfter->getJsonData()["frameCount"];
    int tankFramesAfter = tankJsonAfter->getJsonData()["frameCount"];
    int tankCountAfter = tankJsonAfter->getJsonData()["tanks"].size();
    std::string prodVersionAfter = prodJsonAfter->getJsonData()["version"];

    std::cout << "\nState snapshot AFTER reload:\n";
    std::cout << "  MapModule frames:     " << mapFramesAfter << "\n";
    std::cout << "  TankModule frames:    " << tankFramesAfter << "\n";
    std::cout << "  TankModule tanks:     " << tankCountAfter << "\n";
    std::cout << "  ProductionModule ver: " << prodVersionAfter << "\n\n";

    // Assertions: Isolation
    bool mapUnaffected = (mapFramesAfter == mapFramesBefore);
    bool tankUnaffected = (tankFramesAfter == tankFramesBefore) && (tankCountAfter == tankCountBefore);
    bool prodReloaded = (prodVersionAfter.find("v2.0") != std::string::npos);

    ASSERT_TRUE(mapUnaffected, "MapModule should be UNAFFECTED by ProductionModule reload");
    reporter.addAssertion("map_unaffected", mapUnaffected);

    ASSERT_TRUE(tankUnaffected, "TankModule should be UNAFFECTED by ProductionModule reload");
    reporter.addAssertion("tank_unaffected", tankUnaffected);

    ASSERT_TRUE(prodReloaded, "ProductionModule should be RELOADED with new version");
    reporter.addAssertion("production_reloaded", prodReloaded);

    // === PHASE 2: Post-Reload (15s) ===
    std::cout << "\nPhase 2: Running 15s after reload...\n";

    for (int frame = 0; frame < 900; frame++) { // 15s * 60 FPS
        engine.update(1.0f / 60.0f);

        if (frame % 300 == 0) {
            std::cout << "  Frame " << frame << "/900\n";
        }
    }

    // Vérifier ordre d'exécution (détection via logs ou instrumentation)
    // Pour l'instant, vérifier que tous les modules ont continué de process
    auto mapStateFinal = engine.getModuleState("MapModule");
    auto tankStateFinal = engine.getModuleState("TankModule");
    auto prodStateFinal = engine.getModuleState("ProductionModule");

    auto* mapJsonFinal = dynamic_cast<JsonDataNode*>(mapStateFinal.get());
    auto* tankJsonFinal = dynamic_cast<JsonDataNode*>(tankStateFinal.get());
    auto* prodJsonFinal = dynamic_cast<JsonDataNode*>(prodStateFinal.get());

    int mapFramesFinal = mapJsonFinal->getJsonData()["frameCount"];
    int tankFramesFinal = tankJsonFinal->getJsonData()["frameCount"];
    int prodFramesFinal = prodJsonFinal->getJsonData()["frameCount"];

    // Tous devraient avoir ~1800 frames (30s * 60 FPS)
    ASSERT_WITHIN(mapFramesFinal, 1800, 50, "MapModule should have ~1800 frames");
    ASSERT_WITHIN(tankFramesFinal, 1800, 50, "TankModule should have ~1800 frames");
    ASSERT_WITHIN(prodFramesFinal, 900, 50, "ProductionModule should have ~900 frames (restarted)");

    reporter.addMetric("map_frames_final", mapFramesFinal);
    reporter.addMetric("tank_frames_final", tankFramesFinal);
    reporter.addMetric("prod_frames_final", prodFramesFinal);

    // === VÉRIFICATIONS FINALES ===

    ASSERT_LT(reloadTime, 1000.0f, "Reload time should be < 1s");
    reporter.addMetric("reload_time_ms", reloadTime);

    // === RAPPORT FINAL ===
    std::cout << "\n";
    std::cout << "================================================================================\n";
    std::cout << "MULTI-MODULE ORCHESTRATION SUMMARY\n";
    std::cout << "================================================================================\n";
    std::cout << "Isolation Test:\n";
    std::cout << "  MapModule unaffected:      " << (mapUnaffected ? "✓" : "✗") << "\n";
    std::cout << "  TankModule unaffected:     " << (tankUnaffected ? "✓" : "✗") << "\n";
    std::cout << "  ProductionModule reloaded: " << (prodReloaded ? "✓" : "✗") << "\n\n";

    std::cout << "Final Frame Counts:\n";
    std::cout << "  MapModule:        " << mapFramesFinal << " (~1800 expected)\n";
    std::cout << "  TankModule:       " << tankFramesFinal << " (~1800 expected)\n";
    std::cout << "  ProductionModule: " << prodFramesFinal << " (~900 expected, restarted)\n\n";

    std::cout << "Performance:\n";
    std::cout << "  Reload time:      " << reloadTime << "ms\n";
    std::cout << "================================================================================\n\n";

    reporter.printFinalReport();

    return reporter.getExitCode();
}

📊 Métriques Collectées

Métrique Description Seuil
map_unaffected MapModule non affecté par reload true
tank_unaffected TankModule non affecté par reload true
production_reloaded ProductionModule bien rechargé true
reload_time_ms Temps de reload < 1000ms
map_frames_final Frames processées par MapModule ~1800
tank_frames_final Frames processées par TankModule ~1800
prod_frames_final Frames processées par ProductionModule ~900 (reset)

Critères de Succès

MUST PASS

  1. MapModule unaffected (frameCount et state identiques)
  2. TankModule unaffected (tankCount et frameCount identiques)
  3. ProductionModule reloaded (version mise à jour)
  4. Reload time < 1s
  5. Aucun crash
  6. Ordre d'exécution préservé (Map → Tank → Production)

NICE TO HAVE

  1. Reload time < 500ms
  2. Zero impact sur FPS pendant reload
  3. Communication inter-modules fonctionne après reload

🔧 Execution Order Verification

Option 1: Instrumentation dans SequentialModuleSystem

void SequentialModuleSystem::update(float deltaTime) {
    logger->trace("╔════════════════════════════════════════");
    logger->trace("║ UPDATE CYCLE START");

    for (const auto& [name, module] : modules) {
        logger->trace("║ → Processing: {}", name);
        module->process(deltaTime);
    }

    logger->trace("║ UPDATE CYCLE END");
    logger->trace("╚════════════════════════════════════════");
}

Option 2: Vérification via frameCount delta

// Si l'ordre est Map → Tank → Production:
// - MapModule devrait voir deltaTime en premier
// - Si on reload Production, Map et Tank ne sont pas affectés
// - frameCount de Map et Tank continue linéairement
// - frameCount de Production redémarre à 0

🐛 Cas d'Erreur Attendus

Erreur Cause Action
MapModule affected State mal isolé FAIL - fix state management
TankModule affected Shared memory corruption FAIL - fix isolation
Production not reloaded Reload raté FAIL - check reload logic
Execution order changed Module registry corrupted FAIL - fix registry
Crash during reload Dependency on unloaded module FAIL - fix dependencies

📝 Output Attendu

================================================================================
MULTI-MODULE ORCHESTRATION TEST
================================================================================

Loaded 3 modules: Map, Tank, Production

Phase 1: Running 15s before reload...
  Frame 0/900
  Frame 300/900
  Frame 600/900

State snapshot BEFORE reload:
  MapModule frames:     900
  TankModule frames:    900
  TankModule tanks:     50
  ProductionModule ver: v1.0

Hot-reloading ProductionModule ONLY...
  → Reload completed in 412ms

Verifying isolation...

State snapshot AFTER reload:
  MapModule frames:     900
  TankModule frames:    900
  TankModule tanks:     50
  ProductionModule ver: v2.0 HOT-RELOADED

Phase 2: Running 15s after reload...
  Frame 0/900
  Frame 300/900
  Frame 600/900

================================================================================
MULTI-MODULE ORCHESTRATION SUMMARY
================================================================================
Isolation Test:
  MapModule unaffected:      ✓
  TankModule unaffected:     ✓
  ProductionModule reloaded: ✓

Final Frame Counts:
  MapModule:        1800 (~1800 expected)
  TankModule:       1800 (~1800 expected)
  ProductionModule: 900 (~900 expected, restarted)

Performance:
  Reload time:      412ms
================================================================================

METRICS
================================================================================
  Map unaffected:        true           ✓
  Tank unaffected:       true           ✓
  Production reloaded:   true           ✓
  Reload time:           412ms          (threshold: < 1000ms) ✓

ASSERTIONS
================================================================================
  ✓ map_unaffected
  ✓ tank_unaffected
  ✓ production_reloaded
  ✓ reload_time < 1s

Result: ✅ PASSED

================================================================================

📅 Planning

Jour 1 (3h):

  • Implémenter MapModule, adapter TankModule avec dépendances
  • Implémenter ProductionModule

Jour 2 (3h):

  • Implémenter test_05_multimodule.cpp
  • Instrumentation pour vérifier ordre d'exécution
  • Debug + validation

Prochaine étape: architecture_tests.md (détails des helpers communs)