This commit implements a complete test infrastructure for validating hot-reload stability and robustness across multiple scenarios. ## New Test Infrastructure ### Test Helpers (tests/helpers/) - TestMetrics: FPS, memory, reload time tracking with statistics - TestReporter: Assertion tracking and formatted test reports - SystemUtils: Memory usage monitoring via /proc/self/status - TestAssertions: Macro-based assertion framework ### Test Modules - TankModule: Realistic module with 50 tanks for production testing - ChaosModule: Crash-injection module for robustness validation - StressModule: Lightweight module for long-duration stability tests ## Integration Test Scenarios ### Scenario 1: Production Hot-Reload (test_01_production_hotreload.cpp) ✅ PASSED - End-to-end hot-reload validation - 30 seconds simulation (1800 frames @ 60 FPS) - TankModule with 50 tanks, realistic state - Source modification (v1.0 → v2.0), recompilation, reload - State preservation: positions, velocities, frameCount - Metrics: ~163ms reload time, 0.88MB memory growth ### Scenario 2: Chaos Monkey (test_02_chaos_monkey.cpp) ✅ PASSED - Extreme robustness testing - 150+ random crashes per run (5% crash probability per frame) - 5 crash types: runtime_error, logic_error, out_of_range, domain_error, state corruption - 100% recovery rate via automatic hot-reload - Corrupted state detection and rejection - Random seed for unpredictable crash patterns - Proof of real reload: temporary files in /tmp/grove_module_*.so ### Scenario 3: Stress Test (test_03_stress_test.cpp) ✅ PASSED - Long-duration stability validation - 10 minutes simulation (36000 frames @ 60 FPS) - 120 hot-reloads (every 5 seconds) - 100% reload success rate (120/120) - Memory growth: 2 MB (threshold: 50 MB) - Avg reload time: 160ms (threshold: 500ms) - No memory leaks, no file descriptor leaks ## Core Engine Enhancements ### ModuleLoader (src/ModuleLoader.cpp) - Temporary file copy to /tmp/ for Linux dlopen cache bypass - Robust reload() method: getState() → unload() → load() → setState() - Automatic cleanup of temporary files - Comprehensive error handling and logging ### DebugEngine (src/DebugEngine.cpp) - Automatic recovery in processModuleSystems() - Exception catching → logging → module reload → continue - Module state dump utilities for debugging ### SequentialModuleSystem (src/SequentialModuleSystem.cpp) - extractModule() for safe module extraction - registerModule() for module re-registration - Enhanced processModules() with error handling ## Build System - CMake configuration for test infrastructure - Shared library compilation for test modules (.so) - CTest integration for all scenarios - PIC flag management for spdlog compatibility ## Documentation (planTI/) - Complete test architecture documentation - Detailed scenario specifications with success criteria - Global test plan and validation thresholds ## Validation Results All 3 integration scenarios pass successfully: - Production hot-reload: State preservation validated - Chaos Monkey: 100% recovery from 150+ crashes - Stress Test: Stable over 120 reloads, minimal memory growth 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
16 KiB
16 KiB
Scénario 5: Multi-Module Orchestration
Priorité: ⭐⭐ IMPORTANT Phase: 2 (SHOULD HAVE) Durée estimée: ~2 minutes Effort implémentation: ~4-6 heures
🎯 Objectif
Valider que le hot-reload d'un module spécifique n'affecte pas les autres modules:
- Isolation complète entre modules
- Ordre d'exécution préservé
- State non corrompu dans modules non-reloadés
- Communication inter-modules fonctionnelle (si applicable)
Critique pour systèmes multi-modules en production.
📋 Description
Setup
Charger 3 modules avec dépendances:
- MapModule: Grille 100x100, pas de dépendance
- TankModule: Dépend de MapModule (positions valides)
- ProductionModule: Spawne des tanks, dépend de TankModule
Scénario
- Exécuter pendant 30 secondes avec les 3 modules
- Hot-reload ProductionModule uniquement à t=15s
- Vérifier que:
- MapModule non affecté (state identique)
- TankModule non affecté (tanks toujours présents)
- ProductionModule rechargé (version mise à jour)
- Ordre d'exécution toujours: Map → Tank → Production
🏗️ Implémentation
Module Dependencies
// MapModule.h
class MapModule : public IModule {
public:
bool isPositionValid(float x, float y) const {
int ix = static_cast<int>(x);
int iy = static_cast<int>(y);
return ix >= 0 && ix < mapSize && iy >= 0 && iy < mapSize;
}
void process(float deltaTime) override {
// Update fog of war, etc.
frameCount++;
}
private:
int mapSize = 100;
int frameCount = 0;
};
// TankModule.h
class TankModule : public IModule {
public:
void setMapModule(MapModule* map) { mapModule = map; }
void process(float deltaTime) override {
for (auto& tank : tanks) {
// Vérifier position valide via MapModule
if (mapModule && !mapModule->isPositionValid(tank.x, tank.y)) {
// Correction position
tank.x = std::clamp(tank.x, 0.0f, 99.0f);
tank.y = std::clamp(tank.y, 0.0f, 99.0f);
}
// Update position
tank.x += tank.vx * deltaTime;
tank.y += tank.vy * deltaTime;
}
frameCount++;
}
int getTankCount() const { return tanks.size(); }
private:
std::vector<Tank> tanks;
MapModule* mapModule = nullptr;
int frameCount = 0;
};
// ProductionModule.h
class ProductionModule : public IModule {
public:
void setTankModule(TankModule* tanks) { tankModule = tanks; }
void process(float deltaTime) override {
timeSinceLastSpawn += deltaTime;
if (timeSinceLastSpawn >= 1.0f) {
// Notifier TankModule de spawner un tank
// (Simplification: on log juste ici)
spawned++;
timeSinceLastSpawn -= 1.0f;
logger->debug("Spawned tank #{}", spawned);
}
frameCount++;
}
private:
TankModule* tankModule = nullptr;
int spawned = 0;
float timeSinceLastSpawn = 0.0f;
int frameCount = 0;
std::string version = "v1.0";
};
Test Principal
// test_05_multimodule.cpp
#include "helpers/TestMetrics.h"
#include "helpers/TestReporter.h"
int main() {
TestReporter reporter("Multi-Module Orchestration");
TestMetrics metrics;
std::cout << "================================================================================\n";
std::cout << "MULTI-MODULE ORCHESTRATION TEST\n";
std::cout << "================================================================================\n\n";
// === SETUP - Charger 3 modules ===
DebugEngine engine;
engine.loadModule("MapModule", "build/modules/libMapModule.so");
engine.loadModule("TankModule", "build/modules/libTankModule.so");
engine.loadModule("ProductionModule", "build/modules/libProductionModule.so");
auto mapConfig = createJsonConfig({{"mapSize", 100}});
auto tankConfig = createJsonConfig({{"tankCount", 50}});
auto prodConfig = createJsonConfig({{"version", "v1.0"}});
engine.initializeModule("MapModule", mapConfig);
engine.initializeModule("TankModule", tankConfig);
engine.initializeModule("ProductionModule", prodConfig);
std::cout << "Loaded 3 modules: Map, Tank, Production\n";
// === PHASE 1: Pre-Reload (15s) ===
std::cout << "\nPhase 1: Running 15s before reload...\n";
for (int frame = 0; frame < 900; frame++) { // 15s * 60 FPS
engine.update(1.0f / 60.0f);
if (frame % 300 == 0) { // Progress toutes les 5s
std::cout << " Frame " << frame << "/900\n";
}
}
// Snapshot states AVANT reload
auto mapStateBefore = engine.getModuleState("MapModule");
auto tankStateBefore = engine.getModuleState("TankModule");
auto prodStateBefore = engine.getModuleState("ProductionModule");
auto* mapJsonBefore = dynamic_cast<JsonDataNode*>(mapStateBefore.get());
auto* tankJsonBefore = dynamic_cast<JsonDataNode*>(tankStateBefore.get());
auto* prodJsonBefore = dynamic_cast<JsonDataNode*>(prodStateBefore.get());
int mapFramesBefore = mapJsonBefore->getJsonData()["frameCount"];
int tankFramesBefore = tankJsonBefore->getJsonData()["frameCount"];
int tankCountBefore = tankJsonBefore->getJsonData()["tanks"].size();
std::string prodVersionBefore = prodJsonBefore->getJsonData()["version"];
std::cout << "\nState snapshot BEFORE reload:\n";
std::cout << " MapModule frames: " << mapFramesBefore << "\n";
std::cout << " TankModule frames: " << tankFramesBefore << "\n";
std::cout << " TankModule tanks: " << tankCountBefore << "\n";
std::cout << " ProductionModule ver: " << prodVersionBefore << "\n\n";
// === HOT-RELOAD ProductionModule UNIQUEMENT ===
std::cout << "Hot-reloading ProductionModule ONLY...\n";
// Modifier version dans source
modifySourceFile("tests/modules/ProductionModule.cpp", "v1.0", "v2.0 HOT-RELOADED");
// Recompiler
system("cmake --build build --target ProductionModule 2>&1 | grep -v '^\\['");
// Reload
auto reloadStart = std::chrono::high_resolution_clock::now();
engine.reloadModule("ProductionModule");
auto reloadEnd = std::chrono::high_resolution_clock::now();
float reloadTime = std::chrono::duration<float, std::milli>(reloadEnd - reloadStart).count();
metrics.recordReloadTime(reloadTime);
std::cout << " → Reload completed in " << reloadTime << "ms\n\n";
// === VÉRIFICATIONS POST-RELOAD ===
std::cout << "Verifying isolation...\n";
auto mapStateAfter = engine.getModuleState("MapModule");
auto tankStateAfter = engine.getModuleState("TankModule");
auto prodStateAfter = engine.getModuleState("ProductionModule");
auto* mapJsonAfter = dynamic_cast<JsonDataNode*>(mapStateAfter.get());
auto* tankJsonAfter = dynamic_cast<JsonDataNode*>(tankStateAfter.get());
auto* prodJsonAfter = dynamic_cast<JsonDataNode*>(prodStateAfter.get());
int mapFramesAfter = mapJsonAfter->getJsonData()["frameCount"];
int tankFramesAfter = tankJsonAfter->getJsonData()["frameCount"];
int tankCountAfter = tankJsonAfter->getJsonData()["tanks"].size();
std::string prodVersionAfter = prodJsonAfter->getJsonData()["version"];
std::cout << "\nState snapshot AFTER reload:\n";
std::cout << " MapModule frames: " << mapFramesAfter << "\n";
std::cout << " TankModule frames: " << tankFramesAfter << "\n";
std::cout << " TankModule tanks: " << tankCountAfter << "\n";
std::cout << " ProductionModule ver: " << prodVersionAfter << "\n\n";
// Assertions: Isolation
bool mapUnaffected = (mapFramesAfter == mapFramesBefore);
bool tankUnaffected = (tankFramesAfter == tankFramesBefore) && (tankCountAfter == tankCountBefore);
bool prodReloaded = (prodVersionAfter.find("v2.0") != std::string::npos);
ASSERT_TRUE(mapUnaffected, "MapModule should be UNAFFECTED by ProductionModule reload");
reporter.addAssertion("map_unaffected", mapUnaffected);
ASSERT_TRUE(tankUnaffected, "TankModule should be UNAFFECTED by ProductionModule reload");
reporter.addAssertion("tank_unaffected", tankUnaffected);
ASSERT_TRUE(prodReloaded, "ProductionModule should be RELOADED with new version");
reporter.addAssertion("production_reloaded", prodReloaded);
// === PHASE 2: Post-Reload (15s) ===
std::cout << "\nPhase 2: Running 15s after reload...\n";
for (int frame = 0; frame < 900; frame++) { // 15s * 60 FPS
engine.update(1.0f / 60.0f);
if (frame % 300 == 0) {
std::cout << " Frame " << frame << "/900\n";
}
}
// Vérifier ordre d'exécution (détection via logs ou instrumentation)
// Pour l'instant, vérifier que tous les modules ont continué de process
auto mapStateFinal = engine.getModuleState("MapModule");
auto tankStateFinal = engine.getModuleState("TankModule");
auto prodStateFinal = engine.getModuleState("ProductionModule");
auto* mapJsonFinal = dynamic_cast<JsonDataNode*>(mapStateFinal.get());
auto* tankJsonFinal = dynamic_cast<JsonDataNode*>(tankStateFinal.get());
auto* prodJsonFinal = dynamic_cast<JsonDataNode*>(prodStateFinal.get());
int mapFramesFinal = mapJsonFinal->getJsonData()["frameCount"];
int tankFramesFinal = tankJsonFinal->getJsonData()["frameCount"];
int prodFramesFinal = prodJsonFinal->getJsonData()["frameCount"];
// Tous devraient avoir ~1800 frames (30s * 60 FPS)
ASSERT_WITHIN(mapFramesFinal, 1800, 50, "MapModule should have ~1800 frames");
ASSERT_WITHIN(tankFramesFinal, 1800, 50, "TankModule should have ~1800 frames");
ASSERT_WITHIN(prodFramesFinal, 900, 50, "ProductionModule should have ~900 frames (restarted)");
reporter.addMetric("map_frames_final", mapFramesFinal);
reporter.addMetric("tank_frames_final", tankFramesFinal);
reporter.addMetric("prod_frames_final", prodFramesFinal);
// === VÉRIFICATIONS FINALES ===
ASSERT_LT(reloadTime, 1000.0f, "Reload time should be < 1s");
reporter.addMetric("reload_time_ms", reloadTime);
// === RAPPORT FINAL ===
std::cout << "\n";
std::cout << "================================================================================\n";
std::cout << "MULTI-MODULE ORCHESTRATION SUMMARY\n";
std::cout << "================================================================================\n";
std::cout << "Isolation Test:\n";
std::cout << " MapModule unaffected: " << (mapUnaffected ? "✓" : "✗") << "\n";
std::cout << " TankModule unaffected: " << (tankUnaffected ? "✓" : "✗") << "\n";
std::cout << " ProductionModule reloaded: " << (prodReloaded ? "✓" : "✗") << "\n\n";
std::cout << "Final Frame Counts:\n";
std::cout << " MapModule: " << mapFramesFinal << " (~1800 expected)\n";
std::cout << " TankModule: " << tankFramesFinal << " (~1800 expected)\n";
std::cout << " ProductionModule: " << prodFramesFinal << " (~900 expected, restarted)\n\n";
std::cout << "Performance:\n";
std::cout << " Reload time: " << reloadTime << "ms\n";
std::cout << "================================================================================\n\n";
reporter.printFinalReport();
return reporter.getExitCode();
}
📊 Métriques Collectées
| Métrique | Description | Seuil |
|---|---|---|
| map_unaffected | MapModule non affecté par reload | true |
| tank_unaffected | TankModule non affecté par reload | true |
| production_reloaded | ProductionModule bien rechargé | true |
| reload_time_ms | Temps de reload | < 1000ms |
| map_frames_final | Frames processées par MapModule | ~1800 |
| tank_frames_final | Frames processées par TankModule | ~1800 |
| prod_frames_final | Frames processées par ProductionModule | ~900 (reset) |
✅ Critères de Succès
MUST PASS
- ✅ MapModule unaffected (frameCount et state identiques)
- ✅ TankModule unaffected (tankCount et frameCount identiques)
- ✅ ProductionModule reloaded (version mise à jour)
- ✅ Reload time < 1s
- ✅ Aucun crash
- ✅ Ordre d'exécution préservé (Map → Tank → Production)
NICE TO HAVE
- ✅ Reload time < 500ms
- ✅ Zero impact sur FPS pendant reload
- ✅ Communication inter-modules fonctionne après reload
🔧 Execution Order Verification
Option 1: Instrumentation dans SequentialModuleSystem
void SequentialModuleSystem::update(float deltaTime) {
logger->trace("╔════════════════════════════════════════");
logger->trace("║ UPDATE CYCLE START");
for (const auto& [name, module] : modules) {
logger->trace("║ → Processing: {}", name);
module->process(deltaTime);
}
logger->trace("║ UPDATE CYCLE END");
logger->trace("╚════════════════════════════════════════");
}
Option 2: Vérification via frameCount delta
// Si l'ordre est Map → Tank → Production:
// - MapModule devrait voir deltaTime en premier
// - Si on reload Production, Map et Tank ne sont pas affectés
// - frameCount de Map et Tank continue linéairement
// - frameCount de Production redémarre à 0
🐛 Cas d'Erreur Attendus
| Erreur | Cause | Action |
|---|---|---|
| MapModule affected | State mal isolé | FAIL - fix state management |
| TankModule affected | Shared memory corruption | FAIL - fix isolation |
| Production not reloaded | Reload raté | FAIL - check reload logic |
| Execution order changed | Module registry corrupted | FAIL - fix registry |
| Crash during reload | Dependency on unloaded module | FAIL - fix dependencies |
📝 Output Attendu
================================================================================
MULTI-MODULE ORCHESTRATION TEST
================================================================================
Loaded 3 modules: Map, Tank, Production
Phase 1: Running 15s before reload...
Frame 0/900
Frame 300/900
Frame 600/900
State snapshot BEFORE reload:
MapModule frames: 900
TankModule frames: 900
TankModule tanks: 50
ProductionModule ver: v1.0
Hot-reloading ProductionModule ONLY...
→ Reload completed in 412ms
Verifying isolation...
State snapshot AFTER reload:
MapModule frames: 900
TankModule frames: 900
TankModule tanks: 50
ProductionModule ver: v2.0 HOT-RELOADED
Phase 2: Running 15s after reload...
Frame 0/900
Frame 300/900
Frame 600/900
================================================================================
MULTI-MODULE ORCHESTRATION SUMMARY
================================================================================
Isolation Test:
MapModule unaffected: ✓
TankModule unaffected: ✓
ProductionModule reloaded: ✓
Final Frame Counts:
MapModule: 1800 (~1800 expected)
TankModule: 1800 (~1800 expected)
ProductionModule: 900 (~900 expected, restarted)
Performance:
Reload time: 412ms
================================================================================
METRICS
================================================================================
Map unaffected: true ✓
Tank unaffected: true ✓
Production reloaded: true ✓
Reload time: 412ms (threshold: < 1000ms) ✓
ASSERTIONS
================================================================================
✓ map_unaffected
✓ tank_unaffected
✓ production_reloaded
✓ reload_time < 1s
Result: ✅ PASSED
================================================================================
📅 Planning
Jour 1 (3h):
- Implémenter MapModule, adapter TankModule avec dépendances
- Implémenter ProductionModule
Jour 2 (3h):
- Implémenter test_05_multimodule.cpp
- Instrumentation pour vérifier ordre d'exécution
- Debug + validation
Prochaine étape: architecture_tests.md (détails des helpers communs)