GroveEngine/docs/architecture/data-tree-system.md

IDataTree System Architecture

Overview

The IDataTree system is a unified hierarchical data management system for configuration, persistent data, and runtime state. It provides a flexible, abstract interface that can be backed by multiple storage formats (JSON, Binary, Database, etc.).

Core Concepts

Three Data Domains

The system manages three separate data trees with different characteristics:

root/
├── config/    (Read-only, hot-reload enabled, distributed)
├── data/      (Read-write, saved to disk, local per Engine)
└── runtime/   (Read-write, temporary, never saved)

config/ - Shared Configuration

• Purpose: Game configuration, unit stats, modding
• Access: Read-only for modules
• Source: Loaded from files, synchronized from Coordinator
• Hot-reload: Automatic detection and reload of changed files
• Distribution: Synchronized across all Engines via Coordinator

data/ - Persistent Data

• Purpose: Player saves, campaign progress, unlocks, statistics
• Access: Read-write for modules (via IIO publish)
• Source: Local files per Engine
• Persistence: Saved to disk on request
• Isolation: Each Engine maintains its own data/

runtime/ - Temporary State

• Purpose: Current game state, caches, temporary calculations
• Access: Read-write for modules
• Lifecycle: Exists only in memory, never saved
• Scope: Strictly local to each Engine instance

Architecture Layers

Layer 1: Interfaces (Abstract)

IDataValue    // Abstract data value (type-safe access)
IDataNode     // Tree node (navigation, search, modification)
IDataTree     // Root container (config/data/runtime management)

Key principle: Modules and systems work ONLY with interfaces, never concrete implementations.

Layer 2: Concrete Implementations

JsonDataValue     // nlohmann::json backed value
JsonDataNode      // JSON tree node with full feature set
JsonDataTree      // File-based JSON storage

// Future implementations:
BinaryDataValue   // Custom binary format
DatabaseDataNode  // SQL backend
NetworkDataTree   // Remote data server

Pluggable backends: Change implementation via DataTreeFactory::create(type, path)

Layer 3: Module System Integration

IModuleSystem
├── Owns IDataTree instance
├── Provides config to modules via setConfiguration()
├── Listens to IIO for save requests
└── Persists data to Tree

Responsibilities:

• Initialize Tree (local or synchronized)
• Extract and provide config nodes to modules
• Handle save/load requests from modules via IIO
• Trigger hot-reload notifications

Layer 4: Distributed Coordination

CoordinationModule (Master)
├── Master config tree
├── Hot-reload detection
└── Broadcast updates via IIO

DebugEngine (Workers)
├── Local tree replica
├── Subscribe to config updates
└── Apply synchronized config

Data Flow Patterns

Pattern 1: Module Reads Configuration

// Engine startup
auto tankConfig = tree->getConfigRoot()->getChild("tanks")->getChild("heavy");
tankModule->setConfiguration(*tankConfig, io, scheduler);

// Module uses config
void TankModule::setConfiguration(const IDataNode& config, ...) {
    m_armor = config.getInt("armor");
    m_speed = config.getDouble("speed");
    m_weaponType = config.getString("weapon_type");
}

Pattern 2: Module Saves State

// Module publishes save request via IIO
void TankModule::onDestroy() {
    auto state = createDataNode({
        {"position", {x, y}},
        {"health", currentHealth},
        {"ammo", ammoCount}
    });

    m_io->publish("save:tank:state:123", std::move(state));
}

// Engine listens and persists
void Engine::handleSaveRequests() {
    while (m_io->hasMessages()) {
        auto msg = m_io->pullMessage();

        if (msg.topic.starts_with("save:")) {
            std::string path = extractPath(msg.topic);
            m_tree->getDataRoot()->setChild(path, std::move(msg.data));
            m_tree->saveNode("data/" + path);
        }
    }
}

Pattern 3: Config Hot-Reload (Distributed)

// COORDINATEUR: Master config with hot-reload
void CoordinationModule::tick() {
    if (m_masterTree->reloadIfChanged()) {
        auto config = m_masterTree->getConfigRoot();
        m_networkIO->publish("config:reload", std::move(config));
    }
}

// ENGINE: Receives and applies config
void DebugEngine::processNetworkMessages() {
    auto msg = m_networkIO->pullMessage();

    if (msg.topic == "config:reload") {
        auto configRoot = m_tree->getNode("config");
        configRoot->clearChildren();
        configRoot->setChild("updated", std::move(msg.data));

        // Notify all modules
        for (auto& module : m_modules) {
            auto moduleConfig = configRoot->getChild(module->getType());
            module->setConfiguration(*moduleConfig, m_io, m_scheduler);
        }
    }
}

Advanced Features

Pattern Matching Search

// Find all heavy units
auto heavyUnits = configRoot->getChildrenByNameMatch("*_heavy_*");

// Find all tank variants
auto tanks = configRoot->getChildrenByNameMatch("tank_*");

Property-Based Queries

// Find all units with armor > 150
auto heavyArmored = configRoot->queryByProperty("armor",
    [](const IDataValue& val) {
        return val.isNumber() && val.asInt() > 150;
    });

Hash-Based Validation

// Check if config changed
std::string currentHash = configNode->getTreeHash();
if (currentHash != lastKnownHash) {
    // Config changed, refresh caches
    rebuildLookupTables();
    lastKnownHash = currentHash;
}

Implementation Guidelines

For Module Developers

DO:

• Read config via getInt(), getString(), getBool(), etc.
• Publish save requests via IIO: m_io->publish("save:module:state", data)
• Keep modules under 300 lines
• Use IDataNode interface only, never concrete types

DON'T:

• Directly access IDataTree
• Create nodes manually (use factory or IIO)
• Assume JSON format (work with IDataValue interface)
• Cache config pointers (config can be reloaded)

For Engine Implementers

DO:

• Create one IDataTree per Engine instance
• Subscribe to "save:*" pattern for save requests
• Call reloadIfChanged() periodically (or use callbacks)
• Provide isolated config subtrees to modules

DON'T:

• Share IDataTree between Engines (use synchronization instead)
• Allow modules to access root directly
• Block on save operations (make async if needed)

For System Architects

DO:

• Use Coordinator pattern for config distribution
• Keep data/ isolated per Engine
• Use IIO for all cross-component communication
• Consider multiple IDataTree implementations for different needs

DON'T:

• Synchronize data/ across Engines (defeats isolation)
• Put business logic in IDataTree implementations
• Create tight coupling between Tree and Modules

File Structure Example

gamedata/
├── config/
│   ├── tanks.json
│   ├── weapons.json
│   ├── buildings.json
│   └── mods/
│       └── super_mod/
│           ├── new_tanks.json
│           └── new_weapons.json
├── data/
│   ├── campaign_progress.json
│   ├── unlocked_tech.json
│   └── player_stats.json
└── runtime/
    (in-memory only, not on disk)

Benefits

For Development

• Modular: Swap backends without changing module code
• Testable: Mock IDataTree for unit tests
• Hot-reload: Instant feedback during development
• Type-safe: Compile-time checks with IDataNode interface

For Operations

• Distributed: Config synchronized across cluster
• Isolated: Each Engine manages its own saves
• Auditable: Hash validation for data integrity
• Flexible: JSON for dev, Binary for production

For Modding

• Accessible: JSON files are human-readable
• Safe: Read-only access prevents corruption
• Extensible: Mods add files without conflicts
• Hot-loadable: Changes apply without restart

Future Enhancements

Planned Features

• Binary format implementation (BinaryDataTree)
• Database backend (DatabaseDataTree)
• Differential updates (send only changed config)
• Compression for network sync
• Versioning and migration system

Potential Extensions

• Encryption for sensitive data
• Cloud storage backend (S3DataTree)
• Real-time collaboration (multiple writers)
• Conflict resolution for distributed writes
• Schema validation and type checking

---

Status: Production-ready ✅ Version: 1.0.0 Last Updated: 2025-10-28