GroveEngine/docs/USER_GUIDE.md

GroveEngine User Guide

GroveEngine is a C++17 hot-reload module system designed for building modular applications with runtime code replacement capabilities.

⚠️ IMPORTANT: GroveEngine is experimental and development-ready, NOT production-ready. It is non-deterministic and optimized for rapid prototyping. See main README.md for full limitations.

Table of Contents

1. Overview
2. Core Concepts
3. Project Setup
4. Creating Modules
5. Module Lifecycle
6. Inter-Module Communication
7. Hot-Reload
8. Configuration Management
9. Task Scheduling
10. API Reference

---

Overview

GroveEngine provides:

• Hot-Reload: Replace module code at runtime without losing state
• Modular Architecture: Self-contained modules with clear interfaces
• Pub/Sub Communication: Decoupled inter-module messaging via topics
• State Preservation: Automatic state serialization across reloads
• Configuration Hot-Reload: Update module configuration without code changes

Design Philosophy

• Modules contain pure business logic (200-300 lines recommended)
• No infrastructure code in modules (threading, networking, persistence)
• All data via IDataNode abstraction (backend agnostic)
• Pull-based message processing with callback dispatch (modules control WHEN to process, callbacks handle HOW)

---

Core Concepts

IModule

The base interface all modules implement. Defines the contract for:

• Processing logic (process())
• Configuration (setConfiguration(), getConfiguration())
• State management (getState(), setState())
• Lifecycle (shutdown())

IDataNode

Hierarchical data structure for configuration, state, and messages. Supports:

• Typed accessors (getString(), getInt(), getDouble(), getBool())
• Tree navigation (getChild(), getChildNames())
• Pattern matching (getChildrenByNameMatch())

IIO

Pull-based pub/sub communication with callback dispatch:

• publish(): Send messages to topics
• subscribe(): Register callback handler for topic pattern
• pullAndDispatch(): Pull and auto-dispatch message to handler

ModuleLoader

Handles dynamic loading of .so files:

• load(): Load a module from shared library
• reload(): Hot-reload with state preservation
• unload(): Clean unload

---

Project Setup

Directory Structure

MyProject/
├── CMakeLists.txt
├── external/
│   └── GroveEngine/          # GroveEngine (submodule or symlink)
├── src/
│   ├── main.cpp
│   └── modules/
│       ├── MyModule.h
│       └── MyModule.cpp
└── config/
    └── mymodule.json

CMakeLists.txt

cmake_minimum_required(VERSION 3.20)
project(MyProject VERSION 0.1.0 LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

# ============================================================================
# GroveEngine Integration
# ============================================================================
set(GROVE_BUILD_TESTS OFF CACHE BOOL "Disable GroveEngine tests" FORCE)
add_subdirectory(external/GroveEngine)

# ============================================================================
# Main Executable
# ============================================================================
add_executable(myapp src/main.cpp)

target_link_libraries(myapp PRIVATE
    GroveEngine::impl
    spdlog::spdlog
)

# ============================================================================
# Hot-Reloadable Modules (.so)
# ============================================================================
add_library(MyModule SHARED
    src/modules/MyModule.cpp
)

target_link_libraries(MyModule PRIVATE
    GroveEngine::impl
    spdlog::spdlog
)

set_target_properties(MyModule PROPERTIES
    PREFIX "lib"
    LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/modules
)

# ============================================================================
# Copy config files to build directory
# ============================================================================
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/config/
     DESTINATION ${CMAKE_BINARY_DIR}/config)

# ============================================================================
# Convenience targets
# ============================================================================
add_custom_target(modules
    DEPENDS MyModule
    COMMENT "Building hot-reloadable modules only"
)

Linking GroveEngine

Option 1: Git submodule

git submodule add <grove-engine-repo> external/GroveEngine

Option 2: Symlink (local development)

ln -s /path/to/GroveEngine external/GroveEngine

---

Creating Modules

Module Header

// src/modules/MyModule.h
#pragma once

#include <grove/IModule.h>
#include <grove/IDataNode.h>
#include <grove/IIO.h>
#include <spdlog/spdlog.h>
#include <memory>

namespace myapp {

class MyModule : public grove::IModule {
public:
    MyModule();

    // Required IModule interface
    void process(const grove::IDataNode& input) override;
    void setConfiguration(const grove::IDataNode& configNode,
                          grove::IIO* io,
                          grove::ITaskScheduler* scheduler) override;
    const grove::IDataNode& getConfiguration() override;
    std::unique_ptr<grove::IDataNode> getHealthStatus() override;
    void shutdown() override;
    std::unique_ptr<grove::IDataNode> getState() override;
    void setState(const grove::IDataNode& state) override;
    std::string getType() const override { return "mymodule"; }
    bool isIdle() const override { return true; }

private:
    std::shared_ptr<spdlog::logger> m_logger;
    std::unique_ptr<grove::IDataNode> m_config;
    grove::IIO* m_io = nullptr;

    // Module state (will be preserved across hot-reloads)
    int m_counter = 0;
    std::string m_status;
};

} // namespace myapp

// Required C exports for dynamic loading
extern "C" {
    grove::IModule* createModule();
    void destroyModule(grove::IModule* module);
}

Module Implementation

// src/modules/MyModule.cpp
#include "MyModule.h"
#include <grove/JsonDataNode.h>
#include <spdlog/sinks/stdout_color_sinks.h>

namespace myapp {

MyModule::MyModule() {
    m_logger = spdlog::get("MyModule");
    if (!m_logger) {
        m_logger = spdlog::stdout_color_mt("MyModule");
    }
    m_config = std::make_unique<grove::JsonDataNode>("config");
}

void MyModule::setConfiguration(const grove::IDataNode& configNode,
                                 grove::IIO* io,
                                 grove::ITaskScheduler* scheduler) {
    m_io = io;
    m_config = std::make_unique<grove::JsonDataNode>("config");

    // Read configuration values with defaults
    m_status = configNode.getString("initialStatus", "ready");
    int startCount = configNode.getInt("startCount", 0);

    m_logger->info("MyModule configured: status={}, startCount={}",
                   m_status, startCount);
}

const grove::IDataNode& MyModule::getConfiguration() {
    return *m_config;
}

void MyModule::process(const grove::IDataNode& input) {
    // Get frame timing from input
    double deltaTime = input.getDouble("deltaTime", 0.016);
    int frameCount = input.getInt("frameCount", 0);

    // Your processing logic here
    m_counter++;

    // Process incoming messages (dispatch to registered callbacks)
    while (m_io && m_io->hasMessages() > 0) {
        m_io->pullAndDispatch();  // Callbacks invoked automatically
    }

    // Publish events if needed
    if (m_counter % 100 == 0) {
        auto event = std::make_unique<grove::JsonDataNode>("event");
        event->setInt("counter", m_counter);
        m_io->publish("mymodule:milestone", std::move(event));
    }
}

std::unique_ptr<grove::IDataNode> MyModule::getHealthStatus() {
    auto status = std::make_unique<grove::JsonDataNode>("health");
    status->setString("status", "running");
    status->setInt("counter", m_counter);
    return status;
}

void MyModule::shutdown() {
    m_logger->info("MyModule shutting down, counter={}", m_counter);
}

// ============================================================================
// State Serialization (Critical for Hot-Reload)
// ============================================================================

std::unique_ptr<grove::IDataNode> MyModule::getState() {
    auto state = std::make_unique<grove::JsonDataNode>("state");

    // Serialize all state that must survive hot-reload
    state->setInt("counter", m_counter);
    state->setString("status", m_status);

    m_logger->debug("State saved: counter={}", m_counter);
    return state;
}

void MyModule::setState(const grove::IDataNode& state) {
    // Restore state after hot-reload
    m_counter = state.getInt("counter", 0);
    m_status = state.getString("status", "ready");

    m_logger->info("State restored: counter={}", m_counter);
}

} // namespace myapp

// ============================================================================
// C Export Functions (Required for dlopen)
// ============================================================================

extern "C" {

grove::IModule* createModule() {
    return new myapp::MyModule();
}

void destroyModule(grove::IModule* module) {
    delete module;
}

}

---

Module Lifecycle

┌─────────────────────────────────────────────────────────────────┐
│                        Module Lifecycle                          │
└─────────────────────────────────────────────────────────────────┘

1. LOAD
   ModuleLoader::load(path, name)
        │
        ▼
   dlopen() → createModule()
        │
        ▼
   setConfiguration(config, io, scheduler)
        │
        ▼
   Module Ready

2. PROCESS (Main Loop)
   ┌──────────────────────┐
   │  process(input)      │◄────┐
   │    - Read deltaTime  │     │
   │    - Update state    │     │
   │    - Pull messages   │     │
   │    - Publish events  │     │
   └──────────────────────┘     │
            │                    │
            └────────────────────┘

3. HOT-RELOAD
   File change detected
        │
        ▼
   getState() → Save state
        │
        ▼
   unload() → dlclose()
        │
        ▼
   load(path, name, isReload=true)
        │
        ▼
   setConfiguration(config, io, scheduler)
        │
        ▼
   setState(savedState)
        │
        ▼
   Module continues with preserved state

4. SHUTDOWN
   shutdown()
        │
        ▼
   destroyModule()
        │
        ▼
   dlclose()

---

Inter-Module Communication

IIO Pub/Sub System

Modules communicate via topics using publish/subscribe pattern with callback dispatch.

Key Design: Pull-Based with Callback Dispatch

Unlike traditional push-based systems, IIO gives modules control over WHEN to process messages while callbacks handle HOW to process them:

1. Subscribe with Callback (in setConfiguration): Register handlers for topic patterns
2. Pull and Dispatch (in process): Module controls when to process - callbacks invoked automatically

Benefits:

• No if-forest dispatch: Logic registered at subscription, not scattered in process()
• Module controls timing: Pull-based means deterministic frame ordering
• Thread-safe: Callbacks invoked in module's own thread context
• Clean separation: Subscription setup vs. message processing

Publishing Messages

void MyModule::process(const grove::IDataNode& input) {
    // Create message data
    auto data = std::make_unique<grove::JsonDataNode>("data");
    data->setString("event", "player_moved");
    data->setDouble("x", 100.5);
    data->setDouble("y", 200.3);

    // Publish to topic
    m_io->publish("game:player:position", std::move(data));
}

Subscribing to Topics with Callbacks

void MyModule::setConfiguration(const grove::IDataNode& configNode,
                                 grove::IIO* io,
                                 grove::ITaskScheduler* scheduler) {
    m_io = io;

    // Subscribe to specific topic with callback handler
    m_io->subscribe("game:player:position", [this](const grove::Message& msg) {
        double x = msg.data->getDouble("x", 0.0);
        double y = msg.data->getDouble("y", 0.0);
        // Handle position update...
    });

    // Subscribe with wildcard pattern
    m_io->subscribe("game:player:*", [this](const grove::Message& msg) {
        handlePlayerEvent(msg);
    });

    // Subscribe with low-frequency batching (for non-critical updates)
    grove::SubscriptionConfig config;
    config.batchInterval = 1000;  // 1 second batches
    m_io->subscribeLowFreq("analytics:*", [this](const grove::Message& msg) {
        processBatchedAnalytics(msg);
    }, config);
}

Processing Messages with Callback Dispatch

void MyModule::process(const grove::IDataNode& input) {
    // Pull-based: module controls WHEN to process messages
    // Callbacks registered at subscribe() handle HOW to process
    while (m_io->hasMessages() > 0) {
        m_io->pullAndDispatch();  // Automatically invokes registered callback
    }
}

// No more if-forest dispatch - callbacks were registered at subscription:
// subscribe("game:player:position", [this](const Message& msg) { ... });

Topic Patterns

Topics support wildcard matching:

Pattern	Matches
game:player:*	game:player:position, game:player:health
economy:*	economy:prices, economy:trade
*:error	network:error, database:error

---

Hot-Reload

How It Works

1. File Watcher detects .so modification
2. State Extraction: getState() serializes module state
3. Unload: Old library closed with dlclose()
4. Load: New library loaded with dlopen() (cache bypass via temp copy)
5. Configure: setConfiguration() called with same config
6. Restore: setState() restores serialized state

Implementing Hot-Reload Support

// Critical: Serialize ALL state that must survive reload
std::unique_ptr<grove::IDataNode> MyModule::getState() {
    auto state = std::make_unique<grove::JsonDataNode>("state");

    // Primitives
    state->setInt("counter", m_counter);
    state->setDouble("health", m_health);
    state->setString("name", m_name);
    state->setBool("active", m_active);

    // Complex state: serialize to JSON child nodes
    auto entitiesNode = std::make_unique<grove::JsonDataNode>("entities");
    for (const auto& entity : m_entities) {
        auto entityNode = std::make_unique<grove::JsonDataNode>(entity.id);
        entityNode->setDouble("x", entity.x);
        entityNode->setDouble("y", entity.y);
        entitiesNode->setChild(entity.id, std::move(entityNode));
    }
    state->setChild("entities", std::move(entitiesNode));

    return state;
}

void MyModule::setState(const grove::IDataNode& state) {
    // Restore primitives
    m_counter = state.getInt("counter", 0);
    m_health = state.getDouble("health", 100.0);
    m_name = state.getString("name", "default");
    m_active = state.getBool("active", true);

    // Restore complex state
    m_entities.clear();
    auto* entitiesNode = state.getChildReadOnly("entities");
    if (entitiesNode) {
        for (const auto& name : entitiesNode->getChildNames()) {
            auto* entityNode = entitiesNode->getChildReadOnly(name);
            if (entityNode) {
                Entity e;
                e.id = name;
                e.x = entityNode->getDouble("x", 0.0);
                e.y = entityNode->getDouble("y", 0.0);
                m_entities.push_back(e);
            }
        }
    }
}

File Watcher Example

class FileWatcher {
public:
    void watch(const std::string& path) {
        if (fs::exists(path)) {
            m_lastModified[path] = fs::last_write_time(path);
        }
    }

    bool hasChanged(const std::string& path) {
        if (!fs::exists(path)) return false;

        auto currentTime = fs::last_write_time(path);
        auto it = m_lastModified.find(path);

        if (it == m_lastModified.end()) {
            m_lastModified[path] = currentTime;
            return false;
        }

        if (currentTime != it->second) {
            it->second = currentTime;
            return true;
        }
        return false;
    }

private:
    std::unordered_map<std::string, fs::file_time_type> m_lastModified;
};

---

Configuration Management

JSON Configuration Files

// config/mymodule.json
{
    "initialStatus": "ready",
    "startCount": 0,
    "maxItems": 100,
    "debugMode": false,
    "updateRate": 0.016
}

Loading Configuration

std::unique_ptr<grove::JsonDataNode> loadConfig(const std::string& path) {
    if (fs::exists(path)) {
        std::ifstream file(path);
        nlohmann::json j;
        file >> j;
        return std::make_unique<grove::JsonDataNode>("config", j);
    }
    // Return empty config with defaults
    return std::make_unique<grove::JsonDataNode>("config");
}

Runtime Config Updates

Modules can optionally support runtime configuration changes:

bool MyModule::updateConfig(const grove::IDataNode& newConfig) {
    // Validate new config
    int maxItems = newConfig.getInt("maxItems", 100);
    if (maxItems < 1 || maxItems > 10000) {
        m_logger->warn("Invalid maxItems: {}", maxItems);
        return false;
    }

    // Apply changes
    m_maxItems = maxItems;
    m_debugMode = newConfig.getBool("debugMode", false);

    m_logger->info("Config updated: maxItems={}", m_maxItems);
    return true;
}

---

Task Scheduling

For computationally expensive operations, delegate to ITaskScheduler:

void MyModule::setConfiguration(const grove::IDataNode& configNode,
                                 grove::IIO* io,
                                 grove::ITaskScheduler* scheduler) {
    m_scheduler = scheduler;  // Store reference
}

void MyModule::process(const grove::IDataNode& input) {
    // Delegate expensive pathfinding calculation
    if (needsPathfinding) {
        auto taskData = std::make_unique<grove::JsonDataNode>("task");
        taskData->setDouble("startX", unit.x);
        taskData->setDouble("startY", unit.y);
        taskData->setDouble("targetX", target.x);
        taskData->setDouble("targetY", target.y);
        taskData->setString("unitId", unit.id);

        m_scheduler->scheduleTask("pathfinding", std::move(taskData));
    }

    // Check for completed tasks
    while (m_scheduler->hasCompletedTasks() > 0) {
        auto result = m_scheduler->getCompletedTask();
        std::string unitId = result->getString("unitId", "");
        // Apply pathfinding result...
    }
}

---

API Reference

IDataNode

Method	Description
getString(name, default)	Get string property
getInt(name, default)	Get integer property
getDouble(name, default)	Get double property
getBool(name, default)	Get boolean property
setString(name, value)	Set string property
setInt(name, value)	Set integer property
setDouble(name, value)	Set double property
setBool(name, value)	Set boolean property
hasProperty(name)	Check if property exists
getChild(name)	Get child node (transfers ownership)
getChildReadOnly(name)	Get child node (no ownership transfer)
setChild(name, node)	Add/replace child node
getChildNames()	Get names of all children

IIO

Method	Description
publish(topic, data)	Publish message to topic
subscribe(pattern, handler, config)	Subscribe with callback handler
subscribeLowFreq(pattern, handler, config)	Subscribe with batching and callback
hasMessages()	Count of pending messages
pullAndDispatch()	Pull and auto-dispatch message to handler
getHealth()	Get IO health metrics

IModule

Method	Description
process(input)	Main processing method
setConfiguration(config, io, scheduler)	Initialize module
getConfiguration()	Get current config
getState()	Serialize state for hot-reload
setState(state)	Restore state after hot-reload
getHealthStatus()	Get module health report
shutdown()	Clean shutdown
isIdle()	Check if safe to hot-reload
getType()	Get module type identifier

ModuleLoader

Method	Description
load(path, name, isReload)	Load module from .so
reload(module)	Hot-reload with state preservation
unload()	Unload current module
isLoaded()	Check if module is loaded
getLoadedPath()	Get path of loaded module

IOFactory

Method	Description
create(type, instanceId)	Create IO by type string
create(IOType, instanceId)	Create IO by enum
createFromConfig(config, instanceId)	Create IO from config

IO Types:

• "intra" / IOType::INTRA - Same-process (development)
• "local" / IOType::LOCAL - Same-machine (production single-server)
• "network" / IOType::NETWORK - Distributed (MMO scale)

---

Building and Running

# Configure
cmake -B build

# Build everything
cmake --build build -j4

# Build modules only (for hot-reload workflow)
cmake --build build --target modules

# Run
./build/myapp

Hot-Reload Workflow

1. Start application: ./build/myapp
2. Edit module source: src/modules/MyModule.cpp
3. Rebuild module: cmake --build build --target modules
4. Application detects change and hot-reloads automatically

---

Best Practices

1. Keep modules small: 200-300 lines of pure business logic
2. No infrastructure code: Let GroveEngine handle threading, persistence
3. Serialize all state: Everything in getState() survives hot-reload
4. Use typed accessors: getInt(), getString() with sensible defaults
5. Pull-based messaging: Process messages in process(), not callbacks
6. Validate config: Check configuration values in setConfiguration()
7. Log appropriately: Debug for development, Info for production events