StillHammer/seo-generator-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9dbb6be5dc
seo-generator-server/CLAUDE.md

15 KiB
Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Node.js-based SEO content generation server that creates SEO-optimized content using multiple LLMs with anti-detection mechanisms. The system operates in two exclusive modes: MANUAL (web interface + API) or AUTO (batch processing from Google Sheets).

Development Commands

Server Operations

npm start                    # Start in MANUAL mode (default)
npm start -- --mode=manual  # Explicitly start MANUAL mode  
npm start -- --mode=auto    # Start in AUTO mode
SERVER_MODE=auto npm start  # Start AUTO mode via environment

Production Workflow Execution

# Execute real production workflow from Google Sheets
node -e "const main = require('./lib/Main'); main.handleFullWorkflow({ rowNumber: 2, source: 'production' });"

# Test with different rows
node -e "const main = require('./lib/Main'); main.handleFullWorkflow({ rowNumber: 3, source: 'production' });"

Testing Commands

# Test suites
npm run test:all             # Complete test suite
npm run test:light           # Light test runner
npm run test:smoke           # Smoke tests only
npm run test:llm             # LLM connectivity tests
npm run test:content         # Content generation tests
npm run test:integration     # Integration tests
npm run test:systematic      # Systematic module testing
npm run test:basic           # Basic validation only

# Individual test categories
npm run test:ai-validation   # AI content validation
npm run test:dashboard       # Test dashboard server

Google Sheets Integration Tests

# Test personality loading
node -e "const {getPersonalities} = require('./lib/BrainConfig'); getPersonalities().then(p => console.log(\`\${p.length} personalities loaded\`));"

# Test CSV data loading
node -e "const {readInstructionsData} = require('./lib/BrainConfig'); readInstructionsData(2).then(d => console.log('Data:', d));"

# Test random personality selection  
node -e "const {selectPersonalityWithAI, getPersonalities} = require('./lib/BrainConfig'); getPersonalities().then(p => selectPersonalityWithAI('test', 'test', p)).then(r => console.log('Selected:', r.nom));"

LLM Connectivity Tests

node -e "require('./lib/LLMManager').testLLMManager()"         # Basic LLM connectivity
node -e "require('./lib/LLMManager').testLLMManagerComplete()" # Full LLM provider test suite

Complete System Test

node -e "
const main = require('./lib/Main');
const testData = {
  csvData: {
    mc0: 'plaque personnalisée',
    t0: 'Créer une plaque personnalisée unique',
    personality: { nom: 'Marc', style: 'professionnel' },
    tMinus1: 'décoration personnalisée',
    mcPlus1: 'plaque gravée,plaque métal,plaque bois,plaque acrylique',
    tPlus1: 'Plaque Gravée Premium,Plaque Métal Moderne,Plaque Bois Naturel,Plaque Acrylique Design'
  },
  xmlTemplate: Buffer.from(\`<?xml version='1.0' encoding='UTF-8'?>
<article>
  <h1>|Titre_Principal{{T0}}{Rédige un titre H1 accrocheur}|</h1>
  <intro>|Introduction{{MC0}}{Rédige une introduction engageante}|</intro>
</article>\`).toString('base64'),
  source: 'node_server_test'
};
main.handleFullWorkflow(testData);
"

Architecture Overview

Dual Mode System

The server operates in two mutually exclusive modes controlled by lib/modes/ModeManager.js:

  • MANUAL Mode (lib/modes/ManualServer.js): Web interface, API endpoints, WebSocket for real-time logs
  • AUTO Mode (lib/modes/AutoProcessor.js): Batch processing from Google Sheets without web interface

Core Workflow Pipeline (lib/Main.js)

  1. Data Preparation - Read from Google Sheets (CSV data + XML templates)
  2. Element Extraction - Parse XML elements with embedded instructions
  3. Missing Keywords Generation - Auto-complete missing data using LLMs
  4. Direct Content Generation - Generate all content elements in parallel
  5. Multi-LLM Enhancement - 4-stage processing pipeline across different LLM providers
  6. Content Assembly - Inject generated content back into XML structure
  7. Organic Compilation & Storage - Save clean text to Google Sheets

Google Sheets Integration

  • Authentication: Via GOOGLE_SERVICE_ACCOUNT_EMAIL and GOOGLE_PRIVATE_KEY environment variables
  • Data Sources:
    • Instructions sheet: Columns A-I (slug, T0, MC0, T-1, L-1, MC+1, T+1, L+1, XML template)
    • Personnalites sheet: 15 AI personalities for content variety
    • Generated_Articles sheet: Final compiled text output with metadata

Multi-LLM Modular Enhancement System

Architecture 100% Modulaire avec sauvegarde versionnée :

Workflow Principal (lib/Main.js)

  1. Data Preparation - Read from Google Sheets (CSV data + XML templates)
  2. Element Extraction - Parse XML elements with embedded instructions
  3. Missing Keywords Generation - Auto-complete missing data using LLMs
  4. Simple Generation - Generate base content with Claude
  5. Selective Enhancement - Couches modulaires configurables
  6. Adversarial Enhancement - Anti-détection modulaire
  7. Human Simulation - Erreurs humaines réalistes
  8. Pattern Breaking - Cassage patterns LLM
  9. Content Assembly & Storage - Final compilation avec versioning

Couches Modulaires Disponibles

  • 5 Selective Stacks : lightEnhancement → fullEnhancement → adaptive
  • 5 Adversarial Modes : none → light → standard → heavy → adaptive
  • 6 Human Simulation Modes : none → lightSimulation → personalityFocus → adaptive
  • 7 Pattern Breaking Modes : none → syntaxFocus → connectorsFocus → adaptive

Sauvegarde Versionnée

  • v1.0 : Génération initiale Claude
  • v1.1 : Post Selective Enhancement
  • v1.2 : Post Adversarial Enhancement
  • v1.3 : Post Human Simulation
  • v1.4 : Post Pattern Breaking
  • v2.0 : Version finale

Supported LLM providers: Claude, OpenAI, Gemini, Deepseek, Moonshot, Mistral

Personality System (lib/BrainConfig.js:265-340)

Random Selection Process:

  1. Load 15 personalities from Google Sheets
  2. Fisher-Yates shuffle for true randomness
  3. Select 60% (9 personalities) per generation
  4. AI chooses best match within random subset
  5. Temperature = 1.0 for maximum variability

15 Available Personalities: Marc (technical), Sophie (déco), Laurent (commercial), Julie (architecture), Kévin (terrain), Amara (engineering), Mamadou (artisan), Émilie (digital), Pierre-Henri (heritage), Yasmine (greentech), Fabrice (metallurgy), Chloé (content), Linh (manufacturing), Minh (design), Thierry (creole)

Centralized Logging System (LogSh)

Architecture

  • All logging must go through logSh() function in lib/ErrorReporting.js
  • Multi-output streams: Console (formatted) + File (JSON) + WebSocket (real-time)
  • Never use console.* or other loggers directly

Log Levels and Usage

  • TRACE: Hierarchical workflow execution with parameters (▶ ✔ ✖ symbols)
  • DEBUG: Detailed debugging information (visible in files with debug level)
  • INFO: Standard operational messages
  • WARN: Warning conditions
  • ERROR: Error conditions with stack traces

File Logging

  • Format: JSON structured logs in timestamped files
  • Location: logs/seo-generator-YYYY-MM-DD_HH-MM-SS.log
  • Flush behavior: Immediate flush on every log call to prevent buffer loss
  • Level: DEBUG and above (includes all TRACE logs)

Trace System

  • Hierarchical execution tracking: Using AsyncLocalStorage for span context
  • Function parameters: All tracer.run() calls include relevant parameters
  • Format: Function names with file prefixes (e.g., "Main.handleFullWorkflow()")
  • Performance timing: Start/end with duration measurements
  • Error handling: Automatic stack trace logging on failures

Log Consultation (LogViewer)

Les logs ne sont plus envoyés en console.log (trop verbeux). Tous les événements sont enregistrés dans logs/app.log au format JSONL Pino.

Un outil tools/logViewer.js permet d'interroger facilement ce fichier:

# Voir les 200 dernières lignes formatées
node tools/logViewer.js --pretty

# Rechercher un mot-clé dans les messages
node tools/logViewer.js --search --includes "Claude" --pretty

# Rechercher par plage de temps (tous les logs du 2 septembre 2025)
node tools/logViewer.js --since 2025-09-02T00:00:00Z --until 2025-09-02T23:59:59Z --pretty

# Filtrer par niveau d'erreur
node tools/logViewer.js --last 300 --level ERROR --pretty

Filtres disponibles:

  • --level: 30=INFO, 40=WARN, 50=ERROR (ou INFO, WARN, ERROR)
  • --module: filtre par path ou module
  • --includes: mot-clé dans msg
  • --regex: expression régulière sur msg
  • --since / --until: bornes temporelles (ISO ou YYYY-MM-DD)

Real-time Log Viewing

  • WebSocket server on port 8081
  • Auto-launched tools/logs-viewer.html in Edge browser
  • Features: Search, level filtering, scroll preservation

Key Components

lib/Main.js

Architecture Modulaire Complète - Orchestration workflow avec pipeline configurable et sauvegarde versionnée.

lib/selective-enhancement/

Couches Selective Modulaires :

  • SelectiveCore.js - Application couche par couche
  • SelectiveLayers.js - 5 stacks prédéfinis + adaptatif
  • TechnicalLayer.js - Enhancement technique OpenAI
  • TransitionLayer.js - Enhancement transitions Gemini
  • StyleLayer.js - Enhancement style Mistral
  • SelectiveUtils.js - Utilitaires + génération simple (remplace ContentGeneration.js)

lib/adversarial-generation/

Anti-détection Modulaire :

  • AdversarialCore.js - Moteur adversarial principal
  • AdversarialLayers.js - 5 modes défense configurables
  • DetectorStrategies.js - Stratégies anti-détection interchangeables

lib/human-simulation/

Simulation Erreurs Humaines :

  • HumanSimulationCore.js - Moteur simulation principal
  • HumanSimulationLayers.js - 6 modes simulation
  • FatiguePatterns.js - Patterns fatigue réalistes
  • PersonalityErrors.js - Erreurs spécifiques personnalité
  • TemporalStyles.js - Variations temporelles

lib/pattern-breaking/

Cassage Patterns LLM :

  • PatternBreakingCore.js - Moteur pattern breaking
  • PatternBreakingLayers.js - 7 modes cassage
  • LLMFingerprints.js - Suppression empreintes LLM
  • SyntaxVariations.js - Variations syntaxiques
  • NaturalConnectors.js - Connecteurs naturels

lib/post-processing/

Post-traitement Legacy (remplacé par modules ci-dessus)

lib/LLMManager.js

Multi-LLM provider management with retry logic, rate limiting, and provider rotation.

lib/BrainConfig.js

Google Sheets integration, personality system, and random selection algorithms.

lib/ElementExtraction.js

XML parsing and element extraction with instruction parsing ({{variables}} vs {instructions}).

lib/ArticleStorage.js

Organic text compilation maintaining natural hierarchy and Google Sheets storage.

lib/ErrorReporting.js

Centralized logging system with hierarchical tracing and multi-output streams.

Environment Configuration

Required environment variables in .env:

# Google Sheets Integration
GOOGLE_SERVICE_ACCOUNT_EMAIL=your-service-account@project.iam.gserviceaccount.com
GOOGLE_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
GOOGLE_SHEETS_ID=your_sheets_id

# LLM API Keys
ANTHROPIC_API_KEY=your_anthropic_key
OPENAI_API_KEY=your_openai_key  
GOOGLE_API_KEY=your_google_key
DEEPSEEK_API_KEY=your_deepseek_key
MOONSHOT_API_KEY=your_moonshot_key
MISTRAL_API_KEY=your_mistral_key

# Optional Configuration
LOG_LEVEL=INFO
MAX_COST_PER_ARTICLE=1.00
SERVER_MODE=manual

Tools

Bundle Tool

node tools/pack-lib.cjs              # default → code.js
node tools/pack-lib.cjs --out out.js # custom output
node tools/pack-lib.cjs --order alpha
node tools/pack-lib.cjs --entry lib/test-manual.js

pack-lib.cjs creates a single code.js from all files in lib/. Each file is concatenated with an ASCII header showing its path. Imports/exports are kept, so the bundle is for reading/audit only, not execution.

Unused Code Audit

node tools/audit-unused.cjs # Report dead files and unused exports

Important Development Notes

  • Architecture 100% Modulaire: Ancien système séquentiel supprimé, backup dans /backup/sequential-system/
  • Configuration Granulaire: Chaque couche modulaire indépendamment configurable
  • Sauvegarde Versionnée: v1.0 → v1.1 → v1.2 → v1.3 → v1.4 → v2.0 pour traçabilité complète
  • Compatibility Layer: Interface handleFullWorkflow() maintenue pour rétrocompatibilité
  • Personality system uses randomization: 60% of 15 personalities selected per generation run
  • All data sourced from Google Sheets: No hardcoded JSON files or static data
  • Default XML templates: Auto-generated when column I contains filenames
  • Organic compilation: Maintains natural text flow in final output
  • Temperature = 1.0: Ensures maximum variability in AI responses
  • Trace system: Uses AsyncLocalStorage for hierarchical execution tracking
  • 5/6 LLM providers operational: Gemini may be geo-blocked in some regions

Migration Legacy → Modulaire

  • ❌ Supprimé: lib/ContentGeneration.js + lib/generation/ (pipeline séquentiel fixe)
  • ✅ Remplacé par: Modules selective/adversarial/human-simulation/pattern-breaking
  • ✅ Avantage: Flexibilité totale, stacks adaptatifs, parallélisation possible

File Structure

  • server.js - Express server entry point with mode selection
  • lib/Main.js - Core workflow orchestration
  • lib/modes/ - Mode management (Manual/Auto)
  • lib/BrainConfig.js - Google Sheets integration + personality system
  • lib/LLMManager.js - Multi-LLM provider management
  • lib/ContentGeneration.js - Content generation and enhancement pipeline
  • lib/ElementExtraction.js - XML parsing and element extraction
  • lib/ArticleStorage.js - Content compilation and Google Sheets storage
  • lib/ErrorReporting.js - Centralized logging and error handling
  • tools/ - Development utilities (log viewer, bundler, audit)
  • tests/ - Comprehensive test suite with multiple categories
  • .env - Environment configuration (Google credentials, API keys)

Key Dependencies

  • googleapis - Google Sheets API integration
  • axios - HTTP client for LLM APIs
  • dotenv - Environment variable management
  • express - Web server framework
  • nodemailer - Email notifications (needs setup)

Workflow Sources

  • production - Real Google Sheets data processing
  • test_random_personality - Testing with personality randomization
  • node_server - Direct API processing
  • Legacy: make_com, digital_ocean_autonomous