Restructuration complète du projet ConfluentTranslator
- Nouvelle architecture modulaire avec src/api, src/core, src/utils - Séparation claire docs/ (admin, changelog, dev, security) et tests/ (unit, integration, scripts) - server.js devient un simple point d'entrée - Ajout de STRUCTURE.md documentant l'architecture - Archivage ancien-confluent/ avec générateur de lexique complet 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
272a05b3fe
commit
4b0f916d1c
187
ConfluentTranslator/STRUCTURE.md
Normal file
187
ConfluentTranslator/STRUCTURE.md
Normal file
@ -0,0 +1,187 @@
|
||||
# Structure du projet ConfluentTranslator
|
||||
|
||||
Ce document décrit l'organisation du projet après la réorganisation.
|
||||
|
||||
## Arborescence
|
||||
|
||||
```
|
||||
ConfluentTranslator/
|
||||
├── server.js # Point d'entrée principal (lance src/api/server.js)
|
||||
├── package.json # Dépendances et scripts npm
|
||||
├── .env / .env.example # Configuration environnement
|
||||
├── README.md # Documentation utilisateur
|
||||
│
|
||||
├── src/ # Code source organisé
|
||||
│ ├── api/ # Serveur et routes HTTP
|
||||
│ │ ├── server.js # Serveur Express principal
|
||||
│ │ └── adminRoutes.js # Routes d'administration
|
||||
│ ├── core/ # Logique métier
|
||||
│ │ ├── translation/ # Modules de traduction
|
||||
│ │ │ ├── confluentToFrench.js # Traduction Confluent → FR
|
||||
│ │ │ ├── contextAnalyzer.js # Analyse contextuelle
|
||||
│ │ │ └── promptBuilder.js # Construction des prompts LLM
|
||||
│ │ ├── morphology/ # Morphologie et décomposition
|
||||
│ │ │ ├── morphologicalDecomposer.js # Décomposition morphologique
|
||||
│ │ │ ├── radicalMatcher.js # Recherche par radicaux
|
||||
│ │ │ └── reverseIndexBuilder.js # Construction d'index inversés
|
||||
│ │ └── numbers/ # Traitement des nombres
|
||||
│ │ ├── numberConverter.js # Conversion FR → Confluent
|
||||
│ │ └── numberPreprocessor.js # Prétraitement des nombres
|
||||
│ └── utils/ # Utilitaires
|
||||
│ ├── auth.js # Authentification et tokens
|
||||
│ ├── lexiqueLoader.js # Chargement des lexiques
|
||||
│ ├── logger.js # Système de logs
|
||||
│ └── rateLimiter.js # Rate limiting
|
||||
│
|
||||
├── docs/ # Documentation
|
||||
│ ├── admin/ # Documentation admin
|
||||
│ │ ├── ADMIN_GUIDE.md
|
||||
│ │ └── QUICKSTART_ADMIN.md
|
||||
│ ├── security/ # Documentation sécurité
|
||||
│ │ ├── README_SECURITY.md
|
||||
│ │ ├── SECURITY_TEST.md
|
||||
│ │ └── CHANGELOG_SECURITY.md
|
||||
│ ├── dev/ # Documentation développeur
|
||||
│ │ ├── analysis/ # Analyses techniques
|
||||
│ │ │ └── ANALYSE_MOTS_PROBLEMATIQUES.md
|
||||
│ │ └── numbers/ # Documentation nombres
|
||||
│ │ └── NUMBER_PREPROCESSING.md
|
||||
│ └── changelog/ # Historique et résultats
|
||||
│ ├── COMMIT_SUMMARY.md
|
||||
│ ├── TESTS_SUMMARY.md
|
||||
│ ├── TESTS_NOMBRES_RESULTAT.md
|
||||
│ └── test-results-radical-system.md
|
||||
│
|
||||
├── tests/ # Tests
|
||||
│ ├── unit/ # Tests unitaires (.js, .json, .txt)
|
||||
│ ├── integration/ # Tests d'intégration
|
||||
│ │ └── api/ # Tests API (ex: testsAPI/)
|
||||
│ └── scripts/ # Scripts de test (.sh, .bat)
|
||||
│
|
||||
├── data/ # Données du projet
|
||||
│ ├── lexique.json # Lexique principal
|
||||
│ ├── tokens.json # Tokens d'authentification
|
||||
│ └── (autres fichiers JSON de lexique)
|
||||
│
|
||||
├── prompts/ # Prompts système pour LLM
|
||||
│ ├── proto-system.txt
|
||||
│ └── ancien-system.txt
|
||||
│
|
||||
├── public/ # Fichiers statiques
|
||||
│ ├── index.html
|
||||
│ ├── admin.html
|
||||
│ └── (autres fichiers statiques)
|
||||
│
|
||||
├── logs/ # Logs applicatifs
|
||||
│ └── (fichiers de logs générés)
|
||||
│
|
||||
├── plans/ # Plans et documentation de travail
|
||||
│ └── (documents de planification)
|
||||
│
|
||||
└── node_modules/ # Dépendances npm (généré)
|
||||
```
|
||||
|
||||
## Principes d'organisation
|
||||
|
||||
### src/ - Code source
|
||||
|
||||
Le dossier `src/` contient tout le code applicatif organisé par fonction :
|
||||
|
||||
- **api/** : Tout ce qui concerne le serveur HTTP et les routes
|
||||
- **core/** : La logique métier, subdivisée par domaine
|
||||
- `translation/` : Traduction et analyse linguistique
|
||||
- `morphology/` : Analyse morphologique des mots
|
||||
- `numbers/` : Gestion spécifique des nombres
|
||||
- **utils/** : Fonctions utilitaires transverses
|
||||
|
||||
### docs/ - Documentation
|
||||
|
||||
Documentation organisée par audience et type :
|
||||
|
||||
- **admin/** : Guides pour les administrateurs
|
||||
- **security/** : Documentation sécurité
|
||||
- **dev/** : Documentation technique pour développeurs
|
||||
- **changelog/** : Historique des changements et résultats de tests
|
||||
|
||||
### tests/ - Tests
|
||||
|
||||
Tests organisés par type :
|
||||
|
||||
- **unit/** : Tests unitaires des modules individuels
|
||||
- **integration/** : Tests d'intégration entre modules
|
||||
- **scripts/** : Scripts shell/batch pour lancer les tests
|
||||
|
||||
## Imports et chemins
|
||||
|
||||
### Depuis src/api/ (server.js, adminRoutes.js)
|
||||
|
||||
```javascript
|
||||
// Utilitaires
|
||||
require('../utils/auth')
|
||||
require('../utils/logger')
|
||||
require('../utils/lexiqueLoader')
|
||||
require('../utils/rateLimiter')
|
||||
|
||||
// Translation
|
||||
require('../core/translation/contextAnalyzer')
|
||||
require('../core/translation/promptBuilder')
|
||||
require('../core/translation/confluentToFrench')
|
||||
|
||||
// Morphology
|
||||
require('../core/morphology/reverseIndexBuilder')
|
||||
|
||||
// Chemins vers ressources
|
||||
path.join(__dirname, '..', '..', 'public')
|
||||
path.join(__dirname, '..', '..', 'prompts')
|
||||
path.join(__dirname, '..', '..', 'data')
|
||||
```
|
||||
|
||||
### Depuis src/core/translation/
|
||||
|
||||
```javascript
|
||||
// Vers numbers
|
||||
require('../numbers/numberConverter')
|
||||
require('../numbers/numberPreprocessor')
|
||||
|
||||
// Vers morphology
|
||||
require('../morphology/radicalMatcher')
|
||||
require('../morphology/morphologicalDecomposer')
|
||||
```
|
||||
|
||||
### Depuis src/core/morphology/ ou src/core/numbers/
|
||||
|
||||
```javascript
|
||||
// Vers data
|
||||
require('../../data/lexique.json')
|
||||
```
|
||||
|
||||
## Démarrage
|
||||
|
||||
Le point d'entrée est `server.js` à la racine qui importe `src/api/server.js` :
|
||||
|
||||
```bash
|
||||
node server.js
|
||||
```
|
||||
|
||||
ou
|
||||
|
||||
```bash
|
||||
npm start
|
||||
```
|
||||
|
||||
## Migrations futures
|
||||
|
||||
Si nécessaire, cette structure permet facilement :
|
||||
|
||||
- D'ajouter de nouveaux modules dans `src/core/`
|
||||
- De créer des sous-modules dans `src/api/` (ex: routes métier)
|
||||
- D'ajouter des catégories de tests
|
||||
- D'organiser la documentation par projets
|
||||
|
||||
## Avantages
|
||||
|
||||
- **Clarté** : Chaque fichier a sa place logique
|
||||
- **Maintenabilité** : Structure modulaire et organisée
|
||||
- **Scalabilité** : Facile d'ajouter de nouveaux modules
|
||||
- **Découvrabilité** : On trouve rapidement ce qu'on cherche
|
||||
- **Séparation des préoccupations** : Code / Docs / Tests séparés
|
||||
@ -6,9 +6,7 @@
|
||||
"apiKey": "d9be0765-c454-47e9-883c-bcd93dd19eae",
|
||||
"createdAt": "2025-12-02T06:57:35.077Z",
|
||||
"active": true,
|
||||
"requestsToday": 35,
|
||||
"dailyLimit": -1,
|
||||
"lastUsed": "2025-12-02T08:02:37.203Z",
|
||||
"lastUsed": "2025-12-02T12:54:49.316Z",
|
||||
"llmTokens": {
|
||||
"totalInput": 0,
|
||||
"totalOutput": 0,
|
||||
@ -28,19 +26,17 @@
|
||||
"apiKey": "008d38c2-e6ed-4852-9b8b-a433e197719a",
|
||||
"createdAt": "2025-12-02T07:06:17.791Z",
|
||||
"active": true,
|
||||
"requestsToday": 100,
|
||||
"dailyLimit": 100,
|
||||
"lastUsed": "2025-12-02T08:09:45.029Z",
|
||||
"lastUsed": "2025-12-02T12:51:17.345Z",
|
||||
"llmTokens": {
|
||||
"totalInput": 0,
|
||||
"totalOutput": 0,
|
||||
"totalInput": 40852,
|
||||
"totalOutput": 596,
|
||||
"today": {
|
||||
"input": 0,
|
||||
"output": 0,
|
||||
"input": 40852,
|
||||
"output": 596,
|
||||
"date": "2025-12-02"
|
||||
}
|
||||
},
|
||||
"llmRequestsToday": 0,
|
||||
"llmRequestsToday": 20,
|
||||
"llmDailyLimit": 20
|
||||
}
|
||||
}
|
||||
@ -536,7 +536,7 @@
|
||||
border-radius: 6px;
|
||||
display: none;
|
||||
">
|
||||
<span id="llm-limit-text">Requêtes LLM: 20/20</span>
|
||||
<span id="llm-limit-text">Requêtes LLM: 0/20</span>
|
||||
</div>
|
||||
<div>
|
||||
<button class="logout-btn" onclick="goToAdmin()" id="admin-btn" style="display:none; margin-right: 10px;">🔐 Admin</button>
|
||||
@ -810,24 +810,6 @@
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<h2 style="margin-top: 30px;">🔑 API Keys (optionnel)</h2>
|
||||
|
||||
<div class="form-group">
|
||||
<label>Anthropic API Key</label>
|
||||
<input type="password" id="settings-anthropic-key" placeholder="sk-ant-...">
|
||||
<small style="color: #888; display: block; margin-top: 5px;">
|
||||
Laisser vide pour utiliser la clé du serveur
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label>OpenAI API Key</label>
|
||||
<input type="password" id="settings-openai-key" placeholder="sk-...">
|
||||
<small style="color: #888; display: block; margin-top: 5px;">
|
||||
Laisser vide pour utiliser la clé du serveur
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<button onclick="saveSettings()" style="margin-top: 20px;">💾 Sauvegarder les paramètres</button>
|
||||
<div id="settings-saved-message" style="display: none; color: #4a9eff; margin-top: 10px; font-weight: 600;">
|
||||
✓ Paramètres sauvegardés !
|
||||
@ -885,8 +867,8 @@
|
||||
// User with limited requests
|
||||
counter.style.display = 'block';
|
||||
const limit = data.limit || 20;
|
||||
const remaining = data.remaining !== undefined ? data.remaining : (limit - (data.used || 0));
|
||||
text.textContent = `Requêtes LLM restantes: ${remaining}/${limit}`;
|
||||
const used = data.used || 0;
|
||||
text.textContent = `Requêtes LLM: ${used}/${limit}`;
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error loading LLM limit:', error);
|
||||
@ -951,9 +933,6 @@
|
||||
login();
|
||||
}
|
||||
});
|
||||
|
||||
// Note: LLM limit counter is updated after each translation
|
||||
// No need for automatic polling every few seconds
|
||||
});
|
||||
|
||||
// Authenticated fetch wrapper with auto-logout on 401/403
|
||||
@ -1195,8 +1174,6 @@
|
||||
document.getElementById('temp-value').textContent = settings.temperature;
|
||||
document.getElementById('settings-theme').value = settings.theme;
|
||||
document.getElementById('settings-verbose').checked = settings.verbose;
|
||||
document.getElementById('settings-anthropic-key').value = settings.anthropicKey;
|
||||
document.getElementById('settings-openai-key').value = settings.openaiKey;
|
||||
|
||||
// Apply theme
|
||||
applyTheme(settings.theme);
|
||||
@ -1208,14 +1185,18 @@
|
||||
};
|
||||
|
||||
const saveSettings = () => {
|
||||
// Load existing settings to preserve API keys
|
||||
const existingSettings = JSON.parse(localStorage.getItem('confluentSettings') || '{}');
|
||||
|
||||
const settings = {
|
||||
provider: document.getElementById('settings-provider').value,
|
||||
model: document.getElementById('settings-model').value,
|
||||
temperature: parseFloat(document.getElementById('settings-temperature').value),
|
||||
theme: document.getElementById('settings-theme').value,
|
||||
verbose: document.getElementById('settings-verbose').checked,
|
||||
anthropicKey: document.getElementById('settings-anthropic-key').value,
|
||||
openaiKey: document.getElementById('settings-openai-key').value
|
||||
// Preserve API keys from localStorage
|
||||
anthropicKey: existingSettings.anthropicKey || '',
|
||||
openaiKey: existingSettings.openaiKey || ''
|
||||
};
|
||||
|
||||
localStorage.setItem('confluentSettings', JSON.stringify(settings));
|
||||
|
||||
@ -1,872 +1,3 @@
|
||||
require('dotenv').config();
|
||||
const express = require('express');
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
const { Anthropic } = require('@anthropic-ai/sdk');
|
||||
const OpenAI = require('openai');
|
||||
const {
|
||||
loadAllLexiques,
|
||||
searchLexique,
|
||||
generateLexiqueSummary,
|
||||
buildReverseIndex
|
||||
} = require('./lexiqueLoader');
|
||||
const { analyzeContext } = require('./contextAnalyzer');
|
||||
const { buildContextualPrompt, getBasePrompt, getPromptStats } = require('./promptBuilder');
|
||||
const { buildReverseIndex: buildConfluentIndex } = require('./reverseIndexBuilder');
|
||||
const { translateConfluentToFrench, translateConfluentDetailed } = require('./confluentToFrench');
|
||||
|
||||
// Security modules
|
||||
const { authenticate, requireAdmin, createToken, listTokens, disableToken, enableToken, deleteToken, getGlobalStats, trackLLMUsage, checkLLMLimit } = require('./auth');
|
||||
const { adminLimiter } = require('./rateLimiter');
|
||||
const { requestLogger, getLogs, getLogStats } = require('./logger');
|
||||
|
||||
const app = express();
|
||||
const PORT = process.env.PORT || 3000;
|
||||
|
||||
// Middlewares
|
||||
app.use(express.json());
|
||||
app.use(requestLogger); // Log toutes les requêtes
|
||||
// Rate limiting: on utilise uniquement checkLLMLimit() par API key, pas de rate limit global par IP
|
||||
|
||||
// Route protégée pour admin.html (AVANT express.static)
|
||||
// Vérifie l'auth seulement si API key présente, sinon laisse passer (le JS client vérifiera)
|
||||
app.get('/admin.html', (req, res, next) => {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
|
||||
// Si pas d'API key, c'est une requête browser normale -> laisser passer
|
||||
if (!apiKey) {
|
||||
return res.sendFile(path.join(__dirname, 'public', 'admin.html'));
|
||||
}
|
||||
|
||||
// Si API key présente, vérifier qu'elle est admin
|
||||
authenticate(req, res, (authErr) => {
|
||||
if (authErr) return next(authErr);
|
||||
requireAdmin(req, res, (adminErr) => {
|
||||
if (adminErr) return next(adminErr);
|
||||
res.sendFile(path.join(__dirname, 'public', 'admin.html'));
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
app.use(express.static('public'));
|
||||
|
||||
// Load prompts
|
||||
const protoPrompt = fs.readFileSync(path.join(__dirname, 'prompts', 'proto-system.txt'), 'utf-8');
|
||||
const ancienPrompt = fs.readFileSync(path.join(__dirname, 'prompts', 'ancien-system.txt'), 'utf-8');
|
||||
|
||||
// Load lexiques dynamically from JSON files
|
||||
const baseDir = path.join(__dirname, '..');
|
||||
let lexiques = { proto: null, ancien: null };
|
||||
let reverseIndexes = { proto: null, ancien: null };
|
||||
let confluentIndexes = { proto: null, ancien: null };
|
||||
|
||||
function reloadLexiques() {
|
||||
console.log('Loading lexiques...');
|
||||
lexiques = loadAllLexiques(baseDir);
|
||||
reverseIndexes = {
|
||||
proto: buildReverseIndex(lexiques.proto),
|
||||
ancien: buildReverseIndex(lexiques.ancien)
|
||||
};
|
||||
confluentIndexes = {
|
||||
proto: buildConfluentIndex(lexiques.proto),
|
||||
ancien: buildConfluentIndex(lexiques.ancien)
|
||||
};
|
||||
console.log('Lexiques loaded successfully');
|
||||
console.log(`Confluent→FR index: ${Object.keys(confluentIndexes.ancien || {}).length} entries`);
|
||||
}
|
||||
|
||||
// Initial load
|
||||
reloadLexiques();
|
||||
|
||||
// Health check endpoint (public - for login validation)
|
||||
app.get('/api/health', (req, res) => {
|
||||
res.json({
|
||||
status: 'ok',
|
||||
timestamp: new Date().toISOString(),
|
||||
version: '1.0.0'
|
||||
});
|
||||
});
|
||||
|
||||
// Auth validation endpoint (tests API key without exposing data)
|
||||
app.get('/api/validate', authenticate, (req, res) => {
|
||||
res.json({
|
||||
valid: true,
|
||||
user: req.user?.name || 'anonymous',
|
||||
role: req.user?.role || 'user'
|
||||
});
|
||||
});
|
||||
|
||||
// LLM limit check endpoint - Always returns 200 with info
|
||||
app.get('/api/llm/limit', authenticate, (req, res) => {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
|
||||
console.log('[/api/llm/limit] Check result:', limitCheck); // Debug
|
||||
|
||||
// TOUJOURS retourner 200 avec les données
|
||||
// Cet endpoint ne bloque jamais, il informe seulement
|
||||
res.status(200).json(limitCheck);
|
||||
});
|
||||
|
||||
// Legacy lexique endpoint (for backward compatibility) - SECURED
|
||||
app.get('/lexique', authenticate, (req, res) => {
|
||||
// Return ancien-confluent by default (legacy behavior)
|
||||
if (!lexiques.ancien) {
|
||||
return res.status(500).json({ error: 'Lexique not loaded' });
|
||||
}
|
||||
res.json(lexiques.ancien);
|
||||
});
|
||||
|
||||
// New lexique endpoints - SECURED
|
||||
app.get('/api/lexique/:variant', authenticate, (req, res) => {
|
||||
const { variant } = req.params;
|
||||
|
||||
if (variant !== 'proto' && variant !== 'ancien') {
|
||||
return res.status(400).json({ error: 'Invalid variant. Use "proto" or "ancien"' });
|
||||
}
|
||||
|
||||
if (!lexiques[variant]) {
|
||||
return res.status(500).json({ error: `Lexique ${variant} not loaded` });
|
||||
}
|
||||
|
||||
res.json(lexiques[variant]);
|
||||
});
|
||||
|
||||
// Stats endpoint - SECURED
|
||||
app.get('/api/stats', authenticate, (req, res) => {
|
||||
const { variant = 'ancien' } = req.query;
|
||||
|
||||
if (variant !== 'proto' && variant !== 'ancien') {
|
||||
return res.status(400).json({ error: 'Invalid variant. Use "proto" or "ancien"' });
|
||||
}
|
||||
|
||||
if (!lexiques[variant]) {
|
||||
return res.status(500).json({ error: `Lexique ${variant} not loaded` });
|
||||
}
|
||||
|
||||
const lexique = lexiques[variant];
|
||||
const stats = {
|
||||
motsCF: 0, // Mots Confluent uniques
|
||||
motsFR: 0, // Mots français uniques
|
||||
totalTraductions: 0, // Total de traductions
|
||||
racines: 0, // Racines (racine, racine_sacree)
|
||||
racinesSacrees: 0, // Racines sacrées
|
||||
racinesStandards: 0, // Racines standards
|
||||
compositions: 0, // Compositions
|
||||
verbes: 0, // Verbes
|
||||
verbesIrreguliers: 0, // Verbes irréguliers
|
||||
particules: 0, // Particules grammaticales (negation, particule, interrogation, demonstratif)
|
||||
nomsPropes: 0, // Noms propres
|
||||
marqueurs: 0, // Marqueurs (temps, aspect, nombre)
|
||||
pronoms: 0, // Pronoms (pronom, possessif, relatif, determinant)
|
||||
autres: 0 // Autres types (auxiliaire, quantificateur, etc.)
|
||||
};
|
||||
|
||||
const motsCFSet = new Set();
|
||||
const motsFRSet = new Set();
|
||||
|
||||
// Le lexique peut avoir une structure {dictionnaire: {...}} ou être directement un objet
|
||||
const dict = lexique.dictionnaire || lexique;
|
||||
|
||||
// Parcourir le dictionnaire
|
||||
Object.keys(dict).forEach(motFR => {
|
||||
const entry = dict[motFR];
|
||||
motsFRSet.add(motFR);
|
||||
|
||||
if (entry.traductions) {
|
||||
entry.traductions.forEach(trad => {
|
||||
stats.totalTraductions++;
|
||||
|
||||
// Compter les mots CF uniques
|
||||
if (trad.confluent) {
|
||||
motsCFSet.add(trad.confluent);
|
||||
}
|
||||
|
||||
// Compter par type
|
||||
const type = trad.type || '';
|
||||
if (type === 'racine') {
|
||||
stats.racines++;
|
||||
stats.racinesStandards++;
|
||||
} else if (type === 'racine_sacree') {
|
||||
stats.racines++;
|
||||
stats.racinesSacrees++;
|
||||
} else if (type === 'composition' || type === 'racine_sacree_composee') {
|
||||
stats.compositions++;
|
||||
} else if (type === 'verbe') {
|
||||
stats.verbes++;
|
||||
} else if (type === 'verbe_irregulier') {
|
||||
stats.verbes++;
|
||||
stats.verbesIrreguliers++;
|
||||
} else if (type === 'negation' || type === 'particule' || type === 'interrogation' || type === 'demonstratif') {
|
||||
stats.particules++;
|
||||
} else if (type === 'nom_propre') {
|
||||
stats.nomsPropes++;
|
||||
} else if (type === 'marqueur_temps' || type === 'marqueur_aspect' || type === 'marqueur_nombre') {
|
||||
stats.marqueurs++;
|
||||
} else if (type === 'pronom' || type === 'possessif' || type === 'relatif' || type === 'determinant') {
|
||||
stats.pronoms++;
|
||||
} else if (type !== '') {
|
||||
stats.autres++;
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
stats.motsCF = motsCFSet.size;
|
||||
stats.motsFR = motsFRSet.size;
|
||||
|
||||
res.json(stats);
|
||||
});
|
||||
|
||||
// Search endpoint - SECURED
|
||||
app.get('/api/search', authenticate, (req, res) => {
|
||||
const { q, variant = 'ancien', direction = 'fr2conf' } = req.query;
|
||||
|
||||
if (!q) {
|
||||
return res.status(400).json({ error: 'Missing query parameter "q"' });
|
||||
}
|
||||
|
||||
if (variant !== 'proto' && variant !== 'ancien') {
|
||||
return res.status(400).json({ error: 'Invalid variant. Use "proto" or "ancien"' });
|
||||
}
|
||||
|
||||
const results = searchLexique(lexiques[variant], q, direction);
|
||||
res.json({ query: q, variant, direction, results });
|
||||
});
|
||||
|
||||
// Reload endpoint (for development) - SECURED (admin only)
|
||||
app.post('/api/reload', authenticate, requireAdmin, (req, res) => {
|
||||
try {
|
||||
reloadLexiques();
|
||||
res.json({
|
||||
success: true,
|
||||
message: 'Lexiques reloaded',
|
||||
stats: {
|
||||
proto: lexiques.proto?.meta?.total_entries || 0,
|
||||
ancien: lexiques.ancien?.meta?.total_entries || 0
|
||||
}
|
||||
});
|
||||
} catch (error) {
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Build enhanced prompt with lexique data
|
||||
function buildEnhancedPrompt(basePrompt, variant) {
|
||||
const lexique = lexiques[variant];
|
||||
if (!lexique) return basePrompt;
|
||||
|
||||
const summary = generateLexiqueSummary(lexique, 300);
|
||||
|
||||
return `${basePrompt}
|
||||
|
||||
# LEXIQUE COMPLET (${lexique.meta.total_entries} entrées)
|
||||
${summary}
|
||||
`;
|
||||
}
|
||||
|
||||
// Debug endpoint: Generate prompt without calling LLM - SECURED
|
||||
app.post('/api/debug/prompt', authenticate, (req, res) => {
|
||||
const { text, target = 'ancien', useLexique = true } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
let systemPrompt;
|
||||
let contextMetadata = null;
|
||||
|
||||
// MÊME CODE QUE /translate
|
||||
if (useLexique) {
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
systemPrompt = buildContextualPrompt(contextResult, variant, text);
|
||||
|
||||
const promptStats = getPromptStats(systemPrompt, contextResult);
|
||||
contextMetadata = {
|
||||
wordsFound: contextResult.metadata.wordsFound,
|
||||
wordsNotFound: contextResult.metadata.wordsNotFound,
|
||||
entriesUsed: contextResult.metadata.entriesUsed,
|
||||
totalLexiqueSize: contextResult.metadata.totalLexiqueSize,
|
||||
tokensFullLexique: promptStats.fullLexiqueTokens,
|
||||
tokensUsed: promptStats.promptTokens,
|
||||
tokensSaved: promptStats.tokensSaved,
|
||||
savingsPercent: promptStats.savingsPercent,
|
||||
useFallback: contextResult.useFallback,
|
||||
expansionLevel: contextResult.metadata.expansionLevel
|
||||
};
|
||||
} else {
|
||||
systemPrompt = getBasePrompt(variant);
|
||||
}
|
||||
|
||||
res.json({
|
||||
prompt: systemPrompt,
|
||||
metadata: contextMetadata,
|
||||
stats: {
|
||||
promptLength: systemPrompt.length,
|
||||
promptLines: systemPrompt.split('\n').length
|
||||
}
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Prompt generation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Coverage analysis endpoint (analyze French text before translation) - SECURED
|
||||
app.post('/api/analyze/coverage', authenticate, (req, res) => {
|
||||
const { text, target = 'ancien' } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
// Use the same contextAnalyzer as the translation pipeline
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
const metadata = contextResult.metadata;
|
||||
|
||||
// Calculate recommendation
|
||||
const needsFullRoots = metadata.coveragePercent < 90;
|
||||
let recommendation;
|
||||
if (metadata.coveragePercent >= 95) {
|
||||
recommendation = 'Excellent coverage - context only';
|
||||
} else if (metadata.coveragePercent >= 90) {
|
||||
recommendation = 'Good coverage - context only';
|
||||
} else if (metadata.coveragePercent >= 70) {
|
||||
recommendation = 'Moderate coverage - consider adding roots';
|
||||
} else if (metadata.coveragePercent >= 50) {
|
||||
recommendation = 'Low coverage - full roots recommended';
|
||||
} else {
|
||||
recommendation = 'Very low coverage - full roots required';
|
||||
}
|
||||
|
||||
res.json({
|
||||
coverage: metadata.coveragePercent,
|
||||
found: metadata.wordsFound.map(w => ({
|
||||
word: w.input,
|
||||
confluent: w.confluent,
|
||||
type: w.type,
|
||||
score: w.score
|
||||
})),
|
||||
missing: metadata.wordsNotFound.map(word => ({
|
||||
word,
|
||||
suggestions: [] // TODO: add suggestions based on similar words
|
||||
})),
|
||||
stats: {
|
||||
totalWords: metadata.wordCount,
|
||||
uniqueWords: metadata.uniqueWordCount,
|
||||
foundCount: metadata.wordsFound.length,
|
||||
missingCount: metadata.wordsNotFound.length,
|
||||
entriesUsed: metadata.entriesUsed,
|
||||
useFallback: metadata.useFallback
|
||||
},
|
||||
needsFullRoots,
|
||||
recommendation,
|
||||
variant
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Coverage analysis error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Translation endpoint (NOUVEAU SYSTÈME CONTEXTUEL)
|
||||
app.post('/translate', authenticate, async (req, res) => {
|
||||
const { text, target, provider, model, temperature = 1.0, useLexique = true, customAnthropicKey, customOpenAIKey } = req.body;
|
||||
|
||||
if (!text || !target || !provider || !model) {
|
||||
return res.status(400).json({ error: 'Missing parameters' });
|
||||
}
|
||||
|
||||
// Check for custom API keys
|
||||
const usingCustomKey = !!(customAnthropicKey || customOpenAIKey);
|
||||
|
||||
// Only check rate limit if NOT using custom keys
|
||||
if (!usingCustomKey) {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
if (!limitCheck.allowed) {
|
||||
return res.status(429).json({
|
||||
error: limitCheck.error,
|
||||
limit: limitCheck.limit,
|
||||
used: limitCheck.used
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
let systemPrompt;
|
||||
let contextMetadata = null;
|
||||
|
||||
// NOUVEAU: Analyse contextuelle et génération de prompt optimisé
|
||||
if (useLexique) {
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
systemPrompt = buildContextualPrompt(contextResult, variant, text);
|
||||
|
||||
// Générer métadonnées pour Layer 2
|
||||
const promptStats = getPromptStats(systemPrompt, contextResult);
|
||||
contextMetadata = {
|
||||
wordsFound: contextResult.metadata.wordsFound,
|
||||
wordsNotFound: contextResult.metadata.wordsNotFound,
|
||||
entriesUsed: contextResult.metadata.entriesUsed,
|
||||
totalLexiqueSize: contextResult.metadata.totalLexiqueSize,
|
||||
tokensFullLexique: promptStats.fullLexiqueTokens,
|
||||
tokensUsed: promptStats.promptTokens,
|
||||
tokensSaved: promptStats.tokensSaved,
|
||||
savingsPercent: promptStats.savingsPercent,
|
||||
useFallback: contextResult.useFallback,
|
||||
expansionLevel: contextResult.metadata.expansionLevel,
|
||||
rootsUsed: contextResult.rootsFallback?.length || 0 // Nombre de racines envoyées
|
||||
};
|
||||
} else {
|
||||
systemPrompt = getBasePrompt(variant);
|
||||
}
|
||||
|
||||
let translation;
|
||||
let rawResponse;
|
||||
|
||||
if (provider === 'anthropic') {
|
||||
const anthropic = new Anthropic({
|
||||
apiKey: customAnthropicKey || process.env.ANTHROPIC_API_KEY,
|
||||
});
|
||||
|
||||
const message = await anthropic.messages.create({
|
||||
model: model,
|
||||
max_tokens: 8192, // Max pour Claude Sonnet/Haiku 4.5
|
||||
temperature: temperature / 2, // Diviser par 2 pour Claude (max 1.0)
|
||||
system: systemPrompt,
|
||||
messages: [
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = message.content[0].text;
|
||||
translation = rawResponse;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && message.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, message.usage.input_tokens, message.usage.output_tokens);
|
||||
}
|
||||
|
||||
} else if (provider === 'openai') {
|
||||
const openai = new OpenAI({
|
||||
apiKey: customOpenAIKey || process.env.OPENAI_API_KEY,
|
||||
});
|
||||
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: model,
|
||||
max_tokens: 16384, // Max pour GPT-4o et GPT-4o-mini
|
||||
temperature: temperature,
|
||||
messages: [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = completion.choices[0].message.content;
|
||||
translation = rawResponse;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && completion.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, completion.usage.prompt_tokens, completion.usage.completion_tokens);
|
||||
}
|
||||
} else {
|
||||
return res.status(400).json({ error: 'Unknown provider' });
|
||||
}
|
||||
|
||||
// Parser la réponse pour extraire Layer 1 et Layer 3
|
||||
const parsed = parseTranslationResponse(rawResponse);
|
||||
|
||||
// Construire la réponse avec les 3 layers
|
||||
const response = {
|
||||
// Layer 1: Traduction
|
||||
layer1: {
|
||||
translation: parsed.translation
|
||||
},
|
||||
|
||||
// Layer 2: Contexte (COT hors LLM)
|
||||
layer2: contextMetadata,
|
||||
|
||||
// Layer 3: Explications LLM (avec COT)
|
||||
layer3: {
|
||||
analyse: parsed.analyse,
|
||||
strategie: parsed.strategie,
|
||||
decomposition: parsed.decomposition,
|
||||
notes: parsed.notes,
|
||||
wordsCreated: parsed.wordsCreated || []
|
||||
},
|
||||
|
||||
// Compatibilité avec ancien format
|
||||
translation: parsed.translation
|
||||
};
|
||||
|
||||
res.json(response);
|
||||
|
||||
} catch (error) {
|
||||
console.error('Translation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Parse la réponse du LLM pour extraire les différentes sections (avec COT)
|
||||
* @param {string} response - Réponse brute du LLM
|
||||
* @returns {Object} - Sections parsées
|
||||
*/
|
||||
function parseTranslationResponse(response) {
|
||||
const lines = response.split('\n');
|
||||
|
||||
let analyse = '';
|
||||
let strategie = '';
|
||||
let translation = '';
|
||||
let decomposition = '';
|
||||
let notes = '';
|
||||
let currentSection = null;
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
|
||||
// Détecter les sections (nouveau format COT)
|
||||
if (trimmed.match(/^ANALYSE:/i)) {
|
||||
currentSection = 'analyse';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^STRAT[ÉE]GIE:/i)) {
|
||||
currentSection = 'strategie';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^(Ancien )?Confluent:/i)) {
|
||||
currentSection = 'translation';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^D[ée]composition:/i)) {
|
||||
currentSection = 'decomposition';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^Notes?:/i) || trimmed.match(/^Explication:/i)) {
|
||||
currentSection = 'notes';
|
||||
continue;
|
||||
}
|
||||
|
||||
// Ajouter le contenu à la section appropriée
|
||||
if (currentSection === 'analyse' && trimmed && !trimmed.match(/^---/)) {
|
||||
analyse += line + '\n';
|
||||
} else if (currentSection === 'strategie' && trimmed && !trimmed.match(/^---/)) {
|
||||
strategie += line + '\n';
|
||||
} else if (currentSection === 'translation' && trimmed && !trimmed.match(/^---/)) {
|
||||
translation += line + '\n';
|
||||
} else if (currentSection === 'decomposition' && trimmed) {
|
||||
decomposition += line + '\n';
|
||||
} else if (currentSection === 'notes' && trimmed) {
|
||||
notes += line + '\n';
|
||||
} else if (!currentSection && trimmed && !trimmed.match(/^---/) && !trimmed.match(/^\*\*/)) {
|
||||
// Si pas de section détectée, c'est probablement la traduction
|
||||
translation += line + '\n';
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
analyse: analyse.trim(),
|
||||
strategie: strategie.trim(),
|
||||
translation: translation.trim() || response.trim(),
|
||||
decomposition: decomposition.trim(),
|
||||
notes: notes.trim()
|
||||
};
|
||||
}
|
||||
|
||||
// Raw translation endpoint (for debugging - returns unprocessed LLM output) - SECURED
|
||||
app.post('/api/translate/raw', authenticate, async (req, res) => {
|
||||
const { text, target, provider, model, useLexique = true, customAnthropicKey, customOpenAIKey } = req.body;
|
||||
|
||||
if (!text || !target || !provider || !model) {
|
||||
return res.status(400).json({ error: 'Missing parameters' });
|
||||
}
|
||||
|
||||
// Check for custom API keys
|
||||
const usingCustomKey = !!(customAnthropicKey || customOpenAIKey);
|
||||
|
||||
// Only check rate limit if NOT using custom keys
|
||||
if (!usingCustomKey) {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
if (!limitCheck.allowed) {
|
||||
return res.status(429).json({
|
||||
error: limitCheck.error,
|
||||
limit: limitCheck.limit,
|
||||
used: limitCheck.used
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
let systemPrompt;
|
||||
let contextMetadata = null;
|
||||
|
||||
if (useLexique) {
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
systemPrompt = buildContextualPrompt(contextResult, variant, text);
|
||||
|
||||
const promptStats = getPromptStats(systemPrompt, contextResult);
|
||||
contextMetadata = {
|
||||
wordsFound: contextResult.metadata.wordsFound,
|
||||
wordsNotFound: contextResult.metadata.wordsNotFound,
|
||||
entriesUsed: contextResult.metadata.entriesUsed,
|
||||
totalLexiqueSize: contextResult.metadata.totalLexiqueSize,
|
||||
tokensFullLexique: promptStats.fullLexiqueTokens,
|
||||
tokensUsed: promptStats.promptTokens,
|
||||
tokensSaved: promptStats.tokensSaved,
|
||||
savingsPercent: promptStats.savingsPercent,
|
||||
useFallback: contextResult.useFallback,
|
||||
expansionLevel: contextResult.metadata.expansionLevel
|
||||
};
|
||||
} else {
|
||||
systemPrompt = getBasePrompt(variant);
|
||||
}
|
||||
|
||||
let rawResponse;
|
||||
|
||||
if (provider === 'anthropic') {
|
||||
const anthropic = new Anthropic({
|
||||
apiKey: customAnthropicKey || process.env.ANTHROPIC_API_KEY,
|
||||
});
|
||||
|
||||
const message = await anthropic.messages.create({
|
||||
model: model,
|
||||
max_tokens: 8192, // Max pour Claude Sonnet/Haiku 4.5
|
||||
system: systemPrompt,
|
||||
messages: [
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = message.content[0].text;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && message.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, message.usage.input_tokens, message.usage.output_tokens);
|
||||
}
|
||||
|
||||
} else if (provider === 'openai') {
|
||||
const openai = new OpenAI({
|
||||
apiKey: customOpenAIKey || process.env.OPENAI_API_KEY,
|
||||
});
|
||||
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: model,
|
||||
max_tokens: 16384, // Max pour GPT-4o et GPT-4o-mini
|
||||
messages: [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = completion.choices[0].message.content;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && completion.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, completion.usage.prompt_tokens, completion.usage.completion_tokens);
|
||||
}
|
||||
} else {
|
||||
return res.status(400).json({ error: 'Unknown provider' });
|
||||
}
|
||||
|
||||
// Retourner la réponse BRUTE sans parsing
|
||||
res.json({
|
||||
raw_output: rawResponse,
|
||||
metadata: contextMetadata,
|
||||
length: rawResponse.length,
|
||||
lines: rawResponse.split('\n').length
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Translation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Batch translation endpoint - SECURED
|
||||
app.post('/api/translate/batch', authenticate, async (req, res) => {
|
||||
const { words, target = 'ancien' } = req.body;
|
||||
|
||||
if (!words || !Array.isArray(words)) {
|
||||
return res.status(400).json({ error: 'Missing or invalid "words" array' });
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
const results = {};
|
||||
|
||||
for (const word of words) {
|
||||
const found = searchLexique(lexiques[variant], word, 'fr2conf');
|
||||
if (found.length > 0 && found[0].traductions?.length > 0) {
|
||||
results[word] = {
|
||||
found: true,
|
||||
traduction: found[0].traductions[0].confluent,
|
||||
all_traductions: found[0].traductions
|
||||
};
|
||||
} else {
|
||||
results[word] = { found: false };
|
||||
}
|
||||
}
|
||||
|
||||
res.json({ target, results });
|
||||
});
|
||||
|
||||
// Confluent → French translation endpoint (traduction brute) - SECURED
|
||||
app.post('/api/translate/conf2fr', authenticate, (req, res) => {
|
||||
const { text, variant = 'ancien', detailed = false } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
const variantKey = variant === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
if (!confluentIndexes[variantKey]) {
|
||||
return res.status(500).json({ error: `Confluent index for ${variantKey} not loaded` });
|
||||
}
|
||||
|
||||
try {
|
||||
if (detailed) {
|
||||
const result = translateConfluentDetailed(text, confluentIndexes[variantKey]);
|
||||
res.json(result);
|
||||
} else {
|
||||
const result = translateConfluentToFrench(text, confluentIndexes[variantKey]);
|
||||
res.json(result);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Confluent→FR translation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// NEW: Confluent → French with LLM refinement
|
||||
app.post('/api/translate/conf2fr/llm', authenticate, async (req, res) => {
|
||||
const { text, variant = 'ancien', provider = 'anthropic', model = 'claude-sonnet-4-20250514', customAnthropicKey, customOpenAIKey } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
// Check for custom API keys
|
||||
const usingCustomKey = !!(customAnthropicKey || customOpenAIKey);
|
||||
|
||||
// Only check rate limit if NOT using custom keys
|
||||
if (!usingCustomKey) {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
if (!limitCheck.allowed) {
|
||||
return res.status(429).json({
|
||||
error: limitCheck.error,
|
||||
limit: limitCheck.limit,
|
||||
used: limitCheck.used
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const variantKey = variant === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
if (!confluentIndexes[variantKey]) {
|
||||
return res.status(500).json({ error: `Confluent index for ${variantKey} not loaded` });
|
||||
}
|
||||
|
||||
try {
|
||||
// Step 1: Get raw word-by-word translation
|
||||
const rawTranslation = translateConfluentToFrench(text, confluentIndexes[variantKey]);
|
||||
|
||||
// Step 2: Load refinement prompt
|
||||
const refinementPrompt = fs.readFileSync(path.join(__dirname, 'prompts', 'cf2fr-refinement.txt'), 'utf-8');
|
||||
|
||||
// Step 3: Use LLM to refine translation
|
||||
let refinedText;
|
||||
|
||||
if (provider === 'anthropic') {
|
||||
const anthropic = new Anthropic({
|
||||
apiKey: customAnthropicKey || process.env.ANTHROPIC_API_KEY,
|
||||
});
|
||||
|
||||
const message = await anthropic.messages.create({
|
||||
model: model,
|
||||
max_tokens: 2048,
|
||||
system: refinementPrompt,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: `Voici la traduction brute mot-à-mot du Confluent vers le français. Transforme-la en français fluide et naturel:\n\n${rawTranslation.translation}`
|
||||
}
|
||||
]
|
||||
});
|
||||
|
||||
refinedText = message.content[0].text.trim();
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && message.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, message.usage.input_tokens, message.usage.output_tokens);
|
||||
}
|
||||
} else if (provider === 'openai') {
|
||||
const openai = new OpenAI({
|
||||
apiKey: customOpenAIKey || process.env.OPENAI_API_KEY,
|
||||
});
|
||||
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: model,
|
||||
messages: [
|
||||
{ role: 'system', content: refinementPrompt },
|
||||
{ role: 'user', content: `Voici la traduction brute mot-à-mot du Confluent vers le français. Transforme-la en français fluide et naturel:\n\n${rawTranslation.translation}` }
|
||||
]
|
||||
});
|
||||
|
||||
refinedText = completion.choices[0].message.content.trim();
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && completion.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, completion.usage.prompt_tokens, completion.usage.completion_tokens);
|
||||
}
|
||||
} else {
|
||||
return res.status(400).json({ error: 'Unsupported provider. Use "anthropic" or "openai".' });
|
||||
}
|
||||
|
||||
// Return both raw and refined versions with detailed token info
|
||||
res.json({
|
||||
confluentText: text,
|
||||
rawTranslation: rawTranslation.translation,
|
||||
refinedTranslation: refinedText,
|
||||
translation: refinedText, // For compatibility
|
||||
tokens: rawTranslation.tokens || [],
|
||||
coverage: rawTranslation.coverage || 0,
|
||||
wordsTranslated: rawTranslation.wordsTranslated,
|
||||
wordsNotTranslated: rawTranslation.wordsNotTranslated,
|
||||
provider,
|
||||
model
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Confluent→FR LLM refinement error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Admin routes
|
||||
const adminRoutes = require('./adminRoutes');
|
||||
app.use('/api/admin', authenticate, adminRoutes);
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`ConfluentTranslator running on http://localhost:${PORT}`);
|
||||
console.log(`Loaded: ${lexiques.ancien?.meta?.total_entries || 0} ancien entries, ${lexiques.proto?.meta?.total_entries || 0} proto entries`);
|
||||
});
|
||||
// Point d'entrée du serveur ConfluentTranslator
|
||||
// Importe le serveur depuis la structure organisée
|
||||
require('./src/api/server');
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
const express = require('express');
|
||||
const router = express.Router();
|
||||
const { requireAdmin, createToken, listTokens, disableToken, enableToken, deleteToken, getGlobalStats } = require('./auth');
|
||||
const { getLogs, getLogStats } = require('./logger');
|
||||
const { adminLimiter } = require('./rateLimiter');
|
||||
const { requireAdmin, createToken, listTokens, disableToken, enableToken, deleteToken, getGlobalStats } = require('../utils/auth');
|
||||
const { getLogs, getLogStats } = require('../utils/logger');
|
||||
const { adminLimiter } = require('../utils/rateLimiter');
|
||||
|
||||
// Appliquer l'auth et rate limiting à toutes les routes admin
|
||||
router.use(requireAdmin);
|
||||
@ -16,13 +16,13 @@ router.get('/tokens', (req, res) => {
|
||||
|
||||
// Créer un nouveau token
|
||||
router.post('/tokens', (req, res) => {
|
||||
const { name, role = 'user', dailyLimit = 100 } = req.body;
|
||||
const { name, role = 'user' } = req.body;
|
||||
|
||||
if (!name) {
|
||||
return res.status(400).json({ error: 'Missing parameter: name' });
|
||||
}
|
||||
|
||||
const token = createToken(name, role, dailyLimit);
|
||||
const token = createToken(name, role);
|
||||
res.json({
|
||||
success: true,
|
||||
token: {
|
||||
872
ConfluentTranslator/src/api/server.js
Normal file
872
ConfluentTranslator/src/api/server.js
Normal file
@ -0,0 +1,872 @@
|
||||
require('dotenv').config();
|
||||
const express = require('express');
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
const { Anthropic } = require('@anthropic-ai/sdk');
|
||||
const OpenAI = require('openai');
|
||||
const {
|
||||
loadAllLexiques,
|
||||
searchLexique,
|
||||
generateLexiqueSummary,
|
||||
buildReverseIndex
|
||||
} = require('../utils/lexiqueLoader');
|
||||
const { analyzeContext } = require('../core/translation/contextAnalyzer');
|
||||
const { buildContextualPrompt, getBasePrompt, getPromptStats } = require('../core/translation/promptBuilder');
|
||||
const { buildReverseIndex: buildConfluentIndex } = require('../core/morphology/reverseIndexBuilder');
|
||||
const { translateConfluentToFrench, translateConfluentDetailed } = require('../core/translation/confluentToFrench');
|
||||
|
||||
// Security modules
|
||||
const { authenticate, requireAdmin, createToken, listTokens, disableToken, enableToken, deleteToken, getGlobalStats, trackLLMUsage, checkLLMLimit } = require('../utils/auth');
|
||||
const { adminLimiter } = require('../utils/rateLimiter');
|
||||
const { requestLogger, getLogs, getLogStats } = require('../utils/logger');
|
||||
|
||||
const app = express();
|
||||
const PORT = process.env.PORT || 3000;
|
||||
|
||||
// Middlewares
|
||||
app.use(express.json());
|
||||
app.use(requestLogger); // Log toutes les requêtes
|
||||
// Rate limiting: on utilise uniquement checkLLMLimit() par API key, pas de rate limit global par IP
|
||||
|
||||
// Route protégée pour admin.html (AVANT express.static)
|
||||
// Vérifie l'auth seulement si API key présente, sinon laisse passer (le JS client vérifiera)
|
||||
app.get('/admin.html', (req, res, next) => {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
|
||||
// Si pas d'API key, c'est une requête browser normale -> laisser passer
|
||||
if (!apiKey) {
|
||||
return res.sendFile(path.join(__dirname, '..', '..', 'public', 'admin.html'));
|
||||
}
|
||||
|
||||
// Si API key présente, vérifier qu'elle est admin
|
||||
authenticate(req, res, (authErr) => {
|
||||
if (authErr) return next(authErr);
|
||||
requireAdmin(req, res, (adminErr) => {
|
||||
if (adminErr) return next(adminErr);
|
||||
res.sendFile(path.join(__dirname, '..', '..', 'public', 'admin.html'));
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
app.use(express.static(path.join(__dirname, '..', '..', 'public')));
|
||||
|
||||
// Load prompts
|
||||
const protoPrompt = fs.readFileSync(path.join(__dirname, '..', '..', 'prompts', 'proto-system.txt'), 'utf-8');
|
||||
const ancienPrompt = fs.readFileSync(path.join(__dirname, '..', '..', 'prompts', 'ancien-system.txt'), 'utf-8');
|
||||
|
||||
// Load lexiques dynamically from JSON files
|
||||
const baseDir = path.join(__dirname, '..', '..');
|
||||
let lexiques = { proto: null, ancien: null };
|
||||
let reverseIndexes = { proto: null, ancien: null };
|
||||
let confluentIndexes = { proto: null, ancien: null };
|
||||
|
||||
function reloadLexiques() {
|
||||
console.log('Loading lexiques...');
|
||||
lexiques = loadAllLexiques(baseDir);
|
||||
reverseIndexes = {
|
||||
proto: buildReverseIndex(lexiques.proto),
|
||||
ancien: buildReverseIndex(lexiques.ancien)
|
||||
};
|
||||
confluentIndexes = {
|
||||
proto: buildConfluentIndex(lexiques.proto),
|
||||
ancien: buildConfluentIndex(lexiques.ancien)
|
||||
};
|
||||
console.log('Lexiques loaded successfully');
|
||||
console.log(`Confluent→FR index: ${Object.keys(confluentIndexes.ancien || {}).length} entries`);
|
||||
}
|
||||
|
||||
// Initial load
|
||||
reloadLexiques();
|
||||
|
||||
// Health check endpoint (public - for login validation)
|
||||
app.get('/api/health', (req, res) => {
|
||||
res.json({
|
||||
status: 'ok',
|
||||
timestamp: new Date().toISOString(),
|
||||
version: '1.0.0'
|
||||
});
|
||||
});
|
||||
|
||||
// Auth validation endpoint (tests API key without exposing data)
|
||||
app.get('/api/validate', authenticate, (req, res) => {
|
||||
res.json({
|
||||
valid: true,
|
||||
user: req.user?.name || 'anonymous',
|
||||
role: req.user?.role || 'user'
|
||||
});
|
||||
});
|
||||
|
||||
// LLM limit check endpoint - Always returns 200 with info
|
||||
app.get('/api/llm/limit', authenticate, (req, res) => {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
|
||||
console.log('[/api/llm/limit] Check result:', limitCheck); // Debug
|
||||
|
||||
// TOUJOURS retourner 200 avec les données
|
||||
// Cet endpoint ne bloque jamais, il informe seulement
|
||||
res.status(200).json(limitCheck);
|
||||
});
|
||||
|
||||
// Legacy lexique endpoint (for backward compatibility) - SECURED
|
||||
app.get('/lexique', authenticate, (req, res) => {
|
||||
// Return ancien-confluent by default (legacy behavior)
|
||||
if (!lexiques.ancien) {
|
||||
return res.status(500).json({ error: 'Lexique not loaded' });
|
||||
}
|
||||
res.json(lexiques.ancien);
|
||||
});
|
||||
|
||||
// New lexique endpoints - SECURED
|
||||
app.get('/api/lexique/:variant', authenticate, (req, res) => {
|
||||
const { variant } = req.params;
|
||||
|
||||
if (variant !== 'proto' && variant !== 'ancien') {
|
||||
return res.status(400).json({ error: 'Invalid variant. Use "proto" or "ancien"' });
|
||||
}
|
||||
|
||||
if (!lexiques[variant]) {
|
||||
return res.status(500).json({ error: `Lexique ${variant} not loaded` });
|
||||
}
|
||||
|
||||
res.json(lexiques[variant]);
|
||||
});
|
||||
|
||||
// Stats endpoint - SECURED
|
||||
app.get('/api/stats', authenticate, (req, res) => {
|
||||
const { variant = 'ancien' } = req.query;
|
||||
|
||||
if (variant !== 'proto' && variant !== 'ancien') {
|
||||
return res.status(400).json({ error: 'Invalid variant. Use "proto" or "ancien"' });
|
||||
}
|
||||
|
||||
if (!lexiques[variant]) {
|
||||
return res.status(500).json({ error: `Lexique ${variant} not loaded` });
|
||||
}
|
||||
|
||||
const lexique = lexiques[variant];
|
||||
const stats = {
|
||||
motsCF: 0, // Mots Confluent uniques
|
||||
motsFR: 0, // Mots français uniques
|
||||
totalTraductions: 0, // Total de traductions
|
||||
racines: 0, // Racines (racine, racine_sacree)
|
||||
racinesSacrees: 0, // Racines sacrées
|
||||
racinesStandards: 0, // Racines standards
|
||||
compositions: 0, // Compositions
|
||||
verbes: 0, // Verbes
|
||||
verbesIrreguliers: 0, // Verbes irréguliers
|
||||
particules: 0, // Particules grammaticales (negation, particule, interrogation, demonstratif)
|
||||
nomsPropes: 0, // Noms propres
|
||||
marqueurs: 0, // Marqueurs (temps, aspect, nombre)
|
||||
pronoms: 0, // Pronoms (pronom, possessif, relatif, determinant)
|
||||
autres: 0 // Autres types (auxiliaire, quantificateur, etc.)
|
||||
};
|
||||
|
||||
const motsCFSet = new Set();
|
||||
const motsFRSet = new Set();
|
||||
|
||||
// Le lexique peut avoir une structure {dictionnaire: {...}} ou être directement un objet
|
||||
const dict = lexique.dictionnaire || lexique;
|
||||
|
||||
// Parcourir le dictionnaire
|
||||
Object.keys(dict).forEach(motFR => {
|
||||
const entry = dict[motFR];
|
||||
motsFRSet.add(motFR);
|
||||
|
||||
if (entry.traductions) {
|
||||
entry.traductions.forEach(trad => {
|
||||
stats.totalTraductions++;
|
||||
|
||||
// Compter les mots CF uniques
|
||||
if (trad.confluent) {
|
||||
motsCFSet.add(trad.confluent);
|
||||
}
|
||||
|
||||
// Compter par type
|
||||
const type = trad.type || '';
|
||||
if (type === 'racine') {
|
||||
stats.racines++;
|
||||
stats.racinesStandards++;
|
||||
} else if (type === 'racine_sacree') {
|
||||
stats.racines++;
|
||||
stats.racinesSacrees++;
|
||||
} else if (type === 'composition' || type === 'racine_sacree_composee') {
|
||||
stats.compositions++;
|
||||
} else if (type === 'verbe') {
|
||||
stats.verbes++;
|
||||
} else if (type === 'verbe_irregulier') {
|
||||
stats.verbes++;
|
||||
stats.verbesIrreguliers++;
|
||||
} else if (type === 'negation' || type === 'particule' || type === 'interrogation' || type === 'demonstratif') {
|
||||
stats.particules++;
|
||||
} else if (type === 'nom_propre') {
|
||||
stats.nomsPropes++;
|
||||
} else if (type === 'marqueur_temps' || type === 'marqueur_aspect' || type === 'marqueur_nombre') {
|
||||
stats.marqueurs++;
|
||||
} else if (type === 'pronom' || type === 'possessif' || type === 'relatif' || type === 'determinant') {
|
||||
stats.pronoms++;
|
||||
} else if (type !== '') {
|
||||
stats.autres++;
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
stats.motsCF = motsCFSet.size;
|
||||
stats.motsFR = motsFRSet.size;
|
||||
|
||||
res.json(stats);
|
||||
});
|
||||
|
||||
// Search endpoint - SECURED
|
||||
app.get('/api/search', authenticate, (req, res) => {
|
||||
const { q, variant = 'ancien', direction = 'fr2conf' } = req.query;
|
||||
|
||||
if (!q) {
|
||||
return res.status(400).json({ error: 'Missing query parameter "q"' });
|
||||
}
|
||||
|
||||
if (variant !== 'proto' && variant !== 'ancien') {
|
||||
return res.status(400).json({ error: 'Invalid variant. Use "proto" or "ancien"' });
|
||||
}
|
||||
|
||||
const results = searchLexique(lexiques[variant], q, direction);
|
||||
res.json({ query: q, variant, direction, results });
|
||||
});
|
||||
|
||||
// Reload endpoint (for development) - SECURED (admin only)
|
||||
app.post('/api/reload', authenticate, requireAdmin, (req, res) => {
|
||||
try {
|
||||
reloadLexiques();
|
||||
res.json({
|
||||
success: true,
|
||||
message: 'Lexiques reloaded',
|
||||
stats: {
|
||||
proto: lexiques.proto?.meta?.total_entries || 0,
|
||||
ancien: lexiques.ancien?.meta?.total_entries || 0
|
||||
}
|
||||
});
|
||||
} catch (error) {
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Build enhanced prompt with lexique data
|
||||
function buildEnhancedPrompt(basePrompt, variant) {
|
||||
const lexique = lexiques[variant];
|
||||
if (!lexique) return basePrompt;
|
||||
|
||||
const summary = generateLexiqueSummary(lexique, 300);
|
||||
|
||||
return `${basePrompt}
|
||||
|
||||
# LEXIQUE COMPLET (${lexique.meta.total_entries} entrées)
|
||||
${summary}
|
||||
`;
|
||||
}
|
||||
|
||||
// Debug endpoint: Generate prompt without calling LLM - SECURED
|
||||
app.post('/api/debug/prompt', authenticate, (req, res) => {
|
||||
const { text, target = 'ancien', useLexique = true } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
let systemPrompt;
|
||||
let contextMetadata = null;
|
||||
|
||||
// MÊME CODE QUE /translate
|
||||
if (useLexique) {
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
systemPrompt = buildContextualPrompt(contextResult, variant, text);
|
||||
|
||||
const promptStats = getPromptStats(systemPrompt, contextResult);
|
||||
contextMetadata = {
|
||||
wordsFound: contextResult.metadata.wordsFound,
|
||||
wordsNotFound: contextResult.metadata.wordsNotFound,
|
||||
entriesUsed: contextResult.metadata.entriesUsed,
|
||||
totalLexiqueSize: contextResult.metadata.totalLexiqueSize,
|
||||
tokensFullLexique: promptStats.fullLexiqueTokens,
|
||||
tokensUsed: promptStats.promptTokens,
|
||||
tokensSaved: promptStats.tokensSaved,
|
||||
savingsPercent: promptStats.savingsPercent,
|
||||
useFallback: contextResult.useFallback,
|
||||
expansionLevel: contextResult.metadata.expansionLevel
|
||||
};
|
||||
} else {
|
||||
systemPrompt = getBasePrompt(variant);
|
||||
}
|
||||
|
||||
res.json({
|
||||
prompt: systemPrompt,
|
||||
metadata: contextMetadata,
|
||||
stats: {
|
||||
promptLength: systemPrompt.length,
|
||||
promptLines: systemPrompt.split('\n').length
|
||||
}
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Prompt generation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Coverage analysis endpoint (analyze French text before translation) - SECURED
|
||||
app.post('/api/analyze/coverage', authenticate, (req, res) => {
|
||||
const { text, target = 'ancien' } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
// Use the same contextAnalyzer as the translation pipeline
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
const metadata = contextResult.metadata;
|
||||
|
||||
// Calculate recommendation
|
||||
const needsFullRoots = metadata.coveragePercent < 90;
|
||||
let recommendation;
|
||||
if (metadata.coveragePercent >= 95) {
|
||||
recommendation = 'Excellent coverage - context only';
|
||||
} else if (metadata.coveragePercent >= 90) {
|
||||
recommendation = 'Good coverage - context only';
|
||||
} else if (metadata.coveragePercent >= 70) {
|
||||
recommendation = 'Moderate coverage - consider adding roots';
|
||||
} else if (metadata.coveragePercent >= 50) {
|
||||
recommendation = 'Low coverage - full roots recommended';
|
||||
} else {
|
||||
recommendation = 'Very low coverage - full roots required';
|
||||
}
|
||||
|
||||
res.json({
|
||||
coverage: metadata.coveragePercent,
|
||||
found: metadata.wordsFound.map(w => ({
|
||||
word: w.input,
|
||||
confluent: w.confluent,
|
||||
type: w.type,
|
||||
score: w.score
|
||||
})),
|
||||
missing: metadata.wordsNotFound.map(word => ({
|
||||
word,
|
||||
suggestions: [] // TODO: add suggestions based on similar words
|
||||
})),
|
||||
stats: {
|
||||
totalWords: metadata.wordCount,
|
||||
uniqueWords: metadata.uniqueWordCount,
|
||||
foundCount: metadata.wordsFound.length,
|
||||
missingCount: metadata.wordsNotFound.length,
|
||||
entriesUsed: metadata.entriesUsed,
|
||||
useFallback: metadata.useFallback
|
||||
},
|
||||
needsFullRoots,
|
||||
recommendation,
|
||||
variant
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Coverage analysis error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Translation endpoint (NOUVEAU SYSTÈME CONTEXTUEL)
|
||||
app.post('/translate', authenticate, async (req, res) => {
|
||||
const { text, target, provider, model, temperature = 1.0, useLexique = true, customAnthropicKey, customOpenAIKey } = req.body;
|
||||
|
||||
if (!text || !target || !provider || !model) {
|
||||
return res.status(400).json({ error: 'Missing parameters' });
|
||||
}
|
||||
|
||||
// Check for custom API keys
|
||||
const usingCustomKey = !!(customAnthropicKey || customOpenAIKey);
|
||||
|
||||
// Only check rate limit if NOT using custom keys
|
||||
if (!usingCustomKey) {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
if (!limitCheck.allowed) {
|
||||
return res.status(429).json({
|
||||
error: limitCheck.error,
|
||||
limit: limitCheck.limit,
|
||||
used: limitCheck.used
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
let systemPrompt;
|
||||
let contextMetadata = null;
|
||||
|
||||
// NOUVEAU: Analyse contextuelle et génération de prompt optimisé
|
||||
if (useLexique) {
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
systemPrompt = buildContextualPrompt(contextResult, variant, text);
|
||||
|
||||
// Générer métadonnées pour Layer 2
|
||||
const promptStats = getPromptStats(systemPrompt, contextResult);
|
||||
contextMetadata = {
|
||||
wordsFound: contextResult.metadata.wordsFound,
|
||||
wordsNotFound: contextResult.metadata.wordsNotFound,
|
||||
entriesUsed: contextResult.metadata.entriesUsed,
|
||||
totalLexiqueSize: contextResult.metadata.totalLexiqueSize,
|
||||
tokensFullLexique: promptStats.fullLexiqueTokens,
|
||||
tokensUsed: promptStats.promptTokens,
|
||||
tokensSaved: promptStats.tokensSaved,
|
||||
savingsPercent: promptStats.savingsPercent,
|
||||
useFallback: contextResult.useFallback,
|
||||
expansionLevel: contextResult.metadata.expansionLevel,
|
||||
rootsUsed: contextResult.rootsFallback?.length || 0 // Nombre de racines envoyées
|
||||
};
|
||||
} else {
|
||||
systemPrompt = getBasePrompt(variant);
|
||||
}
|
||||
|
||||
let translation;
|
||||
let rawResponse;
|
||||
|
||||
if (provider === 'anthropic') {
|
||||
const anthropic = new Anthropic({
|
||||
apiKey: customAnthropicKey || process.env.ANTHROPIC_API_KEY,
|
||||
});
|
||||
|
||||
const message = await anthropic.messages.create({
|
||||
model: model,
|
||||
max_tokens: 8192, // Max pour Claude Sonnet/Haiku 4.5
|
||||
temperature: temperature / 2, // Diviser par 2 pour Claude (max 1.0)
|
||||
system: systemPrompt,
|
||||
messages: [
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = message.content[0].text;
|
||||
translation = rawResponse;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && message.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, message.usage.input_tokens, message.usage.output_tokens);
|
||||
}
|
||||
|
||||
} else if (provider === 'openai') {
|
||||
const openai = new OpenAI({
|
||||
apiKey: customOpenAIKey || process.env.OPENAI_API_KEY,
|
||||
});
|
||||
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: model,
|
||||
max_tokens: 16384, // Max pour GPT-4o et GPT-4o-mini
|
||||
temperature: temperature,
|
||||
messages: [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = completion.choices[0].message.content;
|
||||
translation = rawResponse;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && completion.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, completion.usage.prompt_tokens, completion.usage.completion_tokens);
|
||||
}
|
||||
} else {
|
||||
return res.status(400).json({ error: 'Unknown provider' });
|
||||
}
|
||||
|
||||
// Parser la réponse pour extraire Layer 1 et Layer 3
|
||||
const parsed = parseTranslationResponse(rawResponse);
|
||||
|
||||
// Construire la réponse avec les 3 layers
|
||||
const response = {
|
||||
// Layer 1: Traduction
|
||||
layer1: {
|
||||
translation: parsed.translation
|
||||
},
|
||||
|
||||
// Layer 2: Contexte (COT hors LLM)
|
||||
layer2: contextMetadata,
|
||||
|
||||
// Layer 3: Explications LLM (avec COT)
|
||||
layer3: {
|
||||
analyse: parsed.analyse,
|
||||
strategie: parsed.strategie,
|
||||
decomposition: parsed.decomposition,
|
||||
notes: parsed.notes,
|
||||
wordsCreated: parsed.wordsCreated || []
|
||||
},
|
||||
|
||||
// Compatibilité avec ancien format
|
||||
translation: parsed.translation
|
||||
};
|
||||
|
||||
res.json(response);
|
||||
|
||||
} catch (error) {
|
||||
console.error('Translation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Parse la réponse du LLM pour extraire les différentes sections (avec COT)
|
||||
* @param {string} response - Réponse brute du LLM
|
||||
* @returns {Object} - Sections parsées
|
||||
*/
|
||||
function parseTranslationResponse(response) {
|
||||
const lines = response.split('\n');
|
||||
|
||||
let analyse = '';
|
||||
let strategie = '';
|
||||
let translation = '';
|
||||
let decomposition = '';
|
||||
let notes = '';
|
||||
let currentSection = null;
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
|
||||
// Détecter les sections (nouveau format COT)
|
||||
if (trimmed.match(/^ANALYSE:/i)) {
|
||||
currentSection = 'analyse';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^STRAT[ÉE]GIE:/i)) {
|
||||
currentSection = 'strategie';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^(Ancien )?Confluent:/i)) {
|
||||
currentSection = 'translation';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^D[ée]composition:/i)) {
|
||||
currentSection = 'decomposition';
|
||||
continue;
|
||||
}
|
||||
if (trimmed.match(/^Notes?:/i) || trimmed.match(/^Explication:/i)) {
|
||||
currentSection = 'notes';
|
||||
continue;
|
||||
}
|
||||
|
||||
// Ajouter le contenu à la section appropriée
|
||||
if (currentSection === 'analyse' && trimmed && !trimmed.match(/^---/)) {
|
||||
analyse += line + '\n';
|
||||
} else if (currentSection === 'strategie' && trimmed && !trimmed.match(/^---/)) {
|
||||
strategie += line + '\n';
|
||||
} else if (currentSection === 'translation' && trimmed && !trimmed.match(/^---/)) {
|
||||
translation += line + '\n';
|
||||
} else if (currentSection === 'decomposition' && trimmed) {
|
||||
decomposition += line + '\n';
|
||||
} else if (currentSection === 'notes' && trimmed) {
|
||||
notes += line + '\n';
|
||||
} else if (!currentSection && trimmed && !trimmed.match(/^---/) && !trimmed.match(/^\*\*/)) {
|
||||
// Si pas de section détectée, c'est probablement la traduction
|
||||
translation += line + '\n';
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
analyse: analyse.trim(),
|
||||
strategie: strategie.trim(),
|
||||
translation: translation.trim() || response.trim(),
|
||||
decomposition: decomposition.trim(),
|
||||
notes: notes.trim()
|
||||
};
|
||||
}
|
||||
|
||||
// Raw translation endpoint (for debugging - returns unprocessed LLM output) - SECURED
|
||||
app.post('/api/translate/raw', authenticate, async (req, res) => {
|
||||
const { text, target, provider, model, useLexique = true, customAnthropicKey, customOpenAIKey } = req.body;
|
||||
|
||||
if (!text || !target || !provider || !model) {
|
||||
return res.status(400).json({ error: 'Missing parameters' });
|
||||
}
|
||||
|
||||
// Check for custom API keys
|
||||
const usingCustomKey = !!(customAnthropicKey || customOpenAIKey);
|
||||
|
||||
// Only check rate limit if NOT using custom keys
|
||||
if (!usingCustomKey) {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
if (!limitCheck.allowed) {
|
||||
return res.status(429).json({
|
||||
error: limitCheck.error,
|
||||
limit: limitCheck.limit,
|
||||
used: limitCheck.used
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
try {
|
||||
let systemPrompt;
|
||||
let contextMetadata = null;
|
||||
|
||||
if (useLexique) {
|
||||
const contextResult = analyzeContext(text, lexiques[variant]);
|
||||
systemPrompt = buildContextualPrompt(contextResult, variant, text);
|
||||
|
||||
const promptStats = getPromptStats(systemPrompt, contextResult);
|
||||
contextMetadata = {
|
||||
wordsFound: contextResult.metadata.wordsFound,
|
||||
wordsNotFound: contextResult.metadata.wordsNotFound,
|
||||
entriesUsed: contextResult.metadata.entriesUsed,
|
||||
totalLexiqueSize: contextResult.metadata.totalLexiqueSize,
|
||||
tokensFullLexique: promptStats.fullLexiqueTokens,
|
||||
tokensUsed: promptStats.promptTokens,
|
||||
tokensSaved: promptStats.tokensSaved,
|
||||
savingsPercent: promptStats.savingsPercent,
|
||||
useFallback: contextResult.useFallback,
|
||||
expansionLevel: contextResult.metadata.expansionLevel
|
||||
};
|
||||
} else {
|
||||
systemPrompt = getBasePrompt(variant);
|
||||
}
|
||||
|
||||
let rawResponse;
|
||||
|
||||
if (provider === 'anthropic') {
|
||||
const anthropic = new Anthropic({
|
||||
apiKey: customAnthropicKey || process.env.ANTHROPIC_API_KEY,
|
||||
});
|
||||
|
||||
const message = await anthropic.messages.create({
|
||||
model: model,
|
||||
max_tokens: 8192, // Max pour Claude Sonnet/Haiku 4.5
|
||||
system: systemPrompt,
|
||||
messages: [
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = message.content[0].text;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && message.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, message.usage.input_tokens, message.usage.output_tokens);
|
||||
}
|
||||
|
||||
} else if (provider === 'openai') {
|
||||
const openai = new OpenAI({
|
||||
apiKey: customOpenAIKey || process.env.OPENAI_API_KEY,
|
||||
});
|
||||
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: model,
|
||||
max_tokens: 16384, // Max pour GPT-4o et GPT-4o-mini
|
||||
messages: [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
{ role: 'user', content: text }
|
||||
]
|
||||
});
|
||||
|
||||
rawResponse = completion.choices[0].message.content;
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && completion.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, completion.usage.prompt_tokens, completion.usage.completion_tokens);
|
||||
}
|
||||
} else {
|
||||
return res.status(400).json({ error: 'Unknown provider' });
|
||||
}
|
||||
|
||||
// Retourner la réponse BRUTE sans parsing
|
||||
res.json({
|
||||
raw_output: rawResponse,
|
||||
metadata: contextMetadata,
|
||||
length: rawResponse.length,
|
||||
lines: rawResponse.split('\n').length
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Translation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Batch translation endpoint - SECURED
|
||||
app.post('/api/translate/batch', authenticate, async (req, res) => {
|
||||
const { words, target = 'ancien' } = req.body;
|
||||
|
||||
if (!words || !Array.isArray(words)) {
|
||||
return res.status(400).json({ error: 'Missing or invalid "words" array' });
|
||||
}
|
||||
|
||||
const variant = target === 'proto' ? 'proto' : 'ancien';
|
||||
const results = {};
|
||||
|
||||
for (const word of words) {
|
||||
const found = searchLexique(lexiques[variant], word, 'fr2conf');
|
||||
if (found.length > 0 && found[0].traductions?.length > 0) {
|
||||
results[word] = {
|
||||
found: true,
|
||||
traduction: found[0].traductions[0].confluent,
|
||||
all_traductions: found[0].traductions
|
||||
};
|
||||
} else {
|
||||
results[word] = { found: false };
|
||||
}
|
||||
}
|
||||
|
||||
res.json({ target, results });
|
||||
});
|
||||
|
||||
// Confluent → French translation endpoint (traduction brute) - SECURED
|
||||
app.post('/api/translate/conf2fr', authenticate, (req, res) => {
|
||||
const { text, variant = 'ancien', detailed = false } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
const variantKey = variant === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
if (!confluentIndexes[variantKey]) {
|
||||
return res.status(500).json({ error: `Confluent index for ${variantKey} not loaded` });
|
||||
}
|
||||
|
||||
try {
|
||||
if (detailed) {
|
||||
const result = translateConfluentDetailed(text, confluentIndexes[variantKey]);
|
||||
res.json(result);
|
||||
} else {
|
||||
const result = translateConfluentToFrench(text, confluentIndexes[variantKey]);
|
||||
res.json(result);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Confluent→FR translation error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// NEW: Confluent → French with LLM refinement
|
||||
app.post('/api/translate/conf2fr/llm', authenticate, async (req, res) => {
|
||||
const { text, variant = 'ancien', provider = 'anthropic', model = 'claude-sonnet-4-20250514', customAnthropicKey, customOpenAIKey } = req.body;
|
||||
|
||||
if (!text) {
|
||||
return res.status(400).json({ error: 'Missing parameter: text' });
|
||||
}
|
||||
|
||||
// Check for custom API keys
|
||||
const usingCustomKey = !!(customAnthropicKey || customOpenAIKey);
|
||||
|
||||
// Only check rate limit if NOT using custom keys
|
||||
if (!usingCustomKey) {
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
const limitCheck = checkLLMLimit(apiKey);
|
||||
if (!limitCheck.allowed) {
|
||||
return res.status(429).json({
|
||||
error: limitCheck.error,
|
||||
limit: limitCheck.limit,
|
||||
used: limitCheck.used
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const variantKey = variant === 'proto' ? 'proto' : 'ancien';
|
||||
|
||||
if (!confluentIndexes[variantKey]) {
|
||||
return res.status(500).json({ error: `Confluent index for ${variantKey} not loaded` });
|
||||
}
|
||||
|
||||
try {
|
||||
// Step 1: Get raw word-by-word translation
|
||||
const rawTranslation = translateConfluentToFrench(text, confluentIndexes[variantKey]);
|
||||
|
||||
// Step 2: Load refinement prompt
|
||||
const refinementPrompt = fs.readFileSync(path.join(__dirname, 'prompts', 'cf2fr-refinement.txt'), 'utf-8');
|
||||
|
||||
// Step 3: Use LLM to refine translation
|
||||
let refinedText;
|
||||
|
||||
if (provider === 'anthropic') {
|
||||
const anthropic = new Anthropic({
|
||||
apiKey: customAnthropicKey || process.env.ANTHROPIC_API_KEY,
|
||||
});
|
||||
|
||||
const message = await anthropic.messages.create({
|
||||
model: model,
|
||||
max_tokens: 2048,
|
||||
system: refinementPrompt,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: `Voici la traduction brute mot-à-mot du Confluent vers le français. Transforme-la en français fluide et naturel:\n\n${rawTranslation.translation}`
|
||||
}
|
||||
]
|
||||
});
|
||||
|
||||
refinedText = message.content[0].text.trim();
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && message.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, message.usage.input_tokens, message.usage.output_tokens);
|
||||
}
|
||||
} else if (provider === 'openai') {
|
||||
const openai = new OpenAI({
|
||||
apiKey: customOpenAIKey || process.env.OPENAI_API_KEY,
|
||||
});
|
||||
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: model,
|
||||
messages: [
|
||||
{ role: 'system', content: refinementPrompt },
|
||||
{ role: 'user', content: `Voici la traduction brute mot-à-mot du Confluent vers le français. Transforme-la en français fluide et naturel:\n\n${rawTranslation.translation}` }
|
||||
]
|
||||
});
|
||||
|
||||
refinedText = completion.choices[0].message.content.trim();
|
||||
|
||||
// Track LLM usage (only increment counter if NOT using custom key)
|
||||
const apiKey = req.headers['x-api-key'] || req.query.apiKey;
|
||||
if (apiKey && completion.usage && !usingCustomKey) {
|
||||
trackLLMUsage(apiKey, completion.usage.prompt_tokens, completion.usage.completion_tokens);
|
||||
}
|
||||
} else {
|
||||
return res.status(400).json({ error: 'Unsupported provider. Use "anthropic" or "openai".' });
|
||||
}
|
||||
|
||||
// Return both raw and refined versions with detailed token info
|
||||
res.json({
|
||||
confluentText: text,
|
||||
rawTranslation: rawTranslation.translation,
|
||||
refinedTranslation: refinedText,
|
||||
translation: refinedText, // For compatibility
|
||||
tokens: rawTranslation.tokens || [],
|
||||
coverage: rawTranslation.coverage || 0,
|
||||
wordsTranslated: rawTranslation.wordsTranslated,
|
||||
wordsNotTranslated: rawTranslation.wordsNotTranslated,
|
||||
provider,
|
||||
model
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Confluent→FR LLM refinement error:', error);
|
||||
res.status(500).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Admin routes
|
||||
const adminRoutes = require('./adminRoutes');
|
||||
app.use('/api/admin', authenticate, adminRoutes);
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`ConfluentTranslator running on http://localhost:${PORT}`);
|
||||
console.log(`Loaded: ${lexiques.ancien?.meta?.total_entries || 0} ancien entries, ${lexiques.proto?.meta?.total_entries || 0} proto entries`);
|
||||
});
|
||||
@ -2,7 +2,7 @@
|
||||
// Système de décomposition morphologique pour le Confluent
|
||||
// Permet de décomposer les mots composés selon le pattern Racine-Liaison-Racine
|
||||
|
||||
const lexique = require('../data/lexique.json');
|
||||
const lexique = require('../../data/lexique.json');
|
||||
|
||||
// ============================================================================
|
||||
// CHARGEMENT DYNAMIQUE DES LIAISONS DEPUIS LE LEXIQUE
|
||||
@ -2,7 +2,7 @@
|
||||
// Système de recherche par radicaux pour le traducteur Confluent→Français
|
||||
// Permet de trouver les formes conjuguées et dérivées à partir des racines
|
||||
|
||||
const lexique = require('../data/lexique.json');
|
||||
const lexique = require('../../data/lexique.json');
|
||||
|
||||
// ============================================================================
|
||||
// CHARGEMENT DYNAMIQUE DES SUFFIXES DEPUIS LE LEXIQUE
|
||||
@ -9,8 +9,8 @@
|
||||
* 5. Décomposition morphologique (nouveauté)
|
||||
*/
|
||||
|
||||
const { extractRadicals } = require('./radicalMatcher');
|
||||
const { decomposeWord } = require('./morphologicalDecomposer');
|
||||
const { extractRadicals } = require('../morphology/radicalMatcher');
|
||||
const { decomposeWord } = require('../morphology/morphologicalDecomposer');
|
||||
|
||||
/**
|
||||
* Tokenize un texte Confluent
|
||||
@ -10,7 +10,7 @@
|
||||
* 6. Conversion automatique des nombres français → Confluent
|
||||
*/
|
||||
|
||||
const { convertFrenchNumber, isNumber } = require('./numberConverter');
|
||||
const { convertFrenchNumber, isNumber } = require('../numbers/numberConverter');
|
||||
|
||||
/**
|
||||
* FONCTION CENTRALE DE NORMALISATION
|
||||
@ -10,7 +10,7 @@
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { preprocessNumbers } = require('./numberPreprocessor');
|
||||
const { preprocessNumbers } = require('../numbers/numberPreprocessor');
|
||||
|
||||
/**
|
||||
* Charge le template de prompt de base depuis les fichiers
|
||||
@ -33,8 +33,6 @@ function loadTokens() {
|
||||
apiKey: uuidv4(),
|
||||
createdAt: new Date().toISOString(),
|
||||
active: true,
|
||||
requestsToday: 0,
|
||||
dailyLimit: -1, // illimité
|
||||
// Tracking des tokens LLM
|
||||
llmTokens: {
|
||||
totalInput: 0,
|
||||
@ -44,10 +42,7 @@ function loadTokens() {
|
||||
output: 0,
|
||||
date: new Date().toISOString().split('T')[0]
|
||||
}
|
||||
},
|
||||
// Rate limiting LLM (illimité pour admin)
|
||||
llmRequestsToday: 0,
|
||||
llmDailyLimit: -1
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@ -89,20 +84,7 @@ function authenticate(req, res, next) {
|
||||
return res.status(403).json({ error: 'Token disabled' });
|
||||
}
|
||||
|
||||
// Vérifier la limite quotidienne
|
||||
const today = new Date().toISOString().split('T')[0];
|
||||
const tokenToday = token.lastUsed?.split('T')[0];
|
||||
|
||||
if (tokenToday !== today) {
|
||||
token.requestsToday = 0;
|
||||
}
|
||||
|
||||
if (token.dailyLimit > 0 && token.requestsToday >= token.dailyLimit) {
|
||||
return res.status(429).json({ error: 'Daily limit reached' });
|
||||
}
|
||||
|
||||
// Mettre à jour les stats
|
||||
token.requestsToday++;
|
||||
token.lastUsed = new Date().toISOString();
|
||||
saveTokens();
|
||||
|
||||
@ -125,7 +107,7 @@ function requireAdmin(req, res, next) {
|
||||
}
|
||||
|
||||
// Créer un nouveau token
|
||||
function createToken(name, role = 'user', dailyLimit = 100) {
|
||||
function createToken(name, role = 'user') {
|
||||
const id = uuidv4();
|
||||
const apiKey = uuidv4();
|
||||
|
||||
@ -136,8 +118,6 @@ function createToken(name, role = 'user', dailyLimit = 100) {
|
||||
apiKey,
|
||||
createdAt: new Date().toISOString(),
|
||||
active: true,
|
||||
requestsToday: 0,
|
||||
dailyLimit,
|
||||
// Tracking des tokens LLM
|
||||
llmTokens: {
|
||||
totalInput: 0,
|
||||
@ -147,10 +127,7 @@ function createToken(name, role = 'user', dailyLimit = 100) {
|
||||
output: 0,
|
||||
date: new Date().toISOString().split('T')[0]
|
||||
}
|
||||
},
|
||||
// Rate limiting LLM
|
||||
llmRequestsToday: 0,
|
||||
llmDailyLimit: 20
|
||||
}
|
||||
};
|
||||
|
||||
saveTokens();
|
||||
@ -166,8 +143,6 @@ function listTokens() {
|
||||
apiKey: t.apiKey.substring(0, 8) + '...',
|
||||
createdAt: t.createdAt,
|
||||
active: t.active,
|
||||
requestsToday: t.requestsToday,
|
||||
dailyLimit: t.dailyLimit,
|
||||
lastUsed: t.lastUsed
|
||||
}));
|
||||
}
|
||||
@ -210,8 +185,7 @@ function getGlobalStats() {
|
||||
const tokenList = Object.values(tokens);
|
||||
return {
|
||||
totalTokens: tokenList.length,
|
||||
activeTokens: tokenList.filter(t => t.active).length,
|
||||
totalRequestsToday: tokenList.reduce((sum, t) => sum + t.requestsToday, 0)
|
||||
activeTokens: tokenList.filter(t => t.active).length
|
||||
};
|
||||
}
|
||||
|
||||
@ -225,7 +199,7 @@ function checkLLMLimit(apiKey) {
|
||||
if (token.llmRequestsToday === undefined) {
|
||||
token.llmRequestsToday = 0;
|
||||
token.llmDailyLimit = token.role === 'admin' ? -1 : 20;
|
||||
saveTokens(); // Sauvegarder l'initialisation
|
||||
saveTokens();
|
||||
}
|
||||
|
||||
// Initialiser llmTokens.today.date si n'existe pas
|
||||
119
ancien-confluent/README.md
Normal file
119
ancien-confluent/README.md
Normal file
@ -0,0 +1,119 @@
|
||||
# Ancien Confluent - Lexique
|
||||
|
||||
Ce dossier contient le lexique complet de la langue Confluent dans sa version "ancien".
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
ancien-confluent/
|
||||
├── lexique/ # Fichiers JSON du lexique (31 catégories)
|
||||
│ ├── 00-grammaire.json
|
||||
│ ├── 01-racines-sacrees.json
|
||||
│ ├── 02-racines-standards.json
|
||||
│ └── ... (28 autres fichiers)
|
||||
│
|
||||
├── docs/ # Documentation générée
|
||||
│ └── LEXIQUE-COMPLET.md # Lexique complet en Markdown (généré)
|
||||
│
|
||||
├── generer-lexique-complet.js # Script de génération (Node.js)
|
||||
└── generer-lexique-complet.bat # Script pour Windows
|
||||
```
|
||||
|
||||
## Génération du lexique complet
|
||||
|
||||
Le lexique complet est généré automatiquement à partir des fichiers JSON.
|
||||
|
||||
### Sous Linux/Mac/WSL
|
||||
|
||||
```bash
|
||||
node generer-lexique-complet.js
|
||||
```
|
||||
|
||||
### Sous Windows
|
||||
|
||||
Double-cliquez sur `generer-lexique-complet.bat` ou exécutez :
|
||||
|
||||
```cmd
|
||||
generer-lexique-complet.bat
|
||||
```
|
||||
|
||||
### Résultat
|
||||
|
||||
Le script génère le fichier `docs/LEXIQUE-COMPLET.md` qui contient :
|
||||
- Une table des matières cliquable
|
||||
- 31 catégories organisées
|
||||
- 835+ entrées de lexique
|
||||
- Pour chaque entrée :
|
||||
- Le mot français
|
||||
- La traduction en Confluent
|
||||
- La forme liée (si applicable)
|
||||
- Le type (racine sacrée, racine standard, etc.)
|
||||
- Le domaine
|
||||
- Les notes
|
||||
- Les synonymes français
|
||||
|
||||
## Format des fichiers JSON
|
||||
|
||||
Chaque fichier JSON suit cette structure :
|
||||
|
||||
```json
|
||||
{
|
||||
"_comment": "Description de la catégorie",
|
||||
"_mots_a_gerer": [],
|
||||
"dictionnaire": {
|
||||
"mot_francais": {
|
||||
"traductions": [
|
||||
{
|
||||
"confluent": "motsconfluent",
|
||||
"type": "racine_sacree",
|
||||
"forme_liee": "form",
|
||||
"domaine": "domaine_concept",
|
||||
"note": "Note explicative"
|
||||
}
|
||||
],
|
||||
"synonymes_fr": ["synonyme1", "synonyme2"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Statistiques
|
||||
|
||||
- **31 catégories** de vocabulaire
|
||||
- **835+ entrées** au total
|
||||
- **19 racines sacrées** (commencent par une voyelle)
|
||||
- **67 racines standards**
|
||||
|
||||
## Catégories disponibles
|
||||
|
||||
1. Grammaire et Règles
|
||||
2. Racines Sacrées
|
||||
3. Racines Standards
|
||||
4. Castes
|
||||
5. Lieux
|
||||
6. Corps et Sens
|
||||
7. Actions
|
||||
8. Émotions
|
||||
9. Nature et Éléments
|
||||
10. Institutions
|
||||
11. Animaux
|
||||
12. Armes et Outils
|
||||
13. Concepts Abstraits
|
||||
14. Rituels
|
||||
15. Géographie
|
||||
16. Rôles et Titres
|
||||
17. Communication
|
||||
18. Temps
|
||||
19. Couleurs
|
||||
20. Santé et Dangers
|
||||
21. Objets et Matériaux
|
||||
22. Famille
|
||||
23. Nombres
|
||||
24. Nourriture
|
||||
25. Habitat
|
||||
26. Navigation
|
||||
27. Architecture
|
||||
28. Concepts Philosophiques
|
||||
29. Étrangers
|
||||
30. Actions Militaires
|
||||
31. Vêtements et Apparence
|
||||
8456
ancien-confluent/docs/LEXIQUE-COMPLET.md
Normal file
8456
ancien-confluent/docs/LEXIQUE-COMPLET.md
Normal file
File diff suppressed because it is too large
Load Diff
25
ancien-confluent/generer-lexique-complet.bat
Normal file
25
ancien-confluent/generer-lexique-complet.bat
Normal file
@ -0,0 +1,25 @@
|
||||
@echo off
|
||||
REM Script batch pour générer le lexique complet sous Windows
|
||||
REM Appelle simplement le script Node.js
|
||||
|
||||
echo.
|
||||
echo ========================================
|
||||
echo Generation du lexique complet
|
||||
echo ========================================
|
||||
echo.
|
||||
|
||||
node "%~dp0generer-lexique-complet.js"
|
||||
|
||||
if %ERRORLEVEL% EQU 0 (
|
||||
echo.
|
||||
echo ========================================
|
||||
echo Terminé avec succès !
|
||||
echo ========================================
|
||||
) else (
|
||||
echo.
|
||||
echo ========================================
|
||||
echo Erreur lors de la génération
|
||||
echo ========================================
|
||||
)
|
||||
|
||||
pause
|
||||
191
ancien-confluent/generer-lexique-complet.js
Normal file
191
ancien-confluent/generer-lexique-complet.js
Normal file
@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Script de génération du lexique complet en Markdown
|
||||
* Lit tous les fichiers JSON du dossier ./lexique/ et génère ./docs/LEXIQUE-COMPLET.md
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
// Chemins relatifs (pas de hard path)
|
||||
const LEXIQUE_DIR = path.join(__dirname, 'lexique');
|
||||
const OUTPUT_FILE = path.join(__dirname, 'docs', 'LEXIQUE-COMPLET.md');
|
||||
|
||||
// Mapping des noms de fichiers vers des titres lisibles
|
||||
const CATEGORIES = {
|
||||
'00-grammaire': 'Grammaire et Règles',
|
||||
'01-racines-sacrees': 'Racines Sacrées',
|
||||
'02-racines-standards': 'Racines Standards',
|
||||
'03-castes': 'Castes',
|
||||
'04-lieux': 'Lieux',
|
||||
'05-corps-sens': 'Corps et Sens',
|
||||
'06-actions': 'Actions',
|
||||
'07-emotions': 'Émotions',
|
||||
'08-nature-elements': 'Nature et Éléments',
|
||||
'09-institutions': 'Institutions',
|
||||
'10-animaux': 'Animaux',
|
||||
'11-armes-outils': 'Armes et Outils',
|
||||
'12-abstraits': 'Concepts Abstraits',
|
||||
'13-rituels': 'Rituels',
|
||||
'14-geographie': 'Géographie',
|
||||
'15-roles-titres': 'Rôles et Titres',
|
||||
'16-communication': 'Communication',
|
||||
'17-temps': 'Temps',
|
||||
'18-couleurs': 'Couleurs',
|
||||
'19-sante-dangers': 'Santé et Dangers',
|
||||
'20-objets-materiaux': 'Objets et Matériaux',
|
||||
'21-famille': 'Famille',
|
||||
'22-nombres': 'Nombres',
|
||||
'23-nourriture': 'Nourriture',
|
||||
'24-habitat': 'Habitat',
|
||||
'25-navigation': 'Navigation',
|
||||
'26-architecture': 'Architecture',
|
||||
'27-concepts-philosophiques': 'Concepts Philosophiques',
|
||||
'28-etrangers': 'Étrangers',
|
||||
'29-actions-militaires': 'Actions Militaires',
|
||||
'30-vetements-apparence': 'Vêtements et Apparence'
|
||||
};
|
||||
|
||||
/**
|
||||
* Génère une section Markdown pour une catégorie
|
||||
*/
|
||||
function generateCategorySection(categoryName, data) {
|
||||
let markdown = `## ${categoryName}\n\n`;
|
||||
|
||||
if (!data.dictionnaire) {
|
||||
return markdown + '*Aucune entrée*\n\n';
|
||||
}
|
||||
|
||||
// Trier les mots français par ordre alphabétique
|
||||
const sortedWords = Object.keys(data.dictionnaire).sort();
|
||||
|
||||
for (const motFr of sortedWords) {
|
||||
const entry = data.dictionnaire[motFr];
|
||||
|
||||
markdown += `### ${motFr}\n\n`;
|
||||
|
||||
// Traductions en Confluent
|
||||
if (entry.traductions && entry.traductions.length > 0) {
|
||||
for (const trad of entry.traductions) {
|
||||
markdown += `**Confluent:** ${trad.confluent}`;
|
||||
|
||||
if (trad.forme_liee) {
|
||||
markdown += ` *(forme liée: ${trad.forme_liee})*`;
|
||||
}
|
||||
|
||||
markdown += `\n`;
|
||||
|
||||
if (trad.type) {
|
||||
markdown += `- Type: ${trad.type}\n`;
|
||||
}
|
||||
|
||||
if (trad.composition) {
|
||||
markdown += `- Composition: \`${trad.composition}\`\n`;
|
||||
}
|
||||
|
||||
if (trad.domaine) {
|
||||
markdown += `- Domaine: ${trad.domaine}\n`;
|
||||
}
|
||||
|
||||
if (trad.note) {
|
||||
markdown += `- Note: ${trad.note}\n`;
|
||||
}
|
||||
|
||||
markdown += '\n';
|
||||
}
|
||||
}
|
||||
|
||||
// Synonymes français
|
||||
if (entry.synonymes_fr && entry.synonymes_fr.length > 0) {
|
||||
markdown += `*Synonymes français:* ${entry.synonymes_fr.join(', ')}\n\n`;
|
||||
}
|
||||
|
||||
markdown += '---\n\n';
|
||||
}
|
||||
|
||||
return markdown;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fonction principale
|
||||
*/
|
||||
function main() {
|
||||
console.log('🔨 Génération du lexique complet...\n');
|
||||
|
||||
// Vérifier que le dossier lexique existe
|
||||
if (!fs.existsSync(LEXIQUE_DIR)) {
|
||||
console.error(`❌ Erreur: Le dossier ${LEXIQUE_DIR} n'existe pas`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Créer le dossier docs s'il n'existe pas
|
||||
const docsDir = path.dirname(OUTPUT_FILE);
|
||||
if (!fs.existsSync(docsDir)) {
|
||||
fs.mkdirSync(docsDir, { recursive: true });
|
||||
console.log(`📁 Dossier créé: ${docsDir}`);
|
||||
}
|
||||
|
||||
// Lire tous les fichiers JSON du lexique
|
||||
const files = fs.readdirSync(LEXIQUE_DIR)
|
||||
.filter(f => f.endsWith('.json') && !f.startsWith('_') && !f.endsWith('.backup'))
|
||||
.sort();
|
||||
|
||||
console.log(`📚 ${files.length} fichiers de lexique trouvés\n`);
|
||||
|
||||
// Générer le header Markdown
|
||||
let markdown = `# Lexique Complet du Confluent\n\n`;
|
||||
markdown += `*Généré automatiquement le ${new Date().toLocaleDateString('fr-FR')} à ${new Date().toLocaleTimeString('fr-FR')}*\n\n`;
|
||||
markdown += `---\n\n`;
|
||||
markdown += `## Table des matières\n\n`;
|
||||
|
||||
// Générer la table des matières
|
||||
for (const file of files) {
|
||||
const baseName = path.basename(file, '.json');
|
||||
const categoryName = CATEGORIES[baseName] || baseName;
|
||||
markdown += `- [${categoryName}](#${categoryName.toLowerCase().replace(/\s+/g, '-').replace(/[éè]/g, 'e').replace(/[àâ]/g, 'a')})\n`;
|
||||
}
|
||||
|
||||
markdown += `\n---\n\n`;
|
||||
|
||||
// Générer les sections pour chaque catégorie
|
||||
let totalEntries = 0;
|
||||
|
||||
for (const file of files) {
|
||||
const baseName = path.basename(file, '.json');
|
||||
const categoryName = CATEGORIES[baseName] || baseName;
|
||||
const filePath = path.join(LEXIQUE_DIR, file);
|
||||
|
||||
console.log(`📖 Traitement de: ${categoryName}`);
|
||||
|
||||
try {
|
||||
const data = JSON.parse(fs.readFileSync(filePath, 'utf-8'));
|
||||
|
||||
if (data.dictionnaire) {
|
||||
const entryCount = Object.keys(data.dictionnaire).length;
|
||||
totalEntries += entryCount;
|
||||
console.log(` → ${entryCount} entrées`);
|
||||
}
|
||||
|
||||
markdown += generateCategorySection(categoryName, data);
|
||||
|
||||
} catch (err) {
|
||||
console.error(`❌ Erreur lors de la lecture de ${file}:`, err.message);
|
||||
markdown += `## ${categoryName}\n\n*Erreur lors du chargement de cette catégorie*\n\n`;
|
||||
}
|
||||
}
|
||||
|
||||
// Écrire le fichier de sortie
|
||||
fs.writeFileSync(OUTPUT_FILE, markdown, 'utf-8');
|
||||
|
||||
console.log(`\n✅ Lexique généré avec succès!`);
|
||||
console.log(`📊 Total: ${totalEntries} entrées`);
|
||||
console.log(`📝 Fichier créé: ${OUTPUT_FILE}\n`);
|
||||
}
|
||||
|
||||
// Exécuter le script
|
||||
if (require.main === module) {
|
||||
main();
|
||||
}
|
||||
|
||||
module.exports = { generateCategorySection };
|
||||
Loading…
Reference in New Issue
Block a user