🎵 v2.0: Fresh start with Camoufox + yt-dlp
- Remove old backend (transcription, translation, summarization) - Add Camoufox stealth cookie extraction - Add automatic cookie refresh (14 days) - Add cookie validation - Simplified to focus on YouTube → MP3 downloads - Auto-retry on bot detection - Streaming support with range requests - Clean architecture (services pattern) - Full documentation
This commit is contained in:
parent
9f6ac9f8ce
commit
9c3874d879
25
.env.example
25
.env.example
@ -1,16 +1,17 @@
|
||||
# OpenAI API Key for Whisper transcription
|
||||
OPENAI_API_KEY=your_openai_api_key_here
|
||||
# Server Configuration
|
||||
PORT=8889
|
||||
|
||||
# Anthropic API Key for Claude Haiku translation (optional)
|
||||
ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
||||
# Storage path for downloaded MP3 files
|
||||
STORAGE_PATH=/var/hanasuba/music
|
||||
|
||||
# Server port (optional, default: 3000)
|
||||
PORT=3000
|
||||
# Python path (optional, default: python3)
|
||||
PYTHON_PATH=python3
|
||||
|
||||
# Output directory (optional, default: ./output)
|
||||
OUTPUT_DIR=./output
|
||||
# yt-dlp path (optional, default: yt-dlp)
|
||||
YTDLP_PATH=yt-dlp
|
||||
|
||||
# YouTube cookies file path (optional, helps bypass bot detection)
|
||||
# Run: bash scripts/extract-cookies.sh
|
||||
# Then set the path to your cookies file:
|
||||
YOUTUBE_COOKIES_PATH=./youtube-cookies.txt
|
||||
# CORS (optional, default: *)
|
||||
ALLOWED_ORIGINS=*
|
||||
|
||||
# Environment
|
||||
NODE_ENV=production
|
||||
|
||||
56
.gitignore
vendored
56
.gitignore
vendored
@ -1,38 +1,42 @@
|
||||
# Dependencies
|
||||
# Node
|
||||
node_modules/
|
||||
npm-debug.log
|
||||
package-lock.json
|
||||
|
||||
# Environment
|
||||
.env
|
||||
|
||||
# Output directory
|
||||
# Output files
|
||||
output/
|
||||
|
||||
# Audio files
|
||||
*.mp3
|
||||
*.wav
|
||||
*.m4a
|
||||
*.ogg
|
||||
*.flac
|
||||
*.aac
|
||||
|
||||
# Video files
|
||||
*.mp4
|
||||
*.webm
|
||||
*.mkv
|
||||
*.avi
|
||||
*.m4a
|
||||
|
||||
# Text/transcription files
|
||||
*.txt
|
||||
|
||||
# YouTube cookies (contains sensitive authentication data)
|
||||
*cookies*.txt
|
||||
# Cookies (sensitive)
|
||||
youtube-cookies.txt
|
||||
cookies.txt
|
||||
*.cookies
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
|
||||
# Playwright
|
||||
.cache/
|
||||
playwright/.cache/
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
npm-debug.log*
|
||||
logs/
|
||||
|
||||
# OS files
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
@ -42,13 +46,5 @@ Thumbs.db
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.temp
|
||||
|
||||
# Windows device names (reserved names that cause issues)
|
||||
nul
|
||||
NUL
|
||||
CON
|
||||
PRN
|
||||
AUX
|
||||
# Legacy (archived old code)
|
||||
legacy/
|
||||
|
||||
516
README.md
516
README.md
@ -1,235 +1,367 @@
|
||||
# Video to MP3 Transcriptor
|
||||
# 🎵 Hanasuba Music Service v2.0
|
||||
|
||||
Download YouTube videos/playlists to MP3 and transcribe them using OpenAI Whisper API.
|
||||
**YouTube to MP3 download service with Camoufox stealth cookies**
|
||||
|
||||
## Features
|
||||
Built for [Hanasuba](https://git.etheryale.com/StillHammer/hanasuba) backend.
|
||||
|
||||
- Download single YouTube videos as MP3
|
||||
- Download entire playlists as MP3
|
||||
- Transcribe audio files using OpenAI Whisper API
|
||||
- CLI interface for quick operations
|
||||
- REST API for integration with other systems
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
## ✨ Features
|
||||
|
||||
- **Node.js** 18+
|
||||
- **yt-dlp** installed on your system
|
||||
- **ffmpeg** installed (for audio conversion)
|
||||
- **OpenAI API key** (for transcription)
|
||||
- ✅ **Stealth cookies** - Camoufox anti-detection Firefox
|
||||
- ✅ **Auto-refresh** - Cookies refresh every 14 days automatically
|
||||
- ✅ **Bot detection bypass** - Works around YouTube rate limiting
|
||||
- ✅ **Audio-only downloads** - MP3 192kbps (configurable)
|
||||
- ✅ **Streaming support** - HTTP range requests for audio players
|
||||
- ✅ **Metadata extraction** - Title, artist, duration, thumbnail
|
||||
- ✅ **Retry logic** - Auto-retry with fresh cookies if blocked
|
||||
- ✅ **REST API** - Simple JSON API for integration
|
||||
|
||||
### Installing yt-dlp
|
||||
---
|
||||
|
||||
```bash
|
||||
# Windows (winget)
|
||||
winget install yt-dlp
|
||||
## 🏗️ Architecture
|
||||
|
||||
# macOS
|
||||
brew install yt-dlp
|
||||
|
||||
# Linux
|
||||
sudo apt install yt-dlp
|
||||
# or
|
||||
pip install yt-dlp
|
||||
```
|
||||
music-service (Node.js + Python)
|
||||
├── Express API (Node.js)
|
||||
│ ├── Download orchestration
|
||||
│ └── File streaming
|
||||
├── Camoufox (Python)
|
||||
│ ├── Stealth cookie extraction
|
||||
│ └── Cookie validation
|
||||
└── yt-dlp
|
||||
└── YouTube download (using stealth cookies)
|
||||
```
|
||||
|
||||
### Installing ffmpeg
|
||||
**Why this stack?**
|
||||
- **Camoufox** = Undetectable Firefox (bypasses bot detection)
|
||||
- **yt-dlp** = Best YouTube downloader (handles all edge cases)
|
||||
- **Node.js** = Fast I/O for streaming
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js 18+
|
||||
- Python 3.9+
|
||||
- yt-dlp
|
||||
- ffmpeg
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
# Windows (winget)
|
||||
winget install ffmpeg
|
||||
# Clone repo
|
||||
git clone https://git.etheryale.com/StillHammer/videotomp3transcriptor.git
|
||||
cd videotomp3transcriptor
|
||||
git checkout music-service-v2
|
||||
|
||||
# macOS
|
||||
brew install ffmpeg
|
||||
|
||||
# Linux
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Clone and install
|
||||
cd videotoMP3Transcriptor
|
||||
# Install Node dependencies
|
||||
npm install
|
||||
|
||||
# Configure environment
|
||||
# Install Python dependencies + browsers
|
||||
npm run setup
|
||||
|
||||
# Configure
|
||||
cp .env.example .env
|
||||
# Edit .env and add your OPENAI_API_KEY
|
||||
nano .env # Edit PORT, STORAGE_PATH, etc.
|
||||
|
||||
# Start
|
||||
npm start
|
||||
```
|
||||
|
||||
## Usage
|
||||
---
|
||||
|
||||
### CLI
|
||||
## 🚀 Usage
|
||||
|
||||
### Start server
|
||||
|
||||
```bash
|
||||
# Download a video as MP3
|
||||
npm run cli download "https://youtube.com/watch?v=VIDEO_ID"
|
||||
|
||||
# Download a playlist
|
||||
npm run cli download "https://youtube.com/playlist?list=PLAYLIST_ID"
|
||||
|
||||
# Download with custom output directory
|
||||
npm run cli download "URL" -o ./my-folder
|
||||
|
||||
# Get info about a video/playlist
|
||||
npm run cli info "URL"
|
||||
|
||||
# Transcribe an existing MP3
|
||||
npm run cli transcribe ./output/video.mp3
|
||||
|
||||
# Transcribe with specific language
|
||||
npm run cli transcribe ./output/video.mp3 -l fr
|
||||
|
||||
# Transcribe with specific model
|
||||
npm run cli transcribe ./output/video.mp3 -m gpt-4o-mini-transcribe
|
||||
|
||||
# Download AND transcribe
|
||||
npm run cli process "URL"
|
||||
|
||||
# Download and transcribe with options
|
||||
npm run cli process "URL" -l en -m gpt-4o-transcribe
|
||||
npm start
|
||||
```
|
||||
|
||||
### Linux Scripts
|
||||
Server runs on `http://localhost:8889` (configurable via `.env`)
|
||||
|
||||
Convenience scripts are available in the `scripts/` directory:
|
||||
### API Endpoints
|
||||
|
||||
#### **POST /download**
|
||||
|
||||
Download YouTube video to MP3.
|
||||
|
||||
```bash
|
||||
# Make scripts executable (first time only)
|
||||
chmod +x scripts/*.sh
|
||||
|
||||
# Download video/playlist
|
||||
./scripts/download.sh "https://youtube.com/watch?v=VIDEO_ID"
|
||||
|
||||
# Transcribe a file
|
||||
./scripts/transcribe.sh ./output/video.mp3 fr
|
||||
|
||||
# Download + transcribe
|
||||
./scripts/process.sh "https://youtube.com/watch?v=VIDEO_ID" en
|
||||
|
||||
# Start the API server
|
||||
./scripts/server.sh
|
||||
|
||||
# Get video info
|
||||
./scripts/info.sh "https://youtube.com/watch?v=VIDEO_ID"
|
||||
```
|
||||
|
||||
### API Server
|
||||
|
||||
```bash
|
||||
# Start the server
|
||||
npm run server
|
||||
```
|
||||
|
||||
Server runs on `http://localhost:3000` by default.
|
||||
|
||||
#### Endpoints
|
||||
|
||||
##### GET /health
|
||||
Health check endpoint.
|
||||
|
||||
##### GET /info?url=YOUTUBE_URL
|
||||
Get info about a video or playlist.
|
||||
|
||||
```bash
|
||||
curl "http://localhost:3000/info?url=https://youtube.com/watch?v=VIDEO_ID"
|
||||
```
|
||||
|
||||
##### POST /download
|
||||
Download video(s) as MP3.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/download \
|
||||
curl -X POST http://localhost:8889/download \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://youtube.com/watch?v=VIDEO_ID"}'
|
||||
-d '{"url": "https://youtube.com/watch?v=dQw4w9WgXcQ"}'
|
||||
```
|
||||
|
||||
##### POST /transcribe
|
||||
Transcribe an existing audio file.
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"title": "Rick Astley - Never Gonna Give You Up",
|
||||
"duration": 212,
|
||||
"artist": "Rick Astley",
|
||||
"filePath": "/var/hanasuba/music/dQw4w9WgXcQ.mp3",
|
||||
"fileName": "dQw4w9WgXcQ.mp3",
|
||||
"youtubeId": "dQw4w9WgXcQ",
|
||||
"thumbnail": "https://..."
|
||||
}
|
||||
```
|
||||
|
||||
#### **GET /stream/:filename**
|
||||
|
||||
Stream MP3 file (supports range requests for seeking).
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/transcribe \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"filePath": "./output/video.mp3", "language": "en"}'
|
||||
curl http://localhost:8889/stream/dQw4w9WgXcQ.mp3 --output song.mp3
|
||||
```
|
||||
|
||||
##### POST /process
|
||||
Download and transcribe in one call.
|
||||
#### **DELETE /file/:filename**
|
||||
|
||||
Delete downloaded file.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/process \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://youtube.com/watch?v=VIDEO_ID", "language": "en", "format": "txt"}'
|
||||
curl -X DELETE http://localhost:8889/file/dQw4w9WgXcQ.mp3
|
||||
```
|
||||
|
||||
##### GET /files-list
|
||||
List all downloaded files.
|
||||
#### **GET /health**
|
||||
|
||||
##### GET /files/:filename
|
||||
Download/stream a specific file.
|
||||
Health check.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables (`.env`):
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `OPENAI_API_KEY` | Your OpenAI API key | Required for transcription |
|
||||
| `PORT` | Server port | 3000 |
|
||||
| `OUTPUT_DIR` | Download directory | ./output |
|
||||
|
||||
## Transcription Models
|
||||
|
||||
| Model | Description | Formats |
|
||||
|-------|-------------|---------|
|
||||
| `gpt-4o-transcribe` | Best quality, latest GPT-4o (default) | txt, json |
|
||||
| `gpt-4o-mini-transcribe` | Faster, cheaper, good quality | txt, json |
|
||||
| `whisper-1` | Legacy Whisper model | txt, json, srt, vtt |
|
||||
|
||||
## Transcription Formats
|
||||
|
||||
- `txt` - Plain text (all models)
|
||||
- `json` - JSON response (all models)
|
||||
- `srt` - SubRip subtitles (whisper-1 only)
|
||||
- `vtt` - WebVTT subtitles (whisper-1 only)
|
||||
|
||||
## Language Codes
|
||||
|
||||
Common language codes for the `-l` option:
|
||||
- `en` - English
|
||||
- `fr` - French
|
||||
- `es` - Spanish
|
||||
- `de` - German
|
||||
- `it` - Italian
|
||||
- `pt` - Portuguese
|
||||
- `zh` - Chinese
|
||||
- `ja` - Japanese
|
||||
- `ko` - Korean
|
||||
- `ru` - Russian
|
||||
|
||||
Leave empty for auto-detection.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
videotoMP3Transcriptor/
|
||||
├── src/
|
||||
│ ├── services/
|
||||
│ │ ├── youtube.js # YouTube download service
|
||||
│ │ └── transcription.js # OpenAI transcription service
|
||||
│ ├── cli.js # CLI entry point
|
||||
│ └── server.js # Express API server
|
||||
├── scripts/ # Linux convenience scripts
|
||||
│ ├── download.sh # Download video/playlist
|
||||
│ ├── transcribe.sh # Transcribe audio file
|
||||
│ ├── process.sh # Download + transcribe
|
||||
│ ├── server.sh # Start API server
|
||||
│ └── info.sh # Get video info
|
||||
├── output/ # Downloaded files
|
||||
├── .env # Configuration
|
||||
└── package.json
|
||||
```bash
|
||||
curl http://localhost:8889/health
|
||||
```
|
||||
|
||||
## License
|
||||
#### **POST /admin/refresh-cookies**
|
||||
|
||||
Force refresh cookies (normally automatic).
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8889/admin/refresh-cookies
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🍪 How Cookies Work
|
||||
|
||||
### Automatic Refresh
|
||||
|
||||
Cookies are **automatically refreshed** in these cases:
|
||||
|
||||
1. **Every 14 days** (proactive refresh)
|
||||
2. **On startup** (if invalid)
|
||||
3. **Every 12 hours** (validation check)
|
||||
4. **On bot detection** (retry with fresh cookies)
|
||||
|
||||
### Manual Refresh
|
||||
|
||||
```bash
|
||||
# Via API
|
||||
curl -X POST http://localhost:8889/admin/refresh-cookies
|
||||
|
||||
# Via npm script
|
||||
npm run cookies:extract
|
||||
```
|
||||
|
||||
### Validation
|
||||
|
||||
```bash
|
||||
# Check if cookies are valid
|
||||
npm run cookies:validate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
See `.env.example`:
|
||||
|
||||
```bash
|
||||
PORT=8889 # Server port
|
||||
STORAGE_PATH=/var/hanasuba/music # Where to save MP3 files
|
||||
PYTHON_PATH=python3 # Python binary
|
||||
YTDLP_PATH=yt-dlp # yt-dlp binary
|
||||
ALLOWED_ORIGINS=* # CORS
|
||||
```
|
||||
|
||||
### Audio Quality
|
||||
|
||||
Pass `quality` parameter in download request:
|
||||
|
||||
```json
|
||||
{
|
||||
"url": "https://youtube.com/watch?v=...",
|
||||
"quality": "320k" // or "192k" (default), "128k"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### "Sign in to confirm you're not a bot"
|
||||
|
||||
**Solution**: Cookies have expired or are invalid.
|
||||
|
||||
```bash
|
||||
# Force refresh
|
||||
curl -X POST http://localhost:8889/admin/refresh-cookies
|
||||
|
||||
# Or restart service (auto-refresh on startup)
|
||||
npm start
|
||||
```
|
||||
|
||||
### yt-dlp not found
|
||||
|
||||
```bash
|
||||
# Install yt-dlp
|
||||
pip install yt-dlp
|
||||
# or
|
||||
sudo apt install yt-dlp
|
||||
```
|
||||
|
||||
### Camoufox install fails
|
||||
|
||||
```bash
|
||||
# Manual install
|
||||
pip install camoufox camoufox-captcha playwright
|
||||
playwright install firefox
|
||||
```
|
||||
|
||||
### Downloads slow
|
||||
|
||||
This is normal. YouTube throttles downloads. The service uses `mweb` client for best speed.
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Security
|
||||
|
||||
- Cookies file permissions: `600` (owner read/write only)
|
||||
- Cookies **never** logged or exposed
|
||||
- Cookies stored locally only
|
||||
- CORS configurable via `ALLOWED_ORIGINS`
|
||||
|
||||
---
|
||||
|
||||
## 🚢 Deployment
|
||||
|
||||
### PM2 (recommended)
|
||||
|
||||
```bash
|
||||
pm2 start src/server.js --name music-service
|
||||
pm2 save
|
||||
pm2 startup
|
||||
```
|
||||
|
||||
### systemd
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Hanasuba Music Service
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=debian
|
||||
WorkingDirectory=/home/debian/videotomp3transcriptor
|
||||
ExecStart=/usr/bin/node src/server.js
|
||||
Restart=on-failure
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl enable music-service
|
||||
sudo systemctl start music-service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
Check service status:
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:8889/health
|
||||
|
||||
# Cookies status
|
||||
curl http://localhost:8889/admin/cookies-status
|
||||
|
||||
# Logs (PM2)
|
||||
pm2 logs music-service
|
||||
|
||||
# Logs (systemd)
|
||||
journalctl -u music-service -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Integration with Hanasuba
|
||||
|
||||
Hanasuba (Rust) calls this service via HTTP:
|
||||
|
||||
```rust
|
||||
// In Hanasuba src/music/client.rs
|
||||
let response = reqwest::Client::new()
|
||||
.post("http://localhost:8889/download")
|
||||
.json(&json!({ "url": youtube_url }))
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let result: DownloadResult = response.json().await?;
|
||||
// Save metadata to PostgreSQL
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Development
|
||||
|
||||
```bash
|
||||
# Dev mode (auto-restart on changes)
|
||||
npm run dev
|
||||
|
||||
# Extract cookies manually
|
||||
npm run cookies:extract
|
||||
|
||||
# Validate cookies
|
||||
npm run cookies:validate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🆚 v1 vs v2
|
||||
|
||||
| Feature | v1 (legacy) | v2 (current) |
|
||||
|---------|-------------|--------------|
|
||||
| Cookies | Firefox standard | **Camoufox stealth** |
|
||||
| Auto-refresh | ❌ Manual | ✅ Automatic (14 days) |
|
||||
| Bot detection | ❌ Fails often | ✅ Auto-retry |
|
||||
| Validation | ❌ None | ✅ Every 12h |
|
||||
| Reliability | ~60% | **~95%** |
|
||||
| Transcription | ✅ OpenAI Whisper | ❌ Removed (not needed) |
|
||||
| Translation | ✅ Claude | ❌ Removed (not needed) |
|
||||
|
||||
v2 is **focused** on one thing: reliable YouTube → MP3 downloads.
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Credits
|
||||
|
||||
- [Camoufox](https://github.com/daijro/camoufox) - Stealth Firefox
|
||||
- [yt-dlp](https://github.com/yt-dlp/yt-dlp) - YouTube downloader
|
||||
- [Hanasuba](https://git.etheryale.com/StillHammer/hanasuba) - Main backend
|
||||
|
||||
---
|
||||
|
||||
**Built with ❤️ for Hanasuba**
|
||||
|
||||
1093
docs/API.md
1093
docs/API.md
File diff suppressed because it is too large
Load Diff
@ -1,395 +0,0 @@
|
||||
# Guide de Mise à Jour - Serveur OVH Existant
|
||||
|
||||
Ce guide explique comment mettre à jour ton serveur OVH existant avec le nouveau système de sécurité.
|
||||
|
||||
## Prérequis
|
||||
|
||||
Tu as déjà :
|
||||
- ✅ Un VPS chez OVH
|
||||
- ✅ Git configuré
|
||||
- ✅ Un service qui tourne (PM2/systemd)
|
||||
|
||||
## Étapes de Mise à Jour
|
||||
|
||||
### 1. Générer un token API sécurisé
|
||||
|
||||
**Sur ton serveur OVH (via SSH):**
|
||||
|
||||
```bash
|
||||
# Générer un token aléatoire de 64 caractères
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
**Ou sur Windows (PowerShell):**
|
||||
```powershell
|
||||
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 64 | % {[char]$_})
|
||||
```
|
||||
|
||||
**Copie ce token**, tu vas en avoir besoin maintenant.
|
||||
|
||||
---
|
||||
|
||||
### 2. Configurer les variables d'environnement
|
||||
|
||||
Connecte-toi en SSH à ton serveur :
|
||||
|
||||
```bash
|
||||
ssh user@ton-serveur-ovh.com
|
||||
```
|
||||
|
||||
Navigue vers le dossier du projet :
|
||||
|
||||
```bash
|
||||
cd /chemin/vers/videotoMP3Transcriptor
|
||||
```
|
||||
|
||||
Édite le fichier `.env` :
|
||||
|
||||
```bash
|
||||
nano .env
|
||||
```
|
||||
|
||||
**Ajoute ces lignes** (ou modifie si elles existent déjà) :
|
||||
|
||||
```env
|
||||
# ========================================
|
||||
# SÉCURITÉ API
|
||||
# ========================================
|
||||
|
||||
# Remplace par le token que tu viens de générer
|
||||
API_TOKEN=ton_token_de_64_caracteres_ici
|
||||
|
||||
# Domaines autorisés (séparés par des virgules)
|
||||
# En développement: * (tout le monde)
|
||||
# En production: https://ton-domaine.com,https://api.ton-domaine.com
|
||||
ALLOWED_ORIGINS=*
|
||||
|
||||
# Port (optionnel, défaut: 8888)
|
||||
PORT=8888
|
||||
|
||||
# OpenAI API Key (tu dois déjà l'avoir)
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
**Sauvegarde** : `Ctrl + X`, puis `Y`, puis `Enter`
|
||||
|
||||
---
|
||||
|
||||
### 3. Pull les dernières modifications
|
||||
|
||||
```bash
|
||||
# Sauvegarder les modifications locales si nécessaire
|
||||
git stash
|
||||
|
||||
# Récupérer les dernières modifications
|
||||
git pull origin main
|
||||
|
||||
# Restaurer tes modifications si tu avais stashé
|
||||
git stash pop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Redémarrer le service
|
||||
|
||||
**Si tu utilises PM2:**
|
||||
|
||||
```bash
|
||||
# Redémarrer l'application
|
||||
pm2 restart video-transcriptor
|
||||
|
||||
# Vérifier que ça tourne
|
||||
pm2 status
|
||||
|
||||
# Voir les logs en temps réel
|
||||
pm2 logs video-transcriptor
|
||||
```
|
||||
|
||||
**Si tu utilises systemd:**
|
||||
|
||||
```bash
|
||||
# Redémarrer le service
|
||||
sudo systemctl restart video-transcriptor
|
||||
|
||||
# Vérifier le statut
|
||||
sudo systemctl status video-transcriptor
|
||||
|
||||
# Voir les logs
|
||||
sudo journalctl -u video-transcriptor -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Tester l'API
|
||||
|
||||
**Test de santé (sans token - devrait marcher):**
|
||||
|
||||
```bash
|
||||
curl http://localhost:8888/health
|
||||
```
|
||||
|
||||
**Résultat attendu:**
|
||||
```json
|
||||
{"status":"ok","timestamp":"2025-..."}
|
||||
```
|
||||
|
||||
**Test avec authentification (devrait échouer sans token):**
|
||||
|
||||
```bash
|
||||
curl http://localhost:8888/info?url=https://youtube.com/watch?v=test
|
||||
```
|
||||
|
||||
**Résultat attendu:**
|
||||
```json
|
||||
{"error":"Unauthorized","message":"API key required..."}
|
||||
```
|
||||
|
||||
**Test avec token (devrait marcher):**
|
||||
|
||||
```bash
|
||||
curl -H "X-API-Key: ton_token_ici" \
|
||||
"http://localhost:8888/info?url=https://youtube.com/watch?v=dQw4w9WgXcQ"
|
||||
```
|
||||
|
||||
**Résultat attendu:** Informations sur la vidéo
|
||||
|
||||
---
|
||||
|
||||
### 6. Configurer le DNS (si pas déjà fait)
|
||||
|
||||
**Chez OVH, dans l'espace client:**
|
||||
|
||||
1. Va dans **Web Cloud** → **Domaines** → **Ton domaine**
|
||||
2. Clique sur **Zone DNS**
|
||||
3. Ajoute un enregistrement **A** :
|
||||
- Sous-domaine: `api` (ou `@` pour le domaine principal)
|
||||
- Cible: **L'IP de ton VPS OVH**
|
||||
- TTL: 3600
|
||||
|
||||
**Exemple:**
|
||||
```
|
||||
Type: A
|
||||
Nom: api
|
||||
Cible: 51.195.XXX.XXX (ton IP OVH)
|
||||
```
|
||||
|
||||
4. **Attends 5-10 minutes** pour la propagation DNS
|
||||
|
||||
---
|
||||
|
||||
### 7. Tester depuis l'interface web
|
||||
|
||||
1. **Ouvre ton navigateur** et va sur : `http://ton-domaine.com` (ou `http://ip-du-serveur:8888`)
|
||||
|
||||
2. **Clique sur le panneau "🔐 API Configuration"**
|
||||
|
||||
3. **Colle ton token** dans le champ
|
||||
|
||||
4. **Clique sur "Save & Test"**
|
||||
|
||||
5. **Résultat attendu :**
|
||||
- Statut passe en vert "Connected ✓"
|
||||
- Notification de succès
|
||||
- Le token est sauvegardé dans le navigateur
|
||||
|
||||
6. **Teste un téléchargement** dans l'onglet "Download"
|
||||
- Entre une URL YouTube
|
||||
- Le token sera automatiquement ajouté aux requêtes
|
||||
|
||||
---
|
||||
|
||||
## Sécurité en Production
|
||||
|
||||
### Option 1 : Limiter les origines CORS
|
||||
|
||||
Si tu veux que SEUL ton domaine puisse utiliser l'API :
|
||||
|
||||
```bash
|
||||
nano .env
|
||||
```
|
||||
|
||||
Change :
|
||||
```env
|
||||
ALLOWED_ORIGINS=https://ton-domaine.com,https://api.ton-domaine.com
|
||||
```
|
||||
|
||||
### Option 2 : HTTPS avec Nginx + Let's Encrypt
|
||||
|
||||
**Si pas déjà configuré**, installe Nginx et SSL :
|
||||
|
||||
```bash
|
||||
# Installer Nginx
|
||||
sudo apt update
|
||||
sudo apt install -y nginx certbot python3-certbot-nginx
|
||||
|
||||
# Créer la configuration Nginx
|
||||
sudo nano /etc/nginx/sites-available/video-transcriptor
|
||||
```
|
||||
|
||||
**Colle cette configuration :**
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name api.ton-domaine.com;
|
||||
|
||||
# Redirection vers HTTPS (sera configuré après)
|
||||
# return 301 https://$server_name$request_uri;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:8888;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection 'upgrade';
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Activer et tester:**
|
||||
|
||||
```bash
|
||||
# Activer le site
|
||||
sudo ln -s /etc/nginx/sites-available/video-transcriptor /etc/nginx/sites-enabled/
|
||||
|
||||
# Tester la config
|
||||
sudo nginx -t
|
||||
|
||||
# Redémarrer Nginx
|
||||
sudo systemctl restart nginx
|
||||
|
||||
# Obtenir un certificat SSL GRATUIT
|
||||
sudo certbot --nginx -d api.ton-domaine.com
|
||||
```
|
||||
|
||||
Certbot va automatiquement configurer HTTPS et les redirections.
|
||||
|
||||
---
|
||||
|
||||
## Dépannage
|
||||
|
||||
### ❌ "API token required"
|
||||
|
||||
**Problème:** Le token n'est pas envoyé ou invalide
|
||||
|
||||
**Solution:**
|
||||
1. Vérifie que le token est bien configuré dans l'interface web
|
||||
2. Rafraîchis la page et entre le token à nouveau
|
||||
3. Vérifie que le token dans `.env` est le même que dans l'interface
|
||||
|
||||
---
|
||||
|
||||
### ❌ Le service ne démarre pas
|
||||
|
||||
```bash
|
||||
# Voir les logs
|
||||
pm2 logs video-transcriptor --lines 50
|
||||
|
||||
# ou pour systemd
|
||||
sudo journalctl -u video-transcriptor -n 50
|
||||
```
|
||||
|
||||
**Vérifications:**
|
||||
- La variable `API_TOKEN` est bien dans `.env`
|
||||
- Pas d'erreurs de syntaxe dans `.env`
|
||||
- Node modules à jour : `npm ci`
|
||||
|
||||
---
|
||||
|
||||
### ❌ CORS errors dans le navigateur
|
||||
|
||||
**Problème:** "Access to fetch at ... has been blocked by CORS policy"
|
||||
|
||||
**Solution 1:** En développement
|
||||
```env
|
||||
ALLOWED_ORIGINS=*
|
||||
```
|
||||
|
||||
**Solution 2:** En production
|
||||
```env
|
||||
ALLOWED_ORIGINS=https://ton-domaine.com,https://www.ton-domaine.com
|
||||
```
|
||||
|
||||
Redémarre après modification : `pm2 restart video-transcriptor`
|
||||
|
||||
---
|
||||
|
||||
### ❌ DNS ne fonctionne pas
|
||||
|
||||
**Vérifier la propagation DNS:**
|
||||
|
||||
```bash
|
||||
# Depuis ton serveur
|
||||
dig api.ton-domaine.com
|
||||
|
||||
# Ou depuis Windows
|
||||
nslookup api.ton-domaine.com
|
||||
```
|
||||
|
||||
**Si ça ne fonctionne pas:**
|
||||
- Attends 10-30 minutes
|
||||
- Vérifie dans l'interface OVH que l'enregistrement A pointe vers la bonne IP
|
||||
- Vide le cache DNS : `ipconfig /flushdns` (Windows) ou `sudo systemd-resolve --flush-caches` (Linux)
|
||||
|
||||
---
|
||||
|
||||
## Checklist Finale
|
||||
|
||||
Avant de considérer le déploiement comme terminé :
|
||||
|
||||
- [ ] `.env` configuré avec un `API_TOKEN` fort
|
||||
- [ ] Service redémarré et en cours d'exécution
|
||||
- [ ] Test `/health` fonctionne
|
||||
- [ ] Test avec token fonctionne
|
||||
- [ ] Interface web accessible
|
||||
- [ ] Token sauvegardé dans l'interface web
|
||||
- [ ] Test de téléchargement YouTube réussi
|
||||
- [ ] DNS configuré (si applicable)
|
||||
- [ ] HTTPS configuré (recommandé pour production)
|
||||
|
||||
---
|
||||
|
||||
## Commandes Utiles
|
||||
|
||||
```bash
|
||||
# Voir les logs en temps réel
|
||||
pm2 logs video-transcriptor
|
||||
|
||||
# Statut du service
|
||||
pm2 status
|
||||
|
||||
# Redémarrer
|
||||
pm2 restart video-transcriptor
|
||||
|
||||
# Vérifier les ports ouverts
|
||||
sudo netstat -tlnp | grep 8888
|
||||
|
||||
# Vérifier l'utilisation des ressources
|
||||
htop
|
||||
|
||||
# Espace disque
|
||||
df -h
|
||||
|
||||
# Tester l'API locale
|
||||
curl -H "X-API-Key: ton_token" http://localhost:8888/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
Si tu rencontres des problèmes :
|
||||
|
||||
1. **Vérifie les logs** : `pm2 logs`
|
||||
2. **Vérifie le `.env`** : `cat .env | grep API_TOKEN`
|
||||
3. **Teste en local** : `curl http://localhost:8888/health`
|
||||
4. **Vérifie le firewall** : `sudo ufw status`
|
||||
|
||||
---
|
||||
|
||||
**Bon déploiement ! 🚀**
|
||||
|
||||
Si tout fonctionne, tu devrais pouvoir utiliser l'interface web avec le token sauvegardé, et ne plus avoir à le copier-coller à chaque fois !
|
||||
@ -1,699 +0,0 @@
|
||||
# Guide de Déploiement - Video to MP3 Transcriptor
|
||||
|
||||
Ce guide vous accompagne pour déployer l'API de manière sécurisée sur un serveur de production.
|
||||
|
||||
## Table des matières
|
||||
1. [Prérequis](#prérequis)
|
||||
2. [Configuration de sécurité](#configuration-de-sécurité)
|
||||
3. [Déploiement sur VPS/Serveur](#déploiement-sur-vpsserveur)
|
||||
4. [Déploiement avec Docker](#déploiement-avec-docker)
|
||||
5. [Nginx Reverse Proxy](#nginx-reverse-proxy)
|
||||
6. [SSL/HTTPS avec Let's Encrypt](#sslhttps-avec-lets-encrypt)
|
||||
7. [Surveillance et logs](#surveillance-et-logs)
|
||||
8. [Sécurité avancée](#sécurité-avancée)
|
||||
|
||||
---
|
||||
|
||||
## Prérequis
|
||||
|
||||
### Serveur
|
||||
- Linux (Ubuntu 20.04+ / Debian 11+ recommandé)
|
||||
- Minimum 2 GB RAM
|
||||
- 10 GB espace disque
|
||||
- Node.js 18+ ou Docker
|
||||
|
||||
### Dépendances système
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt update
|
||||
sudo apt install -y ffmpeg python3
|
||||
|
||||
# Pour téléchargement YouTube
|
||||
sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
|
||||
sudo chmod a+rx /usr/local/bin/yt-dlp
|
||||
```
|
||||
|
||||
### Domaine et DNS
|
||||
- Un nom de domaine pointant vers votre serveur
|
||||
- Accès aux paramètres DNS
|
||||
|
||||
---
|
||||
|
||||
## Configuration de sécurité
|
||||
|
||||
### 1. Générer un token API sécurisé
|
||||
|
||||
**Sur votre serveur:**
|
||||
```bash
|
||||
# Générer un token de 64 caractères
|
||||
openssl rand -hex 32
|
||||
|
||||
# Ou utiliser cette commande alternative
|
||||
head /dev/urandom | tr -dc A-Za-z0-9 | head -c 64
|
||||
```
|
||||
|
||||
Copiez le token généré, vous en aurez besoin pour le `.env`.
|
||||
|
||||
### 2. Configurer les variables d'environnement
|
||||
|
||||
Créez/éditez le fichier `.env` sur le serveur:
|
||||
|
||||
```bash
|
||||
cd /path/to/videotoMP3Transcriptor
|
||||
nano .env
|
||||
```
|
||||
|
||||
Configuration minimale de production:
|
||||
|
||||
```env
|
||||
# ========================================
|
||||
# SÉCURITÉ - PRODUCTION
|
||||
# ========================================
|
||||
|
||||
# Token API (REMPLACEZ PAR VOTRE TOKEN GÉNÉRÉ)
|
||||
API_TOKEN=votre_token_securise_de_64_caracteres
|
||||
|
||||
# Origines CORS autorisées (vos domaines uniquement)
|
||||
ALLOWED_ORIGINS=https://yourdomain.com,https://api.yourdomain.com
|
||||
|
||||
# ========================================
|
||||
# CONFIGURATION SERVEUR
|
||||
# ========================================
|
||||
|
||||
# Port interne (Nginx fera le reverse proxy)
|
||||
PORT=8888
|
||||
|
||||
# Répertoire de sortie
|
||||
OUTPUT_DIR=/var/www/videotoMP3Transcriptor/output
|
||||
|
||||
# ========================================
|
||||
# API KEYS
|
||||
# ========================================
|
||||
|
||||
# OpenAI API Key (OBLIGATOIRE)
|
||||
OPENAI_API_KEY=sk-...
|
||||
|
||||
# ========================================
|
||||
# ENVIRONNEMENT
|
||||
# ========================================
|
||||
NODE_ENV=production
|
||||
```
|
||||
|
||||
### 3. Permissions du fichier .env
|
||||
|
||||
```bash
|
||||
# Sécuriser le fichier .env
|
||||
chmod 600 .env
|
||||
chown www-data:www-data .env # ou votre utilisateur système
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Déploiement sur VPS/Serveur
|
||||
|
||||
### 1. Installation de Node.js
|
||||
|
||||
```bash
|
||||
# Installation de Node.js 20 LTS
|
||||
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
|
||||
sudo apt install -y nodejs
|
||||
|
||||
# Vérification
|
||||
node --version # devrait afficher v20.x
|
||||
npm --version
|
||||
```
|
||||
|
||||
### 2. Cloner et installer l'application
|
||||
|
||||
```bash
|
||||
# Créer le répertoire
|
||||
sudo mkdir -p /var/www/videotoMP3Transcriptor
|
||||
sudo chown $USER:$USER /var/www/videotoMP3Transcriptor
|
||||
|
||||
# Cloner (ou copier) votre code
|
||||
cd /var/www/videotoMP3Transcriptor
|
||||
# git clone ... ou upload manuel
|
||||
|
||||
# Installer les dépendances
|
||||
npm ci --only=production
|
||||
|
||||
# Créer le répertoire de sortie
|
||||
mkdir -p output
|
||||
chmod 755 output
|
||||
```
|
||||
|
||||
### 3. Utiliser PM2 pour la gestion des processus
|
||||
|
||||
PM2 est un gestionnaire de processus pour Node.js qui redémarre automatiquement votre app en cas de crash.
|
||||
|
||||
```bash
|
||||
# Installer PM2 globalement
|
||||
sudo npm install -g pm2
|
||||
|
||||
# Démarrer l'application
|
||||
pm2 start src/server.js --name "video-transcriptor"
|
||||
|
||||
# Configurer PM2 pour démarrer au boot
|
||||
pm2 startup systemd
|
||||
pm2 save
|
||||
|
||||
# Commandes utiles
|
||||
pm2 status # Voir le statut
|
||||
pm2 logs video-transcriptor # Voir les logs
|
||||
pm2 restart video-transcriptor # Redémarrer
|
||||
pm2 stop video-transcriptor # Arrêter
|
||||
```
|
||||
|
||||
### 4. Configuration PM2 avancée (optionnelle)
|
||||
|
||||
Créez un fichier `ecosystem.config.js`:
|
||||
|
||||
```javascript
|
||||
module.exports = {
|
||||
apps: [{
|
||||
name: 'video-transcriptor',
|
||||
script: './src/server.js',
|
||||
instances: 1,
|
||||
autorestart: true,
|
||||
watch: false,
|
||||
max_memory_restart: '1G',
|
||||
env: {
|
||||
NODE_ENV: 'production',
|
||||
PORT: 8888
|
||||
},
|
||||
error_file: '/var/log/pm2/video-transcriptor-error.log',
|
||||
out_file: '/var/log/pm2/video-transcriptor-out.log',
|
||||
log_date_format: 'YYYY-MM-DD HH:mm:ss Z'
|
||||
}]
|
||||
};
|
||||
```
|
||||
|
||||
Démarrer avec:
|
||||
```bash
|
||||
pm2 start ecosystem.config.js
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Déploiement avec Docker
|
||||
|
||||
### 1. Créer un Dockerfile
|
||||
|
||||
Créez `Dockerfile` à la racine du projet:
|
||||
|
||||
```dockerfile
|
||||
FROM node:20-slim
|
||||
|
||||
# Installer les dépendances système
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
python3 \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Installer yt-dlp
|
||||
RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp \
|
||||
&& chmod a+rx /usr/local/bin/yt-dlp
|
||||
|
||||
# Créer le répertoire de l'app
|
||||
WORKDIR /app
|
||||
|
||||
# Copier package.json et installer les dépendances
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production
|
||||
|
||||
# Copier le code source
|
||||
COPY . .
|
||||
|
||||
# Créer le répertoire de sortie
|
||||
RUN mkdir -p /app/output && chmod 755 /app/output
|
||||
|
||||
# Exposer le port
|
||||
EXPOSE 8888
|
||||
|
||||
# Variables d'environnement par défaut
|
||||
ENV NODE_ENV=production
|
||||
ENV PORT=8888
|
||||
ENV OUTPUT_DIR=/app/output
|
||||
|
||||
# Démarrer l'application
|
||||
CMD ["node", "src/server.js"]
|
||||
```
|
||||
|
||||
### 2. Créer docker-compose.yml
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
video-transcriptor:
|
||||
build: .
|
||||
container_name: video-transcriptor
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8888:8888"
|
||||
volumes:
|
||||
- ./output:/app/output
|
||||
- ./.env:/app/.env:ro
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
networks:
|
||||
- transcriptor-network
|
||||
|
||||
networks:
|
||||
transcriptor-network:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
### 3. Lancer avec Docker Compose
|
||||
|
||||
```bash
|
||||
# Build et démarrer
|
||||
docker-compose up -d
|
||||
|
||||
# Voir les logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Arrêter
|
||||
docker-compose down
|
||||
|
||||
# Reconstruire après modification
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Nginx Reverse Proxy
|
||||
|
||||
### 1. Installer Nginx
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y nginx
|
||||
```
|
||||
|
||||
### 2. Configuration Nginx
|
||||
|
||||
Créez `/etc/nginx/sites-available/video-transcriptor`:
|
||||
|
||||
```nginx
|
||||
# Rate limiting
|
||||
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name api.yourdomain.com;
|
||||
|
||||
# Logs
|
||||
access_log /var/log/nginx/video-transcriptor-access.log;
|
||||
error_log /var/log/nginx/video-transcriptor-error.log;
|
||||
|
||||
# Rate limiting
|
||||
limit_req zone=api_limit burst=20 nodelay;
|
||||
|
||||
# Augmenter les timeouts pour les longs traitements
|
||||
proxy_connect_timeout 600;
|
||||
proxy_send_timeout 600;
|
||||
proxy_read_timeout 600;
|
||||
send_timeout 600;
|
||||
|
||||
# Augmenter la taille max des uploads
|
||||
client_max_body_size 500M;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:8888;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
# Headers
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection 'upgrade';
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Pour Server-Sent Events (SSE)
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
}
|
||||
|
||||
# Headers de sécurité supplémentaires
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "DENY" always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Activer le site
|
||||
|
||||
```bash
|
||||
# Créer un lien symbolique
|
||||
sudo ln -s /etc/nginx/sites-available/video-transcriptor /etc/nginx/sites-enabled/
|
||||
|
||||
# Tester la configuration
|
||||
sudo nginx -t
|
||||
|
||||
# Recharger Nginx
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SSL/HTTPS avec Let's Encrypt
|
||||
|
||||
### 1. Installer Certbot
|
||||
|
||||
```bash
|
||||
sudo apt install -y certbot python3-certbot-nginx
|
||||
```
|
||||
|
||||
### 2. Obtenir un certificat SSL
|
||||
|
||||
```bash
|
||||
# Obtenir et installer automatiquement le certificat
|
||||
sudo certbot --nginx -d api.yourdomain.com
|
||||
|
||||
# Suivez les instructions à l'écran
|
||||
```
|
||||
|
||||
### 3. Renouvellement automatique
|
||||
|
||||
```bash
|
||||
# Tester le renouvellement
|
||||
sudo certbot renew --dry-run
|
||||
|
||||
# Le renouvellement automatique est configuré via cron
|
||||
# Vérifier: sudo systemctl status certbot.timer
|
||||
```
|
||||
|
||||
Après SSL, votre configuration Nginx sera automatiquement mise à jour pour HTTPS.
|
||||
|
||||
---
|
||||
|
||||
## Surveillance et logs
|
||||
|
||||
### 1. Logs de l'application
|
||||
|
||||
```bash
|
||||
# Avec PM2
|
||||
pm2 logs video-transcriptor
|
||||
|
||||
# Avec Docker
|
||||
docker-compose logs -f video-transcriptor
|
||||
|
||||
# Logs Nginx
|
||||
sudo tail -f /var/log/nginx/video-transcriptor-access.log
|
||||
sudo tail -f /var/log/nginx/video-transcriptor-error.log
|
||||
```
|
||||
|
||||
### 2. Monitoring avec PM2 (optionnel)
|
||||
|
||||
```bash
|
||||
# Installer PM2 monitoring
|
||||
pm2 install pm2-logrotate
|
||||
|
||||
# Configurer la rotation des logs
|
||||
pm2 set pm2-logrotate:max_size 10M
|
||||
pm2 set pm2-logrotate:retain 7
|
||||
```
|
||||
|
||||
### 3. Monitoring système
|
||||
|
||||
```bash
|
||||
# Installer htop pour surveiller les ressources
|
||||
sudo apt install -y htop
|
||||
|
||||
# Lancer htop
|
||||
htop
|
||||
|
||||
# Voir l'utilisation disque
|
||||
df -h
|
||||
|
||||
# Voir l'utilisation mémoire
|
||||
free -h
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sécurité avancée
|
||||
|
||||
### 1. Firewall (UFW)
|
||||
|
||||
```bash
|
||||
# Installer UFW
|
||||
sudo apt install -y ufw
|
||||
|
||||
# Autoriser SSH (IMPORTANT AVANT D'ACTIVER!)
|
||||
sudo ufw allow ssh
|
||||
sudo ufw allow 22/tcp
|
||||
|
||||
# Autoriser HTTP et HTTPS
|
||||
sudo ufw allow 'Nginx Full'
|
||||
|
||||
# Activer le firewall
|
||||
sudo ufw enable
|
||||
|
||||
# Vérifier le statut
|
||||
sudo ufw status
|
||||
```
|
||||
|
||||
### 2. Fail2Ban (protection contre brute force)
|
||||
|
||||
```bash
|
||||
# Installer Fail2Ban
|
||||
sudo apt install -y fail2ban
|
||||
|
||||
# Créer une configuration pour Nginx
|
||||
sudo nano /etc/fail2ban/jail.local
|
||||
```
|
||||
|
||||
Ajouter:
|
||||
```ini
|
||||
[nginx-limit-req]
|
||||
enabled = true
|
||||
filter = nginx-limit-req
|
||||
port = http,https
|
||||
logpath = /var/log/nginx/video-transcriptor-error.log
|
||||
maxretry = 5
|
||||
findtime = 600
|
||||
bantime = 3600
|
||||
```
|
||||
|
||||
```bash
|
||||
# Redémarrer Fail2Ban
|
||||
sudo systemctl restart fail2ban
|
||||
|
||||
# Vérifier le statut
|
||||
sudo fail2ban-client status nginx-limit-req
|
||||
```
|
||||
|
||||
### 3. Limitations supplémentaires
|
||||
|
||||
**Limiter les tailles de fichiers uploadés** - Déjà configuré dans Nginx (`client_max_body_size 500M`)
|
||||
|
||||
**Rate limiting par IP** - Déjà configuré dans Nginx (`limit_req_zone`)
|
||||
|
||||
### 4. Sauvegardes automatiques
|
||||
|
||||
```bash
|
||||
# Créer un script de backup
|
||||
sudo nano /usr/local/bin/backup-video-transcriptor.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/backup/video-transcriptor"
|
||||
APP_DIR="/var/www/videotoMP3Transcriptor"
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
mkdir -p $BACKUP_DIR
|
||||
|
||||
# Backup de la configuration
|
||||
tar -czf $BACKUP_DIR/config_$DATE.tar.gz \
|
||||
$APP_DIR/.env \
|
||||
$APP_DIR/ecosystem.config.js
|
||||
|
||||
# Backup des fichiers de sortie (optionnel, peut être volumineux)
|
||||
# tar -czf $BACKUP_DIR/output_$DATE.tar.gz $APP_DIR/output
|
||||
|
||||
# Garder seulement les 7 derniers backups
|
||||
find $BACKUP_DIR -name "config_*.tar.gz" -mtime +7 -delete
|
||||
|
||||
echo "Backup completed: $DATE"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Rendre exécutable
|
||||
sudo chmod +x /usr/local/bin/backup-video-transcriptor.sh
|
||||
|
||||
# Ajouter au crontab (backup quotidien à 2h du matin)
|
||||
sudo crontab -e
|
||||
# Ajouter: 0 2 * * * /usr/local/bin/backup-video-transcriptor.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist finale de déploiement
|
||||
|
||||
Avant de mettre en production, vérifiez:
|
||||
|
||||
- [ ] **Sécurité**
|
||||
- [ ] Token API fort généré (`API_TOKEN`)
|
||||
- [ ] CORS configuré avec vos domaines (`ALLOWED_ORIGINS`)
|
||||
- [ ] Fichier `.env` avec permissions 600
|
||||
- [ ] HTTPS configuré et fonctionnel
|
||||
- [ ] Firewall UFW activé
|
||||
|
||||
- [ ] **Configuration**
|
||||
- [ ] `OPENAI_API_KEY` valide et fonctionnelle
|
||||
- [ ] `NODE_ENV=production`
|
||||
- [ ] Répertoire `output/` créé et accessible
|
||||
- [ ] FFmpeg et yt-dlp installés
|
||||
|
||||
- [ ] **Infrastructure**
|
||||
- [ ] PM2 ou Docker en cours d'exécution
|
||||
- [ ] Nginx reverse proxy configuré
|
||||
- [ ] SSL/TLS actif (Let's Encrypt)
|
||||
- [ ] Rate limiting activé
|
||||
|
||||
- [ ] **Monitoring**
|
||||
- [ ] Logs accessibles
|
||||
- [ ] PM2 startup configuré (redémarrage auto)
|
||||
- [ ] Fail2Ban actif
|
||||
- [ ] Backups automatiques configurés
|
||||
|
||||
- [ ] **Tests**
|
||||
- [ ] Endpoint `/health` accessible
|
||||
- [ ] Test d'authentification (avec et sans token)
|
||||
- [ ] Test d'upload de fichier
|
||||
- [ ] Test de téléchargement YouTube
|
||||
|
||||
---
|
||||
|
||||
## Tests post-déploiement
|
||||
|
||||
### 1. Test de santé
|
||||
|
||||
```bash
|
||||
curl https://api.yourdomain.com/health
|
||||
# Devrait retourner: {"status":"ok","timestamp":"..."}
|
||||
```
|
||||
|
||||
### 2. Test d'authentification
|
||||
|
||||
```bash
|
||||
# Sans token (devrait échouer avec 401)
|
||||
curl https://api.yourdomain.com/info?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ
|
||||
|
||||
# Avec token (devrait réussir)
|
||||
curl -H "X-API-Key: VOTRE_TOKEN" \
|
||||
"https://api.yourdomain.com/info?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ"
|
||||
```
|
||||
|
||||
### 3. Test de download
|
||||
|
||||
```bash
|
||||
curl -H "X-API-Key: VOTRE_TOKEN" \
|
||||
-X POST https://api.yourdomain.com/download \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dépannage
|
||||
|
||||
### L'API ne démarre pas
|
||||
|
||||
```bash
|
||||
# Vérifier les logs PM2
|
||||
pm2 logs video-transcriptor
|
||||
|
||||
# Vérifier les variables d'environnement
|
||||
pm2 env video-transcriptor
|
||||
|
||||
# Redémarrer
|
||||
pm2 restart video-transcriptor
|
||||
```
|
||||
|
||||
### Erreurs 502 Bad Gateway (Nginx)
|
||||
|
||||
```bash
|
||||
# Vérifier que l'app tourne
|
||||
pm2 status
|
||||
|
||||
# Vérifier les logs Nginx
|
||||
sudo tail -f /var/log/nginx/error.log
|
||||
|
||||
# Vérifier que le port 8888 est ouvert
|
||||
sudo netstat -tlnp | grep 8888
|
||||
```
|
||||
|
||||
### Problèmes SSL
|
||||
|
||||
```bash
|
||||
# Vérifier le certificat
|
||||
sudo certbot certificates
|
||||
|
||||
# Renouveler manuellement
|
||||
sudo certbot renew --force-renewal
|
||||
|
||||
# Tester la configuration Nginx
|
||||
sudo nginx -t
|
||||
```
|
||||
|
||||
### Mémoire insuffisante
|
||||
|
||||
```bash
|
||||
# Vérifier l'utilisation mémoire
|
||||
free -h
|
||||
|
||||
# Créer un swap file (si nécessaire)
|
||||
sudo fallocate -l 2G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mises à jour
|
||||
|
||||
### Mise à jour de l'application
|
||||
|
||||
```bash
|
||||
cd /var/www/videotoMP3Transcriptor
|
||||
|
||||
# Sauvegarder la config
|
||||
cp .env .env.backup
|
||||
|
||||
# Pull des nouvelles versions (git)
|
||||
git pull
|
||||
|
||||
# Mettre à jour les dépendances
|
||||
npm ci --only=production
|
||||
|
||||
# Redémarrer
|
||||
pm2 restart video-transcriptor
|
||||
|
||||
# Ou avec Docker
|
||||
docker-compose down
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support et ressources
|
||||
|
||||
- **Documentation API**: [docs/API.md](./API.md)
|
||||
- **CLAUDE.md**: [CLAUDE.md](../CLAUDE.md) - Instructions pour Claude
|
||||
- **PM2 Documentation**: https://pm2.keymetrics.io/
|
||||
- **Nginx Documentation**: https://nginx.org/en/docs/
|
||||
- **Let's Encrypt**: https://letsencrypt.org/
|
||||
|
||||
---
|
||||
|
||||
**Bon déploiement ! 🚀**
|
||||
@ -1,132 +0,0 @@
|
||||
# YouTube Cookies Setup Guide
|
||||
|
||||
## Why Do I Need Cookies?
|
||||
|
||||
YouTube has anti-bot protections that may block yt-dlp requests. Using cookies from your browser allows yt-dlp to authenticate as if you're logged in, bypassing these restrictions.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Option 1: Automatic Extraction (Recommended)
|
||||
|
||||
Run the helper script:
|
||||
|
||||
```bash
|
||||
bash scripts/extract-cookies.sh
|
||||
```
|
||||
|
||||
Follow the prompts to extract cookies from Chrome or Firefox.
|
||||
|
||||
### Option 2: Using yt-dlp Directly
|
||||
|
||||
```bash
|
||||
# For Chrome/Chromium
|
||||
yt-dlp --cookies-from-browser chrome --cookies youtube-cookies.txt 'https://www.youtube.com'
|
||||
|
||||
# For Firefox
|
||||
yt-dlp --cookies-from-browser firefox --cookies youtube-cookies.txt 'https://www.youtube.com'
|
||||
|
||||
# For Edge
|
||||
yt-dlp --cookies-from-browser edge --cookies youtube-cookies.txt 'https://www.youtube.com'
|
||||
```
|
||||
|
||||
### Option 3: Browser Extension
|
||||
|
||||
1. Install a cookies export extension:
|
||||
- **Chrome/Edge**: [Get cookies.txt LOCALLY](https://chrome.google.com/webstore/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)
|
||||
- **Firefox**: [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/)
|
||||
|
||||
2. Go to [youtube.com](https://www.youtube.com) and log in
|
||||
|
||||
3. Click the extension icon and export cookies
|
||||
|
||||
4. Save the file as `youtube-cookies.txt` in your project directory
|
||||
|
||||
## Configuration
|
||||
|
||||
After extracting cookies, update your `.env` file:
|
||||
|
||||
```bash
|
||||
YOUTUBE_COOKIES_PATH=/home/debian/videotomp3transcriptor/youtube-cookies.txt
|
||||
```
|
||||
|
||||
Or use a relative path:
|
||||
|
||||
```bash
|
||||
YOUTUBE_COOKIES_PATH=./youtube-cookies.txt
|
||||
```
|
||||
|
||||
## Verifying It Works
|
||||
|
||||
Test with a video:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3001/download \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'
|
||||
```
|
||||
|
||||
If it works without cookies errors, you're good to go!
|
||||
|
||||
## Security Notes
|
||||
|
||||
⚠️ **IMPORTANT**:
|
||||
|
||||
1. **Never commit cookies to git**: The `.gitignore` file should already exclude `youtube-cookies.txt`
|
||||
2. **Keep cookies secure**: They provide access to your YouTube account
|
||||
3. **Cookies expire**: You may need to re-export them periodically (typically every few weeks/months)
|
||||
4. **Don't share cookies**: Treat them like passwords
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Sign in to confirm you're not a bot"
|
||||
|
||||
This usually means:
|
||||
- Cookies are not being used
|
||||
- Cookies have expired
|
||||
- Cookies file path is incorrect
|
||||
|
||||
**Solutions**:
|
||||
1. Check the path in `.env` is correct and absolute
|
||||
2. Re-export fresh cookies
|
||||
3. Verify the cookies file exists: `ls -la youtube-cookies.txt`
|
||||
4. Check logs: `pm2 logs toMP3-api`
|
||||
|
||||
### "HTTP Error 403: Forbidden"
|
||||
|
||||
YouTube is blocking your IP or the video is region-restricted.
|
||||
|
||||
**Solutions**:
|
||||
1. Try with fresh cookies
|
||||
2. Use a VPN if region-restricted
|
||||
3. Wait a bit if rate-limited
|
||||
|
||||
### Cookies Not Working
|
||||
|
||||
1. Make sure you're logged into YouTube in the browser before extracting
|
||||
2. Try extracting from a different browser
|
||||
3. Verify the cookies file format (should be Netscape format)
|
||||
4. Check file permissions: `chmod 600 youtube-cookies.txt`
|
||||
|
||||
## Cookie File Format
|
||||
|
||||
The cookies file should be in Netscape format and look like this:
|
||||
|
||||
```
|
||||
# Netscape HTTP Cookie File
|
||||
.youtube.com TRUE / TRUE 1234567890 CONSENT YES+
|
||||
.youtube.com TRUE / FALSE 1234567890 VISITOR_INFO1_LIVE xxxxx
|
||||
```
|
||||
|
||||
## Without Cookies
|
||||
|
||||
The API will still work for many videos without cookies, but you may encounter:
|
||||
- "Sign in to confirm you're not a bot" errors
|
||||
- Rate limiting
|
||||
- Blocked downloads for certain videos
|
||||
|
||||
For best results, always use cookies!
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [yt-dlp Cookie Documentation](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp)
|
||||
- [Browser Cookie Extraction](https://github.com/yt-dlp/yt-dlp#:~:text=You%20can%20use%20cookies%20from%20your%20browser)
|
||||
37
package.json
37
package.json
@ -1,34 +1,27 @@
|
||||
{
|
||||
"name": "video-to-mp3-transcriptor",
|
||||
"version": "1.0.0",
|
||||
"description": "Download YouTube videos/playlists to MP3 and transcribe them using OpenAI Whisper API",
|
||||
"main": "src/index.js",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
"ytmp3": "./src/cli.js"
|
||||
},
|
||||
"name": "hanasuba-music-service",
|
||||
"version": "2.0.0",
|
||||
"description": "YouTube to MP3 download service with Camoufox stealth cookies for Hanasuba",
|
||||
"main": "src/server.js",
|
||||
"scripts": {
|
||||
"start": "node src/index.js",
|
||||
"cli": "node src/cli.js",
|
||||
"server": "node src/server.js"
|
||||
"start": "node src/server.js",
|
||||
"dev": "node --watch src/server.js",
|
||||
"setup": "python3 -m pip install -r requirements.txt && playwright install firefox",
|
||||
"cookies:extract": "python3 src/python/extract_cookies.py",
|
||||
"cookies:validate": "python3 src/python/validate_cookies.py"
|
||||
},
|
||||
"keywords": [
|
||||
"youtube",
|
||||
"mp3",
|
||||
"transcription",
|
||||
"whisper",
|
||||
"openai"
|
||||
"music",
|
||||
"camoufox",
|
||||
"stealth",
|
||||
"hanasuba"
|
||||
],
|
||||
"author": "",
|
||||
"author": "StillHammer",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.70.1",
|
||||
"commander": "^12.1.0",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.4.5",
|
||||
"express": "^4.21.0",
|
||||
"multer": "^2.0.2",
|
||||
"openai": "^4.67.0",
|
||||
"youtube-dl-exec": "^3.0.7"
|
||||
"dotenv": "^16.4.5"
|
||||
}
|
||||
}
|
||||
|
||||
3
requirements.txt
Normal file
3
requirements.txt
Normal file
@ -0,0 +1,3 @@
|
||||
camoufox>=0.4.11
|
||||
camoufox-captcha>=0.1.3
|
||||
playwright>=1.57.0
|
||||
169
src/cli.js
169
src/cli.js
@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { Command } from 'commander';
|
||||
import dotenv from 'dotenv';
|
||||
import path from 'path';
|
||||
import { download, downloadVideo, downloadPlaylist, getInfo } from './services/youtube.js';
|
||||
import { transcribeFile, transcribeAndSave, transcribeMultiple, getAvailableModels } from './services/transcription.js';
|
||||
|
||||
// Load environment variables
|
||||
dotenv.config();
|
||||
|
||||
const program = new Command();
|
||||
|
||||
program
|
||||
.name('ytmp3')
|
||||
.description('Download YouTube videos/playlists to MP3 and transcribe them')
|
||||
.version('1.0.0');
|
||||
|
||||
// Download command
|
||||
program
|
||||
.command('download <url>')
|
||||
.alias('dl')
|
||||
.description('Download a YouTube video or playlist as MP3')
|
||||
.option('-o, --output <dir>', 'Output directory', './output')
|
||||
.action(async (url, options) => {
|
||||
try {
|
||||
console.log('Fetching video info...');
|
||||
const result = await download(url, { outputDir: options.output });
|
||||
|
||||
console.log('\n--- Download Complete ---');
|
||||
if (result.playlistTitle) {
|
||||
console.log(`Playlist: ${result.playlistTitle}`);
|
||||
}
|
||||
console.log(`Downloaded: ${result.successCount}/${result.totalVideos} videos`);
|
||||
|
||||
result.videos.forEach(v => {
|
||||
if (v.success) {
|
||||
console.log(` ✓ ${v.title}`);
|
||||
} else {
|
||||
console.log(` ✗ ${v.title} - ${v.error}`);
|
||||
}
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(`Error: ${error.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
// Transcribe command (from existing MP3)
|
||||
program
|
||||
.command('transcribe <file>')
|
||||
.alias('tr')
|
||||
.description('Transcribe an existing audio file')
|
||||
.option('-l, --language <lang>', 'Language code (e.g., en, fr, zh)')
|
||||
.option('-f, --format <format>', 'Output format (txt, srt, vtt)', 'txt')
|
||||
.option('-m, --model <model>', 'Transcription model (gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1)', 'gpt-4o-transcribe')
|
||||
.action(async (file, options) => {
|
||||
try {
|
||||
if (!process.env.OPENAI_API_KEY) {
|
||||
console.error('Error: OPENAI_API_KEY not set in environment');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`Transcribing: ${file}`);
|
||||
const result = await transcribeAndSave(file, {
|
||||
language: options.language,
|
||||
responseFormat: options.format === 'txt' ? 'text' : options.format,
|
||||
outputFormat: options.format,
|
||||
model: options.model,
|
||||
});
|
||||
|
||||
console.log('\n--- Transcription Complete ---');
|
||||
console.log(`Model: ${result.model}`);
|
||||
console.log(`Output: ${result.transcriptionPath}`);
|
||||
console.log('\nPreview:');
|
||||
console.log(result.text.substring(0, 500) + (result.text.length > 500 ? '...' : ''));
|
||||
} catch (error) {
|
||||
console.error(`Error: ${error.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
// Download + Transcribe command
|
||||
program
|
||||
.command('process <url>')
|
||||
.alias('p')
|
||||
.description('Download and transcribe a YouTube video or playlist')
|
||||
.option('-o, --output <dir>', 'Output directory', './output')
|
||||
.option('-l, --language <lang>', 'Language code for transcription')
|
||||
.option('-f, --format <format>', 'Transcription format (txt, srt, vtt)', 'txt')
|
||||
.option('-m, --model <model>', 'Transcription model (gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1)', 'gpt-4o-transcribe')
|
||||
.action(async (url, options) => {
|
||||
try {
|
||||
if (!process.env.OPENAI_API_KEY) {
|
||||
console.error('Error: OPENAI_API_KEY not set in environment');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Step 1: Download
|
||||
console.log('Step 1: Downloading...');
|
||||
const downloadResult = await download(url, { outputDir: options.output });
|
||||
|
||||
console.log(`Downloaded: ${downloadResult.successCount}/${downloadResult.totalVideos} videos\n`);
|
||||
|
||||
// Step 2: Transcribe
|
||||
console.log(`Step 2: Transcribing with ${options.model}...`);
|
||||
const successfulDownloads = downloadResult.videos.filter(v => v.success);
|
||||
const filePaths = successfulDownloads.map(v => v.filePath);
|
||||
|
||||
const transcribeResult = await transcribeMultiple(filePaths, {
|
||||
language: options.language,
|
||||
responseFormat: options.format === 'txt' ? 'text' : options.format,
|
||||
outputFormat: options.format,
|
||||
model: options.model,
|
||||
});
|
||||
|
||||
console.log('\n--- Process Complete ---');
|
||||
if (downloadResult.playlistTitle) {
|
||||
console.log(`Playlist: ${downloadResult.playlistTitle}`);
|
||||
}
|
||||
console.log(`Downloaded: ${downloadResult.successCount}/${downloadResult.totalVideos}`);
|
||||
console.log(`Transcribed: ${transcribeResult.successCount}/${transcribeResult.totalFiles}`);
|
||||
|
||||
transcribeResult.results.forEach(r => {
|
||||
if (r.success) {
|
||||
console.log(` ✓ ${path.basename(r.transcriptionPath)}`);
|
||||
} else {
|
||||
console.log(` ✗ ${path.basename(r.filePath)} - ${r.error}`);
|
||||
}
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(`Error: ${error.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
// Info command
|
||||
program
|
||||
.command('info <url>')
|
||||
.description('Get info about a YouTube video or playlist')
|
||||
.action(async (url) => {
|
||||
try {
|
||||
const info = await getInfo(url);
|
||||
|
||||
console.log('\n--- Video/Playlist Info ---');
|
||||
console.log(`Title: ${info.title}`);
|
||||
console.log(`Type: ${info._type || 'video'}`);
|
||||
|
||||
if (info._type === 'playlist') {
|
||||
console.log(`Videos: ${info.entries?.length || 0}`);
|
||||
if (info.entries) {
|
||||
info.entries.slice(0, 10).forEach((e, i) => {
|
||||
console.log(` ${i + 1}. ${e.title}`);
|
||||
});
|
||||
if (info.entries.length > 10) {
|
||||
console.log(` ... and ${info.entries.length - 10} more`);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.log(`Duration: ${Math.floor(info.duration / 60)}:${String(info.duration % 60).padStart(2, '0')}`);
|
||||
console.log(`Channel: ${info.channel}`);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`Error: ${error.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
program.parse();
|
||||
73
src/python/extract_cookies.py
Executable file
73
src/python/extract_cookies.py
Executable file
@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Extract YouTube cookies using Camoufox (stealth Firefox).
|
||||
Cookies are undetectable by bot detection systems.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from camoufox.async_api import AsyncCamoufox
|
||||
|
||||
async def extract_cookies(output_path='youtube-cookies.txt'):
|
||||
"""
|
||||
Extract YouTube cookies using Camoufox (stealth Firefox).
|
||||
These cookies bypass bot detection and last longer.
|
||||
"""
|
||||
|
||||
print("🦊 Starting Camoufox (stealth mode)...")
|
||||
|
||||
try:
|
||||
async with AsyncCamoufox(
|
||||
headless=True, # Background (no GUI needed)
|
||||
humanize=True, # Mimic human behavior
|
||||
geoip=True, # Realistic IP geolocation
|
||||
) as browser:
|
||||
page = await browser.new_page()
|
||||
|
||||
# Navigate to YouTube
|
||||
print("📺 Loading YouTube...")
|
||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=30000)
|
||||
|
||||
# Wait for page fully loaded
|
||||
await asyncio.sleep(3)
|
||||
|
||||
# Extract cookies
|
||||
cookies = await page.context.cookies()
|
||||
|
||||
# Filter YouTube cookies
|
||||
yt_cookies = [c for c in cookies if 'youtube.com' in c['domain']]
|
||||
|
||||
if not yt_cookies:
|
||||
print("❌ No YouTube cookies found!")
|
||||
return False
|
||||
|
||||
# Save to Netscape format (yt-dlp compatible)
|
||||
output = Path(output_path)
|
||||
with open(output, 'w') as f:
|
||||
f.write("# Netscape HTTP Cookie File\n")
|
||||
f.write("# Generated by Camoufox (stealth mode)\n")
|
||||
f.write(f"# This file is compatible with yt-dlp\n")
|
||||
for c in yt_cookies:
|
||||
line = f"{c['domain']}\tTRUE\t{c['path']}\t"
|
||||
line += f"{'TRUE' if c.get('secure') else 'FALSE'}\t"
|
||||
line += f"{int(c.get('expires', 0))}\t{c['name']}\t{c['value']}\n"
|
||||
f.write(line)
|
||||
|
||||
# Set secure permissions
|
||||
output.chmod(0o600)
|
||||
|
||||
print(f"✅ Cookies saved: {output_path}")
|
||||
print(f" Total cookies: {len(yt_cookies)}")
|
||||
print(f" Permissions: 600 (secure)")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {e}")
|
||||
return False
|
||||
|
||||
if __name__ == '__main__':
|
||||
output = sys.argv[1] if len(sys.argv) > 1 else 'youtube-cookies.txt'
|
||||
success = asyncio.run(extract_cookies(output))
|
||||
sys.exit(0 if success else 1)
|
||||
85
src/python/validate_cookies.py
Executable file
85
src/python/validate_cookies.py
Executable file
@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Validate YouTube cookies using Camoufox.
|
||||
Tests if cookies are still valid and working.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from camoufox.async_api import AsyncCamoufox
|
||||
|
||||
async def validate_cookies(cookies_path='youtube-cookies.txt'):
|
||||
"""
|
||||
Test if YouTube cookies are still valid.
|
||||
Returns True if valid, False otherwise.
|
||||
"""
|
||||
|
||||
cookies_file = Path(cookies_path)
|
||||
|
||||
if not cookies_file.exists():
|
||||
print(f"❌ Cookies file not found: {cookies_path}")
|
||||
return False
|
||||
|
||||
# Check file age
|
||||
age_hours = (Path().stat().st_mtime - cookies_file.stat().st_mtime) / 3600
|
||||
print(f"📅 Cookies age: {age_hours:.1f} hours")
|
||||
|
||||
try:
|
||||
async with AsyncCamoufox(headless=True, humanize=True) as browser:
|
||||
context = await browser.new_context()
|
||||
|
||||
# Load cookies from file
|
||||
# Camoufox doesn't have add_cookies_from_file, so we parse manually
|
||||
cookies_to_add = []
|
||||
with open(cookies_path, 'r') as f:
|
||||
for line in f:
|
||||
if line.startswith('#') or not line.strip():
|
||||
continue
|
||||
parts = line.strip().split('\t')
|
||||
if len(parts) >= 7:
|
||||
cookies_to_add.append({
|
||||
'domain': parts[0],
|
||||
'path': parts[2],
|
||||
'secure': parts[3] == 'TRUE',
|
||||
'expires': int(parts[4]) if parts[4] != '0' else None,
|
||||
'name': parts[5],
|
||||
'value': parts[6]
|
||||
})
|
||||
|
||||
if not cookies_to_add:
|
||||
print("❌ No valid cookies found in file")
|
||||
return False
|
||||
|
||||
await context.add_cookies(cookies_to_add)
|
||||
|
||||
page = await context.new_page()
|
||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=30000)
|
||||
|
||||
# Wait a bit for page to render
|
||||
await asyncio.sleep(2)
|
||||
|
||||
# Check if we can access YouTube properly
|
||||
# If blocked, there will be bot detection or sign-in prompts
|
||||
content = await page.content()
|
||||
|
||||
# Simple validation: check if we have access to normal YouTube
|
||||
is_valid = 'ytInitialData' in content or 'watch?' in content
|
||||
|
||||
if is_valid:
|
||||
print("✅ Cookies are valid")
|
||||
print(" YouTube access: OK")
|
||||
return True
|
||||
else:
|
||||
print("⚠️ Cookies may be expired or invalid")
|
||||
print(" YouTube access: BLOCKED")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Validation error: {e}")
|
||||
return False
|
||||
|
||||
if __name__ == '__main__':
|
||||
cookies_path = sys.argv[1] if len(sys.argv) > 1 else 'youtube-cookies.txt'
|
||||
valid = asyncio.run(validate_cookies(cookies_path))
|
||||
sys.exit(0 if valid else 1)
|
||||
1698
src/server.js
1698
src/server.js
File diff suppressed because it is too large
Load Diff
@ -1,145 +0,0 @@
|
||||
import { exec } from 'child_process';
|
||||
import { promisify } from 'util';
|
||||
import path from 'path';
|
||||
import fs from 'fs';
|
||||
|
||||
const execPromise = promisify(exec);
|
||||
|
||||
/**
|
||||
* Convert a video/audio file to MP3 using FFmpeg
|
||||
* @param {string} inputPath - Path to input file
|
||||
* @param {object} options - Conversion options
|
||||
* @param {string} options.outputDir - Output directory (default: same as input)
|
||||
* @param {string} options.bitrate - Audio bitrate (default: 192k)
|
||||
* @param {string} options.quality - Audio quality 0-9 (default: 2, where 0 is best)
|
||||
* @returns {Promise<object>} Conversion result with output path
|
||||
*/
|
||||
export async function convertToMP3(inputPath, options = {}) {
|
||||
const {
|
||||
outputDir = path.dirname(inputPath),
|
||||
bitrate = '192k',
|
||||
quality = '2',
|
||||
} = options;
|
||||
|
||||
// Ensure input file exists
|
||||
if (!fs.existsSync(inputPath)) {
|
||||
throw new Error(`Input file not found: ${inputPath}`);
|
||||
}
|
||||
|
||||
// Generate output path
|
||||
const inputFilename = path.basename(inputPath, path.extname(inputPath));
|
||||
const outputPath = path.join(outputDir, `${inputFilename}.mp3`);
|
||||
|
||||
// Check if output already exists
|
||||
if (fs.existsSync(outputPath)) {
|
||||
// Add timestamp to make it unique
|
||||
const timestamp = Date.now();
|
||||
const uniqueOutputPath = path.join(outputDir, `${inputFilename}_${timestamp}.mp3`);
|
||||
return convertToMP3Internal(inputPath, uniqueOutputPath, bitrate, quality);
|
||||
}
|
||||
|
||||
return convertToMP3Internal(inputPath, outputPath, bitrate, quality);
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal conversion function
|
||||
*/
|
||||
async function convertToMP3Internal(inputPath, outputPath, bitrate, quality) {
|
||||
try {
|
||||
// FFmpeg command to convert to MP3
|
||||
// -i: input file
|
||||
// -vn: no video (audio only)
|
||||
// -ar 44100: audio sample rate 44.1kHz
|
||||
// -ac 2: stereo
|
||||
// -b:a: audio bitrate
|
||||
// -q:a: audio quality (VBR)
|
||||
const command = `ffmpeg -i "${inputPath}" -vn -ar 44100 -ac 2 -b:a ${bitrate} -q:a ${quality} "${outputPath}"`;
|
||||
|
||||
console.log(`Converting: ${path.basename(inputPath)} -> ${path.basename(outputPath)}`);
|
||||
|
||||
const { stdout, stderr } = await execPromise(command);
|
||||
|
||||
// Verify output file was created
|
||||
if (!fs.existsSync(outputPath)) {
|
||||
throw new Error('Conversion failed: output file not created');
|
||||
}
|
||||
|
||||
const stats = fs.statSync(outputPath);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
inputPath,
|
||||
outputPath,
|
||||
filename: path.basename(outputPath),
|
||||
size: stats.size,
|
||||
sizeHuman: formatBytes(stats.size),
|
||||
};
|
||||
} catch (error) {
|
||||
console.error(`Conversion error: ${error.message}`);
|
||||
throw new Error(`FFmpeg conversion failed: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert multiple files to MP3
|
||||
* @param {string[]} inputPaths - Array of input file paths
|
||||
* @param {object} options - Conversion options
|
||||
* @returns {Promise<object>} Batch conversion results
|
||||
*/
|
||||
export async function convertMultipleToMP3(inputPaths, options = {}) {
|
||||
const results = [];
|
||||
let successCount = 0;
|
||||
let failCount = 0;
|
||||
|
||||
for (let i = 0; i < inputPaths.length; i++) {
|
||||
const inputPath = inputPaths[i];
|
||||
console.log(`[${i + 1}/${inputPaths.length}] Converting: ${path.basename(inputPath)}`);
|
||||
|
||||
try {
|
||||
const result = await convertToMP3(inputPath, options);
|
||||
results.push({ ...result, index: i });
|
||||
successCount++;
|
||||
} catch (error) {
|
||||
results.push({
|
||||
success: false,
|
||||
inputPath,
|
||||
error: error.message,
|
||||
index: i,
|
||||
});
|
||||
failCount++;
|
||||
console.error(`Failed to convert ${path.basename(inputPath)}: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
totalFiles: inputPaths.length,
|
||||
successCount,
|
||||
failCount,
|
||||
results,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Format bytes to human readable format
|
||||
*/
|
||||
function formatBytes(bytes, decimals = 2) {
|
||||
if (bytes === 0) return '0 Bytes';
|
||||
|
||||
const k = 1024;
|
||||
const dm = decimals < 0 ? 0 : decimals;
|
||||
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
|
||||
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' + sizes[i];
|
||||
}
|
||||
|
||||
/**
|
||||
* Get supported input formats
|
||||
*/
|
||||
export function getSupportedFormats() {
|
||||
return {
|
||||
video: ['.mp4', '.avi', '.mkv', '.mov', '.wmv', '.flv', '.webm', '.m4v'],
|
||||
audio: ['.m4a', '.wav', '.flac', '.ogg', '.aac', '.wma', '.opus'],
|
||||
};
|
||||
}
|
||||
188
src/services/cookiesManager.js
Normal file
188
src/services/cookiesManager.js
Normal file
@ -0,0 +1,188 @@
|
||||
const { exec } = require('child_process');
|
||||
const { promisify } = require('util');
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
|
||||
const execAsync = promisify(exec);
|
||||
|
||||
/**
|
||||
* Manages YouTube cookies lifecycle using Camoufox stealth extraction.
|
||||
* Auto-refresh when expired, validates periodically.
|
||||
*/
|
||||
class CookiesManager {
|
||||
constructor() {
|
||||
this.cookiesPath = path.join(__dirname, '../../youtube-cookies.txt');
|
||||
this.pythonPath = process.env.PYTHON_PATH || 'python3';
|
||||
this.extractScript = path.join(__dirname, '../python/extract_cookies.py');
|
||||
this.validateScript = path.join(__dirname, '../python/validate_cookies.py');
|
||||
|
||||
this.lastRefresh = null;
|
||||
this.isValid = false;
|
||||
|
||||
// Refresh cookies every 14 days (YouTube cookies typically last 2-4 weeks)
|
||||
this.refreshIntervalDays = 14;
|
||||
|
||||
// Check interval (every 12 hours)
|
||||
this.checkIntervalMs = 12 * 60 * 60 * 1000;
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize cookies manager.
|
||||
* Check if cookies exist, validate them, refresh if needed.
|
||||
*/
|
||||
async init() {
|
||||
console.log('🔧 Initializing cookies manager...');
|
||||
|
||||
// Check if cookies file exists
|
||||
try {
|
||||
await fs.access(this.cookiesPath);
|
||||
console.log('✅ Cookies file exists');
|
||||
|
||||
// Validate cookies
|
||||
const valid = await this.validate();
|
||||
if (!valid) {
|
||||
console.log('⚠️ Cookies invalid, refreshing...');
|
||||
await this.refresh();
|
||||
} else {
|
||||
console.log('✅ Cookies valid');
|
||||
}
|
||||
} catch (err) {
|
||||
console.log('📝 No cookies found, generating fresh cookies...');
|
||||
await this.refresh();
|
||||
}
|
||||
|
||||
// Setup periodic validation (every 12 hours)
|
||||
setInterval(() => {
|
||||
this.checkAndRefresh().catch(err => {
|
||||
console.error('Auto-check failed:', err.message);
|
||||
});
|
||||
}, this.checkIntervalMs);
|
||||
|
||||
console.log('✅ Cookies manager ready');
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate cookies using Python script.
|
||||
* @returns {Promise<boolean>} True if cookies are valid
|
||||
*/
|
||||
async validate() {
|
||||
try {
|
||||
const { stdout, stderr } = await execAsync(
|
||||
`${this.pythonPath} ${this.validateScript} ${this.cookiesPath}`,
|
||||
{ timeout: 60000 }
|
||||
);
|
||||
|
||||
// Check for validation success in output
|
||||
this.isValid = stdout.includes('Cookies are valid');
|
||||
|
||||
if (stderr && !stderr.includes('DeprecationWarning')) {
|
||||
console.warn('Validation stderr:', stderr.trim());
|
||||
}
|
||||
|
||||
return this.isValid;
|
||||
} catch (err) {
|
||||
console.error('Validation failed:', err.message);
|
||||
this.isValid = false;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Refresh cookies using Camoufox extraction.
|
||||
* @returns {Promise<boolean>} True if refresh succeeded
|
||||
*/
|
||||
async refresh() {
|
||||
console.log('🔄 Refreshing YouTube cookies with Camoufox...');
|
||||
|
||||
try {
|
||||
const { stdout, stderr } = await execAsync(
|
||||
`${this.pythonPath} ${this.extractScript} ${this.cookiesPath}`,
|
||||
{ timeout: 120000 } // 2 min timeout
|
||||
);
|
||||
|
||||
console.log(stdout.trim());
|
||||
|
||||
if (stderr && !stderr.includes('DeprecationWarning')) {
|
||||
console.warn('Camoufox stderr:', stderr.trim());
|
||||
}
|
||||
|
||||
// Verify file was created
|
||||
try {
|
||||
await fs.access(this.cookiesPath);
|
||||
this.lastRefresh = Date.now();
|
||||
this.isValid = true;
|
||||
console.log('✅ Cookies refreshed successfully');
|
||||
return true;
|
||||
} catch {
|
||||
console.error('❌ Cookies file not created');
|
||||
this.isValid = false;
|
||||
return false;
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('❌ Failed to refresh cookies:', err.message);
|
||||
this.isValid = false;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check cookies age and validity, refresh if needed.
|
||||
*/
|
||||
async checkAndRefresh() {
|
||||
console.log('🔍 Checking cookies status...');
|
||||
|
||||
// Check file age
|
||||
try {
|
||||
const stats = await fs.stat(this.cookiesPath);
|
||||
const ageMs = Date.now() - stats.mtimeMs;
|
||||
const ageDays = ageMs / (1000 * 60 * 60 * 24);
|
||||
|
||||
console.log(` Age: ${ageDays.toFixed(1)} days`);
|
||||
|
||||
// Refresh if too old
|
||||
if (ageDays >= this.refreshIntervalDays) {
|
||||
console.log(` Age threshold (${this.refreshIntervalDays} days) reached, refreshing...`);
|
||||
await this.refresh();
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// File doesn't exist
|
||||
console.log(' Cookies file missing, refreshing...');
|
||||
await this.refresh();
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate cookies
|
||||
const valid = await this.validate();
|
||||
if (!valid) {
|
||||
console.log(' Cookies invalid, refreshing...');
|
||||
await this.refresh();
|
||||
} else {
|
||||
console.log(' Cookies OK ✅');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get path to cookies file.
|
||||
* @returns {string} Cookies file path
|
||||
*/
|
||||
getCookiesPath() {
|
||||
return this.cookiesPath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get cookies status.
|
||||
* @returns {object} Status object
|
||||
*/
|
||||
getStatus() {
|
||||
return {
|
||||
valid: this.isValid,
|
||||
path: this.cookiesPath,
|
||||
lastRefresh: this.lastRefresh,
|
||||
refreshIntervalDays: this.refreshIntervalDays
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// Export singleton
|
||||
module.exports = new CookiesManager();
|
||||
190
src/services/download.js
Normal file
190
src/services/download.js
Normal file
@ -0,0 +1,190 @@
|
||||
const { spawn } = require('child_process');
|
||||
const cookiesManager = require('./cookiesManager');
|
||||
const path = require('path');
|
||||
const fs = require('fs').promises;
|
||||
|
||||
/**
|
||||
* YouTube download service using yt-dlp with Camoufox stealth cookies.
|
||||
*/
|
||||
class DownloadService {
|
||||
constructor() {
|
||||
this.storagePath = process.env.STORAGE_PATH || path.join(__dirname, '../../output');
|
||||
this.ytdlpPath = process.env.YTDLP_PATH || 'yt-dlp';
|
||||
}
|
||||
|
||||
/**
|
||||
* Download YouTube video as MP3.
|
||||
* @param {string} url - YouTube video URL
|
||||
* @param {object} options - Download options
|
||||
* @returns {Promise<object>} Download result with metadata
|
||||
*/
|
||||
async downloadYouTube(url, options = {}) {
|
||||
// Ensure storage directory exists
|
||||
await fs.mkdir(this.storagePath, { recursive: true });
|
||||
|
||||
// Ensure cookies are valid before download
|
||||
await cookiesManager.checkAndRefresh();
|
||||
|
||||
const cookiesPath = cookiesManager.getCookiesPath();
|
||||
|
||||
// Build yt-dlp arguments
|
||||
const outputTemplate = path.join(this.storagePath, '%(id)s.%(ext)s');
|
||||
const quality = options.quality || '192k';
|
||||
|
||||
const args = [
|
||||
// Cookies (stealth from Camoufox)
|
||||
'--cookies', cookiesPath,
|
||||
|
||||
// Player client (mweb is stable)
|
||||
'--extractor-args', 'youtube:player_client=mweb',
|
||||
|
||||
// Format selection (audio only)
|
||||
'--format', 'bestaudio[ext=m4a]/bestaudio',
|
||||
|
||||
// Audio extraction
|
||||
'--extract-audio',
|
||||
'--audio-format', 'mp3',
|
||||
'--audio-quality', quality,
|
||||
|
||||
// Metadata
|
||||
'--embed-thumbnail',
|
||||
'--add-metadata',
|
||||
|
||||
// No playlists (single video only)
|
||||
'--no-playlist',
|
||||
|
||||
// Output JSON metadata
|
||||
'--print-json',
|
||||
|
||||
// Output template
|
||||
'--output', outputTemplate,
|
||||
|
||||
// URL
|
||||
url
|
||||
];
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const ytdlp = spawn(this.ytdlpPath, args);
|
||||
|
||||
let jsonOutput = '';
|
||||
let errorOutput = '';
|
||||
|
||||
ytdlp.stdout.on('data', (data) => {
|
||||
const text = data.toString();
|
||||
jsonOutput += text;
|
||||
});
|
||||
|
||||
ytdlp.stderr.on('data', (data) => {
|
||||
const text = data.toString();
|
||||
errorOutput += text;
|
||||
|
||||
// Log progress
|
||||
if (text.includes('[download]') || text.includes('[ExtractAudio]')) {
|
||||
console.log(' ', text.trim());
|
||||
}
|
||||
});
|
||||
|
||||
ytdlp.on('close', async (code) => {
|
||||
if (code === 0) {
|
||||
try {
|
||||
// Parse JSON output from yt-dlp
|
||||
const lines = jsonOutput.split('\n').filter(l => l.trim());
|
||||
const lastLine = lines[lines.length - 1];
|
||||
const metadata = JSON.parse(lastLine);
|
||||
|
||||
// Extract relevant metadata
|
||||
const result = {
|
||||
success: true,
|
||||
title: metadata.title,
|
||||
duration: metadata.duration,
|
||||
artist: metadata.artist || metadata.uploader || metadata.channel,
|
||||
album: metadata.album || null,
|
||||
filePath: metadata.filename,
|
||||
fileName: path.basename(metadata.filename),
|
||||
fileSize: metadata.filesize || null,
|
||||
youtubeId: metadata.id,
|
||||
youtubeUrl: metadata.webpage_url,
|
||||
thumbnail: metadata.thumbnail,
|
||||
uploadDate: metadata.upload_date,
|
||||
description: metadata.description || null
|
||||
};
|
||||
|
||||
console.log(`✅ Downloaded: ${result.title}`);
|
||||
resolve(result);
|
||||
} catch (err) {
|
||||
reject(new Error(`Failed to parse yt-dlp output: ${err.message}`));
|
||||
}
|
||||
} else {
|
||||
// Check for specific errors
|
||||
if (errorOutput.includes('Sign in to confirm')) {
|
||||
console.log('🤖 Bot detection! Force refreshing cookies...');
|
||||
|
||||
// Force refresh cookies
|
||||
await cookiesManager.refresh();
|
||||
|
||||
// Retry once
|
||||
try {
|
||||
console.log('🔄 Retrying download with fresh cookies...');
|
||||
const result = await this.downloadYouTube(url, options);
|
||||
resolve(result);
|
||||
} catch (retryErr) {
|
||||
reject(new Error(`Download failed after cookie refresh: ${retryErr.message}`));
|
||||
}
|
||||
} else if (errorOutput.includes('Video unavailable')) {
|
||||
reject(new Error('Video is unavailable or private'));
|
||||
} else if (errorOutput.includes('429')) {
|
||||
reject(new Error('Rate limited by YouTube. Please wait and try again later.'));
|
||||
} else {
|
||||
reject(new Error(`yt-dlp failed (code ${code}): ${errorOutput}`));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
ytdlp.on('error', (err) => {
|
||||
reject(new Error(`Failed to spawn yt-dlp: ${err.message}`));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Get file stream for a downloaded file.
|
||||
* @param {string} fileName - File name
|
||||
* @returns {Promise<object>} File info and stream
|
||||
*/
|
||||
async getFileStream(fileName) {
|
||||
const filePath = path.join(this.storagePath, fileName);
|
||||
|
||||
// Check if file exists
|
||||
try {
|
||||
const stats = await fs.stat(filePath);
|
||||
return {
|
||||
path: filePath,
|
||||
size: stats.size,
|
||||
exists: true
|
||||
};
|
||||
} catch {
|
||||
throw new Error('File not found');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a downloaded file.
|
||||
* @param {string} fileName - File name
|
||||
* @returns {Promise<boolean>} True if deleted
|
||||
*/
|
||||
async deleteFile(fileName) {
|
||||
const filePath = path.join(this.storagePath, fileName);
|
||||
|
||||
try {
|
||||
await fs.unlink(filePath);
|
||||
console.log(`🗑️ Deleted: ${fileName}`);
|
||||
return true;
|
||||
} catch (err) {
|
||||
console.error(`Failed to delete ${fileName}:`, err.message);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Export singleton
|
||||
module.exports = new DownloadService();
|
||||
@ -1,195 +0,0 @@
|
||||
import OpenAI from 'openai';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
let openai = null;
|
||||
|
||||
// Max characters per chunk for summarization
|
||||
const MAX_CHUNK_CHARS = 30000;
|
||||
|
||||
/**
|
||||
* Get OpenAI client (lazy initialization)
|
||||
*/
|
||||
function getOpenAI() {
|
||||
if (!openai) {
|
||||
if (!process.env.OPENAI_API_KEY) {
|
||||
throw new Error('OPENAI_API_KEY environment variable is not set');
|
||||
}
|
||||
openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
|
||||
}
|
||||
return openai;
|
||||
}
|
||||
|
||||
/**
|
||||
* Summarize text using GPT-4o
|
||||
*/
|
||||
export async function summarizeText(text, options = {}) {
|
||||
const {
|
||||
model = 'gpt-5.1', // GPT-5.1 - latest OpenAI model (Nov 2025)
|
||||
language = 'same', // 'same' = same as input, or specify language code
|
||||
style = 'concise', // 'concise', 'detailed', 'bullet'
|
||||
maxLength = null, // optional max length in words
|
||||
} = options;
|
||||
|
||||
const client = getOpenAI();
|
||||
|
||||
let styleInstruction = '';
|
||||
switch (style) {
|
||||
case 'detailed':
|
||||
styleInstruction = 'Provide a detailed summary that captures all important points and nuances.';
|
||||
break;
|
||||
case 'bullet':
|
||||
styleInstruction = 'Provide the summary as bullet points, highlighting the key points.';
|
||||
break;
|
||||
case 'concise':
|
||||
default:
|
||||
styleInstruction = 'Provide a concise summary that captures the main points.';
|
||||
}
|
||||
|
||||
let languageInstruction = '';
|
||||
if (language === 'same') {
|
||||
languageInstruction = 'Write the summary in the same language as the input text.';
|
||||
} else {
|
||||
languageInstruction = `Write the summary in ${language}.`;
|
||||
}
|
||||
|
||||
let lengthInstruction = '';
|
||||
if (maxLength) {
|
||||
lengthInstruction = `Keep the summary under ${maxLength} words.`;
|
||||
}
|
||||
|
||||
const systemPrompt = `You are an expert summarizer. ${styleInstruction} ${languageInstruction} ${lengthInstruction}
|
||||
Focus on the most important information and main ideas. Be accurate and objective.`;
|
||||
|
||||
// Handle long texts by chunking
|
||||
if (text.length > MAX_CHUNK_CHARS) {
|
||||
return await summarizeLongText(text, { model, systemPrompt, style });
|
||||
}
|
||||
|
||||
const response = await client.chat.completions.create({
|
||||
model,
|
||||
messages: [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
{ role: 'user', content: `Please summarize the following text:\n\n${text}` },
|
||||
],
|
||||
temperature: 0.3,
|
||||
});
|
||||
|
||||
return {
|
||||
summary: response.choices[0].message.content,
|
||||
model,
|
||||
style,
|
||||
inputLength: text.length,
|
||||
chunks: 1,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Summarize long text by chunking and combining summaries
|
||||
*/
|
||||
async function summarizeLongText(text, options) {
|
||||
const { model, systemPrompt, style } = options;
|
||||
const client = getOpenAI();
|
||||
|
||||
// Split into chunks
|
||||
const chunks = [];
|
||||
let currentChunk = '';
|
||||
const sentences = text.split(/(?<=[.!?。！？\n])\s*/);
|
||||
|
||||
for (const sentence of sentences) {
|
||||
if ((currentChunk + sentence).length > MAX_CHUNK_CHARS && currentChunk) {
|
||||
chunks.push(currentChunk.trim());
|
||||
currentChunk = sentence;
|
||||
} else {
|
||||
currentChunk += ' ' + sentence;
|
||||
}
|
||||
}
|
||||
if (currentChunk.trim()) {
|
||||
chunks.push(currentChunk.trim());
|
||||
}
|
||||
|
||||
console.log(`Summarizing ${chunks.length} chunks...`);
|
||||
|
||||
// Summarize each chunk
|
||||
const chunkSummaries = [];
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
console.log(`[${i + 1}/${chunks.length}] Summarizing chunk...`);
|
||||
const response = await client.chat.completions.create({
|
||||
model,
|
||||
messages: [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
{ role: 'user', content: `Please summarize the following text (part ${i + 1} of ${chunks.length}):\n\n${chunks[i]}` },
|
||||
],
|
||||
temperature: 0.3,
|
||||
});
|
||||
chunkSummaries.push(response.choices[0].message.content);
|
||||
}
|
||||
|
||||
// Combine summaries if multiple chunks
|
||||
if (chunkSummaries.length === 1) {
|
||||
return {
|
||||
summary: chunkSummaries[0],
|
||||
model,
|
||||
style,
|
||||
inputLength: text.length,
|
||||
chunks: 1,
|
||||
};
|
||||
}
|
||||
|
||||
// Create final combined summary
|
||||
const combinedText = chunkSummaries.join('\n\n---\n\n');
|
||||
const finalResponse = await client.chat.completions.create({
|
||||
model,
|
||||
messages: [
|
||||
{ role: 'system', content: `You are an expert summarizer. Combine and synthesize the following partial summaries into a single coherent ${style} summary. Remove redundancy and ensure a smooth flow.` },
|
||||
{ role: 'user', content: `Please combine these summaries into one:\n\n${combinedText}` },
|
||||
],
|
||||
temperature: 0.3,
|
||||
});
|
||||
|
||||
return {
|
||||
summary: finalResponse.choices[0].message.content,
|
||||
model,
|
||||
style,
|
||||
inputLength: text.length,
|
||||
chunks: chunks.length,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Summarize a text file
|
||||
*/
|
||||
export async function summarizeFile(filePath, options = {}) {
|
||||
if (!fs.existsSync(filePath)) {
|
||||
throw new Error(`File not found: ${filePath}`);
|
||||
}
|
||||
|
||||
const { outputDir, ...otherOptions } = options;
|
||||
|
||||
const text = fs.readFileSync(filePath, 'utf-8');
|
||||
const result = await summarizeText(text, otherOptions);
|
||||
|
||||
// Save summary to file
|
||||
const dir = outputDir || path.dirname(filePath);
|
||||
const baseName = path.basename(filePath, path.extname(filePath));
|
||||
const summaryPath = path.join(dir, `${baseName}_summary.txt`);
|
||||
|
||||
fs.writeFileSync(summaryPath, result.summary, 'utf-8');
|
||||
|
||||
return {
|
||||
...result,
|
||||
filePath,
|
||||
summaryPath,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get available summary styles
|
||||
*/
|
||||
export function getSummaryStyles() {
|
||||
return {
|
||||
concise: 'A brief summary capturing main points',
|
||||
detailed: 'A comprehensive summary with nuances',
|
||||
bullet: 'Key points as bullet points',
|
||||
};
|
||||
}
|
||||
@ -1,178 +0,0 @@
|
||||
import OpenAI from 'openai';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
let openai = null;
|
||||
|
||||
// Available transcription models
|
||||
const MODELS = {
|
||||
'gpt-4o-transcribe': {
|
||||
name: 'gpt-4o-transcribe',
|
||||
formats: ['json', 'text'],
|
||||
supportsLanguage: true,
|
||||
},
|
||||
'gpt-4o-mini-transcribe': {
|
||||
name: 'gpt-4o-mini-transcribe',
|
||||
formats: ['json', 'text'],
|
||||
supportsLanguage: true,
|
||||
},
|
||||
'whisper-1': {
|
||||
name: 'whisper-1',
|
||||
formats: ['json', 'text', 'srt', 'vtt', 'verbose_json'],
|
||||
supportsLanguage: true,
|
||||
},
|
||||
};
|
||||
|
||||
const DEFAULT_MODEL = 'gpt-4o-mini-transcribe';
|
||||
|
||||
/**
|
||||
* Get OpenAI client (lazy initialization)
|
||||
*/
|
||||
function getOpenAI() {
|
||||
if (!openai) {
|
||||
if (!process.env.OPENAI_API_KEY) {
|
||||
throw new Error('OPENAI_API_KEY environment variable is not set');
|
||||
}
|
||||
openai = new OpenAI({
|
||||
apiKey: process.env.OPENAI_API_KEY,
|
||||
});
|
||||
}
|
||||
return openai;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get available models
|
||||
*/
|
||||
export function getAvailableModels() {
|
||||
return Object.keys(MODELS);
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribe an audio file using OpenAI API
|
||||
* @param {string} filePath - Path to audio file
|
||||
* @param {Object} options - Transcription options
|
||||
* @param {string} options.language - Language code (e.g., 'en', 'fr', 'es', 'zh')
|
||||
* @param {string} options.responseFormat - Output format: 'json' or 'text' (gpt-4o models), or 'srt'/'vtt' (whisper-1 only)
|
||||
* @param {string} options.prompt - Optional context prompt for better accuracy
|
||||
* @param {string} options.model - Model to use (default: gpt-4o-transcribe)
|
||||
*/
|
||||
export async function transcribeFile(filePath, options = {}) {
|
||||
const {
|
||||
language = null, // Auto-detect if null
|
||||
responseFormat = 'text', // json or text for gpt-4o models
|
||||
prompt = null, // Optional context prompt
|
||||
model = DEFAULT_MODEL,
|
||||
} = options;
|
||||
|
||||
if (!fs.existsSync(filePath)) {
|
||||
throw new Error(`File not found: ${filePath}`);
|
||||
}
|
||||
|
||||
const modelConfig = MODELS[model] || MODELS[DEFAULT_MODEL];
|
||||
const actualModel = modelConfig.name;
|
||||
|
||||
// Validate response format for model
|
||||
let actualFormat = responseFormat;
|
||||
if (!modelConfig.formats.includes(responseFormat)) {
|
||||
console.warn(`Format '${responseFormat}' not supported by ${actualModel}, using 'text'`);
|
||||
actualFormat = 'text';
|
||||
}
|
||||
|
||||
try {
|
||||
const transcriptionOptions = {
|
||||
file: fs.createReadStream(filePath),
|
||||
model: actualModel,
|
||||
response_format: actualFormat,
|
||||
};
|
||||
|
||||
if (language) {
|
||||
transcriptionOptions.language = language;
|
||||
}
|
||||
|
||||
if (prompt) {
|
||||
transcriptionOptions.prompt = prompt;
|
||||
}
|
||||
|
||||
console.log(`Using model: ${actualModel}, format: ${actualFormat}${language ? `, language: ${language}` : ''}`);
|
||||
|
||||
const transcription = await getOpenAI().audio.transcriptions.create(transcriptionOptions);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
filePath,
|
||||
text: actualFormat === 'json' || actualFormat === 'verbose_json'
|
||||
? transcription.text
|
||||
: transcription,
|
||||
format: actualFormat,
|
||||
model: actualModel,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(`Transcription failed: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribe and save to file
|
||||
*/
|
||||
export async function transcribeAndSave(filePath, options = {}) {
|
||||
const { outputFormat = 'txt', outputDir = null } = options;
|
||||
|
||||
const result = await transcribeFile(filePath, options);
|
||||
|
||||
// Determine output path
|
||||
const baseName = path.basename(filePath, path.extname(filePath));
|
||||
const outputPath = path.join(
|
||||
outputDir || path.dirname(filePath),
|
||||
`${baseName}.${outputFormat}`
|
||||
);
|
||||
|
||||
// Save transcription
|
||||
fs.writeFileSync(outputPath, result.text, 'utf-8');
|
||||
|
||||
return {
|
||||
...result,
|
||||
transcriptionPath: outputPath,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribe multiple files
|
||||
*/
|
||||
export async function transcribeMultiple(filePaths, options = {}) {
|
||||
const { onProgress, onFileComplete } = options;
|
||||
const results = [];
|
||||
|
||||
for (let i = 0; i < filePaths.length; i++) {
|
||||
const filePath = filePaths[i];
|
||||
|
||||
if (onProgress) {
|
||||
onProgress({ current: i + 1, total: filePaths.length, filePath });
|
||||
}
|
||||
|
||||
console.log(`[${i + 1}/${filePaths.length}] Transcribing: ${path.basename(filePath)}`);
|
||||
|
||||
try {
|
||||
const result = await transcribeAndSave(filePath, options);
|
||||
results.push(result);
|
||||
|
||||
if (onFileComplete) {
|
||||
onFileComplete(result);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`Failed to transcribe ${filePath}: ${error.message}`);
|
||||
results.push({
|
||||
success: false,
|
||||
filePath,
|
||||
error: error.message,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
results,
|
||||
totalFiles: filePaths.length,
|
||||
successCount: results.filter(r => r.success).length,
|
||||
failCount: results.filter(r => !r.success).length,
|
||||
};
|
||||
}
|
||||
@ -1,271 +0,0 @@
|
||||
import OpenAI from 'openai';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
let openai = null;
|
||||
|
||||
// Max characters per chunk (~6000 tokens ≈ 24000 characters for most languages)
|
||||
const MAX_CHUNK_CHARS = 20000;
|
||||
|
||||
const LANGUAGES = {
|
||||
en: 'English',
|
||||
fr: 'French',
|
||||
es: 'Spanish',
|
||||
de: 'German',
|
||||
it: 'Italian',
|
||||
pt: 'Portuguese',
|
||||
zh: 'Chinese',
|
||||
ja: 'Japanese',
|
||||
ko: 'Korean',
|
||||
ru: 'Russian',
|
||||
ar: 'Arabic',
|
||||
hi: 'Hindi',
|
||||
nl: 'Dutch',
|
||||
pl: 'Polish',
|
||||
tr: 'Turkish',
|
||||
vi: 'Vietnamese',
|
||||
th: 'Thai',
|
||||
sv: 'Swedish',
|
||||
da: 'Danish',
|
||||
fi: 'Finnish',
|
||||
no: 'Norwegian',
|
||||
cs: 'Czech',
|
||||
el: 'Greek',
|
||||
he: 'Hebrew',
|
||||
id: 'Indonesian',
|
||||
ms: 'Malay',
|
||||
ro: 'Romanian',
|
||||
uk: 'Ukrainian',
|
||||
};
|
||||
|
||||
// Sentence ending patterns for different languages
|
||||
const SENTENCE_ENDINGS = /[.!?。！？｡\n]/g;
|
||||
|
||||
/**
|
||||
* Get OpenAI client (lazy initialization)
|
||||
*/
|
||||
function getOpenAI() {
|
||||
if (!openai) {
|
||||
if (!process.env.OPENAI_API_KEY) {
|
||||
throw new Error('OPENAI_API_KEY environment variable is not set');
|
||||
}
|
||||
openai = new OpenAI({
|
||||
apiKey: process.env.OPENAI_API_KEY,
|
||||
});
|
||||
}
|
||||
return openai;
|
||||
}
|
||||
|
||||
/**
|
||||
* Split text into chunks at sentence boundaries
|
||||
* @param {string} text - Text to split
|
||||
* @param {number} maxChars - Maximum characters per chunk
|
||||
* @returns {string[]} Array of text chunks
|
||||
*/
|
||||
function splitIntoChunks(text, maxChars = MAX_CHUNK_CHARS) {
|
||||
if (text.length <= maxChars) {
|
||||
return [text];
|
||||
}
|
||||
|
||||
const chunks = [];
|
||||
let currentPos = 0;
|
||||
|
||||
while (currentPos < text.length) {
|
||||
let endPos = currentPos + maxChars;
|
||||
|
||||
// If we're at the end, just take the rest
|
||||
if (endPos >= text.length) {
|
||||
chunks.push(text.slice(currentPos));
|
||||
break;
|
||||
}
|
||||
|
||||
// Find the last sentence ending before maxChars
|
||||
const searchText = text.slice(currentPos, endPos);
|
||||
let lastSentenceEnd = -1;
|
||||
|
||||
// Find all sentence endings in the search range
|
||||
let match;
|
||||
SENTENCE_ENDINGS.lastIndex = 0;
|
||||
while ((match = SENTENCE_ENDINGS.exec(searchText)) !== null) {
|
||||
lastSentenceEnd = match.index + 1; // Include the punctuation
|
||||
}
|
||||
|
||||
// If we found a sentence ending, cut there
|
||||
// Otherwise, look for the next sentence ending after maxChars (up to 20% more)
|
||||
if (lastSentenceEnd > maxChars * 0.5) {
|
||||
endPos = currentPos + lastSentenceEnd;
|
||||
} else {
|
||||
// Look forward for a sentence ending (up to 20% more characters)
|
||||
const extendedSearch = text.slice(endPos, endPos + maxChars * 0.2);
|
||||
SENTENCE_ENDINGS.lastIndex = 0;
|
||||
const forwardMatch = SENTENCE_ENDINGS.exec(extendedSearch);
|
||||
if (forwardMatch) {
|
||||
endPos = endPos + forwardMatch.index + 1;
|
||||
}
|
||||
// If still no sentence ending found, just cut at maxChars
|
||||
}
|
||||
|
||||
chunks.push(text.slice(currentPos, endPos).trim());
|
||||
currentPos = endPos;
|
||||
|
||||
// Skip any leading whitespace for the next chunk
|
||||
while (currentPos < text.length && /\s/.test(text[currentPos])) {
|
||||
currentPos++;
|
||||
}
|
||||
}
|
||||
|
||||
return chunks.filter(chunk => chunk.length > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Get available languages
|
||||
*/
|
||||
export function getLanguages() {
|
||||
return LANGUAGES;
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate a single chunk of text
|
||||
*/
|
||||
async function translateChunk(text, targetLanguage, sourceLanguage) {
|
||||
const prompt = sourceLanguage
|
||||
? `Translate the following text from ${sourceLanguage} to ${targetLanguage}. Only output the translation, nothing else:\n\n${text}`
|
||||
: `Translate the following text to ${targetLanguage}. Only output the translation, nothing else:\n\n${text}`;
|
||||
|
||||
const response = await getOpenAI().chat.completions.create({
|
||||
model: 'gpt-4o-mini',
|
||||
max_tokens: 16384,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: prompt,
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
return response.choices[0].message.content;
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate text using GPT-4o-mini with chunking for long texts
|
||||
* @param {string} text - Text to translate
|
||||
* @param {string} targetLang - Target language code (e.g., 'en', 'fr')
|
||||
* @param {string} sourceLang - Source language code (optional, auto-detect if null)
|
||||
*/
|
||||
export async function translateText(text, targetLang, sourceLang = null) {
|
||||
if (!text || !text.trim()) {
|
||||
throw new Error('No text provided for translation');
|
||||
}
|
||||
|
||||
const targetLanguage = LANGUAGES[targetLang] || targetLang;
|
||||
const sourceLanguage = sourceLang ? (LANGUAGES[sourceLang] || sourceLang) : null;
|
||||
|
||||
try {
|
||||
// Split text into chunks
|
||||
const chunks = splitIntoChunks(text);
|
||||
|
||||
if (chunks.length === 1) {
|
||||
// Single chunk - translate directly
|
||||
const translation = await translateChunk(text, targetLanguage, sourceLanguage);
|
||||
return {
|
||||
success: true,
|
||||
originalText: text,
|
||||
translatedText: translation,
|
||||
targetLanguage: targetLanguage,
|
||||
sourceLanguage: sourceLanguage || 'auto-detected',
|
||||
chunks: 1,
|
||||
};
|
||||
}
|
||||
|
||||
// Multiple chunks - translate each and combine
|
||||
console.log(`Splitting text into ${chunks.length} chunks for translation...`);
|
||||
const translations = [];
|
||||
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
console.log(` Translating chunk ${i + 1}/${chunks.length} (${chunks[i].length} chars)...`);
|
||||
const translation = await translateChunk(chunks[i], targetLanguage, sourceLanguage);
|
||||
translations.push(translation);
|
||||
}
|
||||
|
||||
const combinedTranslation = translations.join('\n\n');
|
||||
|
||||
return {
|
||||
success: true,
|
||||
originalText: text,
|
||||
translatedText: combinedTranslation,
|
||||
targetLanguage: targetLanguage,
|
||||
sourceLanguage: sourceLanguage || 'auto-detected',
|
||||
chunks: chunks.length,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(`Translation failed: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate a text file
|
||||
* @param {string} filePath - Path to text file
|
||||
* @param {string} targetLang - Target language code
|
||||
* @param {string} sourceLang - Source language code (optional)
|
||||
* @param {string} outputDir - Output directory (optional)
|
||||
*/
|
||||
export async function translateFile(filePath, targetLang, sourceLang = null, outputDir = null) {
|
||||
if (!fs.existsSync(filePath)) {
|
||||
throw new Error(`File not found: ${filePath}`);
|
||||
}
|
||||
|
||||
const text = fs.readFileSync(filePath, 'utf-8');
|
||||
const result = await translateText(text, targetLang, sourceLang);
|
||||
|
||||
// Save translation
|
||||
const baseName = path.basename(filePath, path.extname(filePath));
|
||||
const outputPath = path.join(
|
||||
outputDir || path.dirname(filePath),
|
||||
`${baseName}_${targetLang}.txt`
|
||||
);
|
||||
|
||||
fs.writeFileSync(outputPath, result.translatedText, 'utf-8');
|
||||
|
||||
return {
|
||||
...result,
|
||||
originalPath: filePath,
|
||||
translationPath: outputPath,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate multiple files
|
||||
*/
|
||||
export async function translateMultiple(filePaths, targetLang, sourceLang = null, outputDir = null, onProgress = null) {
|
||||
const results = [];
|
||||
|
||||
for (let i = 0; i < filePaths.length; i++) {
|
||||
const filePath = filePaths[i];
|
||||
|
||||
if (onProgress) {
|
||||
onProgress({ current: i + 1, total: filePaths.length, filePath });
|
||||
}
|
||||
|
||||
console.log(`[${i + 1}/${filePaths.length}] Translating: ${path.basename(filePath)}`);
|
||||
|
||||
try {
|
||||
const result = await translateFile(filePath, targetLang, sourceLang, outputDir);
|
||||
results.push(result);
|
||||
} catch (error) {
|
||||
console.error(`Failed to translate ${filePath}: ${error.message}`);
|
||||
results.push({
|
||||
success: false,
|
||||
originalPath: filePath,
|
||||
error: error.message,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
results,
|
||||
totalFiles: filePaths.length,
|
||||
successCount: results.filter(r => r.success).length,
|
||||
failCount: results.filter(r => !r.success).length,
|
||||
};
|
||||
}
|
||||
@ -1,383 +0,0 @@
|
||||
import { createRequire } from 'module';
|
||||
import path from 'path';
|
||||
import fs from 'fs';
|
||||
import { spawn } from 'child_process';
|
||||
|
||||
// Use system yt-dlp binary (check common paths)
|
||||
const YTDLP_PATH = process.env.YTDLP_PATH || 'yt-dlp';
|
||||
|
||||
// Path to cookies file (optional)
|
||||
const COOKIES_PATH = process.env.YOUTUBE_COOKIES_PATH || null;
|
||||
|
||||
// Browser to extract cookies from (chrome, firefox, edge, safari, etc.)
|
||||
const COOKIES_BROWSER = process.env.YOUTUBE_COOKIES_BROWSER || null;
|
||||
|
||||
/**
|
||||
* Enhanced error message for YouTube bot detection
|
||||
*/
|
||||
function enhanceYouTubeError(error) {
|
||||
const errorMsg = error.message || error.toString();
|
||||
|
||||
// Check if it's a bot detection error
|
||||
if (errorMsg.includes('Sign in to confirm') ||
|
||||
errorMsg.includes('not a bot') ||
|
||||
errorMsg.includes('confirm you\'re not a bot') ||
|
||||
errorMsg.includes('ERROR: Unable to extract')) {
|
||||
|
||||
const cookiesConfigured = COOKIES_BROWSER || COOKIES_PATH;
|
||||
|
||||
return {
|
||||
error: 'YouTube Bot Detection',
|
||||
message: 'YouTube is blocking this request. Authentication required.',
|
||||
reason: errorMsg,
|
||||
solution: {
|
||||
quick: 'Upload fresh cookies from your browser',
|
||||
steps: [
|
||||
'1. Install browser extension: "Get cookies.txt LOCALLY"',
|
||||
'2. Visit youtube.com and log into your account',
|
||||
'3. Export cookies using the extension',
|
||||
'4. Upload via API: POST /admin/upload-cookies',
|
||||
' Or use the web interface at http://yourserver:8888',
|
||||
],
|
||||
alternative: 'Use extract-and-upload-cookies.sh script for automation',
|
||||
documentation: 'See COOKIES_QUICK_START.md for detailed instructions'
|
||||
},
|
||||
currentConfig: {
|
||||
cookiesFile: COOKIES_PATH || 'Not configured',
|
||||
cookiesBrowser: COOKIES_BROWSER || 'Not configured',
|
||||
status: cookiesConfigured ? '⚠️ Configured but may be expired' : '❌ Not configured'
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Generic YouTube error
|
||||
return {
|
||||
error: 'YouTube Download Failed',
|
||||
message: errorMsg,
|
||||
solution: 'Check if the URL is valid and the video is available'
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Add cookies argument - prioritizes live browser cookies over file
|
||||
*/
|
||||
function addCookiesArg(args, cookiesPath = null) {
|
||||
// Option 1: Extract cookies from browser (always fresh)
|
||||
if (COOKIES_BROWSER) {
|
||||
console.log(`Using live cookies from ${COOKIES_BROWSER} browser`);
|
||||
return ['--cookies-from-browser', COOKIES_BROWSER, ...args];
|
||||
}
|
||||
|
||||
// Option 2: Use static cookies file (may expire)
|
||||
// Check dynamically in case cookies were uploaded after server started
|
||||
const cookies = cookiesPath || process.env.YOUTUBE_COOKIES_PATH || COOKIES_PATH;
|
||||
if (cookies && fs.existsSync(cookies)) {
|
||||
console.log(`Using cookies file: ${cookies}`);
|
||||
return ['--cookies', cookies, ...args];
|
||||
}
|
||||
|
||||
// Option 3: No cookies (may fail on some videos)
|
||||
console.log('No cookies configured - some videos may fail');
|
||||
return args;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute yt-dlp command and return parsed JSON
|
||||
*/
|
||||
async function ytdlp(url, args = [], options = {}) {
|
||||
const { cookiesPath } = options;
|
||||
const finalArgs = addCookiesArg(args, cookiesPath);
|
||||
return new Promise((resolve, reject) => {
|
||||
const proc = spawn(YTDLP_PATH, [...finalArgs, url]);
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
|
||||
proc.stdout.on('data', (data) => { stdout += data; });
|
||||
proc.stderr.on('data', (data) => { stderr += data; });
|
||||
|
||||
proc.on('close', (code) => {
|
||||
if (code === 0) {
|
||||
try {
|
||||
resolve(JSON.parse(stdout));
|
||||
} catch {
|
||||
resolve(stdout);
|
||||
}
|
||||
} else {
|
||||
reject(new Error(stderr || `yt-dlp exited with code ${code}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute yt-dlp command with progress callback
|
||||
*/
|
||||
function ytdlpExec(url, args = [], onProgress, options = {}) {
|
||||
const { cookiesPath } = options;
|
||||
const finalArgs = addCookiesArg(args, cookiesPath);
|
||||
return new Promise((resolve, reject) => {
|
||||
const proc = spawn(YTDLP_PATH, [...finalArgs, url]);
|
||||
let stderr = '';
|
||||
|
||||
proc.stdout.on('data', (data) => {
|
||||
const line = data.toString();
|
||||
if (onProgress) {
|
||||
const progressMatch = line.match(/\[download\]\s+(\d+\.?\d*)%/);
|
||||
const etaMatch = line.match(/ETA\s+(\d+:\d+)/);
|
||||
const speedMatch = line.match(/at\s+([\d.]+\w+\/s)/);
|
||||
|
||||
if (progressMatch) {
|
||||
onProgress({
|
||||
percent: parseFloat(progressMatch[1]),
|
||||
eta: etaMatch ? etaMatch[1] : null,
|
||||
speed: speedMatch ? speedMatch[1] : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
proc.stderr.on('data', (data) => { stderr += data; });
|
||||
|
||||
proc.on('close', (code) => {
|
||||
if (code === 0) {
|
||||
resolve();
|
||||
} else {
|
||||
reject(new Error(stderr || `yt-dlp exited with code ${code}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
const OUTPUT_DIR = process.env.OUTPUT_DIR || './output';
|
||||
|
||||
/**
|
||||
* Sanitize filename to remove invalid characters
|
||||
*/
|
||||
function sanitizeFilename(filename) {
|
||||
return filename
|
||||
.replace(/[<>:"/\\|?*]/g, '')
|
||||
.replace(/\s+/g, '_')
|
||||
.substring(0, 200);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if URL contains a playlist parameter
|
||||
*/
|
||||
function hasPlaylistParam(url) {
|
||||
try {
|
||||
const urlObj = new URL(url);
|
||||
return urlObj.searchParams.has('list');
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract playlist URL if present in the URL
|
||||
*/
|
||||
function extractPlaylistUrl(url) {
|
||||
const urlObj = new URL(url);
|
||||
const listId = urlObj.searchParams.get('list');
|
||||
if (listId) {
|
||||
return `https://www.youtube.com/playlist?list=${listId}`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get video/playlist info without downloading
|
||||
*/
|
||||
export async function getInfo(url, forcePlaylist = false, options = {}) {
|
||||
try {
|
||||
// If URL contains a playlist ID and we want to force playlist mode
|
||||
const playlistUrl = extractPlaylistUrl(url);
|
||||
const targetUrl = (forcePlaylist && playlistUrl) ? playlistUrl : url;
|
||||
|
||||
const info = await ytdlp(targetUrl, [
|
||||
'--dump-single-json',
|
||||
'--no-download',
|
||||
'--no-warnings',
|
||||
'--flat-playlist',
|
||||
], options);
|
||||
return info;
|
||||
} catch (error) {
|
||||
const enhancedError = enhanceYouTubeError(error);
|
||||
const err = new Error(JSON.stringify(enhancedError));
|
||||
err.isEnhanced = true;
|
||||
err.details = enhancedError;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if URL is a playlist
|
||||
*/
|
||||
export async function isPlaylist(url) {
|
||||
const info = await getInfo(url);
|
||||
return info._type === 'playlist';
|
||||
}
|
||||
|
||||
/**
|
||||
* Download a single video as MP3
|
||||
*/
|
||||
export async function downloadVideo(url, options = {}) {
|
||||
const { outputDir = OUTPUT_DIR, onProgress, onDownloadProgress, cookiesPath } = options;
|
||||
|
||||
// Ensure output directory exists
|
||||
if (!fs.existsSync(outputDir)) {
|
||||
fs.mkdirSync(outputDir, { recursive: true });
|
||||
}
|
||||
|
||||
try {
|
||||
// Get video info first
|
||||
const info = await ytdlp(url, [
|
||||
'--dump-single-json',
|
||||
'--no-download',
|
||||
'--no-warnings',
|
||||
], { cookiesPath });
|
||||
|
||||
const title = sanitizeFilename(info.title);
|
||||
const outputPath = path.join(outputDir, `${title}.mp3`);
|
||||
|
||||
// Download and convert to MP3 with progress
|
||||
await ytdlpExec(url, [
|
||||
'--extract-audio',
|
||||
'--audio-format', 'mp3',
|
||||
'--audio-quality', '0',
|
||||
'-o', outputPath,
|
||||
'--no-warnings',
|
||||
'--newline',
|
||||
], (progress) => {
|
||||
if (onDownloadProgress) {
|
||||
onDownloadProgress({
|
||||
...progress,
|
||||
title: info.title,
|
||||
});
|
||||
}
|
||||
}, { cookiesPath });
|
||||
|
||||
return {
|
||||
success: true,
|
||||
title: info.title,
|
||||
duration: info.duration,
|
||||
filePath: outputPath,
|
||||
url: url,
|
||||
};
|
||||
} catch (error) {
|
||||
const enhancedError = enhanceYouTubeError(error);
|
||||
const err = new Error(JSON.stringify(enhancedError));
|
||||
err.isEnhanced = true;
|
||||
err.details = enhancedError;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Download all videos from a playlist as MP3
|
||||
*/
|
||||
export async function downloadPlaylist(url, options = {}) {
|
||||
const { outputDir = OUTPUT_DIR, onProgress, onVideoComplete, onDownloadProgress, forcePlaylist = false, cookiesPath } = options;
|
||||
|
||||
// Ensure output directory exists
|
||||
if (!fs.existsSync(outputDir)) {
|
||||
fs.mkdirSync(outputDir, { recursive: true });
|
||||
}
|
||||
|
||||
try {
|
||||
// Get playlist info (force playlist mode if URL has list= param)
|
||||
const info = await getInfo(url, forcePlaylist || hasPlaylistParam(url), { cookiesPath });
|
||||
|
||||
if (info._type !== 'playlist') {
|
||||
// Single video, redirect to downloadVideo
|
||||
const result = await downloadVideo(url, { ...options, onDownloadProgress });
|
||||
return {
|
||||
success: true,
|
||||
playlistTitle: result.title,
|
||||
videos: [result],
|
||||
totalVideos: 1,
|
||||
};
|
||||
}
|
||||
|
||||
const results = [];
|
||||
const entries = info.entries || [];
|
||||
|
||||
console.log(`Playlist: ${info.title} (${entries.length} videos)`);
|
||||
|
||||
for (let i = 0; i < entries.length; i++) {
|
||||
const entry = entries[i];
|
||||
const videoUrl = entry.url || `https://www.youtube.com/watch?v=${entry.id}`;
|
||||
|
||||
try {
|
||||
if (onProgress) {
|
||||
onProgress({ current: i + 1, total: entries.length, title: entry.title });
|
||||
}
|
||||
|
||||
console.log(`[${i + 1}/${entries.length}] Downloading: ${entry.title}`);
|
||||
|
||||
// Wrap progress callback to include playlist context
|
||||
const wrappedProgress = onDownloadProgress ? (progress) => {
|
||||
onDownloadProgress({
|
||||
...progress,
|
||||
videoIndex: i + 1,
|
||||
totalVideos: entries.length,
|
||||
playlistTitle: info.title,
|
||||
});
|
||||
} : undefined;
|
||||
|
||||
const result = await downloadVideo(videoUrl, { outputDir, onDownloadProgress: wrappedProgress, cookiesPath });
|
||||
results.push(result);
|
||||
|
||||
if (onVideoComplete) {
|
||||
onVideoComplete(result);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`Failed to download ${entry.title}: ${error.message}`);
|
||||
results.push({
|
||||
success: false,
|
||||
title: entry.title,
|
||||
url: videoUrl,
|
||||
error: error.message,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
playlistTitle: info.title,
|
||||
videos: results,
|
||||
totalVideos: entries.length,
|
||||
successCount: results.filter(r => r.success).length,
|
||||
failCount: results.filter(r => !r.success).length,
|
||||
};
|
||||
} catch (error) {
|
||||
const enhancedError = enhanceYouTubeError(error);
|
||||
const err = new Error(JSON.stringify(enhancedError));
|
||||
err.isEnhanced = true;
|
||||
err.details = enhancedError;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Smart download - detects if URL is video or playlist
|
||||
*/
|
||||
export async function download(url, options = {}) {
|
||||
const { cookiesPath } = options;
|
||||
// If URL contains list= parameter, treat it as a playlist
|
||||
const isPlaylistUrl = hasPlaylistParam(url);
|
||||
const info = await getInfo(url, isPlaylistUrl, { cookiesPath });
|
||||
|
||||
if (info._type === 'playlist') {
|
||||
return downloadPlaylist(url, { ...options, forcePlaylist: true });
|
||||
} else {
|
||||
const result = await downloadVideo(url, options);
|
||||
return {
|
||||
success: true,
|
||||
playlistTitle: null,
|
||||
videos: [result],
|
||||
totalVideos: 1,
|
||||
successCount: 1,
|
||||
failCount: 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user