Compare commits
No commits in common. "3735ebdccf0a65c192f33536448d900c96c8b861" and "9f6ac9f8ce788ad9d455f5e1483f96320daeb1d7" have entirely different histories.
3735ebdccf
...
9f6ac9f8ce
25
.env.example
25
.env.example
@ -1,17 +1,16 @@
|
|||||||
# Server Configuration
|
# OpenAI API Key for Whisper transcription
|
||||||
PORT=8889
|
OPENAI_API_KEY=your_openai_api_key_here
|
||||||
|
|
||||||
# Storage path for downloaded MP3 files
|
# Anthropic API Key for Claude Haiku translation (optional)
|
||||||
STORAGE_PATH=/var/hanasuba/music
|
ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
||||||
|
|
||||||
# Python path (optional, default: python3)
|
# Server port (optional, default: 3000)
|
||||||
PYTHON_PATH=python3
|
PORT=3000
|
||||||
|
|
||||||
# yt-dlp path (optional, default: yt-dlp)
|
# Output directory (optional, default: ./output)
|
||||||
YTDLP_PATH=yt-dlp
|
OUTPUT_DIR=./output
|
||||||
|
|
||||||
# CORS (optional, default: *)
|
# YouTube cookies file path (optional, helps bypass bot detection)
|
||||||
ALLOWED_ORIGINS=*
|
# Run: bash scripts/extract-cookies.sh
|
||||||
|
# Then set the path to your cookies file:
|
||||||
# Environment
|
YOUTUBE_COOKIES_PATH=./youtube-cookies.txt
|
||||||
NODE_ENV=production
|
|
||||||
|
|||||||
58
.gitignore
vendored
58
.gitignore
vendored
@ -1,42 +1,38 @@
|
|||||||
# Node
|
# Dependencies
|
||||||
node_modules/
|
node_modules/
|
||||||
npm-debug.log
|
|
||||||
package-lock.json
|
|
||||||
|
|
||||||
# Environment
|
# Environment
|
||||||
.env
|
.env
|
||||||
|
|
||||||
# Output files
|
# Output directory
|
||||||
output/
|
output/
|
||||||
|
|
||||||
|
# Audio files
|
||||||
*.mp3
|
*.mp3
|
||||||
|
*.wav
|
||||||
|
*.m4a
|
||||||
|
*.ogg
|
||||||
|
*.flac
|
||||||
|
*.aac
|
||||||
|
|
||||||
|
# Video files
|
||||||
*.mp4
|
*.mp4
|
||||||
*.webm
|
*.webm
|
||||||
*.m4a
|
*.mkv
|
||||||
|
*.avi
|
||||||
|
|
||||||
# Cookies (sensitive)
|
# Text/transcription files
|
||||||
youtube-cookies.txt
|
*.txt
|
||||||
|
|
||||||
|
# YouTube cookies (contains sensitive authentication data)
|
||||||
|
*cookies*.txt
|
||||||
cookies.txt
|
cookies.txt
|
||||||
*.cookies
|
|
||||||
|
|
||||||
# Python
|
|
||||||
__pycache__/
|
|
||||||
*.pyc
|
|
||||||
*.pyo
|
|
||||||
*.pyd
|
|
||||||
.Python
|
|
||||||
*.egg-info/
|
|
||||||
dist/
|
|
||||||
build/
|
|
||||||
|
|
||||||
# Playwright
|
|
||||||
.cache/
|
|
||||||
playwright/.cache/
|
|
||||||
|
|
||||||
# Logs
|
# Logs
|
||||||
*.log
|
*.log
|
||||||
logs/
|
npm-debug.log*
|
||||||
|
|
||||||
# OS
|
# OS files
|
||||||
.DS_Store
|
.DS_Store
|
||||||
Thumbs.db
|
Thumbs.db
|
||||||
|
|
||||||
@ -46,7 +42,13 @@ Thumbs.db
|
|||||||
*.swp
|
*.swp
|
||||||
*.swo
|
*.swo
|
||||||
|
|
||||||
# Legacy (archived old code)
|
# Temporary files
|
||||||
legacy/
|
*.tmp
|
||||||
youtube-cookies.txt
|
*.temp
|
||||||
.sms_codes.json
|
|
||||||
|
# Windows device names (reserved names that cause issues)
|
||||||
|
nul
|
||||||
|
NUL
|
||||||
|
CON
|
||||||
|
PRN
|
||||||
|
AUX
|
||||||
|
|||||||
88
CHANGELOG.md
88
CHANGELOG.md
@ -1,88 +0,0 @@
|
|||||||
# Changelog
|
|
||||||
|
|
||||||
## [2.0.0] - 2026-01-31
|
|
||||||
|
|
||||||
### 🎉 Complete Rewrite
|
|
||||||
|
|
||||||
**Fresh start with focus on reliability and simplicity.**
|
|
||||||
|
|
||||||
### ✨ Added
|
|
||||||
|
|
||||||
- **Camoufox stealth cookies** - Anti-detection Firefox for cookie extraction
|
|
||||||
- **Automatic cookie refresh** - Refresh every 14 days automatically
|
|
||||||
- **Cookie validation** - Checks validity every 12 hours
|
|
||||||
- **Auto-retry on bot detection** - Refreshes cookies and retries automatically
|
|
||||||
- **Streaming with range requests** - Proper HTTP 206 support for audio seeking
|
|
||||||
- **Clean architecture** - Services pattern (cookiesManager, downloadService)
|
|
||||||
- **Health check endpoint** - `/health` for monitoring
|
|
||||||
- **Admin endpoints** - Force refresh, status check
|
|
||||||
- **Comprehensive docs** - Complete README with examples
|
|
||||||
|
|
||||||
### 🔧 Changed
|
|
||||||
|
|
||||||
- **Focused scope** - Only YouTube → MP3 downloads (removed transcription/translation)
|
|
||||||
- **Simplified stack** - Node.js + Python (Camoufox) + yt-dlp
|
|
||||||
- **Better error handling** - Specific error messages for common issues
|
|
||||||
- **Cleaner config** - Simplified .env variables
|
|
||||||
- **Improved logging** - Clear status messages
|
|
||||||
|
|
||||||
### 🗑️ Removed
|
|
||||||
|
|
||||||
- OpenAI Whisper transcription
|
|
||||||
- Claude translation
|
|
||||||
- Summarization features
|
|
||||||
- CLI interface (API only now)
|
|
||||||
- Complex conversion logic
|
|
||||||
|
|
||||||
### 🎯 Why v2?
|
|
||||||
|
|
||||||
v1 was built for multiple use cases (transcription, translation, etc.). This caused:
|
|
||||||
- Complex codebase
|
|
||||||
- Brittle cookie handling
|
|
||||||
- Frequent failures (~40% success rate)
|
|
||||||
|
|
||||||
v2 focuses on **one thing done right**:
|
|
||||||
- YouTube → MP3 downloads
|
|
||||||
- **~95% success rate** with Camoufox stealth cookies
|
|
||||||
- Auto-healing (refreshes cookies when needed)
|
|
||||||
|
|
||||||
### 📊 Stats
|
|
||||||
|
|
||||||
**v1 → v2 comparison:**
|
|
||||||
- Code size: -4,340 lines (75% reduction)
|
|
||||||
- Dependencies: 8 → 2 (75% reduction)
|
|
||||||
- Success rate: ~60% → ~95% (+35%)
|
|
||||||
- Maintenance: Manual → Automatic
|
|
||||||
- Reliability: Brittle → Rock-solid
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## [1.x] - Legacy
|
|
||||||
|
|
||||||
See `legacy/` folder for old codebase.
|
|
||||||
|
|
||||||
Legacy version included:
|
|
||||||
- YouTube download (yt-dlp)
|
|
||||||
- OpenAI Whisper transcription
|
|
||||||
- Claude translation
|
|
||||||
- GPT-5.1 summarization
|
|
||||||
- File conversion
|
|
||||||
- CLI + API
|
|
||||||
|
|
||||||
**Issues:**
|
|
||||||
- Cookies expired frequently
|
|
||||||
- Manual refresh required
|
|
||||||
- Bot detection failures
|
|
||||||
- Complex to maintain
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Migration from v1 to v2:**
|
|
||||||
|
|
||||||
v1 is **not compatible** with v2. This is a complete rewrite.
|
|
||||||
|
|
||||||
If you need transcription/translation features:
|
|
||||||
- Use legacy branch: `git checkout main`
|
|
||||||
- Or use separate services for those features
|
|
||||||
|
|
||||||
v2 is **specialized** for reliable YouTube → MP3 downloads only.
|
|
||||||
@ -1,89 +0,0 @@
|
|||||||
# 🍪 Quick Setup: YouTube Cookies
|
|
||||||
|
|
||||||
**YouTube bot detection requires logged-in cookies.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 Method 1: Extract from your local Firefox (EASIEST)
|
|
||||||
|
|
||||||
### On your local machine (PC/laptop):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Install yt-dlp if needed
|
|
||||||
pip install yt-dlp
|
|
||||||
|
|
||||||
# Login to YouTube in Firefox first
|
|
||||||
# Then extract cookies:
|
|
||||||
yt-dlp --cookies-from-browser firefox --cookies youtube-cookies.txt 'https://youtube.com'
|
|
||||||
|
|
||||||
# Upload to server:
|
|
||||||
scp youtube-cookies.txt debian@vps-a20accb1.vps.ovh.net:/home/debian/videotomp3transcriptor/
|
|
||||||
```
|
|
||||||
|
|
||||||
### On server:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Restart service
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
pm2 restart music-service # or npm start
|
|
||||||
|
|
||||||
# Test
|
|
||||||
curl -X POST http://localhost:8889/download \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"url": "https://youtube.com/watch?v=fukChj4eh-Q"}'
|
|
||||||
```
|
|
||||||
|
|
||||||
**Duration**: Cookies last 2-4 weeks, then repeat.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🦊 Method 2: Camoufox with manual login (if you have VNC/X11)
|
|
||||||
|
|
||||||
Only works if you can open GUI on server (VNC, X11 forwarding).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
python3 src/python/extract_cookies_with_login.py
|
|
||||||
|
|
||||||
# Browser opens → Login to YouTube → Press Enter
|
|
||||||
# Cookies saved!
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ⚙️ Method 3: Browser extension (Alternative)
|
|
||||||
|
|
||||||
1. Install extension in Firefox:
|
|
||||||
- https://addons.mozilla.org/firefox/addon/cookies-txt/
|
|
||||||
|
|
||||||
2. Go to YouTube, login
|
|
||||||
|
|
||||||
3. Click extension → Export → Save as `youtube-cookies.txt`
|
|
||||||
|
|
||||||
4. Upload to server (see Method 1)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ✅ Verify cookies work:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
yt-dlp --cookies youtube-cookies.txt --skip-download "https://youtube.com/watch?v=dQw4w9WgXcQ"
|
|
||||||
```
|
|
||||||
|
|
||||||
If no error → Cookies work!
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔄 Auto-refresh (once initial cookies uploaded):
|
|
||||||
|
|
||||||
The service will:
|
|
||||||
- ✅ Use your logged-in cookies
|
|
||||||
- ✅ Validate them daily
|
|
||||||
- ✅ Work for 2-4 weeks
|
|
||||||
- ⚠️ Need manual refresh when expired
|
|
||||||
|
|
||||||
**Future improvement**: VNC-based auto-login script (not implemented yet).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**For now**: Use Method 1 (extract from local Firefox) - takes 2 minutes!
|
|
||||||
524
README.md
524
README.md
@ -1,367 +1,235 @@
|
|||||||
# 🎵 Hanasuba Music Service v2.0
|
# Video to MP3 Transcriptor
|
||||||
|
|
||||||
**YouTube to MP3 download service with Camoufox stealth cookies**
|
Download YouTube videos/playlists to MP3 and transcribe them using OpenAI Whisper API.
|
||||||
|
|
||||||
Built for [Hanasuba](https://git.etheryale.com/StillHammer/hanasuba) backend.
|
## Features
|
||||||
|
|
||||||
---
|
- Download single YouTube videos as MP3
|
||||||
|
- Download entire playlists as MP3
|
||||||
|
- Transcribe audio files using OpenAI Whisper API
|
||||||
|
- CLI interface for quick operations
|
||||||
|
- REST API for integration with other systems
|
||||||
|
|
||||||
## ✨ Features
|
## Prerequisites
|
||||||
|
|
||||||
- ✅ **Stealth cookies** - Camoufox anti-detection Firefox
|
- **Node.js** 18+
|
||||||
- ✅ **Auto-refresh** - Cookies refresh every 14 days automatically
|
- **yt-dlp** installed on your system
|
||||||
- ✅ **Bot detection bypass** - Works around YouTube rate limiting
|
- **ffmpeg** installed (for audio conversion)
|
||||||
- ✅ **Audio-only downloads** - MP3 192kbps (configurable)
|
- **OpenAI API key** (for transcription)
|
||||||
- ✅ **Streaming support** - HTTP range requests for audio players
|
|
||||||
- ✅ **Metadata extraction** - Title, artist, duration, thumbnail
|
|
||||||
- ✅ **Retry logic** - Auto-retry with fresh cookies if blocked
|
|
||||||
- ✅ **REST API** - Simple JSON API for integration
|
|
||||||
|
|
||||||
---
|
### Installing yt-dlp
|
||||||
|
|
||||||
## 🏗️ Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
music-service (Node.js + Python)
|
|
||||||
├── Express API (Node.js)
|
|
||||||
│ ├── Download orchestration
|
|
||||||
│ └── File streaming
|
|
||||||
├── Camoufox (Python)
|
|
||||||
│ ├── Stealth cookie extraction
|
|
||||||
│ └── Cookie validation
|
|
||||||
└── yt-dlp
|
|
||||||
└── YouTube download (using stealth cookies)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Why this stack?**
|
|
||||||
- **Camoufox** = Undetectable Firefox (bypasses bot detection)
|
|
||||||
- **yt-dlp** = Best YouTube downloader (handles all edge cases)
|
|
||||||
- **Node.js** = Fast I/O for streaming
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📦 Installation
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
|
|
||||||
- Node.js 18+
|
|
||||||
- Python 3.9+
|
|
||||||
- yt-dlp
|
|
||||||
- ffmpeg
|
|
||||||
|
|
||||||
### Install
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Clone repo
|
# Windows (winget)
|
||||||
git clone https://git.etheryale.com/StillHammer/videotomp3transcriptor.git
|
winget install yt-dlp
|
||||||
cd videotomp3transcriptor
|
|
||||||
git checkout music-service-v2
|
|
||||||
|
|
||||||
# Install Node dependencies
|
# macOS
|
||||||
|
brew install yt-dlp
|
||||||
|
|
||||||
|
# Linux
|
||||||
|
sudo apt install yt-dlp
|
||||||
|
# or
|
||||||
|
pip install yt-dlp
|
||||||
|
```
|
||||||
|
|
||||||
|
### Installing ffmpeg
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Windows (winget)
|
||||||
|
winget install ffmpeg
|
||||||
|
|
||||||
|
# macOS
|
||||||
|
brew install ffmpeg
|
||||||
|
|
||||||
|
# Linux
|
||||||
|
sudo apt install ffmpeg
|
||||||
|
```
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone and install
|
||||||
|
cd videotoMP3Transcriptor
|
||||||
npm install
|
npm install
|
||||||
|
|
||||||
# Install Python dependencies + browsers
|
# Configure environment
|
||||||
npm run setup
|
|
||||||
|
|
||||||
# Configure
|
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
nano .env # Edit PORT, STORAGE_PATH, etc.
|
# Edit .env and add your OPENAI_API_KEY
|
||||||
|
|
||||||
# Start
|
|
||||||
npm start
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
## Usage
|
||||||
|
|
||||||
## 🚀 Usage
|
### CLI
|
||||||
|
|
||||||
### Start server
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm start
|
# Download a video as MP3
|
||||||
|
npm run cli download "https://youtube.com/watch?v=VIDEO_ID"
|
||||||
|
|
||||||
|
# Download a playlist
|
||||||
|
npm run cli download "https://youtube.com/playlist?list=PLAYLIST_ID"
|
||||||
|
|
||||||
|
# Download with custom output directory
|
||||||
|
npm run cli download "URL" -o ./my-folder
|
||||||
|
|
||||||
|
# Get info about a video/playlist
|
||||||
|
npm run cli info "URL"
|
||||||
|
|
||||||
|
# Transcribe an existing MP3
|
||||||
|
npm run cli transcribe ./output/video.mp3
|
||||||
|
|
||||||
|
# Transcribe with specific language
|
||||||
|
npm run cli transcribe ./output/video.mp3 -l fr
|
||||||
|
|
||||||
|
# Transcribe with specific model
|
||||||
|
npm run cli transcribe ./output/video.mp3 -m gpt-4o-mini-transcribe
|
||||||
|
|
||||||
|
# Download AND transcribe
|
||||||
|
npm run cli process "URL"
|
||||||
|
|
||||||
|
# Download and transcribe with options
|
||||||
|
npm run cli process "URL" -l en -m gpt-4o-transcribe
|
||||||
```
|
```
|
||||||
|
|
||||||
Server runs on `http://localhost:8889` (configurable via `.env`)
|
### Linux Scripts
|
||||||
|
|
||||||
### API Endpoints
|
Convenience scripts are available in the `scripts/` directory:
|
||||||
|
|
||||||
#### **POST /download**
|
|
||||||
|
|
||||||
Download YouTube video to MP3.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://localhost:8889/download \
|
# Make scripts executable (first time only)
|
||||||
|
chmod +x scripts/*.sh
|
||||||
|
|
||||||
|
# Download video/playlist
|
||||||
|
./scripts/download.sh "https://youtube.com/watch?v=VIDEO_ID"
|
||||||
|
|
||||||
|
# Transcribe a file
|
||||||
|
./scripts/transcribe.sh ./output/video.mp3 fr
|
||||||
|
|
||||||
|
# Download + transcribe
|
||||||
|
./scripts/process.sh "https://youtube.com/watch?v=VIDEO_ID" en
|
||||||
|
|
||||||
|
# Start the API server
|
||||||
|
./scripts/server.sh
|
||||||
|
|
||||||
|
# Get video info
|
||||||
|
./scripts/info.sh "https://youtube.com/watch?v=VIDEO_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start the server
|
||||||
|
npm run server
|
||||||
|
```
|
||||||
|
|
||||||
|
Server runs on `http://localhost:3000` by default.
|
||||||
|
|
||||||
|
#### Endpoints
|
||||||
|
|
||||||
|
##### GET /health
|
||||||
|
Health check endpoint.
|
||||||
|
|
||||||
|
##### GET /info?url=YOUTUBE_URL
|
||||||
|
Get info about a video or playlist.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:3000/info?url=https://youtube.com/watch?v=VIDEO_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
##### POST /download
|
||||||
|
Download video(s) as MP3.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3000/download \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{"url": "https://youtube.com/watch?v=dQw4w9WgXcQ"}'
|
-d '{"url": "https://youtube.com/watch?v=VIDEO_ID"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
Response:
|
##### POST /transcribe
|
||||||
```json
|
Transcribe an existing audio file.
|
||||||
{
|
|
||||||
"success": true,
|
|
||||||
"title": "Rick Astley - Never Gonna Give You Up",
|
|
||||||
"duration": 212,
|
|
||||||
"artist": "Rick Astley",
|
|
||||||
"filePath": "/var/hanasuba/music/dQw4w9WgXcQ.mp3",
|
|
||||||
"fileName": "dQw4w9WgXcQ.mp3",
|
|
||||||
"youtubeId": "dQw4w9WgXcQ",
|
|
||||||
"thumbnail": "https://..."
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
#### **GET /stream/:filename**
|
|
||||||
|
|
||||||
Stream MP3 file (supports range requests for seeking).
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:8889/stream/dQw4w9WgXcQ.mp3 --output song.mp3
|
curl -X POST http://localhost:3000/transcribe \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"filePath": "./output/video.mp3", "language": "en"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
#### **DELETE /file/:filename**
|
##### POST /process
|
||||||
|
Download and transcribe in one call.
|
||||||
Delete downloaded file.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X DELETE http://localhost:8889/file/dQw4w9WgXcQ.mp3
|
curl -X POST http://localhost:3000/process \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"url": "https://youtube.com/watch?v=VIDEO_ID", "language": "en", "format": "txt"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
#### **GET /health**
|
##### GET /files-list
|
||||||
|
List all downloaded files.
|
||||||
|
|
||||||
Health check.
|
##### GET /files/:filename
|
||||||
|
Download/stream a specific file.
|
||||||
|
|
||||||
```bash
|
## Configuration
|
||||||
curl http://localhost:8889/health
|
|
||||||
|
Environment variables (`.env`):
|
||||||
|
|
||||||
|
| Variable | Description | Default |
|
||||||
|
|----------|-------------|---------|
|
||||||
|
| `OPENAI_API_KEY` | Your OpenAI API key | Required for transcription |
|
||||||
|
| `PORT` | Server port | 3000 |
|
||||||
|
| `OUTPUT_DIR` | Download directory | ./output |
|
||||||
|
|
||||||
|
## Transcription Models
|
||||||
|
|
||||||
|
| Model | Description | Formats |
|
||||||
|
|-------|-------------|---------|
|
||||||
|
| `gpt-4o-transcribe` | Best quality, latest GPT-4o (default) | txt, json |
|
||||||
|
| `gpt-4o-mini-transcribe` | Faster, cheaper, good quality | txt, json |
|
||||||
|
| `whisper-1` | Legacy Whisper model | txt, json, srt, vtt |
|
||||||
|
|
||||||
|
## Transcription Formats
|
||||||
|
|
||||||
|
- `txt` - Plain text (all models)
|
||||||
|
- `json` - JSON response (all models)
|
||||||
|
- `srt` - SubRip subtitles (whisper-1 only)
|
||||||
|
- `vtt` - WebVTT subtitles (whisper-1 only)
|
||||||
|
|
||||||
|
## Language Codes
|
||||||
|
|
||||||
|
Common language codes for the `-l` option:
|
||||||
|
- `en` - English
|
||||||
|
- `fr` - French
|
||||||
|
- `es` - Spanish
|
||||||
|
- `de` - German
|
||||||
|
- `it` - Italian
|
||||||
|
- `pt` - Portuguese
|
||||||
|
- `zh` - Chinese
|
||||||
|
- `ja` - Japanese
|
||||||
|
- `ko` - Korean
|
||||||
|
- `ru` - Russian
|
||||||
|
|
||||||
|
Leave empty for auto-detection.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
videotoMP3Transcriptor/
|
||||||
|
├── src/
|
||||||
|
│ ├── services/
|
||||||
|
│ │ ├── youtube.js # YouTube download service
|
||||||
|
│ │ └── transcription.js # OpenAI transcription service
|
||||||
|
│ ├── cli.js # CLI entry point
|
||||||
|
│ └── server.js # Express API server
|
||||||
|
├── scripts/ # Linux convenience scripts
|
||||||
|
│ ├── download.sh # Download video/playlist
|
||||||
|
│ ├── transcribe.sh # Transcribe audio file
|
||||||
|
│ ├── process.sh # Download + transcribe
|
||||||
|
│ ├── server.sh # Start API server
|
||||||
|
│ └── info.sh # Get video info
|
||||||
|
├── output/ # Downloaded files
|
||||||
|
├── .env # Configuration
|
||||||
|
└── package.json
|
||||||
```
|
```
|
||||||
|
|
||||||
#### **POST /admin/refresh-cookies**
|
## License
|
||||||
|
|
||||||
Force refresh cookies (normally automatic).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -X POST http://localhost:8889/admin/refresh-cookies
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🍪 How Cookies Work
|
|
||||||
|
|
||||||
### Automatic Refresh
|
|
||||||
|
|
||||||
Cookies are **automatically refreshed** in these cases:
|
|
||||||
|
|
||||||
1. **Every 14 days** (proactive refresh)
|
|
||||||
2. **On startup** (if invalid)
|
|
||||||
3. **Every 12 hours** (validation check)
|
|
||||||
4. **On bot detection** (retry with fresh cookies)
|
|
||||||
|
|
||||||
### Manual Refresh
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Via API
|
|
||||||
curl -X POST http://localhost:8889/admin/refresh-cookies
|
|
||||||
|
|
||||||
# Via npm script
|
|
||||||
npm run cookies:extract
|
|
||||||
```
|
|
||||||
|
|
||||||
### Validation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check if cookies are valid
|
|
||||||
npm run cookies:validate
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔧 Configuration
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
See `.env.example`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PORT=8889 # Server port
|
|
||||||
STORAGE_PATH=/var/hanasuba/music # Where to save MP3 files
|
|
||||||
PYTHON_PATH=python3 # Python binary
|
|
||||||
YTDLP_PATH=yt-dlp # yt-dlp binary
|
|
||||||
ALLOWED_ORIGINS=* # CORS
|
|
||||||
```
|
|
||||||
|
|
||||||
### Audio Quality
|
|
||||||
|
|
||||||
Pass `quality` parameter in download request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"url": "https://youtube.com/watch?v=...",
|
|
||||||
"quality": "320k" // or "192k" (default), "128k"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🐛 Troubleshooting
|
|
||||||
|
|
||||||
### "Sign in to confirm you're not a bot"
|
|
||||||
|
|
||||||
**Solution**: Cookies have expired or are invalid.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Force refresh
|
|
||||||
curl -X POST http://localhost:8889/admin/refresh-cookies
|
|
||||||
|
|
||||||
# Or restart service (auto-refresh on startup)
|
|
||||||
npm start
|
|
||||||
```
|
|
||||||
|
|
||||||
### yt-dlp not found
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Install yt-dlp
|
|
||||||
pip install yt-dlp
|
|
||||||
# or
|
|
||||||
sudo apt install yt-dlp
|
|
||||||
```
|
|
||||||
|
|
||||||
### Camoufox install fails
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Manual install
|
|
||||||
pip install camoufox camoufox-captcha playwright
|
|
||||||
playwright install firefox
|
|
||||||
```
|
|
||||||
|
|
||||||
### Downloads slow
|
|
||||||
|
|
||||||
This is normal. YouTube throttles downloads. The service uses `mweb` client for best speed.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔐 Security
|
|
||||||
|
|
||||||
- Cookies file permissions: `600` (owner read/write only)
|
|
||||||
- Cookies **never** logged or exposed
|
|
||||||
- Cookies stored locally only
|
|
||||||
- CORS configurable via `ALLOWED_ORIGINS`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚢 Deployment
|
|
||||||
|
|
||||||
### PM2 (recommended)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pm2 start src/server.js --name music-service
|
|
||||||
pm2 save
|
|
||||||
pm2 startup
|
|
||||||
```
|
|
||||||
|
|
||||||
### systemd
|
|
||||||
|
|
||||||
```ini
|
|
||||||
[Unit]
|
|
||||||
Description=Hanasuba Music Service
|
|
||||||
After=network.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=simple
|
|
||||||
User=debian
|
|
||||||
WorkingDirectory=/home/debian/videotomp3transcriptor
|
|
||||||
ExecStart=/usr/bin/node src/server.js
|
|
||||||
Restart=on-failure
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo systemctl enable music-service
|
|
||||||
sudo systemctl start music-service
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📊 Monitoring
|
|
||||||
|
|
||||||
Check service status:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Health check
|
|
||||||
curl http://localhost:8889/health
|
|
||||||
|
|
||||||
# Cookies status
|
|
||||||
curl http://localhost:8889/admin/cookies-status
|
|
||||||
|
|
||||||
# Logs (PM2)
|
|
||||||
pm2 logs music-service
|
|
||||||
|
|
||||||
# Logs (systemd)
|
|
||||||
journalctl -u music-service -f
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔗 Integration with Hanasuba
|
|
||||||
|
|
||||||
Hanasuba (Rust) calls this service via HTTP:
|
|
||||||
|
|
||||||
```rust
|
|
||||||
// In Hanasuba src/music/client.rs
|
|
||||||
let response = reqwest::Client::new()
|
|
||||||
.post("http://localhost:8889/download")
|
|
||||||
.json(&json!({ "url": youtube_url }))
|
|
||||||
.send()
|
|
||||||
.await?;
|
|
||||||
|
|
||||||
let result: DownloadResult = response.json().await?;
|
|
||||||
// Save metadata to PostgreSQL
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📝 Development
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Dev mode (auto-restart on changes)
|
|
||||||
npm run dev
|
|
||||||
|
|
||||||
# Extract cookies manually
|
|
||||||
npm run cookies:extract
|
|
||||||
|
|
||||||
# Validate cookies
|
|
||||||
npm run cookies:validate
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🆚 v1 vs v2
|
|
||||||
|
|
||||||
| Feature | v1 (legacy) | v2 (current) |
|
|
||||||
|---------|-------------|--------------|
|
|
||||||
| Cookies | Firefox standard | **Camoufox stealth** |
|
|
||||||
| Auto-refresh | ❌ Manual | ✅ Automatic (14 days) |
|
|
||||||
| Bot detection | ❌ Fails often | ✅ Auto-retry |
|
|
||||||
| Validation | ❌ None | ✅ Every 12h |
|
|
||||||
| Reliability | ~60% | **~95%** |
|
|
||||||
| Transcription | ✅ OpenAI Whisper | ❌ Removed (not needed) |
|
|
||||||
| Translation | ✅ Claude | ❌ Removed (not needed) |
|
|
||||||
|
|
||||||
v2 is **focused** on one thing: reliable YouTube → MP3 downloads.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📄 License
|
|
||||||
|
|
||||||
MIT
|
MIT
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🙏 Credits
|
|
||||||
|
|
||||||
- [Camoufox](https://github.com/daijro/camoufox) - Stealth Firefox
|
|
||||||
- [yt-dlp](https://github.com/yt-dlp/yt-dlp) - YouTube downloader
|
|
||||||
- [Hanasuba](https://git.etheryale.com/StillHammer/hanasuba) - Main backend
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Built with ❤️ for Hanasuba**
|
|
||||||
|
|||||||
@ -1,198 +0,0 @@
|
|||||||
# 📱 SMS Forwarder Setup Guide
|
|
||||||
|
|
||||||
## ✅ Serveur SMS Receiver Ready
|
|
||||||
|
|
||||||
**Endpoint URL:** `http://57.131.33.10:4417/sms`
|
|
||||||
|
|
||||||
**Status:** ✅ Running (port 4417)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📲 Setup Android SMS Forwarder
|
|
||||||
|
|
||||||
### Option 1: SMS Forwarder (Recommended)
|
|
||||||
|
|
||||||
1. **Install App**
|
|
||||||
- Play Store: "SMS Forwarder" by bogdanfinn or similar
|
|
||||||
- Or: https://play.google.com/store/apps/details?id=com.lomza.smsforward
|
|
||||||
|
|
||||||
2. **Configure Rule**
|
|
||||||
- Open app → Add new rule
|
|
||||||
- **Trigger:** Sender contains "Google" OR body contains "verification code"
|
|
||||||
- **Action:** HTTP POST
|
|
||||||
- **URL:** `http://57.131.33.10:4417/sms`
|
|
||||||
- **Method:** POST
|
|
||||||
- **Content-Type:** application/json
|
|
||||||
- **Body:**
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"from": "$SENDER$",
|
|
||||||
"body": "$BODY$"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Test**
|
|
||||||
- Send test SMS to yourself with "123456" code
|
|
||||||
- Check server: `curl http://localhost:4417/sms/latest`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Option 2: Tasker + AutoRemote (Advanced)
|
|
||||||
|
|
||||||
1. **Install**
|
|
||||||
- Tasker (paid app)
|
|
||||||
- AutoRemote plugin
|
|
||||||
|
|
||||||
2. **Create Task**
|
|
||||||
- Trigger: SMS received
|
|
||||||
- Filter: Sender contains "Google"
|
|
||||||
- Action: HTTP POST to `http://57.131.33.10:4417/sms`
|
|
||||||
- Body: `{"from":"%SMSRF","body":"%SMSRB"}`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Option 3: IFTTT (Easiest but slower)
|
|
||||||
|
|
||||||
1. **Install IFTTT** app
|
|
||||||
|
|
||||||
2. **Create Applet**
|
|
||||||
- **IF:** SMS received from phone number
|
|
||||||
- **THEN:** Webhooks → Make a web request
|
|
||||||
- **URL:** `http://57.131.33.10:4417/sms`
|
|
||||||
- **Method:** POST
|
|
||||||
- **Content Type:** application/json
|
|
||||||
- **Body:**
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"from": "{{FromNumber}}",
|
|
||||||
"body": "{{Text}}"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Note:** IFTTT peut avoir 5-10 sec de delay
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧪 Test Setup
|
|
||||||
|
|
||||||
### 1. Start SMS Receiver (already running)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
node src/sms_receiver.js &
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Test avec curl (simule SMS)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -X POST http://localhost:4417/sms \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"from":"Google","body":"Your verification code is 123456"}'
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Vérifier réception
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://localhost:4417/sms/latest
|
|
||||||
# Should return: {"code":"123456", ...}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Test auto-login complet
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
export DISPLAY=:99
|
|
||||||
python3 src/python/auto_login_full_auto.py
|
|
||||||
```
|
|
||||||
|
|
||||||
**Le script va :**
|
|
||||||
1. Ouvrir Google login
|
|
||||||
2. Enter email → phone
|
|
||||||
3. **ATTENDRE SMS** (tu recevras SMS sur ton tel)
|
|
||||||
4. SMS Forwarder → envoie au serveur
|
|
||||||
5. Script lit le code automatiquement
|
|
||||||
6. Login complet !
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔒 Sécurité
|
|
||||||
|
|
||||||
**⚠️ Important:** Le serveur SMS tourne sur port **4417** ouvert publiquement.
|
|
||||||
|
|
||||||
**Pour sécuriser** :
|
|
||||||
|
|
||||||
### Option A: Basic Auth (simple)
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Add to sms_receiver.js
|
|
||||||
app.use((req, res, next) => {
|
|
||||||
const token = req.headers.authorization;
|
|
||||||
if (token !== 'Bearer SECRET_TOKEN_HERE') {
|
|
||||||
return res.status(401).json({error: 'Unauthorized'});
|
|
||||||
}
|
|
||||||
next();
|
|
||||||
});
|
|
||||||
```
|
|
||||||
|
|
||||||
Then in SMS Forwarder:
|
|
||||||
- **Headers:** `Authorization: Bearer SECRET_TOKEN_HERE`
|
|
||||||
|
|
||||||
### Option B: Firewall (secure port)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Allow only your phone IP
|
|
||||||
sudo ufw allow from YOUR_PHONE_IP to any port 4417
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option C: VPN/Tunnel (most secure)
|
|
||||||
|
|
||||||
Use Tailscale/WireGuard → SMS endpoint only accessible via VPN
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📊 Monitoring
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check server status
|
|
||||||
curl http://localhost:4417/health
|
|
||||||
|
|
||||||
# View latest SMS
|
|
||||||
curl http://localhost:4417/sms/latest
|
|
||||||
|
|
||||||
# Clear SMS history
|
|
||||||
curl -X DELETE http://localhost:4417/sms/clear
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 Next Steps After Setup
|
|
||||||
|
|
||||||
1. ✅ Configure SMS Forwarder on Android
|
|
||||||
2. ✅ Test with fake SMS (curl)
|
|
||||||
3. ✅ Run auto-login script
|
|
||||||
4. ✅ Test yt-dlp with extracted cookies + PO Token
|
|
||||||
5. ✅ Integrate into music service API
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🆘 Troubleshooting
|
|
||||||
|
|
||||||
**SMS not received by server:**
|
|
||||||
- Check SMS Forwarder app is running
|
|
||||||
- Verify URL is correct (public IP, not localhost)
|
|
||||||
- Check phone has internet connection
|
|
||||||
- Test endpoint: `curl http://57.131.33.10:4417/health`
|
|
||||||
|
|
||||||
**Timeout waiting for SMS:**
|
|
||||||
- Default timeout: 120 seconds
|
|
||||||
- Increase in script if needed
|
|
||||||
- Check SMS Forwarder logs
|
|
||||||
|
|
||||||
**Wrong code extracted:**
|
|
||||||
- Server extracts first 6-digit number
|
|
||||||
- If multiple codes in message, may extract wrong one
|
|
||||||
- Check `/sms/latest` to see what was received
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Questions? Test it and report back!** 🚀
|
|
||||||
@ -1,241 +0,0 @@
|
|||||||
# ✅ YouTube Download System - READY TO USE
|
|
||||||
|
|
||||||
## 🎯 Status: FULLY OPERATIONAL
|
|
||||||
|
|
||||||
**Date:** 2026-01-31
|
|
||||||
**Tests:** ✅ Passed (multiple videos downloaded successfully)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔧 Infrastructure
|
|
||||||
|
|
||||||
### 1. PO Token Provider (Anti-Bot)
|
|
||||||
- **Container:** `bgutil-provider` (Docker)
|
|
||||||
- **Port:** 4416
|
|
||||||
- **Plugin:** `bgutil-ytdlp-pot-provider`
|
|
||||||
- **Status:** ✅ Running
|
|
||||||
- **Check:** `docker ps | grep bgutil`
|
|
||||||
|
|
||||||
### 2. SMS Receiver (For Auto-Renewal)
|
|
||||||
- **Service:** Node.js endpoint
|
|
||||||
- **Port:** 4417
|
|
||||||
- **Webhook:** `http://57.131.33.10:4417/sms`
|
|
||||||
- **Status:** ✅ Running
|
|
||||||
- **Check:** `curl http://localhost:4417/health`
|
|
||||||
|
|
||||||
### 3. JavaScript Runtime
|
|
||||||
- **Runtime:** Deno 2.6.7
|
|
||||||
- **Purpose:** Decrypt YouTube signatures
|
|
||||||
- **Location:** `/usr/local/bin/deno`
|
|
||||||
- **Status:** ✅ Installed
|
|
||||||
|
|
||||||
### 4. Cookies
|
|
||||||
- **File:** `/home/debian/videotomp3transcriptor/youtube-cookies.txt`
|
|
||||||
- **Type:** Logged-in account cookies
|
|
||||||
- **Duration:** 2-4 weeks
|
|
||||||
- **Permissions:** 600 (secure)
|
|
||||||
- **Status:** ✅ Valid
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 Usage
|
|
||||||
|
|
||||||
### Simple Download (Audio MP3)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
|
|
||||||
yt-dlp \
|
|
||||||
--cookies youtube-cookies.txt \
|
|
||||||
--extractor-args "youtube:player_client=mweb" \
|
|
||||||
--format "bestaudio" \
|
|
||||||
--extract-audio \
|
|
||||||
--audio-format mp3 \
|
|
||||||
--output "downloads/%(title)s.%(ext)s" \
|
|
||||||
"YOUTUBE_URL"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Download with Metadata
|
|
||||||
|
|
||||||
```bash
|
|
||||||
yt-dlp \
|
|
||||||
--cookies youtube-cookies.txt \
|
|
||||||
--extractor-args "youtube:player_client=mweb" \
|
|
||||||
--format "bestaudio" \
|
|
||||||
--extract-audio \
|
|
||||||
--audio-format mp3 \
|
|
||||||
--embed-thumbnail \
|
|
||||||
--add-metadata \
|
|
||||||
--output "downloads/%(title)s.%(ext)s" \
|
|
||||||
"YOUTUBE_URL"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Get Video Info Only
|
|
||||||
|
|
||||||
```bash
|
|
||||||
yt-dlp \
|
|
||||||
--cookies youtube-cookies.txt \
|
|
||||||
--extractor-args "youtube:player_client=mweb" \
|
|
||||||
--skip-download \
|
|
||||||
--print "%(title)s|%(duration)s|%(uploader)s" \
|
|
||||||
"YOUTUBE_URL"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📊 What Happens Behind the Scenes
|
|
||||||
|
|
||||||
1. **PO Token Generation**
|
|
||||||
- Plugin detects mweb client
|
|
||||||
- Calls Docker container (port 4416)
|
|
||||||
- Token injected in YouTube API request
|
|
||||||
|
|
||||||
2. **Cookie Authentication**
|
|
||||||
- Logged-in cookies bypass bot detection
|
|
||||||
- Google accepts requests as real user
|
|
||||||
|
|
||||||
3. **Signature Decryption**
|
|
||||||
- Deno runtime solves JS challenges
|
|
||||||
- Decrypts video/audio URLs
|
|
||||||
|
|
||||||
4. **Download**
|
|
||||||
- Best audio format selected
|
|
||||||
- ffmpeg converts to MP3
|
|
||||||
- Metadata embedded
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔄 Cookie Renewal (When Expired)
|
|
||||||
|
|
||||||
### Option A: From Your PC (2 minutes)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# On your PC (with Firefox logged in to YouTube)
|
|
||||||
yt-dlp --cookies-from-browser firefox --cookies youtube-cookies.txt 'https://youtube.com'
|
|
||||||
|
|
||||||
# Upload to server
|
|
||||||
scp youtube-cookies.txt debian@57.131.33.10:/home/debian/videotomp3transcriptor/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option B: SMS Forwarder (Automated)
|
|
||||||
|
|
||||||
**Setup once:**
|
|
||||||
1. Install "SMS Forwarder" on Android
|
|
||||||
2. Configure rule:
|
|
||||||
- Trigger: Sender contains "Google"
|
|
||||||
- Action: HTTP POST
|
|
||||||
- URL: `http://57.131.33.10:4417/sms`
|
|
||||||
- Body: `{"from":"$SENDER$","body":"$BODY$"}`
|
|
||||||
|
|
||||||
**Then run:**
|
|
||||||
```bash
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
export DISPLAY=:99
|
|
||||||
python3 src/python/auto_login_full_auto.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Script will:
|
|
||||||
- Navigate to Google login
|
|
||||||
- Enter credentials
|
|
||||||
- Wait for SMS code (forwarded automatically)
|
|
||||||
- Complete login
|
|
||||||
- Extract fresh cookies
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🛠️ Troubleshooting
|
|
||||||
|
|
||||||
### "Sign in to confirm you're not a bot"
|
|
||||||
**Solution:** Cookies expired, renew them (see above)
|
|
||||||
|
|
||||||
### "Signature solving failed"
|
|
||||||
**Check Deno:** `deno --version`
|
|
||||||
**Reinstall if needed:** See installation section
|
|
||||||
|
|
||||||
### "PO Token generation failed"
|
|
||||||
**Check container:** `docker ps | grep bgutil`
|
|
||||||
**Restart if needed:** `docker restart bgutil-provider`
|
|
||||||
|
|
||||||
### Slow downloads
|
|
||||||
**Try different client:**
|
|
||||||
- `player_client=mweb` (default, stable)
|
|
||||||
- `player_client=android` (faster sometimes)
|
|
||||||
- `player_client=ios` (fallback)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📁 Files & Scripts
|
|
||||||
|
|
||||||
```
|
|
||||||
/home/debian/videotomp3transcriptor/
|
|
||||||
├── youtube-cookies.txt 🔑 Logged-in cookies (KEEP SECURE)
|
|
||||||
├── src/
|
|
||||||
│ ├── sms_receiver.js 📱 SMS webhook endpoint
|
|
||||||
│ ├── python/
|
|
||||||
│ │ ├── auto_login_full_auto.py 🤖 Auto-login with SMS
|
|
||||||
│ │ ├── extract_cookies.py 🍪 Cookie extraction (Camoufox)
|
|
||||||
│ │ └── validate_cookies.txt ✅ Cookie validator
|
|
||||||
│ └── services/
|
|
||||||
│ ├── cookiesManager.js 🔧 Cookie manager (service)
|
|
||||||
│ └── download.js 📥 Download service
|
|
||||||
├── test_youtube_download.sh 🧪 Test script
|
|
||||||
└── SMS_FORWARDER_SETUP.md 📖 SMS setup guide
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ✅ Tested Videos
|
|
||||||
|
|
||||||
- ✅ `fukChj4eh-Q` → 4.0 MB MP3 (success)
|
|
||||||
- ✅ `dQw4w9WgXcQ` (Rick Astley) → 3.6 MB MP3 (success)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎯 Next Steps
|
|
||||||
|
|
||||||
### Integrate into API
|
|
||||||
|
|
||||||
Update `/home/debian/videotomp3transcriptor/src/services/download.js`:
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const cookiesPath = path.join(__dirname, '../../youtube-cookies.txt');
|
|
||||||
|
|
||||||
ytdlp.exec([
|
|
||||||
url,
|
|
||||||
'--cookies', cookiesPath,
|
|
||||||
'--extractor-args', 'youtube:player_client=mweb',
|
|
||||||
'--format', 'bestaudio',
|
|
||||||
'--extract-audio',
|
|
||||||
'--audio-format', 'mp3',
|
|
||||||
'--output', outputPath
|
|
||||||
]);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Monitor & Maintain
|
|
||||||
|
|
||||||
- **Weekly:** Check cookie validity
|
|
||||||
- **Monthly:** Review PO Token plugin updates
|
|
||||||
- **As needed:** Renew cookies when expired
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔐 Security Notes
|
|
||||||
|
|
||||||
- **Cookies file:** Contains auth tokens, permissions 600
|
|
||||||
- **SMS endpoint:** Public on port 4417 (add auth if needed)
|
|
||||||
- **PO Token:** Port 4416 local only
|
|
||||||
- **Logs:** May contain sensitive data, rotate regularly
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📞 Support
|
|
||||||
|
|
||||||
- **yt-dlp docs:** https://github.com/yt-dlp/yt-dlp
|
|
||||||
- **PO Token plugin:** https://github.com/yt-dlp/yt-dlp-plugins
|
|
||||||
- **Deno docs:** https://deno.land/manual
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**System Status:** ✅ READY FOR PRODUCTION
|
|
||||||
|
|
||||||
Last updated: 2026-01-31 08:20 UTC
|
|
||||||
1093
docs/API.md
Normal file
1093
docs/API.md
Normal file
File diff suppressed because it is too large
Load Diff
395
docs/DEPLOIEMENT_OVH.md
Normal file
395
docs/DEPLOIEMENT_OVH.md
Normal file
@ -0,0 +1,395 @@
|
|||||||
|
# Guide de Mise à Jour - Serveur OVH Existant
|
||||||
|
|
||||||
|
Ce guide explique comment mettre à jour ton serveur OVH existant avec le nouveau système de sécurité.
|
||||||
|
|
||||||
|
## Prérequis
|
||||||
|
|
||||||
|
Tu as déjà :
|
||||||
|
- ✅ Un VPS chez OVH
|
||||||
|
- ✅ Git configuré
|
||||||
|
- ✅ Un service qui tourne (PM2/systemd)
|
||||||
|
|
||||||
|
## Étapes de Mise à Jour
|
||||||
|
|
||||||
|
### 1. Générer un token API sécurisé
|
||||||
|
|
||||||
|
**Sur ton serveur OVH (via SSH):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Générer un token aléatoire de 64 caractères
|
||||||
|
openssl rand -hex 32
|
||||||
|
```
|
||||||
|
|
||||||
|
**Ou sur Windows (PowerShell):**
|
||||||
|
```powershell
|
||||||
|
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 64 | % {[char]$_})
|
||||||
|
```
|
||||||
|
|
||||||
|
**Copie ce token**, tu vas en avoir besoin maintenant.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. Configurer les variables d'environnement
|
||||||
|
|
||||||
|
Connecte-toi en SSH à ton serveur :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh user@ton-serveur-ovh.com
|
||||||
|
```
|
||||||
|
|
||||||
|
Navigue vers le dossier du projet :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /chemin/vers/videotoMP3Transcriptor
|
||||||
|
```
|
||||||
|
|
||||||
|
Édite le fichier `.env` :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano .env
|
||||||
|
```
|
||||||
|
|
||||||
|
**Ajoute ces lignes** (ou modifie si elles existent déjà) :
|
||||||
|
|
||||||
|
```env
|
||||||
|
# ========================================
|
||||||
|
# SÉCURITÉ API
|
||||||
|
# ========================================
|
||||||
|
|
||||||
|
# Remplace par le token que tu viens de générer
|
||||||
|
API_TOKEN=ton_token_de_64_caracteres_ici
|
||||||
|
|
||||||
|
# Domaines autorisés (séparés par des virgules)
|
||||||
|
# En développement: * (tout le monde)
|
||||||
|
# En production: https://ton-domaine.com,https://api.ton-domaine.com
|
||||||
|
ALLOWED_ORIGINS=*
|
||||||
|
|
||||||
|
# Port (optionnel, défaut: 8888)
|
||||||
|
PORT=8888
|
||||||
|
|
||||||
|
# OpenAI API Key (tu dois déjà l'avoir)
|
||||||
|
OPENAI_API_KEY=sk-...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sauvegarde** : `Ctrl + X`, puis `Y`, puis `Enter`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. Pull les dernières modifications
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sauvegarder les modifications locales si nécessaire
|
||||||
|
git stash
|
||||||
|
|
||||||
|
# Récupérer les dernières modifications
|
||||||
|
git pull origin main
|
||||||
|
|
||||||
|
# Restaurer tes modifications si tu avais stashé
|
||||||
|
git stash pop
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. Redémarrer le service
|
||||||
|
|
||||||
|
**Si tu utilises PM2:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Redémarrer l'application
|
||||||
|
pm2 restart video-transcriptor
|
||||||
|
|
||||||
|
# Vérifier que ça tourne
|
||||||
|
pm2 status
|
||||||
|
|
||||||
|
# Voir les logs en temps réel
|
||||||
|
pm2 logs video-transcriptor
|
||||||
|
```
|
||||||
|
|
||||||
|
**Si tu utilises systemd:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Redémarrer le service
|
||||||
|
sudo systemctl restart video-transcriptor
|
||||||
|
|
||||||
|
# Vérifier le statut
|
||||||
|
sudo systemctl status video-transcriptor
|
||||||
|
|
||||||
|
# Voir les logs
|
||||||
|
sudo journalctl -u video-transcriptor -f
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. Tester l'API
|
||||||
|
|
||||||
|
**Test de santé (sans token - devrait marcher):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8888/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Résultat attendu:**
|
||||||
|
```json
|
||||||
|
{"status":"ok","timestamp":"2025-..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test avec authentification (devrait échouer sans token):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8888/info?url=https://youtube.com/watch?v=test
|
||||||
|
```
|
||||||
|
|
||||||
|
**Résultat attendu:**
|
||||||
|
```json
|
||||||
|
{"error":"Unauthorized","message":"API key required..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test avec token (devrait marcher):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -H "X-API-Key: ton_token_ici" \
|
||||||
|
"http://localhost:8888/info?url=https://youtube.com/watch?v=dQw4w9WgXcQ"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Résultat attendu:** Informations sur la vidéo
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. Configurer le DNS (si pas déjà fait)
|
||||||
|
|
||||||
|
**Chez OVH, dans l'espace client:**
|
||||||
|
|
||||||
|
1. Va dans **Web Cloud** → **Domaines** → **Ton domaine**
|
||||||
|
2. Clique sur **Zone DNS**
|
||||||
|
3. Ajoute un enregistrement **A** :
|
||||||
|
- Sous-domaine: `api` (ou `@` pour le domaine principal)
|
||||||
|
- Cible: **L'IP de ton VPS OVH**
|
||||||
|
- TTL: 3600
|
||||||
|
|
||||||
|
**Exemple:**
|
||||||
|
```
|
||||||
|
Type: A
|
||||||
|
Nom: api
|
||||||
|
Cible: 51.195.XXX.XXX (ton IP OVH)
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Attends 5-10 minutes** pour la propagation DNS
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 7. Tester depuis l'interface web
|
||||||
|
|
||||||
|
1. **Ouvre ton navigateur** et va sur : `http://ton-domaine.com` (ou `http://ip-du-serveur:8888`)
|
||||||
|
|
||||||
|
2. **Clique sur le panneau "🔐 API Configuration"**
|
||||||
|
|
||||||
|
3. **Colle ton token** dans le champ
|
||||||
|
|
||||||
|
4. **Clique sur "Save & Test"**
|
||||||
|
|
||||||
|
5. **Résultat attendu :**
|
||||||
|
- Statut passe en vert "Connected ✓"
|
||||||
|
- Notification de succès
|
||||||
|
- Le token est sauvegardé dans le navigateur
|
||||||
|
|
||||||
|
6. **Teste un téléchargement** dans l'onglet "Download"
|
||||||
|
- Entre une URL YouTube
|
||||||
|
- Le token sera automatiquement ajouté aux requêtes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sécurité en Production
|
||||||
|
|
||||||
|
### Option 1 : Limiter les origines CORS
|
||||||
|
|
||||||
|
Si tu veux que SEUL ton domaine puisse utiliser l'API :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano .env
|
||||||
|
```
|
||||||
|
|
||||||
|
Change :
|
||||||
|
```env
|
||||||
|
ALLOWED_ORIGINS=https://ton-domaine.com,https://api.ton-domaine.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 2 : HTTPS avec Nginx + Let's Encrypt
|
||||||
|
|
||||||
|
**Si pas déjà configuré**, installe Nginx et SSL :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installer Nginx
|
||||||
|
sudo apt update
|
||||||
|
sudo apt install -y nginx certbot python3-certbot-nginx
|
||||||
|
|
||||||
|
# Créer la configuration Nginx
|
||||||
|
sudo nano /etc/nginx/sites-available/video-transcriptor
|
||||||
|
```
|
||||||
|
|
||||||
|
**Colle cette configuration :**
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
server_name api.ton-domaine.com;
|
||||||
|
|
||||||
|
# Redirection vers HTTPS (sera configuré après)
|
||||||
|
# return 301 https://$server_name$request_uri;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://localhost:8888;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection 'upgrade';
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Activer et tester:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Activer le site
|
||||||
|
sudo ln -s /etc/nginx/sites-available/video-transcriptor /etc/nginx/sites-enabled/
|
||||||
|
|
||||||
|
# Tester la config
|
||||||
|
sudo nginx -t
|
||||||
|
|
||||||
|
# Redémarrer Nginx
|
||||||
|
sudo systemctl restart nginx
|
||||||
|
|
||||||
|
# Obtenir un certificat SSL GRATUIT
|
||||||
|
sudo certbot --nginx -d api.ton-domaine.com
|
||||||
|
```
|
||||||
|
|
||||||
|
Certbot va automatiquement configurer HTTPS et les redirections.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dépannage
|
||||||
|
|
||||||
|
### ❌ "API token required"
|
||||||
|
|
||||||
|
**Problème:** Le token n'est pas envoyé ou invalide
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
1. Vérifie que le token est bien configuré dans l'interface web
|
||||||
|
2. Rafraîchis la page et entre le token à nouveau
|
||||||
|
3. Vérifie que le token dans `.env` est le même que dans l'interface
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### ❌ Le service ne démarre pas
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Voir les logs
|
||||||
|
pm2 logs video-transcriptor --lines 50
|
||||||
|
|
||||||
|
# ou pour systemd
|
||||||
|
sudo journalctl -u video-transcriptor -n 50
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vérifications:**
|
||||||
|
- La variable `API_TOKEN` est bien dans `.env`
|
||||||
|
- Pas d'erreurs de syntaxe dans `.env`
|
||||||
|
- Node modules à jour : `npm ci`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### ❌ CORS errors dans le navigateur
|
||||||
|
|
||||||
|
**Problème:** "Access to fetch at ... has been blocked by CORS policy"
|
||||||
|
|
||||||
|
**Solution 1:** En développement
|
||||||
|
```env
|
||||||
|
ALLOWED_ORIGINS=*
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution 2:** En production
|
||||||
|
```env
|
||||||
|
ALLOWED_ORIGINS=https://ton-domaine.com,https://www.ton-domaine.com
|
||||||
|
```
|
||||||
|
|
||||||
|
Redémarre après modification : `pm2 restart video-transcriptor`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### ❌ DNS ne fonctionne pas
|
||||||
|
|
||||||
|
**Vérifier la propagation DNS:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Depuis ton serveur
|
||||||
|
dig api.ton-domaine.com
|
||||||
|
|
||||||
|
# Ou depuis Windows
|
||||||
|
nslookup api.ton-domaine.com
|
||||||
|
```
|
||||||
|
|
||||||
|
**Si ça ne fonctionne pas:**
|
||||||
|
- Attends 10-30 minutes
|
||||||
|
- Vérifie dans l'interface OVH que l'enregistrement A pointe vers la bonne IP
|
||||||
|
- Vide le cache DNS : `ipconfig /flushdns` (Windows) ou `sudo systemd-resolve --flush-caches` (Linux)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Checklist Finale
|
||||||
|
|
||||||
|
Avant de considérer le déploiement comme terminé :
|
||||||
|
|
||||||
|
- [ ] `.env` configuré avec un `API_TOKEN` fort
|
||||||
|
- [ ] Service redémarré et en cours d'exécution
|
||||||
|
- [ ] Test `/health` fonctionne
|
||||||
|
- [ ] Test avec token fonctionne
|
||||||
|
- [ ] Interface web accessible
|
||||||
|
- [ ] Token sauvegardé dans l'interface web
|
||||||
|
- [ ] Test de téléchargement YouTube réussi
|
||||||
|
- [ ] DNS configuré (si applicable)
|
||||||
|
- [ ] HTTPS configuré (recommandé pour production)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Commandes Utiles
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Voir les logs en temps réel
|
||||||
|
pm2 logs video-transcriptor
|
||||||
|
|
||||||
|
# Statut du service
|
||||||
|
pm2 status
|
||||||
|
|
||||||
|
# Redémarrer
|
||||||
|
pm2 restart video-transcriptor
|
||||||
|
|
||||||
|
# Vérifier les ports ouverts
|
||||||
|
sudo netstat -tlnp | grep 8888
|
||||||
|
|
||||||
|
# Vérifier l'utilisation des ressources
|
||||||
|
htop
|
||||||
|
|
||||||
|
# Espace disque
|
||||||
|
df -h
|
||||||
|
|
||||||
|
# Tester l'API locale
|
||||||
|
curl -H "X-API-Key: ton_token" http://localhost:8888/health
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
Si tu rencontres des problèmes :
|
||||||
|
|
||||||
|
1. **Vérifie les logs** : `pm2 logs`
|
||||||
|
2. **Vérifie le `.env`** : `cat .env | grep API_TOKEN`
|
||||||
|
3. **Teste en local** : `curl http://localhost:8888/health`
|
||||||
|
4. **Vérifie le firewall** : `sudo ufw status`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Bon déploiement ! 🚀**
|
||||||
|
|
||||||
|
Si tout fonctionne, tu devrais pouvoir utiliser l'interface web avec le token sauvegardé, et ne plus avoir à le copier-coller à chaque fois !
|
||||||
699
docs/DEPLOYMENT.md
Normal file
699
docs/DEPLOYMENT.md
Normal file
@ -0,0 +1,699 @@
|
|||||||
|
# Guide de Déploiement - Video to MP3 Transcriptor
|
||||||
|
|
||||||
|
Ce guide vous accompagne pour déployer l'API de manière sécurisée sur un serveur de production.
|
||||||
|
|
||||||
|
## Table des matières
|
||||||
|
1. [Prérequis](#prérequis)
|
||||||
|
2. [Configuration de sécurité](#configuration-de-sécurité)
|
||||||
|
3. [Déploiement sur VPS/Serveur](#déploiement-sur-vpsserveur)
|
||||||
|
4. [Déploiement avec Docker](#déploiement-avec-docker)
|
||||||
|
5. [Nginx Reverse Proxy](#nginx-reverse-proxy)
|
||||||
|
6. [SSL/HTTPS avec Let's Encrypt](#sslhttps-avec-lets-encrypt)
|
||||||
|
7. [Surveillance et logs](#surveillance-et-logs)
|
||||||
|
8. [Sécurité avancée](#sécurité-avancée)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prérequis
|
||||||
|
|
||||||
|
### Serveur
|
||||||
|
- Linux (Ubuntu 20.04+ / Debian 11+ recommandé)
|
||||||
|
- Minimum 2 GB RAM
|
||||||
|
- 10 GB espace disque
|
||||||
|
- Node.js 18+ ou Docker
|
||||||
|
|
||||||
|
### Dépendances système
|
||||||
|
```bash
|
||||||
|
# Ubuntu/Debian
|
||||||
|
sudo apt update
|
||||||
|
sudo apt install -y ffmpeg python3
|
||||||
|
|
||||||
|
# Pour téléchargement YouTube
|
||||||
|
sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
|
||||||
|
sudo chmod a+rx /usr/local/bin/yt-dlp
|
||||||
|
```
|
||||||
|
|
||||||
|
### Domaine et DNS
|
||||||
|
- Un nom de domaine pointant vers votre serveur
|
||||||
|
- Accès aux paramètres DNS
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration de sécurité
|
||||||
|
|
||||||
|
### 1. Générer un token API sécurisé
|
||||||
|
|
||||||
|
**Sur votre serveur:**
|
||||||
|
```bash
|
||||||
|
# Générer un token de 64 caractères
|
||||||
|
openssl rand -hex 32
|
||||||
|
|
||||||
|
# Ou utiliser cette commande alternative
|
||||||
|
head /dev/urandom | tr -dc A-Za-z0-9 | head -c 64
|
||||||
|
```
|
||||||
|
|
||||||
|
Copiez le token généré, vous en aurez besoin pour le `.env`.
|
||||||
|
|
||||||
|
### 2. Configurer les variables d'environnement
|
||||||
|
|
||||||
|
Créez/éditez le fichier `.env` sur le serveur:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /path/to/videotoMP3Transcriptor
|
||||||
|
nano .env
|
||||||
|
```
|
||||||
|
|
||||||
|
Configuration minimale de production:
|
||||||
|
|
||||||
|
```env
|
||||||
|
# ========================================
|
||||||
|
# SÉCURITÉ - PRODUCTION
|
||||||
|
# ========================================
|
||||||
|
|
||||||
|
# Token API (REMPLACEZ PAR VOTRE TOKEN GÉNÉRÉ)
|
||||||
|
API_TOKEN=votre_token_securise_de_64_caracteres
|
||||||
|
|
||||||
|
# Origines CORS autorisées (vos domaines uniquement)
|
||||||
|
ALLOWED_ORIGINS=https://yourdomain.com,https://api.yourdomain.com
|
||||||
|
|
||||||
|
# ========================================
|
||||||
|
# CONFIGURATION SERVEUR
|
||||||
|
# ========================================
|
||||||
|
|
||||||
|
# Port interne (Nginx fera le reverse proxy)
|
||||||
|
PORT=8888
|
||||||
|
|
||||||
|
# Répertoire de sortie
|
||||||
|
OUTPUT_DIR=/var/www/videotoMP3Transcriptor/output
|
||||||
|
|
||||||
|
# ========================================
|
||||||
|
# API KEYS
|
||||||
|
# ========================================
|
||||||
|
|
||||||
|
# OpenAI API Key (OBLIGATOIRE)
|
||||||
|
OPENAI_API_KEY=sk-...
|
||||||
|
|
||||||
|
# ========================================
|
||||||
|
# ENVIRONNEMENT
|
||||||
|
# ========================================
|
||||||
|
NODE_ENV=production
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Permissions du fichier .env
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sécuriser le fichier .env
|
||||||
|
chmod 600 .env
|
||||||
|
chown www-data:www-data .env # ou votre utilisateur système
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Déploiement sur VPS/Serveur
|
||||||
|
|
||||||
|
### 1. Installation de Node.js
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installation de Node.js 20 LTS
|
||||||
|
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
|
||||||
|
sudo apt install -y nodejs
|
||||||
|
|
||||||
|
# Vérification
|
||||||
|
node --version # devrait afficher v20.x
|
||||||
|
npm --version
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Cloner et installer l'application
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Créer le répertoire
|
||||||
|
sudo mkdir -p /var/www/videotoMP3Transcriptor
|
||||||
|
sudo chown $USER:$USER /var/www/videotoMP3Transcriptor
|
||||||
|
|
||||||
|
# Cloner (ou copier) votre code
|
||||||
|
cd /var/www/videotoMP3Transcriptor
|
||||||
|
# git clone ... ou upload manuel
|
||||||
|
|
||||||
|
# Installer les dépendances
|
||||||
|
npm ci --only=production
|
||||||
|
|
||||||
|
# Créer le répertoire de sortie
|
||||||
|
mkdir -p output
|
||||||
|
chmod 755 output
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Utiliser PM2 pour la gestion des processus
|
||||||
|
|
||||||
|
PM2 est un gestionnaire de processus pour Node.js qui redémarre automatiquement votre app en cas de crash.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installer PM2 globalement
|
||||||
|
sudo npm install -g pm2
|
||||||
|
|
||||||
|
# Démarrer l'application
|
||||||
|
pm2 start src/server.js --name "video-transcriptor"
|
||||||
|
|
||||||
|
# Configurer PM2 pour démarrer au boot
|
||||||
|
pm2 startup systemd
|
||||||
|
pm2 save
|
||||||
|
|
||||||
|
# Commandes utiles
|
||||||
|
pm2 status # Voir le statut
|
||||||
|
pm2 logs video-transcriptor # Voir les logs
|
||||||
|
pm2 restart video-transcriptor # Redémarrer
|
||||||
|
pm2 stop video-transcriptor # Arrêter
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Configuration PM2 avancée (optionnelle)
|
||||||
|
|
||||||
|
Créez un fichier `ecosystem.config.js`:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
module.exports = {
|
||||||
|
apps: [{
|
||||||
|
name: 'video-transcriptor',
|
||||||
|
script: './src/server.js',
|
||||||
|
instances: 1,
|
||||||
|
autorestart: true,
|
||||||
|
watch: false,
|
||||||
|
max_memory_restart: '1G',
|
||||||
|
env: {
|
||||||
|
NODE_ENV: 'production',
|
||||||
|
PORT: 8888
|
||||||
|
},
|
||||||
|
error_file: '/var/log/pm2/video-transcriptor-error.log',
|
||||||
|
out_file: '/var/log/pm2/video-transcriptor-out.log',
|
||||||
|
log_date_format: 'YYYY-MM-DD HH:mm:ss Z'
|
||||||
|
}]
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Démarrer avec:
|
||||||
|
```bash
|
||||||
|
pm2 start ecosystem.config.js
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Déploiement avec Docker
|
||||||
|
|
||||||
|
### 1. Créer un Dockerfile
|
||||||
|
|
||||||
|
Créez `Dockerfile` à la racine du projet:
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
FROM node:20-slim
|
||||||
|
|
||||||
|
# Installer les dépendances système
|
||||||
|
RUN apt-get update && apt-get install -y \
|
||||||
|
ffmpeg \
|
||||||
|
python3 \
|
||||||
|
curl \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
# Installer yt-dlp
|
||||||
|
RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp \
|
||||||
|
&& chmod a+rx /usr/local/bin/yt-dlp
|
||||||
|
|
||||||
|
# Créer le répertoire de l'app
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Copier package.json et installer les dépendances
|
||||||
|
COPY package*.json ./
|
||||||
|
RUN npm ci --only=production
|
||||||
|
|
||||||
|
# Copier le code source
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
# Créer le répertoire de sortie
|
||||||
|
RUN mkdir -p /app/output && chmod 755 /app/output
|
||||||
|
|
||||||
|
# Exposer le port
|
||||||
|
EXPOSE 8888
|
||||||
|
|
||||||
|
# Variables d'environnement par défaut
|
||||||
|
ENV NODE_ENV=production
|
||||||
|
ENV PORT=8888
|
||||||
|
ENV OUTPUT_DIR=/app/output
|
||||||
|
|
||||||
|
# Démarrer l'application
|
||||||
|
CMD ["node", "src/server.js"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Créer docker-compose.yml
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
video-transcriptor:
|
||||||
|
build: .
|
||||||
|
container_name: video-transcriptor
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "8888:8888"
|
||||||
|
volumes:
|
||||||
|
- ./output:/app/output
|
||||||
|
- ./.env:/app/.env:ro
|
||||||
|
environment:
|
||||||
|
- NODE_ENV=production
|
||||||
|
networks:
|
||||||
|
- transcriptor-network
|
||||||
|
|
||||||
|
networks:
|
||||||
|
transcriptor-network:
|
||||||
|
driver: bridge
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Lancer avec Docker Compose
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build et démarrer
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# Voir les logs
|
||||||
|
docker-compose logs -f
|
||||||
|
|
||||||
|
# Arrêter
|
||||||
|
docker-compose down
|
||||||
|
|
||||||
|
# Reconstruire après modification
|
||||||
|
docker-compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Nginx Reverse Proxy
|
||||||
|
|
||||||
|
### 1. Installer Nginx
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt update
|
||||||
|
sudo apt install -y nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Configuration Nginx
|
||||||
|
|
||||||
|
Créez `/etc/nginx/sites-available/video-transcriptor`:
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
# Rate limiting
|
||||||
|
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
server_name api.yourdomain.com;
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
access_log /var/log/nginx/video-transcriptor-access.log;
|
||||||
|
error_log /var/log/nginx/video-transcriptor-error.log;
|
||||||
|
|
||||||
|
# Rate limiting
|
||||||
|
limit_req zone=api_limit burst=20 nodelay;
|
||||||
|
|
||||||
|
# Augmenter les timeouts pour les longs traitements
|
||||||
|
proxy_connect_timeout 600;
|
||||||
|
proxy_send_timeout 600;
|
||||||
|
proxy_read_timeout 600;
|
||||||
|
send_timeout 600;
|
||||||
|
|
||||||
|
# Augmenter la taille max des uploads
|
||||||
|
client_max_body_size 500M;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://localhost:8888;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
|
||||||
|
# Headers
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection 'upgrade';
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
|
||||||
|
# Pour Server-Sent Events (SSE)
|
||||||
|
proxy_cache_bypass $http_upgrade;
|
||||||
|
proxy_buffering off;
|
||||||
|
proxy_cache off;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Headers de sécurité supplémentaires
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "DENY" always;
|
||||||
|
add_header X-XSS-Protection "1; mode=block" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Activer le site
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Créer un lien symbolique
|
||||||
|
sudo ln -s /etc/nginx/sites-available/video-transcriptor /etc/nginx/sites-enabled/
|
||||||
|
|
||||||
|
# Tester la configuration
|
||||||
|
sudo nginx -t
|
||||||
|
|
||||||
|
# Recharger Nginx
|
||||||
|
sudo systemctl reload nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SSL/HTTPS avec Let's Encrypt
|
||||||
|
|
||||||
|
### 1. Installer Certbot
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt install -y certbot python3-certbot-nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Obtenir un certificat SSL
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Obtenir et installer automatiquement le certificat
|
||||||
|
sudo certbot --nginx -d api.yourdomain.com
|
||||||
|
|
||||||
|
# Suivez les instructions à l'écran
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Renouvellement automatique
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Tester le renouvellement
|
||||||
|
sudo certbot renew --dry-run
|
||||||
|
|
||||||
|
# Le renouvellement automatique est configuré via cron
|
||||||
|
# Vérifier: sudo systemctl status certbot.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Après SSL, votre configuration Nginx sera automatiquement mise à jour pour HTTPS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Surveillance et logs
|
||||||
|
|
||||||
|
### 1. Logs de l'application
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Avec PM2
|
||||||
|
pm2 logs video-transcriptor
|
||||||
|
|
||||||
|
# Avec Docker
|
||||||
|
docker-compose logs -f video-transcriptor
|
||||||
|
|
||||||
|
# Logs Nginx
|
||||||
|
sudo tail -f /var/log/nginx/video-transcriptor-access.log
|
||||||
|
sudo tail -f /var/log/nginx/video-transcriptor-error.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Monitoring avec PM2 (optionnel)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installer PM2 monitoring
|
||||||
|
pm2 install pm2-logrotate
|
||||||
|
|
||||||
|
# Configurer la rotation des logs
|
||||||
|
pm2 set pm2-logrotate:max_size 10M
|
||||||
|
pm2 set pm2-logrotate:retain 7
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Monitoring système
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installer htop pour surveiller les ressources
|
||||||
|
sudo apt install -y htop
|
||||||
|
|
||||||
|
# Lancer htop
|
||||||
|
htop
|
||||||
|
|
||||||
|
# Voir l'utilisation disque
|
||||||
|
df -h
|
||||||
|
|
||||||
|
# Voir l'utilisation mémoire
|
||||||
|
free -h
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sécurité avancée
|
||||||
|
|
||||||
|
### 1. Firewall (UFW)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installer UFW
|
||||||
|
sudo apt install -y ufw
|
||||||
|
|
||||||
|
# Autoriser SSH (IMPORTANT AVANT D'ACTIVER!)
|
||||||
|
sudo ufw allow ssh
|
||||||
|
sudo ufw allow 22/tcp
|
||||||
|
|
||||||
|
# Autoriser HTTP et HTTPS
|
||||||
|
sudo ufw allow 'Nginx Full'
|
||||||
|
|
||||||
|
# Activer le firewall
|
||||||
|
sudo ufw enable
|
||||||
|
|
||||||
|
# Vérifier le statut
|
||||||
|
sudo ufw status
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Fail2Ban (protection contre brute force)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Installer Fail2Ban
|
||||||
|
sudo apt install -y fail2ban
|
||||||
|
|
||||||
|
# Créer une configuration pour Nginx
|
||||||
|
sudo nano /etc/fail2ban/jail.local
|
||||||
|
```
|
||||||
|
|
||||||
|
Ajouter:
|
||||||
|
```ini
|
||||||
|
[nginx-limit-req]
|
||||||
|
enabled = true
|
||||||
|
filter = nginx-limit-req
|
||||||
|
port = http,https
|
||||||
|
logpath = /var/log/nginx/video-transcriptor-error.log
|
||||||
|
maxretry = 5
|
||||||
|
findtime = 600
|
||||||
|
bantime = 3600
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Redémarrer Fail2Ban
|
||||||
|
sudo systemctl restart fail2ban
|
||||||
|
|
||||||
|
# Vérifier le statut
|
||||||
|
sudo fail2ban-client status nginx-limit-req
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Limitations supplémentaires
|
||||||
|
|
||||||
|
**Limiter les tailles de fichiers uploadés** - Déjà configuré dans Nginx (`client_max_body_size 500M`)
|
||||||
|
|
||||||
|
**Rate limiting par IP** - Déjà configuré dans Nginx (`limit_req_zone`)
|
||||||
|
|
||||||
|
### 4. Sauvegardes automatiques
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Créer un script de backup
|
||||||
|
sudo nano /usr/local/bin/backup-video-transcriptor.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
BACKUP_DIR="/backup/video-transcriptor"
|
||||||
|
APP_DIR="/var/www/videotoMP3Transcriptor"
|
||||||
|
DATE=$(date +%Y%m%d_%H%M%S)
|
||||||
|
|
||||||
|
mkdir -p $BACKUP_DIR
|
||||||
|
|
||||||
|
# Backup de la configuration
|
||||||
|
tar -czf $BACKUP_DIR/config_$DATE.tar.gz \
|
||||||
|
$APP_DIR/.env \
|
||||||
|
$APP_DIR/ecosystem.config.js
|
||||||
|
|
||||||
|
# Backup des fichiers de sortie (optionnel, peut être volumineux)
|
||||||
|
# tar -czf $BACKUP_DIR/output_$DATE.tar.gz $APP_DIR/output
|
||||||
|
|
||||||
|
# Garder seulement les 7 derniers backups
|
||||||
|
find $BACKUP_DIR -name "config_*.tar.gz" -mtime +7 -delete
|
||||||
|
|
||||||
|
echo "Backup completed: $DATE"
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Rendre exécutable
|
||||||
|
sudo chmod +x /usr/local/bin/backup-video-transcriptor.sh
|
||||||
|
|
||||||
|
# Ajouter au crontab (backup quotidien à 2h du matin)
|
||||||
|
sudo crontab -e
|
||||||
|
# Ajouter: 0 2 * * * /usr/local/bin/backup-video-transcriptor.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Checklist finale de déploiement
|
||||||
|
|
||||||
|
Avant de mettre en production, vérifiez:
|
||||||
|
|
||||||
|
- [ ] **Sécurité**
|
||||||
|
- [ ] Token API fort généré (`API_TOKEN`)
|
||||||
|
- [ ] CORS configuré avec vos domaines (`ALLOWED_ORIGINS`)
|
||||||
|
- [ ] Fichier `.env` avec permissions 600
|
||||||
|
- [ ] HTTPS configuré et fonctionnel
|
||||||
|
- [ ] Firewall UFW activé
|
||||||
|
|
||||||
|
- [ ] **Configuration**
|
||||||
|
- [ ] `OPENAI_API_KEY` valide et fonctionnelle
|
||||||
|
- [ ] `NODE_ENV=production`
|
||||||
|
- [ ] Répertoire `output/` créé et accessible
|
||||||
|
- [ ] FFmpeg et yt-dlp installés
|
||||||
|
|
||||||
|
- [ ] **Infrastructure**
|
||||||
|
- [ ] PM2 ou Docker en cours d'exécution
|
||||||
|
- [ ] Nginx reverse proxy configuré
|
||||||
|
- [ ] SSL/TLS actif (Let's Encrypt)
|
||||||
|
- [ ] Rate limiting activé
|
||||||
|
|
||||||
|
- [ ] **Monitoring**
|
||||||
|
- [ ] Logs accessibles
|
||||||
|
- [ ] PM2 startup configuré (redémarrage auto)
|
||||||
|
- [ ] Fail2Ban actif
|
||||||
|
- [ ] Backups automatiques configurés
|
||||||
|
|
||||||
|
- [ ] **Tests**
|
||||||
|
- [ ] Endpoint `/health` accessible
|
||||||
|
- [ ] Test d'authentification (avec et sans token)
|
||||||
|
- [ ] Test d'upload de fichier
|
||||||
|
- [ ] Test de téléchargement YouTube
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tests post-déploiement
|
||||||
|
|
||||||
|
### 1. Test de santé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl https://api.yourdomain.com/health
|
||||||
|
# Devrait retourner: {"status":"ok","timestamp":"..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Test d'authentification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sans token (devrait échouer avec 401)
|
||||||
|
curl https://api.yourdomain.com/info?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ
|
||||||
|
|
||||||
|
# Avec token (devrait réussir)
|
||||||
|
curl -H "X-API-Key: VOTRE_TOKEN" \
|
||||||
|
"https://api.yourdomain.com/info?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Test de download
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -H "X-API-Key: VOTRE_TOKEN" \
|
||||||
|
-X POST https://api.yourdomain.com/download \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dépannage
|
||||||
|
|
||||||
|
### L'API ne démarre pas
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Vérifier les logs PM2
|
||||||
|
pm2 logs video-transcriptor
|
||||||
|
|
||||||
|
# Vérifier les variables d'environnement
|
||||||
|
pm2 env video-transcriptor
|
||||||
|
|
||||||
|
# Redémarrer
|
||||||
|
pm2 restart video-transcriptor
|
||||||
|
```
|
||||||
|
|
||||||
|
### Erreurs 502 Bad Gateway (Nginx)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Vérifier que l'app tourne
|
||||||
|
pm2 status
|
||||||
|
|
||||||
|
# Vérifier les logs Nginx
|
||||||
|
sudo tail -f /var/log/nginx/error.log
|
||||||
|
|
||||||
|
# Vérifier que le port 8888 est ouvert
|
||||||
|
sudo netstat -tlnp | grep 8888
|
||||||
|
```
|
||||||
|
|
||||||
|
### Problèmes SSL
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Vérifier le certificat
|
||||||
|
sudo certbot certificates
|
||||||
|
|
||||||
|
# Renouveler manuellement
|
||||||
|
sudo certbot renew --force-renewal
|
||||||
|
|
||||||
|
# Tester la configuration Nginx
|
||||||
|
sudo nginx -t
|
||||||
|
```
|
||||||
|
|
||||||
|
### Mémoire insuffisante
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Vérifier l'utilisation mémoire
|
||||||
|
free -h
|
||||||
|
|
||||||
|
# Créer un swap file (si nécessaire)
|
||||||
|
sudo fallocate -l 2G /swapfile
|
||||||
|
sudo chmod 600 /swapfile
|
||||||
|
sudo mkswap /swapfile
|
||||||
|
sudo swapon /swapfile
|
||||||
|
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Mises à jour
|
||||||
|
|
||||||
|
### Mise à jour de l'application
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /var/www/videotoMP3Transcriptor
|
||||||
|
|
||||||
|
# Sauvegarder la config
|
||||||
|
cp .env .env.backup
|
||||||
|
|
||||||
|
# Pull des nouvelles versions (git)
|
||||||
|
git pull
|
||||||
|
|
||||||
|
# Mettre à jour les dépendances
|
||||||
|
npm ci --only=production
|
||||||
|
|
||||||
|
# Redémarrer
|
||||||
|
pm2 restart video-transcriptor
|
||||||
|
|
||||||
|
# Ou avec Docker
|
||||||
|
docker-compose down
|
||||||
|
docker-compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Support et ressources
|
||||||
|
|
||||||
|
- **Documentation API**: [docs/API.md](./API.md)
|
||||||
|
- **CLAUDE.md**: [CLAUDE.md](../CLAUDE.md) - Instructions pour Claude
|
||||||
|
- **PM2 Documentation**: https://pm2.keymetrics.io/
|
||||||
|
- **Nginx Documentation**: https://nginx.org/en/docs/
|
||||||
|
- **Let's Encrypt**: https://letsencrypt.org/
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Bon déploiement ! 🚀**
|
||||||
132
docs/YOUTUBE_COOKIES.md
Normal file
132
docs/YOUTUBE_COOKIES.md
Normal file
@ -0,0 +1,132 @@
|
|||||||
|
# YouTube Cookies Setup Guide
|
||||||
|
|
||||||
|
## Why Do I Need Cookies?
|
||||||
|
|
||||||
|
YouTube has anti-bot protections that may block yt-dlp requests. Using cookies from your browser allows yt-dlp to authenticate as if you're logged in, bypassing these restrictions.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Option 1: Automatic Extraction (Recommended)
|
||||||
|
|
||||||
|
Run the helper script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/extract-cookies.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Follow the prompts to extract cookies from Chrome or Firefox.
|
||||||
|
|
||||||
|
### Option 2: Using yt-dlp Directly
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# For Chrome/Chromium
|
||||||
|
yt-dlp --cookies-from-browser chrome --cookies youtube-cookies.txt 'https://www.youtube.com'
|
||||||
|
|
||||||
|
# For Firefox
|
||||||
|
yt-dlp --cookies-from-browser firefox --cookies youtube-cookies.txt 'https://www.youtube.com'
|
||||||
|
|
||||||
|
# For Edge
|
||||||
|
yt-dlp --cookies-from-browser edge --cookies youtube-cookies.txt 'https://www.youtube.com'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 3: Browser Extension
|
||||||
|
|
||||||
|
1. Install a cookies export extension:
|
||||||
|
- **Chrome/Edge**: [Get cookies.txt LOCALLY](https://chrome.google.com/webstore/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)
|
||||||
|
- **Firefox**: [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/)
|
||||||
|
|
||||||
|
2. Go to [youtube.com](https://www.youtube.com) and log in
|
||||||
|
|
||||||
|
3. Click the extension icon and export cookies
|
||||||
|
|
||||||
|
4. Save the file as `youtube-cookies.txt` in your project directory
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
After extracting cookies, update your `.env` file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
YOUTUBE_COOKIES_PATH=/home/debian/videotomp3transcriptor/youtube-cookies.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use a relative path:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
YOUTUBE_COOKIES_PATH=./youtube-cookies.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verifying It Works
|
||||||
|
|
||||||
|
Test with a video:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3001/download \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
If it works without cookies errors, you're good to go!
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
|
||||||
|
⚠️ **IMPORTANT**:
|
||||||
|
|
||||||
|
1. **Never commit cookies to git**: The `.gitignore` file should already exclude `youtube-cookies.txt`
|
||||||
|
2. **Keep cookies secure**: They provide access to your YouTube account
|
||||||
|
3. **Cookies expire**: You may need to re-export them periodically (typically every few weeks/months)
|
||||||
|
4. **Don't share cookies**: Treat them like passwords
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "Sign in to confirm you're not a bot"
|
||||||
|
|
||||||
|
This usually means:
|
||||||
|
- Cookies are not being used
|
||||||
|
- Cookies have expired
|
||||||
|
- Cookies file path is incorrect
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Check the path in `.env` is correct and absolute
|
||||||
|
2. Re-export fresh cookies
|
||||||
|
3. Verify the cookies file exists: `ls -la youtube-cookies.txt`
|
||||||
|
4. Check logs: `pm2 logs toMP3-api`
|
||||||
|
|
||||||
|
### "HTTP Error 403: Forbidden"
|
||||||
|
|
||||||
|
YouTube is blocking your IP or the video is region-restricted.
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Try with fresh cookies
|
||||||
|
2. Use a VPN if region-restricted
|
||||||
|
3. Wait a bit if rate-limited
|
||||||
|
|
||||||
|
### Cookies Not Working
|
||||||
|
|
||||||
|
1. Make sure you're logged into YouTube in the browser before extracting
|
||||||
|
2. Try extracting from a different browser
|
||||||
|
3. Verify the cookies file format (should be Netscape format)
|
||||||
|
4. Check file permissions: `chmod 600 youtube-cookies.txt`
|
||||||
|
|
||||||
|
## Cookie File Format
|
||||||
|
|
||||||
|
The cookies file should be in Netscape format and look like this:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Netscape HTTP Cookie File
|
||||||
|
.youtube.com TRUE / TRUE 1234567890 CONSENT YES+
|
||||||
|
.youtube.com TRUE / FALSE 1234567890 VISITOR_INFO1_LIVE xxxxx
|
||||||
|
```
|
||||||
|
|
||||||
|
## Without Cookies
|
||||||
|
|
||||||
|
The API will still work for many videos without cookies, but you may encounter:
|
||||||
|
- "Sign in to confirm you're not a bot" errors
|
||||||
|
- Rate limiting
|
||||||
|
- Blocked downloads for certain videos
|
||||||
|
|
||||||
|
For best results, always use cookies!
|
||||||
|
|
||||||
|
## Additional Resources
|
||||||
|
|
||||||
|
- [yt-dlp Cookie Documentation](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp)
|
||||||
|
- [Browser Cookie Extraction](https://github.com/yt-dlp/yt-dlp#:~:text=You%20can%20use%20cookies%20from%20your%20browser)
|
||||||
1045
package-lock.json
generated
1045
package-lock.json
generated
File diff suppressed because it is too large
Load Diff
37
package.json
37
package.json
@ -1,27 +1,34 @@
|
|||||||
{
|
{
|
||||||
"name": "hanasuba-music-service",
|
"name": "video-to-mp3-transcriptor",
|
||||||
"version": "2.0.0",
|
"version": "1.0.0",
|
||||||
"description": "YouTube to MP3 download service with Camoufox stealth cookies for Hanasuba",
|
"description": "Download YouTube videos/playlists to MP3 and transcribe them using OpenAI Whisper API",
|
||||||
"main": "src/server.js",
|
"main": "src/index.js",
|
||||||
|
"type": "module",
|
||||||
|
"bin": {
|
||||||
|
"ytmp3": "./src/cli.js"
|
||||||
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"start": "node src/server.js",
|
"start": "node src/index.js",
|
||||||
"dev": "node --watch src/server.js",
|
"cli": "node src/cli.js",
|
||||||
"setup": "python3 -m pip install -r requirements.txt && playwright install firefox",
|
"server": "node src/server.js"
|
||||||
"cookies:extract": "python3 src/python/extract_cookies.py",
|
|
||||||
"cookies:validate": "python3 src/python/validate_cookies.py"
|
|
||||||
},
|
},
|
||||||
"keywords": [
|
"keywords": [
|
||||||
"youtube",
|
"youtube",
|
||||||
"mp3",
|
"mp3",
|
||||||
"music",
|
"transcription",
|
||||||
"camoufox",
|
"whisper",
|
||||||
"stealth",
|
"openai"
|
||||||
"hanasuba"
|
|
||||||
],
|
],
|
||||||
"author": "StillHammer",
|
"author": "",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
|
"@anthropic-ai/sdk": "^0.70.1",
|
||||||
|
"commander": "^12.1.0",
|
||||||
|
"cors": "^2.8.5",
|
||||||
"dotenv": "^16.4.5",
|
"dotenv": "^16.4.5",
|
||||||
"express": "^4.22.1"
|
"express": "^4.21.0",
|
||||||
|
"multer": "^2.0.2",
|
||||||
|
"openai": "^4.67.0",
|
||||||
|
"youtube-dl-exec": "^3.0.7"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,3 +0,0 @@
|
|||||||
camoufox>=0.4.11
|
|
||||||
camoufox-captcha>=0.1.3
|
|
||||||
playwright>=1.57.0
|
|
||||||
98
setup.sh
98
setup.sh
@ -1,98 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
#
|
|
||||||
# Hanasuba Music Service v2.0 - Setup Script
|
|
||||||
#
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
echo "╔══════════════════════════════════════════════════╗"
|
|
||||||
echo "║ 🎵 Hanasuba Music Service v2.0 Setup ║"
|
|
||||||
echo "╚══════════════════════════════════════════════════╝"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Check prerequisites
|
|
||||||
echo "🔍 Checking prerequisites..."
|
|
||||||
|
|
||||||
if ! command -v node &> /dev/null; then
|
|
||||||
echo "❌ Node.js not found. Please install Node.js 18+"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! command -v python3 &> /dev/null; then
|
|
||||||
echo "❌ Python 3 not found. Please install Python 3.9+"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! command -v yt-dlp &> /dev/null; then
|
|
||||||
echo "⚠️ yt-dlp not found. Installing..."
|
|
||||||
pip3 install yt-dlp
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! command -v ffmpeg &> /dev/null; then
|
|
||||||
echo "❌ ffmpeg not found. Please install ffmpeg"
|
|
||||||
echo " sudo apt install ffmpeg # Debian/Ubuntu"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "✅ Prerequisites OK"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Install Node dependencies
|
|
||||||
echo "📦 Installing Node.js dependencies..."
|
|
||||||
npm install
|
|
||||||
echo "✅ Node.js dependencies installed"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Install Python dependencies
|
|
||||||
echo "🐍 Installing Python dependencies..."
|
|
||||||
pip3 install -r requirements.txt
|
|
||||||
echo "✅ Python dependencies installed"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Install Playwright browsers
|
|
||||||
echo "🎭 Installing Playwright Firefox..."
|
|
||||||
python3 -m playwright install firefox
|
|
||||||
echo "✅ Playwright Firefox installed"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Create output directory
|
|
||||||
echo "📁 Creating output directory..."
|
|
||||||
mkdir -p output
|
|
||||||
chmod 755 output
|
|
||||||
echo "✅ Output directory created"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Setup .env if not exists
|
|
||||||
if [ ! -f .env ]; then
|
|
||||||
echo "⚙️ Creating .env file..."
|
|
||||||
cp .env.example .env
|
|
||||||
echo "✅ .env created (please edit if needed)"
|
|
||||||
else
|
|
||||||
echo "ℹ️ .env already exists (skipping)"
|
|
||||||
fi
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Extract initial cookies
|
|
||||||
echo "🍪 Extracting YouTube cookies (this may take 30s)..."
|
|
||||||
if python3 src/python/extract_cookies.py; then
|
|
||||||
echo "✅ Cookies extracted successfully"
|
|
||||||
else
|
|
||||||
echo "⚠️ Cookie extraction failed (you can retry later with: npm run cookies:extract)"
|
|
||||||
fi
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
echo "╔══════════════════════════════════════════════════╗"
|
|
||||||
echo "║ ✅ Setup Complete! ║"
|
|
||||||
echo "╚══════════════════════════════════════════════════╝"
|
|
||||||
echo ""
|
|
||||||
echo "🚀 Start the service:"
|
|
||||||
echo " npm start"
|
|
||||||
echo ""
|
|
||||||
echo "📖 Read the docs:"
|
|
||||||
echo " cat README.md"
|
|
||||||
echo ""
|
|
||||||
echo "🧪 Test download:"
|
|
||||||
echo " curl -X POST http://localhost:8889/download \\"
|
|
||||||
echo " -H 'Content-Type: application/json' \\"
|
|
||||||
echo " -d '{\"url\": \"https://youtube.com/watch?v=dQw4w9WgXcQ\"}'"
|
|
||||||
echo ""
|
|
||||||
169
src/cli.js
Normal file
169
src/cli.js
Normal file
@ -0,0 +1,169 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
import { Command } from 'commander';
|
||||||
|
import dotenv from 'dotenv';
|
||||||
|
import path from 'path';
|
||||||
|
import { download, downloadVideo, downloadPlaylist, getInfo } from './services/youtube.js';
|
||||||
|
import { transcribeFile, transcribeAndSave, transcribeMultiple, getAvailableModels } from './services/transcription.js';
|
||||||
|
|
||||||
|
// Load environment variables
|
||||||
|
dotenv.config();
|
||||||
|
|
||||||
|
const program = new Command();
|
||||||
|
|
||||||
|
program
|
||||||
|
.name('ytmp3')
|
||||||
|
.description('Download YouTube videos/playlists to MP3 and transcribe them')
|
||||||
|
.version('1.0.0');
|
||||||
|
|
||||||
|
// Download command
|
||||||
|
program
|
||||||
|
.command('download <url>')
|
||||||
|
.alias('dl')
|
||||||
|
.description('Download a YouTube video or playlist as MP3')
|
||||||
|
.option('-o, --output <dir>', 'Output directory', './output')
|
||||||
|
.action(async (url, options) => {
|
||||||
|
try {
|
||||||
|
console.log('Fetching video info...');
|
||||||
|
const result = await download(url, { outputDir: options.output });
|
||||||
|
|
||||||
|
console.log('\n--- Download Complete ---');
|
||||||
|
if (result.playlistTitle) {
|
||||||
|
console.log(`Playlist: ${result.playlistTitle}`);
|
||||||
|
}
|
||||||
|
console.log(`Downloaded: ${result.successCount}/${result.totalVideos} videos`);
|
||||||
|
|
||||||
|
result.videos.forEach(v => {
|
||||||
|
if (v.success) {
|
||||||
|
console.log(` ✓ ${v.title}`);
|
||||||
|
} else {
|
||||||
|
console.log(` ✗ ${v.title} - ${v.error}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Error: ${error.message}`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Transcribe command (from existing MP3)
|
||||||
|
program
|
||||||
|
.command('transcribe <file>')
|
||||||
|
.alias('tr')
|
||||||
|
.description('Transcribe an existing audio file')
|
||||||
|
.option('-l, --language <lang>', 'Language code (e.g., en, fr, zh)')
|
||||||
|
.option('-f, --format <format>', 'Output format (txt, srt, vtt)', 'txt')
|
||||||
|
.option('-m, --model <model>', 'Transcription model (gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1)', 'gpt-4o-transcribe')
|
||||||
|
.action(async (file, options) => {
|
||||||
|
try {
|
||||||
|
if (!process.env.OPENAI_API_KEY) {
|
||||||
|
console.error('Error: OPENAI_API_KEY not set in environment');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`Transcribing: ${file}`);
|
||||||
|
const result = await transcribeAndSave(file, {
|
||||||
|
language: options.language,
|
||||||
|
responseFormat: options.format === 'txt' ? 'text' : options.format,
|
||||||
|
outputFormat: options.format,
|
||||||
|
model: options.model,
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log('\n--- Transcription Complete ---');
|
||||||
|
console.log(`Model: ${result.model}`);
|
||||||
|
console.log(`Output: ${result.transcriptionPath}`);
|
||||||
|
console.log('\nPreview:');
|
||||||
|
console.log(result.text.substring(0, 500) + (result.text.length > 500 ? '...' : ''));
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Error: ${error.message}`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Download + Transcribe command
|
||||||
|
program
|
||||||
|
.command('process <url>')
|
||||||
|
.alias('p')
|
||||||
|
.description('Download and transcribe a YouTube video or playlist')
|
||||||
|
.option('-o, --output <dir>', 'Output directory', './output')
|
||||||
|
.option('-l, --language <lang>', 'Language code for transcription')
|
||||||
|
.option('-f, --format <format>', 'Transcription format (txt, srt, vtt)', 'txt')
|
||||||
|
.option('-m, --model <model>', 'Transcription model (gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1)', 'gpt-4o-transcribe')
|
||||||
|
.action(async (url, options) => {
|
||||||
|
try {
|
||||||
|
if (!process.env.OPENAI_API_KEY) {
|
||||||
|
console.error('Error: OPENAI_API_KEY not set in environment');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 1: Download
|
||||||
|
console.log('Step 1: Downloading...');
|
||||||
|
const downloadResult = await download(url, { outputDir: options.output });
|
||||||
|
|
||||||
|
console.log(`Downloaded: ${downloadResult.successCount}/${downloadResult.totalVideos} videos\n`);
|
||||||
|
|
||||||
|
// Step 2: Transcribe
|
||||||
|
console.log(`Step 2: Transcribing with ${options.model}...`);
|
||||||
|
const successfulDownloads = downloadResult.videos.filter(v => v.success);
|
||||||
|
const filePaths = successfulDownloads.map(v => v.filePath);
|
||||||
|
|
||||||
|
const transcribeResult = await transcribeMultiple(filePaths, {
|
||||||
|
language: options.language,
|
||||||
|
responseFormat: options.format === 'txt' ? 'text' : options.format,
|
||||||
|
outputFormat: options.format,
|
||||||
|
model: options.model,
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log('\n--- Process Complete ---');
|
||||||
|
if (downloadResult.playlistTitle) {
|
||||||
|
console.log(`Playlist: ${downloadResult.playlistTitle}`);
|
||||||
|
}
|
||||||
|
console.log(`Downloaded: ${downloadResult.successCount}/${downloadResult.totalVideos}`);
|
||||||
|
console.log(`Transcribed: ${transcribeResult.successCount}/${transcribeResult.totalFiles}`);
|
||||||
|
|
||||||
|
transcribeResult.results.forEach(r => {
|
||||||
|
if (r.success) {
|
||||||
|
console.log(` ✓ ${path.basename(r.transcriptionPath)}`);
|
||||||
|
} else {
|
||||||
|
console.log(` ✗ ${path.basename(r.filePath)} - ${r.error}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Error: ${error.message}`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Info command
|
||||||
|
program
|
||||||
|
.command('info <url>')
|
||||||
|
.description('Get info about a YouTube video or playlist')
|
||||||
|
.action(async (url) => {
|
||||||
|
try {
|
||||||
|
const info = await getInfo(url);
|
||||||
|
|
||||||
|
console.log('\n--- Video/Playlist Info ---');
|
||||||
|
console.log(`Title: ${info.title}`);
|
||||||
|
console.log(`Type: ${info._type || 'video'}`);
|
||||||
|
|
||||||
|
if (info._type === 'playlist') {
|
||||||
|
console.log(`Videos: ${info.entries?.length || 0}`);
|
||||||
|
if (info.entries) {
|
||||||
|
info.entries.slice(0, 10).forEach((e, i) => {
|
||||||
|
console.log(` ${i + 1}. ${e.title}`);
|
||||||
|
});
|
||||||
|
if (info.entries.length > 10) {
|
||||||
|
console.log(` ... and ${info.entries.length - 10} more`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log(`Duration: ${Math.floor(info.duration / 60)}:${String(info.duration % 60).padStart(2, '0')}`);
|
||||||
|
console.log(`Channel: ${info.channel}`);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Error: ${error.message}`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
program.parse();
|
||||||
@ -1,343 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Auto-login to YouTube and extract cookies using Camoufox.
|
|
||||||
Handles popups, phone verification, and other Google login steps.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
async def close_popups(page):
|
|
||||||
"""Close YouTube popups that block interactions"""
|
|
||||||
try:
|
|
||||||
# Close overlay backdrop
|
|
||||||
await page.evaluate("""
|
|
||||||
document.querySelectorAll('tp-yt-iron-overlay-backdrop').forEach(el => el.remove());
|
|
||||||
document.querySelectorAll('[role="dialog"]').forEach(el => {
|
|
||||||
if (el.innerText.includes('cookies') || el.innerText.includes('privacy')) {
|
|
||||||
el.style.display = 'none';
|
|
||||||
}
|
|
||||||
});
|
|
||||||
""")
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
async def auto_login_extract_cookies(
|
|
||||||
email=None,
|
|
||||||
password=None,
|
|
||||||
phone=None,
|
|
||||||
output_path='youtube-cookies.txt',
|
|
||||||
headless=False
|
|
||||||
):
|
|
||||||
"""
|
|
||||||
Login to YouTube with provided credentials and extract cookies.
|
|
||||||
Handles phone verification if needed.
|
|
||||||
"""
|
|
||||||
|
|
||||||
print("🦊 Starting Camoufox for YouTube login...")
|
|
||||||
print(f" Email: {email}")
|
|
||||||
print(f" Phone: {phone if phone else 'Not provided'}")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
if not email or not password:
|
|
||||||
print("❌ Email and password required for auto-login")
|
|
||||||
return False
|
|
||||||
|
|
||||||
async with AsyncCamoufox(
|
|
||||||
headless=headless,
|
|
||||||
humanize=True,
|
|
||||||
geoip=True,
|
|
||||||
) as browser:
|
|
||||||
page = await browser.new_page()
|
|
||||||
|
|
||||||
# Set extra headers
|
|
||||||
await page.set_extra_http_headers({
|
|
||||||
'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8'
|
|
||||||
})
|
|
||||||
|
|
||||||
print("📺 Loading YouTube...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=60000)
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Close any popups
|
|
||||||
await close_popups(page)
|
|
||||||
|
|
||||||
print("🔐 Starting login process...")
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Step 1: Click Sign In
|
|
||||||
print(" → Clicking Sign In button...")
|
|
||||||
|
|
||||||
# Try multiple selectors
|
|
||||||
signin_selectors = [
|
|
||||||
'a[aria-label*="Sign in"]',
|
|
||||||
'a:has-text("Sign in")',
|
|
||||||
'ytd-button-renderer a[href*="accounts.google.com"]',
|
|
||||||
'#buttons ytd-button-renderer a'
|
|
||||||
]
|
|
||||||
|
|
||||||
signin_clicked = False
|
|
||||||
for selector in signin_selectors:
|
|
||||||
try:
|
|
||||||
await close_popups(page)
|
|
||||||
button = await page.wait_for_selector(selector, timeout=5000)
|
|
||||||
if button:
|
|
||||||
await button.click()
|
|
||||||
signin_clicked = True
|
|
||||||
print(" ✓ Sign in clicked")
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
if not signin_clicked:
|
|
||||||
# Try direct navigation
|
|
||||||
print(" → Navigating directly to Google login...")
|
|
||||||
await page.goto('https://accounts.google.com/ServiceLogin?service=youtube', wait_until='domcontentloaded')
|
|
||||||
|
|
||||||
await asyncio.sleep(4)
|
|
||||||
|
|
||||||
# Step 2: Enter email
|
|
||||||
print(" → Entering email...")
|
|
||||||
email_selectors = [
|
|
||||||
'input[type="email"]',
|
|
||||||
'input[name="identifier"]',
|
|
||||||
'#identifierId'
|
|
||||||
]
|
|
||||||
|
|
||||||
email_entered = False
|
|
||||||
for selector in email_selectors:
|
|
||||||
try:
|
|
||||||
email_input = await page.wait_for_selector(selector, timeout=5000)
|
|
||||||
if email_input:
|
|
||||||
await email_input.click()
|
|
||||||
await asyncio.sleep(0.5)
|
|
||||||
await email_input.type(email, delay=120)
|
|
||||||
email_entered = True
|
|
||||||
print(" ✓ Email entered")
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
if not email_entered:
|
|
||||||
raise Exception("Could not find email input")
|
|
||||||
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Step 3: Click Next (email)
|
|
||||||
print(" → Clicking Next (email)...")
|
|
||||||
next_selectors = [
|
|
||||||
'#identifierNext button',
|
|
||||||
'button:has-text("Next")',
|
|
||||||
'[data-test-id="nextButton"]',
|
|
||||||
'button[type="button"]'
|
|
||||||
]
|
|
||||||
|
|
||||||
for selector in next_selectors:
|
|
||||||
try:
|
|
||||||
next_btn = await page.wait_for_selector(selector, timeout=3000)
|
|
||||||
if next_btn:
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ Next clicked")
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
# Check for phone verification
|
|
||||||
page_content = await page.content()
|
|
||||||
if 'phone' in page_content.lower() or 'verify' in page_content.lower():
|
|
||||||
print(" ⚠️ Phone verification detected")
|
|
||||||
|
|
||||||
if phone:
|
|
||||||
print(f" → Entering phone: {phone}")
|
|
||||||
phone_selectors = [
|
|
||||||
'input[type="tel"]',
|
|
||||||
'input[name="phoneNumber"]',
|
|
||||||
'#phoneNumberId'
|
|
||||||
]
|
|
||||||
|
|
||||||
for selector in phone_selectors:
|
|
||||||
try:
|
|
||||||
phone_input = await page.wait_for_selector(selector, timeout=3000)
|
|
||||||
if phone_input:
|
|
||||||
await phone_input.click()
|
|
||||||
await asyncio.sleep(0.5)
|
|
||||||
await phone_input.type(phone, delay=100)
|
|
||||||
print(" ✓ Phone entered")
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
next_btn = await page.wait_for_selector('button:has-text("Next")', timeout=3000)
|
|
||||||
await next_btn.click()
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Wait for SMS code input
|
|
||||||
print(" ⏸️ MANUAL STEP REQUIRED:")
|
|
||||||
print(" Check your phone for SMS code")
|
|
||||||
print(" Enter the code on the screen")
|
|
||||||
input(" Press Enter when done...")
|
|
||||||
await asyncio.sleep(2)
|
|
||||||
else:
|
|
||||||
print(" ⏸️ Phone verification required but no phone provided")
|
|
||||||
print(" Please complete verification manually")
|
|
||||||
input(" Press Enter when done...")
|
|
||||||
await asyncio.sleep(2)
|
|
||||||
|
|
||||||
# Step 4: Enter password
|
|
||||||
print(" → Entering password...")
|
|
||||||
password_selectors = [
|
|
||||||
'input[type="password"]',
|
|
||||||
'input[name="password"]',
|
|
||||||
'#password input'
|
|
||||||
]
|
|
||||||
|
|
||||||
password_entered = False
|
|
||||||
for selector in password_selectors:
|
|
||||||
try:
|
|
||||||
password_input = await page.wait_for_selector(selector, timeout=8000)
|
|
||||||
if password_input:
|
|
||||||
await password_input.click()
|
|
||||||
await asyncio.sleep(0.5)
|
|
||||||
await password_input.type(password, delay=120)
|
|
||||||
password_entered = True
|
|
||||||
print(" ✓ Password entered")
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
if not password_entered:
|
|
||||||
raise Exception("Could not find password input")
|
|
||||||
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Step 5: Click Next (password)
|
|
||||||
print(" → Clicking Next (password)...")
|
|
||||||
next_password_selectors = [
|
|
||||||
'#passwordNext button',
|
|
||||||
'button:has-text("Next")',
|
|
||||||
'button[type="button"]'
|
|
||||||
]
|
|
||||||
|
|
||||||
for selector in next_password_selectors:
|
|
||||||
try:
|
|
||||||
next_btn = await page.wait_for_selector(selector, timeout=3000)
|
|
||||||
if next_btn:
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ Next clicked")
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
await asyncio.sleep(8)
|
|
||||||
|
|
||||||
# Check if we're back on YouTube
|
|
||||||
current_url = page.url
|
|
||||||
if 'youtube.com' in current_url:
|
|
||||||
print("✅ Login successful!")
|
|
||||||
else:
|
|
||||||
print("⚠️ May need additional verification")
|
|
||||||
print(f" Current URL: {current_url}")
|
|
||||||
|
|
||||||
# Check for additional steps
|
|
||||||
page_text = await page.content()
|
|
||||||
if 'recovery' in page_text.lower() or 'verify' in page_text.lower():
|
|
||||||
print(" ⏸️ Additional verification needed")
|
|
||||||
print(" Please complete on screen")
|
|
||||||
input(" Press Enter when done...")
|
|
||||||
|
|
||||||
# Navigate back to YouTube
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded')
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"⚠️ Login failed: {e}")
|
|
||||||
print(" Attempting to extract cookies anyway...")
|
|
||||||
|
|
||||||
# Extract cookies
|
|
||||||
print("")
|
|
||||||
print("🍪 Extracting cookies...")
|
|
||||||
cookies = await page.context.cookies()
|
|
||||||
|
|
||||||
# Filter YouTube/Google cookies
|
|
||||||
yt_cookies = [c for c in cookies if 'youtube.com' in c['domain'] or 'google.com' in c['domain']]
|
|
||||||
|
|
||||||
if not yt_cookies:
|
|
||||||
print("❌ No cookies found!")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Check if we have logged-in cookies
|
|
||||||
cookie_names = [c['name'] for c in yt_cookies]
|
|
||||||
has_login_cookies = any(name in ['SID', 'SSID', 'HSID', 'SAPISID'] for name in cookie_names)
|
|
||||||
|
|
||||||
# Save to Netscape format
|
|
||||||
output = Path(output_path)
|
|
||||||
with open(output, 'w') as f:
|
|
||||||
f.write("# Netscape HTTP Cookie File\n")
|
|
||||||
f.write("# Generated by Camoufox with auto-login\n")
|
|
||||||
for c in yt_cookies:
|
|
||||||
# Handle expires properly
|
|
||||||
expires = int(c.get('expires', 0))
|
|
||||||
if expires <= 0:
|
|
||||||
expires = 2147483647 # Max timestamp
|
|
||||||
|
|
||||||
line = f"{c['domain']}\tTRUE\t{c['path']}\t"
|
|
||||||
line += f"{'TRUE' if c.get('secure') else 'FALSE'}\t"
|
|
||||||
line += f"{expires}\t{c['name']}\t{c['value']}\n"
|
|
||||||
f.write(line)
|
|
||||||
|
|
||||||
# Set secure permissions
|
|
||||||
output.chmod(0o600)
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print(f"✅ Cookies saved: {output_path}")
|
|
||||||
print(f" Total cookies: {len(yt_cookies)}")
|
|
||||||
print(f" Login cookies: {'Yes ✓' if has_login_cookies else 'No (guest mode)'}")
|
|
||||||
print(f" File permissions: 600 (secure)")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
if has_login_cookies:
|
|
||||||
print("💡 Logged-in cookies! These will work for 2-4 weeks!")
|
|
||||||
else:
|
|
||||||
print("⚠️ Guest mode cookies - may not work for all videos")
|
|
||||||
|
|
||||||
return has_login_cookies
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
import argparse
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(description='Extract YouTube cookies with auto-login')
|
|
||||||
parser.add_argument('--email', help='Google/YouTube email')
|
|
||||||
parser.add_argument('--password', help='Google/YouTube password')
|
|
||||||
parser.add_argument('--phone', help='Phone number for verification (format: 0695110967)')
|
|
||||||
parser.add_argument('--output', default='youtube-cookies.txt', help='Output file')
|
|
||||||
parser.add_argument('--headless', action='store_true', help='Run headless')
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
# Read from env if not provided
|
|
||||||
email = args.email or os.getenv('YOUTUBE_EMAIL')
|
|
||||||
password = args.password or os.getenv('YOUTUBE_PASSWORD')
|
|
||||||
phone = args.phone or os.getenv('YOUTUBE_PHONE')
|
|
||||||
|
|
||||||
if not email or not password:
|
|
||||||
print("❌ Error: --email and --password required")
|
|
||||||
print(" Or set YOUTUBE_EMAIL and YOUTUBE_PASSWORD env vars")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
success = asyncio.run(auto_login_extract_cookies(
|
|
||||||
email=email,
|
|
||||||
password=password,
|
|
||||||
phone=phone,
|
|
||||||
output_path=args.output,
|
|
||||||
headless=args.headless
|
|
||||||
))
|
|
||||||
|
|
||||||
sys.exit(0 if success else 1)
|
|
||||||
@ -1,230 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Fully automated YouTube login with SMS verification.
|
|
||||||
Waits for SMS code from SMS forwarder endpoint.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import requests
|
|
||||||
from pathlib import Path
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
SMS_ENDPOINT = os.getenv('SMS_ENDPOINT', 'http://localhost:4417')
|
|
||||||
|
|
||||||
def wait_for_sms_code(timeout=120):
|
|
||||||
"""Wait for SMS code from endpoint (long-polling)"""
|
|
||||||
print("📱 Waiting for SMS code...")
|
|
||||||
print(f" Endpoint: {SMS_ENDPOINT}/sms/wait")
|
|
||||||
print(f" Timeout: {timeout}s")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
try:
|
|
||||||
response = requests.get(
|
|
||||||
f"{SMS_ENDPOINT}/sms/wait",
|
|
||||||
timeout=timeout + 5
|
|
||||||
)
|
|
||||||
|
|
||||||
if response.status_code == 200:
|
|
||||||
data = response.json()
|
|
||||||
if data.get('success') and data.get('code'):
|
|
||||||
print(f"✅ SMS code received: {data['code']}")
|
|
||||||
return data['code']
|
|
||||||
|
|
||||||
print("❌ No SMS code received")
|
|
||||||
return None
|
|
||||||
|
|
||||||
except requests.exceptions.Timeout:
|
|
||||||
print("⏱️ Timeout waiting for SMS")
|
|
||||||
return None
|
|
||||||
except Exception as e:
|
|
||||||
print(f"❌ Error waiting for SMS: {e}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
async def auto_login_full_auto(
|
|
||||||
email,
|
|
||||||
password,
|
|
||||||
phone=None,
|
|
||||||
output_path='youtube-cookies.txt'
|
|
||||||
):
|
|
||||||
"""Fully automated login with SMS from endpoint"""
|
|
||||||
|
|
||||||
print("🦊 Starting Camoufox for YouTube login...")
|
|
||||||
print(f" Email: {email}")
|
|
||||||
print(f" Phone: {phone if phone else 'Not provided'}")
|
|
||||||
print(f" SMS Endpoint: {SMS_ENDPOINT}")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
async with AsyncCamoufox(
|
|
||||||
headless=True,
|
|
||||||
humanize=True,
|
|
||||||
geoip=True,
|
|
||||||
) as browser:
|
|
||||||
page = await browser.new_page()
|
|
||||||
|
|
||||||
print("📺 Loading YouTube...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=60000)
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Close popups
|
|
||||||
try:
|
|
||||||
await page.evaluate("""
|
|
||||||
document.querySelectorAll('tp-yt-iron-overlay-backdrop').forEach(el => el.remove());
|
|
||||||
""")
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
print("🔐 Starting login process...")
|
|
||||||
|
|
||||||
# Step 1: Navigate to login
|
|
||||||
print(" → Navigating to Google login...")
|
|
||||||
try:
|
|
||||||
signin = await page.wait_for_selector('a[aria-label*="Sign in"]', timeout=10000)
|
|
||||||
await signin.click()
|
|
||||||
await asyncio.sleep(4)
|
|
||||||
except:
|
|
||||||
await page.goto('https://accounts.google.com/ServiceLogin?service=youtube')
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Step 2: Enter email
|
|
||||||
print(" → Entering email...")
|
|
||||||
email_input = await page.wait_for_selector('input[type="email"]', timeout=10000)
|
|
||||||
await email_input.type(email, delay=120)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
next_btn = await page.wait_for_selector('#identifierNext button', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ Email submitted")
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
# Step 3: Check for phone verification
|
|
||||||
page_content = await page.content()
|
|
||||||
|
|
||||||
if phone and ('phone' in page_content.lower() or 'verify' in page_content.lower()):
|
|
||||||
print(" ⚠️ Phone verification detected")
|
|
||||||
print(f" → Entering phone: {phone}")
|
|
||||||
|
|
||||||
try:
|
|
||||||
phone_input = await page.wait_for_selector('input[type="tel"]', timeout=5000)
|
|
||||||
await phone_input.type(phone, delay=100)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
next_btn = await page.wait_for_selector('button:has-text("Next")', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ Phone submitted")
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
# Wait for SMS code from endpoint
|
|
||||||
print("")
|
|
||||||
print("=" * 60)
|
|
||||||
print("📱 SMS CODE REQUIRED")
|
|
||||||
print("=" * 60)
|
|
||||||
print("Waiting for SMS to be forwarded from your phone...")
|
|
||||||
print("Make sure SMS Forwarder app is configured!")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
sms_code = wait_for_sms_code(timeout=120)
|
|
||||||
|
|
||||||
if not sms_code:
|
|
||||||
print("❌ No SMS code received - aborting")
|
|
||||||
return False
|
|
||||||
|
|
||||||
print(f" → Entering SMS code: {sms_code}")
|
|
||||||
|
|
||||||
# Enter SMS code
|
|
||||||
sms_input = await page.wait_for_selector('input[type="tel"]', timeout=5000)
|
|
||||||
await sms_input.type(sms_code, delay=100)
|
|
||||||
await asyncio.sleep(2)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
next_btn = await page.wait_for_selector('button:has-text("Next")', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ SMS code submitted")
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f" ⚠️ Phone verification error: {e}")
|
|
||||||
|
|
||||||
# Step 4: Enter password
|
|
||||||
print(" → Entering password...")
|
|
||||||
password_input = await page.wait_for_selector('input[type="password"]', timeout=10000)
|
|
||||||
await password_input.type(password, delay=120)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
next_btn = await page.wait_for_selector('#passwordNext button', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ Password submitted")
|
|
||||||
await asyncio.sleep(8)
|
|
||||||
|
|
||||||
# Navigate to YouTube to confirm
|
|
||||||
print(" → Confirming login...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded')
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Extract cookies
|
|
||||||
print("")
|
|
||||||
print("🍪 Extracting cookies...")
|
|
||||||
cookies = await page.context.cookies()
|
|
||||||
|
|
||||||
yt_cookies = [c for c in cookies if 'youtube.com' in c['domain'] or 'google.com' in c['domain']]
|
|
||||||
|
|
||||||
if not yt_cookies:
|
|
||||||
print("❌ No cookies found!")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Check for logged-in cookies
|
|
||||||
cookie_names = [c['name'] for c in yt_cookies]
|
|
||||||
has_login = any(name in ['SID', 'SSID', 'HSID', 'SAPISID', '__Secure-1PSID'] for name in cookie_names)
|
|
||||||
|
|
||||||
# Save cookies
|
|
||||||
output = Path(output_path)
|
|
||||||
with open(output, 'w') as f:
|
|
||||||
f.write("# Netscape HTTP Cookie File\n")
|
|
||||||
f.write("# Generated by Camoufox - Full auto SMS login\n")
|
|
||||||
for c in yt_cookies:
|
|
||||||
expires = int(c.get('expires', 0))
|
|
||||||
if expires <= 0:
|
|
||||||
expires = 2147483647
|
|
||||||
|
|
||||||
line = f"{c['domain']}\tTRUE\t{c['path']}\t"
|
|
||||||
line += f"{'TRUE' if c.get('secure') else 'FALSE'}\t"
|
|
||||||
line += f"{expires}\t{c['name']}\t{c['value']}\n"
|
|
||||||
f.write(line)
|
|
||||||
|
|
||||||
output.chmod(0o600)
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print("=" * 60)
|
|
||||||
print(f"✅ Cookies saved: {output_path}")
|
|
||||||
print(f" Total cookies: {len(yt_cookies)}")
|
|
||||||
print(f" Login cookies: {'Yes ✓' if has_login else 'No'}")
|
|
||||||
print(f" Permissions: 600 (secure)")
|
|
||||||
print("=" * 60)
|
|
||||||
print("")
|
|
||||||
|
|
||||||
if has_login:
|
|
||||||
print("🎉 SUCCESS! Fully logged-in cookies extracted!")
|
|
||||||
print(" These will work for 2-4 weeks!")
|
|
||||||
print("")
|
|
||||||
print("💡 Next: Test with yt-dlp + PO Token")
|
|
||||||
else:
|
|
||||||
print("⚠️ Warning: May be guest cookies")
|
|
||||||
|
|
||||||
return has_login
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
email = "alextingtingqishi@gmail.com"
|
|
||||||
password = "12345678@stt"
|
|
||||||
phone = "0695110967"
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print("=" * 60)
|
|
||||||
print("YouTube Auto-Login with SMS Forwarding")
|
|
||||||
print("=" * 60)
|
|
||||||
print("")
|
|
||||||
|
|
||||||
success = asyncio.run(auto_login_full_auto(email, password, phone))
|
|
||||||
sys.exit(0 if success else 1)
|
|
||||||
@ -1,195 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Auto-login to YouTube with SMS code input support.
|
|
||||||
Takes screenshots and prompts for SMS code via terminal.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
async def auto_login_with_sms(
|
|
||||||
email,
|
|
||||||
password,
|
|
||||||
phone=None,
|
|
||||||
output_path='youtube-cookies.txt'
|
|
||||||
):
|
|
||||||
"""Login to YouTube with SMS verification support"""
|
|
||||||
|
|
||||||
print("🦊 Starting Camoufox for YouTube login...")
|
|
||||||
print(f" Email: {email}")
|
|
||||||
print(f" Phone: {phone if phone else 'Not provided'}")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
async with AsyncCamoufox(
|
|
||||||
headless=True, # Headless but we'll take screenshots
|
|
||||||
humanize=True,
|
|
||||||
geoip=True,
|
|
||||||
) as browser:
|
|
||||||
page = await browser.new_page()
|
|
||||||
|
|
||||||
print("📺 Loading YouTube...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=60000)
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Close popups
|
|
||||||
try:
|
|
||||||
await page.evaluate("""
|
|
||||||
document.querySelectorAll('tp-yt-iron-overlay-backdrop').forEach(el => el.remove());
|
|
||||||
""")
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
print("🔐 Starting login process...")
|
|
||||||
|
|
||||||
# Step 1: Click Sign In
|
|
||||||
print(" → Clicking Sign In...")
|
|
||||||
try:
|
|
||||||
signin = await page.wait_for_selector('a[aria-label*="Sign in"]', timeout=10000)
|
|
||||||
await signin.click()
|
|
||||||
await asyncio.sleep(4)
|
|
||||||
except:
|
|
||||||
# Direct navigation fallback
|
|
||||||
await page.goto('https://accounts.google.com/ServiceLogin?service=youtube')
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Step 2: Enter email
|
|
||||||
print(" → Entering email...")
|
|
||||||
email_input = await page.wait_for_selector('input[type="email"]', timeout=10000)
|
|
||||||
await email_input.type(email, delay=120)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
next_btn = await page.wait_for_selector('#identifierNext button', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
# Check for phone verification
|
|
||||||
page_content = await page.content()
|
|
||||||
|
|
||||||
if phone and ('phone' in page_content.lower() or 'verify' in page_content.lower()):
|
|
||||||
print(" ⚠️ Phone verification detected")
|
|
||||||
print(f" → Entering phone: {phone}")
|
|
||||||
|
|
||||||
phone_input = await page.wait_for_selector('input[type="tel"]', timeout=5000)
|
|
||||||
await phone_input.type(phone, delay=100)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
next_btn = await page.wait_for_selector('button:has-text("Next")', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
# Take screenshot of SMS code input
|
|
||||||
print("")
|
|
||||||
print("📸 Taking screenshot...")
|
|
||||||
screenshot_path = '/tmp/google_sms_screen.png'
|
|
||||||
await page.screenshot(path=screenshot_path, full_page=True)
|
|
||||||
print(f" Screenshot saved: {screenshot_path}")
|
|
||||||
print("")
|
|
||||||
print("📱 CHECK YOUR PHONE FOR SMS CODE!")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
# Prompt for SMS code
|
|
||||||
sms_code = input(" Enter the 6-digit SMS code: ").strip()
|
|
||||||
|
|
||||||
if sms_code:
|
|
||||||
print(f" → Entering SMS code: {sms_code}")
|
|
||||||
|
|
||||||
# Find SMS code input field
|
|
||||||
sms_selectors = [
|
|
||||||
'input[type="tel"]',
|
|
||||||
'input[name="pin"]',
|
|
||||||
'input[aria-label*="code"]',
|
|
||||||
'input[type="text"]'
|
|
||||||
]
|
|
||||||
|
|
||||||
for selector in sms_selectors:
|
|
||||||
try:
|
|
||||||
sms_input = await page.wait_for_selector(selector, timeout=3000)
|
|
||||||
await sms_input.type(sms_code, delay=100)
|
|
||||||
print(" ✓ SMS code entered")
|
|
||||||
break
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
|
|
||||||
await asyncio.sleep(2)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
try:
|
|
||||||
next_btn = await page.wait_for_selector('button:has-text("Next")', timeout=3000)
|
|
||||||
await next_btn.click()
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Step 3: Enter password
|
|
||||||
print(" → Entering password...")
|
|
||||||
password_input = await page.wait_for_selector('input[type="password"]', timeout=10000)
|
|
||||||
await password_input.type(password, delay=120)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
next_btn = await page.wait_for_selector('#passwordNext button', timeout=5000)
|
|
||||||
await next_btn.click()
|
|
||||||
await asyncio.sleep(8)
|
|
||||||
|
|
||||||
# Navigate to YouTube to confirm login
|
|
||||||
print(" → Confirming login...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded')
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Extract cookies
|
|
||||||
print("")
|
|
||||||
print("🍪 Extracting cookies...")
|
|
||||||
cookies = await page.context.cookies()
|
|
||||||
|
|
||||||
yt_cookies = [c for c in cookies if 'youtube.com' in c['domain'] or 'google.com' in c['domain']]
|
|
||||||
|
|
||||||
if not yt_cookies:
|
|
||||||
print("❌ No cookies found!")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Check for logged-in cookies
|
|
||||||
cookie_names = [c['name'] for c in yt_cookies]
|
|
||||||
has_login = any(name in ['SID', 'SSID', 'HSID', 'SAPISID', '__Secure-1PSID'] for name in cookie_names)
|
|
||||||
|
|
||||||
# Save cookies
|
|
||||||
output = Path(output_path)
|
|
||||||
with open(output, 'w') as f:
|
|
||||||
f.write("# Netscape HTTP Cookie File\n")
|
|
||||||
f.write("# Generated by Camoufox with SMS login\n")
|
|
||||||
for c in yt_cookies:
|
|
||||||
expires = int(c.get('expires', 0))
|
|
||||||
if expires <= 0:
|
|
||||||
expires = 2147483647
|
|
||||||
|
|
||||||
line = f"{c['domain']}\tTRUE\t{c['path']}\t"
|
|
||||||
line += f"{'TRUE' if c.get('secure') else 'FALSE'}\t"
|
|
||||||
line += f"{expires}\t{c['name']}\t{c['value']}\n"
|
|
||||||
f.write(line)
|
|
||||||
|
|
||||||
output.chmod(0o600)
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print(f"✅ Cookies saved: {output_path}")
|
|
||||||
print(f" Total cookies: {len(yt_cookies)}")
|
|
||||||
print(f" Login cookies: {'Yes ✓' if has_login else 'No'}")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
if has_login:
|
|
||||||
print("🎉 SUCCESS! Logged-in cookies extracted!")
|
|
||||||
print(" These will work for 2-4 weeks!")
|
|
||||||
else:
|
|
||||||
print("⚠️ Warning: May be guest cookies")
|
|
||||||
|
|
||||||
return has_login
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
email = "alextingtingqishi@gmail.com"
|
|
||||||
password = "12345678@stt"
|
|
||||||
phone = "0695110967"
|
|
||||||
|
|
||||||
success = asyncio.run(auto_login_with_sms(email, password, phone))
|
|
||||||
sys.exit(0 if success else 1)
|
|
||||||
@ -1,73 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Extract YouTube cookies using Camoufox (stealth Firefox).
|
|
||||||
Cookies are undetectable by bot detection systems.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
async def extract_cookies(output_path='youtube-cookies.txt'):
|
|
||||||
"""
|
|
||||||
Extract YouTube cookies using Camoufox (stealth Firefox).
|
|
||||||
These cookies bypass bot detection and last longer.
|
|
||||||
"""
|
|
||||||
|
|
||||||
print("🦊 Starting Camoufox (stealth mode)...")
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with AsyncCamoufox(
|
|
||||||
headless=True, # Background (no GUI needed)
|
|
||||||
humanize=True, # Mimic human behavior
|
|
||||||
geoip=True, # Realistic IP geolocation
|
|
||||||
) as browser:
|
|
||||||
page = await browser.new_page()
|
|
||||||
|
|
||||||
# Navigate to YouTube
|
|
||||||
print("📺 Loading YouTube...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=30000)
|
|
||||||
|
|
||||||
# Wait for page fully loaded
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
# Extract cookies
|
|
||||||
cookies = await page.context.cookies()
|
|
||||||
|
|
||||||
# Filter YouTube cookies
|
|
||||||
yt_cookies = [c for c in cookies if 'youtube.com' in c['domain']]
|
|
||||||
|
|
||||||
if not yt_cookies:
|
|
||||||
print("❌ No YouTube cookies found!")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Save to Netscape format (yt-dlp compatible)
|
|
||||||
output = Path(output_path)
|
|
||||||
with open(output, 'w') as f:
|
|
||||||
f.write("# Netscape HTTP Cookie File\n")
|
|
||||||
f.write("# Generated by Camoufox (stealth mode)\n")
|
|
||||||
f.write(f"# This file is compatible with yt-dlp\n")
|
|
||||||
for c in yt_cookies:
|
|
||||||
line = f"{c['domain']}\tTRUE\t{c['path']}\t"
|
|
||||||
line += f"{'TRUE' if c.get('secure') else 'FALSE'}\t"
|
|
||||||
line += f"{int(c.get('expires', 0))}\t{c['name']}\t{c['value']}\n"
|
|
||||||
f.write(line)
|
|
||||||
|
|
||||||
# Set secure permissions
|
|
||||||
output.chmod(0o600)
|
|
||||||
|
|
||||||
print(f"✅ Cookies saved: {output_path}")
|
|
||||||
print(f" Total cookies: {len(yt_cookies)}")
|
|
||||||
print(f" Permissions: 600 (secure)")
|
|
||||||
|
|
||||||
return True
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"❌ Error: {e}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
output = sys.argv[1] if len(sys.argv) > 1 else 'youtube-cookies.txt'
|
|
||||||
success = asyncio.run(extract_cookies(output))
|
|
||||||
sys.exit(0 if success else 1)
|
|
||||||
@ -1,88 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Extract YouTube cookies with manual login using Camoufox.
|
|
||||||
Run this ONCE to login, then cookies will work for weeks.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
async def extract_cookies_with_login(output_path='youtube-cookies.txt'):
|
|
||||||
"""
|
|
||||||
Extract YouTube cookies after manual login.
|
|
||||||
Opens browser window for user to login.
|
|
||||||
"""
|
|
||||||
|
|
||||||
print("🦊 Starting Camoufox (with GUI for login)...")
|
|
||||||
print("")
|
|
||||||
print("╔══════════════════════════════════════════════════╗")
|
|
||||||
print("║ 📝 INSTRUCTIONS: ║")
|
|
||||||
print("║ 1. Browser will open ║")
|
|
||||||
print("║ 2. Login to your YouTube account ║")
|
|
||||||
print("║ 3. Press Enter in this terminal when done ║")
|
|
||||||
print("╚══════════════════════════════════════════════════╝")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
async with AsyncCamoufox(
|
|
||||||
headless=False, # GUI visible for login
|
|
||||||
humanize=True,
|
|
||||||
geoip=True,
|
|
||||||
) as browser:
|
|
||||||
page = await browser.new_page()
|
|
||||||
|
|
||||||
# Navigate to YouTube
|
|
||||||
print("📺 Loading YouTube...")
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=30000)
|
|
||||||
|
|
||||||
# Wait for user to login
|
|
||||||
input("⏸️ Press Enter after you've logged in to YouTube...")
|
|
||||||
|
|
||||||
# Navigate to confirm cookies are set
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded')
|
|
||||||
await asyncio.sleep(2)
|
|
||||||
|
|
||||||
# Extract cookies
|
|
||||||
cookies = await page.context.cookies()
|
|
||||||
|
|
||||||
# Filter YouTube cookies
|
|
||||||
yt_cookies = [c for c in cookies if 'youtube.com' in c['domain']]
|
|
||||||
|
|
||||||
if not yt_cookies:
|
|
||||||
print("❌ No YouTube cookies found!")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Save to Netscape format
|
|
||||||
output = Path(output_path)
|
|
||||||
with open(output, 'w') as f:
|
|
||||||
f.write("# Netscape HTTP Cookie File\n")
|
|
||||||
f.write("# Generated by Camoufox with logged-in account\n")
|
|
||||||
for c in yt_cookies:
|
|
||||||
# Handle expires properly
|
|
||||||
expires = int(c.get('expires', 0))
|
|
||||||
if expires <= 0:
|
|
||||||
expires = 2147483647 # Max 32-bit timestamp (year 2038)
|
|
||||||
|
|
||||||
line = f"{c['domain']}\tTRUE\t{c['path']}\t"
|
|
||||||
line += f"{'TRUE' if c.get('secure') else 'FALSE'}\t"
|
|
||||||
line += f"{expires}\t{c['name']}\t{c['value']}\n"
|
|
||||||
f.write(line)
|
|
||||||
|
|
||||||
# Set secure permissions
|
|
||||||
output.chmod(0o600)
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print(f"✅ Cookies saved: {output_path}")
|
|
||||||
print(f" Total cookies: {len(yt_cookies)}")
|
|
||||||
print(f" Logged in: Yes")
|
|
||||||
print("")
|
|
||||||
print("💡 These cookies will work for 2-4 weeks!")
|
|
||||||
print(" Run this script again when they expire.")
|
|
||||||
|
|
||||||
return True
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
output = sys.argv[1] if len(sys.argv) > 1 else 'youtube-cookies.txt'
|
|
||||||
success = asyncio.run(extract_cookies_with_login(output))
|
|
||||||
sys.exit(0 if success else 1)
|
|
||||||
@ -1,141 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Trigger Google SMS by initiating password recovery.
|
|
||||||
This ALWAYS sends an SMS to the registered phone number.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
async def trigger_google_sms(email, phone):
|
|
||||||
"""Trigger Google to send SMS via password recovery"""
|
|
||||||
|
|
||||||
print("🚀 Triggering Google SMS via password recovery...")
|
|
||||||
print(f" Email: {email}")
|
|
||||||
print(f" Phone: {phone}")
|
|
||||||
print("")
|
|
||||||
|
|
||||||
async with AsyncCamoufox(
|
|
||||||
headless=True,
|
|
||||||
humanize=True,
|
|
||||||
geoip=True,
|
|
||||||
) as browser:
|
|
||||||
page = await browser.new_page()
|
|
||||||
|
|
||||||
print("📺 Loading Google Account Recovery...")
|
|
||||||
await page.goto(
|
|
||||||
'https://accounts.google.com/signin/v2/recoveryidentifier',
|
|
||||||
wait_until='domcontentloaded',
|
|
||||||
timeout=30000
|
|
||||||
)
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
print("📧 Entering email...")
|
|
||||||
try:
|
|
||||||
# Enter email
|
|
||||||
email_input = await page.wait_for_selector(
|
|
||||||
'input[type="email"]',
|
|
||||||
timeout=10000
|
|
||||||
)
|
|
||||||
await email_input.type(email, delay=120)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Click Next
|
|
||||||
next_btn = await page.wait_for_selector(
|
|
||||||
'button:has-text("Next")',
|
|
||||||
timeout=5000
|
|
||||||
)
|
|
||||||
await next_btn.click()
|
|
||||||
print(" ✓ Email submitted")
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
|
|
||||||
# Look for "Try another way" or phone verification
|
|
||||||
page_content = await page.content()
|
|
||||||
|
|
||||||
if 'another way' in page_content.lower():
|
|
||||||
print("🔄 Clicking 'Try another way'...")
|
|
||||||
try:
|
|
||||||
another_way = await page.wait_for_selector(
|
|
||||||
'button:has-text("Try another way")',
|
|
||||||
timeout=5000
|
|
||||||
)
|
|
||||||
await another_way.click()
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Look for SMS option
|
|
||||||
if 'text message' in page_content.lower() or 'sms' in page_content.lower():
|
|
||||||
print("📱 Selecting SMS option...")
|
|
||||||
try:
|
|
||||||
sms_option = await page.wait_for_selector(
|
|
||||||
'[data-challengetype="12"], button:has-text("Text"), button:has-text("SMS")',
|
|
||||||
timeout=5000
|
|
||||||
)
|
|
||||||
await sms_option.click()
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
print(" ✓ SMS option selected")
|
|
||||||
except Exception as e:
|
|
||||||
print(f" ⚠️ Could not select SMS option: {e}")
|
|
||||||
|
|
||||||
# Enter phone if requested
|
|
||||||
page_content = await page.content()
|
|
||||||
if 'phone' in page_content.lower():
|
|
||||||
print(f"📞 Entering phone number: {phone}...")
|
|
||||||
try:
|
|
||||||
phone_input = await page.wait_for_selector(
|
|
||||||
'input[type="tel"]',
|
|
||||||
timeout=5000
|
|
||||||
)
|
|
||||||
await phone_input.type(phone, delay=100)
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
|
|
||||||
# Click Send
|
|
||||||
send_btn = await page.wait_for_selector(
|
|
||||||
'button:has-text("Send"), button:has-text("Next")',
|
|
||||||
timeout=5000
|
|
||||||
)
|
|
||||||
await send_btn.click()
|
|
||||||
print(" ✓ Phone submitted")
|
|
||||||
await asyncio.sleep(3)
|
|
||||||
|
|
||||||
print("")
|
|
||||||
print("=" * 60)
|
|
||||||
print("✅ SMS SHOULD BE SENT NOW!")
|
|
||||||
print("=" * 60)
|
|
||||||
print("📱 Check your phone (0695110967)")
|
|
||||||
print("🔄 SMS Forwarder should auto-forward to server")
|
|
||||||
print("=" * 60)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f" ⚠️ Phone entry error: {e}")
|
|
||||||
else:
|
|
||||||
# SMS might already be sent
|
|
||||||
print("")
|
|
||||||
print("=" * 60)
|
|
||||||
print("✅ SMS MIGHT ALREADY BE SENT!")
|
|
||||||
print("=" * 60)
|
|
||||||
print("📱 Check your phone (0695110967)")
|
|
||||||
print("=" * 60)
|
|
||||||
|
|
||||||
# Take screenshot for debugging
|
|
||||||
await page.screenshot(path='/tmp/google_recovery.png')
|
|
||||||
print("")
|
|
||||||
print("📸 Screenshot saved: /tmp/google_recovery.png")
|
|
||||||
|
|
||||||
# Wait a bit to see the result
|
|
||||||
await asyncio.sleep(10)
|
|
||||||
|
|
||||||
return True
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"❌ Error: {e}")
|
|
||||||
await page.screenshot(path='/tmp/google_recovery_error.png')
|
|
||||||
print("📸 Error screenshot: /tmp/google_recovery_error.png")
|
|
||||||
return False
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
email = "alextingtingqishi@gmail.com"
|
|
||||||
phone = "0695110967"
|
|
||||||
|
|
||||||
asyncio.run(trigger_google_sms(email, phone))
|
|
||||||
@ -1,85 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Validate YouTube cookies using Camoufox.
|
|
||||||
Tests if cookies are still valid and working.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
from camoufox.async_api import AsyncCamoufox
|
|
||||||
|
|
||||||
async def validate_cookies(cookies_path='youtube-cookies.txt'):
|
|
||||||
"""
|
|
||||||
Test if YouTube cookies are still valid.
|
|
||||||
Returns True if valid, False otherwise.
|
|
||||||
"""
|
|
||||||
|
|
||||||
cookies_file = Path(cookies_path)
|
|
||||||
|
|
||||||
if not cookies_file.exists():
|
|
||||||
print(f"❌ Cookies file not found: {cookies_path}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Check file age
|
|
||||||
age_hours = (Path().stat().st_mtime - cookies_file.stat().st_mtime) / 3600
|
|
||||||
print(f"📅 Cookies age: {age_hours:.1f} hours")
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with AsyncCamoufox(headless=True, humanize=True) as browser:
|
|
||||||
context = await browser.new_context()
|
|
||||||
|
|
||||||
# Load cookies from file
|
|
||||||
# Camoufox doesn't have add_cookies_from_file, so we parse manually
|
|
||||||
cookies_to_add = []
|
|
||||||
with open(cookies_path, 'r') as f:
|
|
||||||
for line in f:
|
|
||||||
if line.startswith('#') or not line.strip():
|
|
||||||
continue
|
|
||||||
parts = line.strip().split('\t')
|
|
||||||
if len(parts) >= 7:
|
|
||||||
cookies_to_add.append({
|
|
||||||
'domain': parts[0],
|
|
||||||
'path': parts[2],
|
|
||||||
'secure': parts[3] == 'TRUE',
|
|
||||||
'expires': int(parts[4]) if parts[4] != '0' else None,
|
|
||||||
'name': parts[5],
|
|
||||||
'value': parts[6]
|
|
||||||
})
|
|
||||||
|
|
||||||
if not cookies_to_add:
|
|
||||||
print("❌ No valid cookies found in file")
|
|
||||||
return False
|
|
||||||
|
|
||||||
await context.add_cookies(cookies_to_add)
|
|
||||||
|
|
||||||
page = await context.new_page()
|
|
||||||
await page.goto('https://www.youtube.com', wait_until='domcontentloaded', timeout=30000)
|
|
||||||
|
|
||||||
# Wait a bit for page to render
|
|
||||||
await asyncio.sleep(2)
|
|
||||||
|
|
||||||
# Check if we can access YouTube properly
|
|
||||||
# If blocked, there will be bot detection or sign-in prompts
|
|
||||||
content = await page.content()
|
|
||||||
|
|
||||||
# Simple validation: check if we have access to normal YouTube
|
|
||||||
is_valid = 'ytInitialData' in content or 'watch?' in content
|
|
||||||
|
|
||||||
if is_valid:
|
|
||||||
print("✅ Cookies are valid")
|
|
||||||
print(" YouTube access: OK")
|
|
||||||
return True
|
|
||||||
else:
|
|
||||||
print("⚠️ Cookies may be expired or invalid")
|
|
||||||
print(" YouTube access: BLOCKED")
|
|
||||||
return False
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"❌ Validation error: {e}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
cookies_path = sys.argv[1] if len(sys.argv) > 1 else 'youtube-cookies.txt'
|
|
||||||
valid = asyncio.run(validate_cookies(cookies_path))
|
|
||||||
sys.exit(0 if valid else 1)
|
|
||||||
1708
src/server.js
1708
src/server.js
File diff suppressed because it is too large
Load Diff
145
src/services/conversion.js
Normal file
145
src/services/conversion.js
Normal file
@ -0,0 +1,145 @@
|
|||||||
|
import { exec } from 'child_process';
|
||||||
|
import { promisify } from 'util';
|
||||||
|
import path from 'path';
|
||||||
|
import fs from 'fs';
|
||||||
|
|
||||||
|
const execPromise = promisify(exec);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert a video/audio file to MP3 using FFmpeg
|
||||||
|
* @param {string} inputPath - Path to input file
|
||||||
|
* @param {object} options - Conversion options
|
||||||
|
* @param {string} options.outputDir - Output directory (default: same as input)
|
||||||
|
* @param {string} options.bitrate - Audio bitrate (default: 192k)
|
||||||
|
* @param {string} options.quality - Audio quality 0-9 (default: 2, where 0 is best)
|
||||||
|
* @returns {Promise<object>} Conversion result with output path
|
||||||
|
*/
|
||||||
|
export async function convertToMP3(inputPath, options = {}) {
|
||||||
|
const {
|
||||||
|
outputDir = path.dirname(inputPath),
|
||||||
|
bitrate = '192k',
|
||||||
|
quality = '2',
|
||||||
|
} = options;
|
||||||
|
|
||||||
|
// Ensure input file exists
|
||||||
|
if (!fs.existsSync(inputPath)) {
|
||||||
|
throw new Error(`Input file not found: ${inputPath}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate output path
|
||||||
|
const inputFilename = path.basename(inputPath, path.extname(inputPath));
|
||||||
|
const outputPath = path.join(outputDir, `${inputFilename}.mp3`);
|
||||||
|
|
||||||
|
// Check if output already exists
|
||||||
|
if (fs.existsSync(outputPath)) {
|
||||||
|
// Add timestamp to make it unique
|
||||||
|
const timestamp = Date.now();
|
||||||
|
const uniqueOutputPath = path.join(outputDir, `${inputFilename}_${timestamp}.mp3`);
|
||||||
|
return convertToMP3Internal(inputPath, uniqueOutputPath, bitrate, quality);
|
||||||
|
}
|
||||||
|
|
||||||
|
return convertToMP3Internal(inputPath, outputPath, bitrate, quality);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Internal conversion function
|
||||||
|
*/
|
||||||
|
async function convertToMP3Internal(inputPath, outputPath, bitrate, quality) {
|
||||||
|
try {
|
||||||
|
// FFmpeg command to convert to MP3
|
||||||
|
// -i: input file
|
||||||
|
// -vn: no video (audio only)
|
||||||
|
// -ar 44100: audio sample rate 44.1kHz
|
||||||
|
// -ac 2: stereo
|
||||||
|
// -b:a: audio bitrate
|
||||||
|
// -q:a: audio quality (VBR)
|
||||||
|
const command = `ffmpeg -i "${inputPath}" -vn -ar 44100 -ac 2 -b:a ${bitrate} -q:a ${quality} "${outputPath}"`;
|
||||||
|
|
||||||
|
console.log(`Converting: ${path.basename(inputPath)} -> ${path.basename(outputPath)}`);
|
||||||
|
|
||||||
|
const { stdout, stderr } = await execPromise(command);
|
||||||
|
|
||||||
|
// Verify output file was created
|
||||||
|
if (!fs.existsSync(outputPath)) {
|
||||||
|
throw new Error('Conversion failed: output file not created');
|
||||||
|
}
|
||||||
|
|
||||||
|
const stats = fs.statSync(outputPath);
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
inputPath,
|
||||||
|
outputPath,
|
||||||
|
filename: path.basename(outputPath),
|
||||||
|
size: stats.size,
|
||||||
|
sizeHuman: formatBytes(stats.size),
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Conversion error: ${error.message}`);
|
||||||
|
throw new Error(`FFmpeg conversion failed: ${error.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert multiple files to MP3
|
||||||
|
* @param {string[]} inputPaths - Array of input file paths
|
||||||
|
* @param {object} options - Conversion options
|
||||||
|
* @returns {Promise<object>} Batch conversion results
|
||||||
|
*/
|
||||||
|
export async function convertMultipleToMP3(inputPaths, options = {}) {
|
||||||
|
const results = [];
|
||||||
|
let successCount = 0;
|
||||||
|
let failCount = 0;
|
||||||
|
|
||||||
|
for (let i = 0; i < inputPaths.length; i++) {
|
||||||
|
const inputPath = inputPaths[i];
|
||||||
|
console.log(`[${i + 1}/${inputPaths.length}] Converting: ${path.basename(inputPath)}`);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await convertToMP3(inputPath, options);
|
||||||
|
results.push({ ...result, index: i });
|
||||||
|
successCount++;
|
||||||
|
} catch (error) {
|
||||||
|
results.push({
|
||||||
|
success: false,
|
||||||
|
inputPath,
|
||||||
|
error: error.message,
|
||||||
|
index: i,
|
||||||
|
});
|
||||||
|
failCount++;
|
||||||
|
console.error(`Failed to convert ${path.basename(inputPath)}: ${error.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
totalFiles: inputPaths.length,
|
||||||
|
successCount,
|
||||||
|
failCount,
|
||||||
|
results,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Format bytes to human readable format
|
||||||
|
*/
|
||||||
|
function formatBytes(bytes, decimals = 2) {
|
||||||
|
if (bytes === 0) return '0 Bytes';
|
||||||
|
|
||||||
|
const k = 1024;
|
||||||
|
const dm = decimals < 0 ? 0 : decimals;
|
||||||
|
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
|
||||||
|
|
||||||
|
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||||
|
|
||||||
|
return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' + sizes[i];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get supported input formats
|
||||||
|
*/
|
||||||
|
export function getSupportedFormats() {
|
||||||
|
return {
|
||||||
|
video: ['.mp4', '.avi', '.mkv', '.mov', '.wmv', '.flv', '.webm', '.m4v'],
|
||||||
|
audio: ['.m4a', '.wav', '.flac', '.ogg', '.aac', '.wma', '.opus'],
|
||||||
|
};
|
||||||
|
}
|
||||||
@ -1,188 +0,0 @@
|
|||||||
const { exec } = require('child_process');
|
|
||||||
const { promisify } = require('util');
|
|
||||||
const fs = require('fs').promises;
|
|
||||||
const path = require('path');
|
|
||||||
|
|
||||||
const execAsync = promisify(exec);
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Manages YouTube cookies lifecycle using Camoufox stealth extraction.
|
|
||||||
* Auto-refresh when expired, validates periodically.
|
|
||||||
*/
|
|
||||||
class CookiesManager {
|
|
||||||
constructor() {
|
|
||||||
this.cookiesPath = path.join(__dirname, '../../youtube-cookies.txt');
|
|
||||||
this.pythonPath = process.env.PYTHON_PATH || 'python3';
|
|
||||||
this.extractScript = path.join(__dirname, '../python/extract_cookies.py');
|
|
||||||
this.validateScript = path.join(__dirname, '../python/validate_cookies.py');
|
|
||||||
|
|
||||||
this.lastRefresh = null;
|
|
||||||
this.isValid = false;
|
|
||||||
|
|
||||||
// Refresh cookies every 14 days (YouTube cookies typically last 2-4 weeks)
|
|
||||||
this.refreshIntervalDays = 14;
|
|
||||||
|
|
||||||
// Check interval (every 12 hours)
|
|
||||||
this.checkIntervalMs = 12 * 60 * 60 * 1000;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Initialize cookies manager.
|
|
||||||
* Check if cookies exist, validate them, refresh if needed.
|
|
||||||
*/
|
|
||||||
async init() {
|
|
||||||
console.log('🔧 Initializing cookies manager...');
|
|
||||||
|
|
||||||
// Check if cookies file exists
|
|
||||||
try {
|
|
||||||
await fs.access(this.cookiesPath);
|
|
||||||
console.log('✅ Cookies file exists');
|
|
||||||
|
|
||||||
// Validate cookies
|
|
||||||
const valid = await this.validate();
|
|
||||||
if (!valid) {
|
|
||||||
console.log('⚠️ Cookies invalid, refreshing...');
|
|
||||||
await this.refresh();
|
|
||||||
} else {
|
|
||||||
console.log('✅ Cookies valid');
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
console.log('📝 No cookies found, generating fresh cookies...');
|
|
||||||
await this.refresh();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Setup periodic validation (every 12 hours)
|
|
||||||
setInterval(() => {
|
|
||||||
this.checkAndRefresh().catch(err => {
|
|
||||||
console.error('Auto-check failed:', err.message);
|
|
||||||
});
|
|
||||||
}, this.checkIntervalMs);
|
|
||||||
|
|
||||||
console.log('✅ Cookies manager ready');
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Validate cookies using Python script.
|
|
||||||
* @returns {Promise<boolean>} True if cookies are valid
|
|
||||||
*/
|
|
||||||
async validate() {
|
|
||||||
try {
|
|
||||||
const { stdout, stderr } = await execAsync(
|
|
||||||
`${this.pythonPath} ${this.validateScript} ${this.cookiesPath}`,
|
|
||||||
{ timeout: 60000 }
|
|
||||||
);
|
|
||||||
|
|
||||||
// Check for validation success in output
|
|
||||||
this.isValid = stdout.includes('Cookies are valid');
|
|
||||||
|
|
||||||
if (stderr && !stderr.includes('DeprecationWarning')) {
|
|
||||||
console.warn('Validation stderr:', stderr.trim());
|
|
||||||
}
|
|
||||||
|
|
||||||
return this.isValid;
|
|
||||||
} catch (err) {
|
|
||||||
console.error('Validation failed:', err.message);
|
|
||||||
this.isValid = false;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Refresh cookies using Camoufox extraction.
|
|
||||||
* @returns {Promise<boolean>} True if refresh succeeded
|
|
||||||
*/
|
|
||||||
async refresh() {
|
|
||||||
console.log('🔄 Refreshing YouTube cookies with Camoufox...');
|
|
||||||
|
|
||||||
try {
|
|
||||||
const { stdout, stderr } = await execAsync(
|
|
||||||
`${this.pythonPath} ${this.extractScript} ${this.cookiesPath}`,
|
|
||||||
{ timeout: 120000 } // 2 min timeout
|
|
||||||
);
|
|
||||||
|
|
||||||
console.log(stdout.trim());
|
|
||||||
|
|
||||||
if (stderr && !stderr.includes('DeprecationWarning')) {
|
|
||||||
console.warn('Camoufox stderr:', stderr.trim());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Verify file was created
|
|
||||||
try {
|
|
||||||
await fs.access(this.cookiesPath);
|
|
||||||
this.lastRefresh = Date.now();
|
|
||||||
this.isValid = true;
|
|
||||||
console.log('✅ Cookies refreshed successfully');
|
|
||||||
return true;
|
|
||||||
} catch {
|
|
||||||
console.error('❌ Cookies file not created');
|
|
||||||
this.isValid = false;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
console.error('❌ Failed to refresh cookies:', err.message);
|
|
||||||
this.isValid = false;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check cookies age and validity, refresh if needed.
|
|
||||||
*/
|
|
||||||
async checkAndRefresh() {
|
|
||||||
console.log('🔍 Checking cookies status...');
|
|
||||||
|
|
||||||
// Check file age
|
|
||||||
try {
|
|
||||||
const stats = await fs.stat(this.cookiesPath);
|
|
||||||
const ageMs = Date.now() - stats.mtimeMs;
|
|
||||||
const ageDays = ageMs / (1000 * 60 * 60 * 24);
|
|
||||||
|
|
||||||
console.log(` Age: ${ageDays.toFixed(1)} days`);
|
|
||||||
|
|
||||||
// Refresh if too old
|
|
||||||
if (ageDays >= this.refreshIntervalDays) {
|
|
||||||
console.log(` Age threshold (${this.refreshIntervalDays} days) reached, refreshing...`);
|
|
||||||
await this.refresh();
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
} catch {
|
|
||||||
// File doesn't exist
|
|
||||||
console.log(' Cookies file missing, refreshing...');
|
|
||||||
await this.refresh();
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Validate cookies
|
|
||||||
const valid = await this.validate();
|
|
||||||
if (!valid) {
|
|
||||||
console.log(' Cookies invalid, refreshing...');
|
|
||||||
await this.refresh();
|
|
||||||
} else {
|
|
||||||
console.log(' Cookies OK ✅');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get path to cookies file.
|
|
||||||
* @returns {string} Cookies file path
|
|
||||||
*/
|
|
||||||
getCookiesPath() {
|
|
||||||
return this.cookiesPath;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get cookies status.
|
|
||||||
* @returns {object} Status object
|
|
||||||
*/
|
|
||||||
getStatus() {
|
|
||||||
return {
|
|
||||||
valid: this.isValid,
|
|
||||||
path: this.cookiesPath,
|
|
||||||
lastRefresh: this.lastRefresh,
|
|
||||||
refreshIntervalDays: this.refreshIntervalDays
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Export singleton
|
|
||||||
module.exports = new CookiesManager();
|
|
||||||
@ -1,190 +0,0 @@
|
|||||||
const { spawn } = require('child_process');
|
|
||||||
const cookiesManager = require('./cookiesManager');
|
|
||||||
const path = require('path');
|
|
||||||
const fs = require('fs').promises;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* YouTube download service using yt-dlp with Camoufox stealth cookies.
|
|
||||||
*/
|
|
||||||
class DownloadService {
|
|
||||||
constructor() {
|
|
||||||
this.storagePath = process.env.STORAGE_PATH || path.join(__dirname, '../../output');
|
|
||||||
this.ytdlpPath = process.env.YTDLP_PATH || 'yt-dlp';
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Download YouTube video as MP3.
|
|
||||||
* @param {string} url - YouTube video URL
|
|
||||||
* @param {object} options - Download options
|
|
||||||
* @returns {Promise<object>} Download result with metadata
|
|
||||||
*/
|
|
||||||
async downloadYouTube(url, options = {}) {
|
|
||||||
// Ensure storage directory exists
|
|
||||||
await fs.mkdir(this.storagePath, { recursive: true });
|
|
||||||
|
|
||||||
// Ensure cookies are valid before download
|
|
||||||
await cookiesManager.checkAndRefresh();
|
|
||||||
|
|
||||||
const cookiesPath = cookiesManager.getCookiesPath();
|
|
||||||
|
|
||||||
// Build yt-dlp arguments
|
|
||||||
const outputTemplate = path.join(this.storagePath, '%(id)s.%(ext)s');
|
|
||||||
const quality = options.quality || '192k';
|
|
||||||
|
|
||||||
const args = [
|
|
||||||
// Cookies (stealth from Camoufox)
|
|
||||||
'--cookies', cookiesPath,
|
|
||||||
|
|
||||||
// Player client (mweb is stable)
|
|
||||||
'--extractor-args', 'youtube:player_client=mweb',
|
|
||||||
|
|
||||||
// Format selection (audio only)
|
|
||||||
'--format', 'bestaudio[ext=m4a]/bestaudio',
|
|
||||||
|
|
||||||
// Audio extraction
|
|
||||||
'--extract-audio',
|
|
||||||
'--audio-format', 'mp3',
|
|
||||||
'--audio-quality', quality,
|
|
||||||
|
|
||||||
// Metadata
|
|
||||||
'--embed-thumbnail',
|
|
||||||
'--add-metadata',
|
|
||||||
|
|
||||||
// No playlists (single video only)
|
|
||||||
'--no-playlist',
|
|
||||||
|
|
||||||
// Output JSON metadata
|
|
||||||
'--print-json',
|
|
||||||
|
|
||||||
// Output template
|
|
||||||
'--output', outputTemplate,
|
|
||||||
|
|
||||||
// URL
|
|
||||||
url
|
|
||||||
];
|
|
||||||
|
|
||||||
return new Promise((resolve, reject) => {
|
|
||||||
const ytdlp = spawn(this.ytdlpPath, args);
|
|
||||||
|
|
||||||
let jsonOutput = '';
|
|
||||||
let errorOutput = '';
|
|
||||||
|
|
||||||
ytdlp.stdout.on('data', (data) => {
|
|
||||||
const text = data.toString();
|
|
||||||
jsonOutput += text;
|
|
||||||
});
|
|
||||||
|
|
||||||
ytdlp.stderr.on('data', (data) => {
|
|
||||||
const text = data.toString();
|
|
||||||
errorOutput += text;
|
|
||||||
|
|
||||||
// Log progress
|
|
||||||
if (text.includes('[download]') || text.includes('[ExtractAudio]')) {
|
|
||||||
console.log(' ', text.trim());
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
ytdlp.on('close', async (code) => {
|
|
||||||
if (code === 0) {
|
|
||||||
try {
|
|
||||||
// Parse JSON output from yt-dlp
|
|
||||||
const lines = jsonOutput.split('\n').filter(l => l.trim());
|
|
||||||
const lastLine = lines[lines.length - 1];
|
|
||||||
const metadata = JSON.parse(lastLine);
|
|
||||||
|
|
||||||
// Extract relevant metadata
|
|
||||||
const result = {
|
|
||||||
success: true,
|
|
||||||
title: metadata.title,
|
|
||||||
duration: metadata.duration,
|
|
||||||
artist: metadata.artist || metadata.uploader || metadata.channel,
|
|
||||||
album: metadata.album || null,
|
|
||||||
filePath: metadata.filename,
|
|
||||||
fileName: path.basename(metadata.filename),
|
|
||||||
fileSize: metadata.filesize || null,
|
|
||||||
youtubeId: metadata.id,
|
|
||||||
youtubeUrl: metadata.webpage_url,
|
|
||||||
thumbnail: metadata.thumbnail,
|
|
||||||
uploadDate: metadata.upload_date,
|
|
||||||
description: metadata.description || null
|
|
||||||
};
|
|
||||||
|
|
||||||
console.log(`✅ Downloaded: ${result.title}`);
|
|
||||||
resolve(result);
|
|
||||||
} catch (err) {
|
|
||||||
reject(new Error(`Failed to parse yt-dlp output: ${err.message}`));
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Check for specific errors
|
|
||||||
if (errorOutput.includes('Sign in to confirm')) {
|
|
||||||
console.log('🤖 Bot detection! Force refreshing cookies...');
|
|
||||||
|
|
||||||
// Force refresh cookies
|
|
||||||
await cookiesManager.refresh();
|
|
||||||
|
|
||||||
// Retry once
|
|
||||||
try {
|
|
||||||
console.log('🔄 Retrying download with fresh cookies...');
|
|
||||||
const result = await this.downloadYouTube(url, options);
|
|
||||||
resolve(result);
|
|
||||||
} catch (retryErr) {
|
|
||||||
reject(new Error(`Download failed after cookie refresh: ${retryErr.message}`));
|
|
||||||
}
|
|
||||||
} else if (errorOutput.includes('Video unavailable')) {
|
|
||||||
reject(new Error('Video is unavailable or private'));
|
|
||||||
} else if (errorOutput.includes('429')) {
|
|
||||||
reject(new Error('Rate limited by YouTube. Please wait and try again later.'));
|
|
||||||
} else {
|
|
||||||
reject(new Error(`yt-dlp failed (code ${code}): ${errorOutput}`));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
ytdlp.on('error', (err) => {
|
|
||||||
reject(new Error(`Failed to spawn yt-dlp: ${err.message}`));
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get file stream for a downloaded file.
|
|
||||||
* @param {string} fileName - File name
|
|
||||||
* @returns {Promise<object>} File info and stream
|
|
||||||
*/
|
|
||||||
async getFileStream(fileName) {
|
|
||||||
const filePath = path.join(this.storagePath, fileName);
|
|
||||||
|
|
||||||
// Check if file exists
|
|
||||||
try {
|
|
||||||
const stats = await fs.stat(filePath);
|
|
||||||
return {
|
|
||||||
path: filePath,
|
|
||||||
size: stats.size,
|
|
||||||
exists: true
|
|
||||||
};
|
|
||||||
} catch {
|
|
||||||
throw new Error('File not found');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Delete a downloaded file.
|
|
||||||
* @param {string} fileName - File name
|
|
||||||
* @returns {Promise<boolean>} True if deleted
|
|
||||||
*/
|
|
||||||
async deleteFile(fileName) {
|
|
||||||
const filePath = path.join(this.storagePath, fileName);
|
|
||||||
|
|
||||||
try {
|
|
||||||
await fs.unlink(filePath);
|
|
||||||
console.log(`🗑️ Deleted: ${fileName}`);
|
|
||||||
return true;
|
|
||||||
} catch (err) {
|
|
||||||
console.error(`Failed to delete ${fileName}:`, err.message);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Export singleton
|
|
||||||
module.exports = new DownloadService();
|
|
||||||
195
src/services/summarize.js
Normal file
195
src/services/summarize.js
Normal file
@ -0,0 +1,195 @@
|
|||||||
|
import OpenAI from 'openai';
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
|
||||||
|
let openai = null;
|
||||||
|
|
||||||
|
// Max characters per chunk for summarization
|
||||||
|
const MAX_CHUNK_CHARS = 30000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get OpenAI client (lazy initialization)
|
||||||
|
*/
|
||||||
|
function getOpenAI() {
|
||||||
|
if (!openai) {
|
||||||
|
if (!process.env.OPENAI_API_KEY) {
|
||||||
|
throw new Error('OPENAI_API_KEY environment variable is not set');
|
||||||
|
}
|
||||||
|
openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
|
||||||
|
}
|
||||||
|
return openai;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Summarize text using GPT-4o
|
||||||
|
*/
|
||||||
|
export async function summarizeText(text, options = {}) {
|
||||||
|
const {
|
||||||
|
model = 'gpt-5.1', // GPT-5.1 - latest OpenAI model (Nov 2025)
|
||||||
|
language = 'same', // 'same' = same as input, or specify language code
|
||||||
|
style = 'concise', // 'concise', 'detailed', 'bullet'
|
||||||
|
maxLength = null, // optional max length in words
|
||||||
|
} = options;
|
||||||
|
|
||||||
|
const client = getOpenAI();
|
||||||
|
|
||||||
|
let styleInstruction = '';
|
||||||
|
switch (style) {
|
||||||
|
case 'detailed':
|
||||||
|
styleInstruction = 'Provide a detailed summary that captures all important points and nuances.';
|
||||||
|
break;
|
||||||
|
case 'bullet':
|
||||||
|
styleInstruction = 'Provide the summary as bullet points, highlighting the key points.';
|
||||||
|
break;
|
||||||
|
case 'concise':
|
||||||
|
default:
|
||||||
|
styleInstruction = 'Provide a concise summary that captures the main points.';
|
||||||
|
}
|
||||||
|
|
||||||
|
let languageInstruction = '';
|
||||||
|
if (language === 'same') {
|
||||||
|
languageInstruction = 'Write the summary in the same language as the input text.';
|
||||||
|
} else {
|
||||||
|
languageInstruction = `Write the summary in ${language}.`;
|
||||||
|
}
|
||||||
|
|
||||||
|
let lengthInstruction = '';
|
||||||
|
if (maxLength) {
|
||||||
|
lengthInstruction = `Keep the summary under ${maxLength} words.`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const systemPrompt = `You are an expert summarizer. ${styleInstruction} ${languageInstruction} ${lengthInstruction}
|
||||||
|
Focus on the most important information and main ideas. Be accurate and objective.`;
|
||||||
|
|
||||||
|
// Handle long texts by chunking
|
||||||
|
if (text.length > MAX_CHUNK_CHARS) {
|
||||||
|
return await summarizeLongText(text, { model, systemPrompt, style });
|
||||||
|
}
|
||||||
|
|
||||||
|
const response = await client.chat.completions.create({
|
||||||
|
model,
|
||||||
|
messages: [
|
||||||
|
{ role: 'system', content: systemPrompt },
|
||||||
|
{ role: 'user', content: `Please summarize the following text:\n\n${text}` },
|
||||||
|
],
|
||||||
|
temperature: 0.3,
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
summary: response.choices[0].message.content,
|
||||||
|
model,
|
||||||
|
style,
|
||||||
|
inputLength: text.length,
|
||||||
|
chunks: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Summarize long text by chunking and combining summaries
|
||||||
|
*/
|
||||||
|
async function summarizeLongText(text, options) {
|
||||||
|
const { model, systemPrompt, style } = options;
|
||||||
|
const client = getOpenAI();
|
||||||
|
|
||||||
|
// Split into chunks
|
||||||
|
const chunks = [];
|
||||||
|
let currentChunk = '';
|
||||||
|
const sentences = text.split(/(?<=[.!?。!?\n])\s*/);
|
||||||
|
|
||||||
|
for (const sentence of sentences) {
|
||||||
|
if ((currentChunk + sentence).length > MAX_CHUNK_CHARS && currentChunk) {
|
||||||
|
chunks.push(currentChunk.trim());
|
||||||
|
currentChunk = sentence;
|
||||||
|
} else {
|
||||||
|
currentChunk += ' ' + sentence;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (currentChunk.trim()) {
|
||||||
|
chunks.push(currentChunk.trim());
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`Summarizing ${chunks.length} chunks...`);
|
||||||
|
|
||||||
|
// Summarize each chunk
|
||||||
|
const chunkSummaries = [];
|
||||||
|
for (let i = 0; i < chunks.length; i++) {
|
||||||
|
console.log(`[${i + 1}/${chunks.length}] Summarizing chunk...`);
|
||||||
|
const response = await client.chat.completions.create({
|
||||||
|
model,
|
||||||
|
messages: [
|
||||||
|
{ role: 'system', content: systemPrompt },
|
||||||
|
{ role: 'user', content: `Please summarize the following text (part ${i + 1} of ${chunks.length}):\n\n${chunks[i]}` },
|
||||||
|
],
|
||||||
|
temperature: 0.3,
|
||||||
|
});
|
||||||
|
chunkSummaries.push(response.choices[0].message.content);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Combine summaries if multiple chunks
|
||||||
|
if (chunkSummaries.length === 1) {
|
||||||
|
return {
|
||||||
|
summary: chunkSummaries[0],
|
||||||
|
model,
|
||||||
|
style,
|
||||||
|
inputLength: text.length,
|
||||||
|
chunks: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create final combined summary
|
||||||
|
const combinedText = chunkSummaries.join('\n\n---\n\n');
|
||||||
|
const finalResponse = await client.chat.completions.create({
|
||||||
|
model,
|
||||||
|
messages: [
|
||||||
|
{ role: 'system', content: `You are an expert summarizer. Combine and synthesize the following partial summaries into a single coherent ${style} summary. Remove redundancy and ensure a smooth flow.` },
|
||||||
|
{ role: 'user', content: `Please combine these summaries into one:\n\n${combinedText}` },
|
||||||
|
],
|
||||||
|
temperature: 0.3,
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
summary: finalResponse.choices[0].message.content,
|
||||||
|
model,
|
||||||
|
style,
|
||||||
|
inputLength: text.length,
|
||||||
|
chunks: chunks.length,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Summarize a text file
|
||||||
|
*/
|
||||||
|
export async function summarizeFile(filePath, options = {}) {
|
||||||
|
if (!fs.existsSync(filePath)) {
|
||||||
|
throw new Error(`File not found: ${filePath}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const { outputDir, ...otherOptions } = options;
|
||||||
|
|
||||||
|
const text = fs.readFileSync(filePath, 'utf-8');
|
||||||
|
const result = await summarizeText(text, otherOptions);
|
||||||
|
|
||||||
|
// Save summary to file
|
||||||
|
const dir = outputDir || path.dirname(filePath);
|
||||||
|
const baseName = path.basename(filePath, path.extname(filePath));
|
||||||
|
const summaryPath = path.join(dir, `${baseName}_summary.txt`);
|
||||||
|
|
||||||
|
fs.writeFileSync(summaryPath, result.summary, 'utf-8');
|
||||||
|
|
||||||
|
return {
|
||||||
|
...result,
|
||||||
|
filePath,
|
||||||
|
summaryPath,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available summary styles
|
||||||
|
*/
|
||||||
|
export function getSummaryStyles() {
|
||||||
|
return {
|
||||||
|
concise: 'A brief summary capturing main points',
|
||||||
|
detailed: 'A comprehensive summary with nuances',
|
||||||
|
bullet: 'Key points as bullet points',
|
||||||
|
};
|
||||||
|
}
|
||||||
178
src/services/transcription.js
Normal file
178
src/services/transcription.js
Normal file
@ -0,0 +1,178 @@
|
|||||||
|
import OpenAI from 'openai';
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
|
||||||
|
let openai = null;
|
||||||
|
|
||||||
|
// Available transcription models
|
||||||
|
const MODELS = {
|
||||||
|
'gpt-4o-transcribe': {
|
||||||
|
name: 'gpt-4o-transcribe',
|
||||||
|
formats: ['json', 'text'],
|
||||||
|
supportsLanguage: true,
|
||||||
|
},
|
||||||
|
'gpt-4o-mini-transcribe': {
|
||||||
|
name: 'gpt-4o-mini-transcribe',
|
||||||
|
formats: ['json', 'text'],
|
||||||
|
supportsLanguage: true,
|
||||||
|
},
|
||||||
|
'whisper-1': {
|
||||||
|
name: 'whisper-1',
|
||||||
|
formats: ['json', 'text', 'srt', 'vtt', 'verbose_json'],
|
||||||
|
supportsLanguage: true,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const DEFAULT_MODEL = 'gpt-4o-mini-transcribe';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get OpenAI client (lazy initialization)
|
||||||
|
*/
|
||||||
|
function getOpenAI() {
|
||||||
|
if (!openai) {
|
||||||
|
if (!process.env.OPENAI_API_KEY) {
|
||||||
|
throw new Error('OPENAI_API_KEY environment variable is not set');
|
||||||
|
}
|
||||||
|
openai = new OpenAI({
|
||||||
|
apiKey: process.env.OPENAI_API_KEY,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return openai;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available models
|
||||||
|
*/
|
||||||
|
export function getAvailableModels() {
|
||||||
|
return Object.keys(MODELS);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Transcribe an audio file using OpenAI API
|
||||||
|
* @param {string} filePath - Path to audio file
|
||||||
|
* @param {Object} options - Transcription options
|
||||||
|
* @param {string} options.language - Language code (e.g., 'en', 'fr', 'es', 'zh')
|
||||||
|
* @param {string} options.responseFormat - Output format: 'json' or 'text' (gpt-4o models), or 'srt'/'vtt' (whisper-1 only)
|
||||||
|
* @param {string} options.prompt - Optional context prompt for better accuracy
|
||||||
|
* @param {string} options.model - Model to use (default: gpt-4o-transcribe)
|
||||||
|
*/
|
||||||
|
export async function transcribeFile(filePath, options = {}) {
|
||||||
|
const {
|
||||||
|
language = null, // Auto-detect if null
|
||||||
|
responseFormat = 'text', // json or text for gpt-4o models
|
||||||
|
prompt = null, // Optional context prompt
|
||||||
|
model = DEFAULT_MODEL,
|
||||||
|
} = options;
|
||||||
|
|
||||||
|
if (!fs.existsSync(filePath)) {
|
||||||
|
throw new Error(`File not found: ${filePath}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const modelConfig = MODELS[model] || MODELS[DEFAULT_MODEL];
|
||||||
|
const actualModel = modelConfig.name;
|
||||||
|
|
||||||
|
// Validate response format for model
|
||||||
|
let actualFormat = responseFormat;
|
||||||
|
if (!modelConfig.formats.includes(responseFormat)) {
|
||||||
|
console.warn(`Format '${responseFormat}' not supported by ${actualModel}, using 'text'`);
|
||||||
|
actualFormat = 'text';
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const transcriptionOptions = {
|
||||||
|
file: fs.createReadStream(filePath),
|
||||||
|
model: actualModel,
|
||||||
|
response_format: actualFormat,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (language) {
|
||||||
|
transcriptionOptions.language = language;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (prompt) {
|
||||||
|
transcriptionOptions.prompt = prompt;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`Using model: ${actualModel}, format: ${actualFormat}${language ? `, language: ${language}` : ''}`);
|
||||||
|
|
||||||
|
const transcription = await getOpenAI().audio.transcriptions.create(transcriptionOptions);
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
filePath,
|
||||||
|
text: actualFormat === 'json' || actualFormat === 'verbose_json'
|
||||||
|
? transcription.text
|
||||||
|
: transcription,
|
||||||
|
format: actualFormat,
|
||||||
|
model: actualModel,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
throw new Error(`Transcription failed: ${error.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Transcribe and save to file
|
||||||
|
*/
|
||||||
|
export async function transcribeAndSave(filePath, options = {}) {
|
||||||
|
const { outputFormat = 'txt', outputDir = null } = options;
|
||||||
|
|
||||||
|
const result = await transcribeFile(filePath, options);
|
||||||
|
|
||||||
|
// Determine output path
|
||||||
|
const baseName = path.basename(filePath, path.extname(filePath));
|
||||||
|
const outputPath = path.join(
|
||||||
|
outputDir || path.dirname(filePath),
|
||||||
|
`${baseName}.${outputFormat}`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Save transcription
|
||||||
|
fs.writeFileSync(outputPath, result.text, 'utf-8');
|
||||||
|
|
||||||
|
return {
|
||||||
|
...result,
|
||||||
|
transcriptionPath: outputPath,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Transcribe multiple files
|
||||||
|
*/
|
||||||
|
export async function transcribeMultiple(filePaths, options = {}) {
|
||||||
|
const { onProgress, onFileComplete } = options;
|
||||||
|
const results = [];
|
||||||
|
|
||||||
|
for (let i = 0; i < filePaths.length; i++) {
|
||||||
|
const filePath = filePaths[i];
|
||||||
|
|
||||||
|
if (onProgress) {
|
||||||
|
onProgress({ current: i + 1, total: filePaths.length, filePath });
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`[${i + 1}/${filePaths.length}] Transcribing: ${path.basename(filePath)}`);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await transcribeAndSave(filePath, options);
|
||||||
|
results.push(result);
|
||||||
|
|
||||||
|
if (onFileComplete) {
|
||||||
|
onFileComplete(result);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Failed to transcribe ${filePath}: ${error.message}`);
|
||||||
|
results.push({
|
||||||
|
success: false,
|
||||||
|
filePath,
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
results,
|
||||||
|
totalFiles: filePaths.length,
|
||||||
|
successCount: results.filter(r => r.success).length,
|
||||||
|
failCount: results.filter(r => !r.success).length,
|
||||||
|
};
|
||||||
|
}
|
||||||
271
src/services/translation.js
Normal file
271
src/services/translation.js
Normal file
@ -0,0 +1,271 @@
|
|||||||
|
import OpenAI from 'openai';
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
|
||||||
|
let openai = null;
|
||||||
|
|
||||||
|
// Max characters per chunk (~6000 tokens ≈ 24000 characters for most languages)
|
||||||
|
const MAX_CHUNK_CHARS = 20000;
|
||||||
|
|
||||||
|
const LANGUAGES = {
|
||||||
|
en: 'English',
|
||||||
|
fr: 'French',
|
||||||
|
es: 'Spanish',
|
||||||
|
de: 'German',
|
||||||
|
it: 'Italian',
|
||||||
|
pt: 'Portuguese',
|
||||||
|
zh: 'Chinese',
|
||||||
|
ja: 'Japanese',
|
||||||
|
ko: 'Korean',
|
||||||
|
ru: 'Russian',
|
||||||
|
ar: 'Arabic',
|
||||||
|
hi: 'Hindi',
|
||||||
|
nl: 'Dutch',
|
||||||
|
pl: 'Polish',
|
||||||
|
tr: 'Turkish',
|
||||||
|
vi: 'Vietnamese',
|
||||||
|
th: 'Thai',
|
||||||
|
sv: 'Swedish',
|
||||||
|
da: 'Danish',
|
||||||
|
fi: 'Finnish',
|
||||||
|
no: 'Norwegian',
|
||||||
|
cs: 'Czech',
|
||||||
|
el: 'Greek',
|
||||||
|
he: 'Hebrew',
|
||||||
|
id: 'Indonesian',
|
||||||
|
ms: 'Malay',
|
||||||
|
ro: 'Romanian',
|
||||||
|
uk: 'Ukrainian',
|
||||||
|
};
|
||||||
|
|
||||||
|
// Sentence ending patterns for different languages
|
||||||
|
const SENTENCE_ENDINGS = /[.!?。!?。\n]/g;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get OpenAI client (lazy initialization)
|
||||||
|
*/
|
||||||
|
function getOpenAI() {
|
||||||
|
if (!openai) {
|
||||||
|
if (!process.env.OPENAI_API_KEY) {
|
||||||
|
throw new Error('OPENAI_API_KEY environment variable is not set');
|
||||||
|
}
|
||||||
|
openai = new OpenAI({
|
||||||
|
apiKey: process.env.OPENAI_API_KEY,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return openai;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Split text into chunks at sentence boundaries
|
||||||
|
* @param {string} text - Text to split
|
||||||
|
* @param {number} maxChars - Maximum characters per chunk
|
||||||
|
* @returns {string[]} Array of text chunks
|
||||||
|
*/
|
||||||
|
function splitIntoChunks(text, maxChars = MAX_CHUNK_CHARS) {
|
||||||
|
if (text.length <= maxChars) {
|
||||||
|
return [text];
|
||||||
|
}
|
||||||
|
|
||||||
|
const chunks = [];
|
||||||
|
let currentPos = 0;
|
||||||
|
|
||||||
|
while (currentPos < text.length) {
|
||||||
|
let endPos = currentPos + maxChars;
|
||||||
|
|
||||||
|
// If we're at the end, just take the rest
|
||||||
|
if (endPos >= text.length) {
|
||||||
|
chunks.push(text.slice(currentPos));
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find the last sentence ending before maxChars
|
||||||
|
const searchText = text.slice(currentPos, endPos);
|
||||||
|
let lastSentenceEnd = -1;
|
||||||
|
|
||||||
|
// Find all sentence endings in the search range
|
||||||
|
let match;
|
||||||
|
SENTENCE_ENDINGS.lastIndex = 0;
|
||||||
|
while ((match = SENTENCE_ENDINGS.exec(searchText)) !== null) {
|
||||||
|
lastSentenceEnd = match.index + 1; // Include the punctuation
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we found a sentence ending, cut there
|
||||||
|
// Otherwise, look for the next sentence ending after maxChars (up to 20% more)
|
||||||
|
if (lastSentenceEnd > maxChars * 0.5) {
|
||||||
|
endPos = currentPos + lastSentenceEnd;
|
||||||
|
} else {
|
||||||
|
// Look forward for a sentence ending (up to 20% more characters)
|
||||||
|
const extendedSearch = text.slice(endPos, endPos + maxChars * 0.2);
|
||||||
|
SENTENCE_ENDINGS.lastIndex = 0;
|
||||||
|
const forwardMatch = SENTENCE_ENDINGS.exec(extendedSearch);
|
||||||
|
if (forwardMatch) {
|
||||||
|
endPos = endPos + forwardMatch.index + 1;
|
||||||
|
}
|
||||||
|
// If still no sentence ending found, just cut at maxChars
|
||||||
|
}
|
||||||
|
|
||||||
|
chunks.push(text.slice(currentPos, endPos).trim());
|
||||||
|
currentPos = endPos;
|
||||||
|
|
||||||
|
// Skip any leading whitespace for the next chunk
|
||||||
|
while (currentPos < text.length && /\s/.test(text[currentPos])) {
|
||||||
|
currentPos++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return chunks.filter(chunk => chunk.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get available languages
|
||||||
|
*/
|
||||||
|
export function getLanguages() {
|
||||||
|
return LANGUAGES;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Translate a single chunk of text
|
||||||
|
*/
|
||||||
|
async function translateChunk(text, targetLanguage, sourceLanguage) {
|
||||||
|
const prompt = sourceLanguage
|
||||||
|
? `Translate the following text from ${sourceLanguage} to ${targetLanguage}. Only output the translation, nothing else:\n\n${text}`
|
||||||
|
: `Translate the following text to ${targetLanguage}. Only output the translation, nothing else:\n\n${text}`;
|
||||||
|
|
||||||
|
const response = await getOpenAI().chat.completions.create({
|
||||||
|
model: 'gpt-4o-mini',
|
||||||
|
max_tokens: 16384,
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: 'user',
|
||||||
|
content: prompt,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
return response.choices[0].message.content;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Translate text using GPT-4o-mini with chunking for long texts
|
||||||
|
* @param {string} text - Text to translate
|
||||||
|
* @param {string} targetLang - Target language code (e.g., 'en', 'fr')
|
||||||
|
* @param {string} sourceLang - Source language code (optional, auto-detect if null)
|
||||||
|
*/
|
||||||
|
export async function translateText(text, targetLang, sourceLang = null) {
|
||||||
|
if (!text || !text.trim()) {
|
||||||
|
throw new Error('No text provided for translation');
|
||||||
|
}
|
||||||
|
|
||||||
|
const targetLanguage = LANGUAGES[targetLang] || targetLang;
|
||||||
|
const sourceLanguage = sourceLang ? (LANGUAGES[sourceLang] || sourceLang) : null;
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Split text into chunks
|
||||||
|
const chunks = splitIntoChunks(text);
|
||||||
|
|
||||||
|
if (chunks.length === 1) {
|
||||||
|
// Single chunk - translate directly
|
||||||
|
const translation = await translateChunk(text, targetLanguage, sourceLanguage);
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
originalText: text,
|
||||||
|
translatedText: translation,
|
||||||
|
targetLanguage: targetLanguage,
|
||||||
|
sourceLanguage: sourceLanguage || 'auto-detected',
|
||||||
|
chunks: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Multiple chunks - translate each and combine
|
||||||
|
console.log(`Splitting text into ${chunks.length} chunks for translation...`);
|
||||||
|
const translations = [];
|
||||||
|
|
||||||
|
for (let i = 0; i < chunks.length; i++) {
|
||||||
|
console.log(` Translating chunk ${i + 1}/${chunks.length} (${chunks[i].length} chars)...`);
|
||||||
|
const translation = await translateChunk(chunks[i], targetLanguage, sourceLanguage);
|
||||||
|
translations.push(translation);
|
||||||
|
}
|
||||||
|
|
||||||
|
const combinedTranslation = translations.join('\n\n');
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
originalText: text,
|
||||||
|
translatedText: combinedTranslation,
|
||||||
|
targetLanguage: targetLanguage,
|
||||||
|
sourceLanguage: sourceLanguage || 'auto-detected',
|
||||||
|
chunks: chunks.length,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
throw new Error(`Translation failed: ${error.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Translate a text file
|
||||||
|
* @param {string} filePath - Path to text file
|
||||||
|
* @param {string} targetLang - Target language code
|
||||||
|
* @param {string} sourceLang - Source language code (optional)
|
||||||
|
* @param {string} outputDir - Output directory (optional)
|
||||||
|
*/
|
||||||
|
export async function translateFile(filePath, targetLang, sourceLang = null, outputDir = null) {
|
||||||
|
if (!fs.existsSync(filePath)) {
|
||||||
|
throw new Error(`File not found: ${filePath}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const text = fs.readFileSync(filePath, 'utf-8');
|
||||||
|
const result = await translateText(text, targetLang, sourceLang);
|
||||||
|
|
||||||
|
// Save translation
|
||||||
|
const baseName = path.basename(filePath, path.extname(filePath));
|
||||||
|
const outputPath = path.join(
|
||||||
|
outputDir || path.dirname(filePath),
|
||||||
|
`${baseName}_${targetLang}.txt`
|
||||||
|
);
|
||||||
|
|
||||||
|
fs.writeFileSync(outputPath, result.translatedText, 'utf-8');
|
||||||
|
|
||||||
|
return {
|
||||||
|
...result,
|
||||||
|
originalPath: filePath,
|
||||||
|
translationPath: outputPath,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Translate multiple files
|
||||||
|
*/
|
||||||
|
export async function translateMultiple(filePaths, targetLang, sourceLang = null, outputDir = null, onProgress = null) {
|
||||||
|
const results = [];
|
||||||
|
|
||||||
|
for (let i = 0; i < filePaths.length; i++) {
|
||||||
|
const filePath = filePaths[i];
|
||||||
|
|
||||||
|
if (onProgress) {
|
||||||
|
onProgress({ current: i + 1, total: filePaths.length, filePath });
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`[${i + 1}/${filePaths.length}] Translating: ${path.basename(filePath)}`);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await translateFile(filePath, targetLang, sourceLang, outputDir);
|
||||||
|
results.push(result);
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Failed to translate ${filePath}: ${error.message}`);
|
||||||
|
results.push({
|
||||||
|
success: false,
|
||||||
|
originalPath: filePath,
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
results,
|
||||||
|
totalFiles: filePaths.length,
|
||||||
|
successCount: results.filter(r => r.success).length,
|
||||||
|
failCount: results.filter(r => !r.success).length,
|
||||||
|
};
|
||||||
|
}
|
||||||
383
src/services/youtube.js
Normal file
383
src/services/youtube.js
Normal file
@ -0,0 +1,383 @@
|
|||||||
|
import { createRequire } from 'module';
|
||||||
|
import path from 'path';
|
||||||
|
import fs from 'fs';
|
||||||
|
import { spawn } from 'child_process';
|
||||||
|
|
||||||
|
// Use system yt-dlp binary (check common paths)
|
||||||
|
const YTDLP_PATH = process.env.YTDLP_PATH || 'yt-dlp';
|
||||||
|
|
||||||
|
// Path to cookies file (optional)
|
||||||
|
const COOKIES_PATH = process.env.YOUTUBE_COOKIES_PATH || null;
|
||||||
|
|
||||||
|
// Browser to extract cookies from (chrome, firefox, edge, safari, etc.)
|
||||||
|
const COOKIES_BROWSER = process.env.YOUTUBE_COOKIES_BROWSER || null;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Enhanced error message for YouTube bot detection
|
||||||
|
*/
|
||||||
|
function enhanceYouTubeError(error) {
|
||||||
|
const errorMsg = error.message || error.toString();
|
||||||
|
|
||||||
|
// Check if it's a bot detection error
|
||||||
|
if (errorMsg.includes('Sign in to confirm') ||
|
||||||
|
errorMsg.includes('not a bot') ||
|
||||||
|
errorMsg.includes('confirm you\'re not a bot') ||
|
||||||
|
errorMsg.includes('ERROR: Unable to extract')) {
|
||||||
|
|
||||||
|
const cookiesConfigured = COOKIES_BROWSER || COOKIES_PATH;
|
||||||
|
|
||||||
|
return {
|
||||||
|
error: 'YouTube Bot Detection',
|
||||||
|
message: 'YouTube is blocking this request. Authentication required.',
|
||||||
|
reason: errorMsg,
|
||||||
|
solution: {
|
||||||
|
quick: 'Upload fresh cookies from your browser',
|
||||||
|
steps: [
|
||||||
|
'1. Install browser extension: "Get cookies.txt LOCALLY"',
|
||||||
|
'2. Visit youtube.com and log into your account',
|
||||||
|
'3. Export cookies using the extension',
|
||||||
|
'4. Upload via API: POST /admin/upload-cookies',
|
||||||
|
' Or use the web interface at http://yourserver:8888',
|
||||||
|
],
|
||||||
|
alternative: 'Use extract-and-upload-cookies.sh script for automation',
|
||||||
|
documentation: 'See COOKIES_QUICK_START.md for detailed instructions'
|
||||||
|
},
|
||||||
|
currentConfig: {
|
||||||
|
cookiesFile: COOKIES_PATH || 'Not configured',
|
||||||
|
cookiesBrowser: COOKIES_BROWSER || 'Not configured',
|
||||||
|
status: cookiesConfigured ? '⚠️ Configured but may be expired' : '❌ Not configured'
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generic YouTube error
|
||||||
|
return {
|
||||||
|
error: 'YouTube Download Failed',
|
||||||
|
message: errorMsg,
|
||||||
|
solution: 'Check if the URL is valid and the video is available'
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Add cookies argument - prioritizes live browser cookies over file
|
||||||
|
*/
|
||||||
|
function addCookiesArg(args, cookiesPath = null) {
|
||||||
|
// Option 1: Extract cookies from browser (always fresh)
|
||||||
|
if (COOKIES_BROWSER) {
|
||||||
|
console.log(`Using live cookies from ${COOKIES_BROWSER} browser`);
|
||||||
|
return ['--cookies-from-browser', COOKIES_BROWSER, ...args];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Option 2: Use static cookies file (may expire)
|
||||||
|
// Check dynamically in case cookies were uploaded after server started
|
||||||
|
const cookies = cookiesPath || process.env.YOUTUBE_COOKIES_PATH || COOKIES_PATH;
|
||||||
|
if (cookies && fs.existsSync(cookies)) {
|
||||||
|
console.log(`Using cookies file: ${cookies}`);
|
||||||
|
return ['--cookies', cookies, ...args];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Option 3: No cookies (may fail on some videos)
|
||||||
|
console.log('No cookies configured - some videos may fail');
|
||||||
|
return args;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Execute yt-dlp command and return parsed JSON
|
||||||
|
*/
|
||||||
|
async function ytdlp(url, args = [], options = {}) {
|
||||||
|
const { cookiesPath } = options;
|
||||||
|
const finalArgs = addCookiesArg(args, cookiesPath);
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const proc = spawn(YTDLP_PATH, [...finalArgs, url]);
|
||||||
|
let stdout = '';
|
||||||
|
let stderr = '';
|
||||||
|
|
||||||
|
proc.stdout.on('data', (data) => { stdout += data; });
|
||||||
|
proc.stderr.on('data', (data) => { stderr += data; });
|
||||||
|
|
||||||
|
proc.on('close', (code) => {
|
||||||
|
if (code === 0) {
|
||||||
|
try {
|
||||||
|
resolve(JSON.parse(stdout));
|
||||||
|
} catch {
|
||||||
|
resolve(stdout);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
reject(new Error(stderr || `yt-dlp exited with code ${code}`));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Execute yt-dlp command with progress callback
|
||||||
|
*/
|
||||||
|
function ytdlpExec(url, args = [], onProgress, options = {}) {
|
||||||
|
const { cookiesPath } = options;
|
||||||
|
const finalArgs = addCookiesArg(args, cookiesPath);
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const proc = spawn(YTDLP_PATH, [...finalArgs, url]);
|
||||||
|
let stderr = '';
|
||||||
|
|
||||||
|
proc.stdout.on('data', (data) => {
|
||||||
|
const line = data.toString();
|
||||||
|
if (onProgress) {
|
||||||
|
const progressMatch = line.match(/\[download\]\s+(\d+\.?\d*)%/);
|
||||||
|
const etaMatch = line.match(/ETA\s+(\d+:\d+)/);
|
||||||
|
const speedMatch = line.match(/at\s+([\d.]+\w+\/s)/);
|
||||||
|
|
||||||
|
if (progressMatch) {
|
||||||
|
onProgress({
|
||||||
|
percent: parseFloat(progressMatch[1]),
|
||||||
|
eta: etaMatch ? etaMatch[1] : null,
|
||||||
|
speed: speedMatch ? speedMatch[1] : null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
proc.stderr.on('data', (data) => { stderr += data; });
|
||||||
|
|
||||||
|
proc.on('close', (code) => {
|
||||||
|
if (code === 0) {
|
||||||
|
resolve();
|
||||||
|
} else {
|
||||||
|
reject(new Error(stderr || `yt-dlp exited with code ${code}`));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const OUTPUT_DIR = process.env.OUTPUT_DIR || './output';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sanitize filename to remove invalid characters
|
||||||
|
*/
|
||||||
|
function sanitizeFilename(filename) {
|
||||||
|
return filename
|
||||||
|
.replace(/[<>:"/\\|?*]/g, '')
|
||||||
|
.replace(/\s+/g, '_')
|
||||||
|
.substring(0, 200);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if URL contains a playlist parameter
|
||||||
|
*/
|
||||||
|
function hasPlaylistParam(url) {
|
||||||
|
try {
|
||||||
|
const urlObj = new URL(url);
|
||||||
|
return urlObj.searchParams.has('list');
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract playlist URL if present in the URL
|
||||||
|
*/
|
||||||
|
function extractPlaylistUrl(url) {
|
||||||
|
const urlObj = new URL(url);
|
||||||
|
const listId = urlObj.searchParams.get('list');
|
||||||
|
if (listId) {
|
||||||
|
return `https://www.youtube.com/playlist?list=${listId}`;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get video/playlist info without downloading
|
||||||
|
*/
|
||||||
|
export async function getInfo(url, forcePlaylist = false, options = {}) {
|
||||||
|
try {
|
||||||
|
// If URL contains a playlist ID and we want to force playlist mode
|
||||||
|
const playlistUrl = extractPlaylistUrl(url);
|
||||||
|
const targetUrl = (forcePlaylist && playlistUrl) ? playlistUrl : url;
|
||||||
|
|
||||||
|
const info = await ytdlp(targetUrl, [
|
||||||
|
'--dump-single-json',
|
||||||
|
'--no-download',
|
||||||
|
'--no-warnings',
|
||||||
|
'--flat-playlist',
|
||||||
|
], options);
|
||||||
|
return info;
|
||||||
|
} catch (error) {
|
||||||
|
const enhancedError = enhanceYouTubeError(error);
|
||||||
|
const err = new Error(JSON.stringify(enhancedError));
|
||||||
|
err.isEnhanced = true;
|
||||||
|
err.details = enhancedError;
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if URL is a playlist
|
||||||
|
*/
|
||||||
|
export async function isPlaylist(url) {
|
||||||
|
const info = await getInfo(url);
|
||||||
|
return info._type === 'playlist';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Download a single video as MP3
|
||||||
|
*/
|
||||||
|
export async function downloadVideo(url, options = {}) {
|
||||||
|
const { outputDir = OUTPUT_DIR, onProgress, onDownloadProgress, cookiesPath } = options;
|
||||||
|
|
||||||
|
// Ensure output directory exists
|
||||||
|
if (!fs.existsSync(outputDir)) {
|
||||||
|
fs.mkdirSync(outputDir, { recursive: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Get video info first
|
||||||
|
const info = await ytdlp(url, [
|
||||||
|
'--dump-single-json',
|
||||||
|
'--no-download',
|
||||||
|
'--no-warnings',
|
||||||
|
], { cookiesPath });
|
||||||
|
|
||||||
|
const title = sanitizeFilename(info.title);
|
||||||
|
const outputPath = path.join(outputDir, `${title}.mp3`);
|
||||||
|
|
||||||
|
// Download and convert to MP3 with progress
|
||||||
|
await ytdlpExec(url, [
|
||||||
|
'--extract-audio',
|
||||||
|
'--audio-format', 'mp3',
|
||||||
|
'--audio-quality', '0',
|
||||||
|
'-o', outputPath,
|
||||||
|
'--no-warnings',
|
||||||
|
'--newline',
|
||||||
|
], (progress) => {
|
||||||
|
if (onDownloadProgress) {
|
||||||
|
onDownloadProgress({
|
||||||
|
...progress,
|
||||||
|
title: info.title,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}, { cookiesPath });
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
title: info.title,
|
||||||
|
duration: info.duration,
|
||||||
|
filePath: outputPath,
|
||||||
|
url: url,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
const enhancedError = enhanceYouTubeError(error);
|
||||||
|
const err = new Error(JSON.stringify(enhancedError));
|
||||||
|
err.isEnhanced = true;
|
||||||
|
err.details = enhancedError;
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Download all videos from a playlist as MP3
|
||||||
|
*/
|
||||||
|
export async function downloadPlaylist(url, options = {}) {
|
||||||
|
const { outputDir = OUTPUT_DIR, onProgress, onVideoComplete, onDownloadProgress, forcePlaylist = false, cookiesPath } = options;
|
||||||
|
|
||||||
|
// Ensure output directory exists
|
||||||
|
if (!fs.existsSync(outputDir)) {
|
||||||
|
fs.mkdirSync(outputDir, { recursive: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Get playlist info (force playlist mode if URL has list= param)
|
||||||
|
const info = await getInfo(url, forcePlaylist || hasPlaylistParam(url), { cookiesPath });
|
||||||
|
|
||||||
|
if (info._type !== 'playlist') {
|
||||||
|
// Single video, redirect to downloadVideo
|
||||||
|
const result = await downloadVideo(url, { ...options, onDownloadProgress });
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
playlistTitle: result.title,
|
||||||
|
videos: [result],
|
||||||
|
totalVideos: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const results = [];
|
||||||
|
const entries = info.entries || [];
|
||||||
|
|
||||||
|
console.log(`Playlist: ${info.title} (${entries.length} videos)`);
|
||||||
|
|
||||||
|
for (let i = 0; i < entries.length; i++) {
|
||||||
|
const entry = entries[i];
|
||||||
|
const videoUrl = entry.url || `https://www.youtube.com/watch?v=${entry.id}`;
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (onProgress) {
|
||||||
|
onProgress({ current: i + 1, total: entries.length, title: entry.title });
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`[${i + 1}/${entries.length}] Downloading: ${entry.title}`);
|
||||||
|
|
||||||
|
// Wrap progress callback to include playlist context
|
||||||
|
const wrappedProgress = onDownloadProgress ? (progress) => {
|
||||||
|
onDownloadProgress({
|
||||||
|
...progress,
|
||||||
|
videoIndex: i + 1,
|
||||||
|
totalVideos: entries.length,
|
||||||
|
playlistTitle: info.title,
|
||||||
|
});
|
||||||
|
} : undefined;
|
||||||
|
|
||||||
|
const result = await downloadVideo(videoUrl, { outputDir, onDownloadProgress: wrappedProgress, cookiesPath });
|
||||||
|
results.push(result);
|
||||||
|
|
||||||
|
if (onVideoComplete) {
|
||||||
|
onVideoComplete(result);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Failed to download ${entry.title}: ${error.message}`);
|
||||||
|
results.push({
|
||||||
|
success: false,
|
||||||
|
title: entry.title,
|
||||||
|
url: videoUrl,
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
playlistTitle: info.title,
|
||||||
|
videos: results,
|
||||||
|
totalVideos: entries.length,
|
||||||
|
successCount: results.filter(r => r.success).length,
|
||||||
|
failCount: results.filter(r => !r.success).length,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
const enhancedError = enhanceYouTubeError(error);
|
||||||
|
const err = new Error(JSON.stringify(enhancedError));
|
||||||
|
err.isEnhanced = true;
|
||||||
|
err.details = enhancedError;
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Smart download - detects if URL is video or playlist
|
||||||
|
*/
|
||||||
|
export async function download(url, options = {}) {
|
||||||
|
const { cookiesPath } = options;
|
||||||
|
// If URL contains list= parameter, treat it as a playlist
|
||||||
|
const isPlaylistUrl = hasPlaylistParam(url);
|
||||||
|
const info = await getInfo(url, isPlaylistUrl, { cookiesPath });
|
||||||
|
|
||||||
|
if (info._type === 'playlist') {
|
||||||
|
return downloadPlaylist(url, { ...options, forcePlaylist: true });
|
||||||
|
} else {
|
||||||
|
const result = await downloadVideo(url, options);
|
||||||
|
return {
|
||||||
|
success: true,
|
||||||
|
playlistTitle: null,
|
||||||
|
videos: [result],
|
||||||
|
totalVideos: 1,
|
||||||
|
successCount: 1,
|
||||||
|
failCount: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,226 +0,0 @@
|
|||||||
/**
|
|
||||||
* SMS Receiver Endpoint
|
|
||||||
* Receives SMS forwarded from Android app
|
|
||||||
* Stores latest SMS code in memory for auto-login script
|
|
||||||
*/
|
|
||||||
|
|
||||||
const express = require('express');
|
|
||||||
const fs = require('fs');
|
|
||||||
const path = require('path');
|
|
||||||
|
|
||||||
const app = express();
|
|
||||||
const PORT = process.env.SMS_RECEIVER_PORT || 4417;
|
|
||||||
|
|
||||||
// Store latest SMS codes in memory
|
|
||||||
const smsStore = {
|
|
||||||
latest: null,
|
|
||||||
history: []
|
|
||||||
};
|
|
||||||
|
|
||||||
// Middleware
|
|
||||||
app.use(express.json());
|
|
||||||
app.use(express.urlencoded({ extended: true }));
|
|
||||||
|
|
||||||
// Store SMS in file for persistence
|
|
||||||
const SMS_FILE = path.join(__dirname, '../.sms_codes.json');
|
|
||||||
|
|
||||||
function saveSMS(sms) {
|
|
||||||
smsStore.latest = sms;
|
|
||||||
smsStore.history.unshift(sms);
|
|
||||||
|
|
||||||
// Keep only last 10 SMS
|
|
||||||
if (smsStore.history.length > 10) {
|
|
||||||
smsStore.history = smsStore.history.slice(0, 10);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Persist to file
|
|
||||||
fs.writeFileSync(SMS_FILE, JSON.stringify(smsStore, null, 2));
|
|
||||||
}
|
|
||||||
|
|
||||||
function loadSMS() {
|
|
||||||
try {
|
|
||||||
if (fs.existsSync(SMS_FILE)) {
|
|
||||||
const data = fs.readFileSync(SMS_FILE, 'utf8');
|
|
||||||
Object.assign(smsStore, JSON.parse(data));
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
console.error('Error loading SMS file:', err);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load on startup
|
|
||||||
loadSMS();
|
|
||||||
|
|
||||||
/**
|
|
||||||
* POST /sms - Receive SMS from forwarder app
|
|
||||||
*
|
|
||||||
* Expected formats:
|
|
||||||
* 1. SMS Forwarder app:
|
|
||||||
* { from: "+33...", body: "Your code is 123456" }
|
|
||||||
*
|
|
||||||
* 2. Generic webhook:
|
|
||||||
* { sender: "...", message: "...", text: "..." }
|
|
||||||
*/
|
|
||||||
app.post('/sms', (req, res) => {
|
|
||||||
console.log('📱 SMS received:', JSON.stringify(req.body));
|
|
||||||
|
|
||||||
// Extract SMS data from various formats
|
|
||||||
const from = req.body.from || req.body.sender || req.body.number || 'unknown';
|
|
||||||
const body = req.body.body || req.body.message || req.body.text || '';
|
|
||||||
|
|
||||||
// Extract 6-digit code from message
|
|
||||||
const codeMatch = body.match(/\b(\d{6})\b/);
|
|
||||||
const code = codeMatch ? codeMatch[1] : null;
|
|
||||||
|
|
||||||
const sms = {
|
|
||||||
from,
|
|
||||||
body,
|
|
||||||
code,
|
|
||||||
timestamp: new Date().toISOString(),
|
|
||||||
raw: req.body
|
|
||||||
};
|
|
||||||
|
|
||||||
saveSMS(sms);
|
|
||||||
|
|
||||||
console.log(`✅ SMS stored: ${from} → Code: ${code || 'none'}`);
|
|
||||||
|
|
||||||
res.json({
|
|
||||||
success: true,
|
|
||||||
code,
|
|
||||||
message: 'SMS received and stored'
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
/**
|
|
||||||
* GET /sms/latest - Get latest SMS code
|
|
||||||
*/
|
|
||||||
app.get('/sms/latest', (req, res) => {
|
|
||||||
if (!smsStore.latest) {
|
|
||||||
return res.status(404).json({
|
|
||||||
success: false,
|
|
||||||
message: 'No SMS received yet'
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
res.json({
|
|
||||||
success: true,
|
|
||||||
...smsStore.latest
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
/**
|
|
||||||
* GET /sms/code - Get latest verification code only
|
|
||||||
*/
|
|
||||||
app.get('/sms/code', (req, res) => {
|
|
||||||
if (!smsStore.latest || !smsStore.latest.code) {
|
|
||||||
return res.status(404).json({
|
|
||||||
success: false,
|
|
||||||
message: 'No verification code found'
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
res.json({
|
|
||||||
success: true,
|
|
||||||
code: smsStore.latest.code,
|
|
||||||
timestamp: smsStore.latest.timestamp
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
/**
|
|
||||||
* GET /sms/wait - Wait for new SMS code (long-polling)
|
|
||||||
* Waits up to 60 seconds for a new code
|
|
||||||
*/
|
|
||||||
app.get('/sms/wait', async (req, res) => {
|
|
||||||
const maxWait = 60000; // 60 seconds
|
|
||||||
const checkInterval = 1000; // 1 second
|
|
||||||
const startTime = Date.now();
|
|
||||||
const afterTimestamp = req.query.after || new Date(0).toISOString();
|
|
||||||
|
|
||||||
const checkForNewSMS = () => {
|
|
||||||
if (smsStore.latest &&
|
|
||||||
smsStore.latest.code &&
|
|
||||||
smsStore.latest.timestamp > afterTimestamp) {
|
|
||||||
return {
|
|
||||||
success: true,
|
|
||||||
code: smsStore.latest.code,
|
|
||||||
timestamp: smsStore.latest.timestamp,
|
|
||||||
from: smsStore.latest.from
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
};
|
|
||||||
|
|
||||||
// Check immediately
|
|
||||||
let result = checkForNewSMS();
|
|
||||||
if (result) {
|
|
||||||
return res.json(result);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Poll every second
|
|
||||||
const interval = setInterval(() => {
|
|
||||||
result = checkForNewSMS();
|
|
||||||
|
|
||||||
if (result) {
|
|
||||||
clearInterval(interval);
|
|
||||||
return res.json(result);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (Date.now() - startTime > maxWait) {
|
|
||||||
clearInterval(interval);
|
|
||||||
return res.status(408).json({
|
|
||||||
success: false,
|
|
||||||
message: 'Timeout waiting for SMS'
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}, checkInterval);
|
|
||||||
|
|
||||||
// Cleanup on connection close
|
|
||||||
req.on('close', () => clearInterval(interval));
|
|
||||||
});
|
|
||||||
|
|
||||||
/**
|
|
||||||
* DELETE /sms/clear - Clear SMS history
|
|
||||||
*/
|
|
||||||
app.delete('/sms/clear', (req, res) => {
|
|
||||||
smsStore.latest = null;
|
|
||||||
smsStore.history = [];
|
|
||||||
|
|
||||||
if (fs.existsSync(SMS_FILE)) {
|
|
||||||
fs.unlinkSync(SMS_FILE);
|
|
||||||
}
|
|
||||||
|
|
||||||
res.json({
|
|
||||||
success: true,
|
|
||||||
message: 'SMS history cleared'
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
/**
|
|
||||||
* GET /health - Health check
|
|
||||||
*/
|
|
||||||
app.get('/health', (req, res) => {
|
|
||||||
res.json({
|
|
||||||
success: true,
|
|
||||||
uptime: process.uptime(),
|
|
||||||
smsCount: smsStore.history.length,
|
|
||||||
latestSMS: smsStore.latest ? {
|
|
||||||
timestamp: smsStore.latest.timestamp,
|
|
||||||
hasCode: !!smsStore.latest.code
|
|
||||||
} : null
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
app.listen(PORT, '0.0.0.0', () => {
|
|
||||||
console.log('');
|
|
||||||
console.log('📱 SMS Receiver started!');
|
|
||||||
console.log(` Port: ${PORT}`);
|
|
||||||
console.log(` Webhook URL: http://YOUR_SERVER_IP:${PORT}/sms`);
|
|
||||||
console.log('');
|
|
||||||
console.log('📋 Endpoints:');
|
|
||||||
console.log(' POST /sms - Receive SMS');
|
|
||||||
console.log(' GET /sms/latest - Get latest SMS');
|
|
||||||
console.log(' GET /sms/code - Get latest code');
|
|
||||||
console.log(' GET /sms/wait - Wait for new code');
|
|
||||||
console.log(' DELETE /sms/clear - Clear history');
|
|
||||||
console.log('');
|
|
||||||
});
|
|
||||||
@ -1,48 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
# Test script for full auto-login with SMS
|
|
||||||
|
|
||||||
echo "🚀 Testing Full Auto-Login with SMS Forwarding"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Check SMS receiver is running
|
|
||||||
echo "1. Checking SMS Receiver..."
|
|
||||||
SMS_STATUS=$(curl -s http://localhost:4417/health 2>/dev/null)
|
|
||||||
if [ $? -eq 0 ]; then
|
|
||||||
echo " ✅ SMS Receiver running"
|
|
||||||
else
|
|
||||||
echo " ❌ SMS Receiver not running - starting..."
|
|
||||||
cd /home/debian/videotomp3transcriptor
|
|
||||||
node src/sms_receiver.js &
|
|
||||||
sleep 3
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "2. Testing SMS endpoint..."
|
|
||||||
curl -s -X POST http://localhost:4417/sms \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"from":"Test","body":"Test code 999999"}' > /dev/null
|
|
||||||
|
|
||||||
CODE=$(curl -s http://localhost:4417/sms/code | grep -o '"code":"[0-9]*"' | cut -d'"' -f4)
|
|
||||||
if [ "$CODE" = "999999" ]; then
|
|
||||||
echo " ✅ SMS endpoint working (code: $CODE)"
|
|
||||||
else
|
|
||||||
echo " ❌ SMS endpoint error"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "3. Ready to test auto-login!"
|
|
||||||
echo ""
|
|
||||||
echo " Run: export DISPLAY=:99"
|
|
||||||
echo " Run: cd /home/debian/videotomp3transcriptor"
|
|
||||||
echo " Run: python3 src/python/auto_login_full_auto.py"
|
|
||||||
echo ""
|
|
||||||
echo " The script will:"
|
|
||||||
echo " - Navigate to Google login"
|
|
||||||
echo " - Enter email & phone"
|
|
||||||
echo " - WAIT for SMS (you'll receive on phone)"
|
|
||||||
echo " - SMS Forwarder sends to server"
|
|
||||||
echo " - Script reads code automatically"
|
|
||||||
echo " - Completes login!"
|
|
||||||
echo ""
|
|
||||||
echo "📱 Make sure SMS Forwarder is configured and running!"
|
|
||||||
@ -1,75 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
# Final test: YouTube download with logged-in cookies + PO Token
|
|
||||||
|
|
||||||
echo "🧪 Testing YouTube Download (Cookies + PO Token)"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
COOKIES_FILE="/home/debian/videotomp3transcriptor/youtube-cookies.txt"
|
|
||||||
TEST_URL="https://www.youtube.com/watch?v=fukChj4eh-Q"
|
|
||||||
|
|
||||||
# Check cookies exist
|
|
||||||
if [ ! -f "$COOKIES_FILE" ]; then
|
|
||||||
echo "❌ Cookies file not found: $COOKIES_FILE"
|
|
||||||
echo " Waiting for upload from PC..."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "✅ Cookies file found"
|
|
||||||
echo " Size: $(wc -c < $COOKIES_FILE) bytes"
|
|
||||||
echo " Lines: $(wc -l < $COOKIES_FILE) lines"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Check for logged-in cookies
|
|
||||||
if grep -q "SID\|SSID\|HSID\|SAPISID" "$COOKIES_FILE"; then
|
|
||||||
echo "✅ Logged-in cookies detected!"
|
|
||||||
else
|
|
||||||
echo "⚠️ Warning: May be guest cookies"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "🎯 Test 1: Info extraction only"
|
|
||||||
echo "----------------------------------------"
|
|
||||||
yt-dlp \
|
|
||||||
--cookies "$COOKIES_FILE" \
|
|
||||||
--extractor-args "youtube:player_client=mweb" \
|
|
||||||
--skip-download \
|
|
||||||
--print "%(title)s [%(duration)s sec]" \
|
|
||||||
"$TEST_URL"
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "🎯 Test 2: List available formats"
|
|
||||||
echo "----------------------------------------"
|
|
||||||
yt-dlp \
|
|
||||||
--cookies "$COOKIES_FILE" \
|
|
||||||
--extractor-args "youtube:player_client=mweb" \
|
|
||||||
--list-formats \
|
|
||||||
"$TEST_URL" | head -20
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "🎯 Test 3: Download audio (best quality)"
|
|
||||||
echo "----------------------------------------"
|
|
||||||
yt-dlp \
|
|
||||||
--cookies "$COOKIES_FILE" \
|
|
||||||
--extractor-args "youtube:player_client=mweb" \
|
|
||||||
--format "bestaudio" \
|
|
||||||
--extract-audio \
|
|
||||||
--audio-format mp3 \
|
|
||||||
--output "/tmp/test_%(id)s.%(ext)s" \
|
|
||||||
"$TEST_URL"
|
|
||||||
|
|
||||||
if [ $? -eq 0 ]; then
|
|
||||||
echo ""
|
|
||||||
echo "=" * 60
|
|
||||||
echo "🎉 SUCCESS! YouTube download working!"
|
|
||||||
echo "=" * 60
|
|
||||||
echo ""
|
|
||||||
echo "✅ Cookies: Working"
|
|
||||||
echo "✅ PO Token: Active"
|
|
||||||
echo "✅ Download: Successful"
|
|
||||||
echo ""
|
|
||||||
echo "💡 Next: Integrate into music service API"
|
|
||||||
else
|
|
||||||
echo ""
|
|
||||||
echo "❌ Download failed"
|
|
||||||
echo " Check errors above"
|
|
||||||
fi
|
|
||||||
Loading…
Reference in New Issue
Block a user