groovy-zilean/PRODUCTION_ROADMAP.md

Groovy-Zilean Production Roadmap

Goal: Transform groovy-zilean from a personal project into a production-ready Discord music bot with a web dashboard.

Philosophy: Production-quality architecture with manageable complexity for a solo developer. No overkill, no shortcuts.

---

Table of Contents

1. Architecture Overview
2. Tech Stack Decisions
3. Development Phases
4. Python Environment Setup
5. Why This Approach
6. Quick Reference

---

Architecture Overview

The Winning Design: Database-Mediated Architecture

┌─────────────┐    HTTP/SSE  ┌──────────────┐
│   Browser   │◄────────────►│   FastAPI    │
│ (HTMX HTML) │              │   Backend    │
└─────────────┘              └──────┬───────┘
                                    │
                             ┌──────▼───────┐         ┌─────────────┐
                             │  PostgreSQL  │◄────────│ Discord Bot │
                             │   Database   │         │ (discord.py)│
                             └──────────────┘         └─────────────┘

Key Principles

1. Decoupled Services

   • Bot and web can restart independently
   • No tight coupling via IPC
   • Database is the single source of truth

2. Simple Frontend (Initially)

   • HTMX for interactivity (no build step)
   • Jinja2 templates (server-side rendering)
   • Tailwind CSS via CDN (beautiful, no npm)
   • Optional: Upgrade to React later when ready

3. Production-Ready Backend

   • PostgreSQL for reliability
   • FastAPI for modern Python web
   • Discord OAuth2 for authentication
   • Connection pooling, proper error handling

---

Tech Stack Decisions

Backend

Component	Choice	Why
Bot Framework	discord.py 2.6.4+	Industry standard, hybrid commands
Web Framework	FastAPI	Modern, async, auto-docs, large community
Database	PostgreSQL 14+	Production-ready, ACID compliance, better than SQLite
Music Extraction	yt-dlp	Actively maintained, multi-platform support
Auth	Discord OAuth2	Native Discord integration
ORM	SQLAlchemy (optional)	Or use asyncpg directly for simplicity

Frontend (Phase 1)

Component	Choice	Why
Templates	Jinja2	Server-side rendering, no build step
Interactivity	HTMX	Modern interactivity without React complexity
Styling	Tailwind CSS (CDN)	Beautiful UI, no build process
Real-time	Server-Sent Events	Simple polling/updates, no WebSocket complexity

Frontend (Phase 2 - Optional)

Component	Choice	Why
Framework	React 18	If you need more complex UI later
Real-time	WebSocket	For true real-time when needed
State	Zustand/Context	Simpler than Redux

Infrastructure

Component	Choice	Why
Python Version	3.11 or 3.12	Modern features, better performance
Environment	venv	Isolated dependencies, no PATH conflicts
Process Manager	systemd	Reliable, built into Linux
Reverse Proxy	nginx	Standard, handles SSL, static files

---

Development Phases

Phase 0: Current State ✅

• Working Discord bot with music playback
• SQLite database
• Hybrid commands (slash + prefix)
• 17 audio effects
• Queue, loop, shuffle functionality
• Spotify integration

Phase 1: Quick Wins (1-2 hours)

Goal: Immediate improvements for better UX

Tasks:

1. Lower default volume from 100% to 25%

   • Change default in queue.py:119
   • Scale volume command display (user sees 0-200%, internally 0-50%)
   • Prevents earrape for new users

2. Add .mp3 and .mp4 file support

   • Extend translate.py to detect direct file URLs
   • Support HTTP/HTTPS links to audio files
   • Validate file type before processing

Files to modify:

• cogs/music/queue.py
• cogs/music/translate.py

---

Phase 2: Code Refactoring (4-6 hours)

Goal: Clean, maintainable, documented codebase

2.1 Database Abstraction

Create cogs/music/db_manager.py:

• Database connection class with context manager
• Connection pooling preparation
• Centralize all SQL queries
• Remove scattered sqlite3.connect() calls

2.2 Configuration Management

Update config.py:

• Use environment variables for secrets
• Create .env.example template
• Remove hardcoded credentials
• Add validation for required config

2.3 Code Organization

groovy-zilean/
├── bot/
│   ├── __init__.py
│   ├── bot.py              # Main bot class
│   └── cogs/
│       └── music/
│           ├── __init__.py
│           ├── commands.py     # User-facing commands
│           ├── player.py       # Playback logic
│           ├── queue.py        # Queue management
│           ├── effects.py      # Audio effects
│           ├── db_manager.py   # Database abstraction
│           └── translate.py    # URL/playlist parsing
├── web/
│   ├── __init__.py         # (Future web app)
├── shared/
│   ├── __init__.py
│   ├── config.py           # Shared configuration
│   └── models.py           # Data models
├── main.py                 # Entry point
├── requirements.txt
├── .env.example
└── README.md

2.4 Error Handling & Logging

• Wrap all commands in try/except
• User-friendly error messages
• Proper logging setup (rotating file logs)
• Debug mode toggle

2.5 Type Hints & Documentation

• Add type hints to all functions
• Docstrings for all classes/methods
• Inline comments for complex logic

Expected outcome:

• Easy to navigate codebase
• No secrets in code
• Consistent patterns throughout

---

Phase 3: PostgreSQL Migration (3-4 hours)

Goal: Production-ready database layer

3.1 Local PostgreSQL Setup

# Install PostgreSQL
sudo apt update
sudo apt install postgresql postgresql-contrib

# Create database and user
sudo -u postgres psql
CREATE DATABASE groovy_zilean;
CREATE USER groovy WITH PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE groovy_zilean TO groovy;

3.2 Database Schema Design

-- servers table
CREATE TABLE servers (
    server_id BIGINT PRIMARY KEY,
    is_playing BOOLEAN DEFAULT FALSE,
    song_name TEXT,
    song_url TEXT,
    song_thumbnail TEXT,
    loop_mode VARCHAR(10) DEFAULT 'off',
    volume INTEGER DEFAULT 25,  -- NEW default!
    effect VARCHAR(20) DEFAULT 'none',
    song_start_time DOUBLE PRECISION DEFAULT 0,
    song_duration INTEGER DEFAULT 0,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- songs/queue table
CREATE TABLE songs (
    id SERIAL PRIMARY KEY,
    server_id BIGINT NOT NULL REFERENCES servers(server_id) ON DELETE CASCADE,
    song_link TEXT,
    queued_by TEXT,
    position INTEGER NOT NULL,
    title TEXT,
    thumbnail TEXT,
    duration INTEGER,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(server_id, position)
);

-- Future: users table for web auth
CREATE TABLE users (
    discord_id BIGINT PRIMARY KEY,
    username TEXT,
    avatar TEXT,
    access_token TEXT,
    refresh_token TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    last_login TIMESTAMP
);

-- Future: permissions table
CREATE TABLE permissions (
    id SERIAL PRIMARY KEY,
    server_id BIGINT REFERENCES servers(server_id),
    user_id BIGINT,
    role_id BIGINT,
    can_play BOOLEAN DEFAULT TRUE,
    can_skip BOOLEAN DEFAULT FALSE,
    can_clear BOOLEAN DEFAULT FALSE,
    can_modify_settings BOOLEAN DEFAULT FALSE
);

3.3 Migration Script

Create scripts/migrate_to_postgres.py:

• Read all data from SQLite
• Insert into PostgreSQL
• Validate migration
• Backup SQLite file

3.4 Update Database Code

Replace all sqlite3 calls with asyncpg or psycopg3:

# Old (SQLite)
conn = sqlite3.connect(db_path)
cursor = conn.cursor()

# New (PostgreSQL with asyncpg)
async with pool.acquire() as conn:
    result = await conn.fetch("SELECT * FROM servers WHERE server_id = $1", server_id)

3.5 Connection Pooling

# In bot startup
self.db_pool = await asyncpg.create_pool(
    host='localhost',
    database='groovy_zilean',
    user='groovy',
    password=os.getenv('DB_PASSWORD'),
    min_size=5,
    max_size=20
)

Expected outcome:

• Reliable database with ACID guarantees
• Better concurrent access handling
• Ready for multi-server production load

---

Phase 4: Web Dashboard (20-30 hours)

Goal: User-friendly web interface for bot control

4.1 FastAPI Backend Setup

Project structure:

web/
├── __init__.py
├── main.py              # FastAPI app
├── routes/
│   ├── __init__.py
│   ├── auth.py          # Discord OAuth2
│   ├── servers.py       # Server list/select
│   └── playback.py      # Queue/controls
├── templates/
│   ├── base.html
│   ├── index.html
│   ├── dashboard.html
│   └── components/
│       ├── queue.html
│       └── controls.html
├── static/
│   ├── css/
│   └── js/
└── dependencies.py      # Auth dependencies

Core dependencies:

pip install fastapi uvicorn jinja2 python-multipart httpx

4.2 Discord OAuth2 Authentication

Flow:

1. User clicks "Login with Discord"
2. Redirect to Discord OAuth
3. Discord redirects back with code
4. Exchange code for token
5. Fetch user info + guilds
6. Create session
7. Show dashboard with user's servers

Implementation:

# routes/auth.py
@router.get("/login")
async def login():
    # Redirect to Discord OAuth
    discord_auth_url = (
        f"https://discord.com/api/oauth2/authorize"
        f"?client_id={DISCORD_CLIENT_ID}"
        f"&redirect_uri={REDIRECT_URI}"
        f"&response_type=code"
        f"&scope=identify guilds"
    )
    return RedirectResponse(discord_auth_url)

@router.get("/callback")
async def callback(code: str):
    # Exchange code for token
    # Fetch user info
    # Create session
    # Redirect to dashboard

4.3 HTMX Frontend

Example dashboard with HTMX:

<!-- templates/dashboard.html -->
<div class="container">
    <!-- Server Selector -->
    <select hx-get="/api/servers/{value}/queue"
            hx-target="#queue-container"
            hx-trigger="change">
        {% for server in user_servers %}
            <option value="{{ server.id }}">{{ server.name }}</option>
        {% endfor %}
    </select>

    <!-- Now Playing (auto-updates every 5s) -->
    <div id="now-playing"
         hx-get="/api/now-playing"
         hx-trigger="every 5s">
        <!-- Server renders this -->
    </div>

    <!-- Queue (auto-updates) -->
    <div id="queue-container"
         hx-get="/api/queue"
         hx-trigger="every 3s">
        <!-- Queue items here -->
    </div>

    <!-- Controls -->
    <div class="controls">
        <button hx-post="/api/skip"
                hx-target="#queue-container">
            ⏭️ Skip
        </button>

        <input type="range"
               min="0" max="200"
               hx-post="/api/volume"
               hx-trigger="change"
               hx-vals='{"volume": this.value}'>
    </div>

    <!-- Add Song Form -->
    <form hx-post="/api/play"
          hx-target="#queue-container">
        <input name="query" placeholder="YouTube URL or search">
        <button type="submit">Add to Queue</button>
    </form>
</div>

Why HTMX is perfect here:

• No JavaScript needed for interactivity
• Server renders everything (simpler)
• Auto-updates with hx-trigger="every Xs"
• Progressive enhancement (works without JS)

4.4 API Endpoints

Read endpoints:

GET  /api/servers              # User's servers (with bot)
GET  /api/servers/{id}/queue   # Current queue
GET  /api/servers/{id}/status  # Now playing, volume, etc.

Write endpoints:

POST /api/servers/{id}/play    # Add song (body: {query: "..."})
POST /api/servers/{id}/skip    # Skip current song
POST /api/servers/{id}/volume  # Set volume (body: {volume: 150})
POST /api/servers/{id}/effect  # Set effect (body: {effect: "nightcore"})
POST /api/servers/{id}/loop    # Set loop mode (body: {mode: "queue"})
POST /api/servers/{id}/shuffle # Shuffle queue
POST /api/servers/{id}/clear   # Clear queue

Implementation example:

# routes/playback.py
@router.post("/api/servers/{server_id}/skip")
async def skip_song(
    server_id: int,
    user: User = Depends(get_current_user),
    db = Depends(get_db)
):
    # 1. Check user is in server
    if server_id not in user.guild_ids:
        raise HTTPException(403, "Not in this server")

    # 2. Check permissions (future)
    # if not has_permission(user, server_id, "can_skip"):
    #     raise HTTPException(403, "No permission")

    # 3. Write command to database
    await db.execute(
        "INSERT INTO commands (server_id, action, user_id) VALUES ($1, $2, $3)",
        server_id, "skip", user.id
    )

    # 4. Return updated queue
    return await get_queue(server_id, db)

4.5 Bot Integration (Command Processing)

Add to bot:

# In bot.py or new cogs/web_commands.py
@tasks.loop(seconds=1)
async def process_web_commands(self):
    """Process commands from web dashboard"""
    async with self.db_pool.acquire() as conn:
        # Fetch unprocessed commands
        commands = await conn.fetch(
            "SELECT * FROM commands WHERE processed = FALSE"
        )

        for cmd in commands:
            server_id = cmd['server_id']
            action = cmd['action']
            data = cmd['data']

            guild = self.get_guild(server_id)
            if not guild or not guild.voice_client:
                continue

            # Execute command
            if action == "skip":
                guild.voice_client.stop()
            elif action == "volume":
                # Set volume in database, next song picks it up
                await queue.set_volume(server_id, data['volume'])
            elif action == "play":
                # Queue song from web
                # ... (use existing play logic)

            # Mark as processed
            await conn.execute(
                "UPDATE commands SET processed = TRUE WHERE id = $1",
                cmd['id']
            )

4.6 Permissions System

Basic implementation:

# Check if user can control bot
async def can_control_bot(user_id: int, server_id: int, action: str) -> bool:
    # Check if user is in voice channel with bot
    # Check if user has DJ role (configurable per server)
    # Check specific permission for action
    # Default: anyone in VC can control
    pass

Advanced (Phase 5):

• Role-based permissions
• User-specific permissions
• Configurable via web dashboard

Expected outcome:

• Beautiful, functional web dashboard
• Discord OAuth login
• Real-time queue display
• Full playback control from browser
• Works on mobile

---

Phase 5: Permissions & Production (4-6 hours)

Goal: Production-ready deployment

5.1 Permission System

• DJ role configuration
• Per-server permission settings
• Web UI for permission management

5.2 Rate Limiting

from slowapi import Limiter
limiter = Limiter(key_func=get_remote_address)

@app.post("/api/play")
@limiter.limit("10/minute")  # Max 10 songs per minute
async def play_song(...):
    ...

5.3 Logging & Monitoring

import logging
from logging.handlers import RotatingFileHandler

# Setup logging
handler = RotatingFileHandler(
    'logs/bot.log',
    maxBytes=10_000_000,  # 10MB
    backupCount=5
)
logging.basicConfig(
    handlers=[handler],
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

5.4 Systemd Services

# /etc/systemd/system/groovy-bot.service
[Unit]
Description=Groovy Zilean Discord Bot
After=network.target postgresql.service

[Service]
Type=simple
User=groovy
WorkingDirectory=/home/groovy/groovy-zilean
Environment="PATH=/home/groovy/groovy-zilean/venv/bin"
ExecStart=/home/groovy/groovy-zilean/venv/bin/python main.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/groovy-web.service
[Unit]
Description=Groovy Zilean Web Dashboard
After=network.target postgresql.service

[Service]
Type=simple
User=groovy
WorkingDirectory=/home/groovy/groovy-zilean
Environment="PATH=/home/groovy/groovy-zilean/venv/bin"
ExecStart=/home/groovy/groovy-zilean/venv/bin/uvicorn web.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

5.5 Nginx Configuration

server {
    listen 80;
    server_name groovy.yourdomain.com;

    # Redirect to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name groovy.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/groovy.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/groovy.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /static {
        alias /home/groovy/groovy-zilean/web/static;
        expires 30d;
    }
}

5.6 Database Backups

#!/bin/bash
# scripts/backup_db.sh

BACKUP_DIR="/home/groovy/backups"
DATE=$(date +%Y%m%d_%H%M%S)

pg_dump groovy_zilean > "$BACKUP_DIR/groovy_zilean_$DATE.sql"

# Keep only last 7 days
find $BACKUP_DIR -name "groovy_zilean_*.sql" -mtime +7 -delete

Add to crontab:

0 2 * * * /home/groovy/groovy-zilean/scripts/backup_db.sh

5.7 Environment Variables

# .env (NEVER commit this!)
DISCORD_TOKEN=your_bot_token_here
DISCORD_CLIENT_ID=your_client_id
DISCORD_CLIENT_SECRET=your_client_secret
SPOTIFY_CLIENT_ID=your_spotify_id
SPOTIFY_CLIENT_SECRET=your_spotify_secret
DB_PASSWORD=your_db_password
SECRET_KEY=random_secret_for_sessions
ENVIRONMENT=production

Expected outcome:

• Production-ready deployment
• Automatic restarts on failure
• HTTPS enabled
• Automated backups
• Proper logging

---

Python Environment Setup

Avoiding Python Version Hell

Step 1: Install Python 3.12

# On Debian/Ubuntu
sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev

# Verify installation
python3.12 --version

Step 2: Create Virtual Environment

cd ~/coding/groovy-zilean

# Create venv (only once)
python3.12 -m venv venv

# Activate (every time you work on project)
source venv/bin/activate

# Your prompt should change to show (venv)

Step 3: Install Dependencies

# Make sure venv is activated!
pip install --upgrade pip
pip install -r requirements.txt

Step 4: Add to .gitignore

venv/
__pycache__/
*.pyc
.env
*.db
logs/

Helpful Aliases (add to ~/.bashrc)

alias groovy='cd ~/coding/groovy-zilean && source venv/bin/activate'

Then just type groovy to activate your environment!

---

Why This Approach?

Rejected: discord-ext-ipc + Quart

❌ discord-ext-ipc is unmaintained (last update 2+ years ago) ❌ Tight coupling between bot and web ❌ Both processes must run together ❌ Quart less popular than FastAPI ❌ Still need polling for real-time updates

Rejected: FastAPI + React + Redis + WebSocket

❌ Overkill for solo developer ❌ Too many moving parts (4+ services) ❌ npm/node_modules complexity ❌ Requires learning React well ❌ WebSocket complexity for minimal gain

Chosen: FastAPI + PostgreSQL + HTMX ✅

✅ Production-quality architecture ✅ Decoupled services (independent restarts) ✅ Modern, well-maintained tools ✅ No frontend build step (initially) ✅ Easy upgrade path to React later ✅ Manageable complexity for solo dev ✅ Database as source of truth ✅ All Python backend

---

Quick Reference

Daily Development Workflow

# Activate environment
cd ~/coding/groovy-zilean
source venv/bin/activate

# Run bot
python main.py

# Run web (in another terminal)
source venv/bin/activate
uvicorn web.main:app --reload --port 8000

# Run tests (future)
pytest

# Database migrations (future)
alembic upgrade head

Project Commands

# Install new package
pip install package_name
pip freeze > requirements.txt

# Database
psql groovy_zilean  # Connect to database
pg_dump groovy_zilean > backup.sql  # Backup

# Systemd
sudo systemctl start groovy-bot
sudo systemctl status groovy-bot
sudo journalctl -u groovy-bot -f  # View logs

File Locations

• Bot entry point: main.py
• Config: shared/config.py + .env
• Database: PostgreSQL (not file-based)
• Logs: logs/bot.log, logs/web.log
• Web templates: web/templates/

---

Success Metrics

By the end of this roadmap, you'll have:

✅ Clean, maintainable codebase ✅ Production-ready database ✅ Web dashboard with Discord auth ✅ Mobile-friendly interface ✅ Automated deployment ✅ Backup system ✅ Proper error handling & logging ✅ Permission system ✅ Rate limiting ✅ No earrape (25% default volume!) ✅ .mp3/.mp4 file support

Most importantly: A bot that real servers can use, that you can maintain solo, and that you're proud of!

---

Next Steps

1. Today: Quick wins (volume + file support)
2. This week: Refactoring
3. Next week: PostgreSQL migration
4. Week after: Web dashboard MVP
5. Final week: Production deployment

Let's build something awesome! 🎵⏱️