Configuration¶

Configure Scrapazoid to meet your needs.

Environment Variables¶

Create a .env file in the project root:

# Flask Configuration
FLASK_ENV=development
SECRET_KEY=your-secret-key-here-change-in-production

# Database
DATABASE_URL=sqlite:///scrapazoid_dev.db

# Execution Settings
MAX_EXECUTION_TIME=300
MAX_CONCURRENT_EXECUTIONS_PER_USER=3

# Screenshot Settings
SCREENSHOT_RETENTION_DAYS=7

Configuration Options¶

MAX_EXECUTION_TIME¶

Default: 300 (5 minutes)

Maximum time in seconds a script can run before being terminated.

MAX_EXECUTION_TIME=600  # 10 minutes

Warning

Setting this too high may allow scripts to consume resources indefinitely if they hang.

MAX_CONCURRENT_EXECUTIONS_PER_USER¶

Default: 3

Maximum number of scripts a single user can run simultaneously.

MAX_CONCURRENT_EXECUTIONS_PER_USER=5

SCREENSHOT_RETENTION_DAYS¶

Default: 7

Number of days to keep screenshots before automatic cleanup.

SCREENSHOT_RETENTION_DAYS=30  # Keep for 30 days

DATABASE_URL¶

Database connection string.

SQLite (Development):

DATABASE_URL=sqlite:///scrapazoid_dev.db

PostgreSQL (Production):

DATABASE_URL=postgresql://user:password@localhost/scrapazoid

SECRET_KEY¶

Flask secret key for session security.

SECRET_KEY=your-very-secret-key-here

Production

Always use a strong, random secret key in production!

Generate one with:

python -c "import secrets; print(secrets.token_hex(32))"

Configuration Files¶

config.py¶

Main configuration file with three environments:

DevelopmentConfig - For local development
ProductionConfig - For production deployment
TestingConfig - For running tests

Select environment:

config_name = os.environ.get('FLASK_ENV', 'development')
app = create_app(config_name)

Production Configuration¶

PostgreSQL Setup¶

Install PostgreSQL:

sudo apt-get install postgresql postgresql-contrib

Create database:

sudo -u postgres createdb scrapazoid
sudo -u postgres createuser scrapazoid_user -P

Configure .env:

DATABASE_URL=postgresql://scrapazoid_user:password@localhost/scrapazoid

Gunicorn Setup¶

Install Gunicorn:
```
pip install gunicorn eventlet
```

Run with Gunicorn:

gunicorn -k eventlet -w 1 --bind 0.0.0.0:5000 run:app

Nginx Reverse Proxy¶

server {
    listen 80;
    server_name scrapazoid.example.com;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Docker Configuration¶

docker-compose.yml¶

version: '3.8'

services:
  web:
    build: .
    ports:
      - "5000:5000"
    environment:
      - FLASK_ENV=production
      - DATABASE_URL=postgresql://postgres:password@db/scrapazoid
      - SECRET_KEY=${SECRET_KEY}
    depends_on:
      - db
    volumes:
      - ./app/static/screenshots:/app/app/static/screenshots

  db:
    image: postgres:15
    environment:
      - POSTGRES_DB=scrapazoid
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Security Best Practices¶

Change default SECRET_KEY
Use HTTPS in production (SESSION_COOKIE_SECURE=True)
Restrict database user permissions
Use environment variables for sensitive data
Regular backups
Keep dependencies updated

Performance Tuning¶

Database¶

For high-traffic deployments:

SQLALCHEMY_POOL_SIZE = 10
SQLALCHEMY_MAX_OVERFLOW = 20
SQLALCHEMY_POOL_RECYCLE = 3600

SocketIO¶

For better concurrency:

SOCKETIO_ASYNC_MODE = 'eventlet'  # or 'gevent'

Monitoring¶

Logging¶

Configure logging in production:

import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    'scrapazoid.log',
    maxBytes=10485760,  # 10MB
    backupCount=10
)
handler.setLevel(logging.INFO)
app.logger.addHandler(handler)