trackpull/docs/job-management.md

Job Management

Overview

Downloads run as background jobs. Each job is tracked in-memory during execution and persisted to the database on completion. The frontend polls for status updates every ~1–3 seconds.

---

Job Lifecycle

User Request → Create Job (queued) → Spawn Thread → status: running
    → [completed | failed | cancelled] → Upsert to DB

1. A job record is created in the in-memory jobs dict with status: queued.
2. A Python thread is spawned to run the download function.
3. The thread updates status, output, and return_code in-memory as it runs.
4. On finish (success, failure, or cancellation), the job is upserted into SQLite.

---

In-Memory Job Structure

{
    "id": str,           # UUID
    "user_id": str,      # Owner
    "urls": list[str],   # Spotify URLs
    "options": dict,     # Download parameters
    "status": str,       # queued | running | completed | failed | cancelled
    "output": list[str], # Log lines (capped at 500)
    "command": str,      # CLI command string (Votify jobs only)
    "return_code": int,  # Process exit code
    "process": Popen,    # Subprocess handle (for cancellation)
    "created_at": float, # Unix timestamp
}

process is only present while the job is running; it is not persisted to the database.

---

Output Streaming

• The subprocess stdout is read line-by-line in the runner thread.
• Each line is appended to job["output"].
• The list is capped at 500 entries (oldest lines are dropped first).
• The frontend reads output via GET /api/jobs/<id> and displays the log incrementally.

---

Cancellation

1. Frontend calls POST /api/jobs/<id>/cancel.
2. Job status is set to cancelled in-memory.
3. process.terminate() is called on the Popen handle.
4. The runner thread detects the cancellation flag and logs a cancellation message.
5. The job is then upserted to the database with status: cancelled.

---

Job Expiry

A background daemon thread runs hourly and calls delete_jobs_older_than(days) where days comes from the job_expiry_days setting (default: 30). This removes old job records from the database but does not delete downloaded files.

---

API Endpoints

Endpoint	Method	Purpose
/api/jobs	GET	List all jobs for the current user
/api/jobs/<id>	GET	Get a single job (status + output)
/api/jobs/<id>/cancel	POST	Cancel a running job
/api/jobs/<id>	DELETE	Delete a completed/failed/cancelled job record
/api/admin/users/<id>/jobs	GET	Admin view of any user's jobs

---

Database Persistence

Jobs are only written to SQLite when they reach a terminal state (completed, failed, cancelled). In-memory jobs from before a restart are lost, but completed jobs survive restarts via the database. The GET /api/jobs endpoint merges both sources: in-memory (for live jobs) and database (for historical).

---

Key Files

File	Relevance
app.py	Job dict, route handlers, runner threads, expiry daemon
db.py	upsert_job, get_job, list_jobs_for_user, delete_jobs_older_than