Building a Document Archival System with the API

The high-level data flow looks like this:

URL Source → Scheduler → Pretty PDF API → Storage → Index

┌─────────────┐    ┌───────────┐    ┌──────────────┐    ┌──────────┐    ┌───────┐
│ URL Source   │───→│ Scheduler │───→│ Pretty PDF   │───→│ Storage  │───→│ Index │
│              │    │           │    │ API          │    │          │    │       │
│ - YAML list  │    │ - cron    │    │ POST /v1/    │    │ - Local  │    │ SQLite│
│ - RSS feed   │    │ - Celery  │    │   generate/  │    │ - S3     │    │ FTS5  │
│ - Sitemap    │    │ - Airflow │    │   url        │    │ - GCS    │    │       │
│ - Web scrape │    │ - schedule│    │              │    │          │    │       │
└─────────────┘    └───────────┘    └──────────────┘    └──────────┘    └───────┘

URL Source provides the list of pages to archive. This can be a static file, a dynamic feed, or a programmatic discovery mechanism.

Scheduler triggers archival runs at defined intervals. A simple cron job works for small archives. Celery or Airflow handle complex workflows with dependencies and retries.

Pretty PDF API fetches each URL, extracts the content, applies a template, and returns a styled PDF. The API handles content extraction, site-specific parsing, and PDF rendering — your system just sends URLs and receives PDFs.

Storage holds the generated PDF files. Start with local filesystem storage and move to cloud object storage (S3, GCS) as your archive grows.

Index stores metadata about each archived document — URL, title, archive date, file path, tags — and provides search capability over the archive.

Processing loop with error handling

Walk through your URL list, call the API for each one, and record the result. Handle failures gracefully — network errors, API rate limits, and timeouts should not stop the entire batch.

import httpx
import time
import json
from datetime import datetime

API_KEY = "your_api_key"
API_URL = "https://api.prettypdfprinter.com/v1/generate/url"

def archive_urls(urls, output_dir="archive/"):
    results = []

    for source in urls:
        url = source["url"]
        status = "pending"
        print(f"Archiving: {url}")

        try:
            response = httpx.post(
                API_URL,
                headers={"X-API-Key": API_KEY},
                json={"url": url, "template": "clean"},
                timeout=60.0
            )

            if response.status_code == 200:
                # Save the PDF
                filename = build_filename(url)
                filepath = f"{output_dir}{filename}"
                with open(filepath, "wb") as f:
                    f.write(response.content)

                status = "completed"
                results.append({
                    "url": url,
                    "status": status,
                    "filepath": filepath,
                    "timestamp": datetime.utcnow().isoformat(),
                    "size_bytes": len(response.content)
                })

            elif response.status_code == 429:
                # Rate limited — wait and retry
                retry_after = int(response.headers.get("Retry-After", 60))
                print(f"Rate limited. Waiting {retry_after}s...")
                time.sleep(retry_after)
                status = "pending"  # Will retry on next run

            else:
                status = "failed"
                results.append({
                    "url": url,
                    "status": status,
                    "error": f"HTTP {response.status_code}",
                    "timestamp": datetime.utcnow().isoformat()
                })

        except httpx.TimeoutException:
            status = "failed"
            results.append({
                "url": url,
                "status": status,
                "error": "timeout",
                "timestamp": datetime.utcnow().isoformat()
            })

        # Respect rate limits: pause between requests
        time.sleep(6)  # 10 requests/min = 1 every 6 seconds

    return results

Status tracking

Track every URL through four states: pending (not yet processed), processing (API call in progress), completed (PDF saved), and failed (error occurred). Store status in your index database so you can resume interrupted batches, retry failures, and report on archive completeness.

Metadata per document

For each archived PDF, store: the source URL, the archive timestamp, the document ID or file path, the page title (from the API response or parsed from HTML), the file size, the category or tag from your URL list, and the content hash for change detection.

Directory structure

Organize archived PDFs by date, source domain, or category. A date-based structure works well for most archives because it naturally partitions files and makes cleanup straightforward:

archive/
├── 2026/
│   ├── 02/
│   │   ├── 13/
│   │   │   ├── example-gov_regulations-update_2026-02-13.pdf
│   │   │   ├── competitor-com_pricing_2026-02-13.pdf
│   │   │   └── docs-supplier-com_api-terms_2026-02-13.pdf
│   │   └── 14/
│   │       └── ...
│   └── 03/
│       └── ...
└── metadata.db

Filename conventions

Use a consistent naming pattern: {domain}_{slug}_{date}.pdf. This makes files identifiable at a glance without needing the index database. Sanitize domain and slug to remove special characters:

import re
from urllib.parse import urlparse
from datetime import date

def build_filename(url):
    parsed = urlparse(url)
    domain = parsed.netloc.replace(".", "-")
    slug = parsed.path.strip("/").replace("/", "-")
    slug = re.sub(r"[^a-zA-Z0-9-]", "", slug)[:80]
    today = date.today().isoformat()
    return f"{domain}_{slug}_{today}.pdf"

Metadata in SQLite

Store metadata in a SQLite database alongside your archive. This gives you fast queries, full-text search, and a single file to back up:

CREATE TABLE archived_documents (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    url TEXT NOT NULL,
    domain TEXT NOT NULL,
    title TEXT,
    filepath TEXT NOT NULL,
    archived_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    file_size INTEGER,
    content_hash TEXT,
    category TEXT,
    frequency TEXT,
    status TEXT DEFAULT 'completed'
);

CREATE INDEX idx_archived_url ON archived_documents(url);
CREATE INDEX idx_archived_domain ON archived_documents(domain);
CREATE INDEX idx_archived_date ON archived_documents(archived_at);

Cloud storage for large archives

For archives exceeding a few thousand documents, consider cloud object storage (S3, Google Cloud Storage) for the PDF files while keeping the metadata index in SQLite locally. This separates the cheap, scalable blob storage from the fast, queryable metadata layer. Use the same filename conventions and organize objects with key prefixes that mirror your date-based directory structure.