PDF Assembly Pipeline

Problem

Many workflows need one PDF built from many source files, in a set order, with a bookmark on each section. The files come from many people with inconsistent names. The storage system could merge files, but not in a template's order.

Approach

Node's PDF libraries either merge or bookmark but none do both, so I built a Redis + BullMQ queue to a Python worker that handles the document assembly. Files are assigned to sections by a case-insensitive pattern match. Templates are selected by exact name, because a wrong template builds the wrong document.

Solution

A web request only adds a job to the queue. A Node worker downloads the files, matches them to sections, and runs the Python script as a subprocess with one JSON argument. Exit code 0 marks the job done.

flowchart LR
  U[user browser] -->|build| API[Express API, Node]
  API --> GATE{template bound?}
  GATE -->|no| NT[no template, build disabled]
  GATE -->|yes| Q[(Redis and BullMQ)]
  Q --> WN[Node worker: download, match, manifest]
  V[document store API, signed requests] --> WN
  WN -->|spawn, template as JSON| WP[Python worker: pypdf, reportlab]
  WP -->|stdout and stderr| LOG[job log]
  WP -->|exit 0 and the PDF| ST[storage]
  ST --> U

// Node side: hand the whole job to Python, treat the exit code as the verdict
const py = spawn(PYTHON, ['assemble.py', JSON.stringify(template), '-o', out])
py.stdout.on('data', (d) => job.log(String(d)))
py.on('close', (code) => (code === 0 ? resolve(out) : reject(new Error(`exit ${code}`))))

A bookmark's page number is only known after every earlier file is merged. The script therefore merges in a first pass, recording each file's start page, and writes the bookmarks in a second pass.

sequenceDiagram
  participant T as Template
  participant M as Merge pass
  participant O as Outline pass
  participant W as Writer
  T->>M: sections in order, files in order
  loop each file
    M->>W: add its pages
    M->>M: note its start page
  end
  M->>O: every start page is known
  loop each section
    O->>W: bookmark the section
    O->>W: bookmark files only if 2 or more
  end
  O->>W: open with the outline showing

# Python side: pages first, outline second
offset = 0
for section in ordered(template):
    for f in section.files:
        pages = PdfReader(f.path).pages
        f.start = offset
        for p in pages: writer.add_page(p)
        offset += len(pages)
for section in ordered(template):
    ref = writer.add_outline_item(section.title, first_page(section), parent_of(section))
    if len(section.files) > 1:
        for f in section.files: writer.add_outline_item(f.name, f.start, ref)

flowchart TB
  P[packet template name] -->|exact match| B{template with that name?}
  B -->|no| X[flagged, not buildable]
  B -->|yes| S[sections]
  F[file section, title, type, filename] -->|loose match| S
  S -->|matched| PL[placed]
  S -->|unmatched| SIL[bookmark on page 1]

Another team had a Python desktop app that handled HMAC authentication, so I ported the auth flow to Node.js to fit the existing application stack.

Outcome

A file that matches no section is still included, bookmarked on page 1, without an error. Build time and time saved were not measured.

Applicability

The pipeline fits any document set with a fixed structure, such as grant applications, court bundles and board packets. A subprocess with JSON input and an exit code connects Node to a Python library.