---
title: "PDF Assembly Pipeline"
summary: "Dozens of PDFs from many people become one ordered, bookmarked document."
date: 2026-09-16
links: []
---

<!-- DRAFT: Axel to rewrite -->

## Problem

Many workflows need one PDF built from many source files, in a set order, with a bookmark on each section. The files come from many people with inconsistent names. The storage system could merge files, but not in a template's order.

## Approach

Node's PDF libraries either merge or bookmark but none do both, so I built a Redis + BullMQ queue to a Python worker that handles the document assembly. Files are assigned to sections by a case-insensitive pattern match. Templates are selected by exact name, because a wrong template builds the wrong document.

## Solution

A web request only adds a job to the queue. A Node worker downloads the files, matches them to sections, and runs the Python script as a subprocess with one JSON argument. Exit code 0 marks the job done.

```mermaid
flowchart LR
  U[user browser] -->|build| API[Express API, Node]
  API --> GATE{template bound?}
  GATE -->|no| NT[no template, build disabled]
  GATE -->|yes| Q[(Redis and BullMQ)]
  Q --> WN[Node worker: download, match, manifest]
  V[document store API, signed requests] --> WN
  WN -->|spawn, template as JSON| WP[Python worker: pypdf, reportlab]
  WP -->|stdout and stderr| LOG[job log]
  WP -->|exit 0 and the PDF| ST[storage]
  ST --> U
```

```js
// Node side: hand the whole job to Python, treat the exit code as the verdict
const py = spawn(PYTHON, ['assemble.py', JSON.stringify(template), '-o', out])
py.stdout.on('data', (d) => job.log(String(d)))
py.on('close', (code) => (code === 0 ? resolve(out) : reject(new Error(`exit ${code}`))))
```

A bookmark's page number is only known after every earlier file is merged. The script therefore merges in a first pass, recording each file's start page, and writes the bookmarks in a second pass.

```mermaid
sequenceDiagram
  participant T as Template
  participant M as Merge pass
  participant O as Outline pass
  participant W as Writer
  T->>M: sections in order, files in order
  loop each file
    M->>W: add its pages
    M->>M: note its start page
  end
  M->>O: every start page is known
  loop each section
    O->>W: bookmark the section
    O->>W: bookmark files only if 2 or more
  end
  O->>W: open with the outline showing
```

```python
# Python side: pages first, outline second
offset = 0
for section in ordered(template):
    for f in section.files:
        pages = PdfReader(f.path).pages
        f.start = offset
        for p in pages: writer.add_page(p)
        offset += len(pages)
for section in ordered(template):
    ref = writer.add_outline_item(section.title, first_page(section), parent_of(section))
    if len(section.files) > 1:
        for f in section.files: writer.add_outline_item(f.name, f.start, ref)
```

```mermaid
flowchart TB
  P[packet template name] -->|exact match| B{template with that name?}
  B -->|no| X[flagged, not buildable]
  B -->|yes| S[sections]
  F[file section, title, type, filename] -->|loose match| S
  S -->|matched| PL[placed]
  S -->|unmatched| SIL[bookmark on page 1]
```

Another team had a Python desktop app that handled HMAC authentication, so I ported the auth flow to Node.js to fit the existing application stack.

## Outcome

| Measure                          | Value                                   |
| -------------------------------- | --------------------------------------- |
| Templates                        | 8, from 10 to 52 sections each          |
| End-to-end test packet           | 41 pages, 51 of 51 rows, 0 unplaced     |
| Real packets, files placed       | 144 of 219, 66%                         |
| Live packets a new template touched at deploy | 1 of 7                     |
| Time to add a template           | 15 days, one case, mostly waiting on the order  |
| Node worker, Python worker       | 374 lines, 741 lines                    |

A file that matches no section is still included, bookmarked on page 1, without an error. Build time and time saved were not measured.

## Applicability

The pipeline fits any document set with a fixed structure, such as grant applications, court bundles and board packets. A subprocess with JSON input and an exit code connects Node to a Python library.