CHAPTER 10 · Document and File Processing Pipelines · 7 / 8
Operational concerns
- Size limits. Enforce maximum file sizes at the edge to protect memory and processing time.
- Storage hygiene. Owner-scoped keys make cleanup and access reasoning easy; delete derived artifacts when the source is deleted.
- Idempotency. If a processing step is retried, it shouldn't create duplicate versions or orphaned files. Make steps safe to re-run (Chapter 13).
- Observability. Log each pipeline stage with the document id so a failed conversion is diagnosable. Document pipelines fail in field-specific ways; you'll want the breadcrumbs.