Pipeline Architecture

Architecture & Script Responsibilities

RFQX Pipeline Architecture

How the RFQX repository works, what each script does, what is deterministic vs heuristic vs AI-assisted, and what happens when input documents change.

1. What RFQX Does

RFQX turns a customer RFQ document set into a supplier-side working environment: customer requirement review, supplier proposals, open points, estimation impact, an initial cybersecurity concept, an initial system/security design, and derived supplier system requirements — all with traceability back to Markdown-derived sources.

2. Input Folders

3. Markdown-First Rule

All analysis runs on Markdown, never on the PDFs directly. PDFs are converted to Markdown first; every requirement keeps its Source document, section path and page reference. This keeps extraction auditable and stable.

4. OCR Policy

OCR is disabled (ocr_enabled: false in config.yaml). Pages with no extractable text are flagged for manual review rather than OCR'd. The built-in converter never performs OCR.

5. Main Scripts and Their Responsibility

See section 10 for the full table.

6. Generated Outputs

Requirements register, supplier proposals, open points, estimation impact, supplier system requirements + coverage, initial cybersecurity concept, system/security design, Jira requirement import payloads, workflow state, the HTML site, and the evidence package.

converted/assets/image_manifest.json is generated during conversion when image extraction is enabled. document_artifacts/ is generated from Markdown-derived tables, figures, diagrams, image references, and that converter sidecar. These artifacts are linked to nearby requirements and feed proposal, SSR, Jira, and per-document site context.

The generated HTML site uses a shared premium dashboard shell from scripts/generate_html_site.py: dark sidebar, topbar, page header, card system, modern filters, modern table wrappers, and user-facing source labels. Detail pages keep full text and traceability while avoiding raw internal Markdown path leakage in visible customer-facing HTML.

7. What Happens When New Documents Are Added

See section 13 (re-run behaviour). In short: convert → quality-check → extract → regenerate register, proposals, open points, SSRs, concept → regenerate site → validate. Customer decisions and feedback are never invented.

Static Feedback Authoring

The deployed static site can author feedback locally in the browser and export it as CSV/JSON. It does not write directly to the repository or server unless a backend is added later. Exported files are placed in configured customer input folder and ingested by scripts/ingest_customer_feedback.py.

Per-PDF Document Intelligence Re-run Behavior

When a new PDF is added and the pipeline is re-run:

• a new document intelligence record is created
• a new per-PDF page is created
• requirements from that PDF are grouped under that page
• SSRs derived from that PDF are linked
• open points caused by that PDF are linked
• design/concept/estimation impacts are recalculated
• overall dashboards are also updated

Per-PDF Content / Section Filtering

• Section filters are generated from converted Markdown, extracted requirement

metadata, section references, and page references.

• Parent section filters include child requirements because each requirement row

stores the selected section plus all parent section IDs.

• New PDFs automatically get section filters when numbered headings or

table-of-contents-like entries are detected.

• If headings are not reliable, the document page falls back to page-based

filtering.

• PDFs are not analyzed directly downstream; they remain source evidence only.

Table / Diagram Artifact Stage

• scripts/convert_pdf_to_markdown.py is the only stage that may open PDFs for

image extraction; OCR remains false.

• Extracted images are written under converted/assets/by_pdf/<slug>/ and

described in converted/assets/image_manifest.json.

• scripts/extract_document_artifacts.py reads converted Markdown only.
• It may read the converter image sidecar, but it does not open PDFs.
• Table and diagram artifacts are written to document_artifacts/.
• High/medium-confidence artifact links can enrich requirement review, supplier

proposal, SSR derivation, Jira description/rationale/verification criteria, and effort estimates.

• Missing image assets are reported as converter coverage gaps; RFQX does not

pretend a visual diagram exists when only text/caption context was extracted.

8. What Is Deterministic / Rule-Based

• PDF→Markdown conversion and page/section tagging.
• Requirement extraction structure and IDs.
• Requirement review register field assembly.
• Supplier system requirement clustering and coverage maths.
• Disposition assignment, traceability matrices, counts, coverage %.
• Site generation and validation.

These produce identical output for identical input.

9. What Is Heuristic

• Supplier position defaults (Accept / Accept-with-Assumption / Partial / …) from

keyword + mapping rules.

• Security-relevance inference and capability/feature/interface mapping.
• SSR category classification and clustering granularity.
• Open-point topic matching and estimation impact levels.

Heuristics are conservative and explainable, but they approximate engineering judgment and should be reviewed.

10. What Requires AI/LLM Judgment

Currently no script calls an AI/LLM API (see analysis/source document). The outputs that would most benefit from AI assistance — nuanced requirement interpretation, bespoke supplier proposals, sharp clarification questions, and concept conclusions — are today produced by rules/heuristics or human review. An optional, controlled AI stage is designed in docs/source document.

11. What Requires Human Review

• Low-confidence / human-review-flagged requirements.
• Every supplier position before it is sent to the customer.
• Responsibility (CIA/RASIC) decisions and open-point answers.
• SSR statements before customer alignment.
• The cybersecurity concept review with the Cybersecurity Manager and leads.

12. Known Limitations

• Heuristic defaults are not a substitute for engineering review.
• No AI stage is implemented; conclusions are only as good as the rules/heuristics.
• Duplicate customer requirement IDs in the source are flagged Duplicate/Merged.
• Screenshots require Playwright (not installed); export records the blocker.
• Git remote is not configured; commits are local only.

13. How to Re-run the Pipeline

python scripts/run_full_analysis.py                       # convert + extract (when new PDFs)
python scripts/convert_pdf_to_markdown.py                 # rerun conversion and image sidecar when converter settings change
python scripts/extract_document_artifacts.py              # preserve Markdown tables/diagrams and link context
python scripts/generate_requirement_review_register.py    # register, proposals, open points, concept, SSRs
python scripts/generate_supplier_system_requirements.py   # SSR derivation + coverage + matrix
python scripts/generate_document_intelligence.py          # per-PDF records, traceability and impact data
python scripts/ingest_customer_feedback.py                # apply any customer feedback
python scripts/generate_expert_synthesis.py               # features/interfaces/capabilities/diagrams
python scripts/generate_jira_requirement_import.py --dry-run # Jira Requirement dry-run CSV/XLSX/payloads
python scripts/check_jira_import_payload.py               # validate Jira import payloads
python scripts/generate_html_site.py                      # build the site
python scripts/check_html_site.py                         # validate
python scripts/export_site_evidence.py                    # evidence package
python scripts/check_git_status.py                        # git status report

14. How to Validate the Result

python scripts/check_html_site.py must print "HTML site validation passed." It fails on missing pages/artifacts, OCR enabled, downstream PDF analysis, weak labels, insufficient SSR coverage, or unmapped active requirements without a disposition. Warnings (heuristic proposals, no AI stage, blocked SSRs, feedback not ingested) are expected and listed.

15. AI Assistance Gap Report

The current pipeline uses AI/LLM API: no. See analysis/source document for the scan result, the current rule/heuristic outputs, and the outputs that would benefit from controlled AI assistance.

16. Optional AI Stage Design

The optional AI stage is design-only. See docs/source document, prompts/, and schemas/. It reads Markdown-derived requirements only, never PDFs directly, returns schema-validated JSON, separates evidence from inference, marks confidence, and stays opt-in.

Script Responsibility Table

None of the above currently call an AI/LLM API.

Conversion-Stage Diagram Images

scripts/extract_pdf_images.py opens PDFs only in the conversion/asset stage. It extracts embedded images, renders diagram-candidate pages for vector figures, creates deterministic thumbnails, and writes converted/assets/image_manifest.json. extract_document_artifacts.py and all later stages consume Markdown and this sidecar only. OCR remains false.

The site generator copies only linked images and thumbnails into site/assets/document_artifacts/ and supplies the shared zoom/pan viewer. See docs/source document.

RFQX-36 — Numeric customer requirement ID extraction

Numeric, section-style customer IDs (6.15, 6.19.1, 4.2.3, 10.5.1) are detected only inside a requirement-table / structured-list context, in scripts/requirement_identity.py and scripts/pipeline_lib.py:

• extract_numeric_customer_id(raw_text, context) + numeric_customer_id_confidence(...) grade a

candidate high / medium / low. Only high/medium replace the RFQX fallback customer ID.

• numeric_requirement_rows(text) parses a chunk into per-row (customer_id, title, statement)

tuples, handling both inline (6.15 System Temperature) and stacked (ID alone on its line) layouts. Page-header noise (P / 1 / Page) and table headers (Req. ID Description) are stripped; one row never bleeds into the next.

Guards against over-detection:

• A title ending in "requirements" with no normative body is treated as a section heading (low).
• Rows are kept only when the block forms a sequence and/or carries shall/must/required.
• A numeric token that wraps a prefixed customer ID (section 5.5.1.2.2 above REQ_UDS 0242) is

skipped so the prefixed ID is preserved.

• Statements must start with a capital/digit (filters wrapped mid-sentence fragments and references

such as (Ref 14.14) / 4.16)).

The requirement title is preserved separately (requirement_title) and flows through the register, documents/by_pdf/*.json, the customer delivery package, the Jira import (Summary = numeric customer ID), and the site. RFQX REQ-AUTO-* IDs remain internal traceability IDs only.

Validation: check_jira_import_payload.py, check_html_site.py, and check_site_text_completeness.py fail on RFQX-as-customer-ID, title-merged-into-ID, or Req. ID Requirement text pollution, and warn on duplicate / low-confidence numeric IDs.

AI Assistance Gap

Optional AI Stage Design

Markdown-derived requirements
        │
        ▼
  (optional) AI Analysis Stage  ──►  structured JSON (schema-validated)
        │                               │
        ▼                               ▼
  merge into review register / SSRs / concept  ──►  site + validation