Original PDF Matching Protocol

Status

Workflow layer: 09_ai_workflows.
Purpose: define how to connect user-supplied original PDFs to already-ingested Atlas source records.
Applies to extension folders and lab-publication source folders.
Critical rule: original PDFs are preserved as source evidence; they should not silently overwrite metadata or source notes without verification.

Source folders covered by this protocol

Concrete examples currently used in this Atlas:

00_sources/extreme_ductility_extension/originals/
00_sources/sustainable_ecc_extension/originals/
00_sources/by_lee_lab_publications/originals/

Associated source indexes:

00_sources/extreme_ductility_extension/metadata/extreme_ductility_extension_source_index.csv
00_sources/sustainable_ecc_extension/metadata/sustainable_ecc_extension_source_index.csv
00_sources/by_lee_lab_publications/metadata/by_lee_lab_publications_source_index.csv

1. User saves PDFs into `00_sources/<extension>/originals/`

When Professor Lee supplies original PDFs, save them directly into the appropriate originals/ folder.

Examples:

00_sources/extreme_ductility_extension/originals/
00_sources/sustainable_ecc_extension/originals/
00_sources/by_lee_lab_publications/originals/

Recommended user request after saving PDFs:

<extension>/originals 폴더에 PDF 저장했어. missing list와 매칭해서 source index 업데이트하고, 원본 연결 검증해줘.

Do not create a new extension just because PDFs were added. PDFs should be matched to the existing source collection unless Professor Lee explicitly requests a new extension topic.

2. Match by exact `pdf_filename`

First-pass matching must use exact pdf_filename from the source index or source note.

Procedure:

Read the source index:

00_sources/<extension>/metadata/<extension>_source_index.csv

For each source record, check:

pdf_filename

If a file with the exact same name exists in:

00_sources/<extension>/originals/

then mark it as a direct match.

Update source index fields:

original_pdf_status = available
original_pdf_path = 00_sources/<extension>/originals/<pdf_filename>

Exact match has the highest confidence.

3. Fuzzy match rules

Use fuzzy matching only after exact pdf_filename fails.

Allowed fuzzy match signals, in preferred order:

DOI match extracted from PDF metadata or first pages.
Normalized title match.
First author + year + first 4–5 meaningful title words.
Existing paper_id slug vs PDF filename slug.
Hash duplicate match with an already matched PDF.

Normalization rules:

lowercase
remove punctuation
replace spaces/underscores with hyphens
remove common stopwords only if needed
normalize accented characters
ignore publisher filename prefixes/suffixes

Examples:

li-2001-pva-ecc-tensile-strain-hardening.pdf
li-2001-tensile-strain-hardening-behavior-of-polyvinyl.pdf

may refer to the same Li 2001 PVA-ECC paper and must be hash-checked before deciding canonical status.

Fuzzy match confidence labels:

exact_pdf_filename
matched_by_doi
matched_by_normalized_title
matched_by_author_year_title
matched_by_hash_duplicate
ambiguous_needs_manual_check
unmatched

Ambiguous matches should not be applied silently. They should be reported and left as needs_manual_check unless the title/DOI/hash is decisive.

4. SHA-256 registry

Every PDF in originals/ must be hashed with SHA-256 and recorded.

Registry location:

00_sources/<extension>/originals/originals_registry.csv

Recommended columns:

stored_filename
source_path
sha256
file_size_bytes
matched_paper_id
matched_pdf_filename
match_status
canonical_filename
alias_of
duplicate_group
notes

Rules:

One row per PDF file.
Hash must be computed from the actual file bytes.
If two different filenames have the same SHA-256, mark them as duplicate aliases.
Do not delete duplicates automatically.
Use canonical_filename to indicate which filename should be referenced by the source index.

5. Duplicate PDF handling

Duplicate PDFs are common when the same paper is saved with different names.

Protocol:

Group PDFs by SHA-256.
For each duplicate group, choose canonical filename by this priority:
exact pdf_filename from source index;
canonical author-year-short-title filename;
cleanest title-based filename;
first user-supplied filename if no better option exists.
Record aliases in originals_registry.csv.
Keep duplicate files unless Professor Lee explicitly asks to remove them.
Source index should point to the canonical file only.

Example duplicate groups already observed:

WangFramcos5.pdf
wang-2004-fracture-mechanics-of-concrete-structures.pdf

li-2001-pva-ecc-tensile-strain-hardening.pdf
li-2001-tensile-strain-hardening-behavior-of-polyvinyl.pdf

6. Source index update

After matching, update the relevant source index.

Required source index fields:

original_pdf_status
original_pdf_path
original_pdf_sha256
original_pdf_match_status
original_pdf_canonical_filename
numeric_page_verification_status
numeric_page_verification_date
verification_status

If a field does not exist, add it.

Recommended values:

original_pdf_status = available | pending_user_pdf | needs_manual_check
original_pdf_match_status = exact_pdf_filename | matched_by_doi | matched_by_normalized_title | matched_by_author_year_title | matched_by_hash_duplicate | ambiguous_needs_manual_check | unmatched
verification_status = verified_from_pdf | verified_from_full_text | verified_from_source_note | pending_user_pdf | needs_check
numeric_page_verification_status = pending | completed_verified_against_original_pdf | needs_check

Rules:

Do not mark verification_status = verified_from_pdf just because a PDF file exists.
verified_from_pdf requires at least basic confirmation that the source note/paper card belongs to that PDF.
Numeric/page verification requires separate checking of key values and page/figure/table references.

7. Missing PDF request list regeneration

After matching, regenerate:

00_sources/<extension>/metadata/missing_original_pdf_request_list.md

Required content:

# Missing Original PDF Request List — <extension>

## Status

- Source records: N
- Original PDFs available: A
- Original PDFs still missing: M
- Needs manual check: K

## Missing PDFs

| paper_id | year | title | expected_pdf_filename | DOI | source_note |
|---|---:|---|---|---|---|

## Needs manual check

| paper_id | candidate_pdf | reason |
|---|---|---|

If no missing PDFs remain, state clearly:

Original PDFs still missing: 0

Do not delete old missing lists without replacing them with an updated status.

8. Full text re-extraction policy

Re-extract full text from original PDF only when necessary.

Re-extraction is required if:

Existing full text is missing.
Existing full text was generated from poor OCR.
Existing full text has page numbering errors.
Key tables/equations/figure captions are missing.
Source note or paper card contains suspect values that need checking.
Original PDF differs from the previously extracted text.

Re-extraction is optional if:

existing full text is complete and page-separated;
extracted text already matches the original PDF;
no numeric/page verification errors are suspected.

Re-extracted file should keep the same filename unless replacing a clearly faulty extraction:

00_sources/<extension>/full_text/<paper_id>_full_text.md

If replacing, preserve old file as:

<paper_id>_full_text_backup_<date>.md

Note: <paper_id>_full_text_backup_<date>.md is a filename pattern, not a concrete file expected to exist before re-extraction.

9. PDF-based numeric/page verification policy

PDF matching and numeric/page verification are separate steps.

9.1 Basic PDF match verification

Confirms:

title matches source record;
DOI matches or is absent but title/author/year match;
PDF is not a wrong article;
source note and paper card correspond to this PDF.

After this, source may be marked:

original_pdf_status = available

but not necessarily:

numeric_page_verification_status = completed_verified_against_original_pdf

9.2 Numeric/page verification

Required checks:

Key numerical claims in source note.
Key numerical claims in paper card.
Claim-evidence matrix rows.
Page/section references.
Figure/table/equation references.
Direct tensile vs flexural ductility distinction.
Compressive strength vs tensile ductility distinction.
Material system and test method correctness.

Only after these checks can rows be marked:

verification_status = verified_from_pdf
numeric_page_verification_status = completed_verified_against_original_pdf

If unclear:

needs_check
needs_ocr_check

9.3 Batch-level policy

If Professor Lee states that source notes have already been verified from PDFs, record that as a batch-level verification note, but still keep a report of who/when/what was marked.

10. Final report template

Create a final report after each matching run:

07_visualization/<extension>_original_pdf_link_report.md

Template:

# Original PDF Link Report — <extension>

## Scope

- Source folder: `00_sources/<extension>/`
- Originals folder: `00_sources/<extension>/originals/`
- Source index: `00_sources/<extension>/metadata/<extension>_source_index.csv`

## Matching summary

- PDFs found in originals folder:
- Source records:
- Exact filename matches:
- Fuzzy matches:
- Hash duplicate matches:
- Unmatched PDFs:
- Source records still missing PDFs:
- Needs manual check:

## Duplicate groups

| duplicate_group | canonical_filename | aliases | sha256 |
|---|---|---|---|

## Updated files

- `00_sources/<extension>/metadata/<extension>_source_index.csv`
- `00_sources/<extension>/originals/originals_registry.csv`
- `00_sources/<extension>/metadata/missing_original_pdf_request_list.md`

## Verification status

- Basic PDF match verification: complete / partial / pending
- Numeric/page verification: complete / partial / pending
- Claim-evidence matrix updated: yes / no

## Remaining tasks

1. [Task]
2. [Task]

Also create/update machine-readable summary if useful:

07_visualization/<extension>_original_pdf_link_summary.json

Suggested fields:

{
  "extension": "<extension>",
  "pdfs_in_originals": 0,
  "source_records": 0,
  "matched_records": 0,
  "missing_records": 0,
  "unmatched_pdfs": 0,
  "duplicate_groups": 0,
  "needs_manual_check": 0,
  "broken_links": 0,
  "numeric_page_verification_status": "pending | partial | complete"
}

Concrete examples

Extreme ductility extension

Folder:

00_sources/extreme_ductility_extension/originals/

Index:

00_sources/extreme_ductility_extension/metadata/extreme_ductility_extension_source_index.csv

Report:

07_visualization/extreme_ductility_original_pdf_link_report.md

Current known status from prior build:

Original PDFs matched for all source records.
Duplicate groups were recorded.
Numeric/page verification was marked complete per Professor Lee's confirmation.

Sustainable ECC extension

Folder:

00_sources/sustainable_ecc_extension/originals/

Index:

00_sources/sustainable_ecc_extension/metadata/sustainable_ecc_extension_source_index.csv

Report to create after PDFs arrive:

07_visualization/sustainable_ecc_extension_original_pdf_link_report.md

Current policy:

Original PDFs are pending user supply.
Current evidence is from full text, paper cards, and source notes.
After PDFs arrive, run matching and numeric/page verification before marking verified_from_pdf.

By Lee lab publications

Folder:

00_sources/by_lee_lab_publications/originals/

Index:

00_sources/by_lee_lab_publications/metadata/by_lee_lab_publications_source_index.csv

Report to create after PDFs arrive:

07_visualization/by_lee_lab_publications_original_pdf_link_report.md

Current policy:

Original PDFs are pending unless already mirrored from another source collection.
Since this layer supports lab positioning, exact matching and conservative verification are important.
Lab positioning claims should not be strengthened unless the corresponding PDF/source evidence supports them.

Original PDF Matching Protocol

Status

Source folders covered by this protocol

1. User saves PDFs into 00_sources/<extension>/originals/

2. Match by exact pdf_filename

3. Fuzzy match rules

4. SHA-256 registry

5. Duplicate PDF handling

6. Source index update

7. Missing PDF request list regeneration

8. Full text re-extraction policy

9. PDF-based numeric/page verification policy

9.1 Basic PDF match verification

9.2 Numeric/page verification

9.3 Batch-level policy

10. Final report template

Concrete examples

Extreme ductility extension

Sustainable ECC extension

By Lee lab publications

1. User saves PDFs into `00_sources/<extension>/originals/`

2. Match by exact `pdf_filename`