09_ai_workflows/new_extension_ingestion_protocol.md

Atlas document

New Extension Ingestion Protocol

Status

  • Workflow layer: `09_ai_workflows`.
  • Purpose: define the repeatable protocol for adding one new topic-specific extension to the ECC/SHCC/EGC Atlas.
  • Scope: future extension topics such as self-sensing ECC, thermal adaptive ECC, structural applications, durability, marine ECC, AI/DIC/XCT, etc.
  • Critical rule: **create a new extension only when Professor Lee explicitly requests that topic extension.**
  • Structural rule: preserve the existing Atlas evidence hierarchy and do not create excessive new nodes when existing nodes can support the evidence.

    • Evidence hierarchy to preserve

    Victor Li book = primary anchor  -> foundational papers = supporting / verification layer    -> topic-specific extension papers = extension layer      -> Lee lab publications = lab positioning layer        -> teaching and AI workflow files = derived use layer

    A new extension folder is not a new theory layer. It is a topic-specific evidence collection attached to the existing book-anchored Atlas.

    ---

    1. When to create a new extension folder

    Create a new extension folder only when all conditions below are met:

    1. Professor Lee explicitly asks to add a new extension topic.

    2. The topic is not already covered adequately by an existing extension folder.

    3. The batch contains multiple papers or a coherent topic cluster.

    4. The papers extend, apply, or update Victor Li 2019 book concepts rather than forming a new primary anchor.

    5. The extension can be mapped to existing Atlas nodes such as `02_concepts/`, `04_material_systems/`, or `05_experiments/`.

    Do **not** create a new extension folder when:

  • only one paper is supplied and it can be added to an existing collection;
  • the material belongs naturally to `foundational_papers`;
  • the content is a Lee lab publication that belongs under `00_sources/by_lee_lab_publications/`;
  • the user has not explicitly requested a new extension topic;
  • existing source folders can absorb the new papers without creating a new topic layer.

    • Recommended naming:

    <topic>_extension

    Examples:

    self_sensing_extensionthermal_adaptive_extensionstructural_application_extensionmarine_ecc_extensionxct_dic_characterization_extension

    ---

    2. Required folder structure

    Use the following folder pattern exactly:

    00_sources/<topic>_extension/├── full_text/├── source_notes/├── paper_cards/├── originals/└── metadata/03_papers/<topic>_extension/07_visualization/<topic>_extension_claim_evidence_matrix.csv07_visualization/<topic>_extension_edges.csv

    Folder meanings:

  • `originals/`: original PDF files supplied by the user.
  • `full_text/`: extracted text files, usually generated externally or mechanically.
  • `source_notes/`: source-grounded paper summaries created after reading full text/PDF.
  • `paper_cards/`: compact paper cards for quick retrieval.
  • `metadata/`: source index, request list, registry, ingestion report, and verification status.
  • `03_papers/<topic>_extension/`: mirror of paper cards for paper-level Atlas browsing.

    • ---

    3. Required files

    Each extension must produce or update the following files:

    3.1 Source collection files

    00_sources/<topic>_extension/metadata/<topic>_extension_source_index.csv00_sources/<topic>_extension/metadata/<topic>_extension_source_index.md00_sources/<topic>_extension/metadata/missing_original_pdf_request_list.md00_sources/<topic>_extension/originals/README.md

    If original PDFs are supplied:

    00_sources/<topic>_extension/originals/originals_registry.csv

    If original PDFs are not supplied:

  • set `original_pdf_status = pending_user_pdf` in the source index;
  • list all missing PDFs in `00_sources/<topic>_extension/metadata/missing_original_pdf_request_list.md`.

    • 3.2 Paper-level files

    For each paper:

    00_sources/<topic>_extension/full_text/<paper_id>_full_text.md00_sources/<topic>_extension/source_notes/<paper_id>_source_note.md00_sources/<topic>_extension/paper_cards/<paper_id>_paper_card.md03_papers/<topic>_extension/<paper_id>_paper_card.md

    3.3 Visualization / graph files

    07_visualization/<topic>_extension_claim_evidence_matrix.csv07_visualization/<topic>_extension_edges.csv07_visualization/<topic>_extension_ingestion_report.md07_visualization/<topic>_extension_ingestion_summary.json

    After graph refinement, extension edges may also be reflected in:

    07_visualization/graph_paper_to_node_edges.csv07_visualization/graph_concept_to_concept_edges.csv

    ---

    4. Source note schema

    Use the schema in:

    09_ai_workflows/external_ai_pdf_processing_request_template.md

    Required frontmatter:

    ---title: ""authors: ""year:journal: ""volume: ""issue: ""pages: ""doi: ""pdf_filename: ""source_collection: "external"atlas_layer: "extension"related_book_chapter: "Chapter N: title"related_atlas_nodes:  - "02_concepts/..."  - "04_material_systems/..."  - "05_experiments/..."verification_status: "source_note_seed | verified_from_pdf | verified_from_full_text | needs_check"---

    Required sections:

    # [Paper Title]## One-line Summary## 1. Document Information## 2. Why this paper matters for the Atlas## 3. Key Contributions## 4. Methodology## 5. Key Results## 6. Atlas Node Links## 7. Claim-Evidence Candidates## 8. Relationship to Victor Li 2019 Book## 9. Limitations and Cautions## 10. Keywords / Glossary

    Rules:

  • Use only PDF/full-text evidence.
  • Mark unclear page/figure/table/equation numbers as `needs_check`.
  • Distinguish direct tensile strain capacity from flexural ductility.
  • Distinguish high compressive strength from high tensile ductility.
  • Do not infer Lee lab positioning unless the user explicitly asks.
  • Do not overstate novelty.

    • ---

    5. Paper card schema

    Use the schema in:

    09_ai_workflows/external_ai_pdf_processing_request_template.md

    Required sections:

    # [Author Year] — [Short Title]## Citation## Why this paper matters## Main contribution## Evidence summary## Linked Atlas nodes## Relationship to Victor Li book## Claim-evidence rows to add## Verification status## Cautions

    The paper card should be concise but still source-grounded.

    Paper card must include:

  • DOI or `needs_check`;
  • Atlas layer;
  • related Victor Li book chapter;
  • source PDF filename;
  • extracted text filename;
  • source note filename;
  • linked Atlas nodes;
  • cautions.

    • ---

    6. Claim-evidence matrix schema

    Create:

    07_visualization/<topic>_extension_claim_evidence_matrix.csv

    Minimum columns:

    paper_idyeartitledoiatlas_nodeatlas_fileclaimevidence_excerpt_or_summarypage_or_sectionfigure_table_equationstatussource_notefull_textpaper_cardverification_statusrelated_book_chapteroriginal_pdf_status

    Classification columns to add during first-pass stabilization:

    mechanism_categorymaterial_systemfiber_typebinder_typetest_methodkey_metriclab_relevanceclaim_strengthverification_levelpriority_for_synthesis

    Allowed verification values:

    verified_from_pdfverified_from_full_textverified_from_source_noteverified_from_source_note_pending_user_pdfpending_user_pdfneeds_checkneeds_ocr_check

    Rules:

  • Each row should contain one clear claim.
  • Numeric values must be tied to page/figure/table if available.
  • If source note or paper card includes a value but PDF is not supplied, use `verified_from_source_note_pending_user_pdf`.
  • Do not upgrade source-note evidence to `verified_from_pdf` without original PDF verification.

    • ---

    7. Node supporting-source update protocol

    For each claim row, update the relevant existing Atlas node whenever possible.

    Preferred target node folders:

    02_concepts/04_material_systems/05_experiments/06_lab_position/07_visualization/08_teaching/09_ai_workflows/

    Taxonomy guard:

  • Do not create ad-hoc top-level folders such as `01_foundations/`, `03_micromechanics/`, `06_sustainability/`, or `07_applications/` during batch ingestion unless explicitly requested.
  • Map general history, foundation, classification, and definition nodes into `02_concepts/`.
  • Map sustainability or green-material system topics into `04_material_systems/` or existing concept nodes.
  • If a source note proposes a non-canonical folder, rewrite the target path to the closest canonical existing folder before creating nodes, matrices, or graph edges.

    • Update protocol:

    1. Identify existing nodes from `related_atlas_nodes` in the source note.

    2. Prefer existing nodes over new node creation.

    3. Add or update a `### <Topic> extension sources` subsection under `## Anchor evidence` or `### Supporting sources`.

    4. Include:

    - paper card path;

    - year and title;

    - DOI;

    - evidence status;

    - short main linked claim;

    - matrix path.

    5. If many papers link to one node, summarize counts and list representative papers rather than dumping all rows.

    Do not create a new node unless:

  • no existing node represents the concept;
  • the concept recurs across multiple papers;
  • the new node will be useful for graph/retrieval/teaching;
  • the user has not prohibited new node creation.

    • New node minimum structure:

    # Node Title## Status- Created during <topic> extension ingestion.## Core idea## Anchor evidence### Supporting sources## Linked nodes## Cautions

    ---

    8. Graph edge update protocol

    Create a dedicated edge file:

    07_visualization/<topic>_extension_edges.csv

    Required columns:

    sourcetargetrelationsource_filetarget_filestatusnotes

    Common paper-to-node relation:

    supports

    Other allowed relation types:

    supportsextendsapplies_tomeasuresmeasured_byrequiresenablescontrolsdepends_onaffectsvariant_ofmechanism_ofcontrastslab_contributes_to

    Rules:

  • Paper-to-node edges normally use `supports`.
  • Concept-to-concept edges use mechanism-specific relations such as `requires`, `enables`, `controls`, `depends_on`, `extends`.
  • Do not use `lab_contributes_to` for external extension papers unless the paper is part of the Lee lab publication corpus.
  • After writing extension edges, later graph refinement should merge them into:

    • - `07_visualization/graph_paper_to_node_edges.csv`

    - `07_visualization/graph_concept_to_concept_edges.csv`

    - `07_visualization/graph_lab_to_global_edges.csv` only if applicable.

    Graph validation:

  • target_file must exist;
  • source_file must exist;
  • duplicate `(source, target, relation)` should be removed;
  • missing targets go to an audit list, not silent deletion.

    • ---

    9. Original PDF matching protocol

    Original PDFs may arrive later. Use this process:

    1. User saves PDFs in:

    00_sources/<topic>_extension/originals/

    2. Match PDFs to source records by:

    - `pdf_filename`;

    - DOI;

    - normalized title;

    - first author + year + short title;

    - SHA-256 hash for duplicate detection.

    3. Update source index:

    original_pdf_status = availableoriginal_pdf_path = 00_sources/<topic>_extension/originals/<filename>.pdforiginal_pdf_sha256 = <hash>

    4. Update registry:

    00_sources/<topic>_extension/originals/originals_registry.csv

    Recommended registry columns:

    stored_filenamesource_pathsha256file_sizematched_paper_idmatch_statusalias_ofduplicate_groupnotes

    5. Regenerate:

    00_sources/<topic>_extension/metadata/missing_original_pdf_request_list.md

    6. If an original PDF belongs to foundational papers or lab publications too, mirror it to the appropriate source collection and record the registry entry.

    7. Do not delete duplicate PDFs automatically; record duplicate hash groups and canonical filename.

    ---

    10. Verification checklist

    Before reporting a new extension complete, verify:

    Source files

  • [ ] `full_text/` files exist.
  • [ ] `source_notes/` files exist.
  • [ ] `paper_cards/` files exist.
  • [ ] `03_papers/<topic>_extension/` mirror exists.
  • [ ] source index CSV/MD exists.
  • [ ] original PDF status is recorded.

    • Evidence files

  • [ ] claim-evidence matrix exists.
  • [ ] every row has `paper_id`, `title`, `claim`, `atlas_file`, `status`, and source paths.
  • [ ] classification columns exist.
  • [ ] numeric claims have page/figure/table if verified.
  • [ ] `pending_user_pdf` is not mislabeled as `verified_from_pdf`.

    • Node updates

  • [ ] relevant existing nodes have Supporting sources updated.
  • [ ] no excessive new nodes were created.
  • [ ] new nodes, if any, have status and source basis.

    • Graph

  • [ ] dedicated edge CSV exists.
  • [ ] target_file exists for all final edges.
  • [ ] source_file exists for all final edges.
  • [ ] duplicate edge count reported.
  • [ ] missing target audit retained if needed.

    • Final report

  • [ ] ingestion report exists.
  • [ ] summary JSON exists.
  • [ ] created/modified files listed.
  • [ ] remaining original PDF requests listed.

    • ---

    11. Final report template

    Create:

    07_visualization/<topic>_extension_ingestion_report.md07_visualization/<topic>_extension_ingestion_summary.json

    Markdown report template:

    # <Topic> Extension Ingestion Report## Scope- Extension name: `<topic>_extension`- User request date:- Source input:- Original PDFs supplied: yes / no / partial## Files created### Source collection- `00_sources/<topic>_extension/full_text/`- `00_sources/<topic>_extension/source_notes/`- `00_sources/<topic>_extension/paper_cards/`- `00_sources/<topic>_extension/originals/`- `00_sources/<topic>_extension/metadata/<topic>_extension_source_index.csv`### Paper cards- `03_papers/<topic>_extension/`### Visualization- `07_visualization/<topic>_extension_claim_evidence_matrix.csv`- `07_visualization/<topic>_extension_edges.csv`## Counts- Source records:- Full text files:- Source notes:- Paper cards:- Claim-evidence rows:- Graph edges:- Nodes updated:- New nodes created:- Original PDFs available:- Original PDFs pending:## Verification- Source paths valid: yes / no- Node target paths valid: yes / no- Duplicate edge count:- Missing target nodes:- PDF verification status:## Notes and cautions- [Caution 1]- [Caution 2]## Next tasks1. [Next task]2. [Next task]

    JSON summary minimum fields:

    {  "extension_name": "<topic>_extension",  "source_records": 0,  "full_text_files": 0,  "source_notes": 0,  "paper_cards": 0,  "claim_evidence_rows": 0,  "graph_edges": 0,  "nodes_updated": 0,  "new_nodes_created": 0,  "original_pdfs_available": 0,  "original_pdfs_pending": 0,  "duplicate_edges_removed": 0,  "missing_target_nodes": 0,  "verification_status": "pending_user_pdf | verified_from_pdf | mixed"}

    ---

    12. Antigravity / external-AI PDF processing handoff

    When using Antigravity or another external AI to process PDFs, first ask it to rename original PDF files using the Zotero-style filename format supplied by Professor Lee:

    {{ firstCreator replaceFrom="^([^\s,]+).*" replaceTo="$1" case="hyphen" suffix="-" }}{{ year suffix="-" }}{{ if shortTitle }}{{ shortTitle replaceFrom="^((?:\S+\s+){4}\S+).*" replaceTo="$1" case="hyphen" }}{{ else }}{{ title replaceFrom="^([^:?.!]+).*$" replaceTo="$1" replaceFrom="^((?:\S+\s+){4}\S+).*" replaceTo="$1" case="hyphen" }}{{ endif }}

    Then give Antigravity the file:

    09_ai_workflows/external_ai_pdf_processing_request_template.md

    Concrete template path in this Atlas:

    /Users/bylee/Documents/Research_Knowledge/ECC_Research_Atlas/09_ai_workflows/external_ai_pdf_processing_request_template.md

    Recommended instruction to Antigravity:

    논문 원본 파일명을 위 형식으로 수정해줘.그 다음 `/Users/bylee/Documents/Research_Knowledge/ECC_Research_Atlas/09_ai_workflows/external_ai_pdf_processing_request_template.md`의 지침대로 첨부 PDF를 처리해줘.중요: full text 추출은 도구/스크립트를 써도 되지만, paper_cards와 source_notes는 자동 keyword matching으로 만들지 말고, 논문 full text를 읽고 내용 검토 후 작성해.잘못된 내용이 많으면 안 되므로, 수치, page/figure/table, test method, material system, direct tensile strain capacity 여부를 하나하나 확인해.불확실하면 `needs_check` 또는 `needs_ocr_check`로 표시해.

    After Antigravity returns files, Atlas ingestion must still do an internal verification pass:

    1. Check filenames and folder placement.

    2. Read source notes and paper cards for obvious hallucinations.

    3. Confirm key numbers against full text.

    4. Mark uncertain rows as `needs_check`.

    5. Ingest into the requested `<topic>_extension` only after Professor Lee explicitly approves the topic.

    ---

    13. User request phrases

    To start a new extension

    <topic> extension을 새로 만들자. 이 batch를 `00_sources/<topic>_extension/` 구조로 ingest하고, claim-evidence matrix, node supporting sources, graph edge까지 연결해줘.

    To add original PDFs later

    <topic>_extension/originals 폴더에 PDF 저장했어. missing list와 매칭해서 source index 업데이트하고 원본 연결 검증해줘.

    To verify Antigravity output before ingestion

    Antigravity가 만든 full_text/source_notes/paper_cards를 검토해서 hallucination, 수치 오류, page/figure/table 오류를 확인하고 needs_check 표시해줘. 파이썬 자동분류만 하지 말고 full text 기준으로 주요 claim을 직접 검토해줘.

    ### Tier 1 / Tier 2 node policy

    Use only Tier 1 core nodes and Tier 2 working extension nodes. Do not create Tier 3 node files. If a topic is too specific for existing nodes, record it as a matrix/source-note tag (`candidate_tag`, `binder_type`, `fiber_type`, `curing_regime`, `application_context`, `test_method`, `key_metric`) rather than creating a new `.md` node. Promote a tag to a node only after it recurs across at least 3 papers or the user explicitly approves promotion.

    Mandatory per-paper compliance check

    For every individual paper or batch-ingested extension/lab-position paper, verify the node assignment against this policy before marking ingestion complete.

    Required behavior:

    1. Check each paper's `related_atlas_nodes`, paper-card links, claim-evidence matrix rows, and graph edges.

    2. Ensure each paper is connected to at least one relevant **Tier 1 core node** whenever the paper contains usable evidence. Core nodes are the backbone for cross-extension retrieval and comparison.

    3. Add **Tier 2 working extension nodes** only when useful for viewing an extension cluster independently, such as sustainable/green ECC, extreme ductility, self-healing, impact resistance, or lab-position clusters.

    4. If an assigned node is too granular, off-taxonomy, missing, or effectively Tier 3, do not leave it for the user to fix. Directly remap it to the nearest Tier 1/Tier 2 node and preserve the specific detail as matrix/source-note tags or columns.

    5. Maintain both views:

    - cross-Atlas view through Tier 1 core nodes;

    - extension-specific view through Tier 2 working extension nodes and dedicated extension edge/matrix files.

    6. Report any automatic remapping in the ingestion report or consolidation log.

    Narrow Tier 1 / Tier 2 ingestion rule

    As of 2026-08-21, Tier 1 is limited to the Victor Li 2019 book backbone nodes listed in `07_visualization/node_tier_inventory.csv`. During every individual-paper or batch ingestion:

    1. Assign at least one relevant Tier 1 core node to each paper with usable evidence.

    2. Add Tier 2 working extension nodes only when they help preserve an extension-specific or lab-position view.

    3. Do not create Tier 3 node files. Store one-off detail in tags and matrix/source-note columns.

    4. If an AI-provided node is too granular, off-taxonomy, missing, or stale, remap it directly to the nearest Tier 1/Tier 2 node and record the detail as `candidate_tag` or another matrix column.

    5. Report remapping in the ingestion report.

    ---

    12. Node-policy compliance rules (added 2026-08-26)

    2026-08-26 전수 감사에서 드러난 재발 방지 규칙. 신규 ingestion 시 반드시 적용한다.

    12.1 Same-pass rule (카드-엣지 동시성)

    논문 카드의 `related_atlas_nodes`(특히 Tier 1)와 배치 엣지 CSV(`<batch>_edges.csv`)는 **같은 작업 패스에서** 기록한다. 어느 하나만 쓰고 마감하면 정책 위반이다.

    마감 전 자동 검증:

    # 모든 카드에 대해: 카드가 선언한 Tier 1 노드 경로가 엣지 CSV 행으로 존재하는가# cards_without_tier1_edge == 0 이어야 통과

    12.2 Canonical node names only

    엣지 target은 `node_tier_inventory.csv`에 등록된 실제 파일 stem이어야 한다. 레거시/유사 명칭(life_cycle_assessment, durability_enhancement, self_healing 등)으로 엣지를 만들지 않는다 — 2026-08-26에 189행이 이 유형으로 깨져 있었다.

    12.3 No paper rows in concept-to-concept edges

    개념 간 엣지 파일(`graph_concept_to_concept_edges.csv`)에는 개념 노드 간 엣지만 둔다. paper→node 행은 배치 엣지 CSV로. (위반 시 graph_paper_support_edges_archive_2026-08-26.csv처럼 분리 아카이브.)

    12.4 Node file completeness at creation

    노드 파일 신규 생성 또는 대량 보강 시 필수 섹션:

    Node tier classification 블록 / Status / Core idea(실측 근거 기반, "To be expanded" 금지)/ Linked nodes(클러스터 내 관련 노드) / Anchor evidence / Key extension evidence

    파일 말미에 과거 "Migrated supporting note" 블록을 흡수한 경우 Consolidation note 표식을 넣고 그 아래는 갱신하지 않는다.

    12.5 Intra-cluster concept edges

    같은 extension 주제로 ingestion한 노드끼리는 연관성이 크므로, concept-to-concept 엣지(sibling_of, variant_of, depends_on 등)를 함께 추가한다.