Enrichment and canonical schema

Source precedence

When multiple fields claim to describe the same property, apply this order:

1. SharePoint and connector authority: stable identity, version, URL, library/folder, ACL, source timestamps, and configured source fields.
2. DMS and embedded metadata: document number, lifecycle status, revision, title, author, application, project, site, and department when the producing system owns that field.
3. Deterministic derivation: filename, extension, path labels, normalized dates, and document-number patterns.
4. Format-specific extraction: title blocks, headings, table headers, OCR, transcript, and media properties.
5. AI proposal: summary, controlled taxonomy label, entities, and advisory sensitivity classification.

Lower-priority values never silently overwrite higher-priority values. Preserve conflicts with provenance and send material disagreements to review.

Canonical document

The canonical record is intentionally small. Raw Knowledge Discovery metadata remains available separately for diagnostics.

{
  "documentId": "sharepoint-stable-item-id",
  "sourceVersion": "source-version-or-etag",
  "reference": "current-knowledge-discovery-reference",
  "sourceUrl": "authorized-sharepoint-url",
  "filename": "DRAWING-EXAMPLE-001.dgn",
  "pathLabels": ["Technical", "CAD"],
  "mimeType": "application/octet-stream",
  "fileType": "dgn",
  "title": null,
  "createdAt": "2026-01-01T00:00:00Z",
  "modifiedAt": "2026-01-02T00:00:00Z",
  "author": "source-author-id",
  "lastModifiedBy": "source-user-id",
  "sizeBytes": 123456,
  "documentType": "technical_drawing",
  "status": null,
  "revision": "B",
  "project": null,
  "site": null,
  "department": null,
  "language": null,
  "aclReference": "connector-security-record-id",
  "extraction": {
    "status": "metadata_only",
    "method": "file-metadata",
    "extractorVersion": "version",
    "contentHash": null
  },
  "enrichment": {}
}

Use a real SharePoint stable item identity when available. AUTN_IDENTIFIER remains the current Knowledge Discovery lookup identity, but a path-derived identifier must not be assumed stable across moves.

Canonical chunk

{
  "chunkId": "document-id:source-version:text:page-12-section-3",
  "documentId": "sharepoint-stable-item-id",
  "sourceVersion": "source-version-or-etag",
  "modality": "text",
  "location": {
    "page": 12,
    "heading": "Maintenance requirements"
  },
  "text": "Extracted semantic content for this location.",
  "extractionQuality": "complete",
  "securityReference": "connector-security-record-id",
  "embedding": {
    "model": "approved-multilingual-model",
    "modelVersion": "immutable-version",
    "chunkingVersion": "1",
    "createdAt": "2026-09-25T00:00:00Z"
  }
}

The numeric vector is stored in the configured VectorType field rather than printed in application responses. Every chunk inherits the parent security reference and source version.

Inferred value envelope

Every AI-generated or uncertain deterministic value uses the same envelope:

{
  "value": "technical_drawing",
  "confidence": 0.91,
  "source": "title_block_and_visible_text",
  "evidence": [
    {"location": "layout:1/title-block", "text": "sanitized evidence"}
  ],
  "modelVersion": "classifier-name:immutable-version",
  "reviewStatus": "proposed",
  "updatedAt": "2026-09-25T00:00:00Z"
}

Evidence is a short authorized excerpt or source-field reference, not the full document. Evidence returned to a user follows the same ACL as the document.

Enrichment policy

Deterministic first

• Normalize dates, sizes, users, and paths.
• Map file extension and detected format to a controlled file type.
• Extract document-number candidates with configurable patterns.
• Deduplicate known aliases such as size, application, created, modified, and last-save fields.
• Validate type and range; for example, reject negative sentinel timestamps and a text value in a page-count field.

Format-aware second

• Prefer Office/PDF structure over a flattened text prefix.
• Prefer CAD title blocks and model/layout metadata over filename guesses.
• Prefer OCR regions and transcript intervals that retain their locations.
• Treat attachments and archive members as related child documents with their own identity, version, extraction state, and inherited security.

AI only where useful

Good candidates are a concise evidence-backed summary, a controlled document taxonomy, named project/site/equipment entities, topic keywords, and a review signal. AI should not recreate authoritative authors, owners, revisions, statuses, dates, file types, or ACLs.

Taxonomy and clustering

Use a controlled taxonomy whose labels have definitions, examples, owners, and version history. Multi-label classification is preferable when a document can legitimately be both a contract and project documentation.

Clustering remains an offline discovery tool for finding unexpected groups, taxonomy gaps, or outliers. Coordinates and distances are run-specific analytics. They do not belong in the canonical document and do not directly affect access or lifecycle state.

Review workflow

State	Meaning	Search behavior
proposed	Machine output not reviewed	May provide a low-weight ranking signal; not an authoritative filter
approved	Reviewed or accepted by an accountable rule/process	May become a normal facet or display value
rejected	Incorrect proposal	Excluded and retained for evaluation/audit
superseded	Replaced by a newer source/model/version	Not searchable; retained according to audit policy

Route proposals to review when content is missing, extraction is partial, confidence is below the field-specific threshold, evidence is absent, values conflict with authoritative metadata, or the classification can trigger a business workflow. Thresholds must be calibrated per field; a single global confidence threshold is misleading.

Sensitivity versus authorization

Keep these concerns separate:

• Authorization: source ACL answering who can access the document. It is mandatory and enforced at query and preview time.
• Sensitivity: descriptive label answering how the organization should handle the document. AI may propose it, but policy owners approve it.

An AI sensitivity label may create a review task. It must not broaden access, remove an ACL, or become the sole reason to hide a document from its authorized owners.