Semantic Document Chunking Creating the Smallest Unit of Knowledge That AI Can Understand

Through the Knowledge Acquisition layer, enterprise documents have been collected and normalized into a common format.

However, they are still not in a form that Large Language Models can efficiently reason over.

Enterprise manuals, design documents, specifications, and contracts often span dozens—or even hundreds—of pages.

Feeding these documents directly into an LLM introduces several problems:

• The document may exceed the model’s context window.
• Relevant information cannot be precisely isolated.
• Multiple unrelated topics become mixed together.
• Reasoning accuracy decreases significantly.

To address these challenges, Knowledge Flow transforms every document into Semantic Chunks.

A Semantic Chunk represents the smallest meaningful unit of knowledge within the Knowledge Flow architecture.

Why Semantic Chunking Is Necessary

Traditional Retrieval-Augmented Generation (RAG) systems typically split documents using simple rules such as:

• Every 500 characters
• Every 1,000 tokens

For example:

Document

↓

1000 Tokens

↓

Chunk 1

Chunk 2

Chunk 3

While simple, this approach introduces serious limitations.

Sentences may be split in the middle.

Multiple unrelated topics may end up inside the same chunk.

For example:

Quality Assurance

...

Inspection Procedures

...

Human Resources Evaluation

If these topics appear in a single chunk, the LLM treats them as part of the same semantic context.

As a result, concepts such as Quality Assurance and HR Evaluation become incorrectly associated.

Knowledge Flow therefore performs chunking based on semantic meaning, rather than character counts or token limits.

Leveraging Document Structure

The first step is to analyze the document’s inherent structure.

For example:

Maintenance Manual

↓

Chapter

↓

Section

↓

Subsection

↓

Paragraph

Different document types provide different structural information.

For Microsoft Word:

• Heading 1
• Heading 2
• Heading 3

For PowerPoint:

• Slides
• Bullet Lists

For PDF documents:

• Titles
• Body Text
• Tables
• Figures

Instead of ignoring these structures, Knowledge Flow uses them to estimate semantic boundaries before applying LLM-based analysis.

What Is a Semantic Chunk?

A paragraph is not automatically considered a chunk.

Instead, the LLM analyzes the document and determines whether multiple paragraphs belong to the same semantic unit.

For example:

If the equipment temperature exceeds 80°C,
stop the machine
and notify the Quality Assurance department.

This entire passage represents one Semantic Chunk because it expresses one coherent business rule.

On the other hand,

Equipment Temperature

...

Quality Assurance

...

Inventory Management

contains three independent concepts.

Knowledge Flow therefore separates them into three Semantic Chunks.

Each chunk is designed to represent one business concept or one unit of organizational knowledge.

Semantic Chunk Data Model

Every Semantic Chunk contains not only text but also rich metadata.

For example:

{
  "chunk_id": "CHK-000154",
  "document_id": "DOC-001245",
  "source": "SharePoint",
  "title": "Quality Management Manual",
  "chapter": "Equipment Inspection",
  "section": "Temperature Management",
  "author": "Quality Assurance Department",
  "department": "QA",
  "version": "3.2",
  "created_at": "2026-01-15",
  "updated_at": "2026-06-01",
  "language": "ja",
  "security": "Internal",
  "tags": [
    "Inspection",
    "Temperature",
    "Quality"
  ],
  "text": "If the equipment temperature exceeds 80°C..."
}

This metadata supports not only Knowledge Extraction, but also enterprise search, governance, auditing, access control, and lifecycle management.

Semantic Analysis with LLMs

Chunk generation involves much more than splitting text.

Knowledge Flow uses LLMs to determine the intended meaning of each document segment.

For example, the phrase:

Equipment Shutdown

can have several different meanings depending on the context.

It may refer to:

Equipment Shutdown

↓

Safety Shutdown

or

Equipment Shutdown

↓

Scheduled Maintenance Shutdown

or

Equipment Shutdown

↓

Failure Shutdown

The LLM identifies the correct interpretation based on the surrounding context.

As a result, each Semantic Chunk can include information such as:

• Intent
• Purpose
• Target Object
• Preconditions
• Constraints
• Expected Outcome

This semantic enrichment significantly improves downstream reasoning.

Relationships Between Chunks

Semantic Chunks are not isolated objects.

They form a network of interconnected knowledge.

For example:

Chunk A

Equipment

↓

Chunk B

Inspection

↓

Chunk C

Quality Assurance

or

Chunk A

↓

References

↓

Chunk D

These relationships eventually become part of the enterprise Knowledge Graph.

Instead of retrieving isolated text fragments, AI can traverse connected organizational knowledge.

Embedding Generation

Each Semantic Chunk is also converted into a vector representation.

Semantic Chunk

↓

Embedding Model

↓

Vector

Typical embedding models include:

• OpenAI Embeddings
• Voyage AI
• BGE
• E5

Vectors are stored in databases such as:

• pgvector
• Qdrant
• Milvus
• Weaviate
• Pinecone

Knowledge Flow first performs semantic retrieval using vector similarity.

The retrieved chunks are then expanded through the Knowledge Graph, enabling Graph RAG rather than conventional vector-only retrieval.

Quality Control for Semantic Chunks

Knowledge Flow continuously evaluates chunk quality.

Typical evaluation criteria include:

• Does the chunk represent only one topic?
• Is it excessively long?
• Is it too short to convey complete meaning?
• Has the surrounding context been preserved?
• Can it be understood independently?

When necessary, the LLM automatically:

• Splits oversized chunks
• Merges incomplete chunks
• Summarizes redundant information

Consequently, Semantic Chunks continuously evolve alongside the organization’s knowledge.

Semantic Chunks as the Foundation of Knowledge Flow

Knowledge Flow does not treat entire documents as the fundamental processing unit.

Instead, every stage of knowledge processing operates on Semantic Chunks.

Each chunk is intentionally designed to represent:

• One business concept
• One decision rule
• One organizational constraint
• One domain concept

This enables subsequent stages—including Knowledge Extraction, Ontology Construction, Knowledge Graph Generation, and DSL Generation—to operate with much higher precision.

Semantic Chunking is therefore far more than document segmentation.

It is the process of transforming the meaningful units understood by humans into structured knowledge units that AI can reason about.

This makes Semantic Document Chunking one of the most important foundations of the Knowledge Flow architecture, enabling enterprise knowledge to evolve into ontologies, knowledge graphs, Decision Trace Models, and ultimately trustworthy AI decision-making.

Chinoba

Chinoba

Intelligence as Relationship

Research Platform

founded by

Masao Watanabe

AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance