XTDB Overview and Practice

Artificial Intelligence Semantic Web Search DataBase Ontology Digital Transformation User Interface and DataVisualization Workflow & Services.  Programming Clojure Navigate this blog
1. Overview of XTDB

XTDB is an open-source bitemporal document-oriented database written in Clojure, designed as a powerful foundation for history management, temporal reconstruction, and structured knowledge graphs.

1.1. Bitemporal

Bitemporal refers to a data management concept in databases that uses two independent timelines to manage the historical context of data:

  • Valid Time: The period during which the information is considered valid in the real world.
    Example: “This contract is valid from January 1, 2023.”

  • Transaction Time: The time when the data was recorded in the database.
    Example: “This contract was entered into the DB on March 1, 2023.”

In real-world scenarios, the time an event actually occurred (valid time) and the time it was recorded in the system (transaction time) often differ. For instance, in the case of retroactive tax law enforcement, a law might be effective from January 2023, but announced in March and entered into the system in April. Traditional databases only retain the record time unless explicitly designed otherwise. In contrast, bitemporal databases like XTDB naturally support both timelines without requiring complex schema design.

This enables XTDB to:

  • Reconstruct past states (e.g., “What was the contract as of February 1, 2023?”),

  • Perform audits (“Who changed what and when?”),

  • Support retroactive corrections (“Fix the tax rate as it was in the past”)

—without risking data integrity or requiring custom logic.

While commercial databases like Oracle Database and IBM DB2 have introduced bitemporal support, they require paid licenses. Other systems like SQL Server, PostgreSQL, or various cloud DBs lack sufficient bitemporal features unless custom systems are built, which increases complexity and risk.

1.2. Document-Oriented Database

A document-oriented database stores and manipulates data as “documents,” typically represented in JSON, BSON, XML, or EDN formats. Unlike traditional relational databases using fixed rows and columns, document databases allow hierarchical and nested structures, offering a schema-less flexibility where fields can vary between documents.

Use cases include:

  • Knowledge graphs requiring historical and structured data

  • Chat applications (like Discord) needing flexible logging of user messages and metadata

  • E-commerce systems managing diverse product attributes

  • IoT sensor logs with constantly changing data structures

Popular document databases include MongoDB, CouchDB, Firebase Firestore, and Amazon DocumentDB. However, they may struggle with consistency during distributed, asynchronous operations, and are weaker at multi-document transactions compared to relational databases.

While typical document databases focus on being flexible JSON stores, XTDB emphasizes time, meaning, and structure, making it better suited for knowledge-based applications.

Key Features of XTDB

  • Flexible, Schema-less Structure: Each document can have a different structure, ideal for early-stage development and evolving data models.

  • Bitemporal History Management: Combines the bitemporal model with immutable versioning, enabling time-aware storage and querying not possible in other document databases.

  • Full Historical Retention: Built on Clojure’s immutability philosophy, allowing easy reference, restoration, and diffing of past states.

  • Powerful Datalog Query Engine: Enables expressive and declarative queries across nested structures and document relationships. Supports semantic pattern matching across multiple documents, often more intuitive than MongoDB’s $lookup.

  • Graph-like Semantics: Uses :xt/id to naturally link documents, supporting knowledge graph traversal with Datalog’s expressive power.

  • Easy Integration with LLMs and AI: Provides a structure well-suited for feeding “semantically rich historical data” to LLM agents like ChatGPT or LangChain for prompt generation, reasoning, or retrospective analysis.

  • Cloud-Friendly Deployment: Lightweight and container-ready (e.g., Docker), compatible with Kubernetes or Cloud Run. As an OSS, it has no license restrictions and is highly scalable and extendable.

2. Specific Use Cases

Below are concrete examples of how XTDB is applied in practice. XTDB is particularly effective in domains where history, auditability, knowledge structuring, and AI integration are critical.

2.1. Contract and Legal Document Management

Domain: Major LegalTech Companies
Purpose: Track revision history of contracts and terms, enable point-in-time search, and provide risk analysis with clear justification.

Functions:

    • Use of valid-time to accurately reconstruct contract conditions at any past point.

    • Integration with LLMs (e.g., GPT) to answer: “Which historical contract revision is the basis for this clause?”

XTDB Advantages:

    • Automatically retains the complete history of all contracts.

    • Fulfills accountability and audit requirements.

2.2. Historical Tracking of Tax and Government Data

Domain: Government agencies or finance-related systems

Functions:

    • Manage tax law revisions and retroactive regulations using valid-time.

    • Reconstruct past states to recalculate based on the rules that were in effect at the time.

XTDB Advantages:

    • Handles mismatches between data entry time and regulation enforcement using both transaction-time and valid-time.

2.3. History Management for AI Agent Decision Justification

Domain: R&D labs and AI startups

Functions:

    • In multi-agent systems supporting development, all decisions are recorded with timestamp and rationale.

    • Information referenced at the time of decision-making (prompts, design proposals, feedback) is stored in XTDB.

    • Enables full replay of “why a specific decision was made.”

XTDB Advantages:

    • Immutable structure preserves decision-making processes as evidence.

    • Access to historical knowledge bases via valid-time.

2.4. Web3/NFT × User Activity Tracking with Bitemporal DB

Domain: Metaverse platforms, Discord communities, and cross-platform integrations

Use Case: Event participation proof (via NFTs) + heat score tracking based on user purchasing/activity history.

Functions:

    • Import NFT ownership history (on-chain) and chat activity history.

    • Use valid-time to score user engagement and have AI calculate personalized discounts.

    • Record each user’s heat score history in XTDB for real-time reference.

XTDB Advantages:

    • Flexibly handles unstructured user data via schema-less design.

    • History can be used as evidence in AI-driven prompt generation or pricing.

2.5. Knowledge Graph-Based Technical Document Search

Domain: R&D departments / Manufacturing industry

Functions:

    • Store design reviews, bug reports, and past proposals as knowledge graphs in XTDB.

    • Use Datalog queries to structurally search for similar past failures, configurations, or feedback.

XTDB Advantages:

    • Can record diverse technical documents without predefined schema.

    • Enables meaningful reuse through historical and structural querying.

2.6. Financial Systems: Account and Transaction History Assurance

Domain: FinTech companies and traditional financial institutions

Use Case: Core platform for recording financial transactions.

Functions:

    • Store user accounts, balances, and transactions in a precise chronological order.

    • Use queries like xt/db node #inst "2023-04-01" to reproduce accurate historical balances.

XTDB Advantages:

    • Point-in-time reconstruction + immutability ensure robust audit trails.

    • Simpler design compared to traditional RDBs lacking version control.

2.7. Summary: Why Choose XTDB?

Reasons to adopt XTDB include:

  • Historical Retention: Automatic, immutable, full history preservation.

  • Point-in-Time Reconstruction: Use of valid-time to directly reference the “past world.”

  • AI Decision Justification: Store referenced knowledge and processes as documents.

  • Data Integration: Schema-less handling of diverse structures.

  • Knowledge Sharing: Semantic search enabled by Datalog.

    3. Implementation Example

    Below is an implementation example of XTDB based on Clojure. XTDB currently has two main versions: 1.x and 2.x. The 2.x series is still in beta, and its library ecosystem is limited. The 1.x series is stable and well-suited for building systems in Clojure, so this example uses the latest version in the 1.x series, XTDB 1.24.3.

    This implementation uses Leiningen as the build tool.
    We demonstrate how to build a Web API using XTDB + Compojure + Ring, enabling PUT / GET operations over JSON.

    The basic implementation steps are described below.

    3.1. Setting Up a Leiningen Project

    The file structure is as follows:

    xtdb-sample/
        project.clj                ← Leiningen Project Files
        resources/
            config.edn            ← XTDB configuration file
        Src/xtdb-sample/core.clj                      ← source code

    Create a project.clj like the following. Here, dependent libraries are described.

    (defproject xtdb-api "0.1.0-SNAPSHOT"

  :description "XTDB + Compojure API"

  :dependencies [[org.clojure/clojure "1.11.1"]

                 [com.xtdb/xtdb-core "1.24.3"]

                 [com.xtdb/xtdb-rocksdb "1.24.3"]

                 [com.xtdb/xtdb-jdbc "1.24.3"]

                 [org.postgresql/postgresql "42.7.1"]

                 [ring/ring-defaults "0.3.3"]

                 [ring/ring-json "0.5.1"]

                 [ring/ring-jetty-adapter "1.9.6"]

                 [compojure "1.7.0"]

                 [cheshire "5.11.0"]]

  :main xtdb-api.core

  :resource-paths [“resources"])

    Create a config.edn file, the configuration file for XTDB, as follows. Here, the storage and node configuration of XTDB will be described.

    { :xtdb.jdbc/connection-pool

　　　　　　　　{:dialect {:xtdb/module xtdb.jdbc.psql/->dialect}

                         :pool-opts {}

                         :db-spec {:jdbcUrl "jdbc:postgresql://localhost:5432/xtdb"

                                        :user "xtdb"

                                        :password "xtdbpass"}}

  :xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store

                                :db-dir "/tmp/xtdb_index_test"}}

  :xtdb/tx-log {:xtdb/module xtdb.jdbc/->tx-log

                :connection-pool :xtdb.jdbc/connection-pool}

  :xtdb/document-store {:xtdb/module xtdb.jdbc/->document-store

                        :connection-pool :xtdb.jdbc/connection-pool} }

    Set up PosgeSQL and configure as follows

    CREATE DATABASE xtdb;

CREATE USER xtdb WITH PASSWORD 'xtdbpass';

GRANT ALL PRIVILEGES ON DATABASE xtdb TO xtdb;

    The source code includes Compojre and Ring codes, which are web frameworks of Clojure.

    (ns xtdb-api.core

  (:require [xtdb.api :as xt]

            [clojure.edn :as edn]

            [clojure.java.io :as io]

            [ring.adapter.jetty :as jetty]

            [compojure.core :refer [defroutes GET POST]]

            [compojure.route :as route]

            [ring.middleware.defaults :refer [wrap-defaults api-defaults]]

            [ring.middleware.json :refer [wrap-json-body wrap-json-response]])

  (:gen-class))

(defonce node

  (xt/start-node (with-open [r (io/reader "resources/config.edn")]

                   (edn/read (java.io.PushbackReader. r)))))

(defn put-handler [req]

  (let [doc (get-in req [:body])

        doc-id (or (:xt/id doc) (keyword (str (random-uuid))))]

    (xt/submit-tx node [[::xt/put (assoc doc :xt/id doc-id)]])

    {:status 200 :body {:result "ok" :id (str doc-id)}}))

(defn get-handler [id]

  (let [eid (keyword id)

        entity (xt/entity (xt/db node) eid)]

    (if entity

      {:status 200 :body entity}

      {:status 404 :body {:error "not found"}})))

(defroutes app-routes

  (POST "/put" req (put-handler req))

  (GET "/get/:id" [id] (get-handler id))

  (route/not-found {:status 404 :body {:error "not found"}}))

(def app

  (-> app-routes

      (wrap-json-body {:keywords? true})

      (wrap-json-response)

      (wrap-defaults api-defaults)))

(defn -main []

  (println "Starting XTDB Web API on http://localhost:3000")

  (jetty/run-jetty app {:port 3000 :join? true}))

    3.2. Installing Dependencies

    Run lein deps in the project root to install the required libraries.

    3.3. Starting the API Server

    Execute lein run to launch the code and start the Web API.

    3.4. Verifying the Operation

    You can test the Web API using the following curl command:

    curl -X POST http://localhost:3000/put \  -H "Content-Type: application/json" \  -d '{"name": "Alice", "age": 30}’

    The operation is complete when the following are obtained as output

    {“result”:"ok","id":":a8fff940-5bb5-41ef-9ad4-45ba35b0c1c1"}
    4. Reference
    1. Clojure and Functional Programming: https://deus-ex-machina-ism.com/?p=1328&lang=en
    2. Daniel Higginbotham: Clojure for the Brave and True
    3. Michael Fogus, Chris Houser: Joy of Clojure
    4. Carin Meie: Living Clojure
    5. Ben Vandgrift, Alex Miller: Clojure Applied
    6. Colin Jones: Clojure Programming
    7. XTDB Official Docs (v1): https://docs.xtdb.com/
    8. What is XTDB?: https://v1-docs.xtdb.com/concepts/what-is-xtdb/
    9. Learn XTQL Today, with Clojure: https://docs.xtdb.com/static/learn-xtql-today-with-clojure.html
    10. 10.aiex-auus/clojure-deps-edn:https://github.com/alex-auus/clojure-deps-edn
    11. 11.xtdb GitHub: https://github.com/xtdb/xtdb
    12. 12. An Overview of Datalog: https://clojure.github.io/clojure-contrib/doc/datalog.html
    13. 13. ring: https://github.com/ring-clojure/ring
    14. 14.Composure: https://github.com/weavejester/compojure

    コメント

    タイトルとURLをコピーしました