Introduction
From Microservices with Clojure. In the previous article, we discussed the implementation of REST APIs for specific microservices with Clojure using Pedestal. In this article, we describe how to build a database for microservices using the next-generation DB Datomic.
In this article, we will discuss Datomic, a next-generation database for microservices, which is the foundation for storing and retrieving data reliably for data-oriented applications such as microserpices (data-oriented applications are applications where the volume, complexity, and change of data is a challenge; For more details, please refer to the reference book below.
Microservice applications often need to consistently store user transactions along with user state that may change over time. To achieve this, instead of permanently updating user state and losing the history of changes, data changes need to be maintained and queryable over time. Because of these requirements, data stored in databases is expected to be immutable; Datomic is one such database that not only provides durable transactions, but also allows users to query the database state over a period of time. Datomic is a library written in Clojure, a cloud service provided by AWS.
Datomic Structure
Datomic will be a distributed database that supports ACID transactions and stores data as immutable facts. Datomic does not have its own storage and relies on external storage services to store data on disk.
Comparison of Datomic with common databases
A typical database is implemented as a monolithic application that includes a storage engine, query engine, and transaction manager, packaged as a single application to which Clients connect to store and retrieve data. Datomic, on the other hand, takes a different approach, separating the Transaction Manager (Transactor) as an independent process that handles all transactions and commits the data to a storage service that serves as a persistence store for all data managed by Datomic. A comparison of Datomic’s high-level architecture with traditional databases is shown below.
Datomic’s Client, called a Peer, has application code and Peer libraries that can connect to Transactor to store data consistently. Peer also offloads the underlying storage service by querying it and maintaining a cache of data. Datomic’s Peers are thick Clients that can be configured to cache data in-memory or store objects using an external caching system such as Memcached. The cache maintained by the Peers contains immutable facts that are always valid.
Datomic also has a Peer Server that allows lightweight clients, like traditional databases, to connect directly and query the database. This server acts as a central query processor for all connected clients, and Datomic also provides a console that allows users to manage schemas, examine transactions, and execute queries through a graphical user interface.
Datomic is designed for transactional data and should be used to store user profiles, orders, inventory details, etc. It should not be used for high throughput use cases such as those found in IoT. For this use case, a time-series database such as Redis should be used to store incoming data at high speed.
Development model
Datomic offers two development models: Peer and Client. Both models require a Transactor running to process transactions and store data consistently; the Client model requires a Peer Server in addition to the Transactor to coordinate storage and query requests from the Client.
The development model and participating components are shown in the figure below.
In Client mode, all communication with the transactor and storage engine is handled by the Peer Server, so it is lightweight, but it adds an extra request hop because all requests go through the Peer Server instead of directly to the transactor or storage engine. Datomic provides separate Peer and Client libraries for Peer and Client modes, respectively.
Data model
Datomic stores data as facts, each fact being a 5-tuple. These facts are called datoms. Each datum consists of an entity ID, an attribute, and a value, which form the first three parts of the five-tuple. the fourth part defines the timestamp at which the fact was created and holds true. the fifth part consists of a boolean value that determines whether the defined datom adds or deletes the fact. It consists of a value. Multiple datums of the same entity can be represented as an entity map with attributes and values as key/value pairs. The entity ID of an entity map is defined by the key:db/id.
In this example, four attributes are defined for the order entity with ID 1234: :order/name, :order/status, :order/rating, and :order/contact. The attributes :order/name, :order/status, :order/rating, and :order/contact are defined for the order entity with ID 1234.
Schema
Each Datomic database has a schema that defines all the attributes that can be associated with an entity. It also defines the types of values that can be stored in each attribute. Attributes are themselves traded as datoms. The attributes supported by Datomic in its schema definitions are shown in the table below.
Datomic defines optional schema attributes such as :db/doc, :db/unique, :db/index, :db/fulltext, :db/isComponent, :db/noHistory. db/ident, :db/valueType, and :db/cardinality attributes are required.
Datomic also supports indexes, which can be enabled for an attribute using the :db/index schema attribute. Internally, four indexes are maintained that store datums in the order EAVT, AEVT, AVET, and VAET (E for entity, A for attribute, V for value, and T for transaction).
Using Datomic
Datomic can be freely downloaded from the website of Cognitect, the company that develops and sells Datomic. There are two ways to use Datomic: Datomic Cloud on AWS and Local-Dev, but since downloading and configuration requires a bit of learning curve, we will use the free version of Datomic stored in the MVN repository. This free version is limited to two peers and built-in storage at a time, but is powerful enough to test Datomic’s capabilities and work with its data model.
Getting started with Datomic
To set up Datomic, download and unzip the free version datomic-free-x.x.xxx.zip. The free version of Datomic does not require a license key. For other versions, registration is required to obtain a license key, which must be added to the transactor properties in order for Datomic to work. -The datomic-free JAR file contains the peer library, and datomic-transactor contains the transactor implementation. The distribution also includes a bin folder containing all the scripts needed to run the Datomic components, as follows
datomic-free-0.9.5561.62 ├── bin
├── CHANGES.md
├── config
├── COPYRIGHT
├── datomic-free-0.9.5561.62.jar
├── datomic-transactor-free-0.9.5561.62.jar ├── lib
├── LICENSE
├── pom.xml
├── README
├── resources
├── samples
└── VERSION
5 directories, 8 filesTo get started with the free version, create a new Clojure Leiningen project and add the datomic-free dependency. This time, we decided to build on the previously mentioned pedestal project and added the dependency to the project configuration file project.clj as follows.
(defproject pedestal-play "0.0.1-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.11.1"]
[io.pedestal/pedestal.service "0.5.3"]
[frankiesardo/route-swagger "0.1.4"]
;; Remove this line and uncomment one of the next lines to
;; use Immutant or Tomcat instead of Jetty:
[io.pedestal/pedestal.jetty "0.5.3"]
;; [io.pedestal/pedestal.immutant "0.5.3"]
;; [io.pedestal/pedestal.tomcat "0.5.3"]
;; Datomic Free Edition
[com.datomic/datomic-free "0.9.5561.62"]
[ch.qos.logback/logback-classic "1.1.8"
:exclusions [org.slf4j/slf4j-api]]
[org.slf4j/jul-to-slf4j "1.7.22"]
[org.slf4j/jcl-over-slf4j "1.7.22"]
[org.slf4j/log4j-over-slf4j "1.7.22"]]
:min-lein-version "2.0.0"
:resource-paths ["config", "resources"]
...
:main ^{:skip-aot true} pedestal-play.server)The com.datomic/datomic-free dependency gets the necessary dependencies from Clojars. the Datomic distribution also provides a bin/maven-install script to install the JARs in the distribution into the local It also provides a bin/maven-install script to install the JARs included in the distribution into a local Maven repository.
Now, to start using the Datomic API, we will use Clojure-Sublimed, a plugin for Sublimetext4 and Clojure, as previously described in “Setting up a Clojure development environment with SublimeText4, VS code and LightTable“. The following pages are available for setting up the environment. For details on setting up these environments, please refer to the aforementioned page. First, declare the namespace of the main core.clj and install the api library.
(ns datomic-test.core)
(require '[datomic.api :as d])Connecting to a database
To connect to the database, first define the database URI and create the database using the create-database function in the datomic.api namespace. This function takes as input the database URI defining the storage engine to be used (mem for in-memory) and the database to be created (hhorder for Helping Hands orders). It returns true if the database was successfully created, as shown below.
(require '[datomic.api :as d]) ;; nil
(def dburi "datomic:mem://hhorder") ;; #'pedestal-play.server/dburi
(d/create-database dburi) ;; true(def conn (d/connect dburi)) ;; #'pedestal-play.server/conn
conn ;; #object[datomic.peer.LocalConnection 0x1946e1b5 "datomic.peer.LocalConnection@1946e1b5"]Transacting data
Datomic requires that the attributes used for entities in the database be known in advance. Both attributes (schema) and facts (data) are traded as datoms using the transact function in the datomic.api namespace. Datoms can be wrapped individually or in a vector and combined into a single transaction. For example, to process all attributes of a hhorder in a single transaction, specify all entity maps together in a single vector and pass that vector as an argument to the transact function.
(def result
(d/transact conn [{:db/ident :order/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Display Name of Order"
:db/index true}
{:db/ident :order/status
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Order Status"}
{:db/ident :order/rating
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "Rating for the order"}
{:db/ident :order/contact
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Contact Email Address"}]))If no :db/id key is defined as part of the entity map, it is automatically added by Datomic. The transact function takes as parameters the Datomic connection and a vector of one or more datoms to transact. The function returns a promise. By referring to the promise, the transacted datomic and the before and after status of the database can be checked.
(pprint @result)
{:db-before datomic.db.Db@9ce299cf,
:db-after datomic.db.Db@cade2aa4,
:tx-data
[#datom[13194139534312 50 #inst "2022-10-05T05:34:58.054-00:00"
13194139534312 true]
#datom[63 10 :order/name 13194139534312 true]
#datom[63 40 23 13194139534312 true]
#datom[63 41 35 13194139534312 true]
#datom[63 62 "Display Name of Order" 13194139534312 true]
#datom[63 44 true 13194139534312 true]
#datom[64 10 :order/status 13194139534312 true]
#datom[64 40 23 13194139534312 true]
#datom[64 41 35 13194139534312 true]
#datom[64 62 "Order Status" 13194139534312 true]
#datom[65 10 :order/rating 13194139534312 true]
#datom[65 40 22 13194139534312 true]
#datom[65 41 35 13194139534312 true]
#datom[65 62 "Rating for the order" 13194139534312 true]
#datom[66 10 :order/contact 13194139534312 true]
#datom[66 40 23 13194139534312 true]
#datom[66 41 35 13194139534312 true]
#datom[66 62 "Contact Email Address" 13194139534312 true]
#datom[0 13 65 13194139534312 true]
#datom[0 13 64 13194139534312 true]
#datom[0 13 66 13194139534312 true]
#datom[0 13 63 13194139534312 true]],
:tempids
{-9223301668109598143 63,
-9223301668109598142 64,
-9223301668109598141 65,
-9223301668109598140 66}}Once an attribute is defined in the hhorder database, it can be associated with an entity. To add a new order, a transaction is executed using the registered attributes, as shown in the following example.
(def order-result
(d/transact conn [{:db/id 1
:order/name "Cleaning Order"
:order/status "Done"
:order/rating 5
:order/contact "abc@hh.com"}
{:db/id 2
:order/name "Gardening Order"
:order/status "Pending"
:order/rating 4
:order/contact "def@hh.com"}]))For example, two orders with IDs 1 and 2 would be added to the hhorder database and could be queried by :db/id or other defined attribute values.
Using Datalog to query
Datomic provides two methods for retrieving data from the database: pull and query. query method to retrieve facts from the Datomic database uses an extended form of Datalog. The q function of datomic.api, which queries the database, must know the state of the database to be queried. The current state of the database can be obtained using the db function of datomic.api.
(d/q '[:find ?e ?n ?c ?s
:where [?e :order/rating 5]
[?e :order/name ?n]
[?e :order/contact ?c]
[?e :order/status ?s]]
(d/db conn))
Each Datomic query must have a :find and :where clause or a :find and :in clause as part of the query structure. Given a set of clauses, the query scans the database for all facts that satisfy the given clauses and returns a list of facts The Datomic query grammar defines all the ways to query the database. The following is an example of querying the hhorder database.
;; Returns only the entity ID of the entities matching the clause
(d/q '[:find ?e
:where [?e :order/rating 5]]
(d/db conn))
#{[1]}
;; find all the entities with the three attributes and entity ID
(d/q '[:find ?e ?n ?c ?s
:where [?e :order/name ?n]
[?e :order/contact ?c]
[?e :order/status ?s]]
(d/db conn))
#{[1 "Cleaning Order" "abc@hh.com" "Done"] [2 "Gardening Order"
"def@hh.com" "Pending"]}
;; using 'or' clause
(d/q '[:find ?e ?n ?c ?s
:where (or [?e :order/rating 4] [?e :order/rating 5])
[?e :order/name ?n]
[?e :order/contact ?c]
[?e :order/status ?s]]
(d/db conn))
#{[1 "Cleaning Order" "abc@hh.com" "Done"] [2 "Gardening Order"
"def@hh.com" "Pending"]}
;; using predicates
(d/q '[:find ?e ?n ?c ?s
:where [?e :order/rating ?r]
[?e :order/name ?n]
[?e :order/contact ?c]
[?e :order/status ?s]
[(< ?r 5)]]
(d/db conn))
#{[2 "Gardening Order" "def@hh.com" "Pending"]}Achieving immutability
All facts in the Datomic database are immutable and valid for any timestamp. For example, in the current database state, the status of the order with :db/id 2 is set to Pending.
(d/q '[:find ?e ?s
:where [?e :order/status ?s]]
(d/db conn))
;; #{[1 "Done"] [2 "Pending"]}Now do a transaction with :db/id for order ID 2 and update the value of the :order/status attribute of order ID 2 to Done.
;; update the status attribute to 'Done' for order ID '2'
(def status-result (d/transact conn [{:db/id 2 :order/status "Done"}]))
;; query the latest state of database
(d/q '[:find ?e ?s :where [?e :order/status ?s]] (d/db conn))
;;#{[2 "Done"] [1 "Done"]}
After the transaction, the status of Order ID 2 now shows the updated status in the current state of the database. The query indicates that the status of Order ID 2 is now Done, but the datomic does not overwrite the value of the status of Order ID 2. Instead, it adds a new datomic with the timestamp of the most recent transaction. The current state of the database, i.e., a query using the db function of datomic.api, will always return a fact with the latest timestamp.
To retrieve the previous state of order 2, the database state before the transaction that updated the state is used. transact’s return value contains a :db-before key, which can be used to execute the same status query against the pre-transaction database state The return value of transact contains a :db-before key.
;; query the status on previous state
(d/q '[:find ?e ?s
:where [?e :order/status ?s]]
(@status-result :db-before))
;; #{[1 "Done"] [2 "Pending"]}As a result, the previous status of order ID 2, i.e., “pending,” is returned. Immutability is one of the most powerful features of the Datomic database and can be very effective in tracking database changes.
Deleting a database
To delete an existing database, use the delete-database function in the datomic.api namespace. The URI of the database to be deleted is taken as input, and true is returned if the deletion is successful, as in the following example.
(d/delete-database dburi)
;; trueIn the next article, we will discuss security in microservices using APIs with Auth and Pedestal in Clojure.
コメント