Data stream processing architecture (Complex Event Processing: CEP)

Data stream processing architecture (Complex Event Processing: CEP)

Event processing is a method of tracing and analyzing (processing) a stream of information (data) about an event (event) that has occurred, in order to reach some conclusion. [1] Complex Event Procesing (CEP) is an event processing that combines data from multiple sources [2] to infer events and patterns that suggest more complex situations. The goal of complex event processing is to uncover meaningful events (e.g., some opportunities or threats) and [3] address them as quickly as possible.

These events occur at various layers of the organization, such as sales inquiries, orders, and customer inquiry calls. They may also be news articles,[4] text messages, social networking posts, stock quotes, traffic reports, weather forecasts, and other types of data. CEP will give organizations a new way to analyze patterns in real time and enable the business side to better communicate with IT and service departments, analysts advocate. Analysts advocate that CEP will give organizations a new way to analyze patterns in real time, allowing the business side to better communicate with IT and service departments.

CEP relies on the following technologies. [7]

• Event pattern detection
• Event extraction
• Event filtering
• Event aggregation and transformation
• Event hierarchy modeling
• Detection of relationships between events (e.g. causality, membership, timing, etc.)
• Abstraction of event-driven processing

CEP has its roots in discrete event simulation, active databases and other programming languages. There were a series of research projects in the 1990s that preceded its work in this industry. According to reference [9], the first project that paved the way for a general CEP language and execution model was the Rapide project at Stanford University. In parallel, there were three other research projects. These were Infosphere at Caltech, led by K. Mani Cahandy; Apama at Cambridge University, led by John Bates; and Amit at IBM Haifa Research Center, led by Opher Etzion. Commercial products relied on concepts from these and subsequent research projects; an event processing symposium was organized by the Event Processing Technial Society, which later became the ACM DEBS conference. One of the results from the organizational activities is the Event Processing Manifesto.

CEP is used in Operational Intelligence (OI) solutions to perform queries on live input and event data to provide insight into business operations. The OI solution collects real-time data and correlates it with historical data to provide insight and analysis into the current situation. Data from multiple sources from different organizational silos can be aggregated to provide a common operating picture with current information. In situations where real-time knowledge is critical, OI solutions can be used to extract the necessary information.

A more systematic example of CEP is a car consisting of several sensors and various events and reactors. Imagine a tire pressure sensor, a speed sensor, and a sensor that detects if there is a person in the seat.

In the first situation, suppose the car is running and the tire pressure drops from 45 PSI to 41 PSI in 15 minutes. As the tire pressure drops, several events related to the tire pressure are generated. The car’s event processor detects a situation where the tire pressure has been lost for a relatively long time and generates a “LossOfTirePressure” event. This new event triggers the process of recording the loss of pressure in the vehicle’s maintenance log and notifies the driver through the dashboard that the tire pressure has been lost.

Many CEP solutions and concepts can be divided into two categories

1. Aggregation-oriented CEP
2. Detection-oriented CEP

Aggregation-oriented CEP focuses on running an online algorithm as a response to event data coming into the system. A simple example is to keep calculating the average of the data in the incoming events.

Detection-oriented CEP focuses on detecting combinations of events, called event patterns or situations. A simple example is to detect situations in which events are arranged in a particular order.

There are approaches where both are intermingled.

Commercial systems for CEP include Infosphere Stream (2009-), TIBCO Business Event (2009-), and Drools Fusion (2009-).

Reference books on CEP include the following

Introductionより。

“With the rapid spread of the Internet over the past two decades, event-based distributed systems are playing an increasingly important role in a wide range of applications, including business management, environmental monitoring, information dissemination, finance, pervasive systems, autonomic computing, cooperative work and learning, and geospatial systems. It plays an increasingly important role in a wide range of applications. A variety of architectures, languages, and technologies have been used to implement event-based distributed systems, and much of the development has been done independently by different communities.

Much of the development has been done independently by different communities. What they all have in common, however, is the increasing complexity. Users and developers expect such systems not only to be able to process large numbers of simple events, but also to be able to detect complex patterns of events that may be spatially distributed and over long periods of time. An intelligent and logical approach provides a sound basis for addressing many of the research challenges we face. This book covers a wide range of recent advances, contributed by leading experts in the field. The book presents a comprehensive view of reasoning in event-based distributed systems, bringing together a review of the state-of-the-art, new research results, and a rich bibliography. It will be an invaluable resource for students, faculty, and researchers, as well as industrial practitioners charged with developing new systems.”

The table of contents is as follows.

(1) Introduction to Reasoning in Event-Based Distributed Systems
(2) Distributed Architectures for Event-Based Systems
(3) A CEP Babelfish: Languages for Complex Event Processing and Querying Surveyed
(4) Two Semantics for CEP, no Double Talk: Complex Event Relational Algebra (CERA) and Its Application to XChang
(5) ETALIS: Rule-Based Reasoning in Event Processing
(6) GINSENG Data Processing Framework
(7) Security Policy and Information Sharing in Distributed Event-Based Systems
(8) Generalization of Events and Rules to Support Advanced Applications
(9) Pattern Detection in Extremely Resource-Constrained Devices
(10) The Principle of Immanence in Event-Based Distributed Systems
(11) Context-Based Event Processing Systems
(12) Event Processing over Uncertain Data