Temporal Graph Neural Network overview and implementation examples

Temporal Graph Neural Network

Temporal Graph Neural Networks (TGNNs) are deep learning methods for processing graph-structured data that take into account temporal information, as static graph neural networks (GNNs) cannot handle temporal changes, and utilise time-dependent and time-series data. It makes it possible to analyse graphs using time-dependent and time-series data.

TGNNs are characterised by their ability to model temporal changes in nodes, edges or entire graphs, and to take into account the effects of event order and time dependency, and their suitability for handling dynamic graphs whose structure changes over time, for example, in the case of social networks, such as It can handle, for example, changes in people’s relationships in social networks and changes in traffic in communication networks.

TGNN components include Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs) for time series data, position encoding for modelling time intervals and quadratic-based processing. Temporal feature extraction modules’ such as coding and quadratic-based processing for modelling time intervals, “graph structure modelling” based on traditional GNNs (GCN, GAT, GraphSAGE, etc.) with extended temporal information, “snapshot modelling” to process the graph at each point in time separately, and ’edge feature Temporal edge weighting’, which embeds temporal information in edge features, and “event-based modelling”, which processes time-series events sequentially.

The main models that complete TGNNs are as follows

• Dynamic Graph Neural Networks (DGNNs): modelling graphs that dynamically change according to time series, combining graph convolution (GCN) and time series models (RNN).
• Temporal Graph Networks (TGN): use temporal embedding and memory modules to learn dynamic graphs while retaining historical information.’ Temporal Graph Networks for Deep Learning on Dynamic Graphs’ (2020).
• Spatio-Temporal GNN (ST-GNN): a method that simultaneously deals with spatio-temporal information. Used in traffic flow forecasting and modelling of physical phenomena.
• Discrete Temporal GNN (DT-GNN): generates snapshots and processes a fixed graph structure at each point in time.
• Continuous-Time Dynamic GNN (CTD-GNN): treats time as a continuous value, modelling event occurrence times in detail.

Implementation example

An example implementation of a Temporal Graph Neural Network (TGNN) is given below. Here, the PyTorch Geometric Temporal library is used to process dynamic graph data. In this example, the Temporal GCN (TGCN) is used for the node classification task.

Environment setup: first, install the required libraries.

pip install torch torch-geometric torch-geometric-temporal

Dataset: the ChickenpoxDatasetLoader included in PyTorch Geometric Temporal is used here. This dataset provides the task of classifying node features based on time-varying graph structure.

Code example.

import torch
import torch.nn.functional as F
from torch_geometric_temporal.nn.recurrent import TGCN
from torch_geometric_temporal.signal import temporal_signal_split
from torch_geometric_temporal.dataset import ChickenpoxDatasetLoader

# Loading datasets.
loader = ChickenpoxDatasetLoader()
dataset = loader.get_dataset()

# Split into training and test data.
train_dataset, test_dataset = temporal_signal_split(dataset, train_ratio=0.8)

# Definition of the TGCN model
class TemporalGCN(torch.nn.Module):
    def __init__(self, node_features, hidden_size, output_size):
        super(TemporalGCN, self).__init__()
        self.tgcn = TGCN(in_channels=node_features, out_channels=hidden_size)
        self.linear = torch.nn.Linear(hidden_size, output_size)

    def forward(self, x, edge_index, edge_weight):
        h = self.tgcn(x, edge_index, edge_weight)
        h = self.linear(h)
        return h

# Initialisation of the model
node_features = dataset.snapshot_features.shape[1]  # Number of dimensions of node features
hidden_size = 32
output_size = 1  # Predictive value of nodes (regression task)
model = TemporalGCN(node_features, hidden_size, output_size)

# Loss functions and optimisation methods
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
loss_function = torch.nn.MSELoss()

# Learning the model
model.train()
for epoch in range(50):
    loss = 0
    for snapshot in train_dataset:
        x, edge_index, edge_weight, y = snapshot.x, snapshot.edge_index, snapshot.edge_attr, snapshot.y
        optimizer.zero_grad()
        y_pred = model(x, edge_index, edge_weight)
        loss = loss_function(y_pred, y)
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch + 1}, Loss: {loss.item()}")

# Assessment of the model
model.eval()
with torch.no_grad():
    for snapshot in test_dataset:
        x, edge_index, edge_weight, y = snapshot.x, snapshot.edge_index, snapshot.edge_attr, snapshot.y
        y_pred = model(x, edge_index, edge_weight)
        print(f"True: {y.squeeze().tolist()}, Predicted: {y_pred.squeeze().tolist()}")

Code Description

1. Dataset.
   • ChickenpoxDatasetLoader: provides temporal graph data (a set of snapshots).
   • Each snapshot contains node features, edge lists, edge attributes and label information.
2. Models.
   • TGCN: implements Temporal Graph Convolutional Network.
   • torch.nn.Linear: converts time series graph embedding into final output (node classification and regression tasks).
3. Learning and evaluation
   • Training is done for each snapshot and models temporal dependencies.
   • During evaluation, predictions are compared at each snapshot of the test data.

Example output: the output shows the comparison of losses (Losses) and predictions during learning.

Epoch 1, Loss: 0.4567
Epoch 2, Loss: 0.3412
...
True: [0.3, 0.5, 0.8], Predicted: [0.28, 0.52, 0.79]

Application examples

Temporal Graph Neural Networks (TGNNs) have attracted attention for tasks involving complex data with temporal and structural dependencies. Specific application examples are given below.

1. traffic flow forecasting

• Problem: forecasting traffic flows and congestion in an urban road network.
• Data:
  – Nodes: individual roads and intersections.
  – Edges: connections between roads.
  – Node features: number of vehicles on the road, average speed, weather information, etc.
  – Time series data: historical traffic flows.
• Role of the model: learn temporal traffic flow patterns within the road network using TGNN.
• Outcomes: improved accuracy of traffic congestion forecasts. Optimisation of public transport operation schedules.
• References: Spatio-Temporal Graph Convolutional Networks for Traffic Forecasting (STGCN, Yu et al. 2018)

2. social network behaviour forecasting

• Problem: Predicting the behaviour of social media users (e.g. when and what they will post next).
• Data:
  – Nodes: user accounts.
  – Edges: follow relationships and messages sent and received between users.
  – Node features: posting frequency, past behaviour history.
  – Time series data: changes in activity.
• Role of the model: learn temporal behaviour patterns of users and predict their next activity using TGNN.
• Outcome: improved accuracy of targeted advertising. Improved efficiency of feed recommendation system.

3. financial market forecasting

• Problem: to predict price fluctuations in the stock and cryptocurrency markets.
• Data:
  – Nodes: companies and assets.
  – Edges: correlations between assets (price correlations and trading pairs).
  – Node features: time-series data such as stock prices, trading volumes, news, etc.
• Role of the model: to capture the overall market structure and changes over time using TGNN and to predict price volatility.
• Outcomes: improved high frequency trading (HFT) and risk management.

4. medical data analysis

• Problem prediction of patient health status and disease progression.
• Data:
  – Nodes: patients.
  – Edges: similarities between patients (commonalities in disease and treatment history).
  – Node features: diagnostic data, treatment history and genetic data for each patient.
  – Time series data: changes in health indicators (e.g. blood pressure, heart rate).
• Role of the model: predicts the temporal progression of a patient’s health status using TGNN, enabling early intervention.
• Outcomes: optimisation of patient care. Efficient allocation of healthcare resources.

5. cyber security

• Problem: attack detection and anomaly detection within the network.
• Data:
  – Nodes: servers and devices.
  – Edge: communication between devices.
  – Node features: activity logs for each device.
  – Time series data: changes in communication traffic.
• Role of the model: learn temporal and structural anomaly patterns using TGNNs to detect attacks in real-time.
• Outcome: improved accuracy of anomaly detection. Early response to cyber-attacks.

6. energy consumption prediction

• Problem: predicting and optimising energy consumption in smart grids described in “Electricity storage technology, smart grids and GNNs“.
• Data:
  – Nodes: power plants, households, businesses, etc.
  – Edge: energy supply pathways between power plants and consumers.
  – Node features: energy consumption, weather conditions.
  – Time series data: historical energy use.
• Role of the model: learn consumption patterns at each node using TGNN to improve demand forecasting.
• Outcome: improved efficiency of electricity supply. Optimisation of the use of renewable energy.

The following points need to be taken into account when applying the model

• Data preparation: collection and pre-processing of graphical data, including temporal variations, is important.
• Model selection: choose a TGNN model (e.g. TGCN, EvolveGCN) suitable for the specific task.
• Computational resources: utilise GPUs as graph training involving time-series data is computationally expensive.

reference book

The following section describes reference books and papers on Temporal Graph Neural Networks (TGNNs) and the analysis of dynamic graph data.

1. basic reference books
Graph Neural Networks in general
– ‘Graph Representation Learning’.
– Author: William L. Hamilton
– Year of publication: 2020
– Abstract: Systematically explains the fundamentals of representation learning for graph data and is suitable for an introduction to dynamic graph models with temporal information.

– ‘Deep Learning on Graphs’.
– Author(s): Yao Ma, Jiliang Tang
– Year of publication: 2021
– Abstract: Covers the theory and applications of graph neural networks (GNNs). Methods for analysing dynamic and time series graphs are included.

2. application-focused book.
Spatio-Temporal Graphs.
– ‘Spatio-Temporal Graph Neural Networks for Traffic Forecasting’.
– Author(s): Xiaolei Ma et al.
– Year of publication: 2021
– Abstract: Describes the theory and implementation of GNNs for spatio-temporal data, with particular practical applications in traffic flow forecasting.

– “A survey of dynamic graph neural networks”

3. main publications
Fundamental models for dynamic graphs.
1. ‘EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs’.
– Author(s): Aldo Pareja et al.
– Year of publication: 2020
– Abstract: Proposes an Evolutionary GCN (EvolveGCN) model dedicated to the analysis of time series graphs.

2. ‘Continuous-Time Dynamic Graph Learning via Neural Interaction Processes’.
– Author(s): Rakshit Trivedi et al.
– Year of publication: 2019
– Abstract: Proposal of a model for learning Continuous-Time Dynamic Graphs.

3. ‘Temporal Graph Networks for Deep Learning on Dynamic Graphs’.
– Author(s): Emanuele Rossi et al.
– Year of publication: 2020
– Abstract: Proposal for Temporal Graph Networks (TGNs). Leverages temporal embedding and memory modules.

Spatio-Temporal Analysis and Applications.
4. ‘Spatio-Temporal Graph Convolutional Networks for Traffic Forecasting’
– Author(s): Yu et al.
– Year of publication: 2018
– Abstract: Graph convolutional networks (STGCNs) in spatio-temporal data and their application to traffic flow forecasting.

4. implementation resources
– PyTorch Geometric Temporal
– Abstract: A PyTorch extension library for easy implementation of dynamic graph neural networks.

– DGL (Deep Graph Library)
– Abstract: A general-purpose library suitable for implementing graph neural networks. Also provides modules for handling dynamic graphs. 5.

5. in relation to contrast learning
– ‘A Simple Framework for Contrastive Learning of Visual Representations’.
– Author(s): Ting Chen et al.
– Year of publication: 2020
– Abstract: Learn the basic concepts of contrastive learning and understand the connection with learning dynamic graphs.

6. teaching materials and online resources
– Coursera

– YouTube:.
– Channel: *Stanford CS224W – Machine Learning with Graphs*.
– Description: Official Stanford University lecture series detailing the fundamentals and applications of GNNs.