Tensor Networks and their Applications

What is a Tensor Network?

A Tensor Network is a powerful numerical and conceptual tool widely used in fields such as quantum physics, quantum information, machine learning, and statistical mechanics. It provides a way to represent and compute with complex high-dimensional data—such as many-body quantum states, probability distributions, or deep neural networks—by decomposing them into a network of low-rank tensors.

What is a Tensor?

A tensor is a generalization of a multi-dimensional array and is used to hierarchically represent data or physical quantities.

• A scalar is a 0th-order tensor (a single number).
• A vector (1D array) is a 1st-order tensor.
• A matrix (2D array) is a 2nd-order tensor.
• Higher-order tensors (3rd order and above) are used to represent more complex data structures, such as:
  • Color images (height × width × RGB),
  • Quantum states,
  • Multivariate time-series data.

What is a Tensor Network?

A tensor network is a method for decomposing large, high-order tensors into a network (graph) of smaller tensors connected by shared indices. This makes it possible to efficiently represent and compute extremely large data structures, such as many-body quantum states, whose size grows exponentially.

One key feature of tensor networks is their ability to exploit local interactions and dependencies to compress information. This reduces the need to handle the full tensor explicitly, leading to significant savings in computational and memory costs. Through tensor contraction, inner products, and trace operations, physical quantities and probabilistic measures can be computed efficiently.

Moreover, tensor networks are deeply connected to quantum entanglement structures and are considered essential tools in quantum physics and quantum information theory.

Representative Tensor Network Structures

• MPS (Matrix Product States):
  Linear chain structure of tensors, commonly used to represent 1D quantum systems efficiently. Also applied in natural language processing.
• PEPS (Projected Entangled Pair States):
  Extension of MPS to 2D grids, suitable for representing 2D quantum spin systems and high-dimensional states.
• MPO (Matrix Product Operator):
  The operator version of MPS, used to express Hamiltonians and time-evolution operators efficiently.
• TT (Tensor Train):
  A structure similar to MPS, widely used for high-dimensional tensor approximation and model compression in machine learning.
• Tree Tensor Networks:
  A tree-shaped connection of tensors, ideal for representing hierarchical physical systems or deep models.
• MERA (Multiscale Entanglement Renormalization Ansatz):
  A multiscale hierarchical structure suitable for capturing complex entanglement near quantum critical points.

These structures allow the tensor network to be optimized for specific physical systems or data types, enabling a balance between expressive power and computational efficiency.

Applications Across Domains

Physics

Tensor networks originated in many-body quantum physics and remain central in quantum simulations. They achieve high-precision approximations of quantum states in 1D systems using feasible computational resources. Applications include:

• Ground state search of quantum spin systems,
• Analysis of quantum entanglement structures,
• Theoretical research in quantum computing and quantum information.

Machine Learning

In recent years, tensor networks have gained attention for their compactness and structured representation, especially in deep learning where both input data and model parameters are high-dimensional.

• Techniques like Tensor Train Layers (TT-Layers) express fully connected layers as tensor networks.
• Weight matrices in RNNs or CNNs can be compressed using tensor decompositions.
• These methods reduce model size without compromising accuracy and enable deployment on memory-constrained edge devices or mobile environments.

Other Fields

Beyond physics and AI, tensor networks are also used in:

• Statistical mechanics (e.g., Ising and Potts models) to compute partition and correlation functions efficiently.
• Data compression and signal processing, using tensor decomposition for dimensionality reduction and denoising.
• Image processing, for compact representation and manipulation of complex visual data.

Summary

Backed by physical intuition and mathematical rigor, tensor networks offer both high representational power and computational efficiency. They have become a powerful framework across diverse fields, bridging theory and real-world applications in quantum systems, AI, and complex data modeling.

Specific Application Examples

1. DMRG (Density Matrix Renormalization Group)

Primarily used in physics, but increasingly applied to quantum chemistry and materials design.

Representative Paper:
Chan & Sharma, 2011 – “Matrix product states and DMRG for quantum chemistry”

2. TEBD (Time Evolving Block Decimation)

Used for simulating time evolution (quantum dynamics, thermalization, information propagation).

Representative Paper:
Vidal, 2003 – “Time-evolving block decimation for numerical simulation of 1D quantum systems”

3. TT Decomposition (Tensor Train Decomposition)

Ideal for compressing and manipulating high-dimensional tensors; widely used in AI.

Representative Papers:

• Novikov et al., NeurIPS 2015 – “Tensorizing Neural Networks”
• Oseledets, 2010 – “TT-cross: A black-box cross approximation algorithm for very high-dimensional tensors”

4. MPS Classifier (Matrix Product State Classifier)

A quantum-inspired method for image and text classification.

Representative Paper:
Stoudenmire & Schwab, NeurIPS 2016 – “Supervised Learning with Tensor Networks”

5. CP / Tucker Decomposition

Widely used in medical imaging, video analysis, and recommender systems.

Representative Paper:
Kolda & Bader, 2009 – “Tensor Decompositions and Applications”

Implementation example

Task: Classify MNIST images using an MPS-based classifier

Each image (28×28) is vectorized into 784 elements. The MPS network contracts across the sequence of pixels:

• Each pixel is encoded as a 1D local tensor (binary state).
• The MPS is contracted from left to right to produce class probabilities.

Prerequisites

pip install torch torchvision matplotlib

Implementation (Simplified Version)

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# --- 1. Load MNIST Data ---
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])
train_loader = DataLoader(datasets.MNIST('.', train=True, download=True, transform=transform), batch_size=64)
test_loader = DataLoader(datasets.MNIST('.', train=False, transform=transform), batch_size=1000)

# --- 2. Define MPS Layer ---
class MPSLayer(nn.Module):
    def __init__(self, input_dim, bond_dim, output_dim):
        super().__init__()
        self.input_dim = input_dim
        self.bond_dim = bond_dim
        self.output_dim = output_dim

        self.mps = nn.ParameterList([
            nn.Parameter(torch.randn(bond_dim, 2, bond_dim)) for _ in range(input_dim)
        ])
        self.first_tensor = nn.Parameter(torch.randn(1, 2, bond_dim))
        self.last_tensor = nn.Parameter(torch.randn(bond_dim, 2, output_dim))

    def forward(self, x):
        B = x.shape[0]
        x = (x > 0).float().long()  # Binarize pixels: 0 or 1
        result = []
        for b in range(B):
            psi = self.first_tensor[:, x[b, 0], :]  # [1, bond_dim]
            for i in range(1, self.input_dim - 1):
                psi = torch.einsum('ab,bcd->acd', psi, self.mps[i])
                psi = psi[:, x[b, i], :]
            out = torch.einsum('ab,bcd->acd', psi, self.last_tensor)
            out = out[:, x[b, -1], :]  # [1, num_classes]
            result.append(out.squeeze(0))
        return torch.stack(result)

# --- 3. Define MPS Classifier ---
class MPSClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.mps = MPSLayer(784, bond_dim=10, output_dim=10)

    def forward(self, x):
        x = self.flatten(x)
        return self.mps(x)

model = MPSClassifier()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# --- 4. Training Loop ---
for epoch in range(3):
    model.train()
    total_loss = 0
    for data, target in train_loader:
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    print(f"Epoch {epoch+1}, Loss: {total_loss:.4f}")

# --- 5. Evaluation ---
model.eval()
correct = 0
with torch.no_grad():
    for data, target in test_loader:
        output = model(data)
        pred = output.argmax(dim=1)
        correct += (pred == target).sum().item()

print(f"Test Accuracy: {correct / len(test_loader.dataset) * 100:.2f}%")

Features and Notes

Applications

• Quantum-inspired image classification
• Deployment in resource-constrained environments (edge AI)
• Introductory hands-on example for supervised learning using tensor networks

Books, papers, and resources on Tensor Networks

General & Theoretical Introductions

1. A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States

• Simons Collaboration on the Many-Electron Problem
• Content: A definitive and well-illustrated introduction to MPS, PEPS, and DMRG; suitable for beginners with mathematical background.

Algorithms and Numerical Methods

2. An Efficient Matrix Product Operator Representation of the Quantum-Chemical Hamiltonian

• Content: Comprehensive guide to applying DMRG in quantum chemistry; includes theory, practical implementations, and case studies.
• Audience: Researchers working on quantum systems or tensor-based simulations.

3. Tensor Spaces and Numerical Tensor Calculus

• Author: Wolfgang Hackbusch
• Content: Covers TT (Tensor Train), HT (Hierarchical Tucker), and other decompositions from a numerical analysis perspective.
• Audience: Applied mathematicians, engineers, and computational scientists.

Tensor Networks in AI & Machine Learning

4. Tensor networks in machine learning

• Content: Explains how tensor networks are applied to deep learning, model compression, generative models, and classification tasks.

Recommended Papers:

• “Supervised Learning with Quantum-Inspired Tensor Networks” – E. M. Stoudenmire & D. J. Schwab
• “Tensorizing Neural Networks” – A. Novikov et al., NeurIPS 2015

5. Tensor Decompositions for Data Science

• Authors: Tamara G. Kolda & Brett W. Bader
• Content: A comprehensive survey of CP, Tucker, TT decompositions and their applications across science and engineering.

Hands-On & Implementation-Focused Books

6. Hands-On Quantum Machine Learning With Python

• Author: Frank Zickert
• Content: Learn quantum machine learning algorithms, including tensor networks, by implementing them in Python.
• Tools Covered: PennyLane, TensorNetwork, Qiskit

Online Resources & Tutorials

Additional Recommended Books

• Applications of Tensor Analysis
• An Introduction to Tensor Networks and Matrix Product States“
• Numerical Methods for Quantum Many-Body Systems

Key Software Libraries