Data anonymisation technology

Information confidentiality technology

Data Anonymisation technology (Data Anonymisation) is an approach used to protect sensitive information such as personal and confidential data, making it a widely used technique for data security and privacy protection. The following describes the main aspects and methods of information anonymisation techniques.

1. anonymisation: Anonymisation is the process of transforming personal information so that it cannot be used to identify individuals, including removing direct identifiers (e.g. names, social security numbers), generalising data (e.g. grouping ages by decade), replacing data (e.g. assigning false identifiers) 2. data masking (e.g. data masking)

2. data masking: data masking is a technique for hiding parts of sensitive information (e.g. credit card numbers, social security numbers), and a common approach would be to replace some characters with foreshadowing or random characters. This makes it possible to hide personal information when the data is displayed.

3. pseudonymisation: pseudonymisation is the process of replacing data with false identifiers or pseudo-values that make the real data very difficult to understand while maintaining the same format as the real data. Pseudo-simulation can be used for data analysis and testing while improving data security.

4. data splitting: data splitting is a method of splitting data into multiple locations and managing the individual parts independently. Each part does not contain personal data in isolation, so that if an individual part is compromised, the overall data is not compromised.

5. Noise Injection: Noise injection is a method of introducing random noise into the data, which makes the data appear inaccurate and makes it difficult to identify individual data points. If the process of adding noise is clear, it can continue to retain useful information in data analysis.

6. differential privacy: differential privacy is a mathematical framework to prevent excessive leakage of query responses to data sets.

Algorithms used in information hiding technology

Various algorithms and methods are used in information hiding techniques. These algorithms are designed to transform data so that individuals cannot be identified. These algorithms and methods are described below.

1. Generalisation: generalisation is a technique that replaces specific attributes of data with more general categories, e.g. generalising age into age groups. This technique is a common way of anonymising data.

2. de-identification: de-identification is the process of transforming data so that individual persons cannot be identified, which involves removing or replacing parts of the data. The specific algorithm depends on the type of data and security requirements.

3. data masking: data masking involves replacing sensitive information (e.g. credit card numbers, social security numbers) with a random string of characters or foreshadowing, which is actualised by replacing the sensitive information with a random string of characters. This process is used to conceal parts of the data and protects privacy without restricting data analysis.

4. pseudonymisation: pseudonymisation is a method of replacing real identifiers with fake identifiers, which can be used to protect personal data while preserving the format of the data. Pseudonymisation tables are used to restore information by managing the mapping between identifiers and pseudo-identifiers.

5. differential privacy: differential privacy is a technique where noise is added to the data to prevent individual data points from being tracked, this approach can ensure the privacy of query responses from the database.

6. k-Anonymity: k-Anonymity is a technique that ensures that each individual in a data set has the same attributes as at least k other individuals, which makes it impossible to identify a particular individual.

7. l-Diversity: l-Diversity is a method that ensures that each attribute value in a dataset is held by at least l different individuals, thereby rendering an individual’s attribute values unidentifiable. k-Anonymity is a grouping of quasi-identifiers, whereas The difference is that l-multi-similarity reduces the risk of identification by attribute values and background knowledge.

Examples of information hiding technology implementations

There are various methods of information hiding techniques, which are applied according to the actual system. Examples of implementations of the main information hiding techniques are described below.

1 K-Anonymity

Example implementation:.

• Purpose: to group data in such a way that individuals cannot be identified.
• Example: In a medical database, a patient’s age and postcode are generalised and changed as follows.
  • Original data: 35 years, 12345
  • After anonymisation: 30-40 years, 123XX

Technique used:

• Generalisation of the data: created a range, e.g. age = [20-30].
• Suppression of data: remove specific attributes (e.g. remove data from minority groups).

Example program (Python):

from pandas import DataFrame
import pandas_anonymizer

data = DataFrame({
    "age": [25, 34, 45, 29],
    "zip": [12345, 12345, 67890, 67890]
})

anonymized_data = pandas_anonymizer.generalize(data, {"age": [(20, 30), (30, 40)], "zip": [123XX]})
print(anonymized_data)

2. differential privacy

Example implementation:

• Objective: to reduce the impact of individual data on results and to make them less susceptible to external re-identification.
• Example: to add noise when publishing statistics from sensitive data sets.
  • Original mean: 52.3
  • Mean value after noise: 52.7

Technique used:

• Laplace or Gaussian noise added.
• Used in practice in Apple and Google’s Data on Human Movement.

Example program (Python):

import numpy as np

def add_noise(data, epsilon):
    noise = np.random.laplace(0, 1/epsilon, size=len(data))
    return data + noise

data = [10, 20, 30]
epsilon = 0.5
noisy_data = add_noise(data, epsilon)
print(noisy_data)

3. Pseudonymisation

Example implementation:

• Purpose: to replace real personal identifiers with pseudo-identifiers.
• Example
  • Original data: Name: Taro Yamada, ID: 123456
  • After pseudonymisation: Name: *****, ID: AB123

Technique used: 

• Hashing (e.g. SHA-256).
• Tokenisation.

Example program (Python):

import hashlib

def pseudonymize(data):
    return hashlib.sha256(data.encode()).hexdigest()

name = "Taro Yamada"
pseudonymized_name = pseudonymize(name)
print(pseudonymized_name)

4. data masking

Example implementation:

• Objective: to hide important information but preserve the overall shape of the data.
• Example.
  • Original data: credit card number: 1234-5678-9101-1121
  • After masking: Credit card number: 1234-****-****-1121

Technique used:.

• Partial data replacement.
• Field-level control.

Example program (Python):

def mask_credit_card(card_number):
    return card_number[:4] + "-****-****-" + card_number[-4:]

card_number = "1234-5678-9101-1121"
masked_card = mask_credit_card(card_number)
print(masked_card)

5. anonymised data generation

Example implementation:

• Objective: to generate completely random data to ensure privacy.
• Example:.
  • Artificial generation and use of patient information from original data.

Technique used:

• Synthetic Data Generation.
• Use of GANs (Generative Adversarial Networks).

Example program (Python):

from faker import Faker

fake = Faker()
for _ in range(5):
    print(fake.name(), fake.address(), fake.email())

6. access control and encryption

Example implementation:

• Objective: to restrict who can access the data itself.
• Examples:
  • AES-encrypted storage of medical data.

Technology used:

• AES (Advanced Encryption Standard).
• Role-based access control (RBAC).

Example program (Python):

from cryptography.fernet import Fernet

# Key generation
key = Fernet.generate_key()
cipher_suite = Fernet(key)

# Data encryption
data = "Medical data of Taro Yamada."
encrypted_data = cipher_suite.encrypt(data.encode())
print(encrypted_data)

# Data decryption
decrypted_data = cipher_suite.decrypt(encrypted_data).decode()
print(decrypted_data)

Key points of choice

• Legal requirements: if GDPR or HIPAA compliance is required, K-anonymisation or differential privacy is common.
• Data use purpose: differential privacy or pseudo-anonymisation if data analysis is required.
• Implementation costs: data masking and pseudo-anonymisation are relatively easy to implement.

Challenges and measures for information secrecy technology

Information confidentiality technology has many advantages, but there are also challenges when it comes to its actual use. Typical challenges and their countermeasures are described below.

1. risk of re-identification
Challenge: Even anonymised data can be re-identified by combining it with external information. For example, even if K-anonymised data is made publicly available, there is a risk of being identified when matched with other datasets that are publicly available.

Solution:
– Ensure Diversity:
Introduce L-Diversity (L-Diversity) to ensure that the values of sensitive attributes within the same group are diverse. Examples include adjustments to include more than one occupation or medical history within the same age group.
– Applying risk assessment models: utilising tools (ARX and sdcMicro) to quantitatively assess the risk of re-identification, and ensuring appropriate anonymisation.

2. reducing the usefulness of the data
Challenges: the loss of data detail through anonymisation and confidentiality reduces the accuracy of the analysis and the usefulness of the results. For example, excessive generalisation leads to a coarsening of the granularity of the data, resulting in a loss of detail necessary for the analysis.

Solution:
– Minimal Anonymisation: data are anonymised to the minimum extent necessary and noise is added to the data in such a way that it does not affect the analysis. In differential privacy, the choice of appropriate ε (privacy parameter) is important.
– Use of data synthesis: synthetic data generation tools (e.g. SDV, CTGAN) are used to create data that mimic the characteristics of the original data.

3. regulatory compliance
Challenge: regulations such as GDPR and CCPA require data anonymisation and confidentiality, but implementation to meet these regulations can be difficult.

Solution:
– Use of regulatory frameworks: utilise tools specialising in data anonymisation (e.g. Aircloak, Privitar).
– Establish an audit regime: have the anonymised data audited by legal experts or third-party organisations.

4. technical challenges of implementation
Challenge: Anonymisation and secrecy techniques require a high degree of expertise and can be complex to implement. The performance of anonymisation techniques can also affect the speed and efficiency of the overall system.

Solution:
– Use of libraries and tools: use libraries that do not require specialist knowledge (e.g. ARX, Anonymizer) to increase efficiency.
– Use of parallel processing and cloud technologies: for large data sets, deploy cloud platforms and distributed processing frameworks to improve performance.

5. data linkage difficulties
Challenge: as anonymised data cannot be directly linked to original personal data, it can be difficult to link and integrate between different data sets.

Solution:
– Pseudonymisation: assign a unique identifier (e.g. hash value) to the data so that it can be linked as required. Examples include the use of hash functions or tokenisation techniques.
– Partial decryption of data for linking: instead of fully encrypting the data, some metadata is left behind to facilitate linking.

6. distortion by noise addition
Challenge:  In methods that add noise, such as differential privacy, excessive noise can cause significant loss of data accuracy.

Solution:
– Noise optimisation: optimise the balance between confidentiality and data accuracy by adjusting the privacy parameter (ε).
– Control the distribution of noise: understand the properties of Laplace and Gaussian distributions and choose the distribution that best suits the data usage scenario.

7. inverse analysis of concealment methods
Challenge: an attacker could reverse-analysis the concealment method and recover hidden information.

Solution:
– Combining several concealment techniques: combining K-anonymisation, L-diversity and differential privacy to enhance attack resilience.
– Continuous updating: continually update concealment techniques and algorithms in order to respond to new attack methods.

Reference information and reference books

Describes reference books on information confidentiality technology and data privacy.

Basic theory and practice.
1. 《Privacy-Preserving Data Publishing: Concepts and Techniques》
Authors: Benjamin C. M. Fung, Ke Wang, Ada Wai-Chee Fu, Philip S. Yu
Abstract: This book systematically explains the theory and implementation of information anonymisation techniques (K-anonymisation, L-diversity, T-approximation, etc.). It also details challenges and applications in anonymisation.

2. 《Data Privacy: Foundations, New Developments and the Big Data Challenge》
Authors: Tania F. Peña, Josep Domingo-Ferrer
Description: This book covers the fundamentals of privacy protection technologies and the challenges in large data environments. Includes differential privacy and cryptography.

Applied technologies and latest topics.
3. 《Big Data Privacy and Security in Smart Cities》

4. 《Privacy and Data Protection Seals》
Author(s): Lee A. Bygrave, Luca Tosoni
Abstract: A practical guide to data privacy protection under regulations such as GDPR. Provides a detailed description of the regulatory compliance framework.

5. 《Security and Privacy in Machine Learning》
Authors: Emiliano De Cristofaro, Clément Canonne
Abstract: Fundamentals and applications of privacy issues and defence techniques (federated learning, differential privacy) in machine learning.

Papers and online resources.
6. 《The Algorithmic Foundations of Differential Privacy》
Authors: Cynthia Dwork, Aaron Roth
Abstract: A collection of papers that delve deeply into the theoretical background of differential privacy. They may be made available online free of charge.

7. ARX – Data Anonymization Tool Documentation
Abstract: Official documentation of the open-source data anonymisation tool ARX. It contains a wealth of concrete implementation methods and examples.