Combining AI technology and the metaverse

The metaverse and how it can be used

A metaverse is a virtual space or virtual world built on the internet. Here, users can interact with other people through avatars and engage in 3D spaces. Technologically, technologies such as virtual reality (VR), augmented reality (AR) and blockchain are often utilised.

Metaverse is characterised by an ‘immersive 3D space’ where users can experience ‘entering’ the virtual space in a form similar to reality, ‘real-time interaction’ where users can communicate, collaborate and play games with other users and AI, and a virtual economy (virtual currency and NFT) where ecosystem’ where transactions and unique community cultures can develop, and “persistence” where the virtual space is always online and continues to operate even when users log out.

Key uses of the metaverse include

• Entertainment
  • Gaming: providing a play experience in virtual space, such as Fortnite or Roblox.
  • Live events: artist concerts and film screenings held in virtual space.
  • E.g. Travis Scott’s Fortnite Live.
• Business and working
  • Remote working: meetings and collaboration in virtual meeting rooms.
  • Example: Meta’s Horizon Workrooms.
  • Virtual offices: building an office in a 3D space, incidental communication with colleagues.
  • Presentations and demos: share 3D models of products in real time.
• Education and training
  • Virtual classrooms: online classes where students can participate through avatars.
  • Simulation learning: advanced training, e.g. medical or aviation, can be conducted safely in virtual space.
• Shopping and fashion
  • Virtual store: try on and buy products in virtual space.
  • Example: Gucci sells exclusive items in Roblox.
  • Digital fashion: own outfits and items as an NFT in virtual space.
• Socialising and entertainment
  • Making friends in virtual space: avatars can socialise and share hobbies.
  • Virtual events: weddings and birthday parties can be held in virtual space.
• Medicine and rehabilitation
  • Rehabilitation: movement training in virtual space.
  • Mental health care: stress reduction and treatment programmes using VR.

• Real estate and architecture
  • Virtual modelling: 3D previews and virtual tours of building projects.
  • Land sales: buy and sell virtual land in the metaverse (e.g. Decentraland, Sandbox).
• Tourism and culture
  • Virtual travel: recreate world landmarks and sites of the past in 3D space.
  • Digital cultural experiences: experience art exhibitions and historical events.
• Sport and fitness
  • VR fitness: games with an exercise component, such as Beat Saber.
  • Virtual sports: experience a match in a virtual space.
• Smart cities and urban planning
  • Virtual city simulations: consider and simulate urban planning.
  • Citizen services: provision of administrative services through the metaverse.

Specific examples include.

• Meta (formerly Facebook): provides virtual spaces in Horizon Worlds for work and entertainment.
• Decentraland: uses blockchain technology to buy and sell virtual land and organise events.
• Nike: extends its brand by offering virtual experiences with NIKELAND (in Roblox).
• Education: medical surgical training and exploration of historical buildings using VR.

How to create a metaverse

Creating a metaverse (virtual space) can be done using a combination of several techniques and tools. The basic steps are outlined below.

1. conceptual design: before creating a metaverse, the following points should be clarified

• Purpose: e.g. games, education, business exchange, art exhibition, etc.
• Target users: who will use it?
• Features and functions: avatars, chat, voice calls, economic systems (virtual currency or NFT), etc.

2. technology selection: select the key technologies to be used to build the metaverse.

Platforms.

• Unity: high-performance 3D game engine. Requires visual scripting and programming.
• Unreal Engine: engine with excellent graphical realism.
• Three.js: 3D graphics library that runs in a web browser.

Development languages

• Unity: C#
• Unreal Engine: C++ or Blueprint
• For the web: JavaScript (Three.js, Babylon.js)

Server/network technologies

• Real-time communication: WebSockets, Photon Engine, Mirror, etc.
• Cloud: server operations on AWS, Google Cloud, Azure.

Storage.

• Databases (user information and asset management): MongoDB, Firebase, MySQL, etc.
• 3D asset storage: Google Cloud Storage and AWS S3.

3. designing avatars and 3D environments

Avatars.

• Avatar creation tools: Ready Player Me and Blender.
• Animation: animations can be added using Mixamo.

3D environments

• Modelling: Blender, Maya and Cinema 4D.
• Assets: materials can be obtained from the Unity Asset Store or Sketchfab. 4.

4 User interface and experience design

• UI/UX design: design of menus and interactions.
• Interaction design: how users move around and interact with others in the environment.

5. implementation

Basic functionality

• 3D space display: virtual space in which users can move around.
• Real-time communication: conversation and cooperation with other users.
• Audio: position-based audio representation with Spatial Audio.
• Economic system: trading of items using virtual currency and NFT.

Example code (basic 3D display using Three.js)

import * as THREE from 'three';

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);

camera.position.z = 5;

function animate() {
  requestAnimationFrame(animate);
  cube.rotation.x += 0.01;
  cube.rotation.y += 0.01;
  renderer.render(scene, camera);
}
animate();

6. testing and deployment

• Testing: testing for user experience and bug fixes.
• Hosting: make the metaverse available online (e.g. Netlify, Vercel, Firebase Hosting).
• Mobile/VR support: optimisation for VR devices, e.g. Oculus Quest.

7. continuous updates.

As the metaverse needs to evolve, the following should be done on an ongoing basis

• Add functionality (e.g. new items and areas)
• Reflecting user feedback
• Performance optimisation

Combining AI technology with the metaverse to increase value

Combining AI technologies with the metaverse can significantly improve the convenience and user experience of the virtual space. The following are specific ideas for increasing the value of the metaverse through the use of AI technologies.

1. personalisation of the user experience

• Automatic avatar generation: realistic 3D avatars are generated by AI by analysing the user’s photo and movement data. Examples: customisation of voice and appearance by AI.
• Content recommendations: AI analyses the user’s behaviour history and tastes and suggests events, products and activities that match their interests.

2. interactive NPCs (non-player characters)

• Equipped with AI chatbots: NPCs are equipped with AI to enable real-time natural conversations. Examples: characters acting as tourist guides or educational assistants.
• Autonomous behaviour of AI characters: using machine learning, NPCs react autonomously based on the environment and user behaviour, providing a more dynamic experience.

3. space generation and optimisation

• Automatic generation of virtual spaces by AI: using Procedural Generation technology to generate custom environments in real-time according to user requirements. Examples: easily create conference rooms, shopping malls, educational facilities, etc.
• Dynamic space adjustment: AI analyses human flow and usage to avoid congestion and optimise space layout.

4. natural interfaces.

• Speech recognition and natural language processing (NLP): understands the user’s voice and text in real time and performs actions. Example: changes 3D objects in response to commands such as ‘make this building red’.
• Gesture Recognition: the AI recognises the user’s hand and body movements for interaction. Example: enabling actions such as touching or lifting objects.

5. AI-enabled virtual economy

• AI trading and economic analysis: analyses transaction data in the virtual space to maintain the health of the economy. Examples: forecasting trends in NFT and virtual currency markets.
• Price optimisation: AI analyses supply and demand and dynamically adjusts prices of virtual items and land.

6. advanced learning and education

• AI teaching assistants: AI teachers that dynamically adjust the curriculum according to the user’s progress. Examples: virtual classroom question answering and assignment feedback.
• Real-time translation: using natural language processing to support smooth communication between users speaking different languages.

7. improved security in the metaverse.

• AI fraud detection: detects and suppresses fraud and inappropriate behaviour in virtual space in real time.
• Protection of digital identities: AI monitors avatars and accounts for unusual activity, enhancing security.

8. environmental simulation and research

• Virtual simulation by AI: reproduce natural phenomena and urban planning in virtual space and apply them to solve real-world problems. Examples: smart city design and disaster evacuation simulations.

9. user matching

• Social matching with AI: analyses users’ interests and behaviour patterns and suggests suitable partners and groups. Examples: networking at virtual events and communities.

10. sustainable development

• Optimising energy efficiency with AI: optimise server load and energy consumption in real-time to improve sustainability.
• Build environmental models: manage ecosystems in virtual space with AI and simulate real-life sustainable technologies.

implementation example

Below are concrete examples of implementing AI technologies in the metaverse, using platforms such as JavaScript, Python or Unity.

1. an example implementation of a natural conversation with an NPC: an NPC interacts with a user using natural language processing (NLP), using OpenAI’s API to build an NPC that answers questions in the virtual space

Technology stack.

• Three.js (3D space)
• OpenAI API (NLP)

Example implementation: natural language response of an NPC

// Displaying characters (NPCs) in Three.js
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// Load NPC models.
const loader = new THREE.GLTFLoader();
loader.load('npc_model.gltf', (gltf) => {
    scene.add(gltf.scene);
});

camera.position.z = 5;

// OpenAI API to process conversations.
async function talkToNPC(input) {
    const response = await fetch('https://api.openai.com/v1/chat/completions', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer YOUR_API_KEY`,
        },
        body: JSON.stringify({
            model: "gpt-4",
            messages: [{ role: "user", content: input }],
        }),
    });
    const data = await response.json();
    return data.choices[0].message.content;
}

// Responding to user input
document.getElementById('chatInput').addEventListener('keydown', async (event) => {
    if (event.key === 'Enter') {
        const input = event.target.value;
        const reply = await talkToNPC(input);
        console.log("NPC:", reply); // Show NPC response.
        event.target.value = '';
    }
});

// rendering loop
function animate() {
    requestAnimationFrame(animate);
    renderer.render(scene, camera);
}
animate();

2. real-time customisable virtual space generation with AI: an example of dynamically generating the structure of a virtual space, using AI to create spaces (buildings, rooms, etc.) according to the user’s preferences.

Technology stack.

• Python (AI models)
• Three.js (3D display)

Example implementation: space generation

Python: generating layouts with AI

import openai

def generate_room_description(preferences):
    response = openai.Completion.create(
        model="text-davinci-003",
        prompt=f"Suggest room layouts based on user preferences.: {preferences}",
        max_tokens=100
    )
    return response.choices[0].text.strip()

user_preferences = "Modern design furniture with emphasis on wide windows and natural light."
layout_description = generate_room_description(user_preferences)
print("Generated rooms:", layout_description)

JavaScript: display in Three.js

Convert Python-generated layouts to JSON format and render them in JavaScript.

const layout = {
    "windows": 2,
    "furniture": ["sofa", "table", "lamp"],
    "style": "modern"
};

// Generate space based on layout.
layout.furniture.forEach(item => {
    const geometry = new THREE.BoxGeometry(1, 1, 1);
    const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
    const mesh = new THREE.Mesh(geometry, material);
    scene.add(mesh);
});

3. interaction through gesture recognition: an example of using AI to recognise the user’s hand and body movements and perform actions in a virtual space.

Technology stack.

• TensorFlow.js (recognition of hand movements)
• Three.js (reflection of movements)

Implementation example: moving objects with gestures

import * as handpose from '@tensorflow-models/handpose';
import '@tensorflow/tfjs-backend-webgl';

// Get hand movements with TensorFlow.
const model = await handpose.load();
const video = document.createElement('video');
navigator.mediaDevices.getUserMedia({ video: true }).then(stream => {
    video.srcObject = stream;
    video.play();
});

async function detectHand() {
    const predictions = await model.estimateHands(video);
    if (predictions.length > 0) {
        const x = predictions[0].landmarks[0][0]; // x-coordinate of the hand
        const y = predictions[0].landmarks[0][1]; // Hand y-coordinate
        // Reflected in objects in 3D space.
        cube.position.set(x / 100, -y / 100, 0);
    }
    requestAnimationFrame(detectHand);
}

// Setting up Three.js
const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);

detectHand();

4. natural language manipulation: an example of manipulating objects in 3D space by analysing natural language commands entered by the user.

Technology stack.

• OpenAI API (NLP analysis)
• Three.js (3D object manipulation)

Example implementation: manipulation in natural language

async function processCommand(command) {
    const response = await fetch('https://api.openai.com/v1/completions', {
        method: 'POST',
        headers: {
            'Authorization': `Bearer YOUR_API_KEY`,
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            model: "text-davinci-003",
            prompt: `Translate this command to Three.js actions: ${command}`,
            max_tokens: 100,
        }),
    });
    const data = await response.json();
    return data.choices[0].text.trim();
}

// working example
const command = "Move this cube to the right.";
const action = await processCommand(command);
eval(action); // Perform Three.js operations

reference book

Reference books on the combination of metaverse and AI technologies are described below.

Metaverse in general
1. ‘The Metaverse: A Critical Introduction’

2. ‘The Metaverse: And How it Will Revolutionise Everything’ by Matthew Ball
– A comprehensive commentary describing the technological foundations, economics and business potential of the metaverse.

AI technology and the metaverse
1. ‘Artificial Intelligence in the Metaverse’ by Nakamoto Kai
– Explains how AI supports the metaverse and its applications such as natural language processing and gesture recognition.

2. ‘AI 2041: Ten Visions for Our Future’ by Kai-Fu Lee & Chen Qiufan
– Presents how the evolution of the metaverse and AI will impact the future in the form of ten stories.

3. ‘Designing Virtual Worlds’ by Richard Bartle
– Classic textbook on how to design metaverse and utilise AI. Also applicable to game design.

3D technology and programming.
1. ‘Three.js Essentials’ by Jos Dirksen
– A practical guide to Three.js for metaverse development.

2. ‘Learning Unity 2021’ by Harrison Ferrone
– A basic to advanced guide to metaverse development with Unity.

3, ‘Mastering Unreal Engine 5’

Natural language processing and gesture recognition
1. ‘Deep Learning for Natural Language Processing’ by Palash Goyal, Sumit Pandey, and Karan Jain
– A guide to building chatbots and natural conversation systems using NLP techniques.

2. ‘Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow’ by Aurélien Géron
– A practical guide to machine learning applied to gesture recognition and computer vision.

3. ‘Gesture Recognition: Principles, Techniques and Applications’ by D.D. Samantaray
– Details AI-based gesture recognition techniques.

Futuristic Orientation and Metaverse Design.
1. ‘Virtual Reality and Augmented Reality’ by M. Claudia tom Dieck and Timothy Jung
– 1. details user experience design in AR/VR technologies and metaverse.

2. ‘Experience on Demand: What Virtual Reality Is, How It Works, and What It Can Do’ by Jeremy Bailenson
– Explores how virtual reality (VR) can be used for education, training and entertainment.

3. 『Web3 Revolution: Blockchain, Cryptocurrency, NFT and Metaverse』