Overview of Unity and its integration with external systems

Machine Learning Artificial Intelligence Natural Language Processing Artificial Intelligence Algorithm Artificial Life and Agent Reasoning Technology Knowledge Information ICT Technology Digital Transformation Automata/State Transitions/Automatic Programming Navigation of this blog

Unity Overview

Unity is an integrated development environment (IDE) for game and application development developed and widely used by Unity Technologies. Unity supports cross-platform development and can be used on a variety of operating systems, including Windows, Mac, and Linux, and can develop games and applications for a variety of platforms, including iOS, Android, PlayStation, Xbox, and Nintendo Switch.

One of Unity’s unique features is its support for the C# scripting language, a general-purpose programming language developed by Microsoft to create applications that run on platforms such as the .NET Framework and .NET Core, allowing developers to create games object behavior and behavior can be controlled by writing in C# scripts.

Unity has an extensive asset store where developers can purchase or download for free assets such as 3D models, textures, and sound effects. Unity also has an advanced graphics and physics engine that allows developers to create realistic game experiences.

Furthermore, Unity has a large developer community and provides a wealth of tutorials, documentation, and forums. In addition to game development, Unity is used in a variety of other areas, including virtual reality (VR) and augmented reality (AR) development, simulation, visualization, and education. Unity is also used in a variety of other areas besides game development, including virtual reality (VR) and augmented reality (AR) development, simulation, visualization and education. This makes it a widely used tool for projects of various sizes, from individual developers to major game studios.

Environment settings for using Unity

To use Unity, follow these steps to set up your environment

Download Unity: Download the Unity Hub from the official website (https://unity.com/). The Unity Hub is a tool for managing Unity projects and switching between versions, Once you have installed Unity Hub, you will be able to download the Unity editor.
Install Unity Hub: Run the Unity Hub installer you downloaded and follow the instructions to install. Once the installation is complete, Unity Hub will launch.
Select Unity Version: Once the Unity Hub is launched, open the “Installs” tab and select the version of Unity you wish to use. Once you have selected a version, you will be able to download the Unity Editor for that version.
Installing the Unity Editor: For the version of Unity you selected, click the “+ ADD” button on the “Installs” tab, select the version of the Unity Editor and start the installation. Once the installation is complete, you will be able to create a new project in the “Projects” tab of the Unity Hub.
Create a new project: Click the “New” button on the “Projects” tab of the Unity Hub to create a new project. Specify a name and save location for the project and launch the Unity editor.

The Unity environment is now set up and you can create a project in the Unity Editor and start developing your game or app. Various settings, such as project settings and platform builds, can also be done within the Unity editor.

Unity’s libraries and features

Unity provides a rich set of libraries and features to support game and application development. Below we describe the main libraries and features that Unity has to offer.

Unity UI: Unity UI is an integrated toolset for creating user interfaces (UIs). It allows you to easily place UI elements such as buttons, text, and images to create interactive UIs.
2D and 3D Graphics Engine: Unity has a powerful graphics engine to create 2D and 3D visual effects. This provides a variety of graphics-related capabilities, including 3D modeling, animation, lighting, and shaders.
Physics Engine: Unity supports physics simulation, which allows users to configure the physical behavior of objects, such as collisions, gravity, and friction. This allows for the creation of realistic physics simulations and game behaviors.
Animation: Unity allows the creation of animations for objects and characters. This allows users to create animation clips and animation events to control the movement, rotation, scaling, etc. of objects.
Sound: Unity supports sound playback and control. This allows you to play audio clips and change volume and pitch. It is also possible to set up 3D sounds so that sounds can be heard based on the position of objects.
Particle System: Unity has a built-in particle system that allows users to create effective particle effects such as flames, smoke, and explosions. These can be used to set particle shape, color, speed, life cycle, etc. to achieve realistic effects.
Database: Unity includes SQLite, a built-in database that can be used to store and load data and persist in-game progress and player data.

In addition to these, Unity also supports a number of asset packages and plug-ins to further extend its functionality, allowing developers to utilize these libraries and features to create a wide variety of games and applications.

Unity Implementation Example

Examples of implementations using Unity will be described.

2D Platformer Game: Unity can be used to create a classic 2D platformer game. This allows you to implement basic game mechanics such as player character movement, jumping, enemy collision detection, and item collection.
3D First Person Shooter: Using Unity, you can create a first person shooter game in a realistic 3D environment. However, action elements and artificial intelligence must be implemented, including control of the player’s perspective, weapon firing, enemy AI, and environment interaction.
Virtual Reality (VR) Experiences: Using Unity, virtual reality experiences can be created, which can be combined with VR headsets to create immersive virtual spaces and environments that can receive VR controller input and provide interactive controls and experiences. The system will be able to
Augmented Reality (AR) Applications: Unity can be used to create augmented reality applications on smartphones and tablets. This allows users to interact with virtual objects in physical space, using cameras to integrate virtual objects with the surrounding real world.
Simulation: Unity is also used to develop simulation software. This is used, for example, to reproduce the physics and behavior of the real world, such as in airplane flight simulators or automobile driving simulators.

In the following sections, we will discuss how Unity can be used to integrate with external systems.

Cooperation between Unity and CMS

A CMS (Content Management System) is software used to manage the content of a website or application, and a CMS makes it easy for non-technical people to create, edit, and publish content. Typical CMSs include WordPress, Drupal, Joomla, Magento, Shopify, MediaWiki, etc. (For more information on MediaWiki, please refer to “How to set up MAMP and media wiki and how to use them easily“). There are several approaches to realize the integration between the CMS and Unity, as shown below.

Using a RESTful API:
- Create a RESTful API endpoint on the CMS side, which allows data to be read and written from Unity.
- Within Unity, use a library for making HTTP requests (e.g. UnityWebRequest) to communicate with the API endpoint.
- If necessary, parse and use the received data within Unity.
Using plug-ins:
- The use of plug-ins for Unity provided by CMS vendors can be used to achieve collaboration. This allows for easy communication and data synchronization with the CMS.
- You can find plug-ins for specific CMSs in the Unity Asset Store and other places. For more information on how to install and use plug-ins, please refer to the documentation and tutorials for each plug-in.
Direct database manipulation:
- One method is to connect to the database from within Unity and read and write data directly. However, care must be taken in terms of security and data integrity.
- Use a database connection library (e.g. SQLite) in Unity and execute queries against the database. This operation must be performed in accordance with the database structure on the CMS side.

Unity and Chatbot Integration

A chatbot is a computer program or artificial intelligence (AI) system, a tool that can provide information or perform tasks through user interaction (for more information on chatbots, see “Chatbots and Question and Answer Technology“). (For more information on Chatbot, see “Chatbots and Question-and-Answer Technology. There are several ways to integrate the chatbot with Unity, including

Interaction via REST API: Using the RESTful API provided by the chatbot service, a request can be sent from Unity to interact with the chatbot. This is accomplished by sending a request to the API endpoint using Unity’s WebRequest or HttpClient and retrieving and displaying the chatbot’s response.
Collaboration using WebSockets: If the chatbot service supports WebSockets, a WebSocket client can be implemented in Unity to interact with the chatbot in real-time. This can be done by establishing a WebSocket connection, sending and receiving messages, and displaying the chatbot’s responses in Unity.
Use of plug-ins: Plug-ins for chatbots, available from the Unity Asset Store and elsewhere, can also be used. These plug-ins integrate chatbot functionality within Unity, making it easy to use, import and configure plug-ins, and implement chatbot interaction in Unity.
Use of Natural Language Processing Libraries: Natural Language Processing (NLP) libraries can also be used to analyze text and process dialogue within Unity. This could, for example, be a way to use Python’s NLP library to parse text and generate responses, and then exchange data with Unity.

Unity and Elasticsearch Integration

Elasticsearch is the default OSS for search engines (for more information on ES, please refer to “Overview of Search Systems and Examples of Implementations Focusing on Elasticsearch“), and no official Unity package or plug-in provides a means of directly linking it with Unity. These are not provided in official Unity packages or plug-ins. They can be linked using the following methods

Connect via REST API: Elasticsearch provides a RESTful API that allows you to send and receive data using HTTP requests from Unity. This means that you can use Unity’s WebRequest or HttpClient to send requests to Elasticsearch’s API endpoints to perform operations such as searching and indexing data.
Use Elasticsearch client libraries: Elasticsearch provides client libraries for each programming language. NET and Nest for C#. These libraries can be incorporated into Unity projects to send/receive data and perform search operations with Elasticsearch.

Since Unity is mainly used for developing mobile and desktop applications, direct integration with Elasticsearch is not common and is generally done by using Unity to develop server-side applications and back-end systems, and then using the The general approach is to develop server-side applications and back-end systems using Unity, and then have those systems work with Elasticsearch.

In addition, when using Unity to integrate with Elasticsearch, it is necessary to consider security and performance aspects, such as appropriate authentication and data encryption, and optimization of network traffic.

Linking Unity with the Machine Learning Module

Machine learning technology is the foundation of AI technology and can be combined with other AI technologies to realize a variety of extended functions (see “Machine Learning Technology” and “Overview of Machine Learning and Data Analysis with Pyhton and Introduction to Typical Libraries” for more information on machine learning technology). There are several approaches to integrating the machine learning module with Unity, as shown below.

Linkage via REST API: The simplest configuration is to configure the machine learning module on the server and link it via the REST API.
TensorFlow Unity Plugin: The TensorFlow Unity Plugin can be used to load and run TensorFlow models within Unity. This plug-in provides an API to use TensorFlow models within Unity scripts, train TensorFlow models, and integrate them into Unity for machine learning capabilities.
Unity ML-Agents: Unity ML-Agents (Machine Learning Agents) will be a machine learning framework provided by Unity that allows Unity’s game environment to be controlled by machine learning agents, and Unity ML-Agents Unity ML-Agents can train agents using a variety of machine learning methods, including reinforcement learning and evolutionary algorithms.
ONNX Runtime: ONNX Runtime is an open machine learning inference engine that can also be used in Unity. ONNX Runtime supports models exported by many machine learning frameworks.
Custom Integration: Since Unity uses C#, it is also possible to use popular machine learning libraries (e.g. TensorFlow.NET, SciSharp.TensorFlow) and machine learning frameworks (e.g. PyTorch, Keras) in C#. Unity. Using this approach, models can be loaded within Unity and custom linkages can be implemented to predict and classify data.

Linking Unity and Natural Language Processing

Natural Language Processing (NLP) is a generic term for technologies and methods that enable computers to understand, interpret, and generate natural language used by humans in everyday life. It is the basis for artificial intelligence technologies used in machine translation, document classification, information extraction, question answering systems, sentiment analysis, conversational agents, and many other applications (see “Natural Language Processing Technologies” for more information on natural language processing technologies). There are several ways to integrate the natural language processing module with Unity.

Integration via REST API: The simplest configuration is to configure the NLP module on the server and integrate it via the REST API.
Plug-ins: This approach uses NLP plug-ins available from the Unity Asset Store and other sources, which provide natural language processing, intent interpretation, and other functionality that can be integrated into Unity projects. Plug-ins may include features such as text analysis, sentiment analysis, keyword extraction, etc.
Use of APIs: This is a way to use external NLP services and APIs. Some well-known NLP service providers (Google Cloud Natural Language API, Microsoft Azure Cognitive Services, IBM Watson Natural Language Understanding, etc.) offer APIs. From within Unity, you can send an HTTP request to communicate with the API and retrieve the results.
Use of open source libraries: Another option is to use open source NLP libraries. These are, for example, NLTK (Natural Language Toolkit), spaCy, Stanford CoreNLP, etc. These libraries can be incorporated into a Unity project, and scripts can be created to perform the necessary NLP tasks.
Building an external NLP environment and linking: There is also a way to build an external NLP environment in python or other languages and use tools such as Python.NET or IronPython to set up and link with the Python runtime for calling Python scripts from Unity. Using this method, you can set up a Python environment outside of the Unity project, install the NLP library, and call the python scripts.
Homegrown implementation: Another option is to implement your own NLP module within Unity. This is where you implement your own NLP tasks such as text processing, intent interpretation, etc. using C# and Unity functionality. However, this is a complex task and requires knowledge of machine learning and natural language processing.

The form that appears in all of the above approaches is to implement the functional modules to be linked on the server, embed REST APIs in them, and link them from the Unity side using WebRequest or HttpClient, etc. This loosely coupled application using REST APIs can be thought of as a microservices structure.

This loosely coupled application using REST API can be thought of as a microservice structure (for details on microservices, please refer to “Microservices and Efficient Application Development and Multi-Agent Systems“).

The following is an example of a concrete implementation using the REST API.

Example of REST API Implementation

The specific implementation of cooperation via REST API is described using the case of Elasticsearch as an example.

In Unity, requests can be sent to the Elasticsearch API endpoint using WebRequest or HttpClient to perform operations such as data retrieval and indexing. The basic steps are as follows

Prepare API endpoint: Prepare an Elasticsearch API endpoint. This includes the Elasticsearch host name, port number, and API path.
Create the request: Create the request in a Unity script. This includes the URL of the API endpoint, the HTTP method (GET, POST, PUT, DELETE, etc.), any necessary header information, and the request body (if necessary).
Sending the request: Use Unity’s WebRequest or HttpClient to send the created request to the API endpoint; if using WebRequest, instantiate the WebRequest class and include the methods (GET, POST, PUT, When using HttpClient, instantiate the HttpClient class and call the necessary methods (GetAsync, PostAsync, PutAsync, DeleteAsync, etc.) to send the Send the request.
Process response: Receive response from Elasticsearch and perform necessary processing. Since the response is returned in JSON format, it is necessary to parse the JSON and extract data within Unity scripts to properly process the data received and use it for display and manipulation.
Error handling: Since errors can occur while sending a request or processing a response, it is important to perform error handling appropriately, displaying error messages and processing in response to errors.

Next, we will discuss the case of chatbot as an example of collaboration via websocket, which is another form of general-purpose collaboration.

Integration between Unity and Chatbot using WebSocket

In order to implement a WebSocket client in Unity and interact with a chatbot in real time, the following steps are generally followed.

Obtain a WebSocket client library: Obtain a WebSocket client library in order to perform WebSocket communication within Unity. Several open-source WebSocket client libraries exist, typical ones include WebSocket-Sharp and NativeWebSocket, etc. Find the library in the Unity Asset Store or GitHub and integrate it into your project. WebSocket client implementation: use WebSocket-Sharp or NativeWebSocket.
WebSocket client implementation: Implement a WebSocket client in Unity using the obtained WebSocket client library. This requires implementing basic functions such as establishing a connection, sending and receiving messages, and disconnecting. Refer to the library documentation and sample code for the appropriate implementation.
Implementation of protocol with chatbot: It is necessary to implement a protocol for exchanging data with the chatbot. In general, WebSocket communication is used to send and receive data in JSON format. We will serialize and deserialize JSON in Unity according to the communication protocol with the chatbot, and implement sending and receiving messages.
UI implementation: Create the user interface using Unity’s UI elements. This includes a display area for the chatbot interaction, a text input field, and a send button. When the user enters and sends a message, ensure that the message is sent to the chatbot via the WebSocket client.
Event processing and UI updates: Perform appropriate event processing and UI updates based on messages received from the WebSocket client and changes in connection status. Parse and display received messages, display error messages when the connection is broken, etc.

Reference Book

Reference books on unity include “Unity From Zero to Proficiency (Foundations) [Fourth Edition]: A step-by-step guide to creating your first game with Unity” “Unity in Action, Third Edition: Multiplatform game development in C#” and “Hands-On Unity 2022 Game Development: Learn to use the latest Unity 2022 features to create your first video game in the simplest way possible, 3rd Edition”.