Revolutionizing Game Development: Unity’s AI Advancements with Unity Muse and Unity Sentis

AI has already begun to be applied in areas where it is necessary to analyze and generate content. In particular, artificial intelligence is implemented in game and app development. AI is getting popular primarily due to its ability to significantly speed up and facilitate the content development process. Instead of spending several hours manually creating graphics for games and applications, you can use AI to generate the necessary components on its own simply. Then the developer will make their edits.

Therefore, it is not surprising that the number of companies using AI is growing. According to  Forbes Advisor, by 2027 the artificial intelligence market will be estimated at 407 billion USD. And 64% of enterprises believe that artificial intelligence will increase their productivity.

Unity, one of the most famous app manufacturers for portable gadgets, is among the companies that are already gradually introducing AI into their work. 

In this article, we will take a detailed look and analyze the new artificial intelligence developments from Unity and what they mean for the development of XR applications.

Leveraging Unity AI for App Development

Unity has been using AI for a few years now, but, firstly, it was mostly for marketing and non-game content to a wider audience. Like, for example, Luna, a platform that aims to improve the advertising strategy of brands. Or Supersonic which is a stand-alone platform that automates and accelerates game advertising.

Secondly, artificial intelligence and neural networks for gaming have previously been created only for developers who have special qualifications for working with applications. Like, for example, ML agents, which allow you not to code the behavior of digital elements, because these elements themselves (smart agents) are independently learning the algorithms of behavior in the environment. For example, in a game from Unity, you can hone the behavior of a digital corgi that runs after a stick. Or the behavior of racing cars, as in this video.

Read also: Transforming Reality: How AI Revolutionizes XR for Next-Level Experiences

“In the carting micro game, our ML agent scenes are typically set up with a few major components: the training environment, the level, that we want our agents to observe and familiarize with. Which, in our case, is our racetracks. The agents are the game objects we want to train. So, that they can accomplish a goal, such as driving without colliding with the track walls. And, finally — our academy, which collects all observations, done by our agents and trains them”, said Unity’s official video.

In March 2023, Unity announced their own AI game and application development features. Then there was the first presentation of an ecosystem with artificial intelligence, which operates according to the principle of many AI generators, such as Midjourney and ChatGPT.

A few days ago, new AI tools for developing games and applications were announced: Unity Muse and Unity Sentis, which we will discuss in more detail below. 

Unity Muse: Empowering AI Creativity

Unity Muse is the newest platform with artificial intelligence, the main purpose of which is to accelerate the creation of applications with 3D graphics. In particular, the option of this platform includes such basic functions as:

  • automatic creation of animated characters and digital doubles;
  • creation of the movements of the animated model — for example, jumps or backflips;
  • creating a texture for the environment;
  • addition to ready-made 3D models — for example, the video shows how you can highlight the roof area on the digital model of the house, enter a prompt and select several options for a red roof from the database.

“The Muse platform provides quick access to detailed information from the documentation and can also generate 2D sprites and 3D animations. This allows developers to quickly get the resources they need to create prototypes or new experiences without waiting for final versions of models and sprites. With Muse chat, developers can effectively communicate with other team members and get the help they need during development”, said Qualium Systems Unity tech lead Arcueid D’athemon.

The main difference between Unity Muse and ML agents is ease of use. The newest platform looks like a chat with a neural network and the principle of operation resembles the sensational ChatGPT. As mentioned above, the user enters the required prompt to generate the 3D content required for the application.

Harness the Power of Unity Sentis AI

Unity Sentis is a cross-platform tool that integrates AI into Unity applications. According to Unity’s official announcement, Sentis enables in-app digital models to work flawlessly on all operating systems, browsers, and devices that support Unity, from smartphones and tablets to game consoles and XR glasses.

Thus, applications with built-in AI models work more efficiently due to the fact that data is processed in real-time.

However, according to Arcueid D’athemon, Unity tech lead at Qualium Systems, the range of  Sentis options is wider, and the platform can also generate voice acting and animation for a character.

“It opens up opportunities to quickly create dynamically generated actions in the application that will depend on the environment or interaction with the user. With Sentis, developers can effectively create realistic character movements and high-quality voice acting that enriches the user experience”, said Arcueid D’athemon.

In the end, the combination of the input of the own neural network can change the process of the application use. For example, the user can directly interact with an NPC, while would respond to him with lines that are not pre-recorded. It enhances the immersive experience of gameplay, using the example of a virtual AI character named Orb.

Unity’s interest in AI in app development is evident with recent innovations such as Unity Muse and Unity Sentis. The first tool allows developers to quickly and efficiently create 3D models of game and non-game characters, environments, and textures. As for Sentis, this program allows you to modify the gameplay by introducing NPCs with their own artificial intelligence. Unity’s AI ecosystem continues to evolve, inspiring game and app developers to explore new horizons and reimagine what’s possible in interactive entertainment.

Latest Articles

April 29, 2024
Apple Vision Pro Software Transforms the Construction Industry: The REEKON Experience

Virtual Reality, Augmented Reality, and Extended Reality technologies are revolutionizing various industries, and construction is no exception. While the utilization of Apple Vision Pro in manufacturing or construction industries may not yet be widespread, an increasing number of companies are endeavoring to integrate virtual reality, augmented reality, and extended reality technologies into their daily operations. Why? Let’s try to figure it out in this article! Construction Industry Problems While one might anticipate that technological advancements would alleviate challenges within the construction sector, construction firms frequently encounter a myriad of obstacles that impede efficiency, escalate costs, and compromise safety. Here are some key challenges in construction: Design Visualization and Communication: Traditional blueprints and 2D drawings can be difficult for stakeholders to interpret accurately. Design Iterations and Prototyping: Iterating on design concepts and prototyping can be time-consuming and costly. Construction Planning and Logistics: Planning construction activities and logistics on-site can be complex and error-prone. Worker Training and Safety: Safety is a paramount concern in construction, yet traditional training methods may not effectively prepare workers for on-site hazards. Quality Control and Inspection: Ensuring quality control and conducting inspections during construction can be labor-intensive and prone to human error. Client Engagement and Marketing: Engaging clients and stakeholders in the design process and marketing new developments can be challenging with traditional presentation methods. Remote Collaboration and Coordination: Coordinating teams and stakeholders who are dispersed across different locations can be challenging and time-consuming. Immersive technologies such as Virtual Reality, Augmented Reality, and Mixed Reality, utilizing Apple Vision Pro, offer innovative solutions to many of these problems. Seamless AR Integration with the ROCK Jobsite App on Apple Vision Pro by REEKON Tools One notable example of this transformative technology in action is the implementation of the ROCK Jobsite App on Apple Vision Pro, as demonstrated by REEKON Tools. The ROCK Jobsite App, designed to streamline construction processes, represents a significant advancement in leveraging AR technology using Apple Vision Pro within the construction industry. Unlike many other VR/AR solutions that require extensive customization and integration efforts, the ROCK Jobsite App boasts seamless functionality on the Apple Vision Pro platform. Within just five minutes of installation, users can experience the full capabilities of this powerful tool, making it incredibly accessible and user-friendly. One of the key features of the ROCK Jobsite App is its ability to display measurements in real-time, providing construction professionals with immediate access to crucial data directly on their screens. The integration of Apple Vision Pro enhances this process, making it both effective and engaging. Whether annotating over photos, adding measurements to calculations, or collaborating with team members remotely, this app serves as a valuable companion throughout the construction process How Immersive Technologies Address Construction Problems The integration of Apple Vision Pro into VR/AR/XR technology marks a significant leap forward in the construction sector’s evolution. By tapping into the immersive capabilities of these technologies, construction companies can not only tackle challenges but also unearth fresh avenues for innovation. Here are some standout benefits: Advanced Visualization: With immersive technologies and Apple Vision Pro, stakeholders can immerse themselves in architectural designs and construction plans. This heightened visualization enables a clearer grasp of project requirements and early detection of potential issues. Enhanced Collaboration: Real-time data sharing and annotations foster more efficient collaboration among project teams, regardless of their physical locations. Boosted Efficiency: By automating tasks like data capture and measurement, Apple Vision Pro-equipped tools help construction professionals save time and resources. Manual efforts are replaced with streamlined processes, leading to heightened efficiency on-site. Cost Reduction: AR technology, when integrated with Apple Vision Pro, minimizes errors, lowers rework, and optimizes resource allocation, resulting in cost savings across the project lifecycle The potential applications of AR technology in construction are boundless, from fortifying safety measures to facilitating training simulations. By addressing industry challenges and equipping construction professionals with AR solutions powered by Apple Vision Pro, are reshaping the construction landscape. They’re paving the way for safer, more efficient, and more sustainable building practices.

February 29, 2024
Everything you’d like to know about visionOS development

If you’re venturing into the realm of developing applications for Apple Vision Pro, it’s crucial to equip yourself with the right knowledge. In this article, we unravel the key aspects you need to know about the visionOS operating system, the secrets of programming for Apple Vision Pro, and the essential tools required for app development. visionOS: The Heart of Apple Vision Pro The foundation of the Vision Pro headset lies in the sophisticated visionOS operating system. Tailored for spatial computing, visionOS seamlessly merges the digital and physical worlds to create captivating experiences. Drawing from Apple’s established operating systems, visionOS introduces a real-time subsystem dedicated to interactive visuals on Vision Pro. This three-dimensional interface liberates apps from conventional display constraints, responding dynamically to natural light. At launch, visionOS will support a variety of apps, including native Unity apps, Adobe’s Lightroom, Microsoft Office, medical software, and engineering apps. These applications will take advantage of the unique features offered by visionOS to deliver immersive and engaging user experiences. Programming Secrets for Apple Vision Pro Programming for Apple Vision Pro involves understanding the concept of spatial computing and the shared space where apps coexist. In this floating virtual reality, users can open windows, each appearing as planes in the virtual environment. These windows support both traditional 2D views and the integration of 3D content. Here are some programming “secrets” for Apple Vision Pro: All apps exist in 3D space, even if they are basic 2D apps ported from iOS. Consider the Field of View and opt for a landscape screen for user-friendly experiences. Prioritize user comfort and posture by placing content at an optimal distance. Older UIKit apps can be recompiled for VisionOS, gaining some 3D presence features. Be mindful of users’ physical surroundings to ensure a seamless and comfortable experience. Tools for Apple Vision Pro Development To initiate the development of applications for Vision Pro, you’ll need a Mac computer running macOS Monterey or a newer version. Additionally, you’ll require the latest release of Xcode and the Vision Pro developer kit. The development process entails downloading the visionOS SDK and employing familiar tools such as SwiftUI, RealityKit, ARKit, Unity, Reality Composer Pro, and Xcode, which are also utilized for constructing applications on other Apple operating systems. While it’s feasible to adapt your existing apps for Vision Pro using the visionOS SDK, be prepared for some adjustments in code to accommodate platform differences. Most macOS and iOS apps seamlessly integrate with Vision Pro, preserving their appearance while presenting content within the user’s surroundings as a distinct window. Now, let’s delve into the essentials for assembling your own Apple Vision Pro development kit: SwiftUI: Ideal for creating immersive experiences by overlaying 3D models onto the real world. Xcode: Apple’s integrated development environment, vital for app development and testing. RealityKit: Exclusively designed for Vision Pro, enabling the creation of lifelike, interactive 3D content. ARKit: Apple’s augmented reality framework for overlaying digital content onto the real world. Unity: A powerful tool for visually stunning games and Vision Pro app development. Unity is currently actively developing its SDK to interface with Apple Vision Pro. What’s the catch? Few people know that to develop on Unity, you need not just any Mac, but a Mac with an “M” processor on board! Here are a few more words about supported versions: Unity 2022 LTS (2022.3.191 or newer): Apple Silicon version only. Xcode 15.2: Note that beta versions of Xcode are a no-go. VisionOS 1.0.3 (21N333) SDK: Beta versions are not supported. Unity editor: Apple Silicon Mac and the Apple Silicon macOS build are in; the Intel version is out. Pay attention to these restrictions during your development journey! Apple Vision Pro SDK: Empowering Developers The visionOS Software Development Kit (SDK) is now available, empowering developers to create groundbreaking app experiences for Vision Pro. With tools like Reality Composer Pro, developers can preview and prepare 3D models, animations, and sounds for stunning visuals on Vision Pro. The SDK ensures built-in support for accessibility features, making spatial computing and visionOS apps inclusive and accessible to all users. As Apple continues to lead the way in spatial computing, developers hold the key to unlocking the full potential of the Vision Pro headset. By understanding the intricacies of visionOS, programming secrets, essential development tools, and the application process for the developer kit, you can position yourself at the forefront of this revolutionary technological landscape.

January 30, 2024
Touching Art: How Haptic Gloves Empower to “See” the World of Art

In the realm of art, visual experiences have long been the primary medium of expression, creating a challenge for those with visual impairments. However, a groundbreaking fusion of haptic technology and VR/AR is reshaping the narrative. Explore the innovative synergy between haptic technology and VR/AR and how this collaboration is not only allowing the blind to “see” art but also feel it in ways previously unimaginable. Artful Touch – Haptic Technology’s Role in Art Appreciation Haptic technology introduces a tactile dimension to art appreciation by translating visual elements into touch sensations. Equipped with sensors and precision, haptic gloves enable users to feel textures, contours, and shapes of artworks. This groundbreaking technology facilitates a profound understanding of art through touch, providing a bridge to the visual arts that was once thought impossible for the blind to cross. VR/AR technologies extend this tactile experience into virtual realms, guiding users through art galleries with spatial precision. Virtual environments created by VR/AR technologies enable users to explore and “touch” artworks as if they were physically present. The combination of haptic feedback and immersive VR/AR experiences not only provides a new means of navigating art spaces but also fosters a sense of independence, making art accessible to all. Prague Gallery Unveils a Touchful Virtual Reality Experience The Prague’s National Gallery has taken a revolutionary step towards inclusivity in art with its groundbreaking exhibition, “Touching Masterpieces.” Developed with support of Leontinka Foundation, a charity dedicated to children with visual impairments, this exhibit redefines the boundaries of art appreciation. Visitors to the exhibition, especially those who are blind or visually impaired, are invited to embark on a sensory journey through iconic sculptural masterpieces. Among them are the enigmatic bust of Nefertiti, the timeless Venus de Milo sculpture, and the immortal David by Michelangelo. What sets this exhibition apart is the integration of cutting-edge technology – haptic gloves. These gloves, dubbed “avatar VR gloves,” have been meticulously customized for the project. Using multi-frequency technology, they create a virtual experience where a user’s hand can touch a 3D object in a virtual world, providing tactile feedback in the form of vibrations. The key innovation lies in the gloves’ ability to stimulate different types of skin cells’ tactile responses, ensuring that users, particularly the blind, receive the most accurate perception of the 3D virtual objects on display. As visitors explore the exhibit, they can virtually “touch” and feel the intricate details of these masterpieces, transcending the limitations of traditional art appreciation. Future Possibilities and Evolving Technologies As technology advances, the future holds even more possibilities for inclusive art experiences. The ongoing collaboration between haptic technology and VR/AR promises further refinements and enhancements. Future iterations may introduce features such as simulating colors through haptic feedback or incorporating multisensory elements, providing an even more immersive and enriching experience for blind art enthusiasts. The collaboration between haptic technology and VR/AR is ushering in a new era of art perception, where touch and virtual exploration converge to create a truly inclusive artistic experience. By enabling the blind to “see” and feel art, these technologies break down barriers, redefine traditional boundaries, and illuminate the world of creativity for everyone, regardless of visual abilities. In this marriage of innovation and accessibility, art becomes a shared experience that transcends limitations and empowers individuals to explore the beauty of the visual arts in ways never thought possible.



Let's discuss your ideas

Contact us