Generative AI and Immersive Tech | MAGES Institute

Generative AI and Immersive Tech

30 March, 2024

See examples of how generative AI can be used in conjunction with immersive technology to drive next generation experiences.

What is Generative AI?

An AI model capable of generative text, images or other data when supplied with a prompt is referred to as Generative AI (or GenAI).

GenAI are trained on existing large datasets to “learn” the patterns and structure of the output, based on which they create or generate content. Examples of GenAI include :

  • large-language models (LLMs) such as ChatGPT, Copilot, Gemini, LLaMA;
  • text-to-image models such as DALLe and Midjourney; and
  • the latest to hit the market : text-to-video model Sora (also by OpenAI).

 

Solving the Content Problem

GenAI is vastly utilised across various industries to solve the problem of content generation, by offloading tedious or cognitively-heavy tasks. It is easier and far more efficient to edit and amend a two-thousand word article written by GenAI than it is to hit the word count organically.

Similarly, various artists have incorporated GenAI outputs within their workflow.

Synergy

GenAI can be considered a catalyst and a growth driver. When applied to “current” technology, GenAI opens up the next realm of what is possible in the respective space.

LLM Models have been used in user-created mods for games, such as the Inworld mod for The Elder Scrolls V : Skyrim, where every NPC interaction is powered by ChatGPT. This allows for a degree of immersiveness not possible before, as every user’s prompt and ChatGPT’s response to the prompt may not be the same as everyone else’s experience.

Similar mods such as NVidia’s ACE aim to do the same thing (We’re written about AI-powered NPCs, here!)

Since GenAI can generate content at such a fast rate, it allows for context aware interactions for platforms more than just games. For example :

Student Project : Fetus Lab

Fetus Lab is an education-focused VR-based student project, which utilises ChatGPT to generate responses about the subject matter in real time.

Users are able to ask questions about what they are seeing – a cross-section of a fetus – and get real-time contextualised answers as if they were asking a human assistant.

Coupled with an AI-based text-to-speech tool, the illusion of a virtual personal assistant guiding you through the immersive experience is no longer a distant reality.

GenAI for Education

The key factor driving the next-generation of education is the real-time contextualised responses, which enables a more immersive and personalised learning environment. The interactive 3D visual component to the learning greatly reduces cognitive load and increases memory recall for complex or abstract topics such as STEM (Source).

Other benefits include :

Personalised Learning

Content can be tailored to individual needs – some students may respond well to one method or lesson-flow while others may not. Sufficiently trained GenAI would be able to recognise and select the most appropriate method and deliver accordingly

Dynamic and Adaptive Content

The interactivity enabled by GenAI allows for dynamic learning environments. Concepts can be explained in various ways through different agents (for example, geopolitical tensions can be studied in the context of both WWI and WWII).

Breaking Barriers

Through GenAI combined with AR/VR/MR, students may be able to experience immersive environments that they otherwise wouldn’t be able to – for example, exploring the bottom of the Mariana Trench or a simulation of the surface of the Moon.

Conclusion

GenAI is a technological breakthrough and a catalyst of the next era of computing. Combined with high-end immersive technology such as the Apple Vision Pro mixed reality headset, the days of “Spatial Computing” are upon us.

To get a head start in the dynamic and evolving field of technology, check out MAGES’ #GetIntoTech programs! Covering Full Stack Development, Product Management with UI/UX Design and Cybersecurity, the #GetIntoTech programs are designed to convert adult learners from no experience in tech roles into landing a tech role as a professional, in 6 months.

    Mages Whatsup