AI-Powered Virtual Humans for Training and Storytelling
On 7 November we will be presenting at the AI-Builders network event in Amsterdam. We will be talking about our AI applications and showcasing some demo’s. Read more about our presentation here below the video.
ImproVive has over a decade of experience in creating VR experiences and games for training and storytelling, primarily using the Unity Game Engine. One of our key challenges has been developing interactive non-player characters (NPCs) or avatars. Traditionally, NPC interactions relied on keyword recognition, multiple-choice menus, and decision trees to create dialogues, a method that was often time-consuming and complex to execute. An example of our past work, the VR game Reaction, used indirect speech interactions yet still delivered an empowering experience for players.
With advancements in large language models (LLMs), we now incorporate LLMs into our game engine alongside speech-to-text, text-to-speech, and retrieval-augmented generation (RAG) technologies. By combining these with realistic 3D avatars, which we refer to as Virtual Humans, we create engaging and meaningful interactive characters. Our Virtual Humans respond verbally to user input and also convey non-verbal cues through emotional expressions and gestures. By blending the strengths of LLMs with game engine dynamics, we ensure a well-controlled flow and pacing for each experience or training session.
Our game engine allows seamless deployment across platforms, including PC, mobile, web, VR, and AR. To streamline storytelling, we developed a no-code editor for rapid iteration, making it easier for creators to refine and enhance experiences.
In VR and AR, our avatars have spatial context and awareness, enabling them to recognize objects a user may be holding or respond to specific objects in the environment, enhancing the realism and depth of the interaction.
In our presentation, we will showcase how these technologies work both on a large screen and in an augmented reality (AR) setup.
Currently, ImproVive is applying this technology in training and interactive storytelling. For example, we developed an installation for Museum de Heksenwaag in Oudewater, where visitors can interact with a woman accused of witchcraft and her prosecutor. For Kamp Amersfoort, we are designing a new experience focused on group interaction. Additionally, we are exploring training and coaching applications for various large organisations.
