The project is a collaboration between Buffalo Buffalo, a local game studio based in Vancouver, BC and Team CanCan, a multidisciplinary team of students from the Centre for Digital Media (CDM).

Hollerena is a 3D voice-controlled arena bot battling game with AI integration. This innovative solution includes incorporating two AI technologies into the core game mechanics: voice-to-text and LLM generation. These additions aim to enhance gameplay, introducing chaotic and unexpected fun through the use of artificial intelligence.

Project Overview

Doppelganger, is an AI-integrated cross-platform party pack that includes four mini-games conceptualized by Buffalo Buffalo. Our team has chosen to develop one of the simulations: Hollerena, a speech-powered multiplayer party game featuring robotic arena battles. The current challenge is to create a dynamic and engaging gaming experience using voice-controlled behavior and leveraging Language Models (LLMs) to introduce variability and unique scenarios based on the game state. Traditional game mechanics can sometimes feel static or predictable, leading to decreased player engagement over time. In modern gaming, players seek immersive experiences that offer variability and adaptability. By integrating voice control and LLM-based adaptability, the goal is to enhance player interaction and create a more dynamic gaming environment. 

Challenges

The primary issue lies in the limitations of traditional game design approaches, which may struggle to provide the level of variability and adaptability desired by players. Additionally, integrating voice control and LLM technologies seamlessly into the game architecture poses technical challenges that need to be addressed. The ideal outcome is a gaming experience that feels immersive, dynamic, and tailored to each player's actions and preferences. Ideally, players should feel empowered to interact with the game world through natural language commands, while the LLM intelligently adjusts game rules and introduces new challenges and opportunities based on the evolving game state. 

Hollerena offers automatic speech recognition that allows players to interact with their characters and engage in GenAI-powered conversations. This feature adds a layer of realism and immersion, making the gameplay more dynamic and engaging.

The game enriches the experience with adaptive gameplay using GenAI to introduce chaos. This AI-driven unpredictability keeps players on their toes, ensuring that each match is unique and full of surprises.

Players can throw generated trash talk at their opponents with the click of a button. This feature adds a fun and competitive edge to the game, allowing players to taunt and distract their rivals using AI-generated banter.

Speech-to-Text technology was employed to control the player and facilitate interactions with the LLM. To create a cost-effective and efficient solution, the model was implemented locally on Azure, effectively minimizing token generation costs. The Whisper Model was selected for this purpose, specifically the TINY variant, which features only 39 million parameters and operates with exceptional speed.

An Ubuntu Linux virtual machine was deployed on Azure to manage a Python/FastAPI backend, utilizing OpenAI's Speech-to-Text transcription. Previously, the local setup restricted gameplay to a single network; however, transitioning to the cloud offered significant advantages, including enhanced scalability. By leveraging Azure's infrastructure, the server functioned as an interface for interacting with the AI models. This migration not only enabled global access and improved performance but also ensured high availability, allowing users to connect seamlessly.

The LLM is designed to be contextually aware of its role as a God figure within the game environment. Gemini LLM was selected for its free API and its capability to effectively manage complex queries. In this interactive setting, Gemini not only makes decisions that players can challenge through speech-based arguments but also expresses its moods through various visual representations.

To tailor the LLM's responses to the specific context of the game, a Retrieval Augmented Generation approach inspired by Dynamic Context Injection was implemented. This method involves feeding the LLM with the current game state and player responses, enabling it to make decisions that directly influence gameplay. Furthermore, the LLM interprets word choices to assess player emotions, which can result in either rewards or punishments. To facilitate the integration of its responses into the game and execute functions based on these decisions, in-context learning was utilized to convert the LLM's output into a consumable JSON format. These constraints are essential for preventing the LLM from hallucinating while maintaining contextual integrity throughout extended chains of thought.

These animations were created to illustrate the various phases of interaction between the players and the AI during the intermission, where players plead for mercy. The AI's disconnected body elements were used intentionally to enhance its personality as the game host, adding a distinctive character to the overall experience. This design choice reinforces the AI's role, making its presence more dynamic and engaging for the players.

This lineup showcases our four current characters, each with a unique occupation.

The Ink Bot serves as a writer, the Boom Box Bot takes on the role of a DJ, the Furnace Knight is, fittingly, a knight, and the Burger Bot is a chef. Each character brings its own personality and profession, adding depth to the gameplay experience.

3D Character development