The project is a collaboration between Buffalo Buffalo, a local game studio based in Vancouver, BC and Team CanCan, a multidisciplinary team of students from the Centre for Digital Media (CDM).

Hollerena is a 3D voice-controlled arena bot battling game with AI integration. This innovative solution includes incorporating two AI technologies into the core game mechanics: voice-to-text and LLM generation. These additions aim to enhance gameplay, introducing chaotic and unexpected fun through the use of artificial intelligence.

Project: Hollerena: Generative AI Integrated Speech-Powered Mech Robot Arena Battle Game

Genre: GenAI, PC, Game

Role: Project Manager | Client Key Contact | Lead UI/UX

Platform: Unity Engine on PC

Timeframe: 4 days sprints for 12 weeks

Tools: Figma, Adobe PS, Ai, PR, Google Suite, Canva, Discord, Trello, Miro.

Project Overview

Doppelganger, is an AI-integrated cross-platform party pack that includes four mini-games conceptualized by Buffalo Buffalo. Our team has chosen to develop one of the simulations: Hollerena, a speech-powered multiplayer party game featuring robotic arena battles. The current challenge is to create a dynamic and engaging gaming experience using voice-controlled behavior and leveraging Language Models (LLMs) to introduce variability and unique scenarios based on the game state. Traditional game mechanics can sometimes feel static or predictable, leading to decreased player engagement over time. In modern gaming, players seek immersive experiences that offer variability and adaptability. By integrating voice control and LLM-based adaptability, the goal is to enhance player interaction and create a more dynamic gaming environment. 

Challenges

The primary issue lies in the limitations of traditional game design approaches, which may struggle to provide the level of variability and adaptability desired by players. Additionally, integrating voice control and LLM technologies seamlessly into the game architecture poses technical challenges that need to be addressed. The ideal outcome is a gaming experience that feels immersive, dynamic, and tailored to each player's actions and preferences. Ideally, players should feel empowered to interact with the game world through natural language commands, while the LLM intelligently adjusts game rules and introduces new challenges and opportunities based on the evolving game state. 

Hollerena offers automatic speech recognition that allows players to interact with their characters and engage in GenAI-powered conversations. This feature adds a layer of realism and immersion, making the gameplay more dynamic and engaging.

The game enriches the experience with adaptive gameplay using GenAI to introduce chaos. This AI-driven unpredictability keeps players on their toes, ensuring that each match is unique and full of surprises.

Players can throw generated trash talk at their opponents with the click of a button. This feature adds a fun and competitive edge to the game, allowing players to taunt and distract their rivals using AI-generated banter.

Speech-to-Text technology was employed to control the player and facilitate interactions with the LLM. To create a cost-effective and efficient solution, the model was implemented locally on Azure, effectively minimizing token generation costs. The Whisper Model was selected for this purpose, specifically the TINY variant, which features only 39 million parameters and operates with exceptional speed.

An Ubuntu Linux virtual machine was deployed on Azure to manage a Python/FastAPI backend, utilizing OpenAI's Speech-to-Text transcription. Previously, the local setup restricted gameplay to a single network; however, transitioning to the cloud offered significant advantages, including enhanced scalability. By leveraging Azure's infrastructure, the server functioned as an interface for interacting with the AI models. This migration not only enabled global access and improved performance but also ensured high availability, allowing users to connect seamlessly.

The LLM is designed to be contextually aware of its role as a God figure within the game environment. Gemini LLM was selected for its free API and its capability to effectively manage complex queries. In this interactive setting, Gemini not only makes decisions that players can challenge through speech-based arguments but also expresses its moods through various visual representations.

To tailor the LLM's responses to the specific context of the game, a Retrieval Augmented Generation approach inspired by Dynamic Context Injection was implemented. This method involves feeding the LLM with the current game state and player responses, enabling it to make decisions that directly influence gameplay. Furthermore, the LLM interprets word choices to assess player emotions, which can result in either rewards or punishments. To facilitate the integration of its responses into the game and execute functions based on these decisions, in-context learning was utilized to convert the LLM's output into a consumable JSON format. These constraints are essential for preventing the LLM from hallucinating while maintaining contextual integrity throughout extended chains of thought.

These animations were created to illustrate the various phases of interaction between the players and the AI during the intermission, where players plead for mercy. The AI's disconnected body elements were used intentionally to enhance its personality as the game host, adding a distinctive character to the overall experience. This design choice reinforces the AI's role, making its presence more dynamic and engaging for the players.

This lineup showcases our four current characters, each with a unique occupation.

The Ink Bot serves as a writer, the Boom Box Bot takes on the role of a DJ, the Furnace Knight is, fittingly, a knight, and the Burger Bot is a chef. Each character brings its own personality and profession, adding depth to the gameplay experience.

3D Character development

The visual identity is designed to complement the Al-master, which is the MVP and innovative aspect of the game. It also aligns with the color schemes of the in-game characters. The user interface channels a 90s retro vibe, creating a fun and light-hearted atmosphere suitable for all ages, while the intermission frames adopt a futuristic look to highlight the game's innovative feature of multiple AI integrations.

The logo design aligns with the aesthetic of the AI-Game Master. For this voice-controlled game, we've replaced the circle in the letter 'O' with three imperfectly rounded circles and added a stalk, resembling a speech bubble. Additionally, we replaced the shapes in the letters 'R' and 'A' with star elements to match the AI-Game Master visual theme. The intricate designs add depth and dimension to the landing page, giving the impression that the AI-Game Master is presenting the game from above. We also incorporated a glitching effect on the surface layer to enhance the tech vibe.

We’ve designed high-fidelity, in-game interfaces for PC from scratch, featuring a duo screen setup for multiplayer, with shared and individual player screens integrated into Unity. The overall user flow progresses through phases: entering the game, character selection, starting the game, battle, intermission, another battle, and finally the end game. The color palette follows an analogous theme of blue, pink, and purple. In this setup, the top row of screens is displayed on a shared screen for all players, while the bottom row is shown on individual player screens, creating an organized and immersive multiplayer experience.

Team CanCan was invited to participate in Flats Fest 2024, alongside our client Buffalo Buffalo, in Vancouver, BC, to showcase our game, *Hollerena*. The game is set to be further developed with Buffalo Buffalo and will be included as one of the games in the *Doppelgänger* Party Pack for the company’s Series C round investment at the end of the year.

We had the distinct honor of being the only team from cohort C18 to showcase *Hollerena* at an AI sharing session with the Head of Research from Electronic Arts, Sweden, and 40 PhD students from the Wallenberg AI, Autonomous Systems, and Software Program. During the session, we explored the potential of GenAI use cases in the gaming industry, setting a new benchmark for innovation in student works at Centre for Digital Media.