DevAIl

DevAIl – AI Integration Prototype for Unreal Engine

Hi! I’m Brejack, an indie developer exploring new possibilities with AI in Unreal Engine. DevAIl is a technical prototype that demonstrates how AI can interact with an Unreal game environment.

How it works

DevAIl runs Unreal Engine alongside a local Python backend (llama_backend.exe). This backend hosts a large language model (LLM) and provides several endpoints, including a websocket API (chat-ws) that allows Unreal to communicate with the AI.

For now, interactions are limited to simple actions such as:

• Making the character jump

• Checking the current time

Each interaction has two steps:

1. Unreal sends a request to the AI via llama_backend.exe.

2. The AI “reads” the request, considers possible function calls, and responds with the action to perform in Unreal (e.g., talk or jump).

To enable AI-environment interaction, you must set the AI mode to “auto”. The “none” option disables AI interactions.

Model requirements

DevAIl uses the llama-cpp-python tool and requires a .gguf model placed in:

DevAIl\DevAIl\Binaries\Win64\server\llama_backend\llm 

The default model is:
Qwen3-4B-Instruct-2507-Q4_K_M
Download link: https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/tree/main

You can use other models by placing them in the same folder and specifying their name in the “Model” input.

Voice recognition (experimental)

DevAIl also integrates speech-to-text using whisper.cpp

.
To enable this feature, you must download a Whisper model and place it inside:

DevAIl\DevAIl\Content\Models 

The recommended model is:

• ggml-small.bin
  👉 Download link

⚠️ Important:

• The model must be named exactly ggml-small.bin, even if you choose another variant (e.g., medium, large).

• Without this file, voice recognition will not work.

How to use it

• Hold down the “1” button (top left of the keyboard) to start listening.

• Release the button to stop listening.

• Your voice input will be transcribed to text and sent to the AI, enabling natural voice interaction.

Performance notes

Currently, voice recognition runs entirely on the processor (CPU) .
This means that during audio interpretation the game may appear to freeze or stutter.
This behavior is expected and due to the lack of optimization — future versions may include GPU support or asynchronous processing.

What’s possible?

This is a prototype, so possibilities are open-ended. With AI + function calling + Python, you could:

• Create more complex interactions, e.g., AI assisting with puzzles or doors

• Generate text on demand

• Integrate external Python tools, such as email checks or data management

• Connect with live chats or other experimental features

Important notes

• DevAIl is experimental; it may break or not work on certain PCs.

• The AI integration is for testing purposes only and does not produce final game content.

• This is not a finished game—it’s a sandbox for testing AI integration in Unreal Engine.

###
Dev Notes

• The server runs with FastAPI.

• Automatic endpoint documentation is available at:
  👉 http://localhost:51000/docs

• Currently, only the REST endpoints are listed in the documentation.

• The WebSocket endpoint /api/chat-ws is not shown yet (this is a current limitation of FastAPI with WebSockets).

DevAIl\DevAIl\Binaries\Win64\server\llama_backend\llm 

DevAIl\DevAIl\Content\Models