Robotic Shooter Game • From DIY to Unreal Engine 5 Masterpiece!

 

Previously, when we showcased our initial project at the national science fair, we presented "Armball Mania," a novel and intuitive sensor-controlled game. This project utilized an accelerometer connected to a compact Arduino Nano, with serial communication enabling the transfer of orientation data to an Unreal Engine game. The game's objective was to balance a ball on a football field and score goals by manipulating the field's orientation with the accelerometer mounted on a DIY glove worn by the user.
The project was a hit at the fair in Kolkata, drawing long lines of enthusiastic students and children eager to experience the game. Despite the success, we encountered significant challenges with the accelerometer. The sensor's sensitivity to minute hand movements caused jittery and unstable outputs, making the gameplay less smooth and precise than desired.
To overcome this, we decided to upgrade to a gyroscope. The gyroscope provided a remarkable improvement in accuracy, responsiveness, and overall smoothness. After some calibration and optimization, we were impressed by its performance, marking the beginning of the next chapter: "Armball Mania NEXT."

 

During the initial testing phase of "Armball Mania NEXT," we rigorously evaluated the new gyroscope's performance within the original game's framework. Our goal was to assess the gyroscope's responsiveness and output accuracy in a familiar environment. Using the trusty old Arduino Nano, we meticulously calibrated the sensor, ensuring it was fine-tuned for optimal performance.
To achieve this, we implemented auto-calibration directly into the codebase, allowing the device to adapt dynamically to various scenarios. We delved deep into the existing code, making necessary modifications to enhance responsiveness and smooth out any inconsistencies. Our testing process was comprehensive, involving numerous iterations to identify and resolve edge cases that could potentially disrupt gameplay.
As we meticulously tested the gyroscope, we discovered that its precision and stability far surpassed our expectations. Each tweak and adjustment brought us closer to a flawless experience. The gyroscope's capability to provide accurate, real-time feedback without lag or jitter was nothing short of revolutionary.
This phase of testing was not just about ironing out bugs; it was about refining every aspect of the system to deliver an unparalleled user experience. We were driven by the excitement of seeing our vision come to life and the mystery of discovering how far we could push the boundaries of what this technology could achieve. Every test brought us a step closer to creating a game that was more immersive, engaging, and intuitive than ever before.

However, using the Arduino Nano was not our ultimate goal. While it provided a smoother experience than before, we sought something even faster, smoother, and far more convenient. We needed to eliminate the cumbersome wires and the reliance on serial communication to transmit data. Enter the ESP32—a microcontroller with built-in Wi-Fi capabilities, enabling us to wirelessly transmit data.
Transitioning to the ESP32 required a complete overhaul of our code. The previous method of data transmission was neither optimized nor efficient. We decided to bundle the orientation data into a JSON string and transmit it over Wi-Fi. This change not only modernized our approach but also significantly improved performance. The ESP32 allowed us to achieve unprecedented accuracy and speed, making the entire system much more convenient. The data now streamed seamlessly over Wi-Fi, reducing latency and enhancing the overall user experience.
Getting everything up and running wasn't without its challenges. We spent considerable time ironing out bugs and optimizing the system to ensure the data was transmitted with blistering speed and minimal latency. This involved setting up socket communication using UDP in Unreal Engine 5—a task that was initially problematic but ultimately successful. We chose UDP because it is ideal for scenarios where speed is paramount, such as real-time gaming.

Previously, our system operated on a feedback loop where Unreal Engine would request data via serial communication, and our microcontroller would respond with the XYZ orientation data. With the new approach, our microcontroller continuously broadcasts the data using UDP, allowing Unreal Engine to pick it up as needed. This method proved to be far more efficient and faster, as it eliminated the need for repeated requests and responses. The data streamed effortlessly, ensuring a smooth and immersive experience for the player.
This transition to the ESP32 and UDP broadcasting marked a significant milestone in the development of "Armball Mania NEXT." The new system not only enhanced the game's performance but also provided a glimpse into the future of seamless, wireless interaction between hardware and software. Each breakthrough brought us closer to realizing our vision of an intuitive and engaging gaming experience.

 

Old Armball Communication

Old Armball Data Transmission

 

New Armball Communication

New Armball Data Transmission

After thorough testing, it was time to elevate the game. The previous version featured a single-level football field that players could balance, but we aspired to create something far superior. The old, dull grey graphics were uninspiring. We envisioned a vibrant, visually stunning experience, complete with cutting-edge graphics and engaging gameplay. So, we decided to rebuild everything from scratch using Unreal Engine 5, leveraging its advanced capabilities to create an entirely new environment.
Our first concept was to design a beautiful, soft, and cozy room—a pinkish, charming space that felt inviting and delightful. The centerpiece of this room would be a matching pink football field, unlike any field seen before. This field would feature dynamic elements like physics-based animations, hanging decorations, and sophisticated lighting. The entire room would be meticulously decorated to ensure it was not only fun to play in but also visually appealing.
We started by developing all the cute, soft models in 3D Studio Max. Every detail, from the furniture to the decorations, was crafted to fit our vision. Once the models were ready, we imported them into Unreal Engine 5, where we meticulously textured them, set up the lighting, and placed everything perfectly within the environment.
In this updated version of the football game, the room's theme was inspired by a locker room—a fitting choice given the football context. This locker room, however, was not the gritty, industrial type one might expect. Instead, it featured a cute window with potted plants, minimalistic and smooth lighting, and RTX (ray tracing) technology enabled to achieve stunning visual effects. The LED light strips hung from the window and ceiling, casting soft, colorful illuminations that interacted beautifully with the walls. A shelf in the distance, slightly blurred to keep the focus on the central football field, added depth to the room.
The floating football field in the middle of the room was a masterpiece. It was adorned with charming tiles and goalposts that floated with a slight, intentional latency, adding a layer of challenge and intrigue to the gameplay. The field was free of traditional markings, emphasizing its whimsical, fantastical nature. The football itself was redesigned to be cute and minimalistic, enhancing the overall aesthetic. When the ball dropped to the ground, it behaved with remarkable realism, its movements governed by advanced physics simulations.
This new environment was not only visually stunning but also deeply engaging. The combination of vibrant colors, smooth animations, and realistic physics created an immersive experience that was both fun and mesmerizing. Every element, from the floating goalposts to the soft lighting, worked together to transport players into a delightful, captivating world.

With the vibrant environment taking shape, our attention turned to refining the sensor and integration aspects of the project. Previously, our setup relied on two breadboards and a tangle of jumper cables to connect the sensors and the microcontroller. This prototypical approach was prone to loose connections and instability—issues that were unacceptable for a polished, production-ready system. We needed a more robust, reliable solution that minimized errors and could be presented confidently to both the public and potential partners.
To achieve this, we decided to move away from the breadboard prototype and create a custom, soldered circuit board. This would involve using single-thread, solid-core wires and header pins to connect the microcontroller and sensors securely. Header pins are particularly advantageous because they allow us to remove and replace the components as needed, ensuring flexibility and longevity of the setup. This way, the components wouldn't go to waste and could be easily reconfigured for future iterations or projects.

The plan was to solder header pins onto a breadboard, creating a DIY prototyping board tailored specifically for our project. This board would house the ESP32 microcontroller and the various sensors, securely connecting them with single-thread wires running underneath. This setup would not only be more stable but also allow us to detach and reattach the components effortlessly for further experimentation or updates.
I began by ordering all the necessary components: header pins, single-thread wires, a soldering iron, and solder. Once the materials arrived, the meticulous process of assembling the circuit board started. First, I soldered the header pins into place, ensuring they were securely attached to the breadboard. Next, I carefully connected the headers using single-thread wires, meticulously routing each connection to avoid any potential short circuits.
Soldering requires precision and attention to detail. Each connection had to be solid and stable, with no loose wires or cold solder joints that could lead to disconnections. Additionally, I had to ensure that no wires crossed paths or touched each other, which could cause short circuits and potentially damage the components. This painstaking process involved applying the right amount of solder to each joint and using the soldering iron to create strong, reliable connections.
As the circuit board took shape, it became clear that this approach would significantly enhance the project's reliability and functionality. The securely soldered connections provided a stable platform that was far less prone to the issues that plagued our initial breadboard prototype. Moreover, the use of header pins meant that the ESP32 and sensors could be easily swapped out or reconfigured, providing the flexibility needed for ongoing development and experimentation.
In the end, this custom circuit board was not only a significant technical improvement but also a crucial step towards making the project production-ready. It provided a solid foundation for integrating the sensors and microcontroller, ensuring that the system would perform reliably during presentations and in real-world applications. This meticulous attention to detail and commitment to quality underscored our dedication to creating a truly exceptional product.

After completing the custom circuit board and ensuring its reliability through rigorous testing, we were ready to move on to our next game concept. This new game promised to be both unique and engaging. The idea was to create a scenario where players would defend a kitchen from an invasion of bees. The kitchen would be meticulously designed with a realistic and inviting environment, complete with windows through which the bees would enter, buzzing around the room.
The objective of the game was to control a cannon-like gun, using our sensor technology to aim and shoot at the bees. However, aiming alone wasn't enough; we needed a mechanism for firing the cannon, which led us to integrate a joystick into our setup. This addition would allow players to not only control the cannon's aim with the gyroscope and ESP32 but also use the joystick to shoot at the bees, adding an extra layer of interactivity and fun.

Integrating the joystick involved some additional planning and modification to our existing setup. The joystick needed to communicate seamlessly with the ESP32, sending input data that would be interpreted by the game to control the firing mechanism. This meant updating our code to handle the new input and ensuring that the joystick's signals were accurately read and transmitted without any lag or errors.
We began by selecting a suitable joystick that was responsive and durable, capable of providing precise control for the game. Once we had the hardware, the next step was to incorporate it into our custom circuit board. This required soldering additional header pins for the joystick and carefully connecting it to the ESP32 with single-thread wires. Each connection was checked and double-checked to ensure there were no loose wires or faulty solder joints that could cause issues during gameplay.
With the joystick integrated, we turned our attention to the software side of things. The game's code needed to be updated to interpret the joystick's input, translating it into firing commands for the cannon. We wrote new functions to handle the joystick data, ensuring that pressing the joystick button would result in a smooth, immediate firing action in the game. This involved fine-tuning the sensitivity and responsiveness to ensure that the cannon would fire accurately and promptly when the player pressed the joystick.

As we tested the new setup, it quickly became apparent that the addition of the joystick significantly enhanced the gameplay experience. The combination of the gyroscope for aiming and the joystick for firing provided a more intuitive and engaging control scheme. Players could easily aim the cannon at the bees using natural hand movements and then press the joystick to shoot, making the game feel dynamic and immersive.
The kitchen environment, brought to life with detailed textures and realistic lighting in Unreal  Engine 5, added to the overall experience. Bees buzzed around the room, requiring players to stay alert and react quickly to defend the kitchen. The game not only challenged players' reflexes and coordination but also provided a visually appealing and entertaining setting.
This new game concept demonstrated the versatility and potential of our sensor-based control system. By integrating the joystick and refining our setup, we created a game that was both fun and technically impressive. It showcased our ability to innovate and push the boundaries of what was possible with our technology, setting the stage for even more exciting projects in the future.

 

To control the cannon in our new game, we needed a joystick for firing while retaining the gyroscope for orientation and movement control. This dual-control approach seemed ideal, allowing for a nuanced and responsive gameplay experience. The gyroscope would handle the aiming, and the joystick would handle firing, including controlling the rate of fire.
To integrate the joystick, I returned to the prototyping board. I inserted another header and carefully soldered and connected the necessary wires to the ESP32. This joystick was a simple module, but it was crucial for the control scheme we envisioned. With the hardware setup complete, I moved on to updating the software.
I rewrote portions of the code to include the joystick's inputs. This wasn't overly difficult, as the modular nature of our setup allowed for relatively straightforward modifications. The data format we were using, JSON, needed to be updated to include the joystick values. Now, in addition to the gyroscope's XYZ orientation data, the JSON string also contained the XY values from the joystick and a boolean indicating whether the joystick button was pressed.
The prototype board now featured the gyroscope, the ESP32, and the joystick, each with its own header, properly soldered and connected underneath with wires. This DIY circuit board was designed for flexibility: each component could be removed or replaced as needed, making it perfect for ongoing development and testing.
In terms of wiring, the video shows a single wire connected to the ESP32 microcontroller board. This wire is used solely for uploading code to the ESP32 and powering it during testing. In the final production stage, the device will be powered by a battery, eliminating the need for this wire. It's important to note that this wire does not transmit data to Unreal Engine; it is purely for power and code uploads during development.
This setup allowed us to seamlessly integrate the joystick with our existing sensor system, creating a robust and flexible control scheme for the game. The joystick's values were now part of the data stream sent to Unreal Engine, enabling precise control over the cannon's firing rate and direction. This integration not only enhanced the gameplay experience but also demonstrated the versatility and expandability of our custom sensor platform.

After finalizing the joystick integration and completing the DIY prototype circuit board, I shifted my focus back to the 3D development and modeling of the kitchen environment for our game. We meticulously added various kitchen elements such as cupboards, fridges, and shelves to create an authentic kitchen vibe. These details were crucial for immersing players in the game's environment.
Looking ahead, we planned to incorporate breakable objects like plates and utensils into the kitchen scene. When the ball collides with these objects during gameplay, they will shatter in real-time, enhancing the satisfaction and realism of the experience.
Next, we turned our attention to designing the bees themselves. The bees were designed to be cute yet believable, featuring black eyes, a textured body, and minimalistic wings. Animating the wings and coding their behavior to fly around the kitchen area naturally was a fun challenge. We ensured that the bees would move randomly with a slight delay, providing players with opportunities to aim and shoot.
At any given time, there were five bees flying around the kitchen. When a bee was hit, it would realistically fall and disappear, only to be replaced by a new bee flying in through the windows. This spawning mechanism added dynamism to the gameplay, ensuring that players always had targets to engage with.
Speaking of controls, the sensor's movements controlled the cannon's orientation: tilting left or right moved the gun horizontally, while tilting forward or backward adjusted its vertical position. We intentionally slowed down the vertical motion to add a layer of realism and challenge to aiming.
Meanwhile, the joystick's sensitivity played a crucial role in controlling the cannon's firing rate. Slight movements of the joystick resulted in balls with lower velocities, while pushing the joystick further increased the projectile speed. This sensitivity allowed players to fine-tune their shooting strategy based on the bees' movement patterns and the game's pacing.
Overall, integrating these elements—detailed kitchen environments, interactive bees with realistic physics, and responsive control schemes—created an engaging and immersive gameplay experience. Each component contributed to making our game not only enjoyable but also challenging, ensuring that players would have fun while honing their shooting skills in the whimsical kitchen setting.

As we approached the final stages of development, we refined the DIY prototype circuit board to ensure it was user-friendly and portable. Adding a small vibrator was the finishing touch, enhancing immersion through haptic feedback. Whenever a bee was hit or the cannon fired, the vibrator provided subtle vibrations, heightening the player's sense of engagement and realism.
In collaboration with Unreal Engine, signals were set up to trigger the vibrator accordingly, syncing seamlessly with gameplay events. This integration not only added tactile feedback but also enhanced the overall experience, making each interaction with the game environment feel more lifelike and responsive.
With these final adjustments completed, our innovative kitchen-themed game was ready to captivate players with its blend of detailed environments, interactive challenges, and immersive feedback mechanisms. Whether aiming at buzzing bees or admiring the intricacies of the kitchen setting, players were sure to enjoy a dynamic and engaging experience from start to finish.