Introduction
Video demonstration: https://www.youtube.com/watch?v=9Bk4u3yt4mI
GitHub repository: https://github.com/Gianou/Haaga-Helia-IoT-Experimental-Project
Our team consisted of three students: David, Isabelle, and Sara. We have diverse backgrounds and sets of skills within IT.
Sara is a second year Business Information Technology student from Haaga-Helia. Her major is in digital services and design, but she studies Front-end programming as her minor. She has had some prior experience in robotics, mostly in working with Arduino Uno, ESP32 and different kinds of sensors. She does not have a background in gaming, but she is familiar with the concept of arcade games.
David is a Double Degree exchange student from HES-SO Valais/Wallis. His major is in software development. He had no prior experience with embedded software development nor IoT, though he is experienced in coding and has made basic games from scratch in Java and JavaScript before.
Isabelle is a Double Degree exchange student from HES-SO Neuchâtel. Her major is in software development. She had no prior experience in IoT nor game development, but she is experienced in coding and loves gaming.
For this project, our goal was to create a game device that would run a game with fun controls using many sensors. The game that we produced is an infinite scroller where the player controls a spaceship and is trying to avoid asteroids. The controls change over the course of the game. At first the ship is controlled with a conventional joystick, then it is controlled with a proximity sensor and finally with a gyroscope!
The game and the device are inspired by retro arcade games and by the Nintendo Gameboy. We also thought that adding Ethernet communication to our game device would give it an interesting anachronistic twist. So, we added an online leaderboard.
Ideas
We started our ideation with 6-3-5 brainwriting session. The aim for this session was to create a large number of ideas that would later be discussed and developed further. After selecting ideas from our brainwriting.
After our brainwriting session we narrowed our choices down to two options: a smart alarm clock or an arcade style handheld gaming device with IoT functions. Lotus blossom was made for both ideas and based on discussions within the team the idea of creating an arcade style handheld gaming device with an online leaderboard was selected for this project.
What did you need to learn
Our project is ambitious so to better understand what we had to learn and what we already knew. So, we used the Lotus blossom to create a scope and learning objectives for the project.
After listing project requirements into the Lotus Blossom, we opened them up further in the subproject excel.
As the excel suggested, there were a lot of things we needed to learn. Regarding hardware we needed to figure out how to incorporate multiple sensors to the device. The first components we integrated to the game were two buttons and a Joystick. We also wanted to include other sensors, for example an ultrasonic distance sensor and a gyroscope, that required some research from us. We wanted to create a retro look and feel to the game, so we needed to figure out how to show pixel art and colours in the display. Regarding software, one of the biggest learning points was to figure out how to transition to VScode from the Arduino IDE. This was instrumental in facilitating collaborative development across multiple files, enhancing our workflow efficiency. Wrangling with C++ and its .cpp and .h files felt like building blocks in a Lego set – sometimes frustrating, but ultimately rewarding as we started to piece together our project’s backbone. We needed to learn more about the integration of Firebase, that provided a robust framework for cloud connectivity and data management. These learning experiences not only fortified our technical competencies but also underscored the imperative of adaptability and continuous learning in IoT innovation.
During our project development process, we convened a focused and dynamic Crazy 8’s innovation session among our team of three. This collaborative brainstorming session served as a catalyst for refining the specifics of the game to be featured on our handheld gaming device platform. Through structured ideation and rapid iteration, we explored various gameplay mechanics, visual aesthetics, and interactive features. Each member contributed unique perspectives and insights, fostering a rich exchange of ideas and sparking creative synergies. The session facilitated strategic decision-making, guiding us towards a cohesive and compelling vision for our game. Ultimately, this iterative process enabled us to distil complex concepts into actionable design elements. The decision of creating a 2D infinite scroller game with a space theme, where player must fly a spaceship and dodge asteroids during the game was made. Also, the idea of switching controls of the game instead of levelling up the difficulty in every level came up during the ideation. This way we were able to incorporate more sensors to the hardware.
The building phase was started immediately after the project scope and requirements were decided.
The building phase
Hardware
During the first classes in the 3D lab, we were able to hook up two buttons, a joystick, and a TFT LCD screen to our ESP32. We assembled two breadboards together and the device was already looking like a handheld gaming device.
After that, it was decided that we should concentrate on building a working code architecture, a base version of the game, and a connection to a database. And then once this would work, we could start adding more, less conventional, controls.
HC-SR04 proximity sensor is the first fun control that we added. The proximity sensor is composed of a speaker and a microphone. In our case, the player has to place their hand closer or farther from the sensor to move the ship.
GY-521 Accelerometer/Gyroscope came next, allowing the player to change the orientation of the gaming device to move the ship. This is similar to balancing a marble on a board.
Software
Making a game from scratch requires a rather complex code architecture and so we opted to use Object Oriented Programming (OOP) for this project. This allowed us to encapsulate logic inside of classes and make development easier. So instead of having a single Sketch (.ino) file, we have many C++ files where we define new classes. Each class is split into a header (.h) and a C++ file (.cpp). This is the same technique that is used to write libraries for Arduino and ESP32 development.
A video game works like a flipbook, each frame, the elements of the scene are redrawn in a slightly different spot and that creates an animation effect. In our game, the ship and the asteroids have x and y attributes that define where they are and therefore where they must be drawn on the screen. Each frame, those value are updated and that makes the ship, and the asteroids move.
Then, all that is required for the player to control the ship, is some sort of input. Making the ship move based on the joystick inputs was straightforward and so was using the sonar and the gyroscope. The challenge was to setup the phases in the game where the controls changes and the tutorial on how to use the new control is displayed.
The general architecture of the game engine is based on a composite design pattern where every element of the game extends the GameObject.h abstract class. The root node of the composite pattern is the GameEngine and this allows us to have a loop() function in the Sketch that looks like this:
All the code can be found on the project’s GitHub repository: https://github.com/Gianou/Haaga-Helia-IoT-Experimental-Project
Database
We decided to use the realtime Firebase database as it is free as long as it stays a small DB, which is perfect for our project. We were also familiar with it, as we used it in a React Native project.
We used the library Firebase_Arduino_Client_Library_for_ESP8266_and_ESP32 (URL: https://github.com/mobizt/Firebase-ESP-Client) which is well documented. We were easily able to link our project to the database.
Sending the data to the database was easy, we used a JSON object:
{
“pseudo”: “MyPseudo”,
“score”: “MyScore”
}
However, getting back the data was trickier. It was easy to get them, but harder to use them. Indeed, the data were sent back to us in String, so to be able to read all the pseudo and scores, we would have needed to create methods to deserialize the String. This would have been possible, but we decided to come up with another solution.
We created an index; the idea is that this index would work a bit like a primary key in the database. So, the index starts at 1 and each time a new score is added to the database, it is incremented by 1. Because of this, we do not need to send JSON object to database anymore. Here is a simplified version of the code for a better understanding:
config.api_key = _apiKey; //the api key of our database
config.database_url = _databaseUrl; //the URL of our database
FirebaseData _fbdo;
Firebase.begin(&config, &auth);
Firebase.RTDB.getInt(&_fbdo, “index”); //get the index from the database
int index = _fbdo.intData(); //set the integer index with the value of the
//index in the database
//sends the pseudo (username) to the database in the specified index
Firebase.RTDB.setString(&_fbdo, “score_board/pseudo/”+String(index), pseudo);
//sends the pseudo (username) to the database in the specified index
Firebase.RTDB.setInt(&_fbdo, “score_board/score/”+String(index), score);
//increment the index by 1 and sends the new value to the database
Firebase.RTDB.setInt(&_fbdo, “index/”, index + 1);
Here is what it looks like in the database:
Leaderboard
As the screen that we used for our project is quite small, we decided to just show the leaderboard, aka the top 10 scores. There were two possibilities for this:
Fetch the whole database, sort it and show on the screen the 10 first usernames and scores
Create a leaderboard in the database which contains the top 10 scores and is updated if needed
every time a score is added to the scoreboard in the database.
We opted for the second possibility; the first one is good if you only have a small amount of data. And, even if we know we’ll never have a huge database, we did not know how much time and memory it would use on the ESP32 to each time sort through all the data before displaying the leaderboard.
We created the leader_board in the database. We added 10 new entries in it (all with a score of 0) and added the entry min, which represent the lowest score of the leaderboard (the score of the entry number 10). The leader_board looks like this:
Each time a new score is added to the score board, the score is compared to min (the lowest score in the leaderboard), if it is greater or equals to it. If it is the case, then the new score value is compared to the leaderboard values (starting at the bottom) until it reaches the rank it should be at (when the new score is lower than one of the scores in the leaderboard). The leaderboard is then updated with the new score and the min is updated with the score of the 10th entry.
Here is a part of the code to explain a bit better how we update the leaderboard:
//check if the score has to be added to leader_board, if true, the data from
//the leader_board are fetched and put in a table
if (score >= min) {
int leader_board_score[10];
String leader_board_pseudo[10];
for (int i = 0; i < 10; i++) {
leader_board_score[i] = receiveDataScore(String(i+1));
leader_board_pseudo[i] = receiveDataUserName(String(i+1));
}
int rank = 0;
//loop to find which rank the score will be at
for (int i = 8; i >= 0; i–) {
if (score < leader_board_score[i]) {
rank = i+1;
break;
}
}
//loop to update the ranks of the leader_board
for (int i = 9; i > rank; i–) {
leader_board_score[i] = leader_board_score[i-1];
leader_board_pseudo[i] = leader_board_pseudo[i-1];
}
//add the new score and pseudo to the leader_board table
leader_board_score[rank] = score;
leader_board_pseudo[rank] = pseudo;
//loop to change the leader_board in the database
for (int i = 0; i < 10; i++) {
String path_leader = “leader_board/”;
String pathIndex_leader = path_leader += String(i+1);
String pathScore_leader = pathIndex_leader += “/score”;
String pathPseudo_leader = path_leader += “/pseudo”;
Firebase.RTDB.setInt(&_fbdo, pathScore_leader, leader_board_score[i]);
Firebase.RTDB.setString(&_fbdo, pathPseudo_leader,
leader_board_pseudo[i]);
}
//change the min value in the database
Firebase.RTDB.setInt(&_fbdo, “leader_board/min”, leader_board_score[9]);
}
Online leaderboard
Once a connection to the database was established an online scoreboard was built for the game. The page was created as a simple ReactJs + Vite application, with a retro look and feel.
Some pixel artwork was created for the site using PixilArt and a “Press Start 2P” Google font was used.
One requirement for the scoreboard was a connection to the Firebase Realtime Database. We’re essentially fetching data from a specific path in the database, which likely contains information about a leaderboard. Whenever there’s a change in the data at that path, our application receives updates in real-time.
This is facilitated by setting up a listener function that constantly monitors for changes. When new data arrives, our application updates its internal state accordingly. It’s important to clean up this listener when the component is no longer in use to prevent any unnecessary resource consumption. This ensures that our React component always stays in sync with the latest data in the database, providing users with up-to-date information without the need for manual refreshes. The scoreboard data is showcased in a simple Material UI table.
The outcome
Throughout our journey in developing our IoT handheld arcade gaming system, we’ve gained invaluable insights and experiences that have not only enhanced our technical skills but also enriched our collaborative dynamics and personal growth.
We obviously improved our understanding of IoT technology, but more importantly, we learned a valuable lesson: the importance of enjoyment in our work. By incorporating fun and creativity into our project, we not only maintained our motivation, but it also helped us overcome obstacles.
While we all were already tinkering at different levels, this course helped us see new possibilities. It reinforced our passion for tinkering and showed us how cool it can be when technology and creativity come together. Now, we’re eager to keep exploring and trying new things in this area.
This course inspired us with ideas for crafting useful items for our homes, trying to build them instead of just buying them and in doing so, allowing us to make thing that suits our exact needs.