HandsFree ✋

I finished with all the basic gestures for my app for now and I moved onto optimizing the processes when I came across a bug that took me very long to fix. The problem was my program was detecting the correct gesture but would not go through with the action. After like a whole hour of trying to debug I finally realized that there was trailing whitespace that made it so the conditions to active the actions would not be triggered 😑. But good news is on the optimization standpoint I lowered the resolution of the model so it doesn’t have to render as much and I also moved the camera reading to another thread to unblock the main thread as well as I finally don’t look at the gesture every frame but rather every other frame which greatly improves the speed of the program. Overall great debugging and optimization progress but the model still struggles with recognizing some edge cases, I plan to focus on creating some kind of application so others can easily use the code by the next log.

Great news! I improved the accuracy of the training model to 98.06% which is up from the 96% and it works much better now because I added an extra parameter that calculates the distance between the thumb and the pointer finger which is very distinct for each of my gestures. I also was able to fix the mirroring issue and now it works in the correct direction so it is actually functional and I can navigate and click around fine. I still need to finish implementing the rest of my gestures besides pinch and cursor but all the infrastructure is there to do so. I hope to push the project even farther in the next log

I worked on collecting data for the model by recording consecutive frames of a given gesture and correctly labeling it (e.g “Thumbs Up”, [data]) then I passed almost a thousand lines of this data (each with 63 columns) into a RandomForest model and after some trial and error I came out with ~96 accuracy but I found that my model was mixing up “Pinch” & “Point” gestures so I need to fix that in the next log. Finally for this log I can actually control my mouse now through the pyautogui library but I still need to fix the fact that the cursor goes in the opposite direction than intuitive.

Using MediaPipe I was able to relatively quickly get the joint of my hand to show up in realtime without having to train my own model from scratch which was very rewarding. In this time I laid the infrastructure for my project and got the program ready for the next step which would be to collect position data for each joint given a label (e.g. “Peace Sign”) and then train my own model using that data (hopefully by the next log).