yibo

Instead of using streamlit to create the website, i decided to change over into using HTML, the reasoning for this was because with streamlit not only was i heavily limited in terms of the UI design, also the camera feed was very glitchy with high latency and low resolution. I decided to use HTML instead, creating a basic template for what the website should look like after drawing out a rough design on paper. However i was having trouble making it look good and in the end, i decided that it woiuld be easier to recreate the website in figma (with the UI that i designed on paper) and add in the functionality from there.

ADDED:
-multi stance capability
-collected data for new stance and trained it
-added spoken aloud feedback.
-added multi stance capability to the ai sensei
I added multi stance capability for the ai model trainer, the data collector and the final ‘sensei’.
It works by having a separate column in the datasheet for each different stance. The ai model training works similarly, using a random forest classifier. it will split the data into 80% training and 20% testing. For each stance it calculates the median values (ideal position) and the standard deviation, which is the amount of acceptable variation.
For the decider, it starts by loading the trained model, karate_model.pkl then it tries to load the stance_endcoder.pkl and stance_ideals.pkl, if it finds them, then it goes to multi stance, if it dosent it will do to single stance. For each camera frame it calculates the features and turns it into the variables (the same as when the data was collected) Then it sends it to the ai, which applies stance specific rules, if stance_encoded is 1 then it is set as kiba dachi, 0 is gedan barai (it has the capability for more but ive only trained it on two different stances at the moment).
If the stance is judged as being incorrect, it will generate feedback, it does this by first weighting the importance of different variables, for example knee issues get 1.5 weight as it is deemed as more important and head issues get 0.7 weight as they are less important, then it sorts them by weighted deviation. the larges weighted deviation will generate specific feedback and speak it aloud to the user.
For the feedback system, ive tried to make it understand what each variable means, so that it can give direct feedback based on the largest deviation without me typing out what the common issues are and it selecting them, this has mixed results and i am going to try to implement this more effectively next time.

After the AI model is trained on what makes a good form and what is a imperfect form, i wrote a ‘decider’ script, that first calculates the angles and joints of the person (the same variables that the model was trained on), then “model.predict(current_data)”, triggers the custom trained model - karate_model.pkl, which contains hundreds of “decision trees” to check wether a stance is perfect or imperfect. If over 85% of the trees agree with the perfect stance, the stance will be judged to be good.

However, this isn’t enough, for a person to properly learn the stance, they need feedback, just like a real sensei would.

Once the program thinks that your stance is ‘incorrect’, it enters the coaching stage. It runs your data through a priority based system (the order was decided by the ai to be the most important variables in a good technique). It will show you the first mistake it finds at the top, once the most important condition is satisfied, it will move on down to the next most important until it deems your stance to be perfect.

Next i am planning to work on the ability to do this for multiple stances, as at the moment the model feedback will only have the correct order of importance for one stance. I am also looking to implement voice commands from the sensei so you don’t have to look at the screen every time.

Using scikit-learn, i trained.a random forest model on the dataset i made previously. By giving it data of ‘perfect’ and ‘imperfect’ technique, this allows the ai what good technique is. From the 127 rows of data i fed it, the ai focuses mostly on the right knee, characterising that as the most important value.
Currently, with my very limited and possibly incorrect dataset the ai has achieved a 96.15% accuracy, however this is only for testing purposes, when my project advances, i will provide the program with more accurate data.

I am also working on a function that will allow me to scrape images of good technique from the web, allowing me to train the model on data without having to collect it manually.

After it is trained, the model is saved as a pkl file in the project folder, ready to be accessed by the executer program that allows the project to tell you if your technique is good or not.

Currently, i am working on getting data to train my ai model on, staying with the basic stance of Gedan Barai, i asked my brother (who is good) and me (who is not as good) to perform perfect versions and inperfect versions. Using MediaPipe, the program finds landmark points on the body in 3d space (ish).

The script converts raw cooordinates into relative features. Using trigonometry to calculate the joint angles, the stance ratio with the distance between your feet and stance, center of gravity and more variables, i am able to form a dataset.

Then with manual labelling (by pressing ‘P’ or ‘I’) for perfect or inperfect, it writes a row to the dataset.
Now i will be working on using the dataset to train the AI to detect the quality of a stance. :)