yibo

There was a few issues with the web scraping not working and then the adding your face. The api was returning HTML images, not JSON which really messed with the program. Also the worker kept timing out so i moved it to a backround thread so that the request returns immediately. There were some issues with the api key being blocked due to too many requests, so i decreased the amount of tries and increased the sleep delays (for faster performance as it should be fine with two calls in a shorter time). Another issue was that the images kep getting rejected as the program kept thinking there were no faces when there was, so the detection threshold got lowered from 0.6-0.5, it should still be detecting faces but no longer with such strict requirements. Also i increased the images from 5 - 10 for duck duck go search to have a better chance of finding a proper image.

I finished hosting the project using a railway free hosting (this took AGESS). Then i decided to add animations for the transition because its taking longer than expected for the program to take the photo and analyse it on railway . I was first having lots of trouble getting the photos uploaded properly to the website, i thought that i couldn’t just upload the images so every time the website restarted it had to re scrape the web for photos and the api key kept getting rate limited but i just uploaded all the photos to a folder in the respiratory and the program encoded all of them into the data at the start of when i build the website. Also i added a feature in which you can better see your data, so you can see how similar each of your individual features are. Also i ran out of railway free hosting credits so i had to make a new GitHub account and host it from there.

Ive almost finished the whole project, im just working on properly hosting it now and i fixed the issue when the script on the website dosent properly load, the issue now is i cannot get it to host on any website. Initially i tried to host it with github pages, but due to the fact that i dont just want a static page, i want it to have functionality i cannot just host it there. If i were to use my original streamilt code i could just host it on streamlit cloud, so i tried to make a version of the UI with streamilt but not only did i run into the previous problem of it being very slow and not really working it also looks no where near as good in terms of the UI. Currently ive tried using Render, Fly.io, Koyeb and Cloudfare, but im running into a lot of trouble. (i dont think its the platforms fault im just not very good at hosting things). At the moment i am using huggingface spaces as people online have said its the easiest way to do this, i am running into a job timeout error, i think its because its trying to install way to many packages but im not too sure. The screenshot attatched is my current progress with huggingface spaces, i think im very close and i want to try to ship this project by the end of this week.

I havent had the opportunity to work on this much, but ive tried to fix up the inference on the LSTM, it is now properly integrated and finally dosent show up as everything being perfect. Currently there is no feedback for the model in place, but i was able to make quite a few fixed and i think ready to progress to the next major step.
I noticed that a lot of the time during training and actual inference stages the wireframe for my body was getting quite tangled and led to some odd data, which may have thrown the model off on inference. Additionally, it seemed to be overfitting on the small data set, to fix this ive added a temperature parameter and added dropout (03) layers and L2 regularisation to prevent the model from overfitting.

Due to the constraints in the camera tracking, causing many inconsistencies ive decided to use body mounted sensors to get better and more accurate readings. Ive also been toying with the idea of mounting the whole system on a drivechain and making it move around. I want to make it navigate autonomously and avoid obstacles, whilst maintaining a constant distance from the person. To do this i plan on implementing a swerve drive with jetson nano. I might start working on the swerve drive when i am looking for the right sensors, designing casing for them and waiting for them to arrive.

I am working on the inference for my project, using the LSTM model. Currently it will decide what action it is through the model, and the percentage of “correctness”. I am trying to make it work at the moment but it dosent seem to be working very well. at the moment i am generating feedback using simple if statements to see if the angles of joints are correct, similar to what i was doing before. However, i am having trouble seeing if it is my inference script that is wrong or the model thats not working, as it has trouble determining if the action im doing is really a punch or just idle. It will often misinterpret me sitting still testing as a punch that is 100 correct. Some feedback that i got recommended me to add a 30 frame buffer at the start but that didnt really work, also instead of automatically detecting movement i made it simpler for me by making it press spacebar to detect but that didnt really work either. For now i want to focus on testing wether its the models fault or mine so i will strip it down to its basic, try a different approach for the inference script and just check if the model works at all, if not i will have to re-code it.

I finished writing the script for training my multi stance moving model, currently when you finish training the data using the data trainer, it formats the data into folders of each move (one for each time you trained it).
FIrstly, the training script opens up the DATA_PATH folder, it loads all the sequences from each indivudial action folder, for each sequence there are 30 frames and 132 features, it will assign each action a number to each action. Then it used 80% of the data to trian the model (it uses the adam optimiser, categorical crossentrophy loss and accuracy as the metric), i am training a LSTM model. The remaining 20% of data is used for testing, it tests for 50 epochs and each epoch it will shuffle the data. Then it saves the model and label mapping, ready to use.

Now ive decided to implement multi - movement stances for my project, originally i tried to use a adapted version of my original mediapipe code to collect data for multiple frames of the stance, however it was having trouble knowing when to stop, and the format the data was collected in wouldn’t work properly if i was to train this. I researched methods of collecting moving data and training them and i decided to use a LSTM (long short term memory) model. Additionally, for the training script, i decided to make it just record for 30 frames, this way i didn’t have to worry about asking someone to stop or start the training bit and i didn’t have to try to code a auto-stop feature. For each frame, the script uses mediapipe to record 33 landmarks, each with 4 values, meaning 132 values per frame saved in a .npy file (in a folder for each individual punch) this way it is a lot easier for the model to open than a csv.
The landmarks are all normalised, so it will work better regardless of your height or position from the camera. Currently i am working on finishing the script for training the model and getting around 100 samples of punches to train the model on.

Using figma make, i converted my rough sketches and plans into a template for the website, then i adapted that into html and css for my website. I used a python flask backend to add functionality, whilst the frontend was made with html and css.
At the moment the main part of the program works, when you press the take photo button it will successfully return your celebrity match, however at the moment im not too sure with the ability to add new celebrities on the website using the web scraping program i made previously. Additionally, i will work on implementing the share button to work by either sending it to slack or to other social media like whatsapp.

Im quite happy at the moment with the aesthetics of my program, however i will be more than happy to get feedback, i went with a retro movie style but i am considering making a version when you can just match with fellow hack club members.

Instead of using streamlit to create the website, i decided to change over into using HTML, the reasoning for this was because with streamlit not only was i heavily limited in terms of the UI design, also the camera feed was very glitchy with high latency and low resolution. I decided to use HTML instead, creating a basic template for what the website should look like after drawing out a rough design on paper. However i was having trouble making it look good and in the end, i decided that it woiuld be easier to recreate the website in figma (with the UI that i designed on paper) and add in the functionality from there.

ADDED:
-multi stance capability
-collected data for new stance and trained it
-added spoken aloud feedback.
-added multi stance capability to the ai sensei
I added multi stance capability for the ai model trainer, the data collector and the final ‘sensei’.
It works by having a separate column in the datasheet for each different stance. The ai model training works similarly, using a random forest classifier. it will split the data into 80% training and 20% testing. For each stance it calculates the median values (ideal position) and the standard deviation, which is the amount of acceptable variation.
For the decider, it starts by loading the trained model, karate_model.pkl then it tries to load the stance_endcoder.pkl and stance_ideals.pkl, if it finds them, then it goes to multi stance, if it dosent it will do to single stance. For each camera frame it calculates the features and turns it into the variables (the same as when the data was collected) Then it sends it to the ai, which applies stance specific rules, if stance_encoded is 1 then it is set as kiba dachi, 0 is gedan barai (it has the capability for more but ive only trained it on two different stances at the moment).
If the stance is judged as being incorrect, it will generate feedback, it does this by first weighting the importance of different variables, for example knee issues get 1.5 weight as it is deemed as more important and head issues get 0.7 weight as they are less important, then it sorts them by weighted deviation. the larges weighted deviation will generate specific feedback and speak it aloud to the user.
For the feedback system, ive tried to make it understand what each variable means, so that it can give direct feedback based on the largest deviation without me typing out what the common issues are and it selecting them, this has mixed results and i am going to try to implement this more effectively next time.

Ive been working on turning my program into a website, using streamlit i made a basic website template. ‘st.camera_input’ (a built in streamlit function), rather than complex webrtc code to create a simple button that takes a photo. Then the program processes the image and returns the celebrity face that looks the most like you. The reason i didn’t opt for a live camera feed was because i was having troubles with the feed glitching and not working properly on the website, i will try to fix this for the next devlog! :)

After the AI model is trained on what makes a good form and what is a imperfect form, i wrote a ‘decider’ script, that first calculates the angles and joints of the person (the same variables that the model was trained on), then “model.predict(current_data)”, triggers the custom trained model - karate_model.pkl, which contains hundreds of “decision trees” to check wether a stance is perfect or imperfect. If over 85% of the trees agree with the perfect stance, the stance will be judged to be good.

However, this isn’t enough, for a person to properly learn the stance, they need feedback, just like a real sensei would.

Once the program thinks that your stance is ‘incorrect’, it enters the coaching stage. It runs your data through a priority based system (the order was decided by the ai to be the most important variables in a good technique). It will show you the first mistake it finds at the top, once the most important condition is satisfied, it will move on down to the next most important until it deems your stance to be perfect.

Next i am planning to work on the ability to do this for multiple stances, as at the moment the model feedback will only have the correct order of importance for one stance. I am also looking to implement voice commands from the sensei so you don’t have to look at the screen every time.

I decided that i wanted to increase the amount of celebrities listed, but im lazy. ;(
I built a web scraping program that uses a variety of free apis, all you have to do is give it a list of celebrities and it will find the photos and leave them in the folder. This process was very fustrating, originally, i planned on using duck duck go api, however i kept getting ratelimited and i fear my ip adress is now blocked. I moved towards using wikipedia api, however not all the people had wikipedia pictures that were easy to find (they need to have a thumbnail property image designated). to improve with the quality of wikipedia, the program can now ask for search suggestions, this means if a direct match fails, the program can ask for the next best match.
FInally, to make the scraper ensure that it works, i added functionality for TMDB, the movie database, it is a professional database for actors and singers, is significantly faster than scraping and almost never blocks you.

The program first goes through TMDB, then wikipedia, finally duck duck go.

Also, for faster opening times, the program will do through the .pickle file, detect if there are any new photos int here, and automatically decode it into the .pickle file before startup. This process only needs to be done once per photo so it makes the whole process a lot faster.

Uhhhh. so basically i forgot to post a devlog previously and i just worked on this new bit without posting my first devlog so this times wrong. I spent around 5 hours on this bit and the rest was on the first devlog.

Using the face-recognition library - built on dilb, the program finds 128 unique points on a face, making a ‘face print’, which is a signature unique to each person. Then, the averaging logic takes multiple photos of celebrities, takes the ‘face prints’ of all of those and calculates the average, for better accuracy. (this was implemented later on).
One of the issues i faced initially was that it took a long time to calculate the math for these faces initially, to fix this the first time the program encounters new files in the library, it saves the finished math into a .pickle file, so next time you open up the app it ‘reads’ the math, instead of re-calculating each time.
Then, for the real time comparison (the cool visuals), it turns your face into a ‘face print’, then using media pipe to map points on your face. It calculates the Euclidean distance between your face and all the celebrities in my database, using media pipes pipe mesh, it turns redder or greener according to how similar (or not) you are to the celebrity.

Using scikit-learn, i trained.a random forest model on the dataset i made previously. By giving it data of ‘perfect’ and ‘imperfect’ technique, this allows the ai what good technique is. From the 127 rows of data i fed it, the ai focuses mostly on the right knee, characterising that as the most important value.
Currently, with my very limited and possibly incorrect dataset the ai has achieved a 96.15% accuracy, however this is only for testing purposes, when my project advances, i will provide the program with more accurate data.

I am also working on a function that will allow me to scrape images of good technique from the web, allowing me to train the model on data without having to collect it manually.

After it is trained, the model is saved as a pkl file in the project folder, ready to be accessed by the executer program that allows the project to tell you if your technique is good or not.

Currently, i am working on getting data to train my ai model on, staying with the basic stance of Gedan Barai, i asked my brother (who is good) and me (who is not as good) to perform perfect versions and inperfect versions. Using MediaPipe, the program finds landmark points on the body in 3d space (ish).

The script converts raw cooordinates into relative features. Using trigonometry to calculate the joint angles, the stance ratio with the distance between your feet and stance, center of gravity and more variables, i am able to form a dataset.

Then with manual labelling (by pressing ‘P’ or ‘I’) for perfect or inperfect, it writes a row to the dataset.
Now i will be working on using the dataset to train the AI to detect the quality of a stance. :)

Using mediapipe, i made a program that finds the position of body joints, joining them up to create a pipe frame of the limbs. From this, i calculated the angles of each arm and leg, this way i was able to detect the position of the body and correct the stance. i decided on trying to first code a still position, the Gedan Barai (downward block). I chose this because it is relatively simple stance and it had leg and arm positions that could be detected mostly through arm and leg angles. I improved on my original detection by using 3d vectors to try to more accurately find the positioning of limbs (results varied). Finally, i added a simple system, in which the colour of the joints shown turn from red to green, in response to the correctness of the positioning and a timing system, that allows you to see how long you hold your correct stance in.

Tomorrow i hope to add more stances, and work on the accuracy of the data as well as adding new variables, such as foot positioning and hip position. :)
I spent a lot of time researching different ways to make this project work, i will try using YOLO for positioning too.