SurvivorLM

Managed to deploy it using Google Cloud Platform. That took a lot of failed builds, and an entire evening but it’s done. I used AI to make the UI but to anyone voting/reviewing please note that the implementation and work actually consists of fine-tuning and trying to train the LLM from scratch. This was one hell of a journey but glad it happened. I have learnt so much about how these large language models work and what we can tweak ourselves to make them fit our needs.
Hope you enjoy the tips provided by SurvivorLM.

NOTE

(The website may take some time to load, please be patient since I don’t have the hosting resources to keep multiple instances running at once, hope you get a clear view of the project from the devlogs and not just the website)

Please note that because running this model is kinda expensive, I am using free compute as of now so I had to cap the daily messages to 10. I’m afraid I don’t have the resources that large companies do.

Thank you for following along. Hope you like the SurvivorLM :))

PS: Fixed the truncated output and added the design file to the root of the repository :)

You can download the model @ https://huggingface.co/Shreyy2305/survival-gguf

FINALYYYYYYY!!!! 😭😭😭😭. It works. Ahhhh. It took soo long and so many hours of debugging, the devlog time doesn’t reflect half of it. But yeayyyyyyy!!!! The gguf file complied successfully. I thought I lost hours of training when google colab downloaded all the safetensors sepearately in shards but thankfully it all recompiled. AHHHH. This was easily the hardest project ever.
Please note that this is majorly based on the theory and the actual functioning and fine-tuning of the LLM and not the UI. Now the only thing that’s left is actually deploying it. It works in the command line right now, but making it so that anyone can use it on their own website/cli is going to be difficult. Right now it only works by downloading the 3GB model and running it using llama.cpp but I’ll try to host it for people to test out. Can’t wait :)) Thank you for being a part of this!!

So I’ve decided to completely pivot, I am still training a model. But now this is gonna consist of fine tuning the gemma 4 model for survival. It includes image/video processing as well which would be a lot more helpful and training one from scratch for all of that would take up more resources and brain power that I have. The one I did train does work, only that is produces gibberish because I couldn’t gather enough data, and dont have enough compute to train everything. I’m sure the output with this fine tuning approach will be 1000% more valuable and faster. I’m sorry I can’t attach photos of the actual output just yet since the training process is so lengthy. Right now I’ve gathered about 2000 examples and am fine tuning the model on those. It still takes a lotttt of time time, but it works!! Hope I can turn this into an app or a cli that people can use since the Gemma 4: 2b model is relatively light-weight.

Seems like I’ve hit a wall, rather the wall has hit me. The model is producing incomprehensible text. I’ve come to understand that it’s a scaling problem. Right now, all the data together is ~30 MB. I need to gather wayy more data, more importantly gather clean data. Seeing the Loss drop did feel good, the model is doing the best it can with the amount of data it has but it doesnt have too much to work on. Regardless, I’m not abandoning this, I hope I can come up with and find more data for it to be comprehensible!! Onwards and Upwards we go!!!!

Ok so, I completed the tokenizer. And followed that up by writing and training the model which took a longg time to to train (devlog time does not count for time spent on training). The vocab size is 8000 words and the maximum sequence allowed is 512. All of this was done with PyTorch (No HuggingFace). And after all of that effort, the output I got is in the picture (it is just text that does not make sense)…
Really regret adding Shakespearean text in the training data but hey… you live and you learn!

Ok so I finally got the data together, it took a lot of time but it’s done. Now the actual work. begins. I got the transformers trained, and it is producing a sane amount of tokens and common words have stayed intact. For the data gathering, it was split into 3 parts, Survival Data (5.5MB), Warm-up Data (28.1MB)[English Language], Instruction Data (254KB)[QnA formatted Data]. And it was all combined into a tokenizer corpus (one large .txt file) and trained using sentence piece, next up is the architecture and the actual implementation! I’m really excited!!

To say finding and cleaning and processing data for training has been difficult would be an understatement. There’s always the fear of is this enough? Is this useful? Will this be enough data? And so far, it has been very messy but step by step I think I’m getting closer to actually useable data. However, the folder strucutre is still a mess. An LLM really needs a lot of data and for survival tips, I have been using Wikipedia and openly available survival manuals and PDFs. Data gathering is a really important step and I dont want to take it for granted so let’s hope all of this works!!