StupidCookGPT

Final devlog before ship!
Not much to say, just tidied stuff up. But gosh, it took hella long.

Did a bit of documentation. Removed some sus lines and redundancies. Overall, minor updates. I have yet to upload binaries. Just need to publish nanogpt training .ipynb and those binaries, think I’ll be done.

Also, I attached two images showing the specific flowchart of the mini/nano/nano-M GPT logic; first is miniGPT, nanoGPT, second is nano-MGPT. Should be self explanatory if you’ve read my other devlogs, but I added it (as a sneak peek from one of my newer draft papers on prompt-injection based memory retention techniques in training GPTs) so it makes more sense. <3

Added a logo. Just pixelated the cook emoji but I like it so that’s that. I also uploaded everything to github. Memory model is also finished! The key logic is like this.

You have a database in json or SQLITE format, that you can add and store to, and it is formatted as an rpg-like “inventory” system. I.e. it stores something like {‘apple’: 2}. Then the context window: key-value pairs for general facts. And there’s the history, which is a log of events/actions.

Before the model generates a single token, the MemoryAwareGenerator performs a memory injection. It converts the current MemoryState into a string: i.e. “Inventory: apple (2). Context: location is kitchen. —”), prefixing into the user’s prompt. The model receives the augmented prompt as if it were part of the conversation history.

The most unique part I call it the GenParUp loop (different from GPU) wherein the model is prompted to output special text commands like set_context(key, val) or add_item(name, qty). It is parsed by CommandParser using regular expressions to find these commands. Then, it removes them from the output so the user doesn’t see the code. Because the ‘memory’ has changed, the generator often restarts the generation process with the new, updated memory injected into the prompt to ensure consistency.

Sorry, I don’t have a MWE trained yet since it’s too big for my computer. I’ll see if I can get a MWE working on colab, but I guess this is enough for my ship!

My final task would be to zip all the models up so I can release it as binaries.

Made a new architecture type. I found out that with small locally trained models (on a consumer laptop, i.e. a 2021 Macbook), models that are trained within a reasonable time frame, (i.e. within 14 hours) often hallucinate a lot. Notably for this project, one has a basic structure:

• First, the ingredients
• Building recursively on the ingredients, step 1
• Building recursively on EVERYTHING before, step 2

And so on. But this would mean a complete complex restructuring of my architecture, which is too difficult. I tried training the same stuff on the original distilgpt2 model on colab, and the effects were the same, so my training dataset was either too small or not ideal enough, or my training was too short. Anyhow all of these are inconvenient, so I added a structure on top of the existing nanoGPT framework: a Text-based memory system. I will provide more details after a successful training implementation and upon I have trained a successful model. The discord bot works tho lol.

Began work on an encoder from scratch. I just wanted to get rid of the GPT-2 encoding and use my own but so far though it seems easy it is pretty hard. The encoding block is not just one component, so I realised, but a flow chart in this sort:
tokens -> embeddings -> [attention + MLP] × N -> hidden states
But so far I got embeddings done (it was easy, use a simple split into chars and assn based on unicode). The attention, MLP and N matrix calculations though are pretty challenging to formalise. Especially special chars such as space, though I have it in my vocab it cannot be identified right now.

Now online at https://stupidcookgpt.vercel.app!
Also made a simple discord bot. Trynna get a MWE up ASAP.

Finally, the website is complete. Not much to add on top of the previous devlog, but I was able to properly format the output by calling .replace(/\n/g, "<br>") on the recipe since divs don’t properly format \ns. I also added the /api_query page for documentation, and made a small demo video. Hunted round for some bugs, so here it is. I will upload it to github in the examples/website/ folder.

Did some major work on the online website. This part I did the quickest out of all other parts since at least I can claim that I am quite acquainted with basic web dev, unlike LLM dev (that’s why I took so long in actually producing the model lol!)

Used quite a classic but simple setup. A bunch of divs vertically with their own classes. Of course, the basic “sections” (all of them have unique classnames, though) look the same with same padding, margin, etc. and I centered on this colour scheme: #f9d5d3, #493535, #f4e5dd for a nice mild look that’s more interesting than black grey and white. I also made a loading css animation so the users don’t panic when their request takes a bit. Now I plan to work on formatting the recipe, documenting the API and committing everything to github.

Made it public on huggingface!
After a lot of work, I publicised the first 124M model on Huggingface spaces. You should definitely have a try, I made it my demo!
Also I resolved a loading bug with miniGPT. It occurred that the Huggingface gradio Space loading was a bit different from transformers, so I had to write a bit of “carry-on” code to make the configuration in miniGPT understand the huggingface parameters so the model will actually load. For example, in my OG architecture using GPT-2 encoding, my saved models were very simple in config.json but the update needed a lot more information on the parameters, introducing some random stuff like “_num_labels” and whatnot, but got it all resolved.

I am planning to make it into an API to actually make it a usable chat interface and publish it using vercel or something of the sort.

Major Update:
NanoGPT Architecture prototype complete, known as miniGPT-0.2r2 featuring the new sdsGPT architecture (super duper small GPT), a production-quality GPT-style transformer language model with several optimisations in less than 700 lines. Wherein:

Core Architecture:
Decoder-only transformer that includes the regular:
Embeddings of text input (from the training dataset) to a d-dimensional vector field wherein d is an adjustable hyperparameter depending on how intricately you want your semantic quality of the training data to be, with optional rotary embeddings (RoPE) for better positional information.
Transformer Blocks: Stack of identical layers (default 12), each containing
Multi-head self-attention with causal masking (can’t attend to future tokens)
Feed-forward MLP (4x expansion in hidden dimension)
Layer normalisation and residual connections
Language Head: Linear projection from embedding dim to vocab size

Optimisations for efficiency are also present to live up to our name of NanoGPT as an computing-friendly alternative to bulkier GPT models. Some key optimisations include:

Gradient Checkpointing to reduce memory usage by recomputation rather than storage.
Torch Compile JIT compiling the model for faster execution.
Scaled Residual Projections to initialise output projections inversely to layer count to stabilise deep networks.
Flash Attention by using optimised scaled_dot_product_attention rather than the old flash attention for faster/more memory-efficient attention.
A prototypical repetition penalty multiplier to prevent hallucinations using superpositioned entropy gaussians.

The model itself can be trained based on mixed precision training that accelerates it more for CUDA devices. Training is also checkpointed to allow saving during longer training sessions.

Using NanoGPT I trained a 54 M model and a 128 M model on corbt/all-recipes.

Trained two models: distilGPT-2 (81.9 M) and a 124 M parameter miniGPT model (1-model124M). Began rough work on documenting them, and I also improved the miniGPT architecture from miniGPT-0.1 to miniGPT-0.2, which I am going to train the next ones on.

MiniGPT-0.2 is a major production variant of miniGPT-0.1 featuring stabilised huggingface usage and execution and whatnot, with minor improvements in architecture: top-k sampling, anti repetition bias and weight/model dual loading features.

I plan to make a UI to upload the better ones on.

Trained a couple different different models using diff batch sizes, comparing their coherency in output.

I tried using a locally built GPT (using simple Gen Transformer model), turned out pretty smooth. Pretty happy with the result. Made a ~50 M Model and trained it on my Mac in 15 minutes, so I will probably expand training to 2 epochs and extend the training dataset from 20000 to ~30000 recipes, while adding more embeddings and layers (640, 12) to extend it to a ~100 M model with a training time of up to 1 hour. Turns out a M1 can do alright

Ran it locally with the Colab-trained model, I can publish it at least

Created the model in Colabfold, complementing my existing efforts. Access to a T4 was nice… until it ran out. Now I have a 300 MB pth file that is probably corrupted (I probably truncated the file somewhere because my internet is slow). Next time I’ll probably train on a local M1 Pro. Anyhow I probably can deploy a working model tomorrow.