ryan

Trained YOLO11n on two different food datasets on Kaggle. Both models could locate objects on-device but couldn’t classify them correctly — always predicted the last class index regardless of input
Debugged for days. Tried every combination of TFLite export settings (float16/float32, with/without NMS, different input normalizations). None worked. YOLO’s classification head breaks during TFLite conversion. Found a GitHub issue confirming this is a known problem with no fix

Gave up on on-device ML. Switched to Gemini 2.5 Flash API -> send photo, get back ingredient labels + bounding box coordinates. Took 30 minutes to implement vs days of YOLO debugging (are we serious)

I fixed the bounding box logic. But the model I trained is pretty bad and innacurate (see attached photo, it detected an apple as a lemon) so i’m currently training another model in hopes that that one is more accurate.

I also still have to:

• make it so once the user presses the “add to pantry” button, it’ll actually add to the db
• fix some race condition errors with the model loading then running inference with the capture button

But after fixing those and some polishing, i’m basically done the MVP for this project!

I’ve made a ton of small UI fixes and changes for a better user experience. for example, adding scrolling to the ingredients list, loading screens for everything. i’ve also started on implementing the ml model to detect ingredients. I’ve added the camera, trained an ml model, did a ton of debugging, and now have code that runs inference on it to detect ingredients (this works), but right now the bounding box logic i’m trying to implement isn’t working so i’m debugging that right now.

The results screen now uses the ingredients inside the ingredients db instead of being hardcoded. And i also fixed a bug with the options on an individual recipe screen not working

added watermelondb, which is a database which uses sqlite but has lazy loading. I first had to include it in my config files, then i created the Ingredient model, table schema, and functions to add and remove ingredients from the db. And finally, i added a “+” button on the main page to allow users to add and remove ingredients from the db.

added equipment, time, and difficulty to the actual ui. Also wrote a little script to find what ingredients the recipe uses that the user has. Also fixed some ts errors.

I added cook time, equipment and difficulty to the database for a better user experience using regexs to scan the directions and ingredients for key words. I first created the test_parsers.py file to test whether my parsers would work, then in enrich_recipes.py, I used the parsers to load new columns in the db. Then I modified the sqlalchemy schemas and pydantic models in the backend to account for the new columns

made it so the frontend will actually display the recipe info based on the ingredients given (those are hard coded for now) on the results page. And on the individual recipes page, it now displays the recipes info like directions, ingredients.

so i didn’t find out about Flavortown until I was already somewhat progressed in this project, hence the 26h devlog. I did create a log of my work though so here it is all at once:

March 14, 2026

• Initialized repo with project scaffolding: .gitignore, LICENSE (MIT), README, placeholder CI workflow
• Added initial Expo frontend (app.json) and backend requirements.txt
• Included a TFLite model binary for early camera/ML experimentation
• Cleaned up tracking — removed package.json and package-lock.json from version control

March 20, 2026

• Built the data pipeline for loading recipe embeddings into PostgreSQL
• Created init.sql to set up the recipes table with pgvector (vector(768)) column
• Wrote load_recipes.py — reads RecipeNLG CSV + a numpy memmap of pre-computed embeddings, batch-inserts into DB (512 rows/batch)
• Added test_query.py to verify cosine similarity search against the DB using sentence-transformers/all-mpnet-base-v2
• Removed the TFLite binary and old plan.txt from repo

March 26, 2026

• Scaffolded the full backend package structure: api/, repositories/, services/, schemas/, tests/
• Defined SQLAlchemy ORM model (Recipe) mapping to the recipes table with pgvector Vector(768)
• Created Pydantic schemas: IngredientsRequest (input) and RecipeResponse (output with similarity score)
• Set up FastAPI app in main.py with CORS middleware and lifespan-based model loading (SentenceTransformer)
• Added config.py using pydantic-settings to pull DB and model config from .env
• Stubbed out the POST /recipe route, repo, and service layer (empty files)

March 27, 2026

• Implemented the core querying logic end-to-end
• dependencies.py — yield-based SQLAlchemy session factory for FastAPI’s Depends()
• recipe_repo.py — pgvector cosine distance query, returns top-k results with similarity scores
• recommendation.py — service layer that encodes ingredient text into a vector and passes it to the repo

March 28, 2026

• Wired everything together to complete the backend API
• Connected the POST /recipe router to the service layer with dependency injection (DB session + ML model)
• Extracted ml_model dict into a separate ml.py module to resolve circular import between main.py and the router
• Added error handling (HTTPException 404) on the recipe endpoint

April 3, 2026

• Built the frontend UI shell in React Native with Expo Router and NativeWind (Tailwind v3)
• Set up file-based routing: (tabs) layout with Kitchen Hub, Scan, and Results screens + recipe/[id].tsx
• Created reusable components: RecipeCard (results list item) and IngredientItem
• Configured Babel, Metro, Tailwind, TypeScript, ESLint, and NativeWind
• Placeholder recipe/[id].tsx page (pending API integration)

April 4, 2026

• Added GET /recipe/{id} backend endpoint for fetching a single recipe by ID
• New repo function get_from_id() and Pydantic schema IdResponse (title, ingredients, directions)
• Built out the recipe/[id].tsx detail screen with collapsible sections: EquipmentOption, IngredientsOption, InstructionOption

April 5, 2026

• Refactored frontend — extracted shared Recipe interface into src/types/index.ts
• Created src/services/api.ts with axios HTTP client pointing to ngrok tunnel
• Two API functions: getRecipe(ingredients[]) → POST /recipe and getById(id) → GET /recipe/{id}

added the faq and how-it-works files. i forgot to do this before haha

So I didn’t hear about Flavortown until I finished this project (except for some small stuff), hence the 35h devlog lmao. but I did keep a log of my progress, so here it is all at once:

March 1, 2026
Initialized repository with MIT license and .gitignore

March 3, 2026
Built out HTML shell (index, FAQ, How It Works pages)
Created initial CSS styling (dark theme, cyan accents, Inter font)
Connected frontend to backend with FastAPI endpoint and fetch API
Added Pydantic response schema (UploadResponse)

March 5, 2026
Mounted /static directory so the browser can access server-side images via URL
Implemented first version of copy-move forgery detection using OpenCV SIFT + FLANN matching

March 6, 2026
Implemented splicing detection model (U-Net with ResNet34 encoder via segmentation-models-pytorch)
Tested end-to-end and confirmed working inference

March 7, 2026
Added animated loading screen with CSS keyframe animations (scan line, pulsing circles, backdrop blur)

March 8, 2026
Built results page skeleton (confidence bar, detected manipulations section, view toggle buttons)
Tuned copy-move detection: reduced sensitivity to lower false positives
Optimized noise detection model post-processing

March 9, 2026
Further tuned SIFT copy-move detection thresholds
Completed results page UI (color-coded scores, per-engine breakdowns, detected/not detected labels)
Built canvas overlay system: red semi-transparent mask for splicing, green bounding boxes for copy-move
Added background cleanup task that deletes uploaded files every 10 minutes
MVP complete

March 10, 2026
Deployed: frontend to Netlify, backend to HuggingFace Spaces (Docker)
Set up CORS middleware for cross-origin requests
Fixed static directory not being found in container
Set up GitHub Actions workflow for auto-sync to HuggingFace
Removed model weights from git tracking (downloaded at runtime instead)
Cleaned up repo structure (removed redundant hf-space folder, converted submodule to normal folder)
Switched to UUID4 for uploaded filenames (was sequential before)
Changed invalid file upload response from 400 to 415 (Unsupported Media Type)
Fixed confidence score ranges and async loading bugs in frontend
Removed redundant code in app.js

March 11, 2026
Added automatic file cleanup via async background task with asynccontextmanager lifespan
Improved splicing detection: added mask ratio cap, morphological open/close, erosion
Introduced dependency injection using FastAPI’s Depends() for the detection service
Created DetectionService class wrapping both engines with ThreadPoolExecutor for parallel execution

March 12, 2026
Set up full pytest suite: conftest.py with fixtures, test_api.py (mock tests), test_detection.py (real inference tests)
Used dependency_overrides to inject mock detection functions for fast API tests
Added pytest.ini with fast/slow markers
Improved false positive reduction: connected component filtering, confidence-scaled blob thresholds, synced mask with confidence output
Set up GitHub Actions CI with two parallel jobs (api-tests, model-tests)
Debugged CI: fixed pytest PATH issue (switched to python -m pytest), fixed model weight download (needed in both jobs since model loads at import time), fixed hardcoded path issues
Removed test images and model weights from git tracking via .gitignore updates

March 13, 2026
Added try/catch error handling in api.js (fetch + response parsing) and app.js (error propagation to UI)

So I started & finished this project after finding about Flavortown, but I did keep a somewhat detailed log of my progress:

Initial planning:

• User will be able to remap one key to another key or combination and vice versa
• Will be a GUI using Java Swing
• There will be a way for users to see what keys they’ve remapped
• Mappings will be stored in a hashmap for O(1) lookup
• Will add a keyboard heatmap cuz it’s cool
• Will add a wrapped feature (like spotify wrapped) but for your keys

Feb 2+3, 2026:

• Used JNativeHook to register user keypresses, played around with methods to get used to the library

Feb 4, 2026:

• created a basic implementation of gui
• created a custom KeyMap class, which stores keycodes of the key and what it’ll map to
• played around with trying to block the user’s initial keypress

Feb 6:

• successfully created the keymapping logic
• allowed for holding keys too
• made it so it’ll save all the users mappings inside a text file and it will reload everything into the hashmap that stores everything upon running
• made some other small changes/features like being able to remove all mappings, hiding the select keybinds options after saving, etc.

Feb 7-9:

• created a virtual keyboard on gui where you can press two buttons and this will create a mapping between the two.
• moved gui code away from main.java file to a separate file for organization
• created custom layouts (100%, 75%, 65%)
• stored all mappings inside a table so the user can keep track of all of them
• Also allowed the user to remove a specific mapping by selecting it from the table

Feb 11:

• fixed a ton of buts
• Finished MVP
• switch from saving mappings and custom keys in a text file to storing everything in json using the Gson library.
• now i can just store everything into one file instead of data from combinations (custom keys) and normal keys being in separate text files

Feb 12:

• finished heatmap + keypresses wrapped
• added ability for user to press “shift” and access the keyboard as if shift was being held
• also allowed users to map shifted keys like @ and !
• fixed a lot of minor bugs

Feb 13:

• small changes (minor fixes)
• basically done

Feb 14-17:

• fixed bug where it would spam a key infinitely
• cleaned up and refactored code

Feb 18:

• added thread safety