TL;DR

Added remote drive capabilities: now you can control the robot from the desktop app or via a game controller, and also get a record of the actions the robot took.

Remote Drive

• Refactored the main code to object oriented style so it could also be used in the remote drive module.
• Added RemoteDriveController which can run in two modes: socket (desktop app) or controller.
• Introduced device code that enables robot control and records the actions to be run later.
• Created a simple desktop app with tkinter that is used for the socket-based control.

WIP: Still need to compile the received “code” into actual code
Devlog note: First time video editing, apologies for the quality.

• Updated the README
• Fixed a little issue with the site tour

This devlogs mostly consists of two 11.59 PM sleep deprived coding sessions. So I don’t remember much other than fixing some issues with the vision module. This is an AI review of the last changes:

• Added a retry mechanism that automatically extends HSV color ranges if no objects are detected.
• Refactored color range extension logic to correctly handle multi-part ranges (e.g., red hue wrapping).
• Enhanced detect_contours to support specific color filtering and more robust area-based filtering.
• Added explicit error handling for unsupported picture-taking methods in the camera module.
• Updated fallback vision configurations and improved type safety for detection filters.

• Added interactive pin configuration
• There were a few issues with installation paths and virtual environment checks. Fixed those. It was much harder than it sounds
• Some enhancements to the vision module

• Created a setup interface that will enable user to select which libraries they want to install. Currently it only has the libraries that are used in the vision module but many integration will come.
• Optimized merge_close_contours function with the help of Claude
• Added configuration options for Camera module compatible with both rpicam-apps and picamera2
• Added examples for vision and camera modules
• Modified the readme to include new camera capabilities
• Made some bug fixes and improvements in ContourDetectorclass

Vision Module

I started working on the vision module. Added,

• Object detection
• Contour detection (to find a single colored continuous area)
  Also fixed a few bugs (I don’t remember exactly what) and handled the installation of vision libraries

Bug fix

I realized with the last commit I introduced a critical bug: when tools are called it doesn’t fit the response schema and errors. I made a temporary fix for now: tries json xml and markdown parsing and if they fail calls the llm with formatting and no tools. I am sure this is not very efficient but this is the best I have right now

I started working on this project again. I am planning to add object detection and maybe some other cv capabilities.

Current change

Since the day I started this project I am trying to make it portable and easy to setup. I changed the project structure a bit and now I will be producing a .pyz file that will be the only thing necessary for running this project!

Bug fixes

• If the LLM hadn’t asked you questions, the output wouldn’t have been structured, causing parsing errors. This is now (almost) handled.
• Also added structured output to all LLM calls.

Choose your reasoning effort

• Now you can choose the reasoning effort enhancer agent will use while enhancing your prompt. Hoping this will reduce the time enhancements take (even medium reasoning takes a lot of time).

I made 12 PRs to Lapse. Currently, 6 of them are merged, 1 is closed and 5 are waiting for review.

My PRs

Short summary of all of them (as much as I remember):

• Added comprehensive testing for API routes
• When the Hackatime login wasn’t properly done authentication was failing, fixed that bug
• Added duration displays to timelapses corners, like it is on YouTube
• Implemented a fallback mechanism for when re-muxing fails
• Implemented a *really *fancy setup script that helps devs set up the environment more easily: navigate them through database (automatically sets everything up if using localstack option) and slack setup
• Sometimes after stopping and resuming, a black screen was recorded instead of the actual source. Fixed that issue by not letting the user leave the recorder null.
• Fixed very minor issue in the build script.
• Added a time display in the recording.
• You couldn’t change recording sources or change the resolution of the screen you are recording (or the output would be corrupted). Fixed that.
• IndexedDB was sometimes closing unexpectedly. Fixed that.
• At the first use of a device the passkey was set to 000000 till the first recording. Now it is generated when the user checks it.

App Tour

I have been putting this off for a very long time. Today, I implemented this with reactour.
I thought it would be easy, but the insufficient documentation (IMO, no offence to the devs) had other plans

• And I also did some bug fixes and mobile view adjustments.

A lot of bug fixes

• Well, turns out I got the message types and edit request parameter mixed up in the edit web socket payload. I wasted a lot of time finding this issue.
• Sometimes the LLM didn’t call a tool, didn’t produce an output and just did reasoning. In this case, output was None and everything was failing. I changed the loop conditions a little, hope it works.

Structured Output

• I was telling the LLM to output in XML and parsing with regex, that was all the output validation (there was no output validation if LLM didn’t output XML it would fail). So, I added a response format with pydantic.
• I adjusted the prompt a little to work with structured output, but it still needs some work.

New feature: Edits

I started on working on a feature that would allow the user to request AI edits on the enhanced prompt

• Implemented the edit request input and web sockets logic on the front end
• Created a edit consumer to handle the edits
• Improved and adjusted prompts to handle edits
  Still did’t test anything. I’m soooo tired

Why is frontend so hard! 😭

• Made the website mobile compatible
• Now you can manually edit the prompt (in the future you will be able to request AI edits)
• Added copy and save prompt buttons

I forgot to devlog and now I don’t remember what I exactly had done! If I remember correctly,

• I added a few more options to prompt style, namely, Plain formatting and CoT to techniques
• The LLM was confusing the requested prompt style and the ultimate output user wants from the AI. I made it more explicit in the prompt but time will show if it worked.
• Finally, added a checkbox to specify the target models reasoning-native-ness, so we don’t depend on the LLMs knowledge to identify that. (also updated the prompt accordingly)

Updated the prompt. It was fully using XML, I learnt that this was not the best practice (I hope this information was correct). So now it is a hybrid: markdown for guidelines and XML for variables
Also, improved the prompt style section a little bit.

Added prompt style options: now you can choose the format of your prompt; XML, Markdown, long, short, few-shot etc.
Divided the prompt.py file to smaller functions for readability
And I unfortunately (or maybe fortunately) learned that full XML prompts are counter productive, which means I need to refactor the prompt agaaaaiiinnnn!

Added user messages to display when HCAI is down, fallback is being used and fallback is also down
Implemented a better fallback mechanism

Updated the README
(I know with all the emojis it looks like AI generated, but I like it this way)

Why everything keeps breaking! Ahhaaaa…
Today I tried to use it for something and backend gave an error: JSON parsing
Why?!! It was working yesterday. I checked the prompt, maybe there was something random making LLM go crazy. Couldn’t figure it out.
Finally added json-repair and it works fine for now.
And now the question Cancel button actually does something.

Please DON’T break again.

Made the project ready for production.
Deployed the frontend to Vercel and backend to Railway

Well, turns out you shouldn’t use CoT prompting with reasoning models.
Learning this I had to go over the prompt again.
I learned that gemini-3-flash works best with XML so I switched to XML
Overall spent (I hope not wasted) a lot of time on the prompt again.

Updated the frontend: prompts are auto saved and displayed neatly on the sidebar, stored in local storage.

Now questions are received, sent and answered as arrays, instead of plain string
Fix bugs with web sockets
Add rate limiting

Made significant improvements on the prompt: now it goes through a prompted chain-of-thought process.
Moved everything about the prompts to a new file.
(I want to apologize from HCAI for the immense costs to come)

Removed Celery and Redis so that deployment be cheaper.
Realized “the prompt” needs a lot of improvement.

Created a bat script to start the necessary services.
Made minor improvements to the prompt.
Also attempted to package the app in a .pyz file but failed :(

Implemented channels and web sockets to allow the LLM to ask questions to the user in prompt enhancement

Migrated to Django + Vite (frontend looks almost exactly the same)
Fixed the bug additional web search query.

The first primitive version.
Can save and load prompts.
Streamlit interface.
Has web search capabilities, not sufficient yet.