Mnemosyne

Okay I realised that If I want to ship this my video capture must be in the front end. So I had to refactor all my API calls in the backend to handle this. Wasn’t too difficult and this actually simplified my code a lot.

Also I improved the multithreading in my backend. Now the response is much faster. Essentially my LLM was streaming an output but it would wait for the TTS to finish converting a chunk to audio before it streamed the next chunk. Now I have the LLM add chunks to a queue and have 3 simultaneous TTS workers which takes as many as 3 chunks and converts them into speech. This way is there is no lag or jitter.

I am fully integrated! (with the exception of one or two things).

I have functional prototype going. It is able to capture memories on a manual trigger and then decide if future queries require memory recall. If so it will fetch those memories for the AI to use in its response.

Next I just need to setup automatic triggers for the system to capture memories. Maybe a loop or something more intelligent.

And optimise as well. Because right now its really slow.

I realised that I need a more efficient way of storing and searching for memories. As the no. of memories in my database grows the search time will take too long. After a lot of research and coding I have implemented the HNSW database. This works by creating a layered hierarchy where similar embeddings are stored in a cluster together. Hence, when looking for the right memory for a given query it is able to navigate the database much faster.

I also experimented with a concept called HyDE. It essentially is a simple idea where instead of comparing query embeddings to memory embeddings it gets an LLM to hypothesise what a memory string would look like for a given query. Therefore, the similarity score should be much higher when comparing the embeddings. However I am not sure how effective this would be for my project as the LLM could hypothesise incorrectly and fetch the wrong memories despite it having high similarity scores. I guess I’ll find out!

Next I want to complete integration and have my back end fully working so that I can deploy onto a website asap.

Integration is underway! I’ve successfully combined and validated that my functions work together smoothly. I am able to capture and store memories and then retrieve them when needed as shown below.

Now I need to setup a continuous capture and verify that this works as the database gets scaled to hold more memories.

Implemented some neat filtering functions using an LLM + a very useful library called dateparser. The LLM takes the query and decides whether or not that query requires memory recall. Then dateparser looks at the query and then finds the appropriate time range that memories would have been stored with. This is great because it will save a lot of processing power for my RAG as only queries that require memory recall activate RAG and only embeddings from the relevant time will be processed for a similarity search.

We are up and running! I have FastVLM running and am able to extract descriptions from images. It’s not as fast as I would like it to be so I’ll see what I can do to optimise it. Next up I need to integrate these descriptions with my vector database and have it capturing and running seamlessly in the background.

First Devlog, Hello FlavourTown! This is a project that I have been working on a while. I am going to build an AI assistant that has perfect visual memory recall. With the goal of eventually deploying onto smart glasses to help memory impaired users navigate their lives easier.

The problem with current AR glasses is that transmitting images 24/7 is very battery intensive and heat inducive which is problematic for a small wearable device. Hence my goal is to build an efficient on-device encoding and storage system that efficiently captures and stores day to day images and stores in the form of embeddings. These embeddings then can be used by an AI to help users remember where they placed things, who they met, where they were, etc.

This devlog I mostly did research and setup and decided on an architecture and pipeline. I plan to use FastVLM by Apple. I am going to attempt Approach C and then maybe approach D afterwards.

From now on I will post more frequent devlogs as development is fully underway. Thanks for reading this far. Stay Tuned!