ScrapMetal1

Okay there was a flavortown glitch not letting me post any devlogs for over a week but nonetheless:

I am officially published onto the google play store!

Fixed a mojor bug which stops the schedule from sending after 10 days. After ~10 days of the user not opening the app, android has a battery saving protocol where it moves it the background worker to a an app standby bucket. On Android 12+, apps in the bucket wont fire exact alarms. Switched from setExactAndAllowWhileIdle() to setAlarmClock().

I fixed up some device incompatibilities in the UI. Android 12+ wouldn’t adjust my design assets to fit on the screen which is weird since you would expected the modern versions to be better. But i fixed it by applying window insets to the root view of both the main activity and the add/edit schedule activity to avoid content being clipped by system bars.

Also updated my landing page site.

I have submitted my extension for review on the chrome extensions store. Hopefully it wont take weeks for it to get approved.

I am now working on creating the website. I have registered my subdomain at lockedin.eliascorp.org and have successfully deployed my scaffold.

I also did a bunch of testing and figured out how to optimise my AI prompt the best way. I added a thinking key in the json so know it must reason before it come up with a verdict. The AI is more likely to get it right if it reasons before it comes up with the verdict instead of stating the verdict first and then trying to justify it. Now it doesnt block sites that are somewhat related to my goals which is a lot less annoying for the users. For example if i said i was working on a coding project before it would block flavortown but now it reasons and understands that flavortown might be related.

However this does take a lot more tokens. I will have to implement caching soon.

omg this took me forever but I finally implemented a usage limit. I spent 5 hours trying to debug why my backend would not deploy only to realise that there was worldwide firebase extensions outage. I hate google.

Anyway after that I spent another couple hours trying to implement usage limit so I dont get bankrupted. That was pretty easy but when I tried to implement a notification to notify users I spent a lot of time debugging my code only to realise that windows was supressing the notification. I should use console.log more.

anyway its working now.

WE ARE UP AND RUNNING! The extension is working perfectly! A little too good. Sometimes its too strict. But for the most part its fair and even can tell the difference between youtube videos that are relevant or distracting.

The only problems is that it does take a while. I dont know how I can speed this up. Might just be forever slow like that. At least it works

Next I need to add a allow list and block list for sites that the AI cant block and start building my site.

Okay I realised that If I want to ship this my video capture must be in the front end. So I had to refactor all my API calls in the backend to handle this. Wasn’t too difficult and this actually simplified my code a lot.

Also I improved the multithreading in my backend. Now the response is much faster. Essentially my LLM was streaming an output but it would wait for the TTS to finish converting a chunk to audio before it streamed the next chunk. Now I have the LLM add chunks to a queue and have 3 simultaneous TTS workers which takes as many as 3 chunks and converts them into speech. This way is there is no lag or jitter.

Okay I’ve finished coding up the user authentication and session management. Now the app should only work if you sign in.
However this uses chrome local storage so I don’t know how secure this is.
Also:

• Updated the popups theme to match my blocked page. The image below will become the popup.
• Stopped using Tailwind. I dont understand the point of it. It just makes life harder.
• Got AI to renamed the variables so I stop confusing myself. so many states 🥴

Made a lot of progress in the front end. I created a html page that will act as the overlay whenever a site is blocked. Really happy with the design. Definitely my best html page I’ve ever built.
Also:

• Completed the POST request function to the backend.
• Created the background worker which runs every time a new tab is created.

Added a new feature.
The ability to customise if you want your message to send late. If you need to send messages at an exact time you can toggle this off. The message will no longer send if delayed.
However if you don’t mind if the message sends late then you select the how long the delay can be and it will send if its within the threshold.

Figuring how to send messages that were supposed to send while the phone was dead straight after the phone boots up was very difficult. But I think I figured it out by creating using a BroadcastReceiver to trigger a function after every reboot. I don’t know if this will work 100% of the time but it does so far so im happy 😁.

Fixed a major bug:

• Android was throttling my WorkManager jobs if the app hadn’t been open in a while (which is to be expected since this is an automation app). So now I am using exact alarms to wake up the android OS and then triggering the work manager. The exact alarm should go off regardless of whether the app has been open recently or not. But I need to test this over the next few weeks.

No visual change so I’ll just attach my favourite graphic that I made for the app :D.

Implemented authentication on the front end. I’ve configured Firebase to create the digital ID card for my website so it can sign users up with google and then routes them to my project in Firebase where I will take their name and email and add them to my user database.

Next I need to complete the JS to send the current website to my backend.

Alright, I set up verification so that no-one can spam my backend and waste a bunch of LLM API calls. It works well. The Unauthorised is what I want to see! That means the verification is working. Now I just need to have the front end to start generating valid tokens so that it can gain access to my backend.

I was very tempted to switch over to the proprietary Firebase functions that would handle authorisation and accounts. I did start to switch over to that but I realised I am not learning how any of it works so I decided to stick with coding it all out manually with just basic http requests using .fetch.

This also makes it easier to switch to a different cloud provider later down the line if I need to.

Backend AI functionality is complete. I implemented the API route for my AI to analyse URL. Using Vertex AI so no API keys needed in the repo which is neat.

The AI will take the URL and return a JSON file which then I can extract the boolean value for whether or not I need to block the site.

I wrote a test function which sends a https post request to index.ts with a google search looking up football results as the example.

Finding the right syntax for the new Unified Google Gen AI SDK was a pain but I am happy with the results. What I have now is very future proof and scaleable.

Okay I realise this devlog tracks a lot of time but I am really stubborn when facing a problem and won’t post a devlog until I fix it. This time it took me a while.

So I changed my plan and decided I am going to build a full stack app using Firebase and Vertex AI. It will be a lot of work but at least by the end of it I will have learnt a lot about hosting and deploying full on applications.

The issue I was stuck on was a compatibility issue where Firebase would not talk to the web request router (Hono) that I was using due to some recent update. So I just built my own adaptor. Took a lot of trial and error but now my requests to the cloud server are successfully reaching the local backend (my pc) and being routed and processed.

I am fully integrated! (with the exception of one or two things).

I have functional prototype going. It is able to capture memories on a manual trigger and then decide if future queries require memory recall. If so it will fetch those memories for the AI to use in its response.

Next I just need to setup automatic triggers for the system to capture memories. Maybe a loop or something more intelligent.

And optimise as well. Because right now its really slow.

Okay did my research, planning and setup. Decided on the architecture I am going to use:

• Framework: Vite (React + TypeScript)
• Styling: Tailwind CSS

Refamiliarised myself with react

Made some final changes so that I can ship.

DONE and LAUNCHED!! I am into the closed testing trials and should get approved in 14 days. Check out my listing below.

• Also I worked on my site a lot and created an app landing page so people can download my app easily while its still in the closed testing phase. Check it out at https://eliascorp.org/autosms/
• Got AI to write me a ToS so I don’t get sued.
• Recruited some closed testers through reddit
• Created a demo video for flavortown iOS users who won’t be able to download the app. It’s a little rough.
• Fixed this bug which stopped cloudflare from autodeploying my site.

Okay its been a while since my last devlog. I should have posted a couple dev logs in between but I’ve been stuck on this small bug and it’s driving me crazy.

The problem is when I open my app the opening animation is staggered and glitchy. This is not an application breaking bug, but it just makes the app feel low quality which I don’t want. Anyway after countless hours…. I still can’t fix it. I’ve tried dozens of things and nothing works so I’m just hoping someone more experienced in app dev can lend me hand at some point. Besides that, I got a lot of other things done while I was trying to figure that one out:

• I created the graphics for my App store listing. Check it out below. Looks pretty neat right? I spent a while learning photoshop and then just gave up used templates to stitch together the images. I also threw in a few pop culture references to create a hook.
• Added a sort button to organise the schedules.
• Fixed the hourly reoccuring logic to work after a reboot.
• Filled in all the paperwork for google
• Fixed some other bugs and polished up the ui.

I think one more devlog then I am done. This time for real!

I’m officially a proud owner of my own domain now! Check it out at (eliascorp.org). I bought this domain through cloudflare so I can host my privacy policy (https://eliascorp.org/autosms/privacy/) for AutoSMS and eventually other projects.

While I’m pretty familiar with xml I want to learn html+css comprehensively for web design so I coded the entire privacy policy in html. This took me a while. I also don’t want to get sued so I made sure to do my research regarding avoiding liabilities if it ever comes to that.

Okay so when I said I was done… that was a lie. I still have a lot of bugs and things I need to fix. Every time I’m close to publishing something comes up, like a certain permission I’m not allowed to use. THEN I have to change that. That change causes more bugs. I have to fix those…. Endless cycle. I want to die. But on the bright side I really touched up all the standard android ui and made it pretty. I have the github release done if anyone wants to try it out.
Almost there.

OKAY! I think I am finally done.

• I fixed up a lot of bugs. a lot.
• Realised that I need a start date for my messages otherwise it send before the scheduled time.
• Created a couple notes so that users know what to expect.
• Generated an icon for my app (AI generated). As shown below. What do you think? Should I keep the AI generate or just make a simple icon myself with photoshop?

Time to launch onto the google play store. However I need 20 testers to test continuously for 14 days. I don’t even know 20 people with android phones. 😭

I think I will ship before that. Gonna make a website that allows people to sign up for testing.

Added custom time intervals. You can send daily, weekly or monthly messages now. You can toggle on and off if you don’t want it to be reoccurring.
Also added custom intervals for every 3 days for example or every 5 hours.

This took a lot of coding and debugging. But it is done!! I am very happy with the app now. It’s clean and simple but highly functional.

Almost time to ship I reckon! Just need to do some testing over the next couple weeks.

I realised that I need a more efficient way of storing and searching for memories. As the no. of memories in my database grows the search time will take too long. After a lot of research and coding I have implemented the HNSW database. This works by creating a layered hierarchy where similar embeddings are stored in a cluster together. Hence, when looking for the right memory for a given query it is able to navigate the database much faster.

I also experimented with a concept called HyDE. It essentially is a simple idea where instead of comparing query embeddings to memory embeddings it gets an LLM to hypothesise what a memory string would look like for a given query. Therefore, the similarity score should be much higher when comparing the embeddings. However I am not sure how effective this would be for my project as the LLM could hypothesise incorrectly and fetch the wrong memories despite it having high similarity scores. I guess I’ll find out!

Next I want to complete integration and have my back end fully working so that I can deploy onto a website asap.

Removed all the AI options 🥲.

I found out that my app won’t be approved onto the google play store because it requires a sensitive permission send_sms. The only way to get this approved is if this permission is to make the sole purpose of the app around send_sms. I don’t want to get risk getting denied so I am ensuring that this is very clearly the only purpose of the app.

OMG, I found the bug I was looking for 3 months!!!! It was a sneaky bug where my PeriodicWorkRequest (The background worker responsible for the alarm in the os) accumulated drift after every message. The problem was when I learnt how to implement a reoccurring task in the android os, I didn’t realise that this would act as a stopwatch rather than an alarm clock. After each execution the object would calculate the next time as (current time + 24 hours) instead of scheduling for the desired time. If there was any delay due to androids battery optimisations or otherwise then this delay would build up and then eventually go above the max delay threshold and never send as shown in the image below.

I fixed this by implementing different logic where I set the time until the next message as (current time - scheduled time).

Next up, I want to implement custom time periods.

Brought over all my existing project for this app and created a remote repo. Still learning git so this took me a while. I have a really annoying bug that I am trying to solve where my messages stop sending after 7 days. I can’t find the issue in the code anywhere. I guess I just gotta keep debugging 😒

Integration is underway! I’ve successfully combined and validated that my functions work together smoothly. I am able to capture and store memories and then retrieve them when needed as shown below.

Now I need to setup a continuous capture and verify that this works as the database gets scaled to hold more memories.

Implemented some neat filtering functions using an LLM + a very useful library called dateparser. The LLM takes the query and decides whether or not that query requires memory recall. Then dateparser looks at the query and then finds the appropriate time range that memories would have been stored with. This is great because it will save a lot of processing power for my RAG as only queries that require memory recall activate RAG and only embeddings from the relevant time will be processed for a similarity search.

We are up and running! I have FastVLM running and am able to extract descriptions from images. It’s not as fast as I would like it to be so I’ll see what I can do to optimise it. Next up I need to integrate these descriptions with my vector database and have it capturing and running seamlessly in the background.

First Devlog, Hello FlavourTown! This is a project that I have been working on a while. I am going to build an AI assistant that has perfect visual memory recall. With the goal of eventually deploying onto smart glasses to help memory impaired users navigate their lives easier.

The problem with current AR glasses is that transmitting images 24/7 is very battery intensive and heat inducive which is problematic for a small wearable device. Hence my goal is to build an efficient on-device encoding and storage system that efficiently captures and stores day to day images and stores in the form of embeddings. These embeddings then can be used by an AI to help users remember where they placed things, who they met, where they were, etc.

This devlog I mostly did research and setup and decided on an architecture and pipeline. I plan to use FastVLM by Apple. I am going to attempt Approach C and then maybe approach D afterwards.

From now on I will post more frequent devlogs as development is fully underway. Thanks for reading this far. Stay Tuned!