Had to lowk delete some changelogs since ts over 2000 chars. But added loads. lowk the frontend is kinda broken

Changelog

• true random projects
• CORs
• CORs
• Hot fixes
• Styles
• Non default banner needed for homepage
• final credits done
• fonts
• Project Descriptions, View User on flavortown, tabs
• Added credits to navbar
• Basic Credit Page
• No case sens for searching
• Excudes deleted projects
• Emoji Sizing
• Fixed random projects
• rename of workflows
• Added theme control to the site
• Shows slackid in user page
• Added random devlog speed up with same logic as random projects logic. Cache prewarming
• Added far faster random devlogs
• Project Styling
• [Removed CSS in heex files, moved it into singular CSS file

Improved Setting UI a lot, and added video clear buttons to easily remove videos you do not want.

I redid the file layout, fixed some performance issues. This is looking really good so far especially since this is my first react project

I added a little settings menu now for this. Added the urls being stored in local storage, and editing the youtube urls to play any youtube video and save. Also added a mute button to allow the user to unmute all videos at once

I have made readme changed for shipwrights, I also added the performance changes I accidentally forgot to unstash earlier. These just ensure chunks are computed at the same time instead of sequentially. This significantly speeds it up on ARM CPUs because of the low clock speed but many cores.

Changelog

• readme
• Updated readme
• Readme updates
• Added stashed performance additions from earlier

I have now added text highlighting so users can see which words are the most AI. I did this by cross referencing the chunk results and then applying highlighting to the text based on the percentage dynamically. 100% AI makes it 50% strength, 50% AI makes it 25% strong etc. Its all dynamic. I have also changed the model because of accuracy issues with previous models not predicting accurately

Changelog

• Added text highlighting
• Changed AI Model used to classification

Uh Oh 2 hour 43 minute devlog

so umm i did a LOT. I have redone the frontend myself so it looks really cool. I made a compose file to make it easier for deploying locally with a file cache. I added a cache for the models so I don’t have to consume all the bandwidth when I am restarting the container lol. I have had some issues with adding darkmode to this, switching back only removed storage, but did not clear the data theme attribute in my code. so yeah. I was trouble shooting my workflow after I migrated from my selfhosted GIT to github. I also adding caching to my dockerfile and my workflow for making and publishing a image.

Changelog

• Fixed darkmode
• Added darkmode
• docker ignore
• Redone frontend
• Made compose for local dev
• Fix
• Dockerfile cache fix
• Fixed cache path in dockerfile
• BB Cache
• Docker Fix
• Made the finished version called “:latest-arm”
• Fixed dockerfile
• New release
• removed copying mix lock
• removed mix.lock
• mix lock
• Docker fix
• Docker changes

I made a dopamine focuser to help focus. This is a react project that splits the browser into 4 panels and displays 4 seperate youtube videos at once. These are currently hardcoded in the config.js. These videos are streamed via youtube and are embedded. I tried doing this as fast as possible.

Deployment, made a dockerfile and a github workflow to make and publish the container. I then routed a cloudflare tunnel that points towards the container that is started via a docker-compose.yml file inside my oracle virtual machine.

I made a way to input what you want to do in the command line interface in any operating system and then get a command to do just that. I also added a way to easily copy this command onto the cli. I have not tried multi command arguments yet. This is built using elixir + react. Elixir is amazing for the scalability the backend offers. React allows for a nice looking frontend to be made. This was very complicated from both aspects from issues with CORs and developing it as a whole in under an hour.

Added a basic frontend to this, as well as a CI/CD pipeline to make this into a OCI image for easy deployment. The frontend is currently vibecoded, I am going to pivot towards a human made interface soon since this is all Proof of Concept

Added POST APIs within phoenix to allow for a basic frontend to be made later on. I have further optimized and improved. idk wat photo since this is backend so yea

Migrated to elixir, a lot of performance gains, from a couple of seconds down to one for inference via chunking and running in parallel. idk what to do for the photo so idk. enjoy a photo of a bad gateway ig

I got a working PoC of the AI classifying model working. I have tested my self and the results are very accurate. It took me a while to find a model that was accurate since many are not. This does not work for code just yet. PS idk what to screen shot.

Added readme changes for optimization sidequest

Changelog

• Non default banner needed for homepage
• final credits done

Added fonts & project descriptions.
Also added a button to take you to the users profile on flavortown. Live is working ish, the flavortown API is slow or smth and not updating

Changelog

• fonts
• Project Descriptions, View User on flavortown, tabs

I added a credits page

Changelog

• Added credits to navbar
• Basic Credit Page
• No case sens for searching
• Excudes deleted projects
• Emoji Sizing

Testing & bug fixing live scraper

FINALLY, got a live scraper to scrape, /projects & /devlogs. This allows for liveish stats. These refresh every 30s.
Fixed missing devlogs & time affecting projects, I had to make another scraper to handle this by using /projects/{id} to grab the specific projects, and /devlogs/{id} for the specific missing devlogs. Hitting some rate limiting right now on the fix missing devlog scraper. Also made a TUI for my scraper to handle all my various scripts. Lots of messing around on my supabase to find devlogs, trouble shoot, make some RPCs etc. It was painful to say the least

Added theme controls, currently has two that you can choose

Added slack ID to user profiles to easily find a user

Changelog

• rename of workflows
• Added theme control to the site
• Shows slackid in user page

Features

• Far faster loading times of random projects & devlogs, down from 5 seconds to ~ 40ms.
  I love caching

• Project Styling fixed

Changelog

• Added random devlog speed up with same logic as random projects logic. Cache prewarming
• Added far faster random devlogs
• Project Styling

Features

• Scrolling arrow buttons added, the old scroll bar was hard to see
• Polish everywhere, same font used,
• layout fixed since it looked weird in some places.
• Fixed layout shift from project / user pages

Misc

• Consolidated all CSS into a singular file to make it reusable. This in turn shrinks the codebase to make it more manageable and maintainable.

Changelog

• Removed CSS in heex files, moved it into singular CSS file to reuse components and make the codebase more maintainable
• Scroll arrows for the smaller tiles

New logo

• Made by @SeradedStripes thanks so much
• Some prod fixes for implementing this
• general bux fixes

Changelog

• prod fix v3
• prod fix v2
• prod fixes
• added logo & favicon.ico, made by @SeradedStripes

Features

• Added new way to view devlogs, looks a lot better now
• Redone the logic behind the random devlgos & projects to utilize caching more effectivley

New Caching

It picks a random number from 1 to how many devlogs / projects
Takes that, puts it in to the cache,

Changelog

• Added more efficient caching logic for random projects by using the logic behind random devlogs
• revamped devlog page, redone devlog logic to use caching more efficiently
• readme

Added API Docs back based on the old openapi.yml but now hosted on scalar to make it look better than hosting it locally within FTPDB
Bugfix with if a project was deleted with a user, it would return nil which the frontend didn’t like, it now returns the projects

Features

• Added styled page to show messages instead of pure JSON

• Added URL_Expander, uses same logic as the shortener but uses my own random word library written in elixir to get a random string.

• Automated github workflows for the new branch expander

• General Styling and bug fixes

• Bug Fixes

Changelog

• Fixed urls being too long
• increased amount of random words to 10K
• Styling
• workflow
• Workflow name changes
• workflows
• bug fixes
• URL_Expander fork
• Revert “updated html,css and added image”
• updated html,css and added image
• Added better way to view message with style, not just json string

Fixes

• Project time now shows on random projects
• Fixed devlogs not being shown due to linking issues in the supabase table

Features

• Added devlog count on the main page

Changelog

• Added load more projects, fixed time not being shown on random projects
• Added devlog count on homepage
• Image rendering in markdown
• Added basic markdown support
• Moved heex within elixir files into seperate .html.heex files for maintainability

Added markdown rendering into this. Renders images as well as full complete markdown.
Refactored html.heex files, moved from .ex into .html.heex.

Changelog

• Image rendering in markdown
• Added basic markdown support
• Moved heex within elixir files into seperate .html.heex files for maintainability
• added search bar back

Fixed navbar, now consistent across all pages. Did some scraping scripts that were not recorded on hackatime
Refreshed leader boards etc. Might ship soon

Migrated from garage to Seaweed FS for faster speeds and async. Took me ages of untracked time

• Shows upload speed
• Previews of ALL files now

Changelog

• Changed url from garage to seaweed
• Added Mbps speed count

Added another node thanks to seraded stripes allowing for cross region backup. If u wanna help DM me :)

I know, I know, a 3 hour devlog

• I will be honest I just forgot

We got some MASSIVE changes

• Migrated 100% to garage
• Custom SDK no heavy (AWS S3) SDKs
• Fast
• Unlimited file size depending on ur internet speed
• Shortened URLs from previous project reported over with APIs
• No CSS @Tom needs to do another PR
• GH Workflow for publishing arm64 via selfhosted runner, GH hosted runner for x86/

Issues

• Had many enospc errors, (server kept running out of memory and space)
• Phoenix Liveview, had issues coordinating events in the correct sequence leading to crashing randomly
• URL API was weird, sometimes nothing being returned, my fault with json bodies.

PS a lot of this was backend work dont complain it doesnt look “pretty”. It took a long time for all of this

Changelog

• branch issue
• amd86 workflow
• Allow publish workflow on ‘garage’ branch
• docker release
• working frontend & backend minimal CSS
• gitignore
• env vars

UI Addition & bug fixes with in it. Thanks to @Tom.

Custom SDK is underway

• get_bucket_info(client, bucket) returns all bucket info in xml
• get_object(client, bucket, file_name) returns the file contents
• put_object(client, bucket, file_name, file_contents) uploads file to S3.

Added DB.ex

• Test scripts

Env vars are secure

• runtime.ex

config :fts3, :S3,
  base_url: System.get_env("S3_BASE_URL"),
  region: System.get_env("S3_REGION"),
  access_key_id: System.get_env("S3_ACCESS_KEY_ID"),
  secret_access_key: System.get_env("S3_SECRET_ACCESS_KEY")

Todo

• Complete SDK
• Simple UI of garage S3 branch
• Docker workflows for garage etc

Changelog

• gitignore (ebee0e7)
• S3 lib (2c3ed80)
• DB.ex for testing (af961e4)
• test scripts etc (904ce70)

Finally got garage setup. Took me a while to do in docker, but I eventually moved to bare metal locally. Just tried a test upload to the API from another device to the domain. It Worked!.

Time to do more testing in python and then make an SDK in elixir

Migrating to Garage S3 after this devlog

Changelog

• Proper elixir function for splitting filenames
• add timestamp adds the timestamp at the first dot, instead of before the exention
• encoding file names to stop errors
• Syntax fix
• Added timeout to 10m
• Fixed default address, appened .uk
• Fixed address iusssues
• unlimited file size, not 5GB

Working PoC with any file size

Added working PoC.
Lots of work within supabase site including configuring buckets,testing etc.

Workflow

• Upload file
• File is sharded and uploading in chunks to bypass cloudflare limits
• File is reassembled then uploaded to a local supabase S3 bucket
• I make the address where the file will live, then take that address and pass it through my url shortener and return that instead.
• When the link is followed, you are taken directly the file in supabase.

Features

• Data & Time are appended to all uploaded files, this is to ensure files are not overwritten by other users.

• Chunking to bypass CF limits

• Custom bucket support, if I wanted to run two of these services on two different buckets I can change which bucket through env vars.

and more

To do

Get a better UI with more CSS

Changelog

• Added docker release & actions
• removed base page
• remove default phoneix styles
• routing APIs and requests
• Used AI for a basic frontend
• Added upload controller, handles everything. Used AI for commenting
• env managment
• Added env secrets
• Init, depends

I fixed a lot of issues and bugs. Reverted API docs, and changes to the backend DB.ex. Calling random projects as an RPS from supabase instead of getting “random” projects manually, this ensures that these are truely random, added a filter to ensure default banner projects are excluded since this are normally default starter projects which have no time or content. Redone projects structure to make it more suitable for more vertical screens. Formatting, redone some DB.ex structure. A lot of work of the supabase tables to make RPCs work correctly, supabase tables needed maintenance etc to ensure they are happy.

Added openAPI schema for @SeradedStripes
Fixed some carasel issues
Tidied up some stuff

Gotta improve how the openAPI is presented

• Chores, OPENAPI docs, and fixes (3718655)

Added dockerfile for containeraztion
idk what image

Chores & Devlog page fix
Making devlog page nice next

• Formatting & Random devlogs API (75dfac6)
• /devlogs page for random devlogs (a0786c3)

Basic devlog page

Fixed scraping media URLs, was not fetching since I was using the incorrect API, i still dk what to do as images

Fix to homepage, refactor of functions under DB.ex, misc, project, user, devlogs reorder of functions for maintainability.

Consolidated frontend to use single CSS style document,

And more!
probably more caching

• Added navbar being the same (a3b8294)
• Consolidated frontend, all uses same CSS styling from one file (fa1912a)
• Homepage fix (e20a8b7)
• Refactor, misc, projects, users, devlogs (831f3b5)
• Refactor, misc, projects, users, devlogs. Pt2 (0fe368a)
• Homepage Fix and misc (ff89f47)

Fixed project page showing total user time instead of total project time

Added faster banner scraping by using 40 concurrent workers, I added the option to rescrape FT for banners not just for Null values in the database

Added a container that updates the database in realtime via the API

Idk what image to do

Chores & Features

Added /page/:d to show user stats when being searched
Added separate panel for searching for users

• Fixed crash when nil was being divaded by 3600, fixed by making default 0 if no value fetched (e0bce39)
• Added searching for people (4747ff2)
• formatting (66349a4)
• Search improvements (074f5e1)
• Added /user/:id page for user stats & projects (8fe4f1c)

Added Cachex caching, this should significantly take off database load and speed up the site. idk what to do for the photo since it as all backend work

• Implemented Caching with cachex (f83753b)
• Fixed bug where when you clicked on a project it brought the user to the /project/:user_id (de490e9)

Added /projects to view projects in a grid view with shuffle to get random projects etc
More buttons everywhere
Slack fix

Be sure to use /suggestions for improvements

• slack fix (6d374f0)
• Added more buttons in project page (4c0cae6)
• formatting (f3cfb3c)
• Added a /projects page that shows random projects or those based off of filters (5383084)

Added suggestions, let me know what things you want now

• Added suggestions page that messages me on slack with suggestons (1a76864)

Added searching with smart complete which takes into account hotness and total likes to correctly find what you are looking for

• Carousel is now working properly with the dots (833f24e)
• Added total likes in project viewer (7eae16c)
• Formatting & Searching (d51140a)

Added native viewing for devlogs and projects within this. What should I add next?

• Added profile pictures and display name (754d480)
• Fixed forwarding to via user_id not project_id (ba2a975)
• get_devlogs function (d66d9ce)
• Renamed api.ex to DB.ex, more routes. (62a6fe1)
• Added /project/:id path to show the project and devlogs locally (48d21dd)

Added banners & Profile pictures and names

• Added banners to homepage (ff1e556)

Added basic frontend work with help from claude to get it done quickly.
I am working on getting project banners next
Added more API routes so we can see who has the most freetime

Added backend APIs for the frontend being built
/hot returns top ten projects determined by an algorith,
/top_this_week returns the top projects from this week
/fan_favourites returns most liked projects.

I am working on a frontend next using the PETAL stack.

ps, IDK what image to add since this is all backend

Scraped most of FT, some devlogs missing project IDs, the APIs are alr to work with but I will be releasing my own that are better. I do not have a frontend yet

This project WAS removed before, this devlog contains all the prior devlogs, all 15 of them. Because of the 2000 character limit in flavortown the devlogs are in this file in my github repo:

https://github.com/jam06452/URL_Shortener/blob/main/devlogs.md