miggy

A bunch of stuf!!!

Fixed up the reminders, added a docker compose, added auto case creation, fixed file sending, added a manifest file, added auto case using keywords, improved ratelimit handling and improved caching :3

Sorry for the massive devlog

Changelog

• fix: race condition + use a retry loop
• fix: use an actual cache library
• fix: add cookies as a trigger for the FT tag
• fix: re add user dm on case closed
• fix: don’t create multiple threads for multiple files
• fix: function description
• fix: update ft-voting triggers and auto response
• feat: add user auto responses
• feat: add case tagging
• fix: bump base reminder time to 24h
• fix: add missing docker env var
• feat: a lot of stuff
• feat: who did it?
• fix: fetch trust levels with admin api
• fix: api docs page
• fix: docker compose + requirements.txt

Improved the README and fixed an issue with uv packaging :D

Changelog

• fix: add pypi url
• fix: ship manifest with package
• fix: bump version
• fix: add warning + uv sync

Slack bot!!

Made a Slack bot that integrates with Otter to view people’s projects within Slack!

Check out #otter on the slack or run /otter to try it out!

Changelog

• feat: inaccurate search disclaimer
• feat: add github button
• feat: add dedicated channel
• fix: no projects handling
• fix: styling fixes
• fix: not in channel handling
• fix: add gzip support to backend
• feat!: slack bot project listing
• feat: slack bot

Added a message deletion mode, fixed bot permissions/scopes and fixed the invisible unicode encode/decode logic

Changelog

• feat: add delete apps command
• fix: use correct permissions
• style: add images to readme
• chore: format + lint
• fix: unicode encode/decode logic

Blur hashing!!

Images are now stored alongside a blur hashed version that gets served to the client on the main search request that serves as a placeholder until the (VERY SLOW) airtable URLs load

Changelog

• fix: expanded image closing
• feat!: add blurhashes + progressive image loading

Tiny update

Added search suggestions and a text with total projects in the DB

Changelog

• feat: add temp favicon
• fix: add query data
• fix: back button
• feat: suggested searches + total projects

Feedback! (round 2)

• Added scroll pagination!!!!
• Added a comically large disclaimer
• Tweaked button positions on mobile
• Improved media handling/scraping

Changelog

• fix: make the warning even more obvious
• feat: add BIG projects disclaimer
• fix: add disclaimer
• fix: make pos absolute on mobile
• fix: theme switch position on mobile
• feat: add last refreshed at text
• feat!: better media handling
• feat!: add scroll pagination
• fix: preconnect to airtable assets url

Feedback!!!

Based on people’s feedback, I’ve improved several things like:

• Added literal results to the search algorithm (searching for projects should feel more consistent)
• Made hour formatting more consistent
• Improved the README.md
• Score display now has colors
• Low score entries are now hidden with a button to show them
• Saved current view mode in query parameters
• Added an endpoint for recently added projects
• Speed improvements
• Stopped using media URLs from the Ships API (to avoid expired URLs from being served)

Changelog

• fix: speed up literal candidates subquery
• feat: add project share button
• fix!: don’t use media URL from the ships api
• feat: recent endpoint + code cleanup + /api/v1
• feat: add recent projects endpoint
• feat: save view + apply score color
• fix: better score display
• feat: hide low score results
• fix: add demo url to literal results
• fix: clickable title
• feat: literal results + score colors
• feat: improve readme
• feat: redirect to frontend + improve README
• fix: more consistent hour formatting
• fix: fallback to homepage + add archived urls

Improved a lot of heartbeat ingestion related stuff

• Ignored a regression alert, woke up with 4 THOUSAND Sentry errors (80% of my montly total)
• Proceeded to fix said bug while panicking. Turns out heartbeat requests were failing in a very specific situation, and it was constantly rolling back database transactions
• Improved skeleton/dynamic SSR loading so pages feel snappier
• Fixed imports, they can now import data until 2013! (was 2024 previously)

Changelog

• fix: “no projects” on invalid search
• fix: bump up skeleton delay
• feat: improve content loading
• fix: use default release settings
• fix: deduplicate heartbeats before insert
• fix: reuse import_store
• fix: improve heartbeat storing logic
• feat: better importing with board searching
• fix: heartbeat ingestion responses
• fix: use native time function in install script
• fix: revert status code change

Finishing touches

• Improved the README
• Fixed inconsistencies in the hour formatter
• Fixed back buttons + missing data

Changelog

• fix: improve readme
• fix: higher quality hero image
• feat: redirect to frontend + improve README
• feat: readme draft
• fix: more consistent hour formatting
• fix: fallback to homepage + add archived urls
• feat: improve back buttons

Cool stuff/QOL update

• Added OG tags! You can now share projects in the Slack with pretty embeds :D
• Added markdown rendering to the descriptions
• Fought against Cloudflare pages for an hour trying to figure out why everything broke (it was the HTML sanitization library )
• Improved the /project/[id] page and tweaked the image display

Changelog

• fix: add project not found embed
• fix: truncate description in embed
• fix: header tags
• fix: use ‘xss’ for html sanitizing
• fix: use the correct url 😭
• fix: enable observability
• fix: add wrangler file
• fix!: switch to cloudflare adapter
• feat: SSR og tags
• fix: description rendering
• feat: markdown rendering + better image layout
• feat: add og tags

More changes!!!

• Added username and project name guessing based on the repository URL
• Made search faster by over 10x!!
• Made the site responsive on mobile
• Made NULLs appear last in queries
• Added pagination to the search endpoint
• Added query timings
• Improved logging and fixed assorted bugs/glitches

Changelog

• fix: search data + double search + error logs
• fix: use actual length
• feat: add search pagination
• feat: use inferred data directly from the db
• feat: search endpoint query speedup + NULLS LAST
• fix: page title
• fix: improve mobile responsiveness
• feat: text input clear button
• fix: improve search endpoint
• feat: guess name from code url + table styling

It’s devlog o’clock!!

• Added user searching! You can now look up projects from a user by typing in user:username
• Improved username and project name guessing based on the code URL
• Improved the look of the big projects table (also added more fields to it)
• Added a way to share search queries using URL parameters

Changelog

• feat: guess name from code url + table styling
• fix: remove useless filter items
• feat: add more fields to table + better name guessing
• fix: numeric filter parsing
• feat: user search
• fix: search views
• feat: query in query params

Refactored the old temporary explorer in SvelteKit!

Featuring:

• Filtering by any field
• Filtering using multiple operations for each field
• Pagination with variable page size
• View sharing

Changelog

• fix: tiny style fixes
• fix: styling + filter tweaks
• feat: jump to page
• fix: remove unused explorer file
• feat: add view sharing
• feat: save search + custom row limit
• fix: improve styling and table
• feat: granular search
• feat: better data table
• feat: add project page
• fix: explorer

Finally started working on a proper frontend with SvelteKit !
Currently, only the search API endpoint has been implemented so it’s still missing quite a lot of stuff

Changelog

• feat: add dark mode
• feat: add image expanding
• feat: add spinner + image component
• feat: add more data to views
• feat: add frontend ci + improve backend ci
• refactor!: add sveltekit frontend

Separated data fetching into 2 separate jobs, so data is now updated every hour instead of every 3 hours!

• Added pagination
• Improved parsing for null/invalid records

Changelog

• fix: explorer + increase timeout
• feat: improve explorer
• feat: add query pagination
• fix: timestamp parsing
• fix: separate screenshot url updates
• feat: upsert airbridge projects
• feat: split airbridge and ships jobs

A ton of stuff again!!

• Added documentation using utoipa and Scalar
• Added observability using OpenTelemetry
• Added true hours syncing using Airbridge
• Added auto rebuilding
• Added a temporary frontend
• Improved the request retry logic

Changelog

• fix: move ysws list endpoint
• feat: add OTLP observability
• fix: endpoint nesting
• feat: temp explorer frontend
• fix: make embedding generation less verbose
• fix: also retry non http errors
• feat: add docs + better query types
• feat: add true hours from airbridge
• feat: add build.rs

I KEEP FORGETTING TO MAKE DEVLOGS

So uh hi, big devlog time
(still no decent images because the frontend isn’t done )

• Got mixed local/API embedding working with fallback handling
• Sped up the data ingestion pipeline/job with some query optimizations and by separating embeddings into their own table (turns out having 1024-dimensional vectors in every row makes insertion/updates slow)
• Added embedding caching to reduce load/AI API usage
• Added full text search
• Added an image endpoint (gets the updated image URL from the DB, this is needed because Airtable image URLs expire very fast)

Changelog

• feat: separate embeddings into separate table
• refactor: improve the query endpoint
• fix: tsv generation + improve fetch job
• fix: faster screenshot updating
• feat: add full embedding cache + api/local mixed mode
• feat: add screenshot url endpoints
• feat: add temp query endpoint
• fix: update screenshot urls separately + downgrade model
• feat: add embed generation cache
• feat: add temp test search endpoint
• feat: add req retry + programs endpoint + api client caching
• fix: job log text
• fix: speed up data ingestion + api dimensions
• feat: add embedding handling

Set up a base for the project and started working on data embedding.

Current progress:

• Data fetching
• Embedding (local/API)
• Search API
• Frontend

Fixed a MASSIVE regression introduced in a previous change!
Gained a speed-up of over 30% in ALL queries (slowest query went from 6.9 seconds to 161.40 milliseconds, that’s a 42.75x speed-up!!!! )
Also optimized the database structure and reduced the backup size by over 25% while bumping the compression ratio from 15x to 33x
(also improved some of the existing graphs and the handling of non-existent profiles)

Changelog

• fix(frontend): better 404 profile handling
• fix: don’t error on profile not found
• fix: project time chart labels
• fix: make bulk responses use the wakatime spec
• feat: db chunk time change + fix ingestion regression

Improved the theme/style and improved overall contrast :D
Started working on some major performance improvements

Changelog

• style: improve contrast + sidebar

Here i go again with forgetting that devlogs are a thing that exist and need to be made!
Added a bunch of stuff like a profile page, skeleton loading, dashboard time ranges, SSR auth, timezone selection, fixed a ton of security vulnerabilities, improved UI/UX, and more… (see changelog below)

Changelog

• feat: add query logging instrumentation
• fix: hopefully fix SSR? please?
• fix: SSR auth?
• refactor: fully migrate to serverside auth
• feat: hydrate sidebar with SSR auth
• refactor: security refactor/fixes
• feat: show admin level in the profile page
• fix: add skeleton to profiles
• feat: add profile urls
• feat: save login redirects + public lb
• … (there’s like 15 more commits that doesn’t fit here 😭)

• Added support for slack’s HTTP mode
• Added thread resolving
• Added auto thread unresolving
• Tweaked some messages and fixed some bugs

Cleaned up the code so it passed pylint on strict type checking mode and modularized it. Also added/fixed stuff such as:

• added thread completion messages for user and fraud squad
• added confirmation for user messages
• fixed trust level fetching
• fixed bugs affecting file sending

Improved the standalone page and fixed rebuilding when that page is changed :>

Refactored the data fetching and caching from Hackatime to increase speed and decrease memory usage :D
(down to ~25MB during normal operation with a few cached requests)

Let’s just say that I forgot to make devlogs 😭

• Added optional Github-like labels
• Added a way to align to the current year (or use a fixed year)
• Improved the UI a bit
• Added some fun metrics
• Improved logging/tracing
• Cleaned up the code
• Some more tiny fixes/changes :3

Implemented message encryption/decryption, the encrypted message is stored on Slack so it can be reverted without a local backup! (just by using the same password)

Finished the message downloading logic (~150 messages/ second). Also added a way to view some cool stats about the data :3 (Slack app creation/authentication is also automated)