Activity

Evan Yu

Made a couple changes:

  • Deleted files are now tracked under a deleted status instead of just being set to failed. I’m still deciding on what I should do to purge these ‘tombstone records’.. Perhaps purge after some time? We’ll see
  • The worker now properly tracks bytes sent over the wire for downloads. Previously, we’d essentially directly pipe the byte stream from R2 to the client, and record the file size as bytes sent over the wire, even though that may not be true. I’ve wrapped that ReadableStream with a TransformStream<Uint8Array, Uint8Array> that implements transform(chunk, controller) . Essentially, it increments a counter for each byte (length) written over the stream
  • Did some dashboard revamping, specifically with how the stats are displayed
Attachment
Attachment
0
Evan Yu

Shipped this project!

Hours: 9.42
Cookies: 🍪 77
Multiplier: 6.84 cookies/hr

I’ve built a wrapper around a pager app “Pagem” so I can give Poke (an AI assistant) and some friends the ability to get ahold of me in emergencies. It’s built with Next.js and Convex. Additionally, I’ve managed to get a MCP server running on Convex HTTP actions using Hono + Muppet.dev

Evan Yu

Voters please read this

This is essentially a website that allows some people (and an AI assistant) to ring my phone. Pagers are old, antequated devices from the mid-1990s that ring and display messages. I have an app installed on my phone called Pagem that does exactly that, since I always have my phone in DND mode, and it’s near impossible for some people to get ahold of me.

However, I’m only allowed to have 3 contacts on their free plan. This project allows me to give access to as many people as I want. (in addition to exposing a MCP server too). There’s not really a way for you to test it (I value the little sleep I get :p), however there is a video that hopefully will suffice. I built this using Next.js and Convex as it’s the stack I’m most used to, and Convex is just goated lol


I’ve finally finished it, and have it deployed on https://pager.evanyu.dev/
Unfortunately, I can’t give out credentials to demo this as it rings my phone extremely loudly lmao

I’ve written detailed instructions on how to deploy this yourself on the repo https://github.com/Badbird5907/pageme

Watch the demo video: https://youtube.com/watch?v=zYOWyxfwU_M

Attachment
Attachment
Attachment
0
Evan Yu

Added an audit log, MCP info, and also a last used at field for API tokens and users.
Most of this was just some UI changes, since the data was already in the database. I just needed to add a Card with some (disabled) Input components for the MCP info, and pretty much copy over the page history datatable and add some columns.

The only thing I added was a lastUsedAt field for API tokens and users. This is updated when the token is used/when the JWT is minted

Attachment
Attachment
Attachment
0
Evan Yu

I just hooked up a MCP to it using Hono running on Convex HTTP Actions + Muppet. Poke seems to be able to send me a page, so this project is nearly done
Most of this was pretty straightforward. Muppet already supports serving over Hono, so I just needed to set up Hono for Convex HTTP actions and pipe the request to Muppet with pretty much the demo code they supply

Attachment
Attachment
0
Evan Yu

I’ve implemented most of the admin dashboard, including a user list, and API key management.
Pretty straightforward stuff with datatables, etc… Convex makes this stuff super easy with their react hooks. The best part of using convex is that if data updates (mutates) on one browser, all connected clients update too

Attachment
Attachment
Attachment
0
Evan Yu

I’ve set up a history page using Tanstack tables and shadcn. Pretty simple; just needed to add a by_createdAt_fromUser index to the convex table and using that index, filter by the user, sorted by creation time.

Attachment
0
Evan Yu

I’ve implemented a ui for paging, and also reworked auth (again) to still mint custom JWTs, and use convex’s Custom JWT Provider. It’s quite jank, but I don’t care, it works lol. Convex exposes an /auth/login http endpoint that validates the login, and mints a JWT token for that user. I then have some jank code on the client that sets the token returned in the browser’s cookies. I then implement a useAuth hook that’s then passed to Convex’s react <ConvexProviderWithAuth> component

Attachment
0
Evan Yu

No idea why i’ve decided to roll my own auth. I’ve reworked the JWT signing and have also implemented enforcing the JWT/admin role on the next server

Attachment
0
Evan Yu

Bootstrapped the next project, and set up auth with convex. Didn’t feel like setting up better-auth (and it wouldn’t fit with the scope of my project), so I’ve decided to instead roll my own “fake-ish” auth. Users log in with a username and PIN I give them, and my convex server just mints a JWT and authenticates agains that. It’s janky but I don’t care

Attachment
0
Evan Yu

I’ve reverse engineered how the Pagem API works, and i’ve reimplemented it in convex. So now i’m able to page my phone. Next i’ll build a webui around it, and also run a MCP server and give it to poke

Attachment
Attachment
0
Evan Yu

Did a bit more work, added bulk actions for the multiselect on the datatable. Also fixed a couple smaller bugs/ui papercuts. Finally made the mobile support for the file info page a tiny bit better

Attachment
0
Evan Yu

I’ve migrated the app to use the Tanstack Table library (datatables). This provides for more consistent pagination support. Also fixed the mobile support, and reworked the filters into a dropdown instead of multiple select boxes.

Attachment
Attachment
Attachment
0
Evan Yu

I reworked how the API keys (token) is generated. Instead of encoding the CDN/Ingest server URL in the token, it’s now another (exposable) env var. This makes it easier to expose the cdn base url to the browser. I’ve also cleaned up the API key creation dialog a bit. Tomorrow i’ll work on writing some proper SDK docs

Attachment
0
Evan Yu

I just spent the ENTIRE DAY trying to get this deployed and fixing all the issues. I somehow managed to run into a rare database consistency issue. TLDR is the supabase transaction pooler doesn’t guarantee the db will be consistent right after writing, only that it will be eventually consistent. This leads to a annoying race condition where a file doesn’t exist in the db when uploading, when it was just created by another endpoint.
It’s 6am, and i think i just got it to work. This was extremely fun to debug 🙃
Also I made a bunch of small changes to how the SDK and API server works, and the schema of the API token

Next i’m going to work on adding custom headers on callback for deployment protection bypass etc…

Attachment
Attachment
Attachment
Attachment
0
Evan Yu

I’ve written the some instructions/docs on how to deploy Silo. In the future i’ll look into using QStash instead of vercel queues for the nextjs app, so we can support deploying on CloudFlare only.
Also, I did some interesting prismjs stuff to enhance the highlighting for the wrangler commands.

Attachment
Attachment
Attachment
Attachment
Attachment
0
Evan Yu

Just spent the last two hours working on a overview/writeup on what exactly silo is. Next i’ll write up some docs on how to deploy silo, and then i’ll get onto actually deploying it on cf + vercel

Attachment
Attachment
0
Evan Yu

I’ve started work on a SDK demo site, with code samples etc…
The demo site is built on Next.js with TailwindCSS and Shadcn UI. I’m using prismjs for code highlighting. Next i’ll work on getting the SDK to work in this project and build out a proper demo

Attachment
Attachment
0
Evan Yu

Shipped this project!

Hours: 16.02
Cookies: 🍪 178
Multiplier: 18.15 cookies/hr

Re-shipping (again) due to lost hours

Evan Yu

I’ve reworked the file lifecycle. Most of this time was dedicated to making sure the database and R2 bucket are in a consistent state. (handling errors/retries gracefully etc…)
It mostly achieves this by utilizing “durable” queues to queue file actions that automatically retry if they fail.

Attachment
0
Evan Yu

I’ve added full RBAC to most/all tRPC routers and API routes. Also implemented project deletion. This is done by implementing a tRPC middleware that checks for permissions on the user’s role with better-auth

Attachment
0
Evan Yu

Added better mime-type support. The server SDK now accepts both shorthands (for example image which maps to every single image mime type), and fully qualified mime types (and wildcards).
Also fixed some server-side validation issues and finally implemented a better “staged upload” react hook in the SDK. This allows for better upload flows where the user can choose and “stage” files before uploading. (for example in a chat app)

Attachment
Attachment
0
Evan Yu

I’ve made the server router be more like a builder pattern, and added support for (async) callbacks for expiry and public ACL

Attachment
0
Evan Yu

I’ve begun polishing up this project to be shipped. I’ve redesigned the projects page, made project slugs unique, and added it to the create project dialog, and also added the project list to the main sidebar.
Also made a bunch of other misc changes

Attachment
Attachment
Attachment
0
Evan Yu

Added file expiry/TTL for uploaded files. File validity is checked on download, and they’re lazily deleted by a cron job on the cf worker every 30 minutes.

Attachment
Attachment
0
Evan Yu

I just implemented the SDK and did some major refactoring of the TUS implementation. There are 4 packages: @silo-storage/sdk-core, @silo-storage/sdk-next, @silo-storage/sdk-react, and @silo-storage/sdk-server

The Core SDK implements the core functions, like URL signing, different API request helpers etc…
The Server SDK implements the uploadthing-like file router ergonomics, including handling callbacks etc…
The Next SDK helps adapt the Server SDK to specifically nextjs, like creating the route handlers, etc…
The React SDK implements the React hooks like useUpload() and unstyled upload buttons/dropzones.

I did use AI to partially generate some of the SDK code, specifically the server SDK. This is because writing the typescript types would be very hard and cumbersome (see image 4)
I also used AI to quickly create an example nextjs app to demo the SDK. I plan on rewriting this part later.

Finally, while testing I ran into issues with my TUS implementation. Specifically with the upload resuming. Before, the worker stored all the metadata into KV, but after looking at Signal’s TUS worker implementation, I decided to refactor the TUS handler routes to instead use Durable Objects instead of storing state in a KV. This helps make recovery and keeping things tied together easier. Did use some AI to help with the migration.

Right now, it uses TUS chunked uploads, but i’m looking into streaming it

Attachment
Attachment
Attachment
0
Evan Yu

Built out the webhooks. It uses vercel queues to dispatch the webhooks with retries etc…
Webhook events are signed with a signing secret provided in the ui when creating the webhook.
(the ui is bad right now, but i’ll work on it later)

Attachment
Attachment
Attachment
0
Evan Yu

hours got unlogged again

Attachment
0
Evan Yu

I’ve revamped the environment system. Now each developer gets their own dev env, makes stuff easier. Also added a environment selector in the sidebar.
Finally, to handle deletion of environments with possibly tens of thousands of files, the worker offloads the deletion task onto cloudflare queues, which provides a durable way of ‘queueing’ the deletion of these objects. Not sure if that makes sense, it’s very late and I want to go to bed :p

Attachment
Attachment
0
Evan Yu

Revamped the dashboard, and also added a time range filter to the analytics page.
Also a bunch of other stuff that I forgot about. Spent some time implementing a thing I forgot I already implemented on another computer :/

Attachment
Attachment
0
Evan Yu

I’ve done a lot of work on the admin dashboard. The review dashboard uses z-score normalization to help us normalize reviewer scores to reduce bias.
Also hooked up the mailrelay.com api for marketing efforts.
Finally i’ve implemented an admin settings page for managing the state of applications (opening soon/open/ending)

Obligatory convex plug

Attachment
Attachment
0
Evan Yu

I’ve implemented an application review dashboard. It scores applications with a 3 question rubric.
By default, it hides PII to prevent bias

Attachment
Attachment
0
Evan Yu

I’ve added some cards to the dashboard, and also made the application form detect if it’s been edited somewhere else (convex ftw again)

Also minor ui fixes, like showing the question number in different places depending on the screen size

Attachment
Attachment
Attachment
0
Evan Yu

I’ve implemented a profile setup flow, and the application flow using Convex. Depending on which role the user selects, they are shown a different application form.

Did use some AI for animations (not fun :p)

0
Evan Yu

I’ve set up better-auth with convex

Attachment
1

Comments

inw
inw about 2 months ago

Convex ultrafastcatppuccinparrot

Evan Yu

Bootstrapped the project, and started work on the landing page. Currently i’m using motion.dev for a parallax effect, and some other scroll animations.

Attachment
0
Evan Yu

I’ve made it so that client SDKs can self-sign upload URLs without needing to hit /api/v1/upload for a presigned url.
Also API keys are no longer req’d to upload a file via the ui. It’s handled behind the scenes.

Attachment
0
Evan Yu

Added file/bandwidth analytics, and also revamped the sidebar + mobile support

Attachment
Attachment
Attachment
1

Comments

Evan Yu
Evan Yu 2 months ago
  • I seeded the db with some mock chart data for the screenshot
Evan Yu

The TUS protocol is now properly implemented, had to go around fixing some bugs etc…

Attachment
0
Evan Yu

I built a file info ui, pretty simple stuff using react query and trpc.

Attachment
0
Evan Yu

I’ve implemented the TUS protocol. It’s essentially a protocol for resumable file uploads, it’s pretty cool!

Attachment
0
Evan Yu

I’ve bootstrapped more of the project, added scoped API keys, etc

Attachment
Attachment
0
Evan Yu

I’ve bootstraped a new project using turbo-kit. I’ve also set up a cloudflare worker microservice, and set up organization provisioning

Attachment
0
Evan Yu

I spent forever fixing a bug that I introduced in a previous commit that was meant to FIX bugs lmfao

Attachment
0
Evan Yu

I’ve added a stats display to the llm’s response message. (tokens per second, time to first token, etc)
Also made a bunch of minor ui changes/fixes

Attachment
Attachment
0
Evan Yu

Made stream resume more robust and also made the optimistic message sending even faster

Attachment
0
Evan Yu

Finally tracked down the source of the error over to better-auth… This marks the fourth issue i’ve encountered with better-auth in my time using it that’s taken over 4h to debug.
(The record is https://github.com/better-auth/better-auth/pull/4724, which took well over 10 hours)

Attachment
0
Evan Yu

I’ve decided to migrate to tanstack start. This is because the app is mostly run fully client-side, and I want a better client-side routing experience. The only real next features I use are server actions and api endpoints.

Currently dealing with a weird bug: https://github.com/TanStack/router/issues/5196

Attachment
Attachment
0
Evan Yu

Shipped this project!

Hours: 7.37
Cookies: 🍪 113
Multiplier: 15.29 cookies/hr

I’ve built out turbo-kit to what I’ve wanted it to be. I’ll continue periodically updating libraries and adding stuff I want in the future, but for now i’m focusing on redux.chat.

Fixed an issue with create-turbo-kit not being able to run with npm. To my reviewer, please make sure you are on node 22 and npm 10.9 (or use pnpm :D)

Evan Yu

Fixed an issue with create-turbo-kit not being able to run with npm.
To my reviewer, please make sure you are on node 22 and npm 10.9 (or use pnpm :D)

Also this devlog re-logs the 7h of work that was in my first devlog that got unlogged for some reason

Attachment
Attachment
0
Evan Yu

I actually managed to do it… I optimized the send message latency from ~750ms to ~130ms! And on top of that, I implemented optimistic updates. Little writeup on what I actually did (copy-pasted from discord):

i rewrote the backend, the bottleneck was I was inserting sequentially as it’s basically a tree, and i need a parent node’s id before i can insert a child node
the solution i came up with is so dumb
basically I don’t want the client (browser) to control the id I give to threads and messages, as that allows for the user to supply basically whatever string they want
so I instead generate 3 ids, and I cryptographically sign them so I can “attest” to them being truly random. These ids are then passed to the client on page load (and regenerated on-demand)
then when the message is sent to the database (convex), it validates the signature and inserts the required rows with the ids that the client already knows about
all of this is so that when enter is pressed, the client already knows the ids of the thread and the message, so it can optimistically route to /chat/ and add their message onto the screen, making it look faster

tl;dr: I did some dumb sh*t with pregenerating ids on the server to make the browser redirect and show stuff faster before the rows are inserted in the database

Attachment
1

Comments

Evan Yu
Evan Yu 3 months ago

*I also shaved about 100ms off from the latency by modifying my auth system for convex functions to instead check the user’s jwt and not try to read their user record from the db on every request

Evan Yu

Everything works, but it isn’t fast enough. At least it doesn’t look fast enough. Apps like t3.chat optimistically route the user to a new chat before it’s even created, so i’m going to do that too. However you can’t trust data created by the client, so i’m having the server generate and sign a id on page load, and that signed id is validated by convex when a message is sent, so the client and optimistically route itself to the new chat page before its even created.

Attachment
Attachment
0
Evan Yu

My initial implementation of resumable streams was flawed and did not properly work. I spent two entire days debugging it and fixing it. Now it works!!

Attachment
0
Evan Yu

I’ve implemented a chat message input bar. It has token estimation, and a feature that highlights tokens. Additionally, any important chat state (i.e submitted, generating, error) is shown in the border of the chat input.

Attachment
Attachment
Attachment
0
Evan Yu

I’ve implemented a core layout, and spent most of the time building the sidebar. It uses tanstack virtual lists (https://tanstack.com/virtual/latest) to dynamically load the chat history from convex.

Attachment
0
Evan Yu

Spent a bunch of time debugging weird issues with turborepo, and getting convex to work properly with better-auth (https://labs.convex.dev/better-auth/features/local-install)
Starting to think I should have gone with what i’m usually used to working with (postgres), or using something like clerk/workos

Attachment
0
Evan Yu

Spent nearly 4 hours figuring out convex and setting it up in a turborepo with better-auth. It finally works!

Currently trying to figure out whether to use convex ents

Attachment
0
Evan Yu

I’ve added more images to the blog post, and also implemented an image carousel

Attachment
Attachment
Attachment
0
Evan Yu

Wrote some docs on the docker containers provided, added react-email (https://react.email/), and bumped next to version 16.1.0
The scaffold script now supports automatically removing unused dependencies

Attachment
Attachment
Attachment
0
Evan Yu

I built out the docs section on the website, it uses contentful, mdx, tailwind typography, and rehype-pretty-code for code highlighting.

I’ve also added redis to the template, and a types package, exporting all db types there.

Attachment
Attachment
1

Comments

Evan Yu
Evan Yu 4 months ago

*content-collections, not contentful

Evan Yu

Ignore this devlog, it is to track the 6 hours spent on the previous devlog that wasn’t tracked as the hackatime wasn’t linked

Attachment
Attachment
0
Evan Yu

I’ve bootstrapped a new project off of create-t3-turbo, built a CLI to bootstrap a new project (with docker!), and started on building out the features I need on turbo-kit. This includes fixes for multiple bugs with tRPC + better auth, and the shadcn/ui CLI for monorepos

Attachment
Attachment
0