Redux.chat

Okay, this is my last devlog for Flavortown! I’m pretty much done the site (for now)! The only thing that really needs to be completed is finishing up the billing/paid plans (tokens aren’t free!). In the last 4-5 hours, I polished up the app and did the following:

• Added a bunch of new models from a bunch of providers
• MCP support!
• this was decently easy, the AI SDK already supports MCP pretty much out of the box.
• This was something I really wanted in other apps like T3 Chat and Perplexity, but i got tired of waiting :L
• Polar, it’s basically stripe but better lol.
• Implemented a credits system, users are “billed” the cost of their tokens + a small markup (don’t worry, there is a generous free tier). This makes use of Polar’s built-in credits/meter system. Polar also has a nice system that lets me track how much my users cost me!
• Got the site deployed at https://redux.chat/
• This was easier than I thought it was! There was only one minor issue (forgot to expose a env var globally) I had to fix to get it working! A lot of it was made easy by convex, vercel, and silo’s support for easily making new production environments.

I added “Instructions”. They’re essentially like Claude’s Styles, or ChatGPT’s Custom GPTs. They allow the user to insert their own custom system prompt. I introduced a new settings page with a registry of these instructions. By default the app ships two: Default, and Learning instructions (I’ll probably add more), but the cool part is that they’re all configurable! If you don’t like some default behavior, you can add that into the default instruction! This is a major things T3Chat doesn’t have, and a learning mode is something I really need to help me study that T3Chat doesn’t have.

I just implemented a really good model picker! It’s heavily inspired from T3Chat and T3Code. The entire thing is a Shadcn Popover component that houses the providers and models that I set up earlier. I did use some AI to get some ui parts right, and also to animate it with motion.
I also implemented a filter dropdown, which again is really another shadcn popover lol… Right now it can filter by model capabilities and knowledge cutoff date.
A lot of the time was spent on getting the feel right, for example support for the arrow keys (for example all 4 should work when there is no filter set, but only up/down should work when there is a filter set), or hover tooltips, etc…
I also implemented favorites, which are draggable to reorganize. To implement them I had to add another table to the Convex db schema modelFavorites, which stores the model id, and the sort order. This way when we query for favorited models, we can just sort it by the sort order!

Did three major things today:

1. I implemented chat branching. This has been a feature that’s been lacking on the ai chat apps I use (T3 chat and Perplexity). This isn’t actually that easy to implement, there’s a bunch of little edge cases and nuances. But I managed to get it working! Only when user messages change are branches created, and on every generation/regeneration are the assistant’s messages branched.
2. I also implemented message queueing. Essentially you can type in and queue multiple messages to be sent after the current generation has finished. This was decently easy to do, the hardest part was probably getting the styling right lol…
3. Added a hotkeys settings page. You can remap the hotkeys on the app in this page. The app uses Tanstack Hotkeys which is a (new) pretty nice library for adding hotkeys to webapps. I also added the settings and hotkeys page to the command palette

Also did a bunch of minor bug fixes that I don’t remember here.
Because FT is ending tomorrow, I have been using a bit more AI to juggle bug fixes, ui changes, and some implementation, as there’s still a lot to do. However, I have reviewed every line of code it’s written.

Did two major things today:

1. Reworked the model registry, now we support different AI providers, and openrouter. These providers are mainly used on the backend to keep track of all the models the app offers, and there’s a lot of routing code to determine if it supports certain features, etc… to finally convert it into an ai-sdk LanguageModel. (see the last screenshot)
   We then also use tokenlens to pull some useful model info like cost, features, etc… It’s backed by models.dev
2. This was the actual hard one: LLMs are different, some support PDFs, some support Word/PowerPoint (docx/pptx), etc…
   I want a unified experience across different models, that means every single model would need to support the same set of features. That includes supporting docx and pptx on models that only support PDF. The solution is to essentially convert those word and powerpoint files into PDFs. This isn’t easy, so the solution I came up with is to use libreoffice. libreoffice has a really robust file conversion toolkit, and it’s available over the CLI. BUT WAIT! We’re running in a vercel serverless environment, we can’t just call libreoffice normally, so we need to ‘expose an api’ for it. This is where Gotenberg comes in. This is a docker container that wraps the libreoffice CLI to provide a web API to convert those docx and pptx files to PDF.

So the flow is kind of like this:

1. The user sends a .docx file
2. <does the model support docx?>
   Yes: Send the file to the model, nothing to do
   No: Send the file to gotenberg to convert to a PDF, and then send to the model
   etc…

In the future I plan to write my own service to do this so I can eliminate some inefficiencies, but it works!

Did a bunch of things, mainly the following:

1. Added Python code execution. This is done by spinning up a E2B sandbox and executing a jupyter notebook there. Pretty simple stuff
2. Added a command palette. It’s a restyled version of shadcn/ui Command. You’re able to use it to start new chats, and to search through your old chats.
3. Implemented the Search tool. Currently search is pretty primative, just using the AI SDK Exa Tool. I plan on adding a mode that’s kind of like Perplexity’s Deep Research.
4. Completely overhauled the markdown rendering pipeline. We used to use Streamdown. It’s a nice library, but it’s slow. I took a page out of T3 chat’s book and instead, we use the marked library (lexer) to quickly parse the markdown into chunks. We then render those individual chunks, making sure to only re-render the currently streaming chunk. This helps out our performance a ton, as we aren’t rerendering a bunch of markdown that we don’t need.
   In addition to that, I used Shiki as the code highlighter. Typically, code highlighting is a very CPU intensive operation, and it isn’t any different in this case. So instead, we spin up a web worker and run the highlighting in that worker (with WASM as shiki has a WASM module). This way we’re not blocking the main UI thread and killing app performance.

Did a bunch of work, but most of it boils down to the following:

1. Set up the Silo Tanstack Start SDK. Silo is a blob store solution (basically S3 built for the modern web) that I spent the last few weeks working on. You can find it’s flavortown project here.

• I then implemented file attachments. I did this by creating a new attachments table in the database, and adding a relation from attachments -> messages. When the user uploads an attachment, it gets an initial draft status, and once the message is sent, the state is then set to attached and the attachment is associated to the message. Only after that do we gather all the attachments and build the conversation history to pass to the ai sdk. There’s also a bunch of work handling message and attachment drafts that I won’t go into depth here, but essentially every thread can have its own draft etc…

1. Updated all dependencies since I haven’t touched this in a while
2. Thread titles are auto generated using google/gemini-3.1-flash-lite-preview on openrouter. This task is scheduled to run asynchronously after a thread is created
3. Fixed a bunch of ui bugs and lint errors

I spent forever fixing a bug that I introduced in a previous commit that was meant to FIX bugs lmfao

I’ve added a stats display to the llm’s response message. (tokens per second, time to first token, etc)
Also made a bunch of minor ui changes/fixes

Made stream resume more robust and also made the optimistic message sending even faster

Finally tracked down the source of the error over to better-auth… This marks the fourth issue i’ve encountered with better-auth in my time using it that’s taken over 4h to debug.
(The record is https://github.com/better-auth/better-auth/pull/4724, which took well over 10 hours)

I’ve decided to migrate to tanstack start. This is because the app is mostly run fully client-side, and I want a better client-side routing experience. The only real next features I use are server actions and api endpoints.

Currently dealing with a weird bug: https://github.com/TanStack/router/issues/5196

I actually managed to do it… I optimized the send message latency from ~750ms to ~130ms! And on top of that, I implemented optimistic updates. Little writeup on what I actually did (copy-pasted from discord):

i rewrote the backend, the bottleneck was I was inserting sequentially as it’s basically a tree, and i need a parent node’s id before i can insert a child node
the solution i came up with is so dumb
basically I don’t want the client (browser) to control the id I give to threads and messages, as that allows for the user to supply basically whatever string they want
so I instead generate 3 ids, and I cryptographically sign them so I can “attest” to them being truly random. These ids are then passed to the client on page load (and regenerated on-demand)
then when the message is sent to the database (convex), it validates the signature and inserts the required rows with the ids that the client already knows about
all of this is so that when enter is pressed, the client already knows the ids of the thread and the message, so it can optimistically route to /chat/ and add their message onto the screen, making it look faster

tl;dr: I did some dumb sh*t with pregenerating ids on the server to make the browser redirect and show stuff faster before the rows are inserted in the database

Everything works, but it isn’t fast enough. At least it doesn’t look fast enough. Apps like t3.chat optimistically route the user to a new chat before it’s even created, so i’m going to do that too. However you can’t trust data created by the client, so i’m having the server generate and sign a id on page load, and that signed id is validated by convex when a message is sent, so the client and optimistically route itself to the new chat page before its even created.

Migrated from radix-ui to base-ui: https://base-ui.com/

My initial implementation of resumable streams was flawed and did not properly work. I spent two entire days debugging it and fixing it. Now it works!!

I’ve implemented a chat message input bar. It has token estimation, and a feature that highlights tokens. Additionally, any important chat state (i.e submitted, generating, error) is shown in the border of the chat input.

Did a bunch of work on basically everything, including the database schema, auth, and implemented the backend functions and routes, with resumable streams!
https://github.com/vercel/resumable-stream

I’ve implemented a core layout, and spent most of the time building the sidebar. It uses tanstack virtual lists (https://tanstack.com/virtual/latest) to dynamically load the chat history from convex.

Spent a bunch of time debugging weird issues with turborepo, and getting convex to work properly with better-auth (https://labs.convex.dev/better-auth/features/local-install)
Starting to think I should have gone with what i’m usually used to working with (postgres), or using something like clerk/workos

Spent nearly 4 hours figuring out convex and setting it up in a turborepo with better-auth. It finally works!

Currently trying to figure out whether to use convex ents