Fix: Change encoding for better browser support

Turns out x265 isn’t very well supported… It works on firefox everywhere, but only works for chromium on Windows!
This was really annoying to figure out because there were zero error logs, videos just wouldn’t play.
So I updated my transcoder to use x264 instead, so future videos will be supported basically everywhere. Unfortunately this only applies to new uploaded videos, not existing ones

Commit 3374566704

Feature: Warehouse Manager

I made the Warehouse setup process even easier, by creating another project Warehouse Manager that lets you setup warehouse with two commands!

To do this, I added 2 flags to the master server, --init, which stops the server after generating tokens, and --json, which sets the structured log output to JSON for easy parsing.

I also updated README.md and added docs for Warehouse Manager, to get ready for shipping tomorrow!

Documentation
Commit a4b31017dc - README
Commit 6eaf6fa943 - Docs

Documentation update

I updated README with full install and usage information. Guess its time to ship!

Feature: Download + list command

I added a download and list command! Guess I’m basically ready to ship now…

v0.4.0 Binaries
Commit 0e978cd481 - List
Commit 2a40b86431 - Download

Feature: Upload command

You can now upload using the command warehouse files up <bucket> <object> <local file>

Next up is download and list.

v0.2.0 Binaries
Commit cdb607a577

Feature: Start command

I added the start command.

I’ve been trying to make binaries, but I think codeberg releases is down or something.

Commit 09fec44e9b

Warehouse Manager

I’m writing a CLI script that will make setting up my project Warehouse - distributed object storage even easier by making setup a single command, so you don’t have to worry about docker.

Right now I just made it so it automatically runs the docker container with volumes, then listens for the API keys to be logged and store it in a config file.
Next I need to add a command to run it.

I’ll also probably add commands for uploading and downloading files

Feature: Create API keys + updating API keys update the list

I added a button to create API keys, and I made it so updating a key will automatically update it in the list. These two features were quite quick to implement, my previous code had done most of the work already.

Now Warehouse is almost ready to ship! I just need to do some polishing first

v0.37.2 Binaries
Commit 541755f49a - Create API keys
Commit 07a9e856df - Automatic updates

Feature: Update API Key permissions

You can now update API key permissions through the API and web UI!
It took a while because I had to create components to manage bitfields, which is a bit complicated. In the demo video, you can see the computed JSON which has the calculate bitfields

I also made it so you can switch API keys from the web UI

Next I’ll automatically update the list of keys with new permissions

v0.37.0 Binaries
Commit 94f335d07b - Edit API keys
Commit 127a3dfe03 - Change API key

Feature: Delete API keys

You can now delete API keys in the web dashboard. You might notice that you can still access the web dashboard after deleting the API key. This is because although the refresh key has been deleted, the JWT has not expired yet. When the client attempts to refresh the JWT, it will fail.

To immediately invalidate all JWTs, you can restart the master server, which will regenerate the private/public keys.

I should probably set a shorter expiry, though.

v0.36.0 Binaries
Commit a74ef49edf

Feature: View API Keys

You can now view API keys through the web dashboard. It shows every permission each bucket token has.

Next, I will allow adding/deleting API keys, and display which API key you are using.

v0.35.0 Binaries
Commit c75811eb6b

Feature: Authentication with JWT

A previous issue was that Warehouse used one API key for everything, and the web UI gave public access. To improve security, I implemented JWTs with Refresh Tokens.

A JWT is simply a signed string, containing the payload (in this case permissions) and a signature, to verify the JWT was created by a trusted source (the master server). JWTs are stateless, meaning they do not require a database call to verify them, so they are faster then session tokens.

However, JWTs can not be revoked. To avoid security issues, a JWT must expire in a relatively short time, and be refreshed before they expire. I updated the SDK to automatically handle JWT refreshing.

I store permissions in the JWT using flagsets. The permissions available are:

• Admin | You can do anything
• ManageBuckets | Self explanatory
• ManageVolumes | You can view and compact volumes
• VolumeServer | Gives access to volume server methods. Not for user use

You can also give each token permissions to read/write specific buckets, for example I can give the token permission to only read a bucket and its objects, or only write to buckets.

This permission system allows you to minimise the risk of leaking tokens by only allowing certain tokens certain actions. Unfortunately it also slightly complicates the setup process, as seen in the attached video and docs

Next I’ll add manual token creations/deletion.

v0.34.0 Binaries
Commit c5c1092c14
Commit 58c820df3b (Docs update)

Feature: Delete entire buckets

I added a button to delete an entire bucket. I also made it so deleting a bucket will mark all of its objects as deleted.

Very close to shipping now!

I guess next I should implement access policies/proper authorisation

v0.33.0 Binaries
Commit ecf7b5be92

Feature: Create buckets from the web UI

You can now create a bucket directly from the web UI. I also sorted buckets by their size in bytes.

Next I will add a button to delete buckets, which will make the Web UI cover all of the RPCs available!

v0.32.0 Binaries
Commit 7f3bc670fc

Docs: Include docker-compose instructions + technical details

I:

• Included docs for deploying using docker compose
• Included docs for configuring each server using environment variables
• Included docs for technical details like the chunking flow, scaling architecture and the volume file format
• Respected TMPDIR, which allows changing the temporary file location to avoid the invalid cross-device link error with docker
• Removed the test binary from archives, wasn’t suppposed to be included

v0.31.1 Binaries
Commit 1cbf1fd647

Feature: Docker images

I made docker images! Each image is distroless, containing only the binary and the master contains a migration tool. This means that they are really lightweight. Using docker means I don’t have to worry about testing for Windows/MacOS

To do this I had to change a few things:

• Volume and web servers now wait for the gRPC API to become available
• The schema file is now embedded inside the binary, making it simple to create a docker image for

Next, I need to update docs to prioritise docker instead of binaries, and show an example docker compose

v0.31.0 Binaries
Commit f0cd30c883
Docker Hub

Chore: Improve code quality

I improved code quality by breaking my one file webserver into multiple, and I ran gofumpt.

Commit b15af0c86c

Feature: Update object list after an upload.

I made it so uploading an object will update the object list. This was super easy to implement thanks to the HTMX Upsert Extension

I also updated the sort order to be based off of recency

v0.30.1 Binaries
Commit d1a882158d

Feature: Upload objects from the web UI

You can now upload files directly from the web UI. This was actually really fun to implement. Maybe JavaScript isn’t that bad…

Currently the object list doesn’t get updated, that’s up next.

I also made it so you can click to dismiss toasts, and cleaned up some code

v0.30.0 Binaries
Commit 3833fb5fe3 - Object Uploads
Commit 447f4e5b1a - Click to dismiss

Feature: Download button

You can now download objects from the web UI.

In chromium browsers, it uses showSaveFilePicker for a streaming download which keeps memory usage low. In firefox, its unimplemented so a ponyfill is used. It seems this ponyfill might load the entire file into memory before writing to disk, not good.

Next up I’ll add uploads

v0.29.0 Binaries
Commit 57dd731d38

Feature: Object content preview

I re-implemented the object content preview with the new chunks system, by loading the first chunk.

Right now, previews are only available for text content types, but I will add more in the future.

v0.28.0 Binaries
Commit dca17d8099

Feature: Configurable volume sizes + Display wasted bytes

1. I added multiple configuration options for volume servers

• SIZE_PER_VOLUME specifies the maximum size of each volume. Smaller volumes means more, so higher write throughput (volumes are locked on each write)
• MAX_VOLUMES specifies the maximum number of volumes, useful if you don’t want to use all available disk space

1. I displayed the amount of wasted space on volume servers and volumes

v0.27.0 Binaries
Commit 96cde3a9b2

Fix: Volume compaction doesn’t increase size

Was basically due to bad tracking of data offset and size.

I explained my debugging step by step here

Unfortunately this did not fix the bug with invalid keys. I’m struggling to replicate it though…

v0.26.1 Binaries
Commit f4a791206e

Feature: Volume delete button

I added a button to delete objects in the web UI. I encountered an issue using HTMX where id selectors with special characters wouldn’t be targeted properly. To fix this, I had to use a CSS escaping library. Apparently, noone has ever needed to do this in go before, so I had to adapt a JavaScript one to Go.

I also found a couple of issues:

1. Compacting volumes INCREASES file size on fully compacted volumes

• I suspect this has is caused by accidentally writing something I shouldn’t to the file, since it always increases by 13 bytes every time

1. Compactions FAIL due to an “invalid key” - Not exactly sure what causes this, but I suspect something to do with deletion
2. Major slowdowns - perhaps due to using SQLite which locks on write. I could switch to Turso (Easiest) or possibly Bitcask (In memory KV)

Maybe I’ll do a rewrite in rust. Writing safe code is not my strong suite…

v0.26.0 Binaries
Commit 1388f39b83 - Delete object button
Commit eb28b88d8d - CSS Escaping package

Feature: Update usage after volume compaction

The new usage after a compaction is now displayed.

HTMX Multi-Target Updates were being annoying… For some reason OOB swaps never work for me, but hx-partials do.

v0.25.0 Binaries
Commit 5aafa71a71

Feature: Volume compaction button

I added a button to compact volumes (remove deleted data from the volume file).

To do this, I had to write a toast system, which was annoying because I had to not use the native dialog element, because it renders on a separate “top-layer” which means the toasts could not be rendered above it.

Next up I will make it update the displayed usage after compacting

v0.24.0 Binaries
Commit 356d3cd255

Feature: Chunking

I implemented chunking! It wasn’t as hard as I expected.

The way it works is that objects are split into 80 MiB chunks, and clients handle splitting uploads, sending a request for each chunk, then the volume confirms each chunk is uploaded, with the master updating the object locations when all chunks have been uploaded. The client must also reassemble chunks. Chunking also spreads files more evenly across volumes

I updated the SDK to handle this automatically, and concurrently so multiple chunks can be downloaded/uploaded at the same time.

Some features were removed:

• Per volume object count, doesn’t make sense anymore because objects are spread over volumes
• Object preview - I just need to update the logic to reassemble the file.
• Presigned PUT. Now you need to know the size of the file before uploading, so I can calculate the number of chunks

This update massively decreases memory usage, but does increase CPU usage depending on how many chunks get downloaded/uploaded in parallel. Large files get uploaded faster, and small files probably get uploaded a tiny bit more slowly. I need to do benchmarks to know for sure, and find out what number of concurrent chunk uploaders/downloaders is optimal.

There’s probably still some optimisation to do to avoid copying and stuff. And I need to update the docs

v0.23.0 Binaries
Commit 8efb1d44a9

Feature/Fixes: Track volume free space

Now the master server keeps track of volumes free space, so it can check if there is enough free space before telling a client to put to it, otherwise checking the next volume.

This update was actually hell. Everything broke :)

What I fixed:

• Volumes were properly refreshed after updates. This made writes after compaction work.
• Volume writes were stored in a buffer before committing to the file, which avoids breaking the entire file when compacting.
• Volume servers gracefully shuts down when master server shuts down.
• Error handling with tracking wasted bytes.
• Probably other things. My brain is so fried because of this update.

Unfortunately memory usage is increased as a result of loading the whole data into a buffer and Go GC taking ages to clean it up. Luckily I have 32GB of RAM, my old PC would have crashed…

I don’t think there’s a way to fix the memory usage apart from chunking the files into far smaller pieces - that is not gonna be fun to implement…

Also updated Docs with a custom footer

v0.22.0 Binaries
Commit d4e827b03b
Commit 815b138a55 (Docs)

Fixed Memory Leak!

I found a memory leak where every object write never got released from memory 😬
Fixed this by not using byte arrays and instead an io.ReadCloser, and writing directly to the file.
Can’t believe I wasn’t doing this before…

I also updated the docs to include object RPCs.

v0.21.0 Binaries
Commit 9246b45848 - Memory Leak
Commit b9c85a3416 - Docs

Feature: Documentation

I wrote docs using Hugo. It includes how to self host, and how to use the new Go SDK (not completed though). I also fixed an issue where bucket deletion wouldn’t work.

Docs

v0.20.1 Binaries
Commit 63c5d29d75 - Docs
Commit cce395e1a5 - Bucket deletion fix

Feature: Golang SDK

I wrote a SDK for go (a wrapper around the generate gRPC client) with nice features like:

• Automatically setting up a client for every service (admin, buckets, objects and volumes)
• Automatically creating a context which passes authorisation to the master server
• The super nice features
  • Put an object in 1-3 lines instead of the previous ~50 (due to having to create a policy and then setting up the request and handling errors)
  • Get an object in 1-3 lines instead of ~28 (same reason as above)

I also fixed a bug where you couldn’t start the master server without migrating the schema

Next up I’ll probably write some proper docs with hugo, or add chunking support (or not, its a lot of effort), or add a compact button to the volumes list

v0.20.0 Binaries
Commit a516de71b7
Commit 00633bf003

Godoc Here

For you: I made Warehouse easier to run

I did this by:

1. Actually including binaries for the volume+web servers (oops)
2. Stopped harcoding the server auth token (double oops)
3. Add an option (--schema) to the server binary that allows for automatically applying database schema, and include the schema in releases
4. Updated README.md with quick start instructions

Releases are available at Codeberg if you want to try it yourself. These include the README with install instructions.

Commit 464f0e739a

Feature: Volumes list

I added an action to list volumes of a server, which shows usage and object count of each volume.

I need to display the wasted space count, and add an option to compact the volume.

I also fixed a bug where terminating the volume server before it had synced the needle locations to disk would result in the data being inaccessible. I fixed this by handling interrupts and syncing before shutdowns.

You might notice that volume 1 has 29 bytes of usage, but 0 objects. This is because the object has been flagged as deleted, but is still in physical storage. The reason for this is because volumes are an append only file, meaning deleted and duplicate files are kept until compaction.

v0.18.0 Binaries

Changelog:

• Commit d0213e80b5 - List volumes in Web UI
• Commit 70f8af63be - Implement the RPC for listing volumes
• Commit 9c73be2d12 - Graceful shutdowns of volume servers

Feature: Object preview

I added a details action that shows a preview of the object, and displayed the last updated date.

Right now previews only show for text/* content types, but I will add support for more, like a specific renderer for CSV, JSON, images and videos, with a toggle between raw and formatted.

v0.17.0 Binaries
Commit c4ccca7a84

Feature: View objects

You can now view a buckets objects in the web UI, including total size and count.

The object viewer is flat for now, meaning there are no virtual folders and everything appears at the top level. I will implement this later.

Next up is object actions.

v0.16.0 Binaries
Commit 433c485c2f

Feature: Buckets page

I added a page to view all buckets, with infinite scroll. It contains the object count and size taken by the bucket.

v0.15.0 Binaries
Commit cd9a66be1d

Feature: Total volume server stats

I added total stats to the volume server page

v0.14.0 Binaries
Commit 9721ea5331

Feature: Display volume server usage

I made the usage of volume servers available (through /usage) and displayed it in the Web UI, with a since meter.

This update took longer than expected, because styling meters is hell and I didn’t even end up using the built in ones. I was also having CORS issues :)

v0.13.0 Binaries
Commit ca24d5fee4

Fixes: Master server untracks volumes when the volume server disconnects, and volume server crashes when the master disconnects

This fixes the issue where server where still marked as offline even after disconnecting. This was fixed by switching from a unary (one time) request to bidirectional stream, where disconnects can be handled.

I also:

• Upgraded to HTMX v4
• Fixed an issue where (DEGRADED) was shown instead of (OFFLINE) when all volume servers are online
• Made border colours and radiuses consistent between pages

v0.12.0 Binaries
Commit cfa141bed9
Commit 6f95c42217

Feature: Volume Server

I added a page with a list of all volume servers, their status, volume count, and capacity.

Next up is:

• Total used space
• Total volume server count / volume count / capacity
• Volume servers to be marked as offline when they disconnect

v0.11.0 Binaries
Commit cf265572c8

Feature: Bucket and object counts

Web UI now displays the accurate bucket/object count. I should probably lazy load this though

v0.10.0 Binaries
Commit 5396408a38 (Object count)
Commit 2526a33759 (Bucket count)

Feature: Volume counts

I connected the website to the master API and added a RPC to count volumes/volume servers.

Commit 77fbbbe0e8

Feature: Basic Admin UI

I implemented the basics of the Admin UI, using Golang, Templ, and TailwindCSS.

The numbers you see are made up, there is currently no integration with the master API. The colour also adapts to the status, if all servers are offline the colour is red and amber if some are offline.

Commit 59978babed
Commit ca97fe6061

Feature: Min/max file size and content type validation

You can now specify a min/max size, or a content type prefix to enforce.

v0.9.0 Binaries
Commit fd3f097deb

Chore: Improve Code Quality

I improved the code quality of the volume server by not making it one giant file

v0.8.1 Binaries
Commit 72de9c7ac9

Feature: Get objects

I reimplemented object getting with the new master volume system, so now it responds with a url and JWT for direct access to needles.

v0.8.0 Binaries
Commit 2bc2c6a9d2

Feature: Volume Compaction Supported Again

I reimplement the admin.CompactVolume RPC to fit with the new one master many volumes architecture.

v0.7.2 Binaries
Commit cba82f930e

Fix: Objects are actually deleted

Turns out I forgot to actually delete the object from the master index after deleting from volume servers (oops)

v0.7.1 Binaries
Commit 08b19ced93

Major Feature: Remote Volume Servers

This is a big feature that allows for scaling horizontally. Each volume server connects to the master to initialise and then starts a REST API which provides direct access to needle management.

One difference from normal S3 is that every request is now pre-signed, and you have to communicate directly with each volume server.

How horizontal scaling works

How a volume server connects to the master:

How a put (overwrite) works:

Problems

• Multiple requests - this is still a performance improvement over proxying data, but it makes DX works. I need to write an SDK that makes uploading 1 simple function call.
• Volume compaction - the admin RPC is unimplemented in this version, I need to add an endpoint to the volume servers
• Object getting - this is implemented on the volume server, however you need to know the needle and volume id. This also does not require authentication right now. I will implement this next.
• Content type/object size limits are not verified.
• Configuration is hardcoded in the volume server
• Code quality
• Error handling

I would’ve fixed all these problems, but this devlog was getting long enough :)

Commit 3bf51c25ed
v0.7.0 Binaries

Feature: Volume Compaction

One problem with using a single, append only, volume file is that deleted files and duplicates are not removed. Over time, this can waste a lot of storage. To fix this, I wrote a compaction tool, which reads the volume file, scanning each needle. If a needle is flagged as deleted, it is ignored and any previous needle with the same id is removed. I also only keep a copy of the latest needle, to keep the latest version and remove duplicates. Then, for each needle, the data is copied to a new volume file, then the old file is replaced with the new clean data.

I added an admin RPC to manually trigger compaction, and a utility to retrieve what proportion of the volume file is wasted.

v0.6.0 Binaries
be0acaa5bf

Big Feature: Volume Files

One problem with my storage server is that each object was stored as a separate file on disk.This means each file retrieval is actually multiple disk operations, which can slow down retrieval

Heavily based on Facebook’s Haystack Paper, I wrote a storage system that uses one large file for many smaller objects, lowering the number of disk operations to read one object.

• Each object (file) is stored as a needle
• A write works by appending the needle to a data file
• A needle contains a small amount of metadata, and the data itself:
  • The ID (8 bytes)
  • The flags (whether or not the file has been deleted) (1 byte)
  • The size of the data (4 bytes)
  • The data itself
  • The checksum of the data (using the CRC hashing algorithm) (4 bytes)
• Only 17 bytes are used for metadata, compared to XFS inodes using 536 bytes
• The size and offset of each needle is stored in a kv store, and persisted to disk
• A read retrieves the size and offset of the needle from kv storage, reads the file at the offset, and decodes each field. If the flag is 1, the file is deleted and an error is returned. The checksum of the data is calculated again and compared to the stored checksum
• A delete sets the flag of the needle to 1, and removes the metadata from the kv store

There are some (fixable) problems with this approach:

• Deleted/Duplicate files take up storage. I need to write a compression system, creating a new data file and only writing non-deleted and the first duplicate to the new data file
• I have not written code to recreate the metadata index from the data store. If the metadata index is lost or corrupted, the metadata would have to be recovered by hand.

This work will allow me write volume servers, which manage multiple volumes, to allow for horizontal scaling and redundancy.

v0.5.0 Binaries
00ae00ba93

Feature: Object Retrieval

You can now retrieve files using the gRPC API. I still need to implement streaming puts/gets.

Note that the shown data field is encoded using base64, the actual data has been stored correctly

970d311e99
v0.4.0 Binaries

Feature: Object deletion

You can now delete objects. Unfortunately not all time was tracked.

I also added a Makefile entry for evans to quickly set up a connection for tests.

90bb08ff79
v0.3.1 Binaries

Feature: Object Creation

Here’s all the changes I made:

• Creating a bucket creates a buckets and backups folder on disk
• Buckets no longer have an ID, solely identifiable by name
• Remove unnecessary stuff and don’t try to restore backups that don’t exit ac054d4c96

And the features I added:

• Object creation (Unary/Single Request - Optimal for small files, but I need to add a streaming version for large files)
• Free space check, don’t start writing if there’s not enough space. (annoying to do, because of windows support)
• Backups - If a file already exists, create a backup and ensure all steps succeed or restore the backup

5dc05e56e4
v0.2.1 Binaries

Feature: Delete Bucket

You can now delete buckets. This is the bucket CRUD basics done, so I can move onto object CRUD

1246d3fdf6
v0.1.0 Binaries

Rewrite + Automatic releases

• I moved from a REST API to a gRPC API, because of all the time gRPC saves. Switching to gRPC greatly reduced the lines of code.
• I made the Buckets.Get endpoint take a name, instead of an ID.
• I added goreleaser, to automatically build the server and distributes it on Codeberg
• I required an API key to use RPCs

New endpoint

I added an endpoint to get a buckets information. I also wrote a function to stringify data and handle errors, to remove duplicated code.

I’m planning on rewriting the API with gRPC, because honestly I cannot be bothered with manually stringifying and parsing data. Protobuf is also way more efficient than JSON. It also allows me to generate clients for many languages automatically

76b8b61dd9

New endpoint

I added a GET /buckets endpoint, which allows you to list buckets, with a limit and offset

0d7deb0ccc

Project Restructure + Bucket name validation

I moved the go files to cmd/server, and moved utility functions into separate files.
I also added environment variable configuration for setting the server port and database location.
For bucket name validation, I used regex to only allow characters a-z, 0-9, ‘.’ and ‘_’, with a max length of 32

a38e257c09

Warehouse

In this devlog, I set up dependencies for Warehouse, and write a REST API with a single POST /buckets endpoint, which creates an entry in the SQLite Database

Motivation

I used S3/SeaweedFS for my project Watchtower. But I discovered a few problems:

• I needed a Message Broker, like RabbitMQ to handle uploads. Some problems were:
  • I had to notify each queue of the upload from the API manually.
  • Clean up is hard. You have to wait for each queue to finish, and then remove it from S3.
  • What if a queue should stop processing, like if NSFW content is detected? How would the queue detect this to avoid wasting time processing?
• I needed to write a custom CDN, due to the lack of pre-signed prefix polices. E.g. an HLS video with multiple files could not be served directly from S3, because a pre-signed get policy only allows access to one file
• Too many services. I have to run 1. S3, 2. RabbitMQ, 3. CDN, and 3 separate queues - What if this could all be one service?

Feature Goals

• Basic Bucket CRUD
• Basic Object CRUD
• Pre-signed Policies
• Pre-signed Prefix Policies
• Web UI
• Authentication
• Graph based upload processing
• FFmpeg integration
• TensorFlow integration
• Golang client
• TypeScript client

NSFW Filter for thumbnails

Thumbnails are now automatically blocked if anything NSFW is detected.

I also moved the clean-up step of jobs to a defer, so errored jobs will still be cleaned up.

I know the logs don’t look the nicest and aren’t consistent, I’ll be rewriting logs to use structured logs soon.

53f859535b

UI/UX Update

Videos are now given a background on hover, and the entire box is clickable to take you to the video

b9bd164d73

UI Update: Infinite Scroll + Loading indicator for suggested videos

I added infinite scroll for suggested videos. This was already implemented in the API, but I guess I forgot to implement it on the website. I also added a loading indicator.

Note: The loading is intentionally slowed down in the demo video, just so you can actually see the indicator

8924fb5425

UI Update: New layout options

I added layout options to suggested videos - a ‘comfy’ and ‘compact’ layout.
Your selected layout is consistent between page refreshes and different videos.

0c6e2273a8

UI Update

I updated to video list to readd the separator, use flex wrapping and lower the font size.

Unfortunately, when the flex element wraps, the separator still shows. I spent around an hour trying to fix this using container queries. Container queries are awful to use with heights. Never again.

2b7319b412

Testing + Code quality

I wrote a test case for users.GetUserById. Since a lot of validation was the same from the test for users.Get, I moved that logic into a separate function to be used for both test cases.

a25a0f14fe

The users.Create RPC returned a uint64 ID instead of an int64, so I updated the code to use int64 instead, and removed any no longer necessary conversions to int64.

a683c56d0d

Testing + Code quality

I added a test case for the Users.Get RPC, which verifies that all returned fields are as expected.

When I was writing this, I realised that type for User contained an email, even though it was not sent on most emails, which could lead to confusion using any RPC that returns a User.

To fix this, I added a separate type that contained the users email (UserWithEmail) that is used by RPCs that need the users email, and I removed the email field from the User type.

bc8cc01aa0

Testing

I wrote a test case for the GetFollowing RPC.

This test case has a similar setup to the GetFollowers RPC (create two users and follow one), so I split the logic into a separate function to reduce duplicated code.

The test case ensures:

• Only one following user is returned
• The returned following user has the correct username and ID
• The CreatedAt timestamp is valid

5d7ed806a2

Testing

I added a test case for the GetFollowers RPC - this is passing :)

Honestly, writing tests is boring as hell. I’ve only written 37% of the tests for the Users service… and there’s around three other services to test for.

d0cd784d54

Bug Fixes

In my previous devlog I discovered users were able to follow themselves.

In this devlog I:

• Rewrote the follow/unfollow RPCs so more logic is done with go than SQL
• Blocked users from following themselves (test case passes)
• Added a nicer error message when users try to follow a nonexistent users

761211acb0

Testing

I added test cases for:

• Following yourself (should not be allowed)
• Following someone (allowed)
• Unfollowing someone (allowed)
• Following a nonexistent person (should not be allowed)

As a result of this, I have found that:

• You can follow yourself through the API
• Error handling should be improved for follow RPCs
• I should improve my testing solution, right now I have to manually log if it passes, and fails are just logged as an error. I should make this automatic

These should be fixed in the next update :)

c7b3ea8e3b

Code quality + Testing

I improved code quality by:

• Check grammar/spelling. I fixed a lot of grammar and spelling errors using harper-ls and harper-cli
• Stricter formatting. I ran gofumpt, which has a lot stricter formatting
• Linting. This took by far the longest time - I ran golanglint-ci, and one by one, fixed every single lint issue. This included many unhandled errors (that wouldn’t of caused a problem, just weren’t being logged)

a93729e78c

I also updated testing by:

• Cleaning up code
• Writing test cases for different password lengths

a5f74ef1f5

Feature: Umami tracking

I added Umami, a privacy focused analytics service to Watchtower. As of now it only tracks page views, but in the future I plant to improve integration by adding events for video likes/dislikes, follows, video creation etc.

I also considered Rybbit, because it has more features like error tracking and web vitals, but I could not get a self hosted set up working. Maybe in the future.

• 4ebc86e022

Feature: Comment deletion

I added the ability to delete comments. This can only be done by the user who created the comment, or any admin account

• 19aac7f632

UX Improvement

After updating the video upload script, an issue was created where errors weren’t displayed properly. I have updated the website to clearly show errors

• 8dd0cad1dd

Bug fixes

On the second test case I wrote I found a bug. Not sure if this is a good thing, but at least it means testing is worth it.

• 686d0d4b70 Add test cases to make sure duplicate emails and usernames are blocked
• e6562c8aae Fix bug discovered above where errors were not
  handled correctly
• 2266cef6fc Allow videos down to 240 pixels to be uploaded, and handle videos that are smaller than 240px properly, by rejecting them and showing an error message instead of transcoding them badly

Testing!

Yes - I’ve finally gotten around to writing tests!

Before I was relying on manual testing, by visiting the website - I’ve probably missed quite a few bugs. But now I’ve started writing automated tests for the API.

I have set up the tests so an ephemeral API is set up, along with temporary PostgreSQL, S3, RabbitMQ, and Valkey (redis). These automatically start when you run the tests, and stop when the tests have finished. I did this using the Golang SDK for docker-compose, and waiting for the API to start. After, migrations are automatically applied to the testing database.

As of now, there is only one test case, but I will setup more and more. I plan to implement E2E testing for the website in the future.

Web Update

• af75360ffa Updated formatting of suggested video details
• 0a782728ce Fix live chat dockerfile and deploy to production

Full deployment

The full version of Watchtower is deployed on my new server (my old pc) which means you can now upload videos! These will be transcoded to multiple different resolutions, and analyzed to block NSFW content. The NSFW detection may be a bit inaccurate so make sure to use the report and appeal features

Changes

• 472ff840c5 Fix transcoder dockerfile
• 1141fb2aec Add analytics dockerfile
• 15c7b46db9 Video duration’s are hidden if they are zero (happens in demo environment)
• Multiple Various dockerfile fixes
• Multiple Switch to old golang version of video analyser
• d58a78215d Sidebar links are given a semitransparent background

Polishing

I refactored some ridiculous code that was inline, and made it a separate function.
This will reduce allot of bandwidth because now the JavaScript is in one place instead of in every single comment

View Commit

Polishing

Spent some time polishing things, and working on docker images, for a move to another server

• 612a3cecad I removed the dependency on go-sqlite3 which was leftover from when I used sqlite as the database. The error messages for duplicate usernames are now prettier
• fc2f524b21 I added docker images
• e6394fea10 I gave the icons when changing/uploading thumbnails a background, so the contrast is still OK when the uploaded image is light or dark
• 1427c28e33 I allowed the recommended videos to be clicked anywhere for better UX, and added a background on hover

If you spot anything that needs polishing - please let me know!

iOS App: Update

1. I added sort options, you can now select between latest, popular, recommended and trending. This took a fair amount of work, I had to guess the type of the gRPC client so I could pass it between Views. This is really hard when you don’t have a functional LSP. I also had to rewrite the way the gRPC client was created so the connection doesn’t close.
2. Moved thumbnails to a separate component for clarity

• 03ee41ded7
• 7944685210

iOS Update

I updated the styling of the onboarding page

1 Commit

Web Update

I added suggested videos right in the view video page.

I also simplified a lot of repeated code, moved an API endpoint from the Users service to the Videos service, and made it so if there is only one comment on a video it says 1 Comment instead of 1 Comments - Crazy right?!

View 3 commits

iOS Update

1. I allowed the user to configure the API + S3 Endpoints
2. I added the user avatar, username, and view count to the video list

View the code

iOS App!

I started working on an iOS app using swift!

This is my first time ever using swift, so it took quite a while.

Initially I tried using NativeScript Svelte, then React Native. Its pretty hard to get gRPC working with them, needing a proxy or something for the gRPC server, so I switched to swift instead.

Xcode (Apples IDE) doesn’t run on linux, so I had to install xtool and vscode/zed. I have to rebuild everytime I make a change, and the builds are pretty slow, but Swift is a really nice language and I may use it in the future

More flag reasons

I added a reason to flag videos that is not nudity.

View the commit

Video Reporting

I finished video reporting, now admins can flag a reported video in the admin dashboard

Reports: Progress

Users can now report videos, and admin page was added to view reports

Next up I need to add actions like removing reports and flagging videos

View the commits

Moderation Appeal

You can now request an appeal if your video gets blocked.

There is a new page in the admin dashboard to view appeals

Admins can either reject or remove appeals

Rejecting will not allow the user to create another appeal

View The Code

Rewrite

I rewrote the video analyzer (NSFW detection) from Go to TypeScript, because the go package used for NSFW broke.
It might be slower, but at least it works

Can’t really have many screenshots for this update…

I also reduced the amount of things you need to install by using go tools,
where instead of installing the binary I run go get -tool and call it using go tool
More work for me, less work for someone trying to set this up themselves.

I also reinstalled my OS 2 times, so thats why this update took a while

You can view the changes on Codeberg

Live chat

I added chats to live streams! (you can chat even when the stream is offline
This was implemented using websockets (this is like the 8th microservice)

I made it so scrolling pauses the chat, so you can read messages.
The colour of the usernames are based on the hash of your username, I should probably let you pick a colour though.

Live streams fix

I fixed live streams by writing a proxy to the stream server

Also did a few bug fixes
View The Changes

Replaced dropzone.js with a custom upload script, so uploading matches the theme better
(4h of work because everything decided to break)
(ps don’t use RustFS, it is buggy and vibecoded)

View the code

Added a live indicator to profile pages

Big Update - Live Streams!

I added live streams

View Changes

+ 502 Additions
- 73 Deletions

I added trending, and did a few other things, like improving security, working on a demo environment, improving code, fixing a bug and cleaning up processed thumbnails

View 5 changes

I added view counter (a view counts once every six hours per users) and refactored the code!
View Changes

Admin Dashboard Improvements

I added a user management page that lists all users (with scrolling to load more) and allows flag management (Admin/Verified flags)

View the change

I should add search next

I added an admin dashboard with user/video count analytics.
(I spent a lot of time perfecting how the graph looks)
View the change

Added milliseconds to durations that show on hover
Accidentally deleted the last devlog

Improved UX by redirecting back to profile page after deleting a video (with a refresh) so deleted videos don’t show after deletion
Also changed some endpoints to use proper http semantics

I hid private/unprocessed videos in the recommended feed
Also had problems with atlas migrations which took hours to fix :)

I added a most popular sort option to the homepage of WatchTower.
I am planning to add a trending option as well.
I should probably remove some of the duplicated logic though and replace /videos/latest and /videos/popular with /videos/{sort_type}

I added a “latest” sort option to my video sharing platform
I also fixed a bug where usernames wouldn’t show on recommended videos
Next up is sorting by popularity