Fixed Memory Leak!

I found a memory leak where every object write never got released from memory 😬
Fixed this by not using byte arrays and instead an io.ReadCloser, and writing directly to the file.
Can’t believe I wasn’t doing this before…

I also updated the docs to include object RPCs.

v0.21.0 Binaries
Commit 9246b45848 - Memory Leak
Commit b9c85a3416 - Docs

Feature: Documentation

I wrote docs using Hugo. It includes how to self host, and how to use the new Go SDK (not completed though). I also fixed an issue where bucket deletion wouldn’t work.

Docs

v0.20.1 Binaries
Commit 63c5d29d75 - Docs
Commit cce395e1a5 - Bucket deletion fix

Feature: Golang SDK

I wrote a SDK for go (a wrapper around the generate gRPC client) with nice features like:

• Automatically setting up a client for every service (admin, buckets, objects and volumes)
• Automatically creating a context which passes authorisation to the master server
• The super nice features
  • Put an object in 1-3 lines instead of the previous ~50 (due to having to create a policy and then setting up the request and handling errors)
  • Get an object in 1-3 lines instead of ~28 (same reason as above)

I also fixed a bug where you couldn’t start the master server without migrating the schema

Next up I’ll probably write some proper docs with hugo, or add chunking support (or not, its a lot of effort), or add a compact button to the volumes list

v0.20.0 Binaries
Commit a516de71b7
Commit 00633bf003

Godoc Here

For you: I made Warehouse easier to run

I did this by:

1. Actually including binaries for the volume+web servers (oops)
2. Stopped harcoding the server auth token (double oops)
3. Add an option (--schema) to the server binary that allows for automatically applying database schema, and include the schema in releases
4. Updated README.md with quick start instructions

Releases are available at Codeberg if you want to try it yourself. These include the README with install instructions.

Commit 464f0e739a

Feature: Volumes list

I added an action to list volumes of a server, which shows usage and object count of each volume.

I need to display the wasted space count, and add an option to compact the volume.

I also fixed a bug where terminating the volume server before it had synced the needle locations to disk would result in the data being inaccessible. I fixed this by handling interrupts and syncing before shutdowns.

You might notice that volume 1 has 29 bytes of usage, but 0 objects. This is because the object has been flagged as deleted, but is still in physical storage. The reason for this is because volumes are an append only file, meaning deleted and duplicate files are kept until compaction.

v0.18.0 Binaries

Changelog:

• Commit d0213e80b5 - List volumes in Web UI
• Commit 70f8af63be - Implement the RPC for listing volumes
• Commit 9c73be2d12 - Graceful shutdowns of volume servers

Feature: Object preview

I added a details action that shows a preview of the object, and displayed the last updated date.

Right now previews only show for text/* content types, but I will add support for more, like a specific renderer for CSV, JSON, images and videos, with a toggle between raw and formatted.

v0.17.0 Binaries
Commit c4ccca7a84

Feature: View objects

You can now view a buckets objects in the web UI, including total size and count.

The object viewer is flat for now, meaning there are no virtual folders and everything appears at the top level. I will implement this later.

Next up is object actions.

v0.16.0 Binaries
Commit 433c485c2f

Feature: Buckets page

I added a page to view all buckets, with infinite scroll. It contains the object count and size taken by the bucket.

v0.15.0 Binaries
Commit cd9a66be1d

Feature: Total volume server stats

I added total stats to the volume server page

v0.14.0 Binaries
Commit 9721ea5331

Feature: Display volume server usage

I made the usage of volume servers available (through /usage) and displayed it in the Web UI, with a since meter.

This update took longer than expected, because styling meters is hell and I didn’t even end up using the built in ones. I was also having CORS issues :)

v0.13.0 Binaries
Commit ca24d5fee4

Fixes: Master server untracks volumes when the volume server disconnects, and volume server crashes when the master disconnects

This fixes the issue where server where still marked as offline even after disconnecting. This was fixed by switching from a unary (one time) request to bidirectional stream, where disconnects can be handled.

I also:

• Upgraded to HTMX v4
• Fixed an issue where (DEGRADED) was shown instead of (OFFLINE) when all volume servers are online
• Made border colours and radiuses consistent between pages

v0.12.0 Binaries
Commit cfa141bed9
Commit 6f95c42217

Feature: Volume Server

I added a page with a list of all volume servers, their status, volume count, and capacity.

Next up is:

• Total used space
• Total volume server count / volume count / capacity
• Volume servers to be marked as offline when they disconnect

v0.11.0 Binaries
Commit cf265572c8

Feature: Bucket and object counts

Web UI now displays the accurate bucket/object count. I should probably lazy load this though

v0.10.0 Binaries
Commit 5396408a38 (Object count)
Commit 2526a33759 (Bucket count)

Feature: Volume counts

I connected the website to the master API and added a RPC to count volumes/volume servers.

Commit 77fbbbe0e8

Feature: Basic Admin UI

I implemented the basics of the Admin UI, using Golang, Templ, and TailwindCSS.

The numbers you see are made up, there is currently no integration with the master API. The colour also adapts to the status, if all servers are offline the colour is red and amber if some are offline.

Commit 59978babed
Commit ca97fe6061

Feature: Min/max file size and content type validation

You can now specify a min/max size, or a content type prefix to enforce.

v0.9.0 Binaries
Commit fd3f097deb

Chore: Improve Code Quality

I improved the code quality of the volume server by not making it one giant file

v0.8.1 Binaries
Commit 72de9c7ac9

Feature: Get objects

I reimplemented object getting with the new master volume system, so now it responds with a url and JWT for direct access to needles.

v0.8.0 Binaries
Commit 2bc2c6a9d2

Feature: Volume Compaction Supported Again

I reimplement the admin.CompactVolume RPC to fit with the new one master many volumes architecture.

v0.7.2 Binaries
Commit cba82f930e

Fix: Objects are actually deleted

Turns out I forgot to actually delete the object from the master index after deleting from volume servers (oops)

v0.7.1 Binaries
Commit 08b19ced93

Major Feature: Remote Volume Servers

This is a big feature that allows for scaling horizontally. Each volume server connects to the master to initialise and then starts a REST API which provides direct access to needle management.

One difference from normal S3 is that every request is now pre-signed, and you have to communicate directly with each volume server.

How horizontal scaling works

How a volume server connects to the master:

How a put (overwrite) works:

Problems

• Multiple requests - this is still a performance improvement over proxying data, but it makes DX works. I need to write an SDK that makes uploading 1 simple function call.
• Volume compaction - the admin RPC is unimplemented in this version, I need to add an endpoint to the volume servers
• Object getting - this is implemented on the volume server, however you need to know the needle and volume id. This also does not require authentication right now. I will implement this next.
• Content type/object size limits are not verified.
• Configuration is hardcoded in the volume server
• Code quality
• Error handling

I would’ve fixed all these problems, but this devlog was getting long enough :)

Commit 3bf51c25ed
v0.7.0 Binaries

Feature: Volume Compaction

One problem with using a single, append only, volume file is that deleted files and duplicates are not removed. Over time, this can waste a lot of storage. To fix this, I wrote a compaction tool, which reads the volume file, scanning each needle. If a needle is flagged as deleted, it is ignored and any previous needle with the same id is removed. I also only keep a copy of the latest needle, to keep the latest version and remove duplicates. Then, for each needle, the data is copied to a new volume file, then the old file is replaced with the new clean data.

I added an admin RPC to manually trigger compaction, and a utility to retrieve what proportion of the volume file is wasted.

v0.6.0 Binaries
be0acaa5bf

Big Feature: Volume Files

One problem with my storage server is that each object was stored as a separate file on disk.This means each file retrieval is actually multiple disk operations, which can slow down retrieval

Heavily based on Facebook’s Haystack Paper, I wrote a storage system that uses one large file for many smaller objects, lowering the number of disk operations to read one object.

• Each object (file) is stored as a needle
• A write works by appending the needle to a data file
• A needle contains a small amount of metadata, and the data itself:
  • The ID (8 bytes)
  • The flags (whether or not the file has been deleted) (1 byte)
  • The size of the data (4 bytes)
  • The data itself
  • The checksum of the data (using the CRC hashing algorithm) (4 bytes)
• Only 17 bytes are used for metadata, compared to XFS inodes using 536 bytes
• The size and offset of each needle is stored in a kv store, and persisted to disk
• A read retrieves the size and offset of the needle from kv storage, reads the file at the offset, and decodes each field. If the flag is 1, the file is deleted and an error is returned. The checksum of the data is calculated again and compared to the stored checksum
• A delete sets the flag of the needle to 1, and removes the metadata from the kv store

There are some (fixable) problems with this approach:

• Deleted/Duplicate files take up storage. I need to write a compression system, creating a new data file and only writing non-deleted and the first duplicate to the new data file
• I have not written code to recreate the metadata index from the data store. If the metadata index is lost or corrupted, the metadata would have to be recovered by hand.

This work will allow me write volume servers, which manage multiple volumes, to allow for horizontal scaling and redundancy.

v0.5.0 Binaries
00ae00ba93

Feature: Object Retrieval

You can now retrieve files using the gRPC API. I still need to implement streaming puts/gets.

Note that the shown data field is encoded using base64, the actual data has been stored correctly

970d311e99
v0.4.0 Binaries

Feature: Object deletion

You can now delete objects. Unfortunately not all time was tracked.

I also added a Makefile entry for evans to quickly set up a connection for tests.

90bb08ff79
v0.3.1 Binaries

Feature: Object Creation

Here’s all the changes I made:

• Creating a bucket creates a buckets and backups folder on disk
• Buckets no longer have an ID, solely identifiable by name
• Remove unnecessary stuff and don’t try to restore backups that don’t exit ac054d4c96

And the features I added:

• Object creation (Unary/Single Request - Optimal for small files, but I need to add a streaming version for large files)
• Free space check, don’t start writing if there’s not enough space. (annoying to do, because of windows support)
• Backups - If a file already exists, create a backup and ensure all steps succeed or restore the backup

5dc05e56e4
v0.2.1 Binaries

Feature: Delete Bucket

You can now delete buckets. This is the bucket CRUD basics done, so I can move onto object CRUD

1246d3fdf6
v0.1.0 Binaries

Rewrite + Automatic releases

• I moved from a REST API to a gRPC API, because of all the time gRPC saves. Switching to gRPC greatly reduced the lines of code.
• I made the Buckets.Get endpoint take a name, instead of an ID.
• I added goreleaser, to automatically build the server and distributes it on Codeberg
• I required an API key to use RPCs

New endpoint

I added an endpoint to get a buckets information. I also wrote a function to stringify data and handle errors, to remove duplicated code.

I’m planning on rewriting the API with gRPC, because honestly I cannot be bothered with manually stringifying and parsing data. Protobuf is also way more efficient than JSON. It also allows me to generate clients for many languages automatically

76b8b61dd9

New endpoint

I added a GET /buckets endpoint, which allows you to list buckets, with a limit and offset

0d7deb0ccc

Project Restructure + Bucket name validation

I moved the go files to cmd/server, and moved utility functions into separate files.
I also added environment variable configuration for setting the server port and database location.
For bucket name validation, I used regex to only allow characters a-z, 0-9, ‘.’ and ‘_’, with a max length of 32

a38e257c09

Warehouse

In this devlog, I set up dependencies for Warehouse, and write a REST API with a single POST /buckets endpoint, which creates an entry in the SQLite Database

Motivation

I used S3/SeaweedFS for my project Watchtower. But I discovered a few problems:

• I needed a Message Broker, like RabbitMQ to handle uploads. Some problems were:
  • I had to notify each queue of the upload from the API manually.
  • Clean up is hard. You have to wait for each queue to finish, and then remove it from S3.
  • What if a queue should stop processing, like if NSFW content is detected? How would the queue detect this to avoid wasting time processing?
• I needed to write a custom CDN, due to the lack of pre-signed prefix polices. E.g. an HLS video with multiple files could not be served directly from S3, because a pre-signed get policy only allows access to one file
• Too many services. I have to run 1. S3, 2. RabbitMQ, 3. CDN, and 3 separate queues - What if this could all be one service?

Feature Goals

• Basic Bucket CRUD
• Basic Object CRUD
• Pre-signed Policies
• Pre-signed Prefix Policies
• Web UI
• Authentication
• Graph based upload processing
• FFmpeg integration
• TensorFlow integration
• Golang client
• TypeScript client

NSFW Filter for thumbnails

Thumbnails are now automatically blocked if anything NSFW is detected.

I also moved the clean-up step of jobs to a defer, so errored jobs will still be cleaned up.

I know the logs don’t look the nicest and aren’t consistent, I’ll be rewriting logs to use structured logs soon.

53f859535b

UI/UX Update

Videos are now given a background on hover, and the entire box is clickable to take you to the video

b9bd164d73

UI Update: Infinite Scroll + Loading indicator for suggested videos

I added infinite scroll for suggested videos. This was already implemented in the API, but I guess I forgot to implement it on the website. I also added a loading indicator.

Note: The loading is intentionally slowed down in the demo video, just so you can actually see the indicator

8924fb5425

UI Update: New layout options

I added layout options to suggested videos - a ‘comfy’ and ‘compact’ layout.
Your selected layout is consistent between page refreshes and different videos.

0c6e2273a8

UI Update

I updated to video list to readd the separator, use flex wrapping and lower the font size.

Unfortunately, when the flex element wraps, the separator still shows. I spent around an hour trying to fix this using container queries. Container queries are awful to use with heights. Never again.

2b7319b412

Testing + Code quality

I wrote a test case for users.GetUserById. Since a lot of validation was the same from the test for users.Get, I moved that logic into a separate function to be used for both test cases.

a25a0f14fe

The users.Create RPC returned a uint64 ID instead of an int64, so I updated the code to use int64 instead, and removed any no longer necessary conversions to int64.

a683c56d0d

Testing + Code quality

I added a test case for the Users.Get RPC, which verifies that all returned fields are as expected.

When I was writing this, I realised that type for User contained an email, even though it was not sent on most emails, which could lead to confusion using any RPC that returns a User.

To fix this, I added a separate type that contained the users email (UserWithEmail) that is used by RPCs that need the users email, and I removed the email field from the User type.

bc8cc01aa0

Testing

I wrote a test case for the GetFollowing RPC.

This test case has a similar setup to the GetFollowers RPC (create two users and follow one), so I split the logic into a separate function to reduce duplicated code.

The test case ensures:

• Only one following user is returned
• The returned following user has the correct username and ID
• The CreatedAt timestamp is valid

5d7ed806a2

Testing

I added a test case for the GetFollowers RPC - this is passing :)

Honestly, writing tests is boring as hell. I’ve only written 37% of the tests for the Users service… and there’s around three other services to test for.

d0cd784d54

Bug Fixes

In my previous devlog I discovered users were able to follow themselves.

In this devlog I:

• Rewrote the follow/unfollow RPCs so more logic is done with go than SQL
• Blocked users from following themselves (test case passes)
• Added a nicer error message when users try to follow a nonexistent users

761211acb0

Testing

I added test cases for:

• Following yourself (should not be allowed)
• Following someone (allowed)
• Unfollowing someone (allowed)
• Following a nonexistent person (should not be allowed)

As a result of this, I have found that:

• You can follow yourself through the API
• Error handling should be improved for follow RPCs
• I should improve my testing solution, right now I have to manually log if it passes, and fails are just logged as an error. I should make this automatic

These should be fixed in the next update :)

c7b3ea8e3b

Code quality + Testing

I improved code quality by:

• Check grammar/spelling. I fixed a lot of grammar and spelling errors using harper-ls and harper-cli
• Stricter formatting. I ran gofumpt, which has a lot stricter formatting
• Linting. This took by far the longest time - I ran golanglint-ci, and one by one, fixed every single lint issue. This included many unhandled errors (that wouldn’t of caused a problem, just weren’t being logged)

a93729e78c

I also updated testing by:

• Cleaning up code
• Writing test cases for different password lengths

a5f74ef1f5

Feature: Umami tracking

I added Umami, a privacy focused analytics service to Watchtower. As of now it only tracks page views, but in the future I plant to improve integration by adding events for video likes/dislikes, follows, video creation etc.

I also considered Rybbit, because it has more features like error tracking and web vitals, but I could not get a self hosted set up working. Maybe in the future.

• 4ebc86e022

Feature: Comment deletion

I added the ability to delete comments. This can only be done by the user who created the comment, or any admin account

• 19aac7f632

UX Improvement

After updating the video upload script, an issue was created where errors weren’t displayed properly. I have updated the website to clearly show errors

• 8dd0cad1dd

Bug fixes

On the second test case I wrote I found a bug. Not sure if this is a good thing, but at least it means testing is worth it.

• 686d0d4b70 Add test cases to make sure duplicate emails and usernames are blocked
• e6562c8aae Fix bug discovered above where errors were not
  handled correctly
• 2266cef6fc Allow videos down to 240 pixels to be uploaded, and handle videos that are smaller than 240px properly, by rejecting them and showing an error message instead of transcoding them badly

Testing!

Yes - I’ve finally gotten around to writing tests!

Before I was relying on manual testing, by visiting the website - I’ve probably missed quite a few bugs. But now I’ve started writing automated tests for the API.

I have set up the tests so an ephemeral API is set up, along with temporary PostgreSQL, S3, RabbitMQ, and Valkey (redis). These automatically start when you run the tests, and stop when the tests have finished. I did this using the Golang SDK for docker-compose, and waiting for the API to start. After, migrations are automatically applied to the testing database.

As of now, there is only one test case, but I will setup more and more. I plan to implement E2E testing for the website in the future.

Web Update

• af75360ffa Updated formatting of suggested video details
• 0a782728ce Fix live chat dockerfile and deploy to production

Full deployment

The full version of Watchtower is deployed on my new server (my old pc) which means you can now upload videos! These will be transcoded to multiple different resolutions, and analyzed to block NSFW content. The NSFW detection may be a bit inaccurate so make sure to use the report and appeal features

Changes

• 472ff840c5 Fix transcoder dockerfile
• 1141fb2aec Add analytics dockerfile
• 15c7b46db9 Video duration’s are hidden if they are zero (happens in demo environment)
• Multiple Various dockerfile fixes
• Multiple Switch to old golang version of video analyser
• d58a78215d Sidebar links are given a semitransparent background

Polishing

I refactored some ridiculous code that was inline, and made it a separate function.
This will reduce allot of bandwidth because now the JavaScript is in one place instead of in every single comment

View Commit

Polishing

Spent some time polishing things, and working on docker images, for a move to another server

• 612a3cecad I removed the dependency on go-sqlite3 which was leftover from when I used sqlite as the database. The error messages for duplicate usernames are now prettier
• fc2f524b21 I added docker images
• e6394fea10 I gave the icons when changing/uploading thumbnails a background, so the contrast is still OK when the uploaded image is light or dark
• 1427c28e33 I allowed the recommended videos to be clicked anywhere for better UX, and added a background on hover

If you spot anything that needs polishing - please let me know!

iOS App: Update

1. I added sort options, you can now select between latest, popular, recommended and trending. This took a fair amount of work, I had to guess the type of the gRPC client so I could pass it between Views. This is really hard when you don’t have a functional LSP. I also had to rewrite the way the gRPC client was created so the connection doesn’t close.
2. Moved thumbnails to a separate component for clarity

• 03ee41ded7
• 7944685210

iOS Update

I updated the styling of the onboarding page

1 Commit

Web Update

I added suggested videos right in the view video page.

I also simplified a lot of repeated code, moved an API endpoint from the Users service to the Videos service, and made it so if there is only one comment on a video it says 1 Comment instead of 1 Comments - Crazy right?!

View 3 commits

iOS Update

1. I allowed the user to configure the API + S3 Endpoints
2. I added the user avatar, username, and view count to the video list

View the code

iOS App!

I started working on an iOS app using swift!

This is my first time ever using swift, so it took quite a while.

Initially I tried using NativeScript Svelte, then React Native. Its pretty hard to get gRPC working with them, needing a proxy or something for the gRPC server, so I switched to swift instead.

Xcode (Apples IDE) doesn’t run on linux, so I had to install xtool and vscode/zed. I have to rebuild everytime I make a change, and the builds are pretty slow, but Swift is a really nice language and I may use it in the future

More flag reasons

I added a reason to flag videos that is not nudity.

View the commit

Video Reporting

I finished video reporting, now admins can flag a reported video in the admin dashboard

Reports: Progress

Users can now report videos, and admin page was added to view reports

Next up I need to add actions like removing reports and flagging videos

View the commits

Moderation Appeal

You can now request an appeal if your video gets blocked.

There is a new page in the admin dashboard to view appeals

Admins can either reject or remove appeals

Rejecting will not allow the user to create another appeal

View The Code

Rewrite

I rewrote the video analyzer (NSFW detection) from Go to TypeScript, because the go package used for NSFW broke.
It might be slower, but at least it works

Can’t really have many screenshots for this update…

I also reduced the amount of things you need to install by using go tools,
where instead of installing the binary I run go get -tool and call it using go tool
More work for me, less work for someone trying to set this up themselves.

I also reinstalled my OS 2 times, so thats why this update took a while

You can view the changes on Codeberg

Live chat

I added chats to live streams! (you can chat even when the stream is offline
This was implemented using websockets (this is like the 8th microservice)

I made it so scrolling pauses the chat, so you can read messages.
The colour of the usernames are based on the hash of your username, I should probably let you pick a colour though.

Live streams fix

I fixed live streams by writing a proxy to the stream server

Also did a few bug fixes
View The Changes

Replaced dropzone.js with a custom upload script, so uploading matches the theme better
(4h of work because everything decided to break)
(ps don’t use RustFS, it is buggy and vibecoded)

View the code

Added a live indicator to profile pages

Big Update - Live Streams!

I added live streams

View Changes

+ 502 Additions
- 73 Deletions

I added trending, and did a few other things, like improving security, working on a demo environment, improving code, fixing a bug and cleaning up processed thumbnails

View 5 changes

I added view counter (a view counts once every six hours per users) and refactored the code!
View Changes

Admin Dashboard Improvements

I added a user management page that lists all users (with scrolling to load more) and allows flag management (Admin/Verified flags)

View the change

I should add search next

I added an admin dashboard with user/video count analytics.
(I spent a lot of time perfecting how the graph looks)
View the change

Added milliseconds to durations that show on hover
Accidentally deleted the last devlog

Improved UX by redirecting back to profile page after deleting a video (with a refresh) so deleted videos don’t show after deletion
Also changed some endpoints to use proper http semantics

I hid private/unprocessed videos in the recommended feed
Also had problems with atlas migrations which took hours to fix :)

I added a most popular sort option to the homepage of WatchTower.
I am planning to add a trending option as well.
I should probably remove some of the duplicated logic though and replace /videos/latest and /videos/popular with /videos/{sort_type}

I added a “latest” sort option to my video sharing platform
I also fixed a bug where usernames wouldn’t show on recommended videos
Next up is sorting by popularity