Server - Sign-in Page

I decided to make a page in the server where users can get a token directly. While it’s just a basic HTML+JS thing, I had to work out a lot of stuff with event listeners that I’m not used to (as I use React in the majority of my projects, but am not using it here). It’s a bit of spaghetti code, but it fully works, and I did it all myself! (like everything else in this project, I am not using AI for any part of this)
This is meant to be used for token-based WebDAV login, if users prefer it to using a username and password. It can also be used to directly call the API, if users would prefer using this token to directly calling POST /auth. It will be more useful if I add SSO or 2FA in the future.

Server - WebDAV

I now have functioning WebDAV! This took a very long time to properly implement, and I encountered many roadblocks. For example, my permissions model wasn’t working in WebDAV, and the fix (below since it was a core function issue). When I fixed it, suddenly clients were getting errors loading directories. A lot of debugging later, I figured out the issue was coming from the webdav-server PhysicalFileSystem, so I had to extend it with my own class and write a WebDAV equivalent to the listDir() core function.
I also now filter out certain auto-generated system files if a new config flag is set to true (default)! This means things like .DS_Store will not actually be saved to the FolderHarbor server’s hard drive if the setting is on.
WebDAV is a bit slow right now, I’m going to try improving it later.

Server - WebDAV Logging

I added logging for WebDAV file changes! This uses a switch statement and some extra checks to appropriately log many changes. I’m not logging directory listing, as trying to caused… issues (to say the least). I got 6 pages (120+ entries) of logs in around a minute logging those.

Server Core - checkPath

Through the above-mentioned debugging, I learned my checkPath() function wasn’t accounting for global exclusions. As they were added later, I forgot to add them in this route! That caused the overly-permissive WebDAV issue, but fixing it exposed the second issue. This is now fixed, though, and accounts for all path attributes. I may need to split some of the functionality more for security reasons, but I will see later.

Removing Web Panel

I’m not going to be able to work on the web panel (given that Flavortown is ending so soon), and WebDAV makes it far less needed. Thus, I removed it from the repo, README, and landing page.

Server - WebDAV

This was most of the time on this devlog! I’ve been working hard on adding WebDAV support, and though it is far from done, I made a lot of progress. I’m using the webdav-server NPM library, and my main struggles were figuring out their user and privilege manager structure (I had to make authentication components to plug FolderHarbor’s auth system into WebDAV). I also managed to allow both token-based auth and username+password auth, as it’s more convenient to just use a username and password, but tokens are easier to revoke (if they get leaked). The key is just entering token_<your token> as the password, the username gets ignored if your password is prefixed with token_ and it checks your auth through a token. There will be a website to get a token with your username and password soon, built into the server!

Server - Better audit logging

Added some better audit logging things! Namely, this is pagination (log pages of 20 entries each), and logging some existing file API route actions.

Server - Audit Logging

I started implementing audit logging! Now, every admin action (except listing things) is logged, as well as users signing in and failed password sign in attempts. Logs are stored in PostgreSQL, and they took so long because I designed a bunch of functions to work with them, as well as letting them contain all kinds of different data (the TypeScript types for this were a bit crazy :heavysob:).

Server - Logs API

Kinda in the same vein, the server has a new GET /admin/logs route that returns all logs. This could use some improvement, such as pagination, but that’s a thing for future me to do :3

CLI - Roles

New CLI command and route for listing roles. I haven’t gotten to add more role features yet, but likely will soon!

Server - Permissions

I added some new stuff for permissions! Namely this is the new getEffectivePermissions()function, which takes a user ID and resolves all of their permissions (permissions directly set on their user, as well as permissions set on their roles). This is also integrated into GET /me and the CLI folderharbor accounts get command, but those were such small changes code-wise that I’m bundling them into this section.

CLI - User Updates

I added the ability to update users in the CLI! This includes:

• folderharbor users update username <id>: Change the username for a user
• folderharbor users update password <id>: Change the password of a user
• folderharbor users update failedlogins <id>: Clear a user’s failed login attempts
• folderharbor users update lock <id> [yes|no]: Lock/unlock a user (will lock if you don’t provide a “no” argument at the end)

Users can do some of these for themselves through the folderharbor accounts command, these are for admins to do them. Granting/removing roles/ACLs/permissions isn’t added yet, but will be later.

Landing

I added a downloads page and made a couple small changes.

Server - Core Functions

Wrote some core functions for file management! Namely, I added writeFile, createFolder, deleteItem, and moveItem. These took some work to design, integrate with the ACL system, and generally implement.

Server - API

I added a new API route, POST /files?path=, which creates an empty file at a specific path, with appropriate checks.

Landing

I started on the landing page! This will be a relatively small page I will host to explain what FolderHarbor is, why you should use it, how to use it, and how to access the demo instance (as well as providing download links through GitHub Releases).

CLI - User Admin

There are some new CLI commands for user administration! Namely:

• folderharbor users list: Get a list of all usernames on the server and their IDs.
• folderharbor users get [id]: Get info about a user by their ID, which could be either full or limited based on your access level (server permission).
• folderharbor users create: Create a new user by providing a username and password, get an ID back
• folderharbor users delete [id]: Delete a user by their ID

CLI - Session Revocation

Users can now revoke their own sessions with the folderharbor account revoke-session [id] command (you can get session IDs from the folderharbor account get command). This allows you to sign out compromised devices remotely.

Server - Access Levels

Tiny change, but the server now includes access levels when responding to a GET /admin/users request, based on permissions (users:read or users:read.full). This just makes it a bit easier to tell what data you’re getting back.

CLI Account command

You can now run CLI commands under the folderharbor account subcommand to do various things! Namely, these:

• folderharbor account get: Gets your current account info, like your username, account ID, and active sessions.
• folderharbor account username: Changes your username to a new one! This command includes some extra protections, like running GET /clientconfig on the server and checking if username changes are allowed before prompting to enter a username.
• folderharbor account password: Changes your password! Due to the server changes (mentioned below), this will also automatically sign you in again with your new password.
• folderharbor account failedlogins: This resets the failed login limit for your account, nice if you were trying to sign in on another device and repeatedly mistyped your password.

PATCH /me

Updated this API route to have more security around password changes! If a user wants to change their password, they have to be using a session created in the past hour. They also will be signed out across all devices after the change succeeds, including the current one.

GET /me

I added info about failed login lockouts here! This means users can see if their accounts are locked out for excessive failed password attempts (which allows existing sessions to continue, unlike an admin-imposed lock).

Server Core

I added a new core function for revoking all sessions for a user! This is used currently for password changes, to invalidate sessions created with the old password.

I did some various improvements to the CLI and server!

Server core

I added two functions to the server core!

• Reading files
• Checking whether an item is a file or a folder
  These are mostly just convenience functions, and the reading one has permission checks.

API

Altered GET /files?path=[something] to check if an item is a file or folder, then call the appropriate function and return the appropriate contents.

CLI

I added configuration through the Viper tool, as well as adding better API error handling, more messages, better auth handling, and generally many improvements. I also now allow specifying a server URL in the login process.

Started on a CLI! I’m teaching myself the Go programming language (mainly by using Go by Example) for this CLI.

CLI auth

I built auth for the CLI! This includes a proper interactive login that uses things like hiding the password in the shell (the way, for example. sudo on most Linux distros does). It points at the FolderHarbor server to do this, but doesn’t save the token yet (I still need to add the config file).

Hi again! :3
I added some new endpoints and generally did server work.

Files Endpoints

New /files route! This allows you to list files you have permission to view by calling GET /files, with an optional path argument (like GET /files?path=/srv to descend into directories. This was fairly complicated, as I needed to design a bunch of code around it (as well as taking Node.JS Dirent objects and re-formatting the response to include more data).

Server Landing Page

As the FolderHarbor API operates on a standard web URL and needs to be publicly accessible and known by users (so they can point their clients appropriately), some people may try to visit the URL in a web browser. Thus, I threw together a fairly basic page to tell them that this is the server, not a client. The CSS will likely be improved later, I just want something that works for now. Express static routing was also so confusing, and took me like 15 minutes of debugging to set up properly.

Note

And yes, I am devlogging from a school Chromebook lol, I left my MacBook home but want to devlog now :3

Hey again! I did some various work here, mainly in the config and server.

Config

I added a new config setting, "globalExclusions"! This is a feature that lets you pass glob paths to force-deny. This works mainly as a security feature to prevent overly permissive ACLs from exposing files that shouldn’t be.
I also added some new API routes:

• GET /admin/config: Read the current config
• PATCH /admin/config: Change certain config settings (namely, global exclusions can’t be changed this way)

Core

I added a new core function to allow editing the config! This has all of the appropriate error checking and security mechanisms, such as making sure that the config is writeable and that read-only settings can’t be changed. I also validate it with Zod before writing it, for extra safety. Figuring out how to handle merging the current config and new updates safely was a little complex, but I made it work! :3

Admin Panel

I templated out the admin panel! This was only like 3 minutes of work (duplicating the web folder and changing a bit of HTML), but I felt it should be mentioned.

Wow, I did quite a bit of work! :3

Error Handling

Putting this first since I worked hard on it! API routes are now much less likely to error, and they have more and better checks (especially on PATCH routes)! The messages if they do error are also more normalized across routes.

Core

Core functions for creating users, roles, and ACLs now all support passing some extra information to them! This means that you can directly provide allow and deny path arrays to an ACL when you create it, instead of having to create an empty one and then edit it right after. This is implemented in the role and ACL creation routes, but not user creation (as user-editing permissions are more fine-grained, but not user-creating ones).

I also re-ordered some existing core functions to match the create-read-update-delete structure.

Roles API

I finished the Roles API routes! Namely, these are:

• GET /admin/roles/:roleID: Get a role’s info by ID
• PATCH /admin/roles/:roleID: Edit a role
• DELETE /admin/roles/:roleID: Delete a role

ACLs API

I implemented the entire ACLs API, and all of its routes! That is:

• GET /admin/acls: List ACLs
• POST /admin/acls: Create a new ACL
• GET /admin/acls/:aclID: Get an ACL’s info by ID
• PATCH /admin/acls/:aclID: Edit an ACL
• DELETE /admin/acls/:aclID: Delete an ACL

This work was basically all just API routes!

Users

I finished user routes, for now at least! This added listing (GET /admin/users) and deleting (GET /admin/users/[id]) accounts.

Roles

I templated out the entire roles.ts file, as well as started making routes! Namely, these are:

• GET /admin/roles: List roles
• POST /admin/roles: Create a role

Permissions

I added GET /admin/permissions for reading permissions for the server! These are hardcoded in the source code, so I made this route available without authentication. It just gives the IDs and descriptions for permission nodes on the server.
I also added some new permissions and cleaned up some outdated ones.

Bugfixes and Clean-Up

I fixed some assorted errors! Mainly this was that you couldn’t remove all permissions from a user, now that is possible. I also added some better error handling in some places.

This last few hours of work were a bit crazy for me, as I did this work across multiple days, but here’s the things I did!

Web Panel

I started working on a web panel! This was really just some basic Vite setup work, and I haven’t really written much code on it yet, but I will need this in the near future.

User Editing

I added some API routes for editing and locking user accounts! These could use some more work, but they function on a basic level, and include things like checking for permission levels (the app splits user editing into users:edit and users:edit.full permissions to give more granular control).

Clean-Up

I removed some error-checking conditions that could never be reached (like username_used in a method that didn’t update usernames), as well as removed the GET /auth endpoint (GET /me gives much more information and is generally better).

Bugfixes

There were a few assorted bugs I fixed, but the main one was that sessions kept revoking after session 1. Turns out, this was because I was checking if a session ID matched a valid user in the wrong way, and I was calling the validation function with session.id (the session’s ID) instead of session.userID (the user behind the session’s ID). I fixed this finally, and it took so much debugging.

Notes

Finally, I hope you enjoy this new, more structured devlog style! Sorry I took such a long break from this project, I was making a Slack bot (as well as school stuff). I’m also now using Insomnia by Kong for my API request screenshots, instead of just raw cURL.

I finally finished the bot! I also added a few new features.

README things

I cleaned up the README and made some improvements! I added more instructions, clarified some things, and added some extras (like the Not by AI badge)

Automatic approvals

The bot now has a config option to automatically approve requests to join

Errors

There were some various errors in different parts of the code, as well as things like ESLint not working properly. I fixed these! I also fixed a config permissions issue.

Restructuring

I restructured the code to be in a /src directory, as that makes it easier for me to normalize paths and work with TypeScript compilation.

Workflows

I added GitHub Actions workflows for building and linting the code. This was a bit complicated, as I had to work around pnpm and Docker Buildx (I’m running this on a Raspberry Pi 5, so I have to build for both x86_64 and ARM64).

The bot is finally ready to ship, so that’s my next step! :3

I added a Dockerfile and configuration system, as well as many improvements to the README!

Config

There is now a config system! This makes it easy to change things, such as the time daily recaps trigger and what channel is used. It uses Zod for validation, and I migrated most of the hardcoded values to config ones.

README

I wrote most of the README.md file, including a very detailed features list and some deployment instructions. It isn’t done, but I still added a lot to it.

Dockerfile

I added a Dockerfile for running the app in Docker! It’s fairly simple, but it should work properly!

I did a few things during this time! One thing is I slimmed the code down quite a bit, and removed some duplicated code (like how I was fully replacing blocks on messages before, instead of just re-using the existing blocks like I do now). I also improved some error handling and code structure!
Some other things:

/spacetime

There’s now a command to join my tier 2 channel! This replaces what I used to use a workflow for, but workflows are sadly gone from the Hack Club Slack now.
This is also better than the workflow approach, as it sends DMs on approval/rejection and has the ability to reject users. It’s also just generally more custom and better.

Universal Status integration

This integrates with a previous project of mine, Universal Status, to show my current status in my daily recaps. It’s just a fun little thing, and I plan to add more of these to the private recap soon!

Recap improvements

Now, the recap message links to my recap directly in the private channel, instead of the bot’s message. This allows people to read the public recap directly from my private channel. If I don’t have any messages in the thread, it will fallback to the bot message (because error handling)! It also has some extra checks, like it will only look at messages from my user ID (since sometimes people type in recap threads while they wait for me to post a recap).

I added daily recaps to the bot! There are two kinds:

Public recaps

These are in my public channel, and they are just standard recaps of my day. They are meant to use bullet-pointed lists, and this message is scheduled to send at 10:30 PM every night. It includes a button to mark it as done, which is used for the next kind of recaps! These buttons also have proper permission checks using my Slack user ID.

Private recaps

These are for my private channel, generally used for more detailed recap info that I don’t feel comfortable sharing publicly, or photos of myself. These are triggered by clicking the done button on a public recap, and they include a link to the public recap.

Both kinds of recaps have single-click “done” buttons, as well as reactions (the bot reacts with the 💤 emoji on message send, then changes to ✨ after the recap is marked done, as a way to indicate to people looking at the recap message).
Recaps are mostly done, and scheduling them works, but I still need to improve some things with them. I also want to add more details to private recaps, likely some of which will be pulled from my previously made Discord bot!

I started building the bot! I’m using Slack’s Bolt library for NodeJS, and I have gotten a fairly functional development setup. I built it to work in either socket or HTTP mode without any changes to the code (just environment variables).
I also added a feature that yells at people for threading on a user group mention! This uses an ephemeral message so that people will hopefully reply on the part that doesn’t ping people for every message. For now, it only detects when the top-level message is a user group mention, not when a user group is mentioned in the thread (though this may change).
I have had to write regex and get around TypeScript problems in this time, and it’s fairly complicated since I’m trying to not use any AI in the process of making this bot.

I added many new API routes! This took so long because I had to write a bunch of code to integrate Express APIs with the underlying functions, while still having proper error handling and a good user experience. I added routes for:

• Revoking sessions other than the active one (DELETE /me/session)
• Getting current user info (GET /me)
• Editing some info for the current user (PATCH /me)
• Admin API to read other users’ info (GET /admin/users/[id])
• Client configuration, aka things clients need to know that vary based on the configuration settings (GET /clientconfig)

I also made some permission changes and changed a few other things that needed improvement!

I built a re-implementation of directory listing that accounts for ACLs! This, as well as the newly improved checkPath() function, runs very quickly (thanks to the micromatch library) and manages what paths a user can access. This new method also accounts for parent directories, so if the user has access to something in, say, /home/nova, they can see /home when listing from the root directory (/) without being able to see other directories (like /etc). In /home, they would only be able to see nova/, not any other users’ folders.

Also, I created a permission system for the admin APIs, so that users can only do things their permission levels allow for. This follows a fairly standard RBAC structure, with permissions like roles:create and users:read.full.
Finally, I added a bunch of assorted helpful methods! These allow for extending sessions, deleting roles, and a few other things.

I added quite a bit of authentication-related code! Most of this time was building strong authentication middleware, so I wouldn’t have to worry about spotty auth implementations. I set this up with two functions, auth() (called on every request) and enforceAuth() (called on some endpoints that require auth). This took a while, as this is my first time making my own Express middleware and handling all of the related TypeScript things (like abstracting the Request to add req.session and req.sessionErr). I also had to figure out cookie handling, as I built the auth middleware to accept header-based auth or cookie-based auth, whichever is better for the client.
I also added logic for ACL path-checking, as well as standard functions for modifying and reading users/sessions/roles/ACLs. This completes the CRUD (Create-Read-Update-Delete) structure for managing all four of these (except for updating sessions i guess).
Finally, I split user handling and session handling into two files, since they are both getting very complex by now.

I started adding roles and ACLs! This isn’t done yet, but I wanted to devlog with my current progress on it. I also migrated user validation to be centered on user IDs, as then usernames can be changed without issue. This includes tying the sessions to users with a relation, so sessions will auto-delete if a user is deleted. Finally, I generally improved the database schema.

I added an API route for handling sign-ins! I also added rate limiting (for brute-force protection) and account locking (so admins can prevent users from using their accounts without changing the accounts past a boolean flag). This code is a bit more complex and went through many iterations, such as when I was going to make locks able to expire (which I ended up not doing when I implemented rate-limiting separately from the locking feature, originally I was going to combine them).

This was a very productive time! I added a database and an entire authentication system, including password hashing and short-lived auth tokens! This system is fairly complex and has some weird things, but the main usefulness is that sessions can be remotely revoked (useful if the token gets leaked) and can easily look up the user (hash token -> check against db -> get the user from the session row).
The first video shows the results of some session testing code (create test user -> authenticate to get a token -> check that the token works -> revoke the token), while the second is most of the code I was working on!

I added more to the API! I also added graceful shutdowns and started working on authentication. I was planning on building it around JWTs, and I spent over 40 minutes trying to implement them, but it all fell apart when I realized they were impractical based on my plans for auth.

I added a ton of permission checks and error handling logic. Also, I started building an Express app for the FolderHarbor API. though I haven’t added much. This time was quite technical, as I was adding a ton of checks related to the config file (such as making sure it was owned by the correct user and wasn’t overly permissive).

Started building the server! This project is structured as a monorepo, and I’m building this server process as a “core” of sorts in TypeScript. During this time, I started structuring the repo/code. I also used Commander for command arguments and Zod for parsing the config file.

I added autosave and a few new pages! Among these, there is now a place to view all journal entries, instead of just ones from the current day. I also got the basic structure prepared for a settings page, which will have more features soon.
Autosave works similarly to my previous projects, but I modified the delay and implementation a lot. React prop serialization doesn’t make it easy, but I’m making it work.

I worked on the journal editor and added a page for viewing journal entries! For the editor, I just built something basic for now. It doesn’t have rich text support yet, though I plan to add that. I also coded the ability to delete journal entries.
For the view page, it’s just a simple display of whatever you wrote in the entry. It will be improved upon, but I wanted the foundation.

I added better error handling, as well as a new API route for updating entries. I’m trying to build a lot of the journaling backend before the frontend, as the frontend has a lot of complexity that I want a stable backend to build on.
(The database integration does work, as shown in the previous devlog. I intentionally triggered an error to show that the messages work.)

I built a basic API for adding journal entries! It’s far from perfect, but it will allow me to start on the entry editor. I also added an entry display to the main page, though it doesn’t really map to anything yet. I will add more to these when I work on this next!

I started building the app! I don’t have much yet, but I have added a database, authentication, and a basic UI.
This was mostly just things to get me started on the app :3

I updated a bunch of things to prepare to ship! I deployed the site to a DigitalOcean VPS, added Docker build support, added CI workflows, and fixed bugs. I also drew an 88x31 button (not tracked on my time). I was originally planning on deploying this to Heroku, so I had some hours working on setting that up (~1 hour) that I had to revert due to an issue.
I had to make some pages “dynamic” for them to work in the production environment. I found a few other bugs as well, like RSS feed links not properly leading to my website. I also made all of the widgets fully operational (including the fancy clock on the front page). Overall, this was very technically complicated to figure out, but I got the site fully operational.

I made a widget on my homepage for blog posts, made my projects page pull from the database, improved mobile support, and added project management to my admin panel! The projects page was originally constructed from an array, but uses the database now, so I can add new projects of mine without needing to edit the code. This also included building an admin panel for actually editing the projects. I also moved buttons around on mobile to make it look much better. I tried to change the background to an image, but my CSS didn’t end up working as I wanted it to, so I had to discard those changes.

I built an RSS feed for my blog! This allows my posts to be seen on other platforms, so people wanting to read my posts don’t have to check my blog for new posts when there aren’t any. I also built an auto-save system on my admin panel for when I’m writing, which made me work very hard to sync state across the database, server, and client (my state hooks were causing issues with how I had them set up). I also added some other things, like the ability to download posts as .md files, improving the viewer for posts, sorting posts by published date, and improving error handling.

I created most of the admin panel to manage blog posts, on both the frontend and backend. I had some challenges, mainly with sorting out how to build the PATCH requests to send as little data as possible (as I commonly write things on low data, and I want this to be practical for me to use). The end result turned out quite well, and I even implemented proper error handling!

I started on an admin panel for the blog! This was mostly backend work, as I wanted a lightweight auth system (it just uses an environment variable for the password).
It also included database work to add things like a “published” field to the schema (to allow me to hide blog posts without deleting them), as well as hooking the blog up to an actual PostgreSQL database (running on my Raspberry Pi for development).

I built the foundations of my blog (and its associated database), a footer, an about page that talks about me and things that I like, and a widget that integrates with another previous project of mine (called Universal Status)! The ongoing Nest outage has made it a bit harder for me to do this, as I typically use Nest PostgreSQL while developing, and Universal Status is hosted there. I got around the latter issue by running a local instance of Universal Status and pointing the site there, and the former issue by editing my function to temporarily use placeholder data (in the same schema the database will follow). This one was quite fun to work on, and I’m planning some very cool things with widgets soon!

got started on the site and built the foundations of my project list, home page, and “accounts” page. also included making a header and figuring out some fairly complex things, like tailwind css gradient text!
a lot of this work was getting started, and i am planning on starting the blog and doing a bunch of refinements soon.