Ghost of Nova

Web - Docker

I borrowed a Dockerfile from a past project of mine and made some changes, and now the web panel builds and runs in Docker!

CI/CD - Changes

I improved a lot of the release CI/CD workflow! This includes making it build the web panel’s Dockerfile, as well as saving attestations for it for security. I also improved the release messages across the board.

CLI - nFPM

I started configuring nFPM for the CLI build flow! This means it will build .deb and .rpm files for the CLI and include them in the releases. I sadly don’t have this finished, just am working on it.

Notes

First of all, 90 hours :dance_catgirl:
Another thing, I added a license finally! FolderHarbor is now under the MIT license.

Web - User Panel

I did a ton of work on the panel for users! This will let admins read and edit user information and grants. I’m currently still working on getting grants to work properly though, I have view-only working but not the actual granting system.
Some cool features I added:

• It checks permissions for a bunch of different objects, like if to show a text box or just formatted text for a username! This is done by checking the cached permissions on the user’s session for certain things, like users:grant and users:edit.
• It will take permission levels into account! There are two permissions, users:read and users:read.full, for reading user data. This impacts how much data is returned by the server, and the UI will automatically adjust (both the CSS and the panels that appear) to keep everything looking balanced and nice! :3

Server - Assorted

I did some assorted server things! Namely, this was:

• Debugging macOS FTP: Finder just kept refusing to connect, I eventually figured out this is because the PASV URL was different from the URL I was connecting with. I tried to fix this, forgot how the PASV resolver in ftp-srv worked, and ended up having to revert.
• Strict config: I was messing around with the CLI config editing thing, and realized it wasn’t erroring for incorrect keys in sub-objects. I fixed this by making them strict as well as the main config.
• FTP metadata filter: This is really just a start, but I began working on a system to filter out metadata files and pretend like they were written successfully if that config setting is on (this is already a thing in the WebDAV server).

Web - Admin

I added a bunch of new admin panels! For now, this is just lists of users, roles, and ACLs, plus the Tools panel! The Tools panel gives a list of all permissions (available to all users) and a log reader (requires the logs:read permission). I also filled out the sidebar with a bunch of available / soon-available things!

Web - Logs Reader

Speaking of the logs reader, I think it’s cool enough for its’ own section! This shows a simple, formatted display of the logs, starting with the latest. There’s a little navigation bar at the bottom, which uses some JavaScript to not allow overscrolling. It also displays the log body (which is dynamic per log) as JSON, as I feel it’s readable enough to look cool :3

Web - API Client

I fixed the API client thingy (fetch wrapper) to support search parameters!

Web - Settings

I improved the user settings page more! This includes adding a permission display for “effective permissions”. FolderHarbor users have two types of permissions: Direct, which are directly set on the user, and Effective, which are the combined, deduplicated permissions from the user’s roles + the direct ones. Effective permissions are calculated at request time.

This settings change also includes “locking” the active session in the revocation list, as you can’t use the DELETE /me/session endpoint to revoke the active session. The server will just reply with an error and ask you to log out instead.

Web - Caching

I now cache the associated user’s permissions when they log in! This is used for the admin panel. There’s also now a background task to refresh the user’s username and permissions without making them log in again.

Web - Admin

I started working on the admin panel! There’s very little so far, but here’s a few random things:

• There’s a button to go to the admin panel from the home page, as long as you have at least one permission set on your account.
• That button leads to /admin, which right now is empty save for the header and a sidebar.
• The sidebar isn’t very finished yet, but is joined with the header and will have links to different admin pages (users, roles, ACLs, config, tools, etc.).

Server - Login

I added the user’s effective permission list to POST /auth, allowing the web client to cache it immediately without costing an extra API call and worsening the UX.

CLI - Config

The CLI can now read and update the server’s configuration with some new commands!

• folderharbor config read: Read the config file
• folderharbor config edit <key> <value>: Edit a config setting (supporting nested settings with keys like ftp.port)

Server - Config Handling

I improved config handling! This involved adding lodash for better deep-merging, as well as making the config “strict” in Zod (so adding invalid keys will fail with an error). I also added globalExclusionBypasses as a read-only setting (it was this before in the schema, but now there is an explicit check for it).

Web - Errors

I added some new “fatal error” pages, namely for invalid and expired settings. I also moved some of this handling to the client side, mainly so I could access session details without using search parameters, as well as using React Hooks like useRouter() (NextJS navigation thing).

Thread Reminders

Forgot to include the thread emoji in your message pinging a user group? NovaBot will now prompt you with an ephemeral (“only you can see this”) message, allowing you to still send a message for people to thread on.

Daylight Savings Time

I migrated Cron libraries from node-cron to croner! I did this because croner is more stable and supports Daylight Savings Time.

Config

I was asked to run a custom instance of NovaBot for the #remixed YSWS, but they didn’t want any of the recap features. Thus, I adjusted the config to make it less strict, so the bot could still work!
I also added a config option for the user’s name, which makes it easy to customize certain messages when self-hosting. Finally, I renamed “public” and “private” -> “primary” and “secondary”, as that is more accurate since the bot doesn’t care much about the actual Slack privacy setting.

Note

This one is a shorter one, I’ve been trying to get to 80 hours for a while now and decided to devlog sooner now that I have reached it!

Server - GET /me

Updated the self-info route thingy to make it hide expired sessions. They aren’t removed until the tokens are used, so already-expired sessions were bloating the sessions list.

Web - Session Revocation

Users can now revoke sessions through the /settings page! Just click on the red trash can next to a session.

Web - Metadata

I added some metadata to pages, namely titles! I also finally got the favicon set up properly.

Web - Settings

I started working on a settings page for users to read and change settings about their own accounts! I haven’t finished much of this yet, but parts are functional. This also reads the server’s “client configuration” (GET /clientconfig) to know if users are allowed to change their usernames, and it locks that text box if not.

Web - Fatal Errors

I built a system for redirecting users to specific error pages for fatal errors. For example, I can redirect to /fatal/locked if a user was locked by their administrator (meaning they cannot take any actions until unlocked).

I also updated my query() fetch wrapper to redirect to the locked error page whenever getting a response with that code.

Web - Login

I improved the login flow a lot! Now, there are two “stages”: one to enter a server address (also showing the default server if one is configured), and one to enter credentials. This looks better than the original (which just had three labeled boxes), and also gives me the ability to more easily add 2FA and SSO support in the future, as well as token-based login.

Web - Header and Profile

There’s now a new header, which contains a profile drop-down! This is only on the home page for now, but will also be on admin pages when I make them. The drop-down currently just has a link to the account switcher and a sign out button, but I’m going to add user settings soon, as well as a button to switch between the admin panel and user pages.

Web - Home

I built a home page! This integrates with the Providers API in the server (GET /providers) to get the WebDAV and FTP URLs the user can use. This also uses the “copy alert” button concept I came up with when making my personal website a while back, where it sends a browser alert when you click on a copy button (not revolutionary, but convenient) :3

Web - Sessions

I added a way to track the active session through local storage, meaning you can now actually have two sessions logged in and switch between them! The sign in, sign out, and account switcher components all use this, as well as anything trying to get the active session.

Web - fetch Wrapper

I made a wrapper around fetch() called query(), which means I can just write await query(session, "me") to get info about the current user (this becomes GET /me). I can also specify a RequestInit like with fetch, so I can change the method, add headers, and add body data there. The wrapper auto-adds the session token and the “Content-Type” header (everything in FolderHarbor uses JSON, but I add this header client-side for extra safety), as well as assembling the URL based on the active server’s URL. It also automatically handles errors and parsing the JSON. This will make it far more convenient and less error-prone for the web panel to talk to the server!

Notes

Much of this was backend work, so the devlog video doesn’t look all that impressive. Sorry about that!

Also, this marks 75 hours of FolderHarbor development! :3

Web - Sessions

I added some session management logic to the web panel! This uses the browser’s IndexedDB to store all of the sessions the user has active in their browser, which allows them to sign in to multiple FolderHarbor servers and/or with multiple accounts. I also added a panel to display active sessions, as well as sign them out. I plan to let people use their tokens for the web panel in things like WebDAV as well, so I added a prompt on sign-out to allow people to just clear the session in their browser without invalidating it server-side. I may alter this in the future though :3

Server - Providers API

There’s now an API route called GET /providers, which returns the public server addresses for FTP and WebDAV. If one isn’t enabled, or the config has an empty string as the address, it will return null for it. If the config has null for it, it will automatically fill in with the server’s local IP. You can explicitly override the address with a new config value, which is nice for, say, reverse proxies!

CLI - Providers

I added a command to the CLI to work with the new providers API. folderharbor tools providers will give you a list of providers’ URLs.

Notes

Providers, Platforms, and Protocols mean the same thing in FolderHarbor for now, and I’m kinda just using them interchangeably. It’s just another word for the FTP and WebDAV support in FolderHarbor (except Platforms and Protocols includes the API in theory). It’s a little confusing, I will probably clarify that later.

CI/CD - CLI Release

I did most of the work for the CLI release pipeline! I am now able to just push a tag in the format of cli/v0.1.0, and the CLI will automatically build and release the files! I added cross-platform and cross-architecture building, build attestation, and many more cool things :3

Server - Better FTP PASV

I improved PASV resolving more! Now, you can specify the port range to use for PASV FTP connections, and you can resolve an external IP not in your LAN (this would just fallback to 127.0.0.1 in the past, which caused issues).

Notes

This devlog is a bit simpler, as a lot of the PASV work is harder to showcase. I added a screenshot where I console.log()-ed calls to the PASV resolver with some example IPs (127.0.0.1, 192.168.5.2, 10.0.0.5, and 8.8.8.8), which is what the FTP server will use when you connect with a passive connection.

The build attestations are mainly cool because you can use them to verify that the downloaded file was built on GitHub Actions, so thus there wasn’t anything in between the source code and build that can’t be audited. It’s nice for assurance’s sake, and many open source projects do this.

CI/CD - Test

I added a testing workflow with GitHub Actions! This lints/validates the code on every push. As FolderHarbor is a monorepo, I used the dorny/paths-filter@v4 action to filter what tests are run based on what is updated in the commit. In other words, if I only change the server’s code, that commit won’t cause tests to run for the CLI or web panel’s code.

CI/CD - Start on Release

I started working on the releasing workflow! This one will test, build, and release packages when I push certain tags to the repository. I am still working on this, so I will explain more in my next devlog.

Server - FTP PASV

I added better PASV resolver support for FTP! Now it actually takes into account your interface IPs, instead of using localhost blindly. I “borrowed” some of the code for this from the ftp-srv README, then adapted it to work better in FolderHarbor.

Notes

I spent a while studying the GitHub CLI workflows to help me implement some of these things, as a lot of these are fairly complicated.
Also, I added a package.json script to the server to allow me to type-check without building the package, using tsc --noEmit. It’s pretty basic, so I didn’t include it above.

Server - FTP

I implemented permission checks on all filesystem methods that the ftp-srv package uses, as well as logging on many of them (some things didn’t need logging, like changing directories). I also added some local PASV support, but I am still working on it!

Once all of that was done, I realized I couldn’t open empty files in KDE programs. Opening files with text worked fine, just empty ones caused issues. I had to use WireShark to learn that the ftp-srv package was responding with 213 File status when asked for the SIZE of empty files, instead of 213 0 like they should have. Thus, I had to write a patch to be able to use it, and that patch fixed the issue. I would have made a pull request, but the library isn’t really maintained anymore and that wouldn’t have helped much. I attached a screenshot of the Wireshark results showing that originally broken behavior that is now fixed!

Notes

The screenshot of the logs is included just to show that these things are now being logged! I didn’t change the CLI at all during this time.

Server - FTP Get and List

I implemented the full ACL engine for the get and list functions in ftp-srv’s FileSystem! This is still a bit finicky, but it does mean that you only see folders and files you can access.

Web - Login

I built a login form for the website! It doesn’t yet save the auth token, as I want to build an account switcher and save it then, but the site can successfully communicate with a FolderHarbor server!

Server - CORS

I added CORS support on the server! This was needed to make the web portal work, as it needs to be able to talk to the server. I added an array in the config where admins can set the origins they want to allow.

Server - Config

I restructured the config! As part of this, you can now disable the WebDAV and FTP servers. As FTP is a bit buggy and insecure, it ships disabled in the default config, though it can be enabled with a simple boolean change.

Server - FTP

I started adding FTP support! It’s quite complicated to work with the ftp-srv NPM package I’m using, so I only have the server and authentication implemented so far. This means the remaining step is adding ACLs, as well as bug fixing and logging.

Web - Starting

I restarted the FolderHarbor web panel, as Flavortown got extended to April 30th and I have time to do it! This time, I’m doing it right. Instead of using Vite, I decided it would be better to stick to what I know and build it with NextJS.

Notes

I mainly just want to devlog now so I can move on to the web panel more, so that’s why this one is a bit short! This devlog was brought to you by the ~250 mg of caffeine I have consumed over the past few hours :3

Server - ACL path editing

I used a very similar format to that of the new granting system to improve changing ACL paths! This means you can PATCH /admin/acls/<ACL ID>/paths with data in the format of { path: string, type: "allow" | "deny", delete: boolean }[] to update paths, instead of having to send the full list of paths and overwrite the current database contents for it. This was explained a bit more in the last devlog.
I’ve been really dreading the creation of this system, but the implementation wasn’t as painful as I expected!

CLI - Role granting + ACL editing

The granting features from the last devlog now extend to roles as well! Also, that ACL editing system from this devlog was also implemented into the CLI, and can now be used to modify ACL paths!

CLI - Permission list

You can now check for a list of all permission nodes with folderharbor tools permissions! This also includes descriptions of what they do, and pulls from the server directly.

Notes

Finally done with one of the most boring, monotonous parts of building this! It’s 2:30 AM and I’m tired, so I hope I mentioned all of the changes. Also, yes, the CLI is a bit clunky and annoying to use. I am going by that one principle of “make it work -> make it fast -> make it good” that I heard somewhere, and I needed something functional to build off of.
Also, the old editing system pre-granting still works for roles and ACLs! I removed it from users for simplicity’s sake, though I may re-add it if I see a need. Hope you enjoy my new work! :3

Most of these were backend changes, so there sadly isn’t as much to show this time!

Server - Overhauled granting

I determined the granting system from my previous devlog was a bit annoying and too restrictive, so I overhauled it! Now, there’s just one route (/admin/users/<user ID>/grant), to which you can PATCH data to in the format of { type: <"role" | "acl" | "permission">, id: <id>, revoke: <boolean> }[]. Thus, now you can mass-update a user’s grants with just one API call!
I did a similar thing with roles, too!

Server - ACL path editing

I started on improved path editing! This works very similarly to the grant system from my previous devlog, just for updating ACLs. It’s designed for my CLI, as this allows me to POST the new changes without knowing the original ACL contents (this is why I am designing these new granting systems btw!)

CLI - User granting

I implemented all of the new granting updates into the CLI, specifically for users! I haven’t handled roles quite yet, but plan to soon, after finishing ACL editing on the server. There are also a few UX improvements I want to make on the CLI.

Server - Logging

I added logging to PATCH /me, so things like changing your own username or password are now placed in the audit log! I also made logs sort in descending order, so the first logs on Page 1 are the newest instead of the oldest.

Server - Granting and Revoking

I added server APIs for granting/revoking roles/ACLs/permissions to/from users! This could be done before through PATCH /admin/users/[id], but that required fully overwriting the object. This lets admins add to / remove from the list more seamlessly, by using PATCH /admin/users/[id]/grant/[roles | acls | permissions] and PATCH /admin/users/[id]/revoke/[roles | acls | permissions]. Another benefit is that specificity, though some may see it as a drawback, and I may not keep it that way.

CLI - Log Viewer

There’s now a CLI command to read server logs! folderharbor tools logs [page] will give you up to 20 logs at a time, starting with the newest, and placing the data into an easy-to-read format.

CLI - Granting

I added user granting to the CLI, but still haven’t added revocation. This works through the CLI command folderharbor users update grant <user ID> <roles | acls | permissions>. It then gives an interactive prompt to input the item ID you want it to grant to the user.

Server - Setup

I added a new setup script! Now, by invoking the server with --setup, this script will run. It allows the user to enter a username and password for the admin user, then creates:

• A wildcard ACL (allow glob pattern of **/*)
• An administrator role (with every permission on the server, as well as that ACL)
• A user (with that role)
  This script can only be run if the server doesn’t have any users. It requires root access (sudo) to run, and it logs every action it takes to the audit log.

Server - Recovery

This is similar to the setup script, but it is intended for if the administrator gets locked out of the server (say, accidentally deletes the admin role or their user), and no other administrators can help. By invoking the server with --recovery, this script will run. It will create all of the same resources as setup, but it will also lock every other user (this part is intended for situations of severe compromise, as hackers couldn’t do any damage with a locked account). It also requires root access, and logs everything it does to the audit log.

Quite a bit of work for 1-3 AM (roughly)!

Server - SSL

The server now fully supports SSL! After enabling the new useSSL config option and restarting the server, it will check for two files in the config directory: fullchain.pem and privkey.pem (named that way to match Certbot). If neither exist, it will auto-generate a self signed certificate and signing key there. However, it’s heavily suggested to use a proper certificate, like through Let’s Encrypt.

CLI - Roles

I added a bunch of new CLI commands for role management! One nice thing is that I designed the server to act similarly across resource types, so I was able to just use the same user boilerplate for a lot of this.

• folderharbor roles create: Creates a new role
• folderharbor roles get <id>: Gets info about a role
• folderharbor roles delete <id>: Deletes a role
• folderharbor roles update name <id>: Updates the name of a role

CLI - ACLs

Commands for ACL management, basically the same deal as the role commands (a ton of boilerplate).

• folderharbor acls list: Gets a list of all ACLs
• folderharbor acls create: Creates a new ACL
• folderharbor acls get <id>: Gets info about an ACL
• folderharbor acls delete <id>: Deletes an ACL
• folderharbor acls update name <id>: Updates the name of an ACL

Notes

I used FolderHarbor to transfer the video demoing SSL from my MacBook to my Linux desktop (I’m doing this devlog from my Linux desktop, the last one was on my MacBook). FolderHarbor works very well for these kinds of transfers (as well as many other use cases), and this is basically one of the main reasons I chose to build it!

Server - Global Exclusion Bypasses

This was a fairly major refactor to add support for explicitly allowing certain directories. I first tried using negation globs (prefixing the glob patterns with !, which is theoretically meant to exclude those paths), however it turns out this isn’t supported in the micromatch library when passing a pattern array (like I do here in FolderHarbor).
My solution to this (after like 30 minutes of debugging this strange behavior) was just using a separate array in the config for negations, and checking them. If a global exclusion matches, the global exclusion bypasses are checked. If it matches one, it is still allowed to continue to ACL checks, else access to that item is immediately rejected. This took a lot of work to implement and caused some strange WebDAV behavior at first, but it works now.

Note in the video that the /etc directory is visible, but only the nginx folder is visible in it. There is a global exclusion for /etc/**, but a global exclusion bypass for /etc/nginx/**.

Server - Sign-in Page

I decided to make a page in the server where users can get a token directly. While it’s just a basic HTML+JS thing, I had to work out a lot of stuff with event listeners that I’m not used to (as I use React in the majority of my projects, but am not using it here). It’s a bit of spaghetti code, but it fully works, and I did it all myself! (like everything else in this project, I am not using AI for any part of this)
This is meant to be used for token-based WebDAV login, if users prefer it to using a username and password. It can also be used to directly call the API, if users would prefer using this token to directly calling POST /auth. It will be more useful if I add SSO or 2FA in the future.

Server - WebDAV

I now have functioning WebDAV! This took a very long time to properly implement, and I encountered many roadblocks. For example, my permissions model wasn’t working in WebDAV, and the fix (below since it was a core function issue). When I fixed it, suddenly clients were getting errors loading directories. A lot of debugging later, I figured out the issue was coming from the webdav-server PhysicalFileSystem, so I had to extend it with my own class and write a WebDAV equivalent to the listDir() core function.
I also now filter out certain auto-generated system files if a new config flag is set to true (default)! This means things like .DS_Store will not actually be saved to the FolderHarbor server’s hard drive if the setting is on.
WebDAV is a bit slow right now, I’m going to try improving it later.

Server - WebDAV Logging

I added logging for WebDAV file changes! This uses a switch statement and some extra checks to appropriately log many changes. I’m not logging directory listing, as trying to caused… issues (to say the least). I got 6 pages (120+ entries) of logs in around a minute logging those.

Server Core - checkPath

Through the above-mentioned debugging, I learned my checkPath() function wasn’t accounting for global exclusions. As they were added later, I forgot to add them in this route! That caused the overly-permissive WebDAV issue, but fixing it exposed the second issue. This is now fixed, though, and accounts for all path attributes. I may need to split some of the functionality more for security reasons, but I will see later.

Removing Web Panel

I’m not going to be able to work on the web panel (given that Flavortown is ending so soon), and WebDAV makes it far less needed. Thus, I removed it from the repo, README, and landing page.

Server - WebDAV

This was most of the time on this devlog! I’ve been working hard on adding WebDAV support, and though it is far from done, I made a lot of progress. I’m using the webdav-server NPM library, and my main struggles were figuring out their user and privilege manager structure (I had to make authentication components to plug FolderHarbor’s auth system into WebDAV). I also managed to allow both token-based auth and username+password auth, as it’s more convenient to just use a username and password, but tokens are easier to revoke (if they get leaked). The key is just entering token_<your token> as the password, the username gets ignored if your password is prefixed with token_ and it checks your auth through a token. There will be a website to get a token with your username and password soon, built into the server!

Server - Better audit logging

Added some better audit logging things! Namely, this is pagination (log pages of 20 entries each), and logging some existing file API route actions.

Server - Audit Logging

I started implementing audit logging! Now, every admin action (except listing things) is logged, as well as users signing in and failed password sign in attempts. Logs are stored in PostgreSQL, and they took so long because I designed a bunch of functions to work with them, as well as letting them contain all kinds of different data (the TypeScript types for this were a bit crazy :heavysob:).

Server - Logs API

Kinda in the same vein, the server has a new GET /admin/logs route that returns all logs. This could use some improvement, such as pagination, but that’s a thing for future me to do :3

CLI - Roles

New CLI command and route for listing roles. I haven’t gotten to add more role features yet, but likely will soon!

Server - Permissions

I added some new stuff for permissions! Namely this is the new getEffectivePermissions()function, which takes a user ID and resolves all of their permissions (permissions directly set on their user, as well as permissions set on their roles). This is also integrated into GET /me and the CLI folderharbor accounts get command, but those were such small changes code-wise that I’m bundling them into this section.

CLI - User Updates

I added the ability to update users in the CLI! This includes:

• folderharbor users update username <id>: Change the username for a user
• folderharbor users update password <id>: Change the password of a user
• folderharbor users update failedlogins <id>: Clear a user’s failed login attempts
• folderharbor users update lock <id> [yes|no]: Lock/unlock a user (will lock if you don’t provide a “no” argument at the end)

Users can do some of these for themselves through the folderharbor accounts command, these are for admins to do them. Granting/removing roles/ACLs/permissions isn’t added yet, but will be later.

Landing

I added a downloads page and made a couple small changes.

Server - Core Functions

Wrote some core functions for file management! Namely, I added writeFile, createFolder, deleteItem, and moveItem. These took some work to design, integrate with the ACL system, and generally implement.

Server - API

I added a new API route, POST /files?path=, which creates an empty file at a specific path, with appropriate checks.

Landing

I started on the landing page! This will be a relatively small page I will host to explain what FolderHarbor is, why you should use it, how to use it, and how to access the demo instance (as well as providing download links through GitHub Releases).

CLI - User Admin

There are some new CLI commands for user administration! Namely:

• folderharbor users list: Get a list of all usernames on the server and their IDs.
• folderharbor users get [id]: Get info about a user by their ID, which could be either full or limited based on your access level (server permission).
• folderharbor users create: Create a new user by providing a username and password, get an ID back
• folderharbor users delete [id]: Delete a user by their ID

CLI - Session Revocation

Users can now revoke their own sessions with the folderharbor account revoke-session [id] command (you can get session IDs from the folderharbor account get command). This allows you to sign out compromised devices remotely.

Server - Access Levels

Tiny change, but the server now includes access levels when responding to a GET /admin/users request, based on permissions (users:read or users:read.full). This just makes it a bit easier to tell what data you’re getting back.

CLI Account command

You can now run CLI commands under the folderharbor account subcommand to do various things! Namely, these:

• folderharbor account get: Gets your current account info, like your username, account ID, and active sessions.
• folderharbor account username: Changes your username to a new one! This command includes some extra protections, like running GET /clientconfig on the server and checking if username changes are allowed before prompting to enter a username.
• folderharbor account password: Changes your password! Due to the server changes (mentioned below), this will also automatically sign you in again with your new password.
• folderharbor account failedlogins: This resets the failed login limit for your account, nice if you were trying to sign in on another device and repeatedly mistyped your password.

PATCH /me

Updated this API route to have more security around password changes! If a user wants to change their password, they have to be using a session created in the past hour. They also will be signed out across all devices after the change succeeds, including the current one.

GET /me

I added info about failed login lockouts here! This means users can see if their accounts are locked out for excessive failed password attempts (which allows existing sessions to continue, unlike an admin-imposed lock).

Server Core

I added a new core function for revoking all sessions for a user! This is used currently for password changes, to invalidate sessions created with the old password.

I did some various improvements to the CLI and server!

Server core

I added two functions to the server core!

• Reading files
• Checking whether an item is a file or a folder
  These are mostly just convenience functions, and the reading one has permission checks.

API

Altered GET /files?path=[something] to check if an item is a file or folder, then call the appropriate function and return the appropriate contents.

CLI

I added configuration through the Viper tool, as well as adding better API error handling, more messages, better auth handling, and generally many improvements. I also now allow specifying a server URL in the login process.

Started on a CLI! I’m teaching myself the Go programming language (mainly by using Go by Example) for this CLI.

CLI auth

I built auth for the CLI! This includes a proper interactive login that uses things like hiding the password in the shell (the way, for example. sudo on most Linux distros does). It points at the FolderHarbor server to do this, but doesn’t save the token yet (I still need to add the config file).

Hi again! :3
I added some new endpoints and generally did server work.

Files Endpoints

New /files route! This allows you to list files you have permission to view by calling GET /files, with an optional path argument (like GET /files?path=/srv to descend into directories. This was fairly complicated, as I needed to design a bunch of code around it (as well as taking Node.JS Dirent objects and re-formatting the response to include more data).

Server Landing Page

As the FolderHarbor API operates on a standard web URL and needs to be publicly accessible and known by users (so they can point their clients appropriately), some people may try to visit the URL in a web browser. Thus, I threw together a fairly basic page to tell them that this is the server, not a client. The CSS will likely be improved later, I just want something that works for now. Express static routing was also so confusing, and took me like 15 minutes of debugging to set up properly.

Note

And yes, I am devlogging from a school Chromebook lol, I left my MacBook home but want to devlog now :3

Hey again! I did some various work here, mainly in the config and server.

Config

I added a new config setting, "globalExclusions"! This is a feature that lets you pass glob paths to force-deny. This works mainly as a security feature to prevent overly permissive ACLs from exposing files that shouldn’t be.
I also added some new API routes:

• GET /admin/config: Read the current config
• PATCH /admin/config: Change certain config settings (namely, global exclusions can’t be changed this way)

Core

I added a new core function to allow editing the config! This has all of the appropriate error checking and security mechanisms, such as making sure that the config is writeable and that read-only settings can’t be changed. I also validate it with Zod before writing it, for extra safety. Figuring out how to handle merging the current config and new updates safely was a little complex, but I made it work! :3

Admin Panel

I templated out the admin panel! This was only like 3 minutes of work (duplicating the web folder and changing a bit of HTML), but I felt it should be mentioned.

Wow, I did quite a bit of work! :3

Error Handling

Putting this first since I worked hard on it! API routes are now much less likely to error, and they have more and better checks (especially on PATCH routes)! The messages if they do error are also more normalized across routes.

Core

Core functions for creating users, roles, and ACLs now all support passing some extra information to them! This means that you can directly provide allow and deny path arrays to an ACL when you create it, instead of having to create an empty one and then edit it right after. This is implemented in the role and ACL creation routes, but not user creation (as user-editing permissions are more fine-grained, but not user-creating ones).

I also re-ordered some existing core functions to match the create-read-update-delete structure.

Roles API

I finished the Roles API routes! Namely, these are:

• GET /admin/roles/:roleID: Get a role’s info by ID
• PATCH /admin/roles/:roleID: Edit a role
• DELETE /admin/roles/:roleID: Delete a role

ACLs API

I implemented the entire ACLs API, and all of its routes! That is:

• GET /admin/acls: List ACLs
• POST /admin/acls: Create a new ACL
• GET /admin/acls/:aclID: Get an ACL’s info by ID
• PATCH /admin/acls/:aclID: Edit an ACL
• DELETE /admin/acls/:aclID: Delete an ACL

This work was basically all just API routes!

Users

I finished user routes, for now at least! This added listing (GET /admin/users) and deleting (GET /admin/users/[id]) accounts.

Roles

I templated out the entire roles.ts file, as well as started making routes! Namely, these are:

• GET /admin/roles: List roles
• POST /admin/roles: Create a role

Permissions

I added GET /admin/permissions for reading permissions for the server! These are hardcoded in the source code, so I made this route available without authentication. It just gives the IDs and descriptions for permission nodes on the server.
I also added some new permissions and cleaned up some outdated ones.

Bugfixes and Clean-Up

I fixed some assorted errors! Mainly this was that you couldn’t remove all permissions from a user, now that is possible. I also added some better error handling in some places.

This last few hours of work were a bit crazy for me, as I did this work across multiple days, but here’s the things I did!

Web Panel

I started working on a web panel! This was really just some basic Vite setup work, and I haven’t really written much code on it yet, but I will need this in the near future.

User Editing

I added some API routes for editing and locking user accounts! These could use some more work, but they function on a basic level, and include things like checking for permission levels (the app splits user editing into users:edit and users:edit.full permissions to give more granular control).

Clean-Up

I removed some error-checking conditions that could never be reached (like username_used in a method that didn’t update usernames), as well as removed the GET /auth endpoint (GET /me gives much more information and is generally better).

Bugfixes

There were a few assorted bugs I fixed, but the main one was that sessions kept revoking after session 1. Turns out, this was because I was checking if a session ID matched a valid user in the wrong way, and I was calling the validation function with session.id (the session’s ID) instead of session.userID (the user behind the session’s ID). I fixed this finally, and it took so much debugging.

Notes

Finally, I hope you enjoy this new, more structured devlog style! Sorry I took such a long break from this project, I was making a Slack bot (as well as school stuff). I’m also now using Insomnia by Kong for my API request screenshots, instead of just raw cURL.

I finally finished the bot! I also added a few new features.

README things

I cleaned up the README and made some improvements! I added more instructions, clarified some things, and added some extras (like the Not by AI badge)

Automatic approvals

The bot now has a config option to automatically approve requests to join

Errors

There were some various errors in different parts of the code, as well as things like ESLint not working properly. I fixed these! I also fixed a config permissions issue.

Restructuring

I restructured the code to be in a /src directory, as that makes it easier for me to normalize paths and work with TypeScript compilation.

Workflows

I added GitHub Actions workflows for building and linting the code. This was a bit complicated, as I had to work around pnpm and Docker Buildx (I’m running this on a Raspberry Pi 5, so I have to build for both x86_64 and ARM64).

The bot is finally ready to ship, so that’s my next step! :3

I added a Dockerfile and configuration system, as well as many improvements to the README!

Config

There is now a config system! This makes it easy to change things, such as the time daily recaps trigger and what channel is used. It uses Zod for validation, and I migrated most of the hardcoded values to config ones.

README

I wrote most of the README.md file, including a very detailed features list and some deployment instructions. It isn’t done, but I still added a lot to it.

Dockerfile

I added a Dockerfile for running the app in Docker! It’s fairly simple, but it should work properly!

I did a few things during this time! One thing is I slimmed the code down quite a bit, and removed some duplicated code (like how I was fully replacing blocks on messages before, instead of just re-using the existing blocks like I do now). I also improved some error handling and code structure!
Some other things:

/spacetime

There’s now a command to join my tier 2 channel! This replaces what I used to use a workflow for, but workflows are sadly gone from the Hack Club Slack now.
This is also better than the workflow approach, as it sends DMs on approval/rejection and has the ability to reject users. It’s also just generally more custom and better.

Universal Status integration

This integrates with a previous project of mine, Universal Status, to show my current status in my daily recaps. It’s just a fun little thing, and I plan to add more of these to the private recap soon!

Recap improvements

Now, the recap message links to my recap directly in the private channel, instead of the bot’s message. This allows people to read the public recap directly from my private channel. If I don’t have any messages in the thread, it will fallback to the bot message (because error handling)! It also has some extra checks, like it will only look at messages from my user ID (since sometimes people type in recap threads while they wait for me to post a recap).

I added daily recaps to the bot! There are two kinds:

Public recaps

These are in my public channel, and they are just standard recaps of my day. They are meant to use bullet-pointed lists, and this message is scheduled to send at 10:30 PM every night. It includes a button to mark it as done, which is used for the next kind of recaps! These buttons also have proper permission checks using my Slack user ID.

Private recaps

These are for my private channel, generally used for more detailed recap info that I don’t feel comfortable sharing publicly, or photos of myself. These are triggered by clicking the done button on a public recap, and they include a link to the public recap.

Both kinds of recaps have single-click “done” buttons, as well as reactions (the bot reacts with the 💤 emoji on message send, then changes to ✨ after the recap is marked done, as a way to indicate to people looking at the recap message).
Recaps are mostly done, and scheduling them works, but I still need to improve some things with them. I also want to add more details to private recaps, likely some of which will be pulled from my previously made Discord bot!

I started building the bot! I’m using Slack’s Bolt library for NodeJS, and I have gotten a fairly functional development setup. I built it to work in either socket or HTTP mode without any changes to the code (just environment variables).
I also added a feature that yells at people for threading on a user group mention! This uses an ephemeral message so that people will hopefully reply on the part that doesn’t ping people for every message. For now, it only detects when the top-level message is a user group mention, not when a user group is mentioned in the thread (though this may change).
I have had to write regex and get around TypeScript problems in this time, and it’s fairly complicated since I’m trying to not use any AI in the process of making this bot.

I added many new API routes! This took so long because I had to write a bunch of code to integrate Express APIs with the underlying functions, while still having proper error handling and a good user experience. I added routes for:

• Revoking sessions other than the active one (DELETE /me/session)
• Getting current user info (GET /me)
• Editing some info for the current user (PATCH /me)
• Admin API to read other users’ info (GET /admin/users/[id])
• Client configuration, aka things clients need to know that vary based on the configuration settings (GET /clientconfig)

I also made some permission changes and changed a few other things that needed improvement!

I built a re-implementation of directory listing that accounts for ACLs! This, as well as the newly improved checkPath() function, runs very quickly (thanks to the micromatch library) and manages what paths a user can access. This new method also accounts for parent directories, so if the user has access to something in, say, /home/nova, they can see /home when listing from the root directory (/) without being able to see other directories (like /etc). In /home, they would only be able to see nova/, not any other users’ folders.

Also, I created a permission system for the admin APIs, so that users can only do things their permission levels allow for. This follows a fairly standard RBAC structure, with permissions like roles:create and users:read.full.
Finally, I added a bunch of assorted helpful methods! These allow for extending sessions, deleting roles, and a few other things.

I added quite a bit of authentication-related code! Most of this time was building strong authentication middleware, so I wouldn’t have to worry about spotty auth implementations. I set this up with two functions, auth() (called on every request) and enforceAuth() (called on some endpoints that require auth). This took a while, as this is my first time making my own Express middleware and handling all of the related TypeScript things (like abstracting the Request to add req.session and req.sessionErr). I also had to figure out cookie handling, as I built the auth middleware to accept header-based auth or cookie-based auth, whichever is better for the client.
I also added logic for ACL path-checking, as well as standard functions for modifying and reading users/sessions/roles/ACLs. This completes the CRUD (Create-Read-Update-Delete) structure for managing all four of these (except for updating sessions i guess).
Finally, I split user handling and session handling into two files, since they are both getting very complex by now.

I started adding roles and ACLs! This isn’t done yet, but I wanted to devlog with my current progress on it. I also migrated user validation to be centered on user IDs, as then usernames can be changed without issue. This includes tying the sessions to users with a relation, so sessions will auto-delete if a user is deleted. Finally, I generally improved the database schema.

I added an API route for handling sign-ins! I also added rate limiting (for brute-force protection) and account locking (so admins can prevent users from using their accounts without changing the accounts past a boolean flag). This code is a bit more complex and went through many iterations, such as when I was going to make locks able to expire (which I ended up not doing when I implemented rate-limiting separately from the locking feature, originally I was going to combine them).

This was a very productive time! I added a database and an entire authentication system, including password hashing and short-lived auth tokens! This system is fairly complex and has some weird things, but the main usefulness is that sessions can be remotely revoked (useful if the token gets leaked) and can easily look up the user (hash token -> check against db -> get the user from the session row).
The first video shows the results of some session testing code (create test user -> authenticate to get a token -> check that the token works -> revoke the token), while the second is most of the code I was working on!

I added more to the API! I also added graceful shutdowns and started working on authentication. I was planning on building it around JWTs, and I spent over 40 minutes trying to implement them, but it all fell apart when I realized they were impractical based on my plans for auth.

I added a ton of permission checks and error handling logic. Also, I started building an Express app for the FolderHarbor API. though I haven’t added much. This time was quite technical, as I was adding a ton of checks related to the config file (such as making sure it was owned by the correct user and wasn’t overly permissive).

Started building the server! This project is structured as a monorepo, and I’m building this server process as a “core” of sorts in TypeScript. During this time, I started structuring the repo/code. I also used Commander for command arguments and Zod for parsing the config file.

I added autosave and a few new pages! Among these, there is now a place to view all journal entries, instead of just ones from the current day. I also got the basic structure prepared for a settings page, which will have more features soon.
Autosave works similarly to my previous projects, but I modified the delay and implementation a lot. React prop serialization doesn’t make it easy, but I’m making it work.

I worked on the journal editor and added a page for viewing journal entries! For the editor, I just built something basic for now. It doesn’t have rich text support yet, though I plan to add that. I also coded the ability to delete journal entries.
For the view page, it’s just a simple display of whatever you wrote in the entry. It will be improved upon, but I wanted the foundation.

I added better error handling, as well as a new API route for updating entries. I’m trying to build a lot of the journaling backend before the frontend, as the frontend has a lot of complexity that I want a stable backend to build on.
(The database integration does work, as shown in the previous devlog. I intentionally triggered an error to show that the messages work.)

I built a basic API for adding journal entries! It’s far from perfect, but it will allow me to start on the entry editor. I also added an entry display to the main page, though it doesn’t really map to anything yet. I will add more to these when I work on this next!

I started building the app! I don’t have much yet, but I have added a database, authentication, and a basic UI.
This was mostly just things to get me started on the app :3

I updated a bunch of things to prepare to ship! I deployed the site to a DigitalOcean VPS, added Docker build support, added CI workflows, and fixed bugs. I also drew an 88x31 button (not tracked on my time). I was originally planning on deploying this to Heroku, so I had some hours working on setting that up (~1 hour) that I had to revert due to an issue.
I had to make some pages “dynamic” for them to work in the production environment. I found a few other bugs as well, like RSS feed links not properly leading to my website. I also made all of the widgets fully operational (including the fancy clock on the front page). Overall, this was very technically complicated to figure out, but I got the site fully operational.

I made a widget on my homepage for blog posts, made my projects page pull from the database, improved mobile support, and added project management to my admin panel! The projects page was originally constructed from an array, but uses the database now, so I can add new projects of mine without needing to edit the code. This also included building an admin panel for actually editing the projects. I also moved buttons around on mobile to make it look much better. I tried to change the background to an image, but my CSS didn’t end up working as I wanted it to, so I had to discard those changes.

I built an RSS feed for my blog! This allows my posts to be seen on other platforms, so people wanting to read my posts don’t have to check my blog for new posts when there aren’t any. I also built an auto-save system on my admin panel for when I’m writing, which made me work very hard to sync state across the database, server, and client (my state hooks were causing issues with how I had them set up). I also added some other things, like the ability to download posts as .md files, improving the viewer for posts, sorting posts by published date, and improving error handling.

I created most of the admin panel to manage blog posts, on both the frontend and backend. I had some challenges, mainly with sorting out how to build the PATCH requests to send as little data as possible (as I commonly write things on low data, and I want this to be practical for me to use). The end result turned out quite well, and I even implemented proper error handling!

I started on an admin panel for the blog! This was mostly backend work, as I wanted a lightweight auth system (it just uses an environment variable for the password).
It also included database work to add things like a “published” field to the schema (to allow me to hide blog posts without deleting them), as well as hooking the blog up to an actual PostgreSQL database (running on my Raspberry Pi for development).

I built the foundations of my blog (and its associated database), a footer, an about page that talks about me and things that I like, and a widget that integrates with another previous project of mine (called Universal Status)! The ongoing Nest outage has made it a bit harder for me to do this, as I typically use Nest PostgreSQL while developing, and Universal Status is hosted there. I got around the latter issue by running a local instance of Universal Status and pointing the site there, and the former issue by editing my function to temporarily use placeholder data (in the same schema the database will follow). This one was quite fun to work on, and I’m planning some very cool things with widgets soon!

got started on the site and built the foundations of my project list, home page, and “accounts” page. also included making a header and figuring out some fairly complex things, like tailwind css gradient text!
a lot of this work was getting started, and i am planning on starting the blog and doing a bunch of refinements soon.