Server - Sign-in Page

I decided to make a page in the server where users can get a token directly. While it’s just a basic HTML+JS thing, I had to work out a lot of stuff with event listeners that I’m not used to (as I use React in the majority of my projects, but am not using it here). It’s a bit of spaghetti code, but it fully works, and I did it all myself! (like everything else in this project, I am not using AI for any part of this)
This is meant to be used for token-based WebDAV login, if users prefer it to using a username and password. It can also be used to directly call the API, if users would prefer using this token to directly calling POST /auth. It will be more useful if I add SSO or 2FA in the future.

Server - WebDAV

I now have functioning WebDAV! This took a very long time to properly implement, and I encountered many roadblocks. For example, my permissions model wasn’t working in WebDAV, and the fix (below since it was a core function issue). When I fixed it, suddenly clients were getting errors loading directories. A lot of debugging later, I figured out the issue was coming from the webdav-server PhysicalFileSystem, so I had to extend it with my own class and write a WebDAV equivalent to the listDir() core function.
I also now filter out certain auto-generated system files if a new config flag is set to true (default)! This means things like .DS_Store will not actually be saved to the FolderHarbor server’s hard drive if the setting is on.
WebDAV is a bit slow right now, I’m going to try improving it later.

Server - WebDAV Logging

I added logging for WebDAV file changes! This uses a switch statement and some extra checks to appropriately log many changes. I’m not logging directory listing, as trying to caused… issues (to say the least). I got 6 pages (120+ entries) of logs in around a minute logging those.

Server Core - checkPath

Through the above-mentioned debugging, I learned my checkPath() function wasn’t accounting for global exclusions. As they were added later, I forgot to add them in this route! That caused the overly-permissive WebDAV issue, but fixing it exposed the second issue. This is now fixed, though, and accounts for all path attributes. I may need to split some of the functionality more for security reasons, but I will see later.

Removing Web Panel

I’m not going to be able to work on the web panel (given that Flavortown is ending so soon), and WebDAV makes it far less needed. Thus, I removed it from the repo, README, and landing page.

Server - WebDAV

This was most of the time on this devlog! I’ve been working hard on adding WebDAV support, and though it is far from done, I made a lot of progress. I’m using the webdav-server NPM library, and my main struggles were figuring out their user and privilege manager structure (I had to make authentication components to plug FolderHarbor’s auth system into WebDAV). I also managed to allow both token-based auth and username+password auth, as it’s more convenient to just use a username and password, but tokens are easier to revoke (if they get leaked). The key is just entering token_<your token> as the password, the username gets ignored if your password is prefixed with token_ and it checks your auth through a token. There will be a website to get a token with your username and password soon, built into the server!

Server - Better audit logging

Added some better audit logging things! Namely, this is pagination (log pages of 20 entries each), and logging some existing file API route actions.

Server - Audit Logging

I started implementing audit logging! Now, every admin action (except listing things) is logged, as well as users signing in and failed password sign in attempts. Logs are stored in PostgreSQL, and they took so long because I designed a bunch of functions to work with them, as well as letting them contain all kinds of different data (the TypeScript types for this were a bit crazy :heavysob:).

Server - Logs API

Kinda in the same vein, the server has a new GET /admin/logs route that returns all logs. This could use some improvement, such as pagination, but that’s a thing for future me to do :3

CLI - Roles

New CLI command and route for listing roles. I haven’t gotten to add more role features yet, but likely will soon!

Server - Permissions

I added some new stuff for permissions! Namely this is the new getEffectivePermissions()function, which takes a user ID and resolves all of their permissions (permissions directly set on their user, as well as permissions set on their roles). This is also integrated into GET /me and the CLI folderharbor accounts get command, but those were such small changes code-wise that I’m bundling them into this section.

CLI - User Updates

I added the ability to update users in the CLI! This includes:

• folderharbor users update username <id>: Change the username for a user
• folderharbor users update password <id>: Change the password of a user
• folderharbor users update failedlogins <id>: Clear a user’s failed login attempts
• folderharbor users update lock <id> [yes|no]: Lock/unlock a user (will lock if you don’t provide a “no” argument at the end)

Users can do some of these for themselves through the folderharbor accounts command, these are for admins to do them. Granting/removing roles/ACLs/permissions isn’t added yet, but will be later.

Landing

I added a downloads page and made a couple small changes.

Server - Core Functions

Wrote some core functions for file management! Namely, I added writeFile, createFolder, deleteItem, and moveItem. These took some work to design, integrate with the ACL system, and generally implement.

Server - API

I added a new API route, POST /files?path=, which creates an empty file at a specific path, with appropriate checks.

Landing

I started on the landing page! This will be a relatively small page I will host to explain what FolderHarbor is, why you should use it, how to use it, and how to access the demo instance (as well as providing download links through GitHub Releases).

CLI - User Admin

There are some new CLI commands for user administration! Namely:

• folderharbor users list: Get a list of all usernames on the server and their IDs.
• folderharbor users get [id]: Get info about a user by their ID, which could be either full or limited based on your access level (server permission).
• folderharbor users create: Create a new user by providing a username and password, get an ID back
• folderharbor users delete [id]: Delete a user by their ID

CLI - Session Revocation

Users can now revoke their own sessions with the folderharbor account revoke-session [id] command (you can get session IDs from the folderharbor account get command). This allows you to sign out compromised devices remotely.

Server - Access Levels

Tiny change, but the server now includes access levels when responding to a GET /admin/users request, based on permissions (users:read or users:read.full). This just makes it a bit easier to tell what data you’re getting back.

CLI Account command

You can now run CLI commands under the folderharbor account subcommand to do various things! Namely, these:

• folderharbor account get: Gets your current account info, like your username, account ID, and active sessions.
• folderharbor account username: Changes your username to a new one! This command includes some extra protections, like running GET /clientconfig on the server and checking if username changes are allowed before prompting to enter a username.
• folderharbor account password: Changes your password! Due to the server changes (mentioned below), this will also automatically sign you in again with your new password.
• folderharbor account failedlogins: This resets the failed login limit for your account, nice if you were trying to sign in on another device and repeatedly mistyped your password.

PATCH /me

Updated this API route to have more security around password changes! If a user wants to change their password, they have to be using a session created in the past hour. They also will be signed out across all devices after the change succeeds, including the current one.

GET /me

I added info about failed login lockouts here! This means users can see if their accounts are locked out for excessive failed password attempts (which allows existing sessions to continue, unlike an admin-imposed lock).

Server Core

I added a new core function for revoking all sessions for a user! This is used currently for password changes, to invalidate sessions created with the old password.

I did some various improvements to the CLI and server!

Server core

I added two functions to the server core!

• Reading files
• Checking whether an item is a file or a folder
  These are mostly just convenience functions, and the reading one has permission checks.

API

Altered GET /files?path=[something] to check if an item is a file or folder, then call the appropriate function and return the appropriate contents.

CLI

I added configuration through the Viper tool, as well as adding better API error handling, more messages, better auth handling, and generally many improvements. I also now allow specifying a server URL in the login process.

Started on a CLI! I’m teaching myself the Go programming language (mainly by using Go by Example) for this CLI.

CLI auth

I built auth for the CLI! This includes a proper interactive login that uses things like hiding the password in the shell (the way, for example. sudo on most Linux distros does). It points at the FolderHarbor server to do this, but doesn’t save the token yet (I still need to add the config file).

Hi again! :3
I added some new endpoints and generally did server work.

Files Endpoints

New /files route! This allows you to list files you have permission to view by calling GET /files, with an optional path argument (like GET /files?path=/srv to descend into directories. This was fairly complicated, as I needed to design a bunch of code around it (as well as taking Node.JS Dirent objects and re-formatting the response to include more data).

Server Landing Page

As the FolderHarbor API operates on a standard web URL and needs to be publicly accessible and known by users (so they can point their clients appropriately), some people may try to visit the URL in a web browser. Thus, I threw together a fairly basic page to tell them that this is the server, not a client. The CSS will likely be improved later, I just want something that works for now. Express static routing was also so confusing, and took me like 15 minutes of debugging to set up properly.

Note

And yes, I am devlogging from a school Chromebook lol, I left my MacBook home but want to devlog now :3

Hey again! I did some various work here, mainly in the config and server.

Config

I added a new config setting, "globalExclusions"! This is a feature that lets you pass glob paths to force-deny. This works mainly as a security feature to prevent overly permissive ACLs from exposing files that shouldn’t be.
I also added some new API routes:

• GET /admin/config: Read the current config
• PATCH /admin/config: Change certain config settings (namely, global exclusions can’t be changed this way)

Core

I added a new core function to allow editing the config! This has all of the appropriate error checking and security mechanisms, such as making sure that the config is writeable and that read-only settings can’t be changed. I also validate it with Zod before writing it, for extra safety. Figuring out how to handle merging the current config and new updates safely was a little complex, but I made it work! :3

Admin Panel

I templated out the admin panel! This was only like 3 minutes of work (duplicating the web folder and changing a bit of HTML), but I felt it should be mentioned.

Wow, I did quite a bit of work! :3

Error Handling

Putting this first since I worked hard on it! API routes are now much less likely to error, and they have more and better checks (especially on PATCH routes)! The messages if they do error are also more normalized across routes.

Core

Core functions for creating users, roles, and ACLs now all support passing some extra information to them! This means that you can directly provide allow and deny path arrays to an ACL when you create it, instead of having to create an empty one and then edit it right after. This is implemented in the role and ACL creation routes, but not user creation (as user-editing permissions are more fine-grained, but not user-creating ones).

I also re-ordered some existing core functions to match the create-read-update-delete structure.

Roles API

I finished the Roles API routes! Namely, these are:

• GET /admin/roles/:roleID: Get a role’s info by ID
• PATCH /admin/roles/:roleID: Edit a role
• DELETE /admin/roles/:roleID: Delete a role

ACLs API

I implemented the entire ACLs API, and all of its routes! That is:

• GET /admin/acls: List ACLs
• POST /admin/acls: Create a new ACL
• GET /admin/acls/:aclID: Get an ACL’s info by ID
• PATCH /admin/acls/:aclID: Edit an ACL
• DELETE /admin/acls/:aclID: Delete an ACL

This work was basically all just API routes!

Users

I finished user routes, for now at least! This added listing (GET /admin/users) and deleting (GET /admin/users/[id]) accounts.

Roles

I templated out the entire roles.ts file, as well as started making routes! Namely, these are:

• GET /admin/roles: List roles
• POST /admin/roles: Create a role

Permissions

I added GET /admin/permissions for reading permissions for the server! These are hardcoded in the source code, so I made this route available without authentication. It just gives the IDs and descriptions for permission nodes on the server.
I also added some new permissions and cleaned up some outdated ones.

Bugfixes and Clean-Up

I fixed some assorted errors! Mainly this was that you couldn’t remove all permissions from a user, now that is possible. I also added some better error handling in some places.

This last few hours of work were a bit crazy for me, as I did this work across multiple days, but here’s the things I did!

Web Panel

I started working on a web panel! This was really just some basic Vite setup work, and I haven’t really written much code on it yet, but I will need this in the near future.

User Editing

I added some API routes for editing and locking user accounts! These could use some more work, but they function on a basic level, and include things like checking for permission levels (the app splits user editing into users:edit and users:edit.full permissions to give more granular control).

Clean-Up

I removed some error-checking conditions that could never be reached (like username_used in a method that didn’t update usernames), as well as removed the GET /auth endpoint (GET /me gives much more information and is generally better).

Bugfixes

There were a few assorted bugs I fixed, but the main one was that sessions kept revoking after session 1. Turns out, this was because I was checking if a session ID matched a valid user in the wrong way, and I was calling the validation function with session.id (the session’s ID) instead of session.userID (the user behind the session’s ID). I fixed this finally, and it took so much debugging.

Notes

Finally, I hope you enjoy this new, more structured devlog style! Sorry I took such a long break from this project, I was making a Slack bot (as well as school stuff). I’m also now using Insomnia by Kong for my API request screenshots, instead of just raw cURL.

I added many new API routes! This took so long because I had to write a bunch of code to integrate Express APIs with the underlying functions, while still having proper error handling and a good user experience. I added routes for:

• Revoking sessions other than the active one (DELETE /me/session)
• Getting current user info (GET /me)
• Editing some info for the current user (PATCH /me)
• Admin API to read other users’ info (GET /admin/users/[id])
• Client configuration, aka things clients need to know that vary based on the configuration settings (GET /clientconfig)

I also made some permission changes and changed a few other things that needed improvement!

I built a re-implementation of directory listing that accounts for ACLs! This, as well as the newly improved checkPath() function, runs very quickly (thanks to the micromatch library) and manages what paths a user can access. This new method also accounts for parent directories, so if the user has access to something in, say, /home/nova, they can see /home when listing from the root directory (/) without being able to see other directories (like /etc). In /home, they would only be able to see nova/, not any other users’ folders.

Also, I created a permission system for the admin APIs, so that users can only do things their permission levels allow for. This follows a fairly standard RBAC structure, with permissions like roles:create and users:read.full.
Finally, I added a bunch of assorted helpful methods! These allow for extending sessions, deleting roles, and a few other things.

I added quite a bit of authentication-related code! Most of this time was building strong authentication middleware, so I wouldn’t have to worry about spotty auth implementations. I set this up with two functions, auth() (called on every request) and enforceAuth() (called on some endpoints that require auth). This took a while, as this is my first time making my own Express middleware and handling all of the related TypeScript things (like abstracting the Request to add req.session and req.sessionErr). I also had to figure out cookie handling, as I built the auth middleware to accept header-based auth or cookie-based auth, whichever is better for the client.
I also added logic for ACL path-checking, as well as standard functions for modifying and reading users/sessions/roles/ACLs. This completes the CRUD (Create-Read-Update-Delete) structure for managing all four of these (except for updating sessions i guess).
Finally, I split user handling and session handling into two files, since they are both getting very complex by now.

I started adding roles and ACLs! This isn’t done yet, but I wanted to devlog with my current progress on it. I also migrated user validation to be centered on user IDs, as then usernames can be changed without issue. This includes tying the sessions to users with a relation, so sessions will auto-delete if a user is deleted. Finally, I generally improved the database schema.

I added an API route for handling sign-ins! I also added rate limiting (for brute-force protection) and account locking (so admins can prevent users from using their accounts without changing the accounts past a boolean flag). This code is a bit more complex and went through many iterations, such as when I was going to make locks able to expire (which I ended up not doing when I implemented rate-limiting separately from the locking feature, originally I was going to combine them).

This was a very productive time! I added a database and an entire authentication system, including password hashing and short-lived auth tokens! This system is fairly complex and has some weird things, but the main usefulness is that sessions can be remotely revoked (useful if the token gets leaked) and can easily look up the user (hash token -> check against db -> get the user from the session row).
The first video shows the results of some session testing code (create test user -> authenticate to get a token -> check that the token works -> revoke the token), while the second is most of the code I was working on!

I added more to the API! I also added graceful shutdowns and started working on authentication. I was planning on building it around JWTs, and I spent over 40 minutes trying to implement them, but it all fell apart when I realized they were impractical based on my plans for auth.

I added a ton of permission checks and error handling logic. Also, I started building an Express app for the FolderHarbor API. though I haven’t added much. This time was quite technical, as I was adding a ton of checks related to the config file (such as making sure it was owned by the correct user and wasn’t overly permissive).

Started building the server! This project is structured as a monorepo, and I’m building this server process as a “core” of sorts in TypeScript. During this time, I started structuring the repo/code. I also used Commander for command arguments and Zod for parsing the config file.