Operation Vault

I’ve made some minor tweaks to the UI to hopefully make sync groups more intuitive. Additionally, there’s a minor fix to how the plugin handles renames.

Sync groups are still a bit messy (you can see that in the video), but they eventually work. It’s not consistent, so it’s not the easiest to debug at the moment.

Changelog

• docs: update note
• fix: clearer sync group instructions
• fix: check sharedItem.length instead of !sharedItem
• docs: update instructions

The automated changelog is too long to attach here, so I’ll just add a screenshot. I’ve made changes to the way sync groups are handled when sharing and when encrypted messages are received. I’ve also altered linting rules and added automated linting via GitLab Workflows so that I can know what to fix before the Obsidian team reviews it.
There were some inconsistencies with how individual and group files where handled, but they’re more streamlined now.

These are mostly really minor changes, mostly to the server to help with deployment. An example of a failure in deployment can be seen below.

Changelog

• fix(server -> user list): added nickname field
• fix(server -> cleanup loop): increase threshold for cleanup activation
• fix(transport): update connection configs
• fix(server -> logging): Add proper type
• fix(plugin transport): add extra safeguards to ensure that there is data being written.
• docs(plugin): bump the version
• feat(server): use env variables for S3

I’ve made changes that help when upgrading from a version <0.1.7 to v0.1.7 or greater. This should’ve resulted in a minor increment to the version number, so that’s my bad.

There is also now a confirmation prompt every time you alter (yes, per character) the global channel name.

Changelog

• fix(plugin): add confirmation for changing global channel name
• feat(plugin): automatically migrate to new port if old one is saved
• docs(plugin): add details about port and restart
• style/

This should be the last devlog before the first ship. I’ve made a bunch of changes to the plugin that are needed in order to comply with Obsidian’s minimum bar for plugins. You can follow the plugin approval process at https://github.com/obsidianmd/obsidian-releases/pull/11784.

I’ve also done some testing on mobile and it doesn’t seem great, so it’ll definitely be something I look at before v1. I’ll also be taking a look at adding comments, which’ll probably arrive with v0.3. A UI overhaul will also be in the back of my mind, not sure if it would be before v1 but I would hope so. I do want to balance out getting v1 out faster but also having it be a functional product.

Well, it sure has been a while. Would’ve made a devlog yesterday except silly me made my Hackatime stats private.

• I’ve changed the way the caret label is laid out (container and a grid layout), displayed and animated. This should mean more consistently expected layouts!
• The server is now hosted on a VPS! You can use https://opal.jchen.au:7200/ws!!!!
• As hinted in the link, I’ve decided on Opal as the name. This will take a while to transition though and I’ll need a logo
• I’ve made some changes that mean I should be aiming for a submission to the Obsidian team later in the afternoon. The approval process could take several months, so you can load it using BRAT in the meantime.

And that’s really about it. All the other changes I’ve made are fairly minor and I haven’t really added new features outside of a dev mode. I am planning to add comments soon, though, so stick with me!

Hello! I’m writing this when I should be studying for my exam block but I should probably write a devlog before too many hours are piled on. I’m making a number of changes and mostly refactored the styling for the dashboard. My works’s kinda all over the place but I do have a proper plan now as well. Most of my work for the frontend client will be pushed to a private repo on GitLab rather than the public GitHub one. Previously, I was pushing work to a private GitHub repo but I’ve decided to move that to GitLab cos I should learn to use that as well. Things are happening, Trust Me Bro™️.

Just a really quick update to show that I am, in fact, doing something! This is the grid layout that I’ll be using :)

Progress has been extremely slow recently with school and I don’t expect that to improve. It’ll honestly probably be even worse as we go, but I’ll try to optimise my time so that I can complete this.

Some minor tweaks in the design and changed some colours to comply with the WCAG normal text AAA standard. I also started on the development a bit.

I’ve made a basic sample of the web UI. This should be made fairly fast. I’ve also made versions of the buttons, which I haven’t tested yet and will do so at some point.

I might have less time to work on this project as school starts, so that probably means less time on devlogs too. Hopefully I can still release it. Next, I’ll be focusing purely on the UI before I implement any features.

Can probably start development tomorrow. Basically finished the designs for the editor today and started on the viewer. That’ll probably come at a later stage in dev, so maybe I can hold that off ’til then.

The editor is fairly simplistic, with a bar at the bottom to bold, italicise, underline, strikethrough or highlight text with markdown. I’m still debating on whether or not to render the raw markdown or to create a live preview instead. I think the raw markdown would be easier for now, and I can add the live preview in a post-v1 launch.

As for the viewer, the fancy serif fonts got to me lol. I had the idea of tags and filtering by date (which should already be in the metadata), alongside a reading progress indicator on the left when in an article. I was planning on having this really cool scrolling thing that Clerk used to have on their blog sites, but I’m afraid that it’ll take me too long to crack. Maybe, again, I’ll add it in a post-v1 launch.

But yeah, that’s about it. I’ll finalise and clean up some designs for tomorrow then probably research all the plugins I’m going to stitch together. Very nice :)

Very quick cleanup on the artboards today. Started on the editor view! Took a while to get what I thought I wanted, and there were def a number of other designs besides what you see (only the best two are shown). I think the one on the right is the best, as it allows me to expand into more side features into the future more easily, while also providing a bigger view for the actual document. It doesn’t have things like a mini file explorer, etc. because this is just an initial product. I expect most people to be using the main Obsidian plugin, while this web UI serves as a stopgap for people who don’t have Obsidian or can’t access it and need to edit something in a pinch.

I’ll probably finish the first draft of the editor view tomorrow, then I’ll get started on the hosted view. It’s then just refining everything, and then it’ll be development again! With any luck, this shouldn’t take as long as the plugin as I can probably stick a bunch of packages together. Hopefully.

It grows, day by day lol. I think the dashboard is nearly there. When everything’s revealed, it obviously very cluttered, but I’ll have some way to hide/reveal them. I think I should also add some labels for when they’re obscured, just for ease of use. I might start on the editor/viewer before I do that though, I usually take a good break before refining one of my designs so I can take a look at it with fresh eyes. Overall though, good progress! No doubt this process would save me hours in development.

A lot of making and deleting today, but I think I’m on the right track. You can see that I had four separate ideas for a dashboard and I’m currently in the process of cooking up the third one. As it’s in fairly early stages, you get to see what my artboard looks like when I’m making something with a more complex grid layout. Yay!

Basically, it’s a 3x3 grid with a large section in the middle for documents. I haven’t decided what the middle will look like first as I’m designing the outer components to get a general idea of a theme. I’m also not sure what to do with the four corners, but I suppose some icons that open up modals would be nice. Maybe one for profile, another for settings, one for support and not sure what to do with the last one just yet. We’ll see. I’m deciding on whether or not the outer rectangles should sort of “flip” to reveal themselves, or if they should just be static. It’d be cool and shouldn’t take horribly long to implement, but I do wonder if it’ll be intuitive enough. I’ll see later I suppose. And I’ll see you later too!

More design work! I mostly worked on finishing the current dashboard design (which I don’t agree with for whatever reason) and started another one so I can choose. Also made a couple of concepts for list-ish items and some buttons! There are four button types (passive, primary, secondary and outline), each of which has its own state for hover and with an icon. The primary button also has an alternate colour. I am liking the second dashboard design a lot better, but I think I’ll work on it and maybe even make a third one. Editor is next, obviously, but this is kind of hurting my nonexistent brain. Kinda liked it when it was just fixing bugs and making features

Design work! Looked at lots of inspiration (which I couldn’t log the hours for smh) and started coming up with some stuff. It’ll be heavily ShadCN/UI inspired, but I hate how some components look, so I’ll be designing and making it from scratch, probably. I might even come up with multiple versions to pick from!

I’m starting off with the dashboard, as I figured that’s probably the hardest part. After that, I’ll design the editor and then somehow figure out the viewer for hosting. I’ll get there eventually. Also somewhat settled on a stack for the web UI: React w/ CodeMirror integration, + more for the hosted view packaged with Vite. Further details when I actually start on that Soon^tm. :)

Honestly, the link feature took way less time than I thought it would. If I knew it was this easy, I would’ve just done it yesterday ngl. Anyhow, it is completed now and a demo can be found in the video. I am looking for another name for the V1 release… Will consider a few options later.

I did take a look at what I should use to make the web UI. I’ve always used vanilla HTML+CSS+JS, but I might use React this time, as I do plan on integrating a “hosted” view, alongside the editor. I might also push back the Excalidraw and Canvas support until a post-V1 update. Otherwise, I’ll probably get started sometime next week.

I made lots of refactors today, improving code health. It wasn’t great lol, but I do think it’s a lot better now. It’s still pretty bad, but everything is functional and it’s not in the red (well, it was—I just made refactors to move it back into the orange). You can kinda see where I just kinda stopped caring abt the code health lol. Oh well. That’s a problem for future me. You’ll meet him tomorrow. See you then!

Version history has finally been implemented! It opens up a sidebar, which shows a reverse list of previous versions that the user can preview and restore to. Restoring is like detaching a commit in Git, so it doesn’t remove newer versions. I’ve added a video of how it works and two screenshots of the UI (after and before). Overall, very happy with this change and I didn’t spend too much time on it :D.

I, however, have not implemented viewing changes in the previews as that would move me deep into using fast-diff. I plan on migrating to CodeMirror’s native implementation sometime soon, so I didn’t want to overcommit on fast-diff.

Tomorrow, I’ll implement direct links to open in Obsidian, and then I’ll begin researching and planning the web UI. So excited! Cya later!

I could’ve sworn I completed the devlog, but I guess it didn’t upload. Oh well.

I see that we have one follower now! Thanks for joining me and I hope you’re excited for this :). It means a lot to me.

I completed snapshotting yesterday, alongside massive improvements in performance, which you can see below. I’ve recorded two scenarios, a before and after for each, where the sample text is narrated in text (one is encrypted, the other is not). The only change I made to make this drastic improvement in performance was the encryption/decryption. I simply implemented a cache so that a key didn’t have to be generated every time something had to be encrypted/decrypted. In hindsight, not sure how I didn’t do that bu, butad that I did eventually.

Most of my time today was spent on re-doing changes that I accidentally undid cos I did them on a detached head and then switched back to master. Not great, but everything does work now, so I suppose it’s fine?

Anyway, I’ll probably spend some time on version histories today. I might skip the automated redaction/section freezing until a post-v1 update. It seems kind of niche and I don’t want to run out of things do add lol. So, yeah, it’ll probably be the version histories, direct links and then we’ll be on the web UI while I work on minor bug fixes and improvements for the plugin. Good time. Cya!

There really isn’t anything to demo today, but I’ll include a short video anyway just to prove that my changes haven’t broken functionality… Yet…

Anyway, I did a lot more than I thought I would get done today. I managed to essentially complete the pipelines for sharing, receiving and reconnecting. Each client now downloads/upload one, two or all three of the actual file, state file and manifest. Most of the time, it’ll download the manifest first, to see if it needs to go further. If changes were made since the last sync, it’ll download a state (.yjs) file. If it’s the first time connecting, it’ll download both the file (.md) and state files (.yjs). All three files can be encrypted using the same key (the pin), but they can be decrypted for easy sharing as well. When downloading, it always checks to see if there’s anyone online first, as it’ll prefer to ask them for the file.

Next, I’ll be working on snapshotting and then testing everything thoroughly. No doubt I’ll spend ages tying to figure out a single bug :(. Not much has changed with the timeline, but I am reconsidering snapshot-based syncs. I’m afraid it’s a feature that not a lot of people will use, so I might skip it and implement it in a post-v1 update. I will be working on version histories though, hate to even imagine the UI I have to make for that. Also still planning on the other features. Hopefully won’t be too much pain, bye! :)

I spent way too much time fixing one bug with the live cursors. I think my compiler failed, so none of my patches were actually being applied. Took me way too long 😭. Anyway, I did a bit of both of what I mentioned yesterday. I planned out what the system should look like.

At first, I was debating between timeouts on the client and then a download or sending a download request unencrypted but I figured since I was going to rewrite the cloud integration anyway, I should probably do it this way. In this method, I have a manifest that the server fetches. The client will receive it and know which file it belongs to by which channel it comes in from (remember, each shared item has its own channel), reducing the need for that field on the manifest. It can then request decrypt the manifest if needed and then ask the server for a snapshot if it’s the first time or a .yjs file if it’s reconnecting. Snapshots and .yjs files will be uploaded by whichever client comes first alphamumerically, logged in the manifest. The naming and data structures also make it possible to reconstruct a version history.

The plan for tomorrow is to implement the manifest and allow file uploading/downloading when the host if offline. After that, it’ll probably be snapshot-based syncs and then version histories spread throughout the rest of the week. After that, I’ll work on section freezing & automated redaction, followed by the direct link to open in Obsidian and/or the web UI (which’ll take a while D:). Lots of fun :)

Today, I finally implemented highlight selection and nicknames! I also improved the performance of the live cursors, following in near-real-time!

I also made some much-needed enhancements to the actual code without making any changes to the actual functionality, including flattening one method and enabling and complying with strict null checks. I’ve also made a utils.ts for commonly used functions in the code.

Still debating what I should do tomorrow. On one hand, my code is all over the place and yet I need to get a lot done. Ig I’ll see if I’m bothered to do the storage rewrite tmr. ’Til then!

Spent wayyy too long on this, but I’m genuinely so happy with it. Live cursors! It doesn’t update as fast as I would want it to, but that’s always fixable (I think). I’m really, really happy with how the tags turned out, especially with the automatic flipping and transitions. It took me a while to realise that the translate property existed in CSS. After that, I was able to animate the transition from the hover but not the left-right anchor.
I also made improvements/fixes in:

• Reconnect attempts (properly clears the recycled heartbeatInterval)
• Removal of .yjs state file
• Messages exiting early due to an unnecessarily strict check
  Tomorrow, I’ll quickly implement selection highlighting and then probably work on nicknames. After those, it’s the dreaded overhaul of communications that’ll bring version histories, snapshots and downloading from the cloud. Fun. Anyway, I’ll see you later hopefully (I’m not schizophrenic I swear)!

Didn’t get a lot of major features today, but a lot of cleanup. I won’t list all of them as they’re in the GitHub commits list anyway, but I did manage to refactor some code and add two new features! The first is automatically removing sync when files are deleted. The second is a method to attempt to reconnect to the server when a connection is dropped. Oh, and a third! Automatic prefills for the file you’re currently on. At the moment, you do have to select the prefilled option for it to work, but I hope I can fix that tomorrow. Oh, and the status bar and channel count notice updates are now for the main channel only! No more clutter - yay!
I couldn’t get to the downloading file today as I realised it’s far too much and it’s actually quite closely linked to the version history and snapshot syncs, so I’ll develop it when I get to those, which is right after live cursors and selection highlights, a multiplayer classic!
Tomorrow, I also need to figure out why the .yjs files aren’t deleting. I’ve already tried three different methods and none of them work. Even worse, there’s no error, so it’s basically a game of trial and error now. Anyway, that’s it for now. Hopefully I can get live cursors working tomorrow :).

Today was… Boring. Lmao. I basically spent all day looking at diagrams and fixing merge conflicts. But it does mean a better and hopefully more functional codebase. Tomorrow, I’ll finally be fixing not being able to get files when the host is offline. Hopefully, this’ll force me to make sync even better (less likely to break lol). In the video, you can also see that I need to properly clean up the .yjs files, which I’ll do first thing tomorrow. If I still have time, I’ll continue working on the live cursors. After that is selection highlights and then version history! :D
In one of the screenshots, you’ll code related to tracking files dynamically for changes to sync groups. Not sure if I’ll want to work on it after all the trauma but maybe I will tomorrow. Who knows. I’ll see you then I suppose (speaking to no one D:)

Today (and by that I mean technically yesterday) was bloody stressful. So many things went wrong and the more things I fixed, the more things broke. I swear I didn’t go anywhere near the merging code and yet it still broke, which you can see in the video. I ended up writing so much more code than I had to and I don’;t think I know where anything is anymore. Maybe I’ll continue cleanup tomorrow and probably rewrite a bunch of things. Maybe I can get Claude/Gemini to make a diagram of how the functions are connected and then simplify things. That sounds like a good idea.
I didn’t even manage to fix the double connected notice bug 😭.
Anyway, time to go get some rest now. Will probably fix up a bunch of things tomorrow, refactor some stuff and continue writing code the day after. I got the free CodeScene subscription from GitHub Education so will use that to analyse my code. That’s it for now though, cya!

I went out today, so didn’t spend much time working.
I did, however, get one new feature complete today. It’s more of a QoL thing, but the file is now tracked even if you move/rename it! However, it’ll only track if the plugin is loaded (i.e. enabled while Obsidian is open and moved with Obsidian). In the future, I plan on detecting whether the file exists (so if it’s been moved/renamed) and then (at the user’s command), hash files in the vault to recommend files to readd to the sync. It’ll be cool :)

I didn’t really get much cleaning done today, but I really do need to fix that duplicate “Currently online: x” notice. In the earlier part of the video below, you can also see a little checklist/roadmap I made for the project so that I stay on track. Hopefully this’ll be fun :D.

Anyway, that’s it for today, but I’ll be back tmr with even more changes and maybe even more feature ideas! Cya!

Edit: In my hurry to make this before the day ends, I forgot to mention what I was working on lmao. I’m currently working on sync “groups”. In the checklist/roadmap, you would’ve seen me crossing out “folders” and replacing it with “groups”. Groups will make it easier to manage multiple files’ syncs at once without the restriction of being in particular folders. Basically, it’s folder syncing but better in every way! I was also figuring out how to handle files nested deeply within a shared folder and this solves that problem so yay :D

Today (I’m going to do day-by-day devlogs from now!) was mostly focused on cleanup and I expect tomorrow to be the same (but more consolidation). I mostly focused on cleaning up the amount of notices the plugin would make and making logs a bit more verbose. This is in addition to properly leaving channels when the plugin is offloaded.

I did implement a feature, though! The “inbox” feature allows the user to configure where files are placed by default. Tomorrow, I plan on working on tracking files as they’re moved around so they aren’t confined to the inbox :). After that, probably tracking entire folders and ensuring that files can be downloaded even when the host is offline. Currently, the host has to be online as the message to download a file is encrypted and only the server as access to the cloud storage. After that, probably either live cursors (finally 😭) or version history. After those two, I’ll work on getting the freezing things working. A lot planned!

Currently halfway reworking encryption but decided I should probably write another one of these before I forgot.

Basically, I implemented file syncing (calculating differences, the one you see in the recording) but then deleted it. The reason being, this implementation didn’t support merging (so the last write won) and synced every file in the folder (unless it was deleting a file). I implemented one that was per-file and am working on customising which files are allowed to be shared. As part of that, I have to properly implement encryption instead of sticking to one hardcoded secret. Soon? Perhaps?

I uh…. Kind of made a teeny tiny amount of changes since my last devlog… I got back from holiday this week so yeah.

Starting off with something small, I made plans for how I’ll implement web workers. Just a quick 15min diagram, nothing much.

There are a number of things after that, unfortunately. I’ll try to list the smaller things I remember:

• Simplification of functions and methods
• Using a [OPV] prefix for logs so they can be easily identified (when there are other plugin logs)
• Using more expressive logs than just .log (.info, .warn and .error)
• Other cleanup and flattening

I did, however, add these major features:

• S3-compatible backup snapshot storage
• Realtime client tracking and status
• Rewrote config to allow for custom endpoints (self host?) and channel names
• UI for the above three features
• Status bar in the bottom right (click to view a list of connected clients!)

This did involve a lot of rewriting code (including changing how streams work, sigh) but it’s all for the better. Files stored in the cloud are encrypted and are deleted upon the shareID being revoked. The PIN is optional. A future improvement would be having the server actually talking back and confirming certain actions so that the client isn’t as blind, but that’s a problem for future me to deal with…

I would also add more screenshots as there are more things I worked on, but I’ve hit the cap :(
Edit: goddammit I refreshed the page and it didn’t show that one of the images I uploaded was still there… :(

Wow… That’s a lot! I’ve added file transfers, which comes with chunking and I’ve also added encryption! Also fixed a few bugs there and there :) Also tested the file transfer feature with a 1.5GB video file (you can see it in the ss). Obsidian is usable while transferring it, although I may move the transfer to another thread in the future to make the performance even better. Next, I’ll be working on refining the current implementation and then adding realtime.

Rewrote everything from the ground-up and I can now connect from Obsidian! I’ll still have to figure out how this’ll work in prod but it’s one step closer :). Theoretically, building everything should be pretty straightforward now.

I’ve managed to make this work with my browser (yay!) but not Obsidian (aw). I think I know why though, but my code’s gotten extremely cluttered, and I’m not that far in, so I’ll just restart this, and hopefully it’ll work with Obsidian :D. At the moment, it’s only establishing a connection to the server.

I mostly thought through the idea and wrote down specifically what features I wanted before planning the tech stack I’ll be using. This is, once again, after a couple of nights just lying in bed thinking about it. Planned in Obsidian ofc :)