i lost a bunch of time to not realizing that wakatime webstorm was bugged ;(

anyway here’s the feature list:

1. you can type in the shell syntax box and it populates the argument list automatically! (that’s the main thing here)
2. keyboard shortcuts for the argument list (see screenshot #1 for details)
3. aria up the wazoo
4. the move and delete buttons don’t work yet

…and this is now the longest set of attributes in the entire codebase, to make Firefox stop complaining about how the contenteditable shell syntax text box is somehow not focusable (…but it has a tab order by default??) and not labeled (ok fine but like if there are multiple of these i can’t have id) and also not a textbox

<div
    contenteditable="plaintext-only"
    spellcheck="false"
    class="-edit-stack-editor"
    role="textbox"
    aria-multiline="true"
    tabindex="0"
    aria-label="Shell syntax entry"
></div>

oh a third one? okay

REAL real parser with a REAL state machine. None of that stupid switch-case nonsense.
(The goal is to make the highlighting parser double as the parser for turning the Shell Syntax into the Argument List. The other direction will probably be much simpler, but has to be separate code anyway. We needed this to stay sane while doing parsing for arguments instead of just syntax constructs; understanding when an argument ends is very complicated if all of your state machinery is just a single switch statement with a whole bunch of cases)

Tip of the day: https://craftinginterpreters.com/
(Even just getting through the first half in Java has taught me so much stuff! A bunch of it got applied here for this devlog.)

Real parser. Stupid regex tricks weren’t enough.

(Remember, we’re trying to parse shell syntax here, so we need "quoted string support".)

In the process, we discovered that:

• contenteditable puts NBSPs (not the same thing as a normal space) when two or more spaces are needed, or a space at the end of a line
• Chrome just puts \ns in, Firefox uses <br> instead
• Chrome implicitly uses <pre> formatting on contenteditable elements
• dashed borders are drawn differently between browsers (i like Firefox’s more tbh)

anyway here’s your cool JS tip: switch (true). (TypeScript: switch (true as boolean) because we need the boolean type here to stop it from complaining about having case (boolean):)
you can use it like when from Kotlin (just make sure you’re actually coercing everything to booleans, truthyness isn’t enough):

switch (true) {
    case dollar && char === "$":
        dollar = false
        newNode("esc", null, i - 1)
        finishNode(i + 1, null)
        break
    case dollar && char === "{":
        dollar = false
        vari = true
        varName = ""
        newNode("var", null, i - 1)
        break
}

(if you can’t tell, we’re using this to make a stateful parser that’s just a single loop with a really big switch statement.)

Syntax highlighting for variables and $$ escape codes!

replaceWith is officially my favorite modern DOM API. Also, TIL that border on display: inline; elements doesn’t take up any space.

not much else to say except that it’s harder than it should be to highlight spans of text in the DOM. we have to walk the tree and search for #text nodes. and then replaceWith modifies the childNodes list (it’s a view, not a copy! and there’s no such thing as ConcurrentModificationException in JS to save my butt smh)

BREAKING NEWS: layout shifts are “terrible, awful, unfortunate, distressing, regrettable, and/or dreadful”

this segment:

• <x-replace> autonomous custom element (that’s the official name for the much less formal process of “make up an element name that has a hyphen in it”) that we scan through and instantiate based on the type, name, and/or list values using TS
• ArgumentEditor class to hold the Shell Syntax and Argument List together
• Try to contain the chaos that is the contenteditable by dissolving a bunch of tags that only provide formatting
• that’s subject.querySelectorAll("span, b, i, u, strike").forEach(it => it.replaceWith(...it.childNodes))

oh. it’s devlog time. okay

we got some basic editing LAYOUT done. THE FUNCTIONALITY IS NOT IN YET.

for highlighting the Shell Syntax box, i have a contenteditable <div> and a replica of it behind which has color:transparent and aria-hidden=true. the highlighting goes on the background element to avoid interfering with text editing.

list editor is really hard to design! for accessibility semantics reasons, it’s an <ol> (ordered list), but HTML spec says only <li> are allowed in there. here, actually, have a code snippet

<ol class="-edit-arglist">
        <li role="none">
            <button class="insert-gutter-button">+</button>
        </li>
        <li class="-argument">
            holy 
            <button>move</button>
            <button>del</button>
        </li>
        <li role="none">
            <button class="insert-gutter-button">+</button>
        </li>
</ol>

see that role="none"? that tells accessibility tools that the li is in fact not a list item. i’m also telling the browser not to increment the counter for the ordered list (unsure if this actually works with screen readers, need to test on Windows/NVDA most likely) with counter-increment: list-item 0;.

for the actual styling of the gutter buttons, li[role="none"] gets relative position to act as reference for the button’s position: absolute; right: calc(100% + 0.5rem); top: -12px; bottom: -12px;.

that’s basically it.

i have been bribed to work on this. because this 1h41m is a month old, this is a reconstruction of what happened based on commits. also yeah the attachment is useless

• something about closing databases
• (in the staging area) additional asserts in the fake storedprocs
• (in the staging area) experimental better stack insert thing?

how has it been two hours already?!

okay so … uhh… so there’s two things.

Start of the editor work

(see image)

editors are much harder than interpreters.
so much css. so many scrollboxes. non-cancellable esc key for some reason (why, <dialog>??)
yes the styling’s a bit broken for the form, and the X button really shouldn’t be there. it’s progress, I suppose.

New step types

• error (crash the recipe)
• branch (conditionals. see when from Kotlin for inspiration)

i’m going to need to add some way to do math at some point

Now that we have the input/output UI down, it’s time to work on the other half of the UI: the part where you actually tell the program what to do. The current idea is a bunch of cards using CSS Grid, where you can click a card to use it / edit it.

The fancy important action buttons are conic-gradient(in okhsl shorter hue .... (tip: you should use okhsl for colors in your website!)

Note: We still don’t have the functionality, so this session was just building out the HTML and CSS.
Continuing with the trend of using modern browser features for as much of the site as possible, we’re using the <dialog> element to provide the backdrop and open/close functionality.
Also, the cards are <button>s. do you know what <button>s aren’t allowed to have in their DOM tree? flow content. (that means div-type block stuff.) even though i’m styling the button as display: flex;, to be standards-conformant we have to use <span>s for the contents of the buttons rahhhhhhh

the new design has arrived! we have a breakpoint to switch between the horizontal and vertical layouts. we’re already relying on modern web features, so container queries are used here instead of media queries
also had to remove the background of the content because it was so jarring

ultra-high-res images (hosted on catbox):

• vertical layout
• horizontal layout

We need to redesign the i/o UX. People shouldn’t have to scroll up and down the page just to figure out which input file correlates with which output file. As a result, we’ve switched the page to grid, and are using elements that span the three columns to contain the wider stuff (like the i/o panel.)
The recipe area now needs to go below both the inputs and outputs, but it should still be visible initially (in the case of shared links, this is especially useful.)

Also, added an error handler that pops up a notification.

i regret to inform you that i had a massive skill issue making this notification with progress bar.

okay so we need to redo the UI a bit. people expect horizontal flows when doing conversions, and having to scroll to correlate files will be a problem.

also, notifications for download progress coming soon?

also also we had to disable the service worker because it was making changes not happen in dev which was very annoying

oh wow i forgot to devlog

we implemented like half of the recipe system. recipes are like scripts but not as good because we’re interpreting them with JS
planned step types:

• ffmpeg
• ffprobe
• set output file (done)
• bulk set variables (done)
• regex replace (done)
• regex match

i also added a ServiceWorker for caching because ffmpeg is like 30 MB. plan is to also have it get that cross-origin isolation so we can use the multithreaded ffmpeg library
(you’re supposed to look at the ‘transferred’ column in the screenshot)

we now support files that aren’t just audio! like images! and videos! just look at the image!

(ffprobe to the rescue here, we’re just parsing the output of ffprobe -v error -show_entries stream $FILE for each file)

i really hate bundlers.

the ones that work work well, except when you need a worker.
and then they all fail.

so we’re resorting to manually copying files out of node_modules.

and patching our output JS

funnnnnn
(in other news, I made Forgejo CI push to GitHub in order for hosting to happen on GitHub pages. try it out @ https://penguinencounter.github.io/fumux-prod/dev/)

did I mention I did a bunch of UI design too? that also happened.

made the drag and drop work! for some reason tampermonkey was not cooperating with it but then it started working so IDK what happened there

the clear button does not work and selecting files does not work

fun fact .ogg is application/ogg and not audio/vorbis. .oga might work, but we need to support .ogg

i got it to read files and tell me if they’re stereo or mono as well as the bitrate.

this is ffprobe -v error -select_streams a -of default=nw=1 -show_entries stream=bit_rate,channels input.ogg -o input-data.txt under the hood, and we’re using JSPM to link our packages (thanks to Lea Verou’s blog for introducing JSPM to me, it uses importmaps which are nice because it’s not a freaking bundler)

Small one:

Avatar initialization works and doesn’t get misattributed to some random event!

(Avatar init is pushed onto an event queue and kinda gets ran whenever it can, meaning it previously got lumped into something random, which led to some poor POST_WORLD_RENDER bearing the burden of an entire avatar’s initialization scripts…)

To do this we had to defer creating an event until we knew what it was, but also support going back and retroactively deciding we had no idea what it was.

Finished off the branding work! I have a 16:9 banner and a 1:1 icon, both with absurdly large resolutions (8K for the banner, 4320x4320 for the icon)

• full resolution banner
• half resolution banner
• full resolution square
• half resolution square
• square 512x
• square 256x

(I also have some images of just the wireframe scene and just the full scene, but they don’t have attribution in them so I’m not going to upload them)

more branding! ft. lapse

i’m using these avatars (you need to be in the FiguraMC discord to use these):

• https://discord.com/channels/1129805506354085959/1253683727750398043
• https://discord.com/channels/1129805506354085959/1318699746625589341
• https://discord.com/channels/1129805506354085959/1251519317019988109

some time was taken trying to figure out why HIPRT wasn’t working in blender, but that’s a problem for later because it seems like I’ll need to compile blender from sources to fix it

Code

Added a /fprofile command to control recordings from in-game and filter them to a specific player. There are still some bugs with the filtering where events aren’t being parented correctly to work out, however.

also, I made a instruction decoder in SQLite. this will be useful somehow, I will make sure of it! (please be faster than the relevant data transfer speeds 🙏)

Lapse

I worked on some branding (trying to make the project banner happen!)
Check it out below I guess

okay maybe we shouldn’t have gotten too excited because all the parenting was wrong.
that should be fixed now though? all it took was even more views and more trigger fanagling yep

I learned how to make SQLite error if a condition is met though, that was useful for debugging because otherwise I can’t really get any debugging info out of my triggers, I just have to hope they work lol
it’s something like this (FAIL makes it so that things don’t get rolled back):

SELECT CASE
           WHEN (SOME CONDITION)
               THEN RAISE(FAIL, 'AAAA debug data: ' || WHATEVER)
           END;

(attachment 1: Select id, parent, type, friendlyName, realtime from tree where parent = 1172;. no, there should not be 500+ frames nested inside this one frame.)
(attachment 2: after I fixed it; how it’s supposed to look when doing that CTE from earlier)

really quick update to note that it worked, letsgooooo

look at these numbers! on average, it’s 46% faster than the old trigger method and 57% faster than two INSERTs!
and at the 99.5%ile it blows past the original trigger, being 70% faster, and it’s even faster than the two-inserts method as well!

…now we just need to check that everything got parented… yep! looks like we’re keeping this one!

new data is in the same place (this spreadsheet).

i absolutely locked in on SQL and cooked up this monstrosity: a SQL-backed stack to avoid doing data transfer as much as possible. we will see if it works in the next devlog

CREATE TABLE IF NOT EXISTS temp.stack (
    ordering INTEGER NOT NULL PRIMARY KEY,
    id       INTEGER NOT NULL,
    type     TEXT    NOT NULL
) STRICT;

you can push new items to it like this:

INSERT INTO stack (ordering, id, type)
VALUES (
    coalesce(
        (SELECT ordering - 1 FROM stack ORDER BY ordering LIMIT 1),
        0
    ), last_insert_rowid(), 'frame'
);

peek the top item:

SELECT id FROM stack ORDER BY ordering LIMIT 1

and pop it:

DELETE FROM stack ORDER BY ordering LIMIT 1
-- in our case we want to unwind in case some internal state gets messed up
DELETE FROM stack
WHERE ordering <= (
    SELECT ordering FROM stack WHERE id = old.id LIMIT 1
);

for the attachment i’ve run this INSERT repeatedly to show how the stack table and triggers automate a bunch of bookkeeping.

INSERT INTO fast_events (timestamp, friendlyName, owner, category)
VALUES (1,
        'Test event',
        '497dcba3-ecbf-4587-a2dd-5eb0665e6880',
        'undefined');

the worst part about rewriting your architecture is actually implementing your architecture.

did you know it takes (relatively) AGES to do an INSERT ... RETURNING + INSERT on the application layer, compared to the equivalent TRIGGER instructions_insert ON instructions INSTEAD OF INSERT which uses last_insert_rowid? yeah me neither but it’s bad as shown by this profiler output (profiling a profiler. hmmmm)
full resolution image

that’s all for now, i’ll go back to trying to figure out how to use a trigger for everything. it looks like sqlite-jdbc is doing a bunch of stuff i don’t want it to do during initial setup smh

edit: want some data? here you go: raw performance data in nanoseconds
warning: RAW DATA sheets have a ton of rows that your browser probably won’t like (as in 450,000ish!)

who likes redesigns? we like redesigns! (a bit too much, probably)

we gave up on the old schema because of two big issues:

• we want to make beautiful waterfall charts, but the relation between instruction steps and the calls that they’re a part of are very unclear
• calls should be part of a tree, not flat? like, this seems obvious but we somehow missed it the first time

so… pure tree structure! (also I added inheritance to the schema so that each individual node is slightly smaller)
I also added some views that do the filter and LEFT JOIN automatically, as well as some INSTEAD OF INSERT triggers to make inserting into views work the way it’s supposed to.

this does mean doing a bunch of work keeping track of relations.

also, have a recursive cte to get the root node!

WITH RECURSIVE parents(id) AS (SELECT ?
                               UNION ALL
                               SELECT tree.parent
                               FROM tree
                                    JOIN parents ON tree.id = parents.id)
SELECT tree.*
FROM tree
     JOIN parents ON tree.id = parents.id
WHERE tree.parent IS NULL;

click to view full-resolution diagram!

Sources are real! Also, I can now throw queries like this at the database…

SELECT z.id,
       e.label               as event,
       sum(z.insn)           as totalInsn,
       sum(z.realtime) / 1e9 as totalTime,
       s.name                as source,
       z.lineNumber          as line,
       z.pc                  as pc,
       c.bytecode,
       e.owner               as owner,
       s.content
FROM steps z
         LEFT JOIN events e on z.event = e.id
         LEFT JOIN main.closures c on z.closure = c.id
         LEFT JOIN main.sources s on z.source = s.uuid
WHERE e.parent = 22876
   OR e.parent = 22812
GROUP BY z.event, z.source;

(instruction & time usage per source & event, plus some more data)

there are still performance issues, however. I need to compare against a baseline

We now have some actual event hierarchies! We did lose the source attribution while we’re preparing to staple sources, though.

Completed

• Indexes are now taking up 70% (!!!) of the database size, somehow!
• Also, implemented autocommitting. This is much more space-efficient than keeping everything in the WAL, wow
• Events are real!
• Steps are associated with events!
• Actually figured out what I’m doing wrt prepared statements! (turns out it just needed another abstraction layer)
• Found a quirk in Figura that means that animation keyframes run before everything else in the game. Hmmm

To fix

• timestamps on root events are very broken
• timestamps in general make very little sense
• drop owner column from steps because it can be derived by crossreferencing with events
• figure out what indexes we actually need and drop the rest (again, 70% of disk space!?!?)

To add

• Sources!
• Filter recordings. We really don’t want to be recording everyone on plaza and blowing through our users’ disk space.
• Capture pings
• More stuff scattered throughout the codebase in //TODO comments

We have a minimum viable product! (very minimum and very not viable)

It just dumps all of the instructions of all loaded avatars with no distinguishing between them. Also, it eats disk space for breakfast and doesn’t commit ever, so the -wal file just grows until you quit the game.

This gave us the opportunity to see the index creation performance. It seems to roughly double(!) the database size (!!) and takes about 5 seconds for ~2 min of recording on Plaza.

aside: do database screenshots count as “not code”? because i don’t have like, UI work to show for any of this…

i guess you get to have a visualization as a treat

select source, count(*) as total from steps group by source order by total desc;

oh we’re so back

Not Databases

• fixed a bug where the game wouldn’t launch in dev environments because of a transitive dependency usurping the direct dependency
• added ‘caused by’ support for translatable exceptions

Databases

• made DbAction for individual processes
• sketched out schema again
• we now run the schema directly from a .sql file instead of having to type it into a .kt file
• indexes are created in an ‘optimize’ action instead of when creating the tables because indexes are slow
• databases can be opened read-only
• foreign keys start disabled by default and are only enabled & checked upon recording completion

Made a LineEdit wrapper that supports custom padding around the actual LineEdit.

features:

• inspector support for managing all the styles on the root element
  • uses the theming & theme variation system!
• live preview
• focus & readonly support

not-yet-features:

• click on panel to focus
• other LineEdit properties

I’m making a launcher (like a thing that launches apps!)

It doesn’t have a name yet.

For now, I’ve decided to try using Godot 4 to do it because it

• comes with built-in support for transparent windows
• also supports wayland, kind of
• has a pretty good UI toolkit too

special thanks to popcar for these blog posts:

• https://popcar.bearblog.dev/how-to-minify-godots-build-size/
• https://popcar.bearblog.dev/using-godot-for-gui-app-development/

note: cyan box will not be in final product. it will either be black or a shader. i also really want this to be customizable

did you know there’s a 2000 character limit on devlogs? I sure didn’t until after I had written a really big one with code examples and then it rejected me.

tldr: rewrote database wrapper. added SQLite WAL support. tried to fix concurrency. found out the hard way that you really need retries on SQL operations.

the whole thing, for all the storytelling score:
https://gist.github.com/penguinencounter/1ec89dc025b8b400c61517d37467728b

also, the reason why the attachment is weird: https://github.com/hackclub/flavortown/issues/1086
(I have to upload a new copy of that image every time I edit the text :P)

Did some database design:

• profile_meta table; single row, contains schema versioning information

• sources: source attachments

• channels: groups that users may want to track, like TICK, or RENDER. could be continuous or not

• events: collections of steps, like a single tick or frame. instruction count and timing are totaled under a single event to measure overall performance / permission level impact

  • can have a parent event (ex. ITEM_RENDER has RENDER as a parent)

• steps: individual actions within a script. instructions, function calls, detailed timings

• Added method to automatically set up a blank database with the current schema version.

• Added method to apply database migrations in the future.

• Added method to copy a on-disk database into a temporary database

  • this is done with a RESTORE FROM command, which is not real SQLite and gets intercepted by the JDBC connector. As a result, you can’t do this with a prepared statement, so manual sanitization goes brrrr

newConn.createStatement().use {
    val syntax = when {
        url.contains("\"") && url.contains("\'") -> throw SQLException("Cannot unmount database: no safe quote type for $url")
        url.contains("\'") -> "RESTORE FROM \"$url\""
        else -> "RESTORE FROM '$url'"
    }
    @Suppress("SqlSourceToSinkFlow")
    it.executeUpdate(syntax)
}

Project Setup

Created a new project based on https://github.com/FallingColors/hexdummy.
This is a Hex Casting addon template, but it has the best Architectury+Kotlin build system I’ve seen so far, so we’re going to use it for something completely unrelated.

• tore out all the Hex Casting stuff
  • kept PKPCPBP for the JSON5 preprocessor
• added Figura dependencies
  • figura itself - figura-common & figura-fabric & figura-forge
  • its transitive dependencies, figura-luaj-core, figura-luaj-jse, nvwebsocketclient, oggus, concentus, and fabricPermissionsApi
• updated MixinExtras
• upgraded Loom to 1.13
  • build JDK must be at least 21
  • upgraded kotlin to 2.2.21
• added SQLite JDBC library (plan is to make profiler files SQLite databases)

Actual Mod Code

• added hooks for the start and end of function calls, as well as every instruction executed in Lua bytecode
• found out that LuaJ’s LuaClosure::execute method has bad control flow and causes @Expression from MixinExtras to get stuck in an infinite loop for some reason??

More matching libraries! In this case it’s DotNetNuke.Log4net. Having a quick way to see all the logging calls is nice

Found a matching library!

HidLibrary appears to be statically linked into the executable.

ILSpy decompilation accuracy is much better than dotPeek’s, so I switched all the sources to that. As a result, I can actually build the application from the decompiled sources! (It doesn’t run ofc but it’s not incorrect at least)

That does mean I lost some of the renames. Though, I am keeping track of the original names of everything now, so in case of some decompiler issues it doesn’t become a pain to track things down.

Mapped a bunch of Windows API bindings and names. Also, figured out that C# async is completely fake and is just 2 delegates and 3 methods in a trenchcoat.

it works! kind of! (well, it doesn’t crash?)

needs: UI, actual merged output, conflict resolver

more untested code! to be tested tomorrow

(it’s like an event loop or something. i am coming to the realization that i will somehow need to write developer documentation for all of this)

in other news I turned full line code completion back on, we’ll see if it actually saves any time

Is it binary? The answer may surprise you.

we have two different kinds of UTF-16 and you have to guess which one is correct

also JS labels are real and you Can use them

built out the planner system a bit. should be able to actually pick the right before transformer by next devlog

also we’re just pretending everything we don’t recognize is text for now. this is a bad idea

JavaScript is a good and well-designed language /s
(maps to the rescue)
(this is a bug fix devlog)

I made a Meyers diff implementation!

Special thanks to James Coglan for this set of 3 articles which makes the algorithm make sense (or, more sense than I got from reading the original paper) and also providing a Ruby implementation that I adapted to TypeScript:
https://blog.jcoglan.com/2017/02/12/the-myers-diff-algorithm-part-1/
https://blog.jcoglan.com/2017/02/15/the-myers-diff-algorithm-part-2/
https://blog.jcoglan.com/2017/02/17/the-myers-diff-algorithm-part-3/

I wish I could get hackatime time for the hour spent reading math papers to figure out how the hell all these algorithms work. oh well

Next: probably a basic 3-way merge, then a MVP

(Also, Rhythm Doctor OST is carrying me through this)

a bit of polish

(yeah it’s 47 minutes of frontend. time to lock in on backend again)

made a quick browser demo for testing purposes (and finished up getting back to before I deleted everything)

next is probably getting text merges to work

I discovered that I probably should start with a better foundation for actions.

…so I deleted most of my stuff :( see commit https://github.com/penguinencounter/blenderize/commit/b2b6a0aca3cdfcb352591b14d5c947f3890b00df

Got the project set up and published on GH.

Also, accidentally made a circular dependency when trying to make a Promise wrapper with a progress callback. That was interesting to solve. (Builder to the rescue?)

(There’s not really much to show, because most of this is library stuff - eventually there will be a browser demo, but we’re not there yet)