SAM2-Fuse

I fixed up a couple of resolve-breaking-bugs and published a beta release for ONLY mac. ONLY MAC IS SUPPORTED RN.

I finished up the deleting points and had to refactor ONLY the frontend. I had to change it from a dropdown to a text box due to Davinci Resolve not allowing a method to remove from dropdowns. I also did a couple bug fixes to make the experience smooth sailing. The attached is the new full demo of the program!

This entire process took around 3.5 minutes to FULLY complete. If you have an NVIDIA gpu with a crap ton of CUDA cores, it’ll be 10x faster.

I finally got the add points function working and made it send it off to the server for propagation. The video is the current demonstration of the model, it’s sped up by 5x. It’s a lot faster if you have a nvidia gpu tho!

I worked on polishing the propagation system and fixed some stuff server sided. Here’s a video of a finished product!

It’s been a long time since a devlog, but for good reason; Davinci Resolve fusion fuse scripting is a pain because a lot of important features I need is locked, so I needed to find alternative. It’s also hard to debug something that doesn’t even have a debugger, so I force a bunch of log statments that freezes everything. The image shows ONE frame that’s finally been propagated and cached in fusion. There’s still a bunch of bugs I need to fix, but this is a preview of what I did in those 7 hours of work time.

I changed how the model loading worked in Lua. I had to implement a really stupid way to get my requests to work when the server starts to boot up. I finally got model changing to work, which requires an entirely new session to be created. I think I could rewrite how the model switching works, though.

I had to basically completely rewrite how the video propagator/session system worked. Originally, it was that it took in video frames via base64, converted to image, added to a directory, then video predicted through. Now, I have all the frames uploaded at front and have the video propagator iterate through the entire thing. I also had to make something that cleared/cleaned up a session if the user wasn’t using it for a while.
(Image is the output of my test api script testing the session manager api. 2 corresponds to testing the video propagator and the 3 is to test cleanup. Cleanup for 1 SESSION went from 9.12 GB RAM -> ~300 MB)

After a couple of hours, I refined the server and how it works. I ended up having to implement a heartbeat system so it could actually shut down while in the background. In addition, I had to handle a cumbersome problem that DaVinci Resolve does NOT provide ANY lua plugins (for http handling, etc) so I had to rely on using the terminal curl to make my requests to my backend. I also coded some of the frontend (without functionality) and got the config loader working. (With a drag and drop library. I was not gonna parse that all myself). Unfinished UI Design down below.

I finally got the API working for the propagation.

Finally made it so the session can make a video propagator. I need to revise the way I do it tho, I need to export a frame and save it to disk rather than using a numpy array and inserting into that.

I finished up my setup.py to install the necessary dependencies and also created a test.py script to test the video masker. I put a point at the grass and it masked it properly!

(The very green thing outlining the grass is the mask generated by the model)

Somehow Hackatime logged 4 hours but it did not feel that long. I made a “setup.py” script to download the proper dependencies (pytorch, torchvision) onto the users computer. I had to handle a case where certain pytorch builds had to be installed based on the cuda version installed on the user’s computer. There’s some problems I ran into: the default “venv” module that python includes does not (for some reason) install pip. This means I have to curl and download a get-pip.py script that installs pip so I could install all the other dependnecies. All stuff is installed with either pip or curl.