Seer

Ok, so there are a few things that I tried doing. Even after making the switch to the grounding DINO model, which perfectly predicted the bboxes of each object, the 3D projections still look like ahhh.

I know the issue is somewhere in the raycasting code, but I can’t lie ts frying me already. I tried making a few changes, but literally nothing worked. So now, I’m gonna switch to manual annotation. It’ll be more tedious than before, but at least it’ll work (hopefully).

Changelog

• FIX: Attempted a raycasting fix to have accurate bounding boxes and projections. Did NOT work. Will try switching to manual annotation
• FIX: Use ONLY the reconstruction with the most images (the best reconstruction) instead of using all of the available ones, since that leads to incorrect camera poses and incorrect environments

After looking at the projections of the Gemma-made bboxes in the environment, I was not having a good time. So, I decided to switch to a second model for bboxes while keeping object detection on Gemma.

I used IDEA-Research’s Grounding DINO model, which, given an object name, estimates a bounding box around it. I took the object names from Gemma’s output and put them into the classifier, creating this beautiful JSON you see below. Haven’t yet run the raycasting algorithm on them yet.

Changelog

• Use the actual object name (no underscore)
• FIX: Unpack box not in the for loop but as a variable to fix ‘cannot unpack non-iterable float object’ error
• FEAT: Add function to normalize the bboxes so that they can be easily run through the raycasting pipeline
• misc
• DOCS: Improve documentation for the Grounding DINO model loader with beautiful type annotations
• REFACTOR: Use folder name instead of file name for images to be consistent
• FEAT: Implement a Grounding DINO model for bbox detection, since Gemini SUCKED. Also, use colorama for better debug messages.
• DEBUG: Add a script to view bboxes in 2D to check Gemma’s outputs
• FIX: Use all the output/i/ folders from previous runs that actually succeeded, and save aggregations to JSON file.
• FIX: Use all the output/i/ folders from previous runs that actually succeeded, and save aggregations to JSON file.

Once the bounding boxes were gotten from Gemma, I only had to make the raycasting pipeline to project the 2D bounding box onto a 3D plane. The first thing I did was calculate camera poses using COLMAP, then I used those camera poses to project rays onto each image, recording all data in dictionaries.

I actually managed to learn a whole lot about looping through dictionaries and their intrinsic qualities, which was really fun. The only drawback is that the time complexity is not favorable: O(n^2), where n is the number of frames/images.

The only bug to fix is in the COLMAP database, where the initial pair of images is not being calculated. This is probably because there is too small of a difference between the camera poses in each frame. I’m not sure yet if this is true or how to even fix this.

Changelog

• FEAT: Make the aggregation method
• FIX: Adapt the raycast method to usethe new ray dictionary format
• FIX: Use frame name instead of full filepath as frame name
• FIX: Add debug messaging
• REFACTOR: Change rays to be a dictionary that contains data for each frame

Switched fully to Gemma for the frame predictions cause its lowkey WAY more accurate with its text output. Also gave it context in the prompt from past frames so that it predicts things across frames more consistently.

Below is an image of the JSON file it outputs for all 450 images.

Changelog

• CHORE: Remove .vscode
• CHORE: Add subfolders inside ./data but .keep the folder in the repo
• CHORE: Remove all data from repo
• CHORE: Update gitignore
• FEAT: Give Gemma last-frame context so that it can be more consistent with its predictions across frames
• FEAT: Get the Nth frame processing and JSON saving down with Gemma
• FEAT: Get Gemma up and running
• CHORE: Ignore the COLMAP datbase folder
• CHORE: Update requirements
• FEAT: Attempt to use Gemma instead of NVIDIA’s model, since Nemotron often has issues with generating coherent text
• FEAT: Make the camera pose analyzer using COLMAP
• FEAT: Make the processing pipeline for creating JSON every N-th frame

I have changed the prompt to estimate bounding boxes for each object in the 2D image. Now, I have to project that into the 3D environment, and make these bounding boxes actually three-dimensional. To do this I will use raycasting and camera pose to estimate distance to objects and therefore their positions in 3D space.

I’m also thinking of switching to Gemma instead of Nemotron, since Gemma is not just a VLM, but is mutlimodal, so will often have better text/JSON outputs.

Changelog

• CHORE: Update requirements
• FEAT: Attempt to use Gemma instead of NVIDIA’s model, since Nemotron often has issues with generating coherent text
• FEAT: Make the camera pose analyzer using COLMAP
• FEAT: Make the processing pipeline for creating JSON every N-th frame

On the movement and environment side of things I added the fabled zoom mechanic (it was really easy) and some speed control.

The most important new thing is that I am now using NVIDIA’s Nemotron with OpenRouter to analyze images and output the object it most likely is and the materials it’s made of.

I initially used Qwen, but it js wasn’t precicse enough for my use case.

Next up, I’ll have to find a way to create bounding boxes around specific objects in the image so that I know where to start and stop each material. Afterwards, it’s the actually fun part of making the lookup table so that each material can be linked to a specific molecule.

Changelog

• IMPROVEMENT: Refine Nemotron’s prompt for better outputs
• FIX: Use NVIDIA Nemotron instead of Qwen (it’s better) and rename some things to be more descriptive
• FEAT: Use Qwen to identify objects in image with OpenRouter
• DOCS: Update README to have more detail on what the project actually is
• FEAT: Add a speed increase/decrease mechanic using +/- keys
• FEAT: Add the zoom mechanic by controlling FOV
• FEAT: Make the WASD movement system (it’s kinda scuffed rn but we live wid it)

Good news! I was able to load the OBJ into Panda3D and create some basic movement capabilities including WASD and mouse movement. The only things that I have to do before I get to the physics are adding zoom and maybe re-recording the video (since it’s kinda low quality, which messes up how the 3D scan looks)

Changelog

• FEAT: Make the WASD movement system (it’s kinda scuffed rn but we live wid it)
• FIX: Fine tune the rotation on the x axis
• FEAT: Add the mouse movement system so that the user can actually move around
• FEAT: Make the basic Panda3D object viewer
• DOCS: Update the reconstruction viewer to have better comments and docstrings
• FEAT: Complete the 3D reconstruction viewer with aspose 3D and Open3D
• CHORE: Update .gitignore with large files

So I made the pipeline that turns videos into .obj files through the intermediary .usdz (Apple’s proprietary 3D file format).
I had to first take the video from many different angles, then turn it into a folder full of images that were taken every two frames, and then, had to use some SwiftUI code to use Apple’s RealityConverter.
Then, I decided to use aspose 3d to convert to OBJ, but ran into some coloring issues, since the USDZ is a zip archive, so I had to extract the texture images from it and insert them into the OBJ.

The result is actually not too far from what it really is!