Janosch

I began with computer vision. I used the Google Play Games emulator, launched Block Blast, and started tracking the board, the individual fields, and then the actual pixels. Based on that, I wrote functions that calculate the average pixel values for each field and compare them to the default background. Using a simple binary representation, I was able to visualize which fields already contain a block and which are still empty.

With that working, I moved on to piece recognition. I first determined the correct area for each of the three pieces and then tried placing a 5x5 grid over each one, since the largest possible piece is 1x5. This works for some shapes, but it breaks for pieces like 2x3 or 2x2, because those are centered within their area and therefore no longer align cleanly with the grid. So I tried another approach: removing the background pixels from each piece area, cropping the bounding box to the remaining foreground pixels, and then determining how many pixels are left and where they are located. From that, I can infer the exact piece type, because I know that each single block is 18x18 pixels. For example, if the bounding box is 36x36 pixels, I know it must be a 2x2 piece. After some tuning, this approach works really well.

After that, I added a live console output of the gameplay to verify that everything behaves as expected. As it turns out, it does, at least for the most part. With this in place, I can now start implementing the feature that automatically drags pieces onto the board. Once that works, I can begin putting a neural network on top of it.

That is all for now. I hope that by the next devlog, the automated gameplay will already be working so that I can start focusing on the neural network, because I think that will be one of the hardest challenges of this project.