Scorer Logic Complete

Im so fucking relieved this part took me forever omg but I managed to select and implement all the data per position I want. I decided to take 2 vectors per position, 1 vector per side. I think I will first find a way to train the weights of the positional analysis, then turn my attention to the tactical detection, though ive already got a full pipeline format ready

Concept Detection

Idk why I said I would do tactical detection in the last devlog cuz this time I was obsessed with improving the positional pipeline and I realized that in order to give valid feedback, I need to know what the user was intending to do. Now the way I was planning to do this was to map each move to the concept it shows by using some rules and labels and then compare it to the concept that the engine lines show and then use that in the explanation. I even coded the concept analysis fully before I realized that wait a minute, this is exactly what my position vector scorers were supposed to calculate. I’m doing the same fricking thing but calling it different things. So I was back to the drawing board, when I thought, why not ship this without intent data but make a tool for users to see what they think certain moves do to collect data to train a model to map moves to intent labels for specific ratings! And I can integrate Maia’s neural networks in leela chess 0 to get human like moves at specific ratings to compare the user’s played move with. But this is like a later thing to integrate.

Tactics

ML tactic detection is probably the way to go, but again I need loads of data, so I will implement rule based detection for now.

Smaller edits

I changed code’s readability, by adding static type hinting, adding fixed dataclasses for clusters, evaluations, positions, etc. Future me will thank me.

Positional Analysis

So this was a pretty long session. I got the positional analysis working by using different methods to group engine moves into similar plans and using the plans to compare and evaluate the correctness of the user’s move with what the position required. Took me long enough since I went back and forth between a lot of different ways of evaluating and sometimes even required changes in how I handle the score, which I finally decided to add as another value in the evaluations object. Next, I’m going to look at a way to detect tactical mistakes and ideas. Btw this isnt the actual positional analysis, but a way to compare the position vector part that I skipped over to the engine position vectors, which helps me decide if the user is following engine plans.

Note: I dont have a pic of any output rn so im just going to attach a pic of a method I used to compare the engine position vector with the user position vector

Not much done

This was a bit of a bad session. I couldn’t rly understand how to develop the positional analysis side except like a broad idea, so I think I’m going to work on the other important parts like the frontend and the rest of the backend using dummy data for now so that I can come back and spend some time on this. The pic is the output schema I have decided

Stockfish Worker

So I worked on the stockfish fen analyzer today. Basically what it does its a fastapi server that waits for batch fen analysis and then it divides the fens across multiple workers (depending on the number of cpus on the device). Each worker processes the fens and updates the result on a redis server. The client (which is my main server analysis pipeline) polls for the status of completion, and when its completed, it will return all of the fens. However, I have a Lichess eval db alr, and it caches a lot of the positions. So when an alr analyzed game is requested, my fastapi server just fetches the eval from the db and immediately returns all of the cached evals. Only the evals not in the db are sent to stockfish. After stockfish analyzes the positions, it adds it to the db, but it analyzes at only depth 14 (for speed). So it adds the position to a redis queue, which my raspberry pi 5 looks up continuously, reanalyzes the position at a higher depth, and updates the db. Next time the same position is entered, its alr cached. Now im going to work on the full positional pipeline

Lichess Db caching

I had an idea to download the lichess evaluations db so that I wont need to run stockfish on each position, saving me a lot of time. However the db is pretty big (18gb compressed, maybe even 100+gb uncompressed), but I didnt realize that before writing a custom encoding and converting to sql script that took me 3.5 HOURS TO COMPLETE 😭. I just realized that even if each position is encoded to 256 bytes, sql will take around 90gb cuz there are 343 million positions. Damn. Anyways, I don’t have enough space on my microsd card for that so when I deploy this project on Digital Ocean or smth, I’ll just convert the .zst file to sql there. So now im going to focus on the analysis part and maybe complete the stockfish worker script first before moving on to the positional analysis

Analysis Pipeline

Started working on the actually analysing games part. I decided to split the stockfish and django servers so that they dont compete for cpu cores or threads. Rn iv done the django side of the fetching stockfish evals. Took some time cuz i was pretty new to celery. Next Im going to work on the stockfish side of it.

Lichess url import

Added a small feature to import games by adding a url instead of pulling all games.

Added Chess.com import

yeah so basically what the title says, I used chess.com’s public api to get add games of a user from the past x months given their username, and stored the important info into my own custom db model. Had to use the python-chess library to convert the pgn string to actual moves to store. Now that im done with this im going to do the actual analyzing part

Lichess Import

Added importing games from last 1 year into db by calling the lichess api and streaming the games in ndjson after getting the token. Didn’t import the pgn cuz json gave me the moves and thats enough. Next, either imma implement chess.com import (without oauth) or move on to analyzing and add chess.com oauth import later.

First Devlog

Ik its the first devlog of this project but I got many things done

Setup Django and User Auth

I got the user authentication part done with logging in and accessing everything. I decided to do jwt cuz its pretty secure and allows me to keep a pretty nice split of the frontend and backend, so that I don’t accidently allow the client side parts to access all of the secrets.

Lichess OAuth

This was pretty long cuz I had to read so much fricking documentation to even understand what was going on. Ik the public api is free and doesn’t need a login but oauth gives a nice feel to my project and it allows me to get user games at higher rates, wait, no, no it doesnt, chess.com does that, ok damn, I spent 3 hours for nothing. At least it looks good lmao.

Finally finished! I added a readme and deployed to pythonanywhere! It took some time cuz I had to iron out a final couple of bugs and I decided to render multiple loops of the coil, which I didn’t do before for some reason.

Soo I completed the styling, though I will admit a good chunk of it was from AI (im sry ik but css is too confusing for me smhw). But it looks great! There were some minor issues and bugs, so got them sorted out as well and made some ease of use adjustments, so I think this is ready to ship! Its reached a good stage and I want to see all of ur thoughts on this as well. Im just going to deploy and make a readme now

Ok so I fixed the problem where the field arrows were all weird and another bug where the force arrow construction caused a large red arrow even when there was no force. Now all I have left to do is make it stylish cuz rn the interface looks depressing. I think after that I’ll ship this proj cuz it has gotten to a pretty good stage and publishable.

Massive session. so I completely made the entire THREE.js frontend part of the sim, linked the python api and managed to get the thing working. But, its very buggy so I spent like an hour or 2 fixing it but no luck, so im going to do that next session. Also as you can see no styling yet, so that will also be part of the next session. Hopefully i dont lose my sanity. wish me luck!

So I worked on making the python script from before into an api instead. I just rly hope that it doesnt slow down. I need to make the frontend part of this next

Ok I’ve hit a problem. I was making the 3d simulation in matplotlib, but its extremely laggy with even small grid sizes and segmentation. I can’t rly continue it here, and I think that even in the future, matplotlib based simulation will be harder for everyone to access, so I am going to make this python script a backend and divert all of the rendering to WebGL, probably Three.js. The thing is I have never used WebGL, sooo this is going to be tough. Oh well, nothing I can do. Y’all have any suggestions for frameworks or should I continue with Three.js?

I worked on calculating the forces on each coil, but I ran into a lot of problems. First the axes were swapped in some places and that took me too much time than it should to figure out, but eventually the problem I found out was that the simulation has to be in 3d cuz I keep ignore the y axis in all calculations, which for some reason only causes the force to go haywire. I got it to work if both coils are lined up (pic 1) but if they’re not, it doesn’t work (pic 2). So my main job for the next session is to make it fully 3d, which is gonna look kinda sick too. I totally forgot abt making the adjustable real time, so im going to do that too.

Ok so good progress, I got the whole class based setup of the coils setup, so now I can easily control number of coils, direction of the coil, current, frequency, phase and other stuff like that individually for each coil with minimal hassle. Took me so much time cuz I was trying to figure out how to make the previous extremely laggy version into something faster. Before, it took like 30s for a 2 coil setup to load, but now it hardly takes 5, so not bad if I do say myself. Next, I might explore calculating the forces on the coils, and movability of the coils without changing the parameters manually each time. (ik the image looks a lot like the first devlog but the internal code is fully changed. Check it out on the repo!)

So I decided to switch the previous code into a neater, more controllable version using a coil class, which allows me to fine tune each coil and easily debug as well. I’m not exactly done yet, but I figured out how to rotate the coil and calculate magnetic field by using rotation matrices. Also sped up the calculation by removing a lot of the for loops and replacing them with numpy functions. Next step is to complete the B at a point function to make the field i was getting before again. But for now look at this 3d graph of the coil I was able to make!

Soooo I started my project. First got a single loop simulation of the magnetic fields of the coil done, had to do some research on the biot savart law. Once I was done with that, I had to add multi loop coil support, which was simply just adding all of the vectors for each coordinate up to get a total net vector. But it was kinda boring cuz it was static, so I added an animation where I varied the current in the loops. Lastly I finished with adding multiple coils in one figure. I ran into the error where i thought the 2 coils together weren’t adding up correctly, but just ended up being a scaling bug (i wasted like 30 mins on this 😭) Next step would be to make the coils adjustable in orientation and direction, and figuring out how to make the coils vary in current or size or other specs.