live match preview per row + proper toast breakdown (Increement 2)

Before

• no way to know how many cells a pair would affect before pressing run
• toast showed “bob” x3 with no replace val
• table had 3 columns, match feedback was invisible until after the operation

Now

• each row has a fourth “matches” column showing a live count as you type the find value
• count shows green for hits, grey for zero, orange for a bad regex pattern mid-typing
• toast reads “alice” → “bobby” (3 cells) | “bob” → “folk” (2 cells) with full old and new values
• replace value shows (deleted) in the toast when the replace field is blank

input placeholder and readme

Before

• input bar was blank with no hint that it was interactive - like user had to find out where to type the command :”)
  readme did not exist in the repo

Now

• input shows “type your command here….” in dim green until the user types
• readme is live in the repo with setup, commands, and concepts documented

demo video attached

improve Ai prompt for Chat feature and readme

The Ai prompt needs some improvement like it was giving away all the results easily. So i improved the prompt and also the readme!!

devlog 1 Setup of boot screen,some commands and input taking

• full retro crt terminal with vt323 font, green glow, scanline overlay, and vignette corners
• boot sequence fires on load using getRandomLetter and getRandomNumber for dump lines and an ascii logo
• handleCommands validates input against the VALID_COMMANDS array and routes to the right function
• help, clear, exit, echo, status, and decrypt commands all working
• bad commands trigger a screen shake animation and a red error line
• web audio api provides a low-frequency hum and a mechanical click on every keypress
• terminal auto-scrolls to bottom on every new line added

HUD, readme button, and concept dialog (see demo video)

• HUD panel appears in the bottom-left corner with two live bars, speed and heat( the thing that is driving this whole concept)
• a small circular button sits in the top-right corner and opens the readme dialog on click
• the dialog ? is on top right corner which explains this project
• the dialog closes via the X button or pressing Esc
• dialog explains six concept sections explain Verlet integration, constraints, multi-pass glow, heat color, and motion trails in Super Simple language(i hope so you get it)

edit: my commits got all messed up, i pushed and only main got pushed cuz ( i just reused my prev terminal commands for commiting again and that command was only pushing one file so it all got messed up :”) then i wanted to remove my commit and it ended up removing the files from my laptop also :”) thank got it worked on ctrl+z otherwise i would have been cooked esp 1 day before ft)

glow, heat color, and motion trails

• the chain now draws in three passes per frame instead of one single stroke and together the three passes fake a neon tube glow
• mouse speed is captured on every move event
• raw speed is set till max of 60
• heat is smoothed across frames so color fades in and out instead of snapping( kinda cool )
• speed decays by 0.85 each frame so the chain ends( end is not a good word here since it dont just disappear but i don’t know what else to say) when the mouse stops

multi-pair find and replace table editor, increement 1

Before

• find and replace only handled one pair at a time, requiring multiple manual runs to standardise a messy column
• no way to queue up “N/A → unknown”, “none → unknown”, “- → unknown” and apply them in one shot
• single find and replace state was two loose text input keys, no concept of a list

Now

• find and replace section renders a dynamic table where each row is one find and replace pair
• rows can be added with “+ add row” and removed individually with the x button
• “run all” applies every pair in order and reports per-pair change counts in the toast
• one global regex toggle covers the entire batch, same pattern as the old single-pair UI
• after a successful run the table resets to one blank row automatically

API key moved to header, enum validation on pipeline JSON imports

Before

• both ai_insights.py and nl_cleaner.py appended the Gemini API key to the URL askey=... (cuz of Key Leak Security Reasons)
• any exception traceback, proxy access log, or browser network tab captured the full key in plaintext
• method, action, output_fmt, and input_fmt were read from an uploaded pipeline JSON and passed directly to cleaning functions with no allowlist check
• a crafted JSON with "method": "../../etc" or "action": "execute" reached cap_outliers and validate_range as-is
  0 threshold from pipeline JSON was parsed with float() but never range-checked, so “threshold”: “1e308” caused a numpy overflow
• the same unvalidated values also reached build_pipeline_script & were embedded in the exported Python script

Now

• both files pass the API key in the x-goog-api-key request header and remove it from the URL entirely
• _safe_enum in pipeline.py allowlists method, action, and email action fields against explicit sets before any cleaning function is called
• _safe_date_fmt validates output format against the four UI dropdown options and caps input format to 40 chars
• _safe_threshold clamps the parsed float to 0.1–100.0, matching the UI slider bounds
• all three helpers are also applied in build_pipeline_script so the exported Python script reflects the same validated values
  Note: Checked this using testing_api_pipeline py file that scans my Backend files and Tell if the testing passes or not! you can see it in attached SS

physics foundation, chain follows mouse

• a fullscreen dark canvas since i don’t wanna design any UI :)
• the head node is pinned to the mouse position every frame
• each tail node moves using Verlet integration, implying velocity from position history
• gravity of 0.18px per frame pulls the tail down naturally
• damping of 0.92 bleeds off energy so the chain settles instead of oscillating forever( FANCY WORDS IK)
• the chain draws as a single continuous white line with rounded caps
• a small white dot marks the cursor position since the system cursor is hidden
• touch events are supported so it works on mobile too

move groq api calls behind a vercel serverless function

(used ai to figure this out, as i didn’t know i had to make a serverless function to set the key in vercel)

Before

• groq api key is assigned to window.ENV_GROQ_KEY in config.js and wasn’t accessible in the deployment link
• both the tipster chat and final scene verify fetch directly to api.groq.com from the frontend

Now

• api key lives only in the vercel dashboard as GROQ_API_KEY
• all groq calls go through /api/chat.js which runs server-side on vercel
• frontend fetches /api/chat and gets back only the text it needs

keyboard login, implemented groq key, tipster fix, readme

Before

• login required clicking on-screen buttons, no keyboard support
• gemini key was used
• tipster was still giving away information directly instead of asking questions
• no readme :(

Now

• typing digits, backspace, and enter on the keyboard works on the login screen
• a small hint below the OS name tells players to check the console or devlog for the password
• groq api key is used since it allows more rps
• both tipster and final scene use the shared helper so both benefit from fallback
• tipster system prompt now enforces Socratic mode like every response must end with a question, player must reach every
• finally write readme

demo video link(contains spoilers, also chat feature still sucks somehow,so try to find the answer without using it)
https://drive.google.com/file/d/1oXJNCq0buNGMcFJXMV2Xquk60hl_5HqE/view?usp=sharing
demo was long so i couldnt attach it

window resizing, final scene font fixes, submit button color

Before

• windows could only be dragged, no way to resize them
• all final scene text was tiny pixel sizes (5px to 11px), basically unreadable at normal screen distance
• ending screens had illegibly small text for the reveal lines

Now

• all windows are resizable by dragging any edge or corner, with a minimum size of 320x240
• a subtle resize grip indicator shows in the bottom right corner of every window
• cursor changes to the appropriate resize direction arrow when hovering near window edges
• final scene title bumped from 11px to 18px, prompt from 8px to 13px, textarea input from 7px to 12px
• choice button text and ending reveal lines also got increases( can’t right its increement also :”) )
  This still didn’t fulfill the Font issue so Next i will make the New window which have change font size feature in it, so user can change size according to their preference

final scene, tipster polish, cursor and font fixes, BG images fixes

Note

read this devlog after you have played the game since it contains all the SPOILERS!

Before

• no final scene existed, no way to submit a theory or reach an ending
• tipster chat header used a plain gradient, no tab_bg image like other windows
• chat bubble font was 6px, barely readable especially against the background image
• start menu was missing Submit Theory as an app option on the desktop (as i need to make it)

Now

• double-clicking Submit Theory opens a fullscreen theory input powered by gemini verification
• correct theory unlocks three ending choices with animated reveal screens
• do your job sends sarah back and shows a case closed stamp with delayed text fade-in
• hide the truth fades in three lines slowly and ends with “some cases are better left unsolved”
• call the police triggers a phone ring animation then injects a threatening email from unknown sender
• wrong theory shows “you still haven’t found the truth. go away.” and lets player try again
• tipster header now uses tab_bg.jpg with luminosity blend matching all other window titlebars
• chat bubble font bumped from 6px to 8px with line-height 2.2 so messages are readable
  note:
  I think text is still not readable enough so in next devlog i will work on making it readable

anonymous tipster - full feature devlog

Before

• no AI assistance in the game at all
• a help me button existed with hardcoded hint strings that was basically useless

Now

• a glowing pink icon sits bottom-right above the taskbar, pulsing to draw attention
• clicking it opens a chat window
• an anonymous tipster persona powered by gemini 3.1 flash lite preview responds to the player
• every message rebuilds the system prompt injecting what files and emails the player has opened this session
• tipster never volunteers information, always asks what the player thinks first
• if the player gives a correct interpretation it confirms and nudges forward
• if the player gives up it switches to direct mode with a concrete folder or file instruction
• gemini key lives in config.js which is in .gitignore and never touches the repo
• last 3 exchanges sent to gemini so context stays!

what i had to suffer for this(aka why it took so long) :”)

• tipster had no awareness of what the player had actually opened before so spend time figuring out what to do here, so i send last 3 messages to gemini also as context and also made levels that gemini will know of what level/files user has explored till now
• gemini volunteered the full truth unprompted without waiting for the player so i had to restrict my prompt way toooo much

hi!!
In this devlog, i restructured all the code, remove the mails or photos that was too obvious. I also brainstorm the whole flow of the Game
like what should go into MAILs, PHOTOS, where should be final stuff and how can user connect the dots
Also, i made sure everything points to some clue!!! before it was too messy but now i made it look clean and delete all the unnecessary stuff! All the stuff in every folder is finalized.
And i added a help me button that guides user, but i think its not so good, so I will replace it with something actually helping and not so obvious in the next ship!
i think now i will only make a new big push of code and get it done as most of the flow is already done in the code!

fix delimiter ReDoS check and replace import with top-level pandas import in recommendations

Before

Fix Redos

• _validate_delimiter tried to catch dangerous patterns by calling re.compile on regex delimiters before running the split
  re.compile("(a+)+$") succeeds but it only blows up at match time on bad input, so the check did nothing

replace import

• _apply_fix called __import__("pandas") four separate times inside the function body
• each call goes through Python’s import machinery on every button click even though pandas is already loaded
• the code was unreadable __import__("pandas").to_numeric(...) buried the actual operation

Now

fix ReDos

• regex metacharacters in delimiters are blocked entirely instead of compiled and trusted
• every accepted delimiter is passed through re.escape so the re engine always treats it as a fixed string

fix impor

• all four __import__("pandas") calls replaced with plain pd.to_numeric(...)
• the percentage conversion chain was also un-nested into a named cleaned variable so it reads as two clear steps instead of one wrapped call
• behavior is identical, zero logic changes

input sanitization and delimiter injection guard across split, merge, and rename

Before

• column rename accepted any string including null bytes, newlines, carriage returns, and pipe characters
• a null byte in a column name corrupted CSV exports without any error
• a pipe character in a name broke _parse_label_meta in pipeline.py bcz pipe is the label delimiter
• a newline in a name broke pipeline JSON label parsing and the history display
• merge and split column had the same exposure for their output column name inputs
• the split delimiter was passed raw to str.split which pandas routes through the re engine
• a pattern like (a+)+$ typed into the delimiter box would cause ReDoS, hanging the server( this is still unsolved though)
• a plain literal like . was treated as a regex wildcard, splitting on every character instead of on a literal dot
• the Apply/Split buttons stayed active even when a pending value was already invalid

Now

• _sanitize_col_name in transforms.py rejects null bytes, newlines, carriage returns, pipes, empty strings, and names over 200 characters
• rename, merge, and split all call _sanitize_col_name before touching the dataframe
• _validate_delimiter checks length, detects whether the input is literal or regex, and calls re.compile on regex delimiters before any data is touched
• plain literal delimiters are passed through re.escape so a . always means a dot, never a wildcard( but this is still not enough so i have to work on it more in next ship)
• the rename UI validates every pending name live and shows a specific error per bad name
• the split UI validates the delimiter live on every keystroke and shows the error inline

Note:

ReDoS issue is still not fully solved because even with validation, a user could still input a complex but valid regex that causes it. I will work on implementing this in later ships, I already discovered another issue that i am so eager to resolve so I will work on that first :”)

Hi!!
So before, it was just a PIXELOS
but then i thought, why not i make it a Crime Investigation Game like now
YOU are A detective and you got this PIXELOS of girl Named Sarah and her parents assign the job to you to find out about their missing girl! Now you are searching through her OS finding all the clues!! WOuld u find out the real kidnapper?!

Ok smthn like this, i wanna go on more but i think i shouldnt as this devlog should contain technical stuff also
so i made a new MAIL window also which shows the mail of the Girl Sarah and then also a case Notes window which shows the basic case notes like initial details!!! i have figured out all the flow of the game tho
so stay tuned!!!!!!!!
(i know i took a longg longg break for this project but…)

hardening across session storage, cleaning, PDF export

Before

• session files were loaded with pickle.load and no integrity check, anyone who could write to the sessions folder could execute code
• MD5 was used for session key generation despite being cryptographically broken
• find and replace passed user regex directly to pandas with no length or compile check
• drop duplicate rows left a gapped index that could break downstream operations
• AI summary and history labels were injected raw into ReportLab paragraphs where XML-like tags get parsed as markup

Now

• session files are signed with HMAC-SHA256 on save and the tag is verified before unpickling
• SHA-256 replaces MD5 everywhere in session key generation
• find and replace validates regex patterns with the same safe regex guard used in validators
• drop duplicate rows calls reset index so downstream code always sees a clean 0-based index
• AI summary filename and step labels are escaped before reaching ReportLab

sandbox hardening for AI cleaner

Before

• sandbox exposed full pd module, allowing pd.read_csv('/etc/passwd') or df.to_csv(...) to access the filesystem
• print was in safe builtins, allowing the entire dataframe to be written to server logs
• _build_prompt sent raw cell values including salary, email, and phone columns to the external Gemini API
  no length limit on user instructions meant a 100 KB string could be used for prompt injection
• no length limit on the code textarea meant AST parsing a massive string could hang the server

Now

• a _BLOCKED_PD_METHODS set blocks all pandas read and write IO methods at both string and AST level
  print removed from safe builtins so sandboxed code cannot leak data to logs
• _build_prompt now uses the same PII column detection logic as ai_insights.py to redact sensitive columns before sending to Gemini
• instructions over 2000 characters are rejected before the API call is made
• code over 8000 characters is rejected before _static_check and exec are called

Security Fixes- PII Safety, Regex Guard, File Limits

Before

• df.head(8).to_string() sent full row data to the Gemini API

• sensitive info like emails, salaries, phone numbers, SSNs leaked externally

• no truncation or filtering, every column fully exposed

• validate_email_col accepted any custom_pattern and compiled it instantly

• patterns like (a+)+$ could hang the process (ReDoS risk)

• protection existed only in pipeline replay, not in direct UI usage

• no file size limit, large files (GBs) could stall the server

• extension parsing used split(".")[-1], trusting every dot

• files like real.csv.xlsx bypassed intended validation logic

Now

• AI prompts send only column name, dtype, null rate, and up to 3 sample values

• sample values truncated to 20 characters max

• columns wth keywords (email, phone, salary, password, ssn) send no values at all

• _safe_regex in validators.py is now the single source of truth

• patterns >300 chars or failing compile fall back to default safely

• every entry point (UI + pipeline) routes through _safe_regex, eliminating ReDoS

• file uploads capped at 200MB, rejected early with clear error

• extension parsing uses rsplit(".", 1) for accurate detection

• unsupported types throw explicit errors instead of silently passing

• overall shift from unsafe defaults → safe-by-design system across AI, validation, and file handling

Note:

I couldn’t found xlsx with 200 mbs :)) i downloaded one with million rows hoping it would be 200 but that one is still 32 mbs :)

sanitize pipeline script export and harden JSON import

Before

• col names from history were f-stringed raw into the exported python script

• a column named with quote characters could produce runnable malicious code in pipeline.py

• regex patterns loaded from a pipeline JSON file were passed to validators without any check

• a crafted ReDoS pattern in a saved JSON could hang the server during pipeline replay

• the JSON importer accepted files of any size with no upper bound

Now

• all column names pass through _safe_col before being embedded in script strings

• _safe_regex validates and compiles every pattern from JSON before it reaches any validator

• patterns longer than 300 characters or that fail re.compile fall back to the safe default

• pipeline JSON files larger than 512 KB are rejected before any parsing begins

• step lists with more than 500 entries are also rejected with a clear error message

first of all, sorry kinda long devlog
because i have to submit it in LOCKIN sidequest and i had very less hours to do so! thats why i didnt bother on devlogging and just stick to my feature and errors correction!

feature overview: ai study material generation and error handling

• resolved tons of errors in app( more detail below )

• the app uses background OCR to extract every word from the images, before it was extracting only 8 words to give to AI but now it extracts all word so AI SUMMARY works well

• an AI classifier reads that text and automatically figures out which class subject the photo belongs to

• this creates an automated, interactive study guide without the student having to write a prompt and to upload all pics to the AI to summarize it!(smart before exams tho cuz u can just quickly remember the stuff)

Tech details

before

• background byte-reading threads _cachePhotoBytes kept running even after users closed the screen, causing massive mem leaks

• network timeouts during AI classification caused setState() lifecycle crashes if the user minimized the app

• rapid batch processing via Future.wait caused hidden I/O file overwrites because photos processed in the exact same ms

• ui fallback messages like “scan failed” or “no internet” were accidentally being saved into the database, poisoning the search index with fake data :))

now

• injected strict if !mounted return; guard before every single UI update to completely eliminate background crashes

• appended _${path.hashCode} to temp filenames to guarantee no concurrent file overwriting during fast batch processing

• wrapped the local storage and database indexing steps in a tight try/catch block so individual photo failures never break the whole batch

• the catch block now parses error strings to detect constraint or exists duplicate exceptions, treating them as a success

• the UI state now safely shows only successful uploads
  (and with that word limit reached!)

search feature complete, index, quality, and navigation

Before

• ocr text saved to the search index was capped at 8 display keywords per photo

• a digital screenshot with 200 words was only searchable by 8 of them

• multi-word queries like “artificial intelligence” only matched if the exact phrase appeared consecutively in the ocr text

• was no search screen

Now

• a separate ocrFull field stores cleanedText plus the entire keyword list so every word on a digital screenshot is indexed
• _confirmAndSave reads ocrFull into SearchService.indexPhoto instead of the 8-keyword display string
• search() splits the query into individual words and requires every word to appear somewhere in the text in any order
• search is the second tab in the bottom navbar

PS i’m never making mobile app again :”) just testing it takes 15 to 20 min, first you have to transfer it on your phone then run it and all :”( cant run it on emulator cuz it has to take photos from android !

full-text search across all notes - backend and frontend devlog

Before

• ocr text was thrown away after gemini classified each photo
• no index existed anywhere on device to search against
• deletePhoto in storage_service removed the file but left ghost entries nowhere to clean
• clearAll in storage_service wiped prefs but had no knowledge of the search index
• no screen existed to type a query or see results

Now

• SearchService.indexPhoto() saves ocr text + subject under a single ocr_index key in SharedPreferences immediately after every successful save in _confirmAndSave()
• SearchService.search() scans the entire index for a query substring and returns List<SearchResult> sorted alphabetically by subject
• SearchService.removePhoto() is called inside StorageService.deletePhoto() so deleting a photo cleans the index atomically
• SearchService.removeSubject() exists for future subject deletion cleanup
• SearchService.clearIndex() is called inside StorageService.clearAll() so a full app reset leaves no fatherless/orphan(funny word ik) data

sandboxed exec to block code injection in AI cleaner

I wasn’t thinking much about the security, but one kind and cool voter commented on one major security issue of the system so i get to work on it! Now i will work on more security features to make this pipeline secure to use!

Before

• global execution environment allowed full access to the os and subprocess modules

• no import blocking meant __import__('os').system(...) worked

• dunder attributes like __class__ and __builtins__ were reachable from the user interface

• manual edits to the generated code bypassed all internal security constraints

Now

• static ast analysis rejects every import statement and forbidden name branch

• regex pattern scanning catches shell style calls before the parser initializes

• execution globals replace the __builtins__ default with a strict twenty five item whitelist

• validation logic triggers on both the initial generation and the final apply action

AI Assistant tab

Before

• AI Cleaner was inside Clean tab and was easy to miss for users :)
• AI Data Analysis and General Guide were both in Guide tab

Now

• new ai_assistant.py replaces guide.pyand single file owns all three AI and guide sections
• tab renamed to “AI Assistant”
• AI Cleaner moved from Clean tab expander to top of AI Assistant tab
• AI Data Analysis is below it
• General Guide moves to the bottom
• clean.py has the AI Cleaner expander and its import fully removed
• app.py updated with new tab variable, new import
  also tested other features thoroughly so i can ship next

filter and inspect v3 match mode(AND OR) + export

Before

• multiple filters always applied as AND ( And means this and this ,both should be satisified to show this data filtered)
• no way to export filtered results( database peeps will know sometimes its usefull to export stuff after applying sql queries of select where value ==)
• no tooltips explaining what filters do or how match mode works

Now

• ALL (AND) vs ANY (OR) radio toggle added at top with tooltip explaining both modes clearly with examples
• ALL mode narrows down by chaining filters on the result of the previous filter
• ANY mode builds a union mask across all filters applied independently to the original df
• export section added below the dataframe with CSV and Excel download buttons
• export always uses the full filtered result not the 500-row display cap i made for performance
• tooltips added to match mode radio, add filter, clear all, all three metrics, and both export buttons

filter preview v2: multi-filter stacking + UI polish

Before

• only one filter could be active at a time
• no way to narrow down by multiple cols simultaneously
• bw operator was missing so range checks needed two separate filters

Now

• filter rows stored as id list in st.session_state.fp_filters so any number can stack
• each row gets a unique id-prefixed key so streamlit tracks them independently
• x button removes individual rows cleanly without affecting others
• add filter and clear all buttons added below the filter rows
• bw operator added for numeric and datetime with two side-by-side inputs (Min/Max, From/To)
• display capped at 500 rows with caption showing match count for performance

filter preview v1: single column filter

Before

• no way to inspect rows matching a condition without running an operation
• users had to apply a fix and undo just to verify the right rows were targeted
• full dataframe display with no filtering made spotting dirty values tedious

Now

• filter_preview.py added as a read-only module, never writes to current_df
• single filter row with column selector and condition selector
• operators auto-detected per dtype: text gets contains/equals/starts with/ends with/is empty, numeric gets equals/gt/lt/is empty, datetime gets equals/before/after/is empty
• match rate metrics row shows matching rows, total rows, and %age
• filtered dataframe renders below with use_container_width=True
• made it as a separate tab after overview so user can go to it to check the data

col_popover selection reset bug

Before

• The columns user selected before get reset when user switch to other tab thus when he came back to the tab again, he had to deselect and select columns again for operations :)
• on_change callbacks were the only sync mechanism between checkbox keys and val_selected
• after a run, val_selected.pop() cleared the list but _vc_section_col keys stayed True in session state
• streamlit never fired on_change again because widget value had not changed from its perspective

Now

• col_popover reads _vc_ keys directly at render time and rebuilds val_selected from scratch every render
• _make_all_handler now resets all individual _vc_ keys when toggling all on or off so checkboxes visually match
• _make_col_handler syncs the all checkbox when individual columns are toggled

The issues i got while doing this:

• col_popover was unconditionally rebuilding current from vc keys on every render. In streamlit, all tab contents execute on every rerun even when the tab isn’t visible (mess i know) .
• so i had to fix this also , and i used this flow
• If val_selected[section] already exists -> it was set by an on_change handler which is the definitive user action. Trust it, sync vc keys to match it for display, done.
• If val_selected[section] is missing -> means either first ever render, or clear_popover just wiped it. Only then fall back to reading vc keys.

I say this earlier and will say it again i hate how streamlit works, worst framework ever, i might change to some other framework after this ship.

button and popover alignment fix

My buttons were kind of inconsistent for the recommendations and validate tab. My focus before was never UI so i didn’t bother changing them. But now, i think i should change them and make them consistent so here i’m

Before

• Fix button and select columns popover rendered inside the same st.columns block as description txt
• st.write("") spacers before buttons were unreliable and column heights varied per section thus making the UI looks inconsistent
• recommendations and validate both had this problem..

Now

• separate st.columns call handles only the popover and button on their own dedicated row
• both widgets are always the same height so they always land on the same line
• button click logic moved outside col blocks so st.rerun() fires cleanly after all columns close

Excecutable Validation JSON feature

Before

• import_pipeline_json had no branches for any custom validation step
• JSON preview only existed on the save side as a raw code dump
• Uploaded JSON had zero preview, and also there was no toast on successful json application TT

Now

• _parse_label_meta() moved to module level so both import_pipeline_json and build_pipeline_script share it
• Each step checks col in tmp.columns before running and skips if column is missing
• threshold, min, max are safely cast to float with fallback so bad values never crash
• _friendly_label() added in history_export.py
• _render_json_preview() added, shows numbered step list with easy to read labels instead of raw JSON so non tech also get it
• uploaded pipeline auto-expands its preview so user sees steps before hitting Apply

Executable Validation Python code Export

(this was to be done for the custom validators i made in prev devlog)

Before

• Custom Validation steps exported was not implemented yet
• Custom regex, country code, date format, thresholds all were not working
• Type Override had parts.strip() bug on a list :))

Now

• _parse_label_meta() reads key=value pairs embedded in label strings
• all 5 validation steps export real runnable code with exact user settings
• each commit_history call embeds col + settings per column
• bug fixed, stubs added for Split/Merge/Rename :”)
  next i will have to make the custom validators work for pipeline.json also

Global Undo and Redo in the Sidebar

Before

• Undo and Redo were only inside History and Export, and this user has to switch tabs and couldnt access them easily
• render_sidebar() had no access to history or redo_stack
• Sidebar bottom stayed unused in Simple mode

Now

• render_undo_redo() added in state.py and called in both sidebar modes
• Reads from st.session_state, returns early means buttons will show even if no undo and redo has been performed so user know in the start that side bar has this feature :)
• Click triggers toast + st.rerun() for full UI sync
• st.caption() shows next undo/redo actions. so user know what is happening and also user will know what will happen if they press UNDO or REDO!

Custom Inputs for Date, Email, and Phone Validators

Now users can provide specific formats, patterns or country codes to bypass general pattern

Before

• validate_date_col() in validate.py relied on a 14-format loop and C-level inference, which could be slow or misinterpret ambiguous dates (eg 01/02/03).

• validate_email_col() used a hardcoded regex pattern that couldnt be overridden for specific corporate domains or stricter requirements.

• validate_phone_col() was hardcoded to US standards, making it difficult to validate international numbers

Now

• validate_date_col accepts a custom_input_format parameter. If no custom input format is set, then it pass the default US format

• validate_email_col features a custom_pattern parameter. A ternary operator replaces the default regex if a custom one is provided. A new try/except block catches invalid regex syntax

• validate_phone_col now accepts default_country_code. The logic strips leading + signs and uses dynamic boolean masks for country code and length.

UI

• the tooltip for the functions explains the user about what to write in this custom pattern and how so they get how to use this feature

Add AI Executive Summary to PDF Report

Now user will get AI summary in PDF report also
Technical details:

Before

• build_report_pdf() in ‘reporting.py’ accepted only ‘original_df’, ‘cleaned_df’, history, and filename with no Ai summary abt the dataset
• The report opened with all the technical details, and a non-technical person couldn’t make sense of what even the data is abt
• also the AI summary string already existed in st.session_state under ai_insights_{file_id} after the user clicked Generate AI Summary or Analyse my data in the Guide tab,
• ‘history_export.py’ called build_report_pdf() with four positional args and moved on

Now

• build_report_pdf() accepts an optional ai_summary: str = "" parameter
• when ai_summary is non-empty, a light blue tinted LIGHT_BLUE background panel is rendered immediately below the title block and says run ai summary first to get the ai summary in the pdf
• history_export.py reads st.session_state.get(f"ai_insights_{file_id}", {}) before calling the builder, pulls .get(“summary”, “”), and passes it through

Fix: Vectorized Phone and Date Validators

Before

• validate_phone_col and validate_date_col both used .apply() to run a Python function row by row having HUGE waiting time
• validate_date_col ran every row through a 14-format try/except loop even when pd.to_datetime would have parsed it fine in one go!

Now

• validate_phone_col is fully vectorized: .str.replace(r"\D", "", regex=True) strips non-digits across the whole column in one C-level pass, then .str.len() and boolean masks build the +1, + prefixes with pd.Series.where
• validate_date_col uses a two-pass strategy: first pass runs pd.to_datetime(..., infer_datetime_format=True) which handles the vast majority of dates at C speed in one go then second pass only touches the rows that failed and tries the explicit format list….so on clean data the slow path never runs at all

Also change the toast message and now it shows this ✓ along with the message!

Fix: Recommendations Tab State Wipe, Duration Regex False Positives

Before

• fixing any issue in the recommendations tab wiped the entire tab state and dropped the user back to the “Scan for Issues” screen bcz scan_key was f"rec_scanned_{df_key}" and df_key is derived from make_df_key(current_df), which changes every time a fix mutates the dataframe, so the old key simply stopped existing in session state
• the duration regex (h|hr|hour|min|minute|sec|second) had no word boundaries, so the bare h matched the letter h inside any word like John”, “the”, any english text at all would fire it and flag name or description columns as duration columns (found this one while testing on a movies dataset)

Now

• scan_key is now a stable hardcoded string "rec_scanned" with no dependency on df_key, so it survives reruns after fixes; state.py explicitly resets it to False on every new file load and fresh session so it never carries over across files
• duration regex tightened to \b(hrs?|hours?|mins?|minutes?|secs?|seconds?)\b with word boundaries in both get_analysis_and_recommendations and get_type_suggestions in cache.py; bare h removed entirely since no real duration data is ever just the letter h standing alone

also i didn’t know we can use ‘’ to show the code things!!! now i am gonna use them more!

Fix: Session Resume Dialog Crash and Loading Spinner

Before

• clicking outside the resume dialog closed it without setting _resume_choice, leaving init_state in a broken half-initialized state and crashing on the next render

• the loading spinner and info message never actually displayed because init_state ran synchronously on the same render pass, so the spinner appeared and vanished :) (handling edge cases are really tough)

Now

• dismissible=False added to the dialog, disabling backdrop click and removing the X button so the user is forced to pick an option (read this thing in human computer interaction course)

• loading is now two-pass: first rerun sets _resume_loading and renders the spinner, second rerun clears the flag and runs init_state normally, making the spinner genuinely visible between reruns

Fix: Recommendations Feedback and Overview Slider

Before

• clicking Fix or Auto-Fix All in the recommendations tab gave no feedback :)

• the overview tab crashed on single-row files with “Slider min_value must be less than max_value” when both values resolved to 1

Now

• every Fix button in recommendations now wraps its operation in a spinner and fires a toast on success matching the exact same pattern used in the validate tab that i did in last devlogs

• Auto-Fix All also shows a spinner and toasts on completion

• the overview slider now guards against min equal to max by skipping the slider entirely on single-row files and always flooring the minimum at 1

Note:

 Also this took me 45 minute to do, but for some reason my devlog didnt got uploaded and i saw that just now :) so

Performance improvement: AI Insights and Page Load Fix

so first thing happen when user opened my app was to load the ai analysis overview and ai guide, thus user have to wait for 10 seconds for small file max to load all the tabs :)
now all tabs load instantly because now i have button buttons that user has to press to get the ai overview and guide!

Before:

• opening the app with a 50k row file meant waiting 10 to 20 seconds before any tab appeared
• guide.py called get_ai_insights automatically on every render which blocked the entire page
• the overview tab had a separate streaming approach using st.write_stream which returned None in most streamlit versions so the summary never showed at all
• also there were no cache that could be stored for one file so gemini don’t have to be called for it again and again on every reload/reupload of the file

Now:

• both tabs share the exact same cache key ai_insights_file_id so one api call populates both
• neither tab calls the api automatically on page load, the user triggers it with a button when they want it
• guide tab shows an Analyse my data button and overview tab shows a Generate AI Summary button
• whichever button the user clicks first, both tabs show their content on the next rerun with zero extra api calls
• st.write_stream and the entire streaming approach were removed since streamlit tabs buffer all output and flush at once, streaming never worked there
• ai_insights.py went from 310 lines to 190 by removing the dead streaming code
• page now loads instantly regardless of file size, gemini is never called until the user asks for it

UI Feedback - Spinners, Toasts, and the Dialog Bug

so in this devlog, i focused on making the app feel responsive :)

Before:

• when a user clicked any button, the app would run the operation, call st.rerun(), and then show a success banner on the next render
• this was handled using _omsg in session state across clean.py and validate.py
• the success message had no auto-dismiss, so it just stayed there until the next interaction
• sometimes the message didn’t even show, depending on which tab rendered first
• there was no loading feedback, so the app felt frozen during heavy operations
• validate.py had zero spinners at all, meaning everything just paused WITHOUT even notifiying user
• spinner messages (where present) were vague like “Processing…” with no real context
• redo / history-related feedback existed but didn’t clearly communicate what was happening

Now:

• replaced the entire _omsg pattern with a cleaner toast-based system
• new flow: set st.session_state[“_toast”] = (msg, icon) -> call st.rerun() -> pop it at render -> show st.toast()
• toasts automatically disappear after ~3 seconds, so no more stuck messages
• they appear in the corner and don’t mess with layout at all (way cleaner UX)
• users now immediately see feedback like “please wait…” instead of thinking the app died
• instead of “Stripping whitespace…”, it now says things like:
  “Scanning 50,000 rows and stripping whitespace…”
• fixed the dialog bug where “Resume previous session?” wouldn’t close
• issue was st.rerun() inside @st.dialog only rerunning the fragment, fixed with: st.rerun(scope=“app”)
• also cleaned up history_export.py removed old show_msg() + last_success_msg usage
• migrated everything to toast systemSome of the functions still have poor feedback :)
  this stuff is really so annoying, esp streamlit, ughh!!! idk whether to improve user feedback or performance or add new functions.

HII!!

History tab UI improvements

So in this devlog, I improved the History tab by adding visual feedback for all operations and making the redo stack visible!

Before:

• when a user clicked undo or redo, or Reset Data, Clear History, Apply Pipeline, or Generate Report, the app froze for a moment and refreshed giving lil hint that operation successful.

• there was no visual feedback or spinner showing that the app was actually processing the request.

• also the user could see their history of past actions but they had no idea what was inside the redo stack waiting to be redone.

• some buttons didn’t even show a green success box after finishing!

Now:

• I wrapped all the major operations inside history_export.py (like undo and redo) with with st.spinner(“… please wait…”): blocks.

• now when a user clicks any button in the history tab they get an immediate spinning message telling them exactly what is happening.

• I also imported show_msg from state.py so every single operation assigns a message to st.session_state.last_success_msg before it calls st.rerun().

• now the user always gets a green success confirmation box once the spinner finishes!

• added a new list right below the main history list that loops through the redo_stack.

• now if the user undoes an action it pops up in the Available to Redo list so they know exactly what will happen if they click the Redo button.

next,i dont know what needs improvement, but i might focus more on User feedback :)

HII!!

Redo operation

So in this devlog, I integrated the redo operation into the history system!
Before:

• the undo history lived in st.session_state and got saved to the .dp_sessions folder, but if a user accidentally clicked undo, they lost that cleaned state forever
• The app could only go backwards, not forwards :”)
• also when a user clicked undo on a large dataset the app just paused for a second and refreshed. there was no visual feedback that the pipeline was working on reverting the data
  .
  Now:
• I added a redo stack to the pipeline. when a user clicks undo, the current dataframe it gets pushed onto a redo stack. if they click redo, it pops from that stack and goes back into the main history.
• whenever the user makes a brand new cleaning change, the redo stack clears itself because the timeline has changed. I updated session_persist.py so the redo stack is saved to the disk and survives page reloads
  .
• UI changes: wrapped the undo and redo functions inside st.spinner blocks inside the history export tab
• now when the user clicks either button they get an immediate spinning message saying Undoing last action please wait or Redoing action please wait
• added a new list right below the main history list that loops through the redo_stack. now if the user undoes an action it pops up in the Available to Redo list so they know exactly what will happen if they click the Redo button

next i will work on making the user feedback for the history tab more better! ik right now it sucks, but dont worry i will improve it!

Valid tab: UI Improvements

• Now each function gives clear feedback
• eg the email validator tab will say X number of emails has been flagged or removed
• same with the phone validator or standardazing the dates
• i didnt had to fix the UI for outlier cap and validate range function as i already have done that before

Clean Tab: Performance and UI Improvements

voters said before that my app sucks in feedback so now im trying to make it more better
first of all, i made these changes in clean tab

• Every operation now calculates and reports the actual difference.

• Strip whitespace counts changed cells per column.

• Duplicate drop shows the percentage of data removed.

• Smart cleaner lists the names of converted columns.

• Missing value handler reports filled counts and dropped columns separately.

• Type override warns when conversion failures create new nulls.

• Find and replace reports the exact match count.

• Outlier capping reports the exact number of values affected.

• Type guesser adds a per-column progress bar.

• Type guesser now runs via button instead of opening automatically for better performance

Next i will update the validate tab

Upload Feedback Fix- Small fix

Before:

• user picked a file and saw nothing. no spinner, no message, nothing.
• the upload tab showed the success message only after everything was done
• so on Excel files with large sheets there was a silent 5 to 10 second gap where the app looked frozen and the user had no idea if the upload worked. some users switched tabs thinking nothing happened,

Now:

• upload.py checks loaded_file_id against the current uploaded file_id

• if they do not match the file was just picked but not yet parsed so it immediately shows Reading filename please wait in blue

• this renders on the very first rerun after file selection before app.py has done any work at all

• once init_state finishes and sets loaded_file_id the next rerun flips the message to the green success confirmation

• user always sees feedback within one rerun of picking the file

One user complained that they often had to upload the file twice, now they will get clear feedback and their file will quickly upload

Lazy Tab Rendering

Before:

• all tabs blocked on page load for 30 to 40 seconds on large files.
• get_analysis_and_recommendations ran unconditionally at the top of recommendations.py
• the correlation heatmap ran inside a closed expander on every render pass
• streamlit executes everything inside expanders even when they are collapsed , meaning expensive functions was being ran even when user dont even need them :(

The cache.py :”) :

• spent 30 minutes adding sampling and large file guards to every analysis function in cache.py thinking that would fix the slowness. it helped memory and accuracy in large files but did not fix the tab loading delay at all
• problem was never the speed of the functions. it was that they were being called in the first place on every render even when the user had not asked for them.
• the real fix had nothing to do with cache.py, i wasted my time working with cache.py tbh!

Now:

• recommendations are behind a Scan for Issues button, and tab opens instantly.
• scan results are cached in session state keyed on df_key so they survive reruns
• but auto invalidate when the dataframe changes after a cleaning step.
• correlation heatmap is behind a Compute Correlation button inside its expander.
• same df_key keying pattern so results persist until data actually changes.
• clean, validate, and history tabs were already instant and needed no changes.
• on a 1 million row file all tabs now open after some wait(better than before tho).

Also now only the history tab is still stuck with only 2 functions, there is a bug thats keeping it from showing other functions instantly, i will work on that in next one, but other all tabs are fine, i know speed is a issue but for 10k rows , i can only do this much now :(
Edit: Ai also gave its summary after 2 min TT big wait time, but its a given for 10k rows :)) maybe one day i might become a pro and improve its performance, but to me performance is really hard to solve rn

Large File Support

Before:

• everything assumed small datasets. pd.read_csv loaded the entire file into memory at once.
• KNN and MICE imputers were called on 500k row files which would freezes or crashes the browser, also all analysis functions scanned every row every time.

Now:

• CSV files are read in 50k row chunks and concatenated so peak memory stays low
• int64 columns are downcast to int32 and float64 to float32 on files over 50k rows
• this cuts memory roughly in half with a warning shown to the user about precision loss
• KNN and MICE imputers now have a row limit, above 50k rows they fall back to mean imputation which runs in linear time and is safe for 500k rows(found this info from Medium article )
• all expensive analysis functions like recommendations, correlation, quality score and type suggestions now run on a 20k row random sample on large files
• histogram KDE is computed on a 10k sample while counts still use the full series
• overview tab Now shows a warning banner when downcasting was applied
• row count metrics now use comma formatting so 100000 renders as 100,000( yey kinda low level detail TT )

Note:

• In the uploaded video, the data get loaded really quickly but the tabs show stuff really late

• it took 30 to 40 second for that it was because of heat correlation and expensive analysis functions. In next update,i will try that only the tab with expensive functions slow down but other tabs should show up like the validate and clean tab as they really don’t have any stuff that should pre run

• Also i attached a screenshot, check it out, my app also loaded a

1 MILLION rows TT

yay i had to write in the bold, also tho its a bit slow but I can always work on this part <3

Persistent Session History

Before:

• undo history lived only in st.session_state, a page refresh and everything is REMOVED
• users lost all cleaning progress on reload which was evil ik

Now:

• added session_persist.py as a standalone persistence module
• history and both dataframes are pickled to home dir under .dp_sessions folder
• sessions are keyed by md5 of filename plus filesize, NOT streamlit file_id
• this was the core bug, file_id is a fresh UUID on every reload so the session was never found
• sessions older than 2 days are auto cleaned on each init
• commit_history and undo_last both autosave to disk after every mutation
• on file load init_state checks for a saved session before doing anything
• if a saved session exists a native st.dialog pops up showing time since save and step count
• user picks continue to restore full history and current df or start fresh to wipe and begin clean
• reset to original and clear history both delete the persisted file

How tough it was:
YES I NEED TO TELL THIS!!!
3 issues no 4:

• first file_id changing every reload so session was never found
• second windows path separator making tmp dir fail silently
• third st.stop halting before current_df was set so the page just crashed quietly
• fourth pycache serving the old pipeline.py so autosave never fired at all
  Literally tried so many methods to get this work TT i hate how session stuff works :(

Two updates and longer devlog cuz HACKATIME was down :((

AI Cleaner Improvements

Before:

Gemini blindly generated code without checking if the data was ready. dirty columns (currency symbols, % signs, units) caused runtime crashes without having no USER FEEDBACK :”)

NOw:

• Gemini gets raw sample values per column
• added feasibility check , now gemini reasons before writing any code
• if app already has a feature for it, Gemini says go to exact tab + feature as its a guarenteed method to work , but it also generates code incase user want to run this
• if column is dirty and operation assumes clean data , then a warning is shown to fix the column first with the right feature, but code is still generated if user wants to run it anyway
• gemini timeout bumped to 30s and it auto-retries once on timeout

Performance Improvements

Before:

• st.cache_data was hashing the ENTIRE dataframe on every cached function call
• with 7 cached functions each receiving the full df, thats 7 full df hashes per rerun.
• also st.cache_data.clear() was nuking ALL cache on every new file upload.
• profile tab was computing correlation, missing heatmap, and before/after on every tab switch.
• result: 20 to 30s load times even on datasets under 1000 rows :”)

Now:

• added make_df_key() in cache.py
  all 7 cached functions now take df_key as first arg instead of the full df

• Now we have 3 phases workflow
• Phase 1 (~instant): All 8 tabs render immediately. user sees the full app right away
• Phase 2 (~2-10s): app.py calls Gemini after all tabs are rendered, before gemini was calling before the loading of all the app which was causing the issue
• Phase 3 (instant): st.rerun() fires, all tabs re-render with the cached insights already in session state

HI!
In this devlog i moved and upgraded the AI analysis feature

AI analysis moved to Guide tab

• before this the AI fixes were showing inside the Recommendations tab
• it felt cluttered there because recommendations already has its own rule-based list
• moved the whole AI analysis into the Guide tab
• Guide tab is where users go to understand what to do, so AI fits there ig

what the Guide tab AI section now includes

• a plain English summary of what the dataset is about
• a list of prioritised fix cards, most important first
• each card shows the issue title, which column it affects, and why it matters
• each card shows exactly which tab to go to and what to click to fix it
• each card shows an AI Cleaner shortcut so the user can just type the fix instead
• the shortcut tells them exactly what to type into the AI Cleaner in the Clean tab
• the cards use real tab names and real button names from the actual UI
• styled with a blue border for the how to fix section and green for the AI shortcut

what the Overview tab now includes

• AI summary is also shown at the top of Overview as a duplicate
• it uses the same cached result so it costs zero extra API calls
• non-tech users see the summary immediately when they open Overview

how caching works

• Gemini is called once per file load
• result is stored in session state under ai insights file id
• Guide tab and Overview tab both read from the same cache key
• new file uploaded means a fresh call, same file means instant load

also

It works for the multi-sheet excels also! but you have to select the Sheet you want to get AI guide on!!!

HI!
In this devlog i built

AI data summary and AI fix suggester

• one Gemini API call does two jobs at the same time
• first job is a plain English summary of what the dataset looks like
• second job is a prioritised list of fixes the AI recommends
• both come back in one JSON object so we only spend one request per file
• the result is cached in session_state keyed to the file id
• this means even if the user switches between Overview and Recommendations tabs the call only happens once
• new file uploaded means new call, same file means cached result

how the summary works

• shows up at the top of the Overview tab
• 2 to 4 plain English sentences about what the data is and its quality
• non-tech users immediately understand what they are looking at

how the fix suggester works

• shows up at the top of the Recommendations tab above the existing rule-based ones
• each fix has a priority number, a column name, a short issue title, and a one-sentence reason
• sorted by priority so the most important thing is always first
• styled as cards with a blue left border and priority number

HI!
In this devlog i built

AI natural language cleaner

• user types a plain English instruction like drop rows where Age is empty
• app sends the instruction plus column names, types, and 5 sample rows to Gemini
• prompt asks Gemini to return a JSON object with two fields, code and explanation
• so i used gemini api key and made this ai natural language cleaner
• using gemini-3.1-flash-lite-preview, fast and cheap and allows 500 requests per day
• temperature set to 0.1 so output is consistent and not creative
• maxOutputTokens set to 1024 to support multi-line code for complex queries
• Gemini can write code of as many lines as needed
• this means complex instructions like normalise then fill then rename all work in one go
• code field is the pandas code to run
• explanation field is a plain English sentence of what the code will do
• app shows the explanation with “Verify that this matches what you want” so users know what will happen once they apply this code
• app shows the code in an editable text area so tech users can changed it if needed
• user clicks Apply to run it, or Cancel to discard
• code runs in an isolated exec() with only df and pd available, nothing else
• if execution fails the error is shown and the dataframe is not changed
• on success the result goes into current_df and gets logged to History
• API key is loaded from a .env file using python-dotenv
• if the key is missing a warning is shown and the section is hidden

Guide ME feature
I literally tried three approaches before landing on this one.

• the first attempt used a fixed-position HTML card with Streamlit buttons rendered below it, relying on CSS position:fixed !important to reposition the button row to visually overlap the card. this failed because Streamlit’s component tree does not guarantee DOM order or stable class names across rerenders, so the CSS selectors targeting the button row were unreliable and the buttons kept appearing at the bottom of whichever tab was open.
• the second attempt used links styled as buttons inside the HTML card, with a handler at the top of render that read st.query_params and updated state. this seemed promising but Streamlit treats query param URL changes as full page navigations, not rerenders. clicking Next reloaded the entire app from scratch and cleared the uploaded file.
• then finally, added a Guide tab to the data cleaning app
  its a static checklist ,it only exports one function: render(tab).
  it takes the Streamlit tab object and renders inside it.
  in app.py, add a Guide tab to the st.tabs call and pass it to guide.render().
  this was the easiest and is done

What i wanted: was a cool guide feature that shows in bottom right and guide users like how many software works
but it was nearly impossible in streamlit for me tho :”) so i ended up making just a guide me tab :)
pushed both files on github tho so no onne can say how can u spend 1 hr 30 min on just one tab that shows only static data TT

Hi! in this devlog i built
data quality score

• new score from 0 to 100 shown at the top of the overview tab, computed across five dimensions each worth 20 points
• completeness penalises missing cells proportionally, so a dataset with 10% missing values loses about 2 points on that dimension
• uniqueness penalises duplicate rows the same way
  type consistency scans every object column and checks what fraction of values are actually numeric strings stored as text. the more columns doing this the lower the score
• outlier cleanliness uses a 3x IQR fence instead of the std 1.5x so only extreme values count against the score, not natural spread( i hate this probabilty stuff but its so imp for data science and ML TT)
• validity checks text columns for common placeholder strings like none, na, n/a, null, unknown, and empty strings
  , its colour coded green above 80, amber( yes we all think of that amber :”) ) between 55 and 80, and red below 55
• five breakdown cards sit below the gauge, one per dimension, each showing the raw score out of 20, the grade, and a plain english explanation of what was found
• the whole thing runs through @st.cache_data so it recomputes automatically whenever the dataframe changes and the user sees the score tick up in real time as they clean

Hi,in this devlog, i did

cleaning report PDF export

• added new reporting.py module that builds the entire PDF using reportlab
• NOW report includes a summary table comparing rows, columns, missing cells, and duplicate rows before and after cleaning with a change column
• full column profile for both the original and cleaned dataset showing type, null count, null percentage, and unique value count
• missing value breakdown table that lists every column that had nulls before or after, and flags whether it was fully resolved
• ordered list of every cleaning step performed with the dataframe shape after each one so you can trace exactly what changed when
• first 10 rows of the cleaned dataset as a sample table at the end
• PDF regenerates automatically only when the history changes, using the same history_len cache key pattern already used for the Excel download
• reportlab added to requirements.txt. so if the import fails the UI shows a pip install hint instead of crashing the whole app

HI!!!
In this devlog i did!

Data Type Guesser

• new section at bottom of clean tab that scans every column and suggests the correct type based on what the values actually contain
• can detects nine patterns: email addresses, boolean values, currency amounts, %ages, measurement units, time durations, datetimes, plain numeric strings, and low cardinality categories
• each suggestion shows the column name, current type, suggested type, a confidence percentage, the reason it was flagged, and three sample values so u can judge before applying
• rendered as an editable table where you tick which suggestions to apply and press one button, so you never have to go column by column manually
• also the confidence is computed from the proportion of values that matched the pattern, so a column where 95% of values look like currency scores higher than one where 60% do

Pipeline JSON Save and Reload

• cleaning history now exports to a small pipeline.json that stores step labels only, not data

• upload that file on any new dataset and THEE app replays every automatable step in order

• Steps that need manual column selection like email validation or outlier capping are skipped and reported back honestly

• Every replayed step is pushed onto the undo stack so the full workflow stays consistent

Correlation Heatmap

• Added to the Profile tab between the missing value heatmap and the before/after comparison

• Symmetric grid where each cell is colour encoded, blue for positive correlation, red for negative, white near zero

• Value labels are drawn inside cells when there are 12 or fewer columns, hidden automatically on wide datasets so the chart stays readable

• Supports Pearson, Spearman, and Kendall methods with a dropdown to switch between them

• A summary line below the chart surfaces the three strongest off-diagonal pairs so you can spot redundant columns at a glance without reading the whole grid

• All computation runs through @st.cache_data so switching methods is instant with no recompute

Also this is only available in ADVANCED mode, don’t want to scare off the users who don;t know much!!

Data Cleaning Pipeline v3.0.0

• I figured out i should do devlogs like this from now on instead of doing long para as these are more readable

Modular Refactor

• Broke a 1800 line single file 19 focused files across a clean folder structure. For me, understanding the code before was easy because streamlit has so less code to do but for reviewers it might not be easy so !
• cleaning/ folder is now Python with zero Streamlit dependency so every function is independently testable
• Each tab gets its own file with a single render() function, app.py is now just 106

Distribution Charts

• Added histogram with KDE curve overlay to the Profile tab for numeric columns
• IQR fence lines drawn directly on the histogram so u can visually judge outliers before deciding to cap them
  categorical columns get a horizontal frequency bar chart with percentage tooltips
• missing value heatmap shows a sampled grid of the whole dataset, red cells are nulls, blue are present
• All chart data goes through @st.cache_data so switching columns has no recompute cost

Column Transforms

• Split column breaks one column into many on any delimiter, extra parts pad with NaN, source column optionally kept
• merge columns concatenates any number of columns into one with a custom separator
• rename columns gives an editable table so you can fix all names in one session without touching code

HI!
in this devlog i built the boot animation.set up array of boot steps where each step has a message, a progress percentage, and a delay in ms. a recursive setTimeout function go thro them one by one updating the loading bar and status text. the start logo spins using a CSS keyframe animation. when all steps get finished it hides the boot screen and shows the login page. restart also goes through the full boot sequence!!!

Splash screen, UI and some fixes!
Hi! In this devlog I improved UI of LectureVault.
I built an animated splash screen!!! Also made logo for LV which is still so meh but i got no artistic sense so :”)
Fixed all the dim buttons across the app. Bottom nav icons, filter pills, sort button, the Change storage location link in settings, Move and Delete in the selection bar, and all dialog action buttons are now solid vivid colors!! because voters before complained my buttons aren’t good so :) before i just made this to just get its functionality right, i didnt bother much on UI but many of voters said my UI is trash so imma fix it now!!!

HI!!!
in this devlog i built the boot and login system
the page now starts with a login screen instead of jumping straight to the desktop
built a tryLogin function that checks the credentials and either dismisses the screen or shakes the form and shows an error. the shutdown button triggers a black overlay saying restarting, user has to reload to get back in. restart does the same black screen but after 2.5 seconds it would) automatically brings the login screen back. empty fields also count as valid so you can just hit enter and get in quickly.

HI!!!
in this devlog i rebuilt the window management system
the old code just toggled a class and had no concept of which window was on top.
so, i replaced it with a z-index counter that increments every time a window is focused so the last clicked window is always on top. minimize now hides the window but keeps the taskbar button visible and grayed out. clicking that grayed button restores the window to its exact last position.
the taskbar button also now correctly toggles!!!!

hi!! in this devlog i setup2 the start menu for PixelOS.
so the structure was already there from last session so this was mostly about getting the interactions right. clicking the start button toggles the menu open and closed and adds a glow to the button so you know its active. clicking any app in the menu opens that window and closes the menu at the same time. i also added a listener on the desktop so clicking anywhere outside the menu closes it. the shut down and restart buttons trigger a toast notification saying restarting and shutting down.
I also changed the wallpaper!

Browser window
hi!! in this devlog i built the Browser window for PixelOS!
i set up a pages object where each key is a pixel url and value is a function that returns html.
this made it really easy to add new pages later on… i also built a proper browser history array with a current index ptr so back and forward work correctly. clicking links inside pages navigates too.

hi!!
in this devlog i built the text editor window for PixelOS!
double clicking a txt file now opens it with the content loaded instead of browser alert that i had set before.
i wrote an openEditor function that takes a file object and populates the textarea. also added live word and character counting that updates on every keystroke, and replaced all alerts with a proper toast notification. the save button and new button r also working!

hi!!
in this devlog i built the file explorer window for PixelOS!
i set up a fake file system as a javascript object with folders and files nested inside each other. clicking the desktop icon or taskbar button opens the window, and you can double click folders to go inside them. i also built a navigation history stack so the back button remembers where you came from. dragging the window also works!!!

hi!!
in this devlog i set up the static website for PixelOS! Just put static icons there for now!well, took me alot of time searching for right wallpaper and icons TT and then i also had to remove the backgrounds of icon using canva :”) wallpaper is in png while all icons are in svg format!
for now, i styled the taskbar with the start button, app shortcuts, and a live clock!!!
i will work in increements and set more things in my webos!

password: 2010 btw u could have find it out by seeing the crack on keys

Hi!
In this devlog I rebuilt the diagnosis engine in MediScribe AI.
The system worked before, but results were kinda unreliable. confidence scores were wrong and way too less.

First, I removed a penalty bug. Normal findings were reducing scores for real diagnoses.
second, I added support for required facts. Now a diagnosis only appears if relevant sections have data.
third, I fixed score calculation. All diagnoses now use one clean formula. Scores stay between 1% and 95%.
fourth, I fixed deselection issues. Old answers from removed followups no longer affect results.
fifth, I unified logic. SOAP and diagnosis screens now use the same scoring system.

Please check the whole readme and demo before voting, i know many of you wont get this project but if u research on it then its really a cool thing to make :”)

NOTE: I rushed alot in demo because FT dont allow long videos :”)
you can watch full demo on this link:

https://drive.google.com/file/d/13VzI_spU9x2eqakv_oEsoGAXvFYAF1iM/view?usp=sharing

hi!!
in this devlog, i did nothing much but find an UI mishap!
the user can only save the records once he is on last screen of history taking
so i made a bottom draft widget that will be shown in every screen of history taking and it says save draft and exit( means go back to homescreen).
this way user won’t have to fill in every steps and can just safe draft!
then i also made my code modular by adding a patient repositary service!

Drawing Canvas Improvements
hi!
in this devlog, i added an eraser, a special pen (filling pen), and an undo button to the shared notebook drawing tools.
you can now erase mistakes, or use the special pen to fill.for this purpose, i just added a local array to track my stroke ids for the undo button and encoded the new tools directly into the stroke color strings (like sending “ERASER” or appending “_FILL” to the hex) so we didnt even have to change the supabase database schema.
Also, playing around with the special pen(filling pen) is so fun, check it out, i found it way more cool than bucket feature in paint, thats why i build like this

Offline Storage & Full Edit Flow
Hi! In this devlog I added offline storage and a complete edit flow to MediScribe AI.
I integrated Hive as the local database, every patient session is now saved as a single object containing all seven data layers: patient info, history, systemic review, vitals, labs, examination findings, and the generated SOAP note.
The Patient Records screen now reads live from Hive using a ValueListenableBuilder… so the list refreshes instantly after saving or deleting.
Tapping any patient record opens a full detail view showing everything entered across all screens. The edit flow now pre-fills every screen. Re-saving overwrites the original record cleanly.
The SOAP screen’s bottom button now saves and navigates home in one tap.
Next feature will be integrating neural network models to give more precise diagnosis based on all history taking

Sorry kinda long devlog but no worries i still document everything in this devlog :)
Hi!!
in this devlog, all i did was to suffer :”))

so i try adding a catch block in upload screen that catches gemini error like reaching the api limit!
but implementing this feature broke my all flow, try to fixed it many times but no vain, so i had to switch to prev commit code :)
also there was the error in toast message of undoing the delete action, it followed user everywhere so i fixed it by clearing the toast message while moving to another activity!
Also, I added the move photos functionality in other folders also, user now can move images among folders
also updated the readme!
moreover, user now cant delete the folder named unclassified cuz obv unclassified images need to go there!
also, image was’t loading sometimes!
Android returns content:// URIs from the image picker, and Image.file() cant read those as it only works with real paths.
so i cache the image bytes immediately after picking (while the URI permission is still active) using XFile.readAsBytes() then i render with Image.memory() instead.

Also good news
before i was using gemini model 2.5 flash lite and it was only 20 requests per day, but now i find out about 3.1 flast lite model and it can do 500 requests per day
so i am using it right now
also, my app send 1 request for 20 pics in one session, so i am saving alot of gemini credit this way!! so you can use application in peace cuz
if u upload 20 images per session, then we classify 1000 pics a day!!!

also, if someone dont wanna worry abt it at all! so he/she can just clone my repo and put their own gemini key in .env file. but i dont think so you need to do this

demo video is attached, i can assure you images i classified in app is 100% correct :”)

Pinch to zoom feature and some UI improvements!
Hi! In this devlog I fixed and added a few things.

i added unclassified folder that gets created automatically during onboarding setup so photos that the AI cannot confidently sort can go there

made delete folder flow so users now get a choice when deleting. They can either remove the folder from the app only, keeping all their files on the device, or delete everything including the physical folder and photos inside it.

Improved the pinch to zoom in the image viewer.
Replaced the keep original copy toggle in settings with a clean info card explaining that my app never touch the camera roll.

also i attached the video testing every feature, check it out!

debugging and debugging and image view feature minimal implementation

Hi! In this devlog I fixed several issues

first issue was the app kept showing the onboarding screen every time it launched even after setup was done. Fixed it by calling markSetupDone() at the end of the folder setup flow so the app now remembers that setup was completed.
Second was the Reset App button in settings was not properly clearing the setup flag. so i fixed it by replacing the individual save calls with clearAll() which wipes everything including the setup done flag.
third was the classification settings toggles in settings were hardcoded before. so i make it save to SharedPreferences and reading them back on load, then wiring each toggle into the actual upload logic
fourth was deleting a folder from the folder view screen did nothing. Fixed it by implementing the delete folder flow properly with a warning dialog that tells you that folder will be deleted from the app but wont be deleted from the device.

moreover, i added a new image viewer screen that opens when you tap on a photo in the folder view. It supports swiping between photos, I made it so that user can see the actual image we are classifying right now
I also removed the About section in history as i really dont need it!
also, i added a create and delete folder option in the homepage also, also fixed navbar issues :)
in the next devlog, i will implement zoom logic more correctly and also will implement sharing fuctionality of pictures!!!
ANd will remove the redundant icons i placed before :”)

OCR + Gemini Classification
hi!
in this devlog, i got the full AI classification pipeline working. here’s how it actually went

the OCR part was kinda smooth. i used Google ML Kit’s text recognizer and ran it locally on each image, so no internet needed. on top of that, i added a small OCR enhancer that fixes common mistakes like “Ca1cu1us” -> “Calculus”, pulls out keywords by removing stop words, and detects academic patterns like “derivative calculus” or “acid base chemistry” using regex. and also enhance my images first before sending it to OCR (also did rotating of image incase they are yk in weird position)
for gemini,
at first, i was sending one request per image, so 10 photos meant 10 requests. this instantly hit the rate limit (free tier only allows 10 per minute). then i switched to batching (mean many images->extract keywords->send it as one prompt to classify it) sounds easy but its not, everything came back as Unclassified with 0% confidence, even though OCR output looked fine. tho single image requests still worked perfectly,
so i added a debug panel on the upload screen to actually see what was going on. turns out the batch prompt was structured completely differently from the single-image prompt.
another issue was maxOutputTokens being set to 1000, which was cutting off the JSON response for larger batches. increasing it to 4000 fixed that part.

then i gave up and i used NotebookLM to refine the batch prompt. the fix was putting the reasoning field before the subject field in the JSON. this made Gemini process the text first before deciding the label. combined with making the batch prompt match the single prompt style (same formatting, examples, tone), it finally started working.

final setup: OCR runs in parallel locally, then a single Gemini request classifies everything and returns a JSON array, mapped back using IDs.

Image Upload Feature
hi!
in this devlog, i added an image upload feature to the shared notebook. you can now select a picture and it drops exactly in the center of your screen.
.for this purpose, just added a native compressImage helper using the browser canvas api to resize big photos to 1024px and compress them into 80% jpegs before sending them. then it uploads the tiny jpeg to the supabase notebook images storage bucket and drops it in using the handlePlaceMedia function.
the tricky part was the coordinate math and upload delays, images were getting placed outside the current page view because it calculated the position after the upload finished and accidentally double counted window.scrolly in the math
.fixed it by capturing the exact startx and starty center coordinates before the upload begins so it fits where you were looking even if you scroll away.
next feature: idk honestly, im too tired cuz of this image upload feature :)

Settings Screen
hi!
in this devlog, i integrated the settings screen to save and load everything from SharedPreferences. before this it was all in memory so changes were lost when you left the screen.
you can now add a new subject, rename an existing one, or delete one.
it was mostly just replacing the local list mutations with calls to StorageService.saveSubjects() after every change. add, rename and delete all update the list then immediately write the whole thing back to prefs.
hmm the tricky part was making sure the home screen reflects the changes after you come back from settings. fixed it by calling _loadData() inside didChangeDependencies!!!
next devlog: OCR using ML Kit and maybe gemini integration also
Note: whole about section is for the future implementation, as i havent made any release on playstore yet

Folder View
hi! in this devlog, i made the folder view screen show real photos from the device instead of grey placeholder boxes. user can tap any subject card on the home screen and see the actual lecture photos you uploaded
it reads the files directly from the subject folder on device storage, pulls real metadata like file name, size, and modified date. deleting a photo removes the file. long pressing enters selection mode where you can delete multiple at once. the bottom sheet also shows a thumbnail of the photo.
did it by using getPhotosInSubject from storage service to list all jpg and png files in the folder, then used Image.file() to render them. added a deletePhoto method that calls file.delete() on the actual path.
the tricky part was reading args passed from the home screen , i had to make sure that was read inside didChangeDependencies so the photos reload every time you navigate back to the screen.
next devlog: settings screen maybe TT

Photo Upload
hi!
in this devlog, i made photo upload work.
when you pick photos, they show up in the photo strip and the classification list with your actual images. you can tap the arrow on any photo to expand it and manually override which subject it belongs to. once you hit confirm and save, the photo gets physically copied to the correct subject folder on your device storage.
it was straightforward for the most part, just wired up image_picker for gallery and camera, and used dart:io to copy files into the right folder path.
the tricky part was permissions. android 13 and below handle storage permissions completely differently, so i had to request photos permission first and fall back to storage permission for older versions otherwise it silently failed.
next devlog: folder view screen showing photos from device

Lock Feature
hi!
in this devlog, i added a lock feature to the shared notebook. you can now hover over any text, sticker, or gif and click the lock button that appears, this locks it so no one can delete it, or edit it. when something is locked, hover again and click lock button to unlock it.
it was easy, just added a locked boolean column to both writings and media_items in supabase. on the frontend, when locked, the delete button is hidden, drag is blocked, and editing is disabled. the lock/unlock just does a simple update to flip that boolean.
the tricky part was realtime sync, the lock state wasn’t broadcasting to other users at all.. fixed it by running ALTER TABLE writings REPLICA IDENTITY FULL
next feature: image upload integration maybe

Multiple Notebooks Feature
hi!
in this devlog, i made the shared notebook supports multiple notebooks, there’s a dropdown in the toolbar where you can switch between them, create new ones of your own..e.g someone can make a notebook just for cat pics, another for music recs, whatever.
for this purpose, i added a notebooks table in supabase and put a notebook_id column through writings, media_items, and drawing_strokes.
i also set up RLS policies, anyone can read and create notebooks. I didnt made feature of deleting someone’s notebook because its just evil
Next feature can be of locking one’s text so no one can delete it or image upload integration!

Debugging and debugging :”)
Hi!!
In this devlog, i Spent a hours doing a full code review and fixing everythin i found out is wrong!
In both smart_column_cleaner and the Recommendations tab conversion paths, converted series with a sparse dropna index were being assigned back to the full dataframe. This simply means it turns the originally null rows into NaN. The fix was applying .reindex(df_clean.index) in all conversion branches. Similar issues existed in validate_email, cap_outliers, and validate_range, where row removals left a non contiguous index and caused mismatches in before/after comparisons. These were fixed by adding .reset_index(drop=True).

moreover, KNNImputer could receive n_neighbors=0 on very small datasets.. so i added a guarded max(1, min(…, len(df_clean))) constraint . IterativeImputer also crashes when only one numeric column exists, so it now automatically falls back to KNNImputer.

the most illegal one :”) history was being pushed even when operations failed, so it now uses snapshot() and commit_history() to record only successful operations
.
find_and_replace previously cast entire columns to strings, converting NaN to “nan”. validate_date_col also used an incorrect missing value type.Also, all pandas inplace=True calls were replaced with reassignment for pandas 3.x compatibility, and Excel downloads are now cached to avoid unnecessary regeneration.

Note: I attached the video explaining each feature, tho video is rushed cuz ft dont allow uploading large videos TT

tooltips, readme update, and testing
in this devlog, i Added help tooltips across the Clean and Validate tabs. Every widget that needed context now has a ? icon, on hover it gives the explanation of what the option does and when to use it. I made it so that even non tech user can understand whats happening with their data!
Also Updated the README and included new features description in it. e.g the tab layout, column profiler, before/after comparison, history and undo, pipeline export, and multi-sheet Excel support. Also added one more multi-sheet excel test files.
Moreover, i performed extensive testing and made sure that my pipeline works. My app works on single sheet excel and multi sheet excel also.
If you guys have any issues with it then feedback is welcome!!!

Before/after comparison, column type override, and pipeline export
Hi!!!
in this devlog, three more features are added.
First was before/after column comparison, you pick a column and it shows original vs current values side by side with a third column that flags exactly which rows changed.
Second was column type override. a dropdown is given to force any column to int, float, string, datetime, boolean, or category.
it was kinda tricky TT cuz booleans needed a string-mapping step first, integers needed nullable Int64 so NaN values don’t cause a cast error, datetime needed errors=‘coerce’.
Third was pipeline export. which I think is the most useful thing. It reads through the cleaning history and writes out a proper .py script you can download and rerun on any new file.

UI CHANGES
HI
This devlog is mostly a styling and restructuring UI rather than new features.
The first thing was removing the big heading at the top of the page. because it was taking alot of space and isn’t necessary in the first place.
Then i made the tabs sit right at the top of the page like a website navbar.. Streamlit’s default tab styling is pretty small, so i had to override it with CSS.
Before, my UI was having a long long one webpage, which was causing the bad user experience
Now i made separate tabs for separate purposes so just by looking at the UI, user know where he will find what features!!!
Streamlit is easy (if we use its own styling) but if we try to make out own css in it :”) then its sucks :)

Implemented Column Profiler, find & replace and undo/history system
Hi!!
In this devlog, I added three features to the data cleaning pipeline that I felt were genuinely missing from the first version.
First was a column profiler ,table that shows you per-column stats like min, max, mean, median, std, skewness, null percentage, and sample values. Cached it with @st.cache_data so it only recomputes when the dataframe actually changes.
Second was find & replace, pick a column, type a search string, type a replacement, and optionally flip on regex mode for pattern-based replacements..
Third,, was a full undo/history system. The history panel shows every step and lets you undo one step at a time or wipe the whole history.

Resolved Performance Issues
HI!!
in this devlog, I optimized the app to stop re-running expensive operations on every single user interaction. The main issue was that analyze_data_issues, which runs regex loops across every column in the dataframe was firing on every button click I fixed this by wrapping it (along with the recommendations generator) into a single @st.cache_data function called get_analysis_and_recommendations. Same thing i did to the file reader and the stats calculator. Also deleted all verbose=True calls inside manual operations asthey were triggering partial rerenders.
Some users said its a bit slow, thats why i spend 3 hours , only figuring out what could be the issue, and came up with these solutions
Also, I made two modes, one simple and one advanced in the settings sidebar, by clicking to advanced mode, user can select the thresholds and model they want to implement for missing values!
in the next devlog, i will implement more functions slowly and will also make it more user friendly.

Hi,
In this devlog, I made the persistent data layer working. the app needed to remember ur subjects and storage folder between sessions.
So,
I used SharedPreferences to save and load everything locally on the device. So now when you go through onboarding, pick your storage folder , add ur subjects . all of that will be saved in the folder you selected.
Next time you open the app, it reads that data and takes you straight to the home screen with your subjects.
In next devlog, i will integrate the photo upload system

HI!!
In this devlog, I made the settings screen. Here, the user will fill in his clinician details (Name, Designation, and Hospital). He can also enable the dark mode, which will be implemented later. moreover, it also has the version and AI engine details about the app.
I did it using the SettingsService singleton and custom Flutter widgets like _SettingsCard and _Field.
Also, One thing I love abt Flutter/Android dev is we can just import Material Icons and make it work!

Hi!!!
So i built the patient_records_screen, for now its having the dummy data because we will integrate hive database later on
it will have all the patient records till now, user can also delete, edit or copy the patient details.
moreover, i also make the navbar working, linked patient records screen with the navbar, and also cleaned the dashboard and remove the buttons i added before but i didnt actually needed them :”)
Also navbar will not be shown when we are entering the patient record because those are the steps user have to perform, what if in between he accidently click the home screen so :)

Hi!!
in this devlog, i made the final diagnosis screen that shows the diagnosis in detailed manner. this diagnosis is gained through the physical examination done by the doctor and through symptoms entered!
And moreover, i made my code clean and modular, i separated the data models from screens dart code and save it in models folder thus making my code more easy to extend in the future
and also this way my code wont have to import big screens.dart files and can just import the data models, now my architecture is pretty clean

HI!!!
In this devlog
i made full structure of my application, i made only the UI of whole system, cuz this way it will be easy to visualize all the flow
and also , i linked all the screen together so now all left is backend working. I also figure out what services to make, so basically i made all the ARCHITECTURE of the system!
also before i was doing it in JAVA but someone suggested i should make it in flutter because flutter is more easier to learn than JAVA
soI here im! I also made a new repo for flutter cuz i pushed it in prev repo with java code and it was causing errors so i made new one!
Also , upload pic functionality isnt working as i havent implemented its backend yet :”)

HI!!!!
In this devlog, i made the SOAP screen
for context, SOAP is literally how doctors write clinical notes.
S = Subjective (what the patient tells you), O = Objective (what YOU find, vitals, labs, examination), A = Assessment (differential diagnosis), P = Plan (what to do next). every doctor writes these.

so what i built is, after you go through the ENTIRE flow (patient info → history → systemic review → vitals → labs → physical examination) the app auto-generates a full SOAP note from ALL of that data.
the S tab pulls the patient’s name, MR number, chief complaints with duration and severity, past medical history, drug history, allergies, family history, social history.
the O tab shows vitals with abnormal flags, and the full lab panel
the A tab is the differential diagnosis section
the P tab is the plan, investigations, treatment, follow-up. its still in the making
also everything is EDITABLE. so the doctor can tap any section and just fix whatever the AI got wrong.

Lab Screen Integration
Hi, in this devlog I made the lab screen where the user will enter the lab data.

I made various lab panels in it, Complete Blood Count, Liver Function Tests,
Renal Function Tests, Electrolytes, Glucose & HbA1c, Lipid Profile, Thyroid
Function, Cardiac Enzymes, Coagulation Profile, Arterial Blood Gas, Urine
Analysis, Iron Studies, and a free-text Cultures section.
made it according
to the official WHO and standard clinical reference ranges for all normal,
warning, and critical values.
And this data will be stored in Hive later on.

This was difficult to make because instead of
hardcoding every panel I had to design a LabTest and LabPanel config system so
the UI renders itself automatically from the data , moreover
each panel is collapsible, theres a live abnormal counter in the app bar, and
a save summary sheet that lists all flagged values before proceeding

Patient Info Screen Feature
Hi
so in this devlog I made the patient_info screen that takes the patient information like age, gender, full name, date of birth, address, marital status, religion, date of admission, and mode of admission (OPD or Emergency). It also auto-generates a unique MR number for each patient.
I built it using Flutter with a Form widget. The date of birth uses three linked dropdowns (Day / Month / Year). There’s also an animated OPD / Emergency toggle pill.
In the future I will implement Hive DB in it so all patient data will be stored locally on the device.
I also spent time linking all the screens together so the final navigation flow is:
Home -> PatientInfo -> HistoryTaking (3 pages) -> SystemicHistory -> Vitals -> Examination

HI!!
i was trying to make my website responsive :”) and i broke the editing feature in it, then i spend some time debugging it and used the prev working one then!!
i have tried many times to make it responsive but it just dont Work, so im leaving it as it is
ALso i check other features and make sure everything is working correctly!
Also updated the readme
Note: please use it on laptops or tablets TT
i have attached the video explaining all the features!!!

hi, so in this devlog, i implemented the lofi music in my notebook, this music will be synced for everyone! You can choose to pause or play it!
When user first enter the notebook, it will not play , it will only play if user decided to play it in the lofi tab!
i did it by using direct MP3 streaming via the HTML element pulling from nightride.fm.,
i only implemented 4 of the playlists in it!

Hi!!!
so in this devlog, i implemented the Reactions feature that makes reaction emojies floats for some seconds then disappear just like how it is in Google Meet
for this purpose, i used useReactions hook which uses Supabase Broadcast. It was easy to implement!!
next feature would be Lofi music integration maybe! or I might just improve the existing code a bit and reship it and then move on!!!

HI!!
so in this devlog, i built a feature that shows who is currently typing right now . i built it using the Supabase realtime presence thing, and the channel lifecycle bug was the toughest part thats why it took me 50 mins TT

every user joins a shared presence channel and calls channel.track() to broadcast their current state (name + whether they’re typing). When anyones state changes, the sync event fires for all other users and they update their UI accordingly. thats how i made it work in the end :”)

HI!
so in this devlog i made a name feature, when you open the notebook for the first time, a little modal pops up asking for your name. then whenever you write a textbox, your name shows up underneath it on hover so everyone knows who wrote what :)
i did this by saving the name to localStorage so you only get asked once, and passing it as an author_name column on every insert to supabase. then in the WritingNode component i just render a small label that fades in on hover. honestly nothing crazy going on under the hood :”)
it was easy work using supabase, i just had to add one column and one label, the modal was like 20 lines.
Also, u see a toast message , [this person] has joined or left the notebook!

HI!!!
So in this devlog i implemented the realtime feature, before this users had to reload the notebook to see other people updates.
but now it works by subscribing to postgres_changes on each table.
and also before i tried implementing this feature but it didn’t work because i had realtime enabled in the supabase dashboard but never actually wrote the .on(“postgres_changes”, …) subscription in the code for writings and media_items.
yes i almost gave up for this but then i said why not try it out one more time :”) Before it didnt work cuz i was not integrating it in my code well, i also overlook this feature back then, but its so cool to have it

Also, it took me almost 30 min to implement this, but since flavortown was down, so i worked on other feature
which is user will enter his name in the start, and my db will save this name and show it alongside with his writings!!!

ERASE DRAWING IMPLEMENTATION
Hi!!
So in this devlog, i worked on drawing feature :”)
ANd IT WAS SO HARD TO MAKE!!!
first i tried making a eraser how we have this in the Paint, but then i read that for this my pic need to be saved as a flat image in supabase so it can erase it accordingly. but i was saving the strokes individually so that wasn’t going to work, then i switched to the cross symbols on the strokes, just like my textboxes,so when user clicks on these cross symbols, the stroke gets deleted. its not as easy as it seems :) dealing w strokes is very tough because the strokes are just arrays of points on a canvas, there’s no DOM element to attach a button to. so i had to do hit-testing on every mousemove i check if the cursor is close enough to any line segment in any stroke, and if yes, show a ✕ button floating at the midpoint. BUT THEN the invisible div i put over the page to detect the hover was blocking all the drag and drop for text nodes and stickers :”)) so i had to scrap that and instead attach the mousemove listener directly to the page element so it wouldn’t block anything underneath.
and i dont know why everything i do breaks the drag n drop functionality, ugh i hate it!!! so got it working also by debugging it
(apparently its cuz of drag events bubble up through the DOM, so any event listener(mousedown/mousemove) before the drag handler gets it will silently KILL the event)

Hi!!!
So, in this devlog, I improved the anti-cheat system again bcz it was not working with tablets with sidebars.
I was checking if
(outerWidth - innerWidth > 160) or (outerHeight - innerHeight > 160) to detect devtools…
but the problem was that on tablets, opening a browser sidebar shrinks the innerWidth by the same amount devtools would.so it was triggering the
CHEATER DETECTED screen.
I removed that size threshold check entirely, and now it relies on the other detection
methods: the debugger timing trick, the Image getter trap, keyboard shortcut blocking,
and right-click prevention.
Yes!! Thats it, i hope you enjoy it

ADDED DRAWING FEATURE
HIII!!!
SO in this devlog, i added a pen drawing feature to the shared notebook :”). SO, you can now switch between Text and Draw mode using a toggle in the toolbar. when you’re in draw mode, you can pick your pen color from 12 presets and choose from 3 pen sizes (small, medium, chunky). then just draw !
for this,i also made a new table in supabase named as strokes!
the TOUGH part was getting the drawing coordinates right across multi-page layouts since the canvas had to match the full scrollHeight of the page, not just the viewport.
(ngl this coordinate calculation always frustrates me TT)

hi!!!
so, this devlog is just about refactoring the code, tho i checked and its pretty nice as it is, Also added more keywords in content-analyzer!!
the thing is i got 6 mult on 11 hrs of work on this project cuz of bot voters, so i have to reship this again
i hope i get my justice now TT!!!
Thats it!!! I hope u like my extension!!!
Also, in the attached Video, i was rushing alot TT because flavortown has the limit for the video upload, it wont let me upload if my video is too long TT!
it still went to 2+ min :”)

BUILT THE ENTIRE EXAMINATION ENGINE!!! (used the prev code also i wrote in first devlog)
HI!!!
SO in this devlog, I built the physical examination screen ,it was the most complex screen in the whole app :”) SO, every question used to have its phase title, instruction, and clinical tip hardcoded in THREE separate giant const maps (like 32 entries each, 96 lines of pure hardcoding). so i deleted all of that and moved it into knowledge_base.json as fields directly on each question object( phase_title, instruction, tip).now KBQuestion.fromJson() just reads them directly.
ALSO the constraint system was a whole thing, you cannot select “Tachycardia” AND “Bradycardia” at the same time (because, you know, physics) .so i built a contradiction checker that rolls back any impossible selection and shows a red banner. there are like 40+ mutually exclusive pairs across CVS, RESP, ABD and NEURO. none of them were hard individually but mapping ALL of them out at once was a little (NOT LITTLE) pain :”))
ALSO the rule engine injects follow-up questions dynamically ,so if you select “Irregularly irregular rhythm” it unlocks an AF-specific question mid-flow. and the results sheet shows diagnoses with certainty percentages calculated from a scoring system.
I ALMOST gave up and considered just making it a static checklist :”) also its not perfect right now,i will spend more time on it :”)

HI!!!!
so before doing a physical examination, doctors actually have to record a TON of stuff first. vitals!! blood pressure, pulse, temperature, respiratory rate, oxygen saturation, blood glucose, BMI. and each one has clinical ranges. like if BP is 145/90 that’s not just a bit high, that’s specifically Hypertension Stage 1 and the doctor needs to know that immediately.
so i built the vitals screen. as you type in any value, the app runs it through a clinical engine i wrote and flags it in real time. the card turns red with a little warning strip telling you exactly what’s wrong. “Hypertensive Crisis,urgent evaluation required.” that kind of thing.
it also auto-generates a flags list at the bottom. so if a patient comes in with high BP AND low SpO2 AND tachycardia, you see all three flagged together in one place before you even move to the next screen. those flags get passed forward through the whole app flow.
also added a custom vitals section for when doctors need to record something outside the standard list, things like CVP or ICP or ETCO2 that you’d see in ICU settings.
the full app flow is: patient info → medical history → systemic history → vitals → lab reports → physical examination → SOAP note → diagnosis (training a neural network for this part!!)
slowly getting there :)

HI!!!!
SO, I found out that before doing a physical examination, doctors first have to take the patient’s history. This includes asking about previous medications, past symptoms, current complaints, checking vitals, and also asking if there is any medical history in the family. Basically, the doctor first tries to understand the patient’s background before actually examining them.

Because of this, I designed the overall flow of my application around how a real clinical process works. The flow goes like this:
take patient history → take vitals → take lab reports → perform physical examination → generate SOAP → give diagnosis.

For the diagnosis part, I am planning to train a neural network model so that the system can suggest possible diagnoses based on all the collected information.

So in this devlog, I worked on building the UI for taking patient history. The process starts with the user entering the basic details of the patient, like name, age, gender, and other identification information. After that, the user can record the patient’s medical history, such as past illnesses, medications they are currently taking, and any relevant family medical history that might affect the diagnosis.

Next comes the systemic history section. In this part, the user records information related to different body systems, for example if the patient has any issues related to the respiratory system, cardiovascular system, digestive system, and so on. This helps doctors quickly understand which body system might be affected and gives better context before moving on to the physical examination stage.

So overall, this part of the application focuses on collecting structured patient history in a clear and organized way, which will later help the system generate a more accurate SOAP summary and diagnosis.

Physical Examination IPPA style JSON building for CARDIOVASCULAR EXAMINATION ONLY devlog
HI!!!
SO in this devlog, i implemented the json file with all the facts and rules!!! So what will it do is, it will match the facts with rules and ask the user questions abt what he is observing during the 4 type of medical examinations which are IPPA so it will first ask abt general examination then IPPA (inspection, palpation, percussion, and auscultation)!!
so my app will help the medical students/doctors in taking the physical examination or history of the patient!! My friend is a medical student and she said she often forgot what steps to perform in each examination, so my application will help her remind all the questions/steps being performed in the examination and then in the end will also give diagnosis!
You can see it working for cardiovascular for now in the video attached!
Also i used the medical book KUNDUS BEDSIDE CLINIC 4th EDITION for building of my JSON so all the data being used for my application comes from this book!

HI!!
So, this is my first devlog, im making an application for junior doctors or medical students, they can run it offline and find out what questions to ask to the patients based on specific examinations, in the end it also gives suggestion what this all symptoms point to. Lets say breathlessness and High bp points to heart failure(it dont always does but its a hypothesis lol)
This is all based on kundu bedside clinic book!!! SO in this 1h 50m devlog, all i did was find out what will be our dart/flutter logic, how will UI works, what main features will User see for this purpose
and figure out how to make our Json with all facts(defined terms/questions used in medical books)
SO, i figured it out and then make the minimal working UI for it, rn it have no questions placed in it, as i will be placing it later on using the JSON i will built , i will make a forward chaining inference engine(see facts first and use it to lead towards the goal) that will look at our json and based on each examination suggest the user what should he examine in patient!!!

NOTE: i will be providing the examinations in IPPA medical styling!!!
I am not a medical student, but i will make sure all of the stuff i will make to be accurate, i will only use kundu book for this! and make it work!!! also making this for a medical friend who needs this :”))

HI!!!
IN this devlog, i again checked all the functionalities and found out that toolbar is just static at the header so i make it sticked to the top regardless of the page position!!!
I just had to add one thing in its css
position: sticky
so, it was easy work, but i found out how dumb i am, i was only testing my website using the first page only TT and didnt thought about adding gifs or stickers in later pages

Note: i added video of the project explaining all the functionalities!!! watch it to find out all the functionalities TT ( but they are so obvious to the users, i dont think so u guys need to hear me yap)

HI!!!
so in this devlog, i debugged the drag n drop feature. I found out that drag and drop isnt working deep down in the page because i was calculating the Y position of updated position wrongly :”)
Spent Hour figuring out whats causing the issue :”) , apart from this, i cleaned my code and tested all the functionalities.
Also, I will decide on new features based on reviews or voting. so suggest me more features please TT

DEBUGGED EDITING FEATURE
HIII!!!
SO in this devlog, I debugged the editing feature :”) SO, clicking a node to edit it would also trigger the global page click handler at the same time, so i stopped that with e.stopPropagation(). ALSO, contentEditable was also not working where the cursor kept jumping to position 0 instead of the end,then i set a Range to fix that. also if someone clicked away without typing anything it would save an empty string, so i just delete the row instead if its blank. none of it was super hard but debugging all of it at once was a little( NOT LITTLE) pain :”)) I ALMOST GAVE UP AND WAS THINKING OF JUST REMOVING THE EDITING FEATURE :”)

GIFS and STICKER INTEGRATION
HIII!!!
SO in this devlog, i integrated the gifs and emojies. For it i also made a table in SUPABASE for stickers and gifs so they updated in real time.
I add a list of emojies in the code(hardcoded) and you can also search it accordingly. I dont use any apis for the emojies and implemented it all in code.
Also, i used api of giphy for gifs, user can now add gifs into the Page. and can also search them. I also added options for gifs user can use to search! like sakura, kawaii and cats(cuz i love cats)
Adding api made life alot easier TT and also giphy apis are free to use, you can also use that to integrate gifs in your project!!!
Also, user can also resize the gifs or stickers!

MORE THEMES + EMOJI PLACEMENT DEVLOG
HII!!!
In this update i did a bunch of theme improvements.
second thing. before there were like 2-4 emojis per theme placed at 6 hardcoded positions (corners and sides). now every theme has 10-12 emojis and they get randomly scattered across 10 positions. I also changed the website background to use the same background/overlay register page was using.
I will keep adding more themes later!!! Sakura is still my fav one tho!!!
Next Update: Writing freely with given pen OR integration of gifs or emojies!!!

THEME INTEGRATION DEVLOG
HII!!!
in this devlog i added japanese-themed page background named SAKURA AND made the add page button.
the theme were the MOST fun part. theme has three layers,
a body background (the color behind the page),
a page style (the actual notebook paper look),
little emoji in the corners.
the theme also sync in realtime across all users through the page_settings table in supabase. so if someone changes the theme everyone sees it instantly.
I will update more themes later on!!!
also spent way too long on fixing the text going outside the page from the right side. TT
Also, added the change font button!!!