search feature complete, index, quality, and navigation

Before

• ocr text saved to the search index was capped at 8 display keywords per photo

• a digital screenshot with 200 words was only searchable by 8 of them

• multi-word queries like “artificial intelligence” only matched if the exact phrase appeared consecutively in the ocr text

• was no search screen

Now

• a separate ocrFull field stores cleanedText plus the entire keyword list so every word on a digital screenshot is indexed
• _confirmAndSave reads ocrFull into SearchService.indexPhoto instead of the 8-keyword display string
• search() splits the query into individual words and requires every word to appear somewhere in the text in any order
• search is the second tab in the bottom navbar

PS i’m never making mobile app again :”) just testing it takes 15 to 20 min, first you have to transfer it on your phone then run it and all :”( cant run it on emulator cuz it has to take photos from android !

full-text search across all notes - backend and frontend devlog

Before

• ocr text was thrown away after gemini classified each photo
• no index existed anywhere on device to search against
• deletePhoto in storage_service removed the file but left ghost entries nowhere to clean
• clearAll in storage_service wiped prefs but had no knowledge of the search index
• no screen existed to type a query or see results

Now

• SearchService.indexPhoto() saves ocr text + subject under a single ocr_index key in SharedPreferences immediately after every successful save in _confirmAndSave()
• SearchService.search() scans the entire index for a query substring and returns List<SearchResult> sorted alphabetically by subject
• SearchService.removePhoto() is called inside StorageService.deletePhoto() so deleting a photo cleans the index atomically
• SearchService.removeSubject() exists for future subject deletion cleanup
• SearchService.clearIndex() is called inside StorageService.clearAll() so a full app reset leaves no fatherless/orphan(funny word ik) data

sandboxed exec to block code injection in AI cleaner

I wasn’t thinking much about the security, but one kind and cool voter commented on one major security issue of the system so i get to work on it! Now i will work on more security features to make this pipeline secure to use!

Before

• global execution environment allowed full access to the os and subprocess modules

• no import blocking meant __import__('os').system(...) worked

• dunder attributes like __class__ and __builtins__ were reachable from the user interface

• manual edits to the generated code bypassed all internal security constraints

Now

• static ast analysis rejects every import statement and forbidden name branch

• regex pattern scanning catches shell style calls before the parser initializes

• execution globals replace the __builtins__ default with a strict twenty five item whitelist

• validation logic triggers on both the initial generation and the final apply action

AI Assistant tab

Before

• AI Cleaner was inside Clean tab and was easy to miss for users :)
• AI Data Analysis and General Guide were both in Guide tab

Now

• new ai_assistant.py replaces guide.pyand single file owns all three AI and guide sections
• tab renamed to “AI Assistant”
• AI Cleaner moved from Clean tab expander to top of AI Assistant tab
• AI Data Analysis is below it
• General Guide moves to the bottom
• clean.py has the AI Cleaner expander and its import fully removed
• app.py updated with new tab variable, new import
  also tested other features thoroughly so i can ship next

filter and inspect v3 match mode(AND OR) + export

Before

• multiple filters always applied as AND ( And means this and this ,both should be satisified to show this data filtered)
• no way to export filtered results( database peeps will know sometimes its usefull to export stuff after applying sql queries of select where value ==)
• no tooltips explaining what filters do or how match mode works

Now

• ALL (AND) vs ANY (OR) radio toggle added at top with tooltip explaining both modes clearly with examples
• ALL mode narrows down by chaining filters on the result of the previous filter
• ANY mode builds a union mask across all filters applied independently to the original df
• export section added below the dataframe with CSV and Excel download buttons
• export always uses the full filtered result not the 500-row display cap i made for performance
• tooltips added to match mode radio, add filter, clear all, all three metrics, and both export buttons

filter preview v2: multi-filter stacking + UI polish

Before

• only one filter could be active at a time
• no way to narrow down by multiple cols simultaneously
• bw operator was missing so range checks needed two separate filters

Now

• filter rows stored as id list in st.session_state.fp_filters so any number can stack
• each row gets a unique id-prefixed key so streamlit tracks them independently
• x button removes individual rows cleanly without affecting others
• add filter and clear all buttons added below the filter rows
• bw operator added for numeric and datetime with two side-by-side inputs (Min/Max, From/To)
• display capped at 500 rows with caption showing match count for performance

filter preview v1: single column filter

Before

• no way to inspect rows matching a condition without running an operation
• users had to apply a fix and undo just to verify the right rows were targeted
• full dataframe display with no filtering made spotting dirty values tedious

Now

• filter_preview.py added as a read-only module, never writes to current_df
• single filter row with column selector and condition selector
• operators auto-detected per dtype: text gets contains/equals/starts with/ends with/is empty, numeric gets equals/gt/lt/is empty, datetime gets equals/before/after/is empty
• match rate metrics row shows matching rows, total rows, and %age
• filtered dataframe renders below with use_container_width=True
• made it as a separate tab after overview so user can go to it to check the data

col_popover selection reset bug

Before

• The columns user selected before get reset when user switch to other tab thus when he came back to the tab again, he had to deselect and select columns again for operations :)
• on_change callbacks were the only sync mechanism between checkbox keys and val_selected
• after a run, val_selected.pop() cleared the list but _vc_section_col keys stayed True in session state
• streamlit never fired on_change again because widget value had not changed from its perspective

Now

• col_popover reads _vc_ keys directly at render time and rebuilds val_selected from scratch every render
• _make_all_handler now resets all individual _vc_ keys when toggling all on or off so checkboxes visually match
• _make_col_handler syncs the all checkbox when individual columns are toggled

The issues i got while doing this:

• col_popover was unconditionally rebuilding current from vc keys on every render. In streamlit, all tab contents execute on every rerun even when the tab isn’t visible (mess i know) .
• so i had to fix this also , and i used this flow
• If val_selected[section] already exists -> it was set by an on_change handler which is the definitive user action. Trust it, sync vc keys to match it for display, done.
• If val_selected[section] is missing -> means either first ever render, or clear_popover just wiped it. Only then fall back to reading vc keys.

I say this earlier and will say it again i hate how streamlit works, worst framework ever, i might change to some other framework after this ship.

button and popover alignment fix

My buttons were kind of inconsistent for the recommendations and validate tab. My focus before was never UI so i didn’t bother changing them. But now, i think i should change them and make them consistent so here i’m

Before

• Fix button and select columns popover rendered inside the same st.columns block as description txt
• st.write("") spacers before buttons were unreliable and column heights varied per section thus making the UI looks inconsistent
• recommendations and validate both had this problem..

Now

• separate st.columns call handles only the popover and button on their own dedicated row
• both widgets are always the same height so they always land on the same line
• button click logic moved outside col blocks so st.rerun() fires cleanly after all columns close

Excecutable Validation JSON feature

Before

• import_pipeline_json had no branches for any custom validation step
• JSON preview only existed on the save side as a raw code dump
• Uploaded JSON had zero preview, and also there was no toast on successful json application TT

Now

• _parse_label_meta() moved to module level so both import_pipeline_json and build_pipeline_script share it
• Each step checks col in tmp.columns before running and skips if column is missing
• threshold, min, max are safely cast to float with fallback so bad values never crash
• _friendly_label() added in history_export.py
• _render_json_preview() added, shows numbered step list with easy to read labels instead of raw JSON so non tech also get it
• uploaded pipeline auto-expands its preview so user sees steps before hitting Apply

Executable Validation Python code Export

(this was to be done for the custom validators i made in prev devlog)

Before

• Custom Validation steps exported was not implemented yet
• Custom regex, country code, date format, thresholds all were not working
• Type Override had parts.strip() bug on a list :))

Now

• _parse_label_meta() reads key=value pairs embedded in label strings
• all 5 validation steps export real runnable code with exact user settings
• each commit_history call embeds col + settings per column
• bug fixed, stubs added for Split/Merge/Rename :”)
  next i will have to make the custom validators work for pipeline.json also

Global Undo and Redo in the Sidebar

Before

• Undo and Redo were only inside History and Export, and this user has to switch tabs and couldnt access them easily
• render_sidebar() had no access to history or redo_stack
• Sidebar bottom stayed unused in Simple mode

Now

• render_undo_redo() added in state.py and called in both sidebar modes
• Reads from st.session_state, returns early means buttons will show even if no undo and redo has been performed so user know in the start that side bar has this feature :)
• Click triggers toast + st.rerun() for full UI sync
• st.caption() shows next undo/redo actions. so user know what is happening and also user will know what will happen if they press UNDO or REDO!

Custom Inputs for Date, Email, and Phone Validators

Now users can provide specific formats, patterns or country codes to bypass general pattern

Before

• validate_date_col() in validate.py relied on a 14-format loop and C-level inference, which could be slow or misinterpret ambiguous dates (eg 01/02/03).

• validate_email_col() used a hardcoded regex pattern that couldnt be overridden for specific corporate domains or stricter requirements.

• validate_phone_col() was hardcoded to US standards, making it difficult to validate international numbers

Now

• validate_date_col accepts a custom_input_format parameter. If no custom input format is set, then it pass the default US format

• validate_email_col features a custom_pattern parameter. A ternary operator replaces the default regex if a custom one is provided. A new try/except block catches invalid regex syntax

• validate_phone_col now accepts default_country_code. The logic strips leading + signs and uses dynamic boolean masks for country code and length.

UI

• the tooltip for the functions explains the user about what to write in this custom pattern and how so they get how to use this feature

Add AI Executive Summary to PDF Report

Now user will get AI summary in PDF report also
Technical details:

Before

• build_report_pdf() in ‘reporting.py’ accepted only ‘original_df’, ‘cleaned_df’, history, and filename with no Ai summary abt the dataset
• The report opened with all the technical details, and a non-technical person couldn’t make sense of what even the data is abt
• also the AI summary string already existed in st.session_state under ai_insights_{file_id} after the user clicked Generate AI Summary or Analyse my data in the Guide tab,
• ‘history_export.py’ called build_report_pdf() with four positional args and moved on

Now

• build_report_pdf() accepts an optional ai_summary: str = "" parameter
• when ai_summary is non-empty, a light blue tinted LIGHT_BLUE background panel is rendered immediately below the title block and says run ai summary first to get the ai summary in the pdf
• history_export.py reads st.session_state.get(f"ai_insights_{file_id}", {}) before calling the builder, pulls .get(“summary”, “”), and passes it through

Fix: Vectorized Phone and Date Validators

Before

• validate_phone_col and validate_date_col both used .apply() to run a Python function row by row having HUGE waiting time
• validate_date_col ran every row through a 14-format try/except loop even when pd.to_datetime would have parsed it fine in one go!

Now

• validate_phone_col is fully vectorized: .str.replace(r"\D", "", regex=True) strips non-digits across the whole column in one C-level pass, then .str.len() and boolean masks build the +1, + prefixes with pd.Series.where
• validate_date_col uses a two-pass strategy: first pass runs pd.to_datetime(..., infer_datetime_format=True) which handles the vast majority of dates at C speed in one go then second pass only touches the rows that failed and tries the explicit format list….so on clean data the slow path never runs at all

Also change the toast message and now it shows this ✓ along with the message!

Fix: Recommendations Tab State Wipe, Duration Regex False Positives

Before

• fixing any issue in the recommendations tab wiped the entire tab state and dropped the user back to the “Scan for Issues” screen bcz scan_key was f"rec_scanned_{df_key}" and df_key is derived from make_df_key(current_df), which changes every time a fix mutates the dataframe, so the old key simply stopped existing in session state
• the duration regex (h|hr|hour|min|minute|sec|second) had no word boundaries, so the bare h matched the letter h inside any word like John”, “the”, any english text at all would fire it and flag name or description columns as duration columns (found this one while testing on a movies dataset)

Now

• scan_key is now a stable hardcoded string "rec_scanned" with no dependency on df_key, so it survives reruns after fixes; state.py explicitly resets it to False on every new file load and fresh session so it never carries over across files
• duration regex tightened to \b(hrs?|hours?|mins?|minutes?|secs?|seconds?)\b with word boundaries in both get_analysis_and_recommendations and get_type_suggestions in cache.py; bare h removed entirely since no real duration data is ever just the letter h standing alone

also i didn’t know we can use ‘’ to show the code things!!! now i am gonna use them more!

Fix: Session Resume Dialog Crash and Loading Spinner

Before

• clicking outside the resume dialog closed it without setting _resume_choice, leaving init_state in a broken half-initialized state and crashing on the next render

• the loading spinner and info message never actually displayed because init_state ran synchronously on the same render pass, so the spinner appeared and vanished :) (handling edge cases are really tough)

Now

• dismissible=False added to the dialog, disabling backdrop click and removing the X button so the user is forced to pick an option (read this thing in human computer interaction course)

• loading is now two-pass: first rerun sets _resume_loading and renders the spinner, second rerun clears the flag and runs init_state normally, making the spinner genuinely visible between reruns

Fix: Recommendations Feedback and Overview Slider

Before

• clicking Fix or Auto-Fix All in the recommendations tab gave no feedback :)

• the overview tab crashed on single-row files with “Slider min_value must be less than max_value” when both values resolved to 1

Now

• every Fix button in recommendations now wraps its operation in a spinner and fires a toast on success matching the exact same pattern used in the validate tab that i did in last devlogs

• Auto-Fix All also shows a spinner and toasts on completion

• the overview slider now guards against min equal to max by skipping the slider entirely on single-row files and always flooring the minimum at 1

Note:

 Also this took me 45 minute to do, but for some reason my devlog didnt got uploaded and i saw that just now :) so

Performance improvement: AI Insights and Page Load Fix

so first thing happen when user opened my app was to load the ai analysis overview and ai guide, thus user have to wait for 10 seconds for small file max to load all the tabs :)
now all tabs load instantly because now i have button buttons that user has to press to get the ai overview and guide!

Before:

• opening the app with a 50k row file meant waiting 10 to 20 seconds before any tab appeared
• guide.py called get_ai_insights automatically on every render which blocked the entire page
• the overview tab had a separate streaming approach using st.write_stream which returned None in most streamlit versions so the summary never showed at all
• also there were no cache that could be stored for one file so gemini don’t have to be called for it again and again on every reload/reupload of the file

Now:

• both tabs share the exact same cache key ai_insights_file_id so one api call populates both
• neither tab calls the api automatically on page load, the user triggers it with a button when they want it
• guide tab shows an Analyse my data button and overview tab shows a Generate AI Summary button
• whichever button the user clicks first, both tabs show their content on the next rerun with zero extra api calls
• st.write_stream and the entire streaming approach were removed since streamlit tabs buffer all output and flush at once, streaming never worked there
• ai_insights.py went from 310 lines to 190 by removing the dead streaming code
• page now loads instantly regardless of file size, gemini is never called until the user asks for it

UI Feedback - Spinners, Toasts, and the Dialog Bug

so in this devlog, i focused on making the app feel responsive :)

Before:

• when a user clicked any button, the app would run the operation, call st.rerun(), and then show a success banner on the next render
• this was handled using _omsg in session state across clean.py and validate.py
• the success message had no auto-dismiss, so it just stayed there until the next interaction
• sometimes the message didn’t even show, depending on which tab rendered first
• there was no loading feedback, so the app felt frozen during heavy operations
• validate.py had zero spinners at all, meaning everything just paused WITHOUT even notifiying user
• spinner messages (where present) were vague like “Processing…” with no real context
• redo / history-related feedback existed but didn’t clearly communicate what was happening

Now:

• replaced the entire _omsg pattern with a cleaner toast-based system
• new flow: set st.session_state[“_toast”] = (msg, icon) -> call st.rerun() -> pop it at render -> show st.toast()
• toasts automatically disappear after ~3 seconds, so no more stuck messages
• they appear in the corner and don’t mess with layout at all (way cleaner UX)
• users now immediately see feedback like “please wait…” instead of thinking the app died
• instead of “Stripping whitespace…”, it now says things like:
  “Scanning 50,000 rows and stripping whitespace…”
• fixed the dialog bug where “Resume previous session?” wouldn’t close
• issue was st.rerun() inside @st.dialog only rerunning the fragment, fixed with: st.rerun(scope=“app”)
• also cleaned up history_export.py removed old show_msg() + last_success_msg usage
• migrated everything to toast systemSome of the functions still have poor feedback :)
  this stuff is really so annoying, esp streamlit, ughh!!! idk whether to improve user feedback or performance or add new functions.

HII!!

History tab UI improvements

So in this devlog, I improved the History tab by adding visual feedback for all operations and making the redo stack visible!

Before:

• when a user clicked undo or redo, or Reset Data, Clear History, Apply Pipeline, or Generate Report, the app froze for a moment and refreshed giving lil hint that operation successful.

• there was no visual feedback or spinner showing that the app was actually processing the request.

• also the user could see their history of past actions but they had no idea what was inside the redo stack waiting to be redone.

• some buttons didn’t even show a green success box after finishing!

Now:

• I wrapped all the major operations inside history_export.py (like undo and redo) with with st.spinner(“… please wait…”): blocks.

• now when a user clicks any button in the history tab they get an immediate spinning message telling them exactly what is happening.

• I also imported show_msg from state.py so every single operation assigns a message to st.session_state.last_success_msg before it calls st.rerun().

• now the user always gets a green success confirmation box once the spinner finishes!

• added a new list right below the main history list that loops through the redo_stack.

• now if the user undoes an action it pops up in the Available to Redo list so they know exactly what will happen if they click the Redo button.

next,i dont know what needs improvement, but i might focus more on User feedback :)

HII!!

Redo operation

So in this devlog, I integrated the redo operation into the history system!
Before:

• the undo history lived in st.session_state and got saved to the .dp_sessions folder, but if a user accidentally clicked undo, they lost that cleaned state forever
• The app could only go backwards, not forwards :”)
• also when a user clicked undo on a large dataset the app just paused for a second and refreshed. there was no visual feedback that the pipeline was working on reverting the data
  .
  Now:
• I added a redo stack to the pipeline. when a user clicks undo, the current dataframe it gets pushed onto a redo stack. if they click redo, it pops from that stack and goes back into the main history.
• whenever the user makes a brand new cleaning change, the redo stack clears itself because the timeline has changed. I updated session_persist.py so the redo stack is saved to the disk and survives page reloads
  .
• UI changes: wrapped the undo and redo functions inside st.spinner blocks inside the history export tab
• now when the user clicks either button they get an immediate spinning message saying Undoing last action please wait or Redoing action please wait
• added a new list right below the main history list that loops through the redo_stack. now if the user undoes an action it pops up in the Available to Redo list so they know exactly what will happen if they click the Redo button

next i will work on making the user feedback for the history tab more better! ik right now it sucks, but dont worry i will improve it!

Valid tab: UI Improvements

• Now each function gives clear feedback
• eg the email validator tab will say X number of emails has been flagged or removed
• same with the phone validator or standardazing the dates
• i didnt had to fix the UI for outlier cap and validate range function as i already have done that before

Clean Tab: Performance and UI Improvements

voters said before that my app sucks in feedback so now im trying to make it more better
first of all, i made these changes in clean tab

• Every operation now calculates and reports the actual difference.

• Strip whitespace counts changed cells per column.

• Duplicate drop shows the percentage of data removed.

• Smart cleaner lists the names of converted columns.

• Missing value handler reports filled counts and dropped columns separately.

• Type override warns when conversion failures create new nulls.

• Find and replace reports the exact match count.

• Outlier capping reports the exact number of values affected.

• Type guesser adds a per-column progress bar.

• Type guesser now runs via button instead of opening automatically for better performance

Next i will update the validate tab

Upload Feedback Fix- Small fix

Before:

• user picked a file and saw nothing. no spinner, no message, nothing.
• the upload tab showed the success message only after everything was done
• so on Excel files with large sheets there was a silent 5 to 10 second gap where the app looked frozen and the user had no idea if the upload worked. some users switched tabs thinking nothing happened,

Now:

• upload.py checks loaded_file_id against the current uploaded file_id

• if they do not match the file was just picked but not yet parsed so it immediately shows Reading filename please wait in blue

• this renders on the very first rerun after file selection before app.py has done any work at all

• once init_state finishes and sets loaded_file_id the next rerun flips the message to the green success confirmation

• user always sees feedback within one rerun of picking the file

One user complained that they often had to upload the file twice, now they will get clear feedback and their file will quickly upload

Lazy Tab Rendering

Before:

• all tabs blocked on page load for 30 to 40 seconds on large files.
• get_analysis_and_recommendations ran unconditionally at the top of recommendations.py
• the correlation heatmap ran inside a closed expander on every render pass
• streamlit executes everything inside expanders even when they are collapsed , meaning expensive functions was being ran even when user dont even need them :(

The cache.py :”) :

• spent 30 minutes adding sampling and large file guards to every analysis function in cache.py thinking that would fix the slowness. it helped memory and accuracy in large files but did not fix the tab loading delay at all
• problem was never the speed of the functions. it was that they were being called in the first place on every render even when the user had not asked for them.
• the real fix had nothing to do with cache.py, i wasted my time working with cache.py tbh!

Now:

• recommendations are behind a Scan for Issues button, and tab opens instantly.
• scan results are cached in session state keyed on df_key so they survive reruns
• but auto invalidate when the dataframe changes after a cleaning step.
• correlation heatmap is behind a Compute Correlation button inside its expander.
• same df_key keying pattern so results persist until data actually changes.
• clean, validate, and history tabs were already instant and needed no changes.
• on a 1 million row file all tabs now open after some wait(better than before tho).

Also now only the history tab is still stuck with only 2 functions, there is a bug thats keeping it from showing other functions instantly, i will work on that in next one, but other all tabs are fine, i know speed is a issue but for 10k rows , i can only do this much now :(
Edit: Ai also gave its summary after 2 min TT big wait time, but its a given for 10k rows :)) maybe one day i might become a pro and improve its performance, but to me performance is really hard to solve rn

Large File Support

Before:

• everything assumed small datasets. pd.read_csv loaded the entire file into memory at once.
• KNN and MICE imputers were called on 500k row files which would freezes or crashes the browser, also all analysis functions scanned every row every time.

Now:

• CSV files are read in 50k row chunks and concatenated so peak memory stays low
• int64 columns are downcast to int32 and float64 to float32 on files over 50k rows
• this cuts memory roughly in half with a warning shown to the user about precision loss
• KNN and MICE imputers now have a row limit, above 50k rows they fall back to mean imputation which runs in linear time and is safe for 500k rows(found this info from Medium article )
• all expensive analysis functions like recommendations, correlation, quality score and type suggestions now run on a 20k row random sample on large files
• histogram KDE is computed on a 10k sample while counts still use the full series
• overview tab Now shows a warning banner when downcasting was applied
• row count metrics now use comma formatting so 100000 renders as 100,000( yey kinda low level detail TT )

Note:

• In the uploaded video, the data get loaded really quickly but the tabs show stuff really late

• it took 30 to 40 second for that it was because of heat correlation and expensive analysis functions. In next update,i will try that only the tab with expensive functions slow down but other tabs should show up like the validate and clean tab as they really don’t have any stuff that should pre run

• Also i attached a screenshot, check it out, my app also loaded a

1 MILLION rows TT

yay i had to write in the bold, also tho its a bit slow but I can always work on this part <3

Persistent Session History

Before:

• undo history lived only in st.session_state, a page refresh and everything is REMOVED
• users lost all cleaning progress on reload which was evil ik

Now:

• added session_persist.py as a standalone persistence module
• history and both dataframes are pickled to home dir under .dp_sessions folder
• sessions are keyed by md5 of filename plus filesize, NOT streamlit file_id
• this was the core bug, file_id is a fresh UUID on every reload so the session was never found
• sessions older than 2 days are auto cleaned on each init
• commit_history and undo_last both autosave to disk after every mutation
• on file load init_state checks for a saved session before doing anything
• if a saved session exists a native st.dialog pops up showing time since save and step count
• user picks continue to restore full history and current df or start fresh to wipe and begin clean
• reset to original and clear history both delete the persisted file

How tough it was:
YES I NEED TO TELL THIS!!!
3 issues no 4:

• first file_id changing every reload so session was never found
• second windows path separator making tmp dir fail silently
• third st.stop halting before current_df was set so the page just crashed quietly
• fourth pycache serving the old pipeline.py so autosave never fired at all
  Literally tried so many methods to get this work TT i hate how session stuff works :(

Two updates and longer devlog cuz HACKATIME was down :((

AI Cleaner Improvements

Before:

Gemini blindly generated code without checking if the data was ready. dirty columns (currency symbols, % signs, units) caused runtime crashes without having no USER FEEDBACK :”)

NOw:

• Gemini gets raw sample values per column
• added feasibility check , now gemini reasons before writing any code
• if app already has a feature for it, Gemini says go to exact tab + feature as its a guarenteed method to work , but it also generates code incase user want to run this
• if column is dirty and operation assumes clean data , then a warning is shown to fix the column first with the right feature, but code is still generated if user wants to run it anyway
• gemini timeout bumped to 30s and it auto-retries once on timeout

Performance Improvements

Before:

• st.cache_data was hashing the ENTIRE dataframe on every cached function call
• with 7 cached functions each receiving the full df, thats 7 full df hashes per rerun.
• also st.cache_data.clear() was nuking ALL cache on every new file upload.
• profile tab was computing correlation, missing heatmap, and before/after on every tab switch.
• result: 20 to 30s load times even on datasets under 1000 rows :”)

Now:

• added make_df_key() in cache.py
  all 7 cached functions now take df_key as first arg instead of the full df

• Now we have 3 phases workflow
• Phase 1 (~instant): All 8 tabs render immediately. user sees the full app right away
• Phase 2 (~2-10s): app.py calls Gemini after all tabs are rendered, before gemini was calling before the loading of all the app which was causing the issue
• Phase 3 (instant): st.rerun() fires, all tabs re-render with the cached insights already in session state

HI!
In this devlog i moved and upgraded the AI analysis feature

AI analysis moved to Guide tab

• before this the AI fixes were showing inside the Recommendations tab
• it felt cluttered there because recommendations already has its own rule-based list
• moved the whole AI analysis into the Guide tab
• Guide tab is where users go to understand what to do, so AI fits there ig

what the Guide tab AI section now includes

• a plain English summary of what the dataset is about
• a list of prioritised fix cards, most important first
• each card shows the issue title, which column it affects, and why it matters
• each card shows exactly which tab to go to and what to click to fix it
• each card shows an AI Cleaner shortcut so the user can just type the fix instead
• the shortcut tells them exactly what to type into the AI Cleaner in the Clean tab
• the cards use real tab names and real button names from the actual UI
• styled with a blue border for the how to fix section and green for the AI shortcut

what the Overview tab now includes

• AI summary is also shown at the top of Overview as a duplicate
• it uses the same cached result so it costs zero extra API calls
• non-tech users see the summary immediately when they open Overview

how caching works

• Gemini is called once per file load
• result is stored in session state under ai insights file id
• Guide tab and Overview tab both read from the same cache key
• new file uploaded means a fresh call, same file means instant load

also

It works for the multi-sheet excels also! but you have to select the Sheet you want to get AI guide on!!!

HI!
In this devlog i built

AI data summary and AI fix suggester

• one Gemini API call does two jobs at the same time
• first job is a plain English summary of what the dataset looks like
• second job is a prioritised list of fixes the AI recommends
• both come back in one JSON object so we only spend one request per file
• the result is cached in session_state keyed to the file id
• this means even if the user switches between Overview and Recommendations tabs the call only happens once
• new file uploaded means new call, same file means cached result

how the summary works

• shows up at the top of the Overview tab
• 2 to 4 plain English sentences about what the data is and its quality
• non-tech users immediately understand what they are looking at

how the fix suggester works

• shows up at the top of the Recommendations tab above the existing rule-based ones
• each fix has a priority number, a column name, a short issue title, and a one-sentence reason
• sorted by priority so the most important thing is always first
• styled as cards with a blue left border and priority number

HI!
In this devlog i built

AI natural language cleaner

• user types a plain English instruction like drop rows where Age is empty
• app sends the instruction plus column names, types, and 5 sample rows to Gemini
• prompt asks Gemini to return a JSON object with two fields, code and explanation
• so i used gemini api key and made this ai natural language cleaner
• using gemini-3.1-flash-lite-preview, fast and cheap and allows 500 requests per day
• temperature set to 0.1 so output is consistent and not creative
• maxOutputTokens set to 1024 to support multi-line code for complex queries
• Gemini can write code of as many lines as needed
• this means complex instructions like normalise then fill then rename all work in one go
• code field is the pandas code to run
• explanation field is a plain English sentence of what the code will do
• app shows the explanation with “Verify that this matches what you want” so users know what will happen once they apply this code
• app shows the code in an editable text area so tech users can changed it if needed
• user clicks Apply to run it, or Cancel to discard
• code runs in an isolated exec() with only df and pd available, nothing else
• if execution fails the error is shown and the dataframe is not changed
• on success the result goes into current_df and gets logged to History
• API key is loaded from a .env file using python-dotenv
• if the key is missing a warning is shown and the section is hidden

Guide ME feature
I literally tried three approaches before landing on this one.

• the first attempt used a fixed-position HTML card with Streamlit buttons rendered below it, relying on CSS position:fixed !important to reposition the button row to visually overlap the card. this failed because Streamlit’s component tree does not guarantee DOM order or stable class names across rerenders, so the CSS selectors targeting the button row were unreliable and the buttons kept appearing at the bottom of whichever tab was open.
• the second attempt used links styled as buttons inside the HTML card, with a handler at the top of render that read st.query_params and updated state. this seemed promising but Streamlit treats query param URL changes as full page navigations, not rerenders. clicking Next reloaded the entire app from scratch and cleared the uploaded file.
• then finally, added a Guide tab to the data cleaning app
  its a static checklist ,it only exports one function: render(tab).
  it takes the Streamlit tab object and renders inside it.
  in app.py, add a Guide tab to the st.tabs call and pass it to guide.render().
  this was the easiest and is done

What i wanted: was a cool guide feature that shows in bottom right and guide users like how many software works
but it was nearly impossible in streamlit for me tho :”) so i ended up making just a guide me tab :)
pushed both files on github tho so no onne can say how can u spend 1 hr 30 min on just one tab that shows only static data TT

Hi! in this devlog i built
data quality score

• new score from 0 to 100 shown at the top of the overview tab, computed across five dimensions each worth 20 points
• completeness penalises missing cells proportionally, so a dataset with 10% missing values loses about 2 points on that dimension
• uniqueness penalises duplicate rows the same way
  type consistency scans every object column and checks what fraction of values are actually numeric strings stored as text. the more columns doing this the lower the score
• outlier cleanliness uses a 3x IQR fence instead of the std 1.5x so only extreme values count against the score, not natural spread( i hate this probabilty stuff but its so imp for data science and ML TT)
• validity checks text columns for common placeholder strings like none, na, n/a, null, unknown, and empty strings
  , its colour coded green above 80, amber( yes we all think of that amber :”) ) between 55 and 80, and red below 55
• five breakdown cards sit below the gauge, one per dimension, each showing the raw score out of 20, the grade, and a plain english explanation of what was found
• the whole thing runs through @st.cache_data so it recomputes automatically whenever the dataframe changes and the user sees the score tick up in real time as they clean

Hi,in this devlog, i did

cleaning report PDF export

• added new reporting.py module that builds the entire PDF using reportlab
• NOW report includes a summary table comparing rows, columns, missing cells, and duplicate rows before and after cleaning with a change column
• full column profile for both the original and cleaned dataset showing type, null count, null percentage, and unique value count
• missing value breakdown table that lists every column that had nulls before or after, and flags whether it was fully resolved
• ordered list of every cleaning step performed with the dataframe shape after each one so you can trace exactly what changed when
• first 10 rows of the cleaned dataset as a sample table at the end
• PDF regenerates automatically only when the history changes, using the same history_len cache key pattern already used for the Excel download
• reportlab added to requirements.txt. so if the import fails the UI shows a pip install hint instead of crashing the whole app

HI!!!
In this devlog i did!

Data Type Guesser

• new section at bottom of clean tab that scans every column and suggests the correct type based on what the values actually contain
• can detects nine patterns: email addresses, boolean values, currency amounts, %ages, measurement units, time durations, datetimes, plain numeric strings, and low cardinality categories
• each suggestion shows the column name, current type, suggested type, a confidence percentage, the reason it was flagged, and three sample values so u can judge before applying
• rendered as an editable table where you tick which suggestions to apply and press one button, so you never have to go column by column manually
• also the confidence is computed from the proportion of values that matched the pattern, so a column where 95% of values look like currency scores higher than one where 60% do

Pipeline JSON Save and Reload

• cleaning history now exports to a small pipeline.json that stores step labels only, not data

• upload that file on any new dataset and THEE app replays every automatable step in order

• Steps that need manual column selection like email validation or outlier capping are skipped and reported back honestly

• Every replayed step is pushed onto the undo stack so the full workflow stays consistent

Correlation Heatmap

• Added to the Profile tab between the missing value heatmap and the before/after comparison

• Symmetric grid where each cell is colour encoded, blue for positive correlation, red for negative, white near zero

• Value labels are drawn inside cells when there are 12 or fewer columns, hidden automatically on wide datasets so the chart stays readable

• Supports Pearson, Spearman, and Kendall methods with a dropdown to switch between them

• A summary line below the chart surfaces the three strongest off-diagonal pairs so you can spot redundant columns at a glance without reading the whole grid

• All computation runs through @st.cache_data so switching methods is instant with no recompute

Also this is only available in ADVANCED mode, don’t want to scare off the users who don;t know much!!

Data Cleaning Pipeline v3.0.0

• I figured out i should do devlogs like this from now on instead of doing long para as these are more readable

Modular Refactor

• Broke a 1800 line single file 19 focused files across a clean folder structure. For me, understanding the code before was easy because streamlit has so less code to do but for reviewers it might not be easy so !
• cleaning/ folder is now Python with zero Streamlit dependency so every function is independently testable
• Each tab gets its own file with a single render() function, app.py is now just 106

Distribution Charts

• Added histogram with KDE curve overlay to the Profile tab for numeric columns
• IQR fence lines drawn directly on the histogram so u can visually judge outliers before deciding to cap them
  categorical columns get a horizontal frequency bar chart with percentage tooltips
• missing value heatmap shows a sampled grid of the whole dataset, red cells are nulls, blue are present
• All chart data goes through @st.cache_data so switching columns has no recompute cost

Column Transforms

• Split column breaks one column into many on any delimiter, extra parts pad with NaN, source column optionally kept
• merge columns concatenates any number of columns into one with a custom separator
• rename columns gives an editable table so you can fix all names in one session without touching code

HI!
in this devlog i built the boot animation.set up array of boot steps where each step has a message, a progress percentage, and a delay in ms. a recursive setTimeout function go thro them one by one updating the loading bar and status text. the start logo spins using a CSS keyframe animation. when all steps get finished it hides the boot screen and shows the login page. restart also goes through the full boot sequence!!!

Splash screen, UI and some fixes!
Hi! In this devlog I improved UI of LectureVault.
I built an animated splash screen!!! Also made logo for LV which is still so meh but i got no artistic sense so :”)
Fixed all the dim buttons across the app. Bottom nav icons, filter pills, sort button, the Change storage location link in settings, Move and Delete in the selection bar, and all dialog action buttons are now solid vivid colors!! because voters before complained my buttons aren’t good so :) before i just made this to just get its functionality right, i didnt bother much on UI but many of voters said my UI is trash so imma fix it now!!!

HI!!!
in this devlog i built the boot and login system
the page now starts with a login screen instead of jumping straight to the desktop
built a tryLogin function that checks the credentials and either dismisses the screen or shakes the form and shows an error. the shutdown button triggers a black overlay saying restarting, user has to reload to get back in. restart does the same black screen but after 2.5 seconds it would) automatically brings the login screen back. empty fields also count as valid so you can just hit enter and get in quickly.

HI!!!
in this devlog i rebuilt the window management system
the old code just toggled a class and had no concept of which window was on top.
so, i replaced it with a z-index counter that increments every time a window is focused so the last clicked window is always on top. minimize now hides the window but keeps the taskbar button visible and grayed out. clicking that grayed button restores the window to its exact last position.
the taskbar button also now correctly toggles!!!!

hi!! in this devlog i setup2 the start menu for PixelOS.
so the structure was already there from last session so this was mostly about getting the interactions right. clicking the start button toggles the menu open and closed and adds a glow to the button so you know its active. clicking any app in the menu opens that window and closes the menu at the same time. i also added a listener on the desktop so clicking anywhere outside the menu closes it. the shut down and restart buttons trigger a toast notification saying restarting and shutting down.
I also changed the wallpaper!

Browser window
hi!! in this devlog i built the Browser window for PixelOS!
i set up a pages object where each key is a pixel url and value is a function that returns html.
this made it really easy to add new pages later on… i also built a proper browser history array with a current index ptr so back and forward work correctly. clicking links inside pages navigates too.

hi!!
in this devlog i built the text editor window for PixelOS!
double clicking a txt file now opens it with the content loaded instead of browser alert that i had set before.
i wrote an openEditor function that takes a file object and populates the textarea. also added live word and character counting that updates on every keystroke, and replaced all alerts with a proper toast notification. the save button and new button r also working!

hi!!
in this devlog i built the file explorer window for PixelOS!
i set up a fake file system as a javascript object with folders and files nested inside each other. clicking the desktop icon or taskbar button opens the window, and you can double click folders to go inside them. i also built a navigation history stack so the back button remembers where you came from. dragging the window also works!!!

hi!!
in this devlog i set up the static website for PixelOS! Just put static icons there for now!well, took me alot of time searching for right wallpaper and icons TT and then i also had to remove the backgrounds of icon using canva :”) wallpaper is in png while all icons are in svg format!
for now, i styled the taskbar with the start button, app shortcuts, and a live clock!!!
i will work in increements and set more things in my webos!

Hi!
In this devlog I rebuilt the diagnosis engine in MediScribe AI.
The system worked before, but results were kinda unreliable. confidence scores were wrong and way too less.

First, I removed a penalty bug. Normal findings were reducing scores for real diagnoses.
second, I added support for required facts. Now a diagnosis only appears if relevant sections have data.
third, I fixed score calculation. All diagnoses now use one clean formula. Scores stay between 1% and 95%.
fourth, I fixed deselection issues. Old answers from removed followups no longer affect results.
fifth, I unified logic. SOAP and diagnosis screens now use the same scoring system.

Please check the whole readme and demo before voting, i know many of you wont get this project but if u research on it then its really a cool thing to make :”)

NOTE: I rushed alot in demo because FT dont allow long videos :”)
you can watch full demo on this link:

https://drive.google.com/file/d/13VzI_spU9x2eqakv_oEsoGAXvFYAF1iM/view?usp=sharing

hi!!
in this devlog, i did nothing much but find an UI mishap!
the user can only save the records once he is on last screen of history taking
so i made a bottom draft widget that will be shown in every screen of history taking and it says save draft and exit( means go back to homescreen).
this way user won’t have to fill in every steps and can just safe draft!
then i also made my code modular by adding a patient repositary service!

Drawing Canvas Improvements
hi!
in this devlog, i added an eraser, a special pen (filling pen), and an undo button to the shared notebook drawing tools.
you can now erase mistakes, or use the special pen to fill.for this purpose, i just added a local array to track my stroke ids for the undo button and encoded the new tools directly into the stroke color strings (like sending “ERASER” or appending “_FILL” to the hex) so we didnt even have to change the supabase database schema.
Also, playing around with the special pen(filling pen) is so fun, check it out, i found it way more cool than bucket feature in paint, thats why i build like this

Offline Storage & Full Edit Flow
Hi! In this devlog I added offline storage and a complete edit flow to MediScribe AI.
I integrated Hive as the local database, every patient session is now saved as a single object containing all seven data layers: patient info, history, systemic review, vitals, labs, examination findings, and the generated SOAP note.
The Patient Records screen now reads live from Hive using a ValueListenableBuilder… so the list refreshes instantly after saving or deleting.
Tapping any patient record opens a full detail view showing everything entered across all screens. The edit flow now pre-fills every screen. Re-saving overwrites the original record cleanly.
The SOAP screen’s bottom button now saves and navigates home in one tap.
Next feature will be integrating neural network models to give more precise diagnosis based on all history taking

Sorry kinda long devlog but no worries i still document everything in this devlog :)
Hi!!
in this devlog, all i did was to suffer :”))

so i try adding a catch block in upload screen that catches gemini error like reaching the api limit!
but implementing this feature broke my all flow, try to fixed it many times but no vain, so i had to switch to prev commit code :)
also there was the error in toast message of undoing the delete action, it followed user everywhere so i fixed it by clearing the toast message while moving to another activity!
Also, I added the move photos functionality in other folders also, user now can move images among folders
also updated the readme!
moreover, user now cant delete the folder named unclassified cuz obv unclassified images need to go there!
also, image was’t loading sometimes!
Android returns content:// URIs from the image picker, and Image.file() cant read those as it only works with real paths.
so i cache the image bytes immediately after picking (while the URI permission is still active) using XFile.readAsBytes() then i render with Image.memory() instead.

Also good news
before i was using gemini model 2.5 flash lite and it was only 20 requests per day, but now i find out about 3.1 flast lite model and it can do 500 requests per day
so i am using it right now
also, my app send 1 request for 20 pics in one session, so i am saving alot of gemini credit this way!! so you can use application in peace cuz
if u upload 20 images per session, then we classify 1000 pics a day!!!

also, if someone dont wanna worry abt it at all! so he/she can just clone my repo and put their own gemini key in .env file. but i dont think so you need to do this

demo video is attached, i can assure you images i classified in app is 100% correct :”)

Pinch to zoom feature and some UI improvements!
Hi! In this devlog I fixed and added a few things.

i added unclassified folder that gets created automatically during onboarding setup so photos that the AI cannot confidently sort can go there

made delete folder flow so users now get a choice when deleting. They can either remove the folder from the app only, keeping all their files on the device, or delete everything including the physical folder and photos inside it.

Improved the pinch to zoom in the image viewer.
Replaced the keep original copy toggle in settings with a clean info card explaining that my app never touch the camera roll.

also i attached the video testing every feature, check it out!

debugging and debugging and image view feature minimal implementation

Hi! In this devlog I fixed several issues

first issue was the app kept showing the onboarding screen every time it launched even after setup was done. Fixed it by calling markSetupDone() at the end of the folder setup flow so the app now remembers that setup was completed.
Second was the Reset App button in settings was not properly clearing the setup flag. so i fixed it by replacing the individual save calls with clearAll() which wipes everything including the setup done flag.
third was the classification settings toggles in settings were hardcoded before. so i make it save to SharedPreferences and reading them back on load, then wiring each toggle into the actual upload logic
fourth was deleting a folder from the folder view screen did nothing. Fixed it by implementing the delete folder flow properly with a warning dialog that tells you that folder will be deleted from the app but wont be deleted from the device.

moreover, i added a new image viewer screen that opens when you tap on a photo in the folder view. It supports swiping between photos, I made it so that user can see the actual image we are classifying right now
I also removed the About section in history as i really dont need it!
also, i added a create and delete folder option in the homepage also, also fixed navbar issues :)
in the next devlog, i will implement zoom logic more correctly and also will implement sharing fuctionality of pictures!!!
ANd will remove the redundant icons i placed before :”)

OCR + Gemini Classification
hi!
in this devlog, i got the full AI classification pipeline working. here’s how it actually went

the OCR part was kinda smooth. i used Google ML Kit’s text recognizer and ran it locally on each image, so no internet needed. on top of that, i added a small OCR enhancer that fixes common mistakes like “Ca1cu1us” -> “Calculus”, pulls out keywords by removing stop words, and detects academic patterns like “derivative calculus” or “acid base chemistry” using regex. and also enhance my images first before sending it to OCR (also did rotating of image incase they are yk in weird position)
for gemini,
at first, i was sending one request per image, so 10 photos meant 10 requests. this instantly hit the rate limit (free tier only allows 10 per minute). then i switched to batching (mean many images->extract keywords->send it as one prompt to classify it) sounds easy but its not, everything came back as Unclassified with 0% confidence, even though OCR output looked fine. tho single image requests still worked perfectly,
so i added a debug panel on the upload screen to actually see what was going on. turns out the batch prompt was structured completely differently from the single-image prompt.
another issue was maxOutputTokens being set to 1000, which was cutting off the JSON response for larger batches. increasing it to 4000 fixed that part.

then i gave up and i used NotebookLM to refine the batch prompt. the fix was putting the reasoning field before the subject field in the JSON. this made Gemini process the text first before deciding the label. combined with making the batch prompt match the single prompt style (same formatting, examples, tone), it finally started working.

final setup: OCR runs in parallel locally, then a single Gemini request classifies everything and returns a JSON array, mapped back using IDs.

Image Upload Feature
hi!
in this devlog, i added an image upload feature to the shared notebook. you can now select a picture and it drops exactly in the center of your screen.
.for this purpose, just added a native compressImage helper using the browser canvas api to resize big photos to 1024px and compress them into 80% jpegs before sending them. then it uploads the tiny jpeg to the supabase notebook images storage bucket and drops it in using the handlePlaceMedia function.
the tricky part was the coordinate math and upload delays, images were getting placed outside the current page view because it calculated the position after the upload finished and accidentally double counted window.scrolly in the math
.fixed it by capturing the exact startx and starty center coordinates before the upload begins so it fits where you were looking even if you scroll away.
next feature: idk honestly, im too tired cuz of this image upload feature :)

Settings Screen
hi!
in this devlog, i integrated the settings screen to save and load everything from SharedPreferences. before this it was all in memory so changes were lost when you left the screen.
you can now add a new subject, rename an existing one, or delete one.
it was mostly just replacing the local list mutations with calls to StorageService.saveSubjects() after every change. add, rename and delete all update the list then immediately write the whole thing back to prefs.
hmm the tricky part was making sure the home screen reflects the changes after you come back from settings. fixed it by calling _loadData() inside didChangeDependencies!!!
next devlog: OCR using ML Kit and maybe gemini integration also
Note: whole about section is for the future implementation, as i havent made any release on playstore yet

Folder View
hi! in this devlog, i made the folder view screen show real photos from the device instead of grey placeholder boxes. user can tap any subject card on the home screen and see the actual lecture photos you uploaded
it reads the files directly from the subject folder on device storage, pulls real metadata like file name, size, and modified date. deleting a photo removes the file. long pressing enters selection mode where you can delete multiple at once. the bottom sheet also shows a thumbnail of the photo.
did it by using getPhotosInSubject from storage service to list all jpg and png files in the folder, then used Image.file() to render them. added a deletePhoto method that calls file.delete() on the actual path.
the tricky part was reading args passed from the home screen , i had to make sure that was read inside didChangeDependencies so the photos reload every time you navigate back to the screen.
next devlog: settings screen maybe TT

Photo Upload
hi!
in this devlog, i made photo upload work.
when you pick photos, they show up in the photo strip and the classification list with your actual images. you can tap the arrow on any photo to expand it and manually override which subject it belongs to. once you hit confirm and save, the photo gets physically copied to the correct subject folder on your device storage.
it was straightforward for the most part, just wired up image_picker for gallery and camera, and used dart:io to copy files into the right folder path.
the tricky part was permissions. android 13 and below handle storage permissions completely differently, so i had to request photos permission first and fall back to storage permission for older versions otherwise it silently failed.
next devlog: folder view screen showing photos from device

Lock Feature
hi!
in this devlog, i added a lock feature to the shared notebook. you can now hover over any text, sticker, or gif and click the lock button that appears, this locks it so no one can delete it, or edit it. when something is locked, hover again and click lock button to unlock it.
it was easy, just added a locked boolean column to both writings and media_items in supabase. on the frontend, when locked, the delete button is hidden, drag is blocked, and editing is disabled. the lock/unlock just does a simple update to flip that boolean.
the tricky part was realtime sync, the lock state wasn’t broadcasting to other users at all.. fixed it by running ALTER TABLE writings REPLICA IDENTITY FULL
next feature: image upload integration maybe

Multiple Notebooks Feature
hi!
in this devlog, i made the shared notebook supports multiple notebooks, there’s a dropdown in the toolbar where you can switch between them, create new ones of your own..e.g someone can make a notebook just for cat pics, another for music recs, whatever.
for this purpose, i added a notebooks table in supabase and put a notebook_id column through writings, media_items, and drawing_strokes.
i also set up RLS policies, anyone can read and create notebooks. I didnt made feature of deleting someone’s notebook because its just evil
Next feature can be of locking one’s text so no one can delete it or image upload integration!

Debugging and debugging :”)
Hi!!
In this devlog, i Spent a hours doing a full code review and fixing everythin i found out is wrong!
In both smart_column_cleaner and the Recommendations tab conversion paths, converted series with a sparse dropna index were being assigned back to the full dataframe. This simply means it turns the originally null rows into NaN. The fix was applying .reindex(df_clean.index) in all conversion branches. Similar issues existed in validate_email, cap_outliers, and validate_range, where row removals left a non contiguous index and caused mismatches in before/after comparisons. These were fixed by adding .reset_index(drop=True).

moreover, KNNImputer could receive n_neighbors=0 on very small datasets.. so i added a guarded max(1, min(…, len(df_clean))) constraint . IterativeImputer also crashes when only one numeric column exists, so it now automatically falls back to KNNImputer.

the most illegal one :”) history was being pushed even when operations failed, so it now uses snapshot() and commit_history() to record only successful operations
.
find_and_replace previously cast entire columns to strings, converting NaN to “nan”. validate_date_col also used an incorrect missing value type.Also, all pandas inplace=True calls were replaced with reassignment for pandas 3.x compatibility, and Excel downloads are now cached to avoid unnecessary regeneration.

Note: I attached the video explaining each feature, tho video is rushed cuz ft dont allow uploading large videos TT

tooltips, readme update, and testing
in this devlog, i Added help tooltips across the Clean and Validate tabs. Every widget that needed context now has a ? icon, on hover it gives the explanation of what the option does and when to use it. I made it so that even non tech user can understand whats happening with their data!
Also Updated the README and included new features description in it. e.g the tab layout, column profiler, before/after comparison, history and undo, pipeline export, and multi-sheet Excel support. Also added one more multi-sheet excel test files.
Moreover, i performed extensive testing and made sure that my pipeline works. My app works on single sheet excel and multi sheet excel also.
If you guys have any issues with it then feedback is welcome!!!

Before/after comparison, column type override, and pipeline export
Hi!!!
in this devlog, three more features are added.
First was before/after column comparison, you pick a column and it shows original vs current values side by side with a third column that flags exactly which rows changed.
Second was column type override. a dropdown is given to force any column to int, float, string, datetime, boolean, or category.
it was kinda tricky TT cuz booleans needed a string-mapping step first, integers needed nullable Int64 so NaN values don’t cause a cast error, datetime needed errors=‘coerce’.
Third was pipeline export. which I think is the most useful thing. It reads through the cleaning history and writes out a proper .py script you can download and rerun on any new file.

UI CHANGES
HI
This devlog is mostly a styling and restructuring UI rather than new features.
The first thing was removing the big heading at the top of the page. because it was taking alot of space and isn’t necessary in the first place.
Then i made the tabs sit right at the top of the page like a website navbar.. Streamlit’s default tab styling is pretty small, so i had to override it with CSS.
Before, my UI was having a long long one webpage, which was causing the bad user experience
Now i made separate tabs for separate purposes so just by looking at the UI, user know where he will find what features!!!
Streamlit is easy (if we use its own styling) but if we try to make out own css in it :”) then its sucks :)

Implemented Column Profiler, find & replace and undo/history system
Hi!!
In this devlog, I added three features to the data cleaning pipeline that I felt were genuinely missing from the first version.
First was a column profiler ,table that shows you per-column stats like min, max, mean, median, std, skewness, null percentage, and sample values. Cached it with @st.cache_data so it only recomputes when the dataframe actually changes.
Second was find & replace, pick a column, type a search string, type a replacement, and optionally flip on regex mode for pattern-based replacements..
Third,, was a full undo/history system. The history panel shows every step and lets you undo one step at a time or wipe the whole history.

Resolved Performance Issues
HI!!
in this devlog, I optimized the app to stop re-running expensive operations on every single user interaction. The main issue was that analyze_data_issues, which runs regex loops across every column in the dataframe was firing on every button click I fixed this by wrapping it (along with the recommendations generator) into a single @st.cache_data function called get_analysis_and_recommendations. Same thing i did to the file reader and the stats calculator. Also deleted all verbose=True calls inside manual operations asthey were triggering partial rerenders.
Some users said its a bit slow, thats why i spend 3 hours , only figuring out what could be the issue, and came up with these solutions
Also, I made two modes, one simple and one advanced in the settings sidebar, by clicking to advanced mode, user can select the thresholds and model they want to implement for missing values!
in the next devlog, i will implement more functions slowly and will also make it more user friendly.

Hi,
In this devlog, I made the persistent data layer working. the app needed to remember ur subjects and storage folder between sessions.
So,
I used SharedPreferences to save and load everything locally on the device. So now when you go through onboarding, pick your storage folder , add ur subjects . all of that will be saved in the folder you selected.
Next time you open the app, it reads that data and takes you straight to the home screen with your subjects.
In next devlog, i will integrate the photo upload system

HI!!
In this devlog, I made the settings screen. Here, the user will fill in his clinician details (Name, Designation, and Hospital). He can also enable the dark mode, which will be implemented later. moreover, it also has the version and AI engine details about the app.
I did it using the SettingsService singleton and custom Flutter widgets like _SettingsCard and _Field.
Also, One thing I love abt Flutter/Android dev is we can just import Material Icons and make it work!

Hi!!!
So i built the patient_records_screen, for now its having the dummy data because we will integrate hive database later on
it will have all the patient records till now, user can also delete, edit or copy the patient details.
moreover, i also make the navbar working, linked patient records screen with the navbar, and also cleaned the dashboard and remove the buttons i added before but i didnt actually needed them :”)
Also navbar will not be shown when we are entering the patient record because those are the steps user have to perform, what if in between he accidently click the home screen so :)

Hi!!
in this devlog, i made the final diagnosis screen that shows the diagnosis in detailed manner. this diagnosis is gained through the physical examination done by the doctor and through symptoms entered!
And moreover, i made my code clean and modular, i separated the data models from screens dart code and save it in models folder thus making my code more easy to extend in the future
and also this way my code wont have to import big screens.dart files and can just import the data models, now my architecture is pretty clean

HI!!!
In this devlog
i made full structure of my application, i made only the UI of whole system, cuz this way it will be easy to visualize all the flow
and also , i linked all the screen together so now all left is backend working. I also figure out what services to make, so basically i made all the ARCHITECTURE of the system!
also before i was doing it in JAVA but someone suggested i should make it in flutter because flutter is more easier to learn than JAVA
soI here im! I also made a new repo for flutter cuz i pushed it in prev repo with java code and it was causing errors so i made new one!
Also , upload pic functionality isnt working as i havent implemented its backend yet :”)

HI!!!!
In this devlog, i made the SOAP screen
for context, SOAP is literally how doctors write clinical notes.
S = Subjective (what the patient tells you), O = Objective (what YOU find, vitals, labs, examination), A = Assessment (differential diagnosis), P = Plan (what to do next). every doctor writes these.

so what i built is, after you go through the ENTIRE flow (patient info → history → systemic review → vitals → labs → physical examination) the app auto-generates a full SOAP note from ALL of that data.
the S tab pulls the patient’s name, MR number, chief complaints with duration and severity, past medical history, drug history, allergies, family history, social history.
the O tab shows vitals with abnormal flags, and the full lab panel
the A tab is the differential diagnosis section
the P tab is the plan, investigations, treatment, follow-up. its still in the making
also everything is EDITABLE. so the doctor can tap any section and just fix whatever the AI got wrong.

Lab Screen Integration
Hi, in this devlog I made the lab screen where the user will enter the lab data.

I made various lab panels in it, Complete Blood Count, Liver Function Tests,
Renal Function Tests, Electrolytes, Glucose & HbA1c, Lipid Profile, Thyroid
Function, Cardiac Enzymes, Coagulation Profile, Arterial Blood Gas, Urine
Analysis, Iron Studies, and a free-text Cultures section.
made it according
to the official WHO and standard clinical reference ranges for all normal,
warning, and critical values.
And this data will be stored in Hive later on.

This was difficult to make because instead of
hardcoding every panel I had to design a LabTest and LabPanel config system so
the UI renders itself automatically from the data , moreover
each panel is collapsible, theres a live abnormal counter in the app bar, and
a save summary sheet that lists all flagged values before proceeding

Patient Info Screen Feature
Hi
so in this devlog I made the patient_info screen that takes the patient information like age, gender, full name, date of birth, address, marital status, religion, date of admission, and mode of admission (OPD or Emergency). It also auto-generates a unique MR number for each patient.
I built it using Flutter with a Form widget. The date of birth uses three linked dropdowns (Day / Month / Year). There’s also an animated OPD / Emergency toggle pill.
In the future I will implement Hive DB in it so all patient data will be stored locally on the device.
I also spent time linking all the screens together so the final navigation flow is:
Home -> PatientInfo -> HistoryTaking (3 pages) -> SystemicHistory -> Vitals -> Examination

HI!!
i was trying to make my website responsive :”) and i broke the editing feature in it, then i spend some time debugging it and used the prev working one then!!
i have tried many times to make it responsive but it just dont Work, so im leaving it as it is
ALso i check other features and make sure everything is working correctly!
Also updated the readme
Note: please use it on laptops or tablets TT
i have attached the video explaining all the features!!!

hi, so in this devlog, i implemented the lofi music in my notebook, this music will be synced for everyone! You can choose to pause or play it!
When user first enter the notebook, it will not play , it will only play if user decided to play it in the lofi tab!
i did it by using direct MP3 streaming via the HTML element pulling from nightride.fm.,
i only implemented 4 of the playlists in it!

Hi!!!
so in this devlog, i implemented the Reactions feature that makes reaction emojies floats for some seconds then disappear just like how it is in Google Meet
for this purpose, i used useReactions hook which uses Supabase Broadcast. It was easy to implement!!
next feature would be Lofi music integration maybe! or I might just improve the existing code a bit and reship it and then move on!!!

HI!!
so in this devlog, i built a feature that shows who is currently typing right now . i built it using the Supabase realtime presence thing, and the channel lifecycle bug was the toughest part thats why it took me 50 mins TT

every user joins a shared presence channel and calls channel.track() to broadcast their current state (name + whether they’re typing). When anyones state changes, the sync event fires for all other users and they update their UI accordingly. thats how i made it work in the end :”)

HI!
so in this devlog i made a name feature, when you open the notebook for the first time, a little modal pops up asking for your name. then whenever you write a textbox, your name shows up underneath it on hover so everyone knows who wrote what :)
i did this by saving the name to localStorage so you only get asked once, and passing it as an author_name column on every insert to supabase. then in the WritingNode component i just render a small label that fades in on hover. honestly nothing crazy going on under the hood :”)
it was easy work using supabase, i just had to add one column and one label, the modal was like 20 lines.
Also, u see a toast message , [this person] has joined or left the notebook!

HI!!!
So in this devlog i implemented the realtime feature, before this users had to reload the notebook to see other people updates.
but now it works by subscribing to postgres_changes on each table.
and also before i tried implementing this feature but it didn’t work because i had realtime enabled in the supabase dashboard but never actually wrote the .on(“postgres_changes”, …) subscription in the code for writings and media_items.
yes i almost gave up for this but then i said why not try it out one more time :”) Before it didnt work cuz i was not integrating it in my code well, i also overlook this feature back then, but its so cool to have it

Also, it took me almost 30 min to implement this, but since flavortown was down, so i worked on other feature
which is user will enter his name in the start, and my db will save this name and show it alongside with his writings!!!

ERASE DRAWING IMPLEMENTATION
Hi!!
So in this devlog, i worked on drawing feature :”)
ANd IT WAS SO HARD TO MAKE!!!
first i tried making a eraser how we have this in the Paint, but then i read that for this my pic need to be saved as a flat image in supabase so it can erase it accordingly. but i was saving the strokes individually so that wasn’t going to work, then i switched to the cross symbols on the strokes, just like my textboxes,so when user clicks on these cross symbols, the stroke gets deleted. its not as easy as it seems :) dealing w strokes is very tough because the strokes are just arrays of points on a canvas, there’s no DOM element to attach a button to. so i had to do hit-testing on every mousemove i check if the cursor is close enough to any line segment in any stroke, and if yes, show a ✕ button floating at the midpoint. BUT THEN the invisible div i put over the page to detect the hover was blocking all the drag and drop for text nodes and stickers :”)) so i had to scrap that and instead attach the mousemove listener directly to the page element so it wouldn’t block anything underneath.
and i dont know why everything i do breaks the drag n drop functionality, ugh i hate it!!! so got it working also by debugging it
(apparently its cuz of drag events bubble up through the DOM, so any event listener(mousedown/mousemove) before the drag handler gets it will silently KILL the event)

Hi!!!
So, in this devlog, I improved the anti-cheat system again bcz it was not working with tablets with sidebars.
I was checking if
(outerWidth - innerWidth > 160) or (outerHeight - innerHeight > 160) to detect devtools…
but the problem was that on tablets, opening a browser sidebar shrinks the innerWidth by the same amount devtools would.so it was triggering the
CHEATER DETECTED screen.
I removed that size threshold check entirely, and now it relies on the other detection
methods: the debugger timing trick, the Image getter trap, keyboard shortcut blocking,
and right-click prevention.
Yes!! Thats it, i hope you enjoy it

ADDED DRAWING FEATURE
HIII!!!
SO in this devlog, i added a pen drawing feature to the shared notebook :”). SO, you can now switch between Text and Draw mode using a toggle in the toolbar. when you’re in draw mode, you can pick your pen color from 12 presets and choose from 3 pen sizes (small, medium, chunky). then just draw !
for this,i also made a new table in supabase named as strokes!
the TOUGH part was getting the drawing coordinates right across multi-page layouts since the canvas had to match the full scrollHeight of the page, not just the viewport.
(ngl this coordinate calculation always frustrates me TT)

hi!!!
so, this devlog is just about refactoring the code, tho i checked and its pretty nice as it is, Also added more keywords in content-analyzer!!
the thing is i got 6 mult on 11 hrs of work on this project cuz of bot voters, so i have to reship this again
i hope i get my justice now TT!!!
Thats it!!! I hope u like my extension!!!
Also, in the attached Video, i was rushing alot TT because flavortown has the limit for the video upload, it wont let me upload if my video is too long TT!
it still went to 2+ min :”)

BUILT THE ENTIRE EXAMINATION ENGINE!!! (used the prev code also i wrote in first devlog)
HI!!!
SO in this devlog, I built the physical examination screen ,it was the most complex screen in the whole app :”) SO, every question used to have its phase title, instruction, and clinical tip hardcoded in THREE separate giant const maps (like 32 entries each, 96 lines of pure hardcoding). so i deleted all of that and moved it into knowledge_base.json as fields directly on each question object( phase_title, instruction, tip).now KBQuestion.fromJson() just reads them directly.
ALSO the constraint system was a whole thing, you cannot select “Tachycardia” AND “Bradycardia” at the same time (because, you know, physics) .so i built a contradiction checker that rolls back any impossible selection and shows a red banner. there are like 40+ mutually exclusive pairs across CVS, RESP, ABD and NEURO. none of them were hard individually but mapping ALL of them out at once was a little (NOT LITTLE) pain :”))
ALSO the rule engine injects follow-up questions dynamically ,so if you select “Irregularly irregular rhythm” it unlocks an AF-specific question mid-flow. and the results sheet shows diagnoses with certainty percentages calculated from a scoring system.
I ALMOST gave up and considered just making it a static checklist :”) also its not perfect right now,i will spend more time on it :”)

HI!!!!
so before doing a physical examination, doctors actually have to record a TON of stuff first. vitals!! blood pressure, pulse, temperature, respiratory rate, oxygen saturation, blood glucose, BMI. and each one has clinical ranges. like if BP is 145/90 that’s not just a bit high, that’s specifically Hypertension Stage 1 and the doctor needs to know that immediately.
so i built the vitals screen. as you type in any value, the app runs it through a clinical engine i wrote and flags it in real time. the card turns red with a little warning strip telling you exactly what’s wrong. “Hypertensive Crisis,urgent evaluation required.” that kind of thing.
it also auto-generates a flags list at the bottom. so if a patient comes in with high BP AND low SpO2 AND tachycardia, you see all three flagged together in one place before you even move to the next screen. those flags get passed forward through the whole app flow.
also added a custom vitals section for when doctors need to record something outside the standard list, things like CVP or ICP or ETCO2 that you’d see in ICU settings.
the full app flow is: patient info → medical history → systemic history → vitals → lab reports → physical examination → SOAP note → diagnosis (training a neural network for this part!!)
slowly getting there :)

HI!!!!
SO, I found out that before doing a physical examination, doctors first have to take the patient’s history. This includes asking about previous medications, past symptoms, current complaints, checking vitals, and also asking if there is any medical history in the family. Basically, the doctor first tries to understand the patient’s background before actually examining them.

Because of this, I designed the overall flow of my application around how a real clinical process works. The flow goes like this:
take patient history → take vitals → take lab reports → perform physical examination → generate SOAP → give diagnosis.

For the diagnosis part, I am planning to train a neural network model so that the system can suggest possible diagnoses based on all the collected information.

So in this devlog, I worked on building the UI for taking patient history. The process starts with the user entering the basic details of the patient, like name, age, gender, and other identification information. After that, the user can record the patient’s medical history, such as past illnesses, medications they are currently taking, and any relevant family medical history that might affect the diagnosis.

Next comes the systemic history section. In this part, the user records information related to different body systems, for example if the patient has any issues related to the respiratory system, cardiovascular system, digestive system, and so on. This helps doctors quickly understand which body system might be affected and gives better context before moving on to the physical examination stage.

So overall, this part of the application focuses on collecting structured patient history in a clear and organized way, which will later help the system generate a more accurate SOAP summary and diagnosis.

Physical Examination IPPA style JSON building for CARDIOVASCULAR EXAMINATION ONLY devlog
HI!!!
SO in this devlog, i implemented the json file with all the facts and rules!!! So what will it do is, it will match the facts with rules and ask the user questions abt what he is observing during the 4 type of medical examinations which are IPPA so it will first ask abt general examination then IPPA (inspection, palpation, percussion, and auscultation)!!
so my app will help the medical students/doctors in taking the physical examination or history of the patient!! My friend is a medical student and she said she often forgot what steps to perform in each examination, so my application will help her remind all the questions/steps being performed in the examination and then in the end will also give diagnosis!
You can see it working for cardiovascular for now in the video attached!
Also i used the medical book KUNDUS BEDSIDE CLINIC 4th EDITION for building of my JSON so all the data being used for my application comes from this book!

HI!!
So, this is my first devlog, im making an application for junior doctors or medical students, they can run it offline and find out what questions to ask to the patients based on specific examinations, in the end it also gives suggestion what this all symptoms point to. Lets say breathlessness and High bp points to heart failure(it dont always does but its a hypothesis lol)
This is all based on kundu bedside clinic book!!! SO in this 1h 50m devlog, all i did was find out what will be our dart/flutter logic, how will UI works, what main features will User see for this purpose
and figure out how to make our Json with all facts(defined terms/questions used in medical books)
SO, i figured it out and then make the minimal working UI for it, rn it have no questions placed in it, as i will be placing it later on using the JSON i will built , i will make a forward chaining inference engine(see facts first and use it to lead towards the goal) that will look at our json and based on each examination suggest the user what should he examine in patient!!!

NOTE: i will be providing the examinations in IPPA medical styling!!!
I am not a medical student, but i will make sure all of the stuff i will make to be accurate, i will only use kundu book for this! and make it work!!! also making this for a medical friend who needs this :”))

HI!!!
IN this devlog, i again checked all the functionalities and found out that toolbar is just static at the header so i make it sticked to the top regardless of the page position!!!
I just had to add one thing in its css
position: sticky
so, it was easy work, but i found out how dumb i am, i was only testing my website using the first page only TT and didnt thought about adding gifs or stickers in later pages

Note: i added video of the project explaining all the functionalities!!! watch it to find out all the functionalities TT ( but they are so obvious to the users, i dont think so u guys need to hear me yap)

HI!!!
so in this devlog, i debugged the drag n drop feature. I found out that drag and drop isnt working deep down in the page because i was calculating the Y position of updated position wrongly :”)
Spent Hour figuring out whats causing the issue :”) , apart from this, i cleaned my code and tested all the functionalities.
Also, I will decide on new features based on reviews or voting. so suggest me more features please TT

DEBUGGED EDITING FEATURE
HIII!!!
SO in this devlog, I debugged the editing feature :”) SO, clicking a node to edit it would also trigger the global page click handler at the same time, so i stopped that with e.stopPropagation(). ALSO, contentEditable was also not working where the cursor kept jumping to position 0 instead of the end,then i set a Range to fix that. also if someone clicked away without typing anything it would save an empty string, so i just delete the row instead if its blank. none of it was super hard but debugging all of it at once was a little( NOT LITTLE) pain :”)) I ALMOST GAVE UP AND WAS THINKING OF JUST REMOVING THE EDITING FEATURE :”)

GIFS and STICKER INTEGRATION
HIII!!!
SO in this devlog, i integrated the gifs and emojies. For it i also made a table in SUPABASE for stickers and gifs so they updated in real time.
I add a list of emojies in the code(hardcoded) and you can also search it accordingly. I dont use any apis for the emojies and implemented it all in code.
Also, i used api of giphy for gifs, user can now add gifs into the Page. and can also search them. I also added options for gifs user can use to search! like sakura, kawaii and cats(cuz i love cats)
Adding api made life alot easier TT and also giphy apis are free to use, you can also use that to integrate gifs in your project!!!
Also, user can also resize the gifs or stickers!

MORE THEMES + EMOJI PLACEMENT DEVLOG
HII!!!
In this update i did a bunch of theme improvements.
second thing. before there were like 2-4 emojis per theme placed at 6 hardcoded positions (corners and sides). now every theme has 10-12 emojis and they get randomly scattered across 10 positions. I also changed the website background to use the same background/overlay register page was using.
I will keep adding more themes later!!! Sakura is still my fav one tho!!!
Next Update: Writing freely with given pen OR integration of gifs or emojies!!!

THEME INTEGRATION DEVLOG
HII!!!
in this devlog i added japanese-themed page background named SAKURA AND made the add page button.
the theme were the MOST fun part. theme has three layers,
a body background (the color behind the page),
a page style (the actual notebook paper look),
little emoji in the corners.
the theme also sync in realtime across all users through the page_settings table in supabase. so if someone changes the theme everyone sees it instantly.
I will update more themes later on!!!
also spent way too long on fixing the text going outside the page from the right side. TT
Also, added the change font button!!!

HII!!!
in this devlog, i made editing existing text work. It wasn’t working and it took forever to figure out why. turns out there were actually two separate problems happening at the same time which made it extra confusing to debug.
first problem was the RLS policies. the writings table only had SELECT and INSERT policies, which meant update and delete calls were being blocked by Supabase. fixed it by adding two policies in the Supabase dashboard: anyone can update and anyone can delete, both with using (true).
second problem was way more _____. the click handler wasn’t reaching the writing nodes at all. tried React’s onClick, tried onMouseDown, tried native event listeners, nothing worked. eventually added console logs and figured out that e.target.closest(“.writing-node”) was returning null because clicks were landing on the inner text div, not the wrapper. fixed it by switching to e.target.closest(“[data-id]”) which catches clicks anywhere inside the node, and moved everything into a single global document.addEventListener to avoid React’s errors TT

HI!!!
so i built a shared notebook app where multiple people can write on the same page at the same time. it has a notebook with lines and a red margin and a pen. you click anywhere on the canvas and write there.
the first version used a basic input that saved on Enter. I know its annoying because you write alot of thing and forgot to hit enter and all your text is gone
so in next update i will make it so that without hitting enter, it will save your work, and also let user to edit and delete current writings.
Also there’s a color picker and font selector in the toolbar. pretty happy with how it turned out

Hi!!!
In this devlog, i tested all the functionalities once again. I am sure they all are working correctly.
I also have attached the demo video along with my voice explaining all the functionalities!!!
I hope you like my work :”)
Demo Video: https://drive.google.com/file/d/1-grGcqJQ6B3npXZOQ_7ZmkfjFL48aCGl/view?usp=sharing

HI!!!
In this devlog, I added a new Validation & Quality section in the data cleaning pipeline where I implemented five new validators to check emails, clean and standardize phone numbers, correctly parse mixed-format dates, detect and handle outliers, and validate value ranges. This update helped fix many hidden data issues, especially the date parser which now handles almost all common formats., and I also kept the UI consistent with the rest

HII!!
So in this devlog,
I turned the app into an intelligent assistant that actually thinks about your data.
It now scans the dataset, finds problems like duplicates, currency symbols, wrong data types, percentages as text, and missing values, and shows them as simple fix cards.
Each issue comes with a one-click “Fix This” button, and I also added checkboxes in front of AI suggestions so users can select exactly which columns they want to apply fixes to.
Added an “Auto-Fix All Issues” button that runs the full pipeline in the best order and fixes everything at once.
Now beginners can clean data without technical knowledge, while advanced users still have full control.

HI!!!
So , in this devlog, i also added milliseconds tracking on the pomodoro! and I used this extension for weeks to figure out if its working correctly or not. I think, I have added the most of features i could in it!!!
I am glad how it turns out <3

HI!!! In this devlog, I finally got the Buy Now button working! When clicked, it shows an alert dialog asking for confirmation, and on confirming it triggers the SMS logic using SmsManager. Testing on emulator shows SMS failed (emulators can’t send real SMS) but the logic is correct! In this process I learned about AlertDialog.Builder and how SmsManager works.

HI!!!
In this devlog, i made an activity(page) of the product where i add buy now button and product details, buy now button isnt working for now, i will update that in next devlog
in this process, i learned how we pass data from one activity to other activity, also keeping track of ids are headache, i should
be more professional and name them professionally so its easier to remember TT

Hi!!
in this devlog, I made a homepage, i used dummy data to fill in the values of the products. I made a usable product_card.xml and used
it instead of making 4 different cards.
I learned how to use setText and how to find View from its id from XML.
In next part,i will connect this activity to product details activity where user can order then.

HII!!! so i am going to build a simple e-commerce app. Well, I am new to android developement. so I will only be making beginner level stuff in it to enhance my skills.
In this devlog, I explored Android animations (TranslateAnimation, AlphaAnimation), XML layouts with RelativeLayout, and Handler.postDelayed() for timed navigation. And using it i built a splash screen with a sliding truck animation that transitions to HomeActivity after 3 seconds.

HII!!
So in this devlog,
I upgraded the website with a super-smart missing value handler. It now detects all kinds of missing data like “NA”, “?”, -999, and more, then fills them intelligently.
Numeric columns get KNN or MICE imputation depending on dataset size, while categorical columns get mode or “Missing” automatically.
Smart threshold drops columns with too many missing values, and everything can be controlled in the sidebar.
Also added a one-click “Full Pipeline” button that runs all cleaning steps in the best order, with detailed feedback showing exactly what changed.

HII!!
So in this devlog,
I upgraded the app into a smart data transformation system by adding intelligent string cleaning and automatic type detection.
The system now cleans text, detects patterns like currency, percentages, units, durations, and numeric values, and converts them automatically.
smart threshold system prevents wrong conversions, and users can control sensitivity using a settings sidebar.
also added a one-click “Run Basic Pipeline” button and detailed feedback showing exactly what was converted.

HI!!!
SO in this devlog, I improved the data cleaning pipeline with a cleaner UI and a real-time statistics dashboard showing rows, columns, missing cells, duplicates, and data types.
I added a flexible preview slider (5–50 rows), a collapsible column info panel, export options (CSV & Excel), and a reset button to restore the original dataset instantly.

Hi, this is my second devlog on the LectureVault! Today I worked on the main page where you can select images from your gallery to start organizing them. I used the Activity Result API with a launcher bcz old startActivityForResult way is now deprecated. I learned that you have to use EXTRA_ALLOW_MULTIPLE to let the user pick more than one photo at a time.

Next update: letting user make the folders first so it will be easy for my classifier to classify images!!

Hi!!!!
So, this is my last devlog of this project!
In this devlog, I finally added the feature of blocking distraction sites while having the pomodoro Focus Session. Whenever you try to open the distraction sites e.g instagram, Facebook etc, it will block it and take you to page that says KEEP SIZZLING.
Also, user can also add or remove sites that he want to block during focus sessions! This made it easy for users to add anime/series pirated sites that i might have skipped TT.
The blocking system was actually pretty tricky to implement! I had to use Chrome’s content scripts to inject code into every single page you visit. The script runs in the background and constantly checks if you’re in an active focus session. If you are, it matches the current website against your block list using proper domain matching (so it catches instagram.com, www.instagram.com, m.instagram.com - all of them!). The overlay I made is persistent too - even if you try to delete it using inspect element, it respawns every 2 seconds
But here’s the thing - I also added a checkbox in the focus tab that says “Block distraction sites during focus” because I realized not everyone wants this feature all the time! Maybe you’re doing study that requires YouTube Lectures, or maybe you just want the timer without the blocking. So now you can toggle it on/off per session!
I made this for myself mainly! because i wanted a platform that can track my study time and can block distractions during my focus sessions! This project also taught me so much about Chrome extension APIs, content script injection, and state management across multiple contexts.

Also , cooking time is for flavortown!!!!

Hi, This is my first devlog on the LectureVault(might change this name later)
and this is also my first time working on android app development, so i will be learning about it along with doing this project!!! I do know java tho so i hope its easy to learn. I will be using java for this project.
What i learn today:
folder structuring mainly
how to push on android studio(tho i still need more details on this one, will research abt it tommorrow)
XML is for all the layout here , also it gave you cool view of design.
In attached video you can see if we move the design layout , like move the icons in it , then it also changes the code accordingly, no worries abt maintaining x axis and y axis TT
Also, im finding this way more harder than web dev :”)

Hi!!!
So in this devlog, i added a feature that if user try to cheat his way by opening the dev console, then a cheater screen will show up preventing him from opening it.
I added this feature in all 3 levels.
Also , make sure you open this website on tablet or laptop, as i dont have made it responsive on phone as its too much of a chaos!!!
Turn on your volume also!!!

HI!!!
So, i made this website more annoying by adding optical illusion background, and more annoying emojies and colors.
I also changed the css(styling) for all levels!!! I hope you find it irritating!!!
I made all levels beatable so, i hope you also find it playable along with it being irritating
You can play it here
https://website-that-fights-you.vercel.app/
This will be my last devlog of this project, I am not planning for any future update!! but i would love to make a rage game like this again!!!!
Also, my attached video keep getting corrupted on level3!!!!! you can just play it to find it out

HI!!!

so i made the Game Over screen for the project.
It’s supposed to celebrate the player finishing all levels, but of course in the worst UI ever. I added huge flashing, spinning, jittering text that says stuff like “CONGRATULATIONS! YOU ARE A LEGEND!”, “GOAT STATUS UNLOCKED!”, and “AURA FARMING COMPLETE!”. The colors are super annoying(well for me tho)
I also made random fake popups and loading bars appear everywhere. They don’t do anything, just to make the player annoy. Clicking anywhere triggers random sounds and shakes the screen. Some icons and images glitch and flicker too, just to mess with the eyes.
The real “Continue” or “Restart” button is hidden in a corner, so the player has to find it by exploring carefully. It was really fun writing all the animations and random effects.

Here’s the demo link to try it:
https://website-that-fights-you.vercel.app/gameover.html

I think this is the most funny page I have made!!!

Also, for voters:
LINK for level2
https://website-that-fights-you.vercel.app/level2.html
For level3 , you can visit here
https://website-that-fights-you.vercel.app/level3.html

HI!!!

so today i finally made Level 3. This one is crazy messy and full of distractions.

The level is all about confusing the player with fake buttons, popups, clocks, progress bars, and glitching stuff everywhere. Nothing works the way it looks, but every level is still beatable.

I added flying buttons that swap positions when you get close, fake countdown timers, fake loading bars, and even random popup sounds. Clicking the wrong buttons makes the screen shake, and some icons and images flicker around to make you panic.

The actual success button is hidden, and the hint is in the screen. You have to ignore all the distractions, watch carefully, and find it yourself.
I also did some work on Game Over screen, I am reusing my previous components for it. I hope I do this project on time. :)

Here’s the link to try it yourself:
https://website-that-fights-you.vercel.app/level3.html
If you want to play all levels:
https://website-that-fights-you.vercel.app/

HI!!!
so today, i finalized level2, Its abt reading terms and conditions, but if you scroll too fast, you will be redirected to the top, you cant skip or accept the page!!! Only when you complete it, you goto next level (but there is a plot twists in the flying buttons, you can play it and find it out)!!!
I made alot of flying buttons and clocks in it to confuse the user. Since it is for bad UI competition , i made many levels of it showing bad UI, but no worries each level is doable
you can find it out by playing it on link provided below,
I think i should write more details in this devlog about the level, but i want users to find it out by playing , i have put alot of effort and thinking in making it, It was fun writing the logic for whole flow of the level2. I thought abt more difficult and bad ui stuff, but since i want this to be a doable game also , i am not going to make it too difficult.
Note: I used ai to write the terms and conditions. But, if i got time later,i will write my own yap!!!
LINK for level2
https://website-that-fights-you.vercel.app/level2.html
For level1 , you can visit here
https://website-that-fights-you.vercel.app/

HI!!!

so this is my first devlog of this project. So, I have already made Level 1.
Level 1 is basically all about a running button.

Whenever you try to click the button, it runs away. But after SOME TIME (you have to figure it out by paying close attention), the button will be clickable for 1 second (should I make it of even smaller duration?!). And if you click it, you will be redirected to Level 2.

And for your ease, I named it CLICK ME, so you will know when this button will appear.
But best would be if I just make it a random button, which would be so worse.
I’ll think about it later.

Anyways, there is still an issue. There is a race condition in my code. I’m sending an alert anytime the user clicks the screen/button and it’s not time yet, so the screen will send annoying alerts, but it’s not working because of the race condition.

The mouse events are occurring so fast, so I’ll try to fix that. For now, I tried my best to research it, but :”)

So I’ll make Level 2 first and then do this.
Also in the attached video, it took me 5 6 tries to do that :”) , I’m a bit slow :”)
Here’s a vercel link of this project, you guys can try it and tell me how many tries did it take for you?!
If you guys managed to do that, then I will have to make it more tough :”)

https://website-that-fights-you.vercel.app/

HI
Well!!! I renamed my extension to Sizzle: Focus & Habit Time Tracker.
Also renamed the tabs. Focus is now Power Sizzle and Stats is now Hottest Streaks.

Then I submitted my extension to Firefox since it is free of cost. I originally made the extension for Chrome, so yeah, I got a lot of errors in manifest.json. I spent around an hour just figuring out their documentation and fixing the manifest. After that, I also had to change other files like background.js and popup.js. VS Code tracked only 1 hour 29 minutes, but honestly it took me almost 3 hours to understand Firefox docs and then code everything. SIGH!!!

Anyways, once my extension gets approved, I will share the link in the next devlog. My GitHub will also have files that support both Chrome and Firefox. Since manifest.json can only support one browser, I made a manifest-firefox.txt file on GitHub. If anyone needs the Firefox manifest, they can just copy paste that file into the original manifest.json.

I also spent another 20 minutes making the logo. Graphic designing is not my thing, so the logo turned out pretty meh, but I did my best.

Future Update:
I tried adding a 3 beeps sound when a focus session or break ends, but it is not noticeable. I will try to find some free sounds and update it. I also have other features in mind, like blocking distraction sites during a focus session. I will try to add that in the future.

HI!!!
In the last update, I only have made the tabs for the Compare and Records.
But now, I made the Compare and Records tabs fully functional! The Compare tab now actively tracks your progress by comparing your current week’s usage against the last, giving you a percentage breakdown of how much your productivity has increased or your procrastination has dropped.
While the Records tab tracks your current and longest streaks, your best study day, total study time, and total focus sessions. To make the progress visual, I added some logic that assigns a fun “level-up” emoji to your streak: it starts at a sleepy 💤 if you’re just beginning, but as you hit 1, 3, or 7 days, it evolves into a full 🔥🔥🔥 to show YOU ARE DOING GREAT!!!

Next update: I still have to think about what new features I will add!!! But I think a feature for blocking entertainment sites while you are in a focus session would be great. However, I can only block things like Instagram, Facebook, or pirated drama/anime sites lol; I will have to figure it out because I can’t just block YouTube since some people watch lectures there. Also, I mainly made this extension because I couldn’t find any that combined a focus pomodoro with time tracking. Enough with the yapping!!! Have a great day everyone!

Hi
Today, I added the cards for the Leaderboard functionality.
The Leaderboard shows your top 10 websites for the week, plus some champion cards - most frequently visited site, top study site, and top entertainment site. I reused my existing renderThisWeek function to display the top 10, and wrote new logic to track the most frequent site.
For future work, I still need to add the Compare and Records tabs. Compare will show your study time vs entertainment time for this week compared to last week, and tell you if you’re being more or less productive. Records will track stuff like your longest streak, most productive day, and other personal bests.
Note: I know I am naming it as stats for my tab, I tried putting Leader Board there but it takes too much space and look congested so I decided to just write Stats instead :”)
Also, I had to reinstall the extension because of an annoying error, which means all my tracking data got wiped and the timer restarted from zero :”) I should probably add a backup/export feature so this doesn’t happen again!

Spent 1hr 14m making focus pomodoro sessions, user now can set pomodoro timer of any durations for their work, all the focus session time will go into the user study/work time. User will also get long break after 4 sessions.
Also , streak badge will also be shown for focus sessions
Now, i think instead of showing the number of sessions in a day, i should show the time spent in focus sessions per day.
Note: (integrated my previous pomodoro code and make it work with the extension)

Moved the this week and this month time tracked tab in the time travel tab
Time travel tab is basically showing the time one spend on a certain day, week or month.
User can see their past data through time travel tab now
Next Feature:
Integration of the Focus pomodoro timer, all the focus time will be counted as study time

Made a pie chart , that shows the percentage of all the websites you are using.
Also made the stats for the sites in descending sorted order, so user can know the sites they use the most. Also added the month and week section ,where user can see the time they spend this week or month.
Added the study and procrastination cards, user will mark the websites they use for study and for entertainment ,and that data will be updated in respective card.
Upcoming feature:
now i will make new feature where user can find the time spent on specific day, week, or month.

I just finished the UI for my new extension and matched the design to the Flavortown dashboard.. It is a web time tracker that stays on your computer, but I added a special “Cooking Time” feature that specifically tracks how long you spend on flavortown.hackclub.com.
Btw, i really like cute UI of flavortown hack club and want to make websites like this one, i will probably make a focus app next with cute ui. I know my skills aren’t good, but i should try :”)

I setup the manifest.json (the heart of any Chrome extension) and initialized the project environment in VS Code. I also made the GitHub repository to track my progress and began coding the background script to detect when a user switches between tabs.

I’ve built a Streamlit UI that lets you upload a CSV and instantly clean it by dropping duplicate rows/columns and stripping extra spaces from text. User will choose what functions he want to run using the provided buttons. I already pushed the code for these three core functions and added a data preview so you can see the results immediately. For the next session, I’m moving on to removing special characters and fixing missing values.

I spent an hour setting up a Streamlit interface to handle the tedious parts of data cleaning. I wrote three core functions that take any uploaded CSV and automatically fix it: one to strip hidden whitespaces from text, one to drop duplicate rows, and a third to find and remove identical columns. The goal was to make something generic so I don’t have to manually clean files every time I start a new project. It’s simple, fast, and handles the “dirty” data work in one click.