Facebook Marketplace Scraper

I finished it up for the ship, currently working on building it correctly so everything works. I quickly realized that using github workflows for every build to find bugs wasn’t gonna work so I did it locally and got everything working!
Now it’s super easy to use, all you do is run main.exe.

Changelog

• update build.yml
• fix example settings
• remove dist files
• fixed small bug
• fix SENDER_ADDRESS vs SENDER_ADDRESSES
• add stuff to /dist
• update build process
• finished up the ui

I got all the ui for the settings page working! I even have a pills type thing so you can select multiple emails for it to go to. It will check if it’s a valid email as well with regex. Next up: categories and search options for things other than cars!

NOTE: the app password in the photo is old I realized I leaked it and then changed it. Now it’s different.

I finished the scraper ui! now you can scrape and search, all from the ui. next I’m working on the settings page and then I’ll work on making the search not just for cars. I also need to work on the category feature.

I’m keeping on working on the ui, and I finished the search page completely! You can filter by make, model, price, year, mileage, and even search the database with a search bar! One feature I want to add soon is being able to scrape into a category so you can distinguish between two different things you’re looking for. For example if I wanted to look for couches but also at the same time cars, I could scrape the cars into the “cars” category and the couches into the “couches” category. Also I got images working, and started on the scraper ui. I have the layout for it currently but it is non functional. Working on that next.

I worked a bunch more on the frontend ui, mostly getting the inputs the way I want them. I spent most of that time coding a dropdown menu when you search for makes and models of cars that suggests cars to you and autocompletes and things like that. Cool, but I need to lock in on getting it working before flavortown ends.

I did get function sharing with eel set up and I’m now calling the search function from my javascript site which is super exciting! Now I just have to create a function to add each listing to the page in js, and call my search() function from js every time the filters change oh and also change the search variables on input of the filters (so you can actually filter the results with the filters). I also started scraping the image urls into the database so I can display them in the ui instead of just words. That was fun.

Here’s a picture of the search results getting logged to the console in javascript, taken from the python function.

I’m working on writing a web ui! I found a python library, eel, that lets me write an html page and interact with my python script from the javascript of that page, allowing me to make a fully functioning ui in the browser. I did some design in figma and I think it turned out pretty well. Now I’m working on the structure of the page, and after that I’ll add the functionality. The ui that I’ve made so far is pretty broken but I’m working on it give me time

The ui will be the best way for the user to interact with the program, as your average person doesn’t know how to use the terminal and code.

I added arguments! Now when you run it you can do things like scraper.exe --windowed --purge-db or even scraper.exe --help which includes help text. I did this for both scraper.py and search.py.

I also created a file called convert-to-csv.py that converts the databse into a csv file, which I’ve been playing around with a bunch.

I added a nice loading bar at the bottom of the terminal when you’re scraping that gives you an estimate of the time remaining and just looks nice.

I converted the whole thing to a all around marketplace scraper! Now it doesn’t just look for cars, but it can look for anything! I spent a long time refactoring everything and probably accidentally sent my mom some emails by mistake

I also updated the setup script and the readme. The readme was just basically find all the instances of “car” and replace them with “listing” lol.

I’m planning to work on a cli type deal soon and being able to send options to it with parameters when you run it as well.

I fixed up a few bugs in how the scraper program runs, including trying to launch chrome multiple times because it wasn’t working consistently (at least on windows).

I’ve finished the setup.py file to provide the best ease of use. It does things like automatically find your chrome extension and create a new profile folder. It can also help you setup email notifications and filters. I also added a build.yml that runs on every push that builds the python scripts into exe files with all the packages built in, no install needed.

Changelog

• fix up some stuff in setup.py
• fix some slight bugs
• tidy up setup ux
• zoomed out and fixed some stuf
• update buid.yml (again)
• update build.yml
• update build.yml
• fixed up some input issues
• make scraping faster
• update build file
• add more setup options and steps
• add build workflow

I’m trying to improve the experience of setting up the scripts, hopefully that makes it so voters can test it more easily. I’ll also compile the scripts into .exe eventually so it’s easier in that respect as well. I have so far made the setup script for setting up the scraper, which opens the chrome profile so you can sign into marketplace, and configures some other things, as well as finding the chrome installation and configuring the profile directory to start in. I also made the script for setting up the filters, and am working on setting up email setup script.

I also added email notifications! I’m using smtplib to connect to smtp.gmail.com and send messages from there. Right now I have it configured with task scheduler on windows to run every day and it’ll notify me if it finds any cars that I haven’t seen before.

Really I’m happy with how it’s coming along and I’m still using it to search for cars!

Sorry for the long devlog it took me a bit to set up the flavortown project. So I built a custom facebook marketplace scraper for scraping cars.

How it works:

You have to do a bit of setup in settings.py for it to work, but all of it is well documented with both python comments and in the readme.

First, you run scraper.py. This opens chrome in a headless mode and goes to the marketplace url provided., scrolls down and every 3 scrolls runs runs a selector and saves all the cars that it finds to a database, along with the metadata mileage, url, title, year, location, and price.

To filter the results, you configure settings.py with the filters you want, then run search.py. This will filter through all the cars in the database, and can even send an email if it finds one it hasn’t found before. I set it up with windows task scheduler to run every day since I’m looking for a car.

The whole thing is really well documented (or so I think) and is relatively easy to use. I’m still working on making the email part of it easier to set up, and I need to add an example secrets.py.

The filters are pretty robust and hopefully I’ll add more categories of items along with more db tables to expand it beyond just cars.