Facebook Car Scraper

I fixed up a few bugs in how the scraper program runs, including trying to launch chrome multiple times because it wasn’t working consistently (at least on windows).

I’ve finished the setup.py file to provide the best ease of use. It does things like automatically find your chrome extension and create a new profile folder. It can also help you setup email notifications and filters. I also added a build.yml that runs on every push that builds the python scripts into exe files with all the packages built in, no install needed.

Changelog

• fix up some stuff in setup.py
• fix some slight bugs
• tidy up setup ux
• zoomed out and fixed some stuf
• update buid.yml (again)
• update build.yml
• update build.yml
• fixed up some input issues
• make scraping faster
• update build file
• add more setup options and steps
• add build workflow

I’m trying to improve the experience of setting up the scripts, hopefully that makes it so voters can test it more easily. I’ll also compile the scripts into .exe eventually so it’s easier in that respect as well. I have so far made the setup script for setting up the scraper, which opens the chrome profile so you can sign into marketplace, and configures some other things, as well as finding the chrome installation and configuring the profile directory to start in. I also made the script for setting up the filters, and am working on setting up email setup script.

I also added email notifications! I’m using smtplib to connect to smtp.gmail.com and send messages from there. Right now I have it configured with task scheduler on windows to run every day and it’ll notify me if it finds any cars that I haven’t seen before.

Really I’m happy with how it’s coming along and I’m still using it to search for cars!

Sorry for the long devlog it took me a bit to set up the flavortown project. So I built a custom facebook marketplace scraper for scraping cars.

How it works:

You have to do a bit of setup in settings.py for it to work, but all of it is well documented with both python comments and in the readme.

First, you run scraper.py. This opens chrome in a headless mode and goes to the marketplace url provided., scrolls down and every 3 scrolls runs runs a selector and saves all the cars that it finds to a database, along with the metadata mileage, url, title, year, location, and price.

To filter the results, you configure settings.py with the filters you want, then run search.py. This will filter through all the cars in the database, and can even send an email if it finds one it hasn’t found before. I set it up with windows task scheduler to run every day since I’m looking for a car.

The whole thing is really well documented (or so I think) and is relatively easy to use. I’m still working on making the email part of it easier to set up, and I need to add an example secrets.py.

The filters are pretty robust and hopefully I’ll add more categories of items along with more db tables to expand it beyond just cars.