One minute silence for NLC who couldn't put a smile on our face this morning...

shsss
may the good Lord save us from minimum wage.

In the early hours of this morning, -or in the morning hours early this morning - I decided out of sheer cynicism of Jumia Black Friday (being very unlucky in the past), to write a script that crawls jumia Pages and get names of items with at least 50% discount.
Well it worked and as it appears, I felt the urged to fork it on github for my fellow programmers to take advantage of the upcoming Black Friday deals, all you have to do is run the script using --scrapy crawl Aragog -- from the root directory.

You can change the threshold discount rate from the wicked.py file in the spider folder and you can even change the urls to crawl from the same folder.

It produces the result in the form of excel files, pretty basic but with a few refactoring you can do wonders with it.

Requirements:
Python 3 (best installed with Condamini or Anaconda)
Pandas (comes with condamini or Anaconda)
Scrapy (pip/conda install scrapy)
Virtualenv (if you will)

Link to the repo on GitHub

https://github.com/Drjimoh/BlackfridayCrawler


Let me know if you need anything. Have fun scraping jumia and pray jumia doesn't change their robot.txt file till after the Black Friday.

By the way, the spider was able to crawl 11 heavy jumia pages and return outputs in 26.93 seconds (for those that believe python is slow).

Wow, that really worked out? I want to do something similar, but I'm not sure how to do that. Could you please help me out with this? It's identical to Jumia Black Friday, but I'm willing to write the script for more websites. I want the script to list at least the items with a 20% discount and more. Considering the number of websites I'm willing to use in my project, and I'm not sure about the scrip and what else I need. I have already contacted https://proxycrawl.com/scraping-with-worldwide-backconnect-proxy for their back connect proxies. I hope they'll make it work.

Good morning Drjimoh. Pls how can I reach you? I’m a newbie to web scraping and I’m in dire need of someone who can direct me. I was really impressed with this project and it’ll be a pleasure to be advised by you.