Mrquote:

In most case it's illegal, but you just have to look up the robots.txt file to see what the site allows and what they don't allow.

I doubt if it is illegal to scrape . Site owners are only scared of the effect of bombarding their site with series of request on their website. So, The way you make use of the data you finally scraped makes it illegal or legal!

This is cool bro.

Why not just use BS4 only instead of starting with selenium, since bs4 is faster and lighter, also considering the fact that the jumia site is not dynamically loaded by Javascript.

Beuberry:
Anjofe died and his house was abandoned!
The African adage goes like this:
Ese girigiri nile Anjofe
Anjofe ku o, a o reni kankan
Erin ku o mogudu fi'je
Efon ku o, mogudu fi'je
Mogudu wa ku ko s'eni ti o je un o.
Na govt be Mogudu!

Am sure many people doesn't know the actual meaning of Mogudu, especially the yoruba folks!
Mogudu means the big Cooking pot (Ishasun)

Hope I am right!

Really want to know what these Chelsea players do in training sessions, especially Caicedo, Jackson and Mudryk.. And they go home smiling to the bank weekly and play nonsense on match day. Eye sore.

DataMina:
Hello Nairalanders,

I want to share my experience in crawling leads from Jiji. As a web scraping enthusiast, I've tackled various sites, but this one proved to be difficult. This is because the phone numbers are in the product detail page and you have to be logged in to get it. Another issue I faced was that the website is loaded with java script and uses infinite scrolling.

I was able to circumvent these road blocks by using selenium to login to the site with my personal details after which I saved the cookies as a json file. As the cite uses infinite scrolling I scraped all the sellers url links to a csv file. I then read the url links so as to crawl the site using the links.

Remember my login details have been saved as cookies in a json file, In the process of crawling the site again using the URLs, I applied the cookies from the json file. The crawler visited each seller's detail page using the saved URL links and clicked the "Show Contact" button to extract the contact information.

Retailers specializing in second-hand items like laptops can leverage this approach coupled with applied analytics on the scraped the data to gain swift access to mouth-watering deals, faster than regular visitor

You can check out the code for executing the project in my github repo: https://github.com/StephDAnalyst/JijiLeadScraping

Great job! Being able to handle the infinite scrolling on jiji page, different websites with different technicalities when it comes to data extraction. I did a project for a client recently, Airbnb and zillow, omoh the zillow gave me wahala small, reasons were, all the Web elements did not load except you scroll the page bit by bit down to the button, and to be able to scroll the page you have to minimise the driver window., very interesting.
Nice one op.

blaskikv:
Is webscraping lucrative?

Data Extraction! Absolutely, especially if you can by pass anti bots/capcha mechanism on website + being able to integrate the data into a database and automation

airsaylongcome:


Illegal is pushing it a bit. Unethical, yes. But if NL doesn’t expose APIs for devs to legally consume data, then people have no option than to scrape shege from it.

To scrape 'SHEGE' from it. Lol.

DataMina:

I was just experimenting with it because when I tried using Octoparse (a badass no code tool) to scrap nairaland website, it couldn't work because the site didn't appear structured. So I decided to experiment with Selenium and it worked

Tho, it seems the page loads dynamically making Bs4 hard to easily get the data out, selenium can load the page and render the javascript, then you may now use Bs4 to soup it and get the stuff (combination of Sele&Bs4). Scrappy works too easily.

You can as well grab all the front page topic links first and then loop through it using Bs4 to get all the data points , to improve the speed.

You have done so well.

Can we work on a portfolio project together using scrapy with splash or scrapy with playwright to generate leads, then we dump it into a database plus scheduling using airflow?

Paddybaci:
I intend diving into data analytics courses like Advance Excel, SQL, powerB, Tableu etc.
I wish to know if it requires a specific Laptop?
And how about a laptop with the under listed specification
Brand : HP
Model : EliteBook 2540p
Type : Notebook
Processor : Intel Core i5 M540 @ 2.53GHz 2.53GHz
Storage Capacity : 500GB HDD
Maximum Storage Capacity : 2TB HDD
RAM : 4GB
Maximum RAM Upgrade : 8GB
Supported OS : Windows 10

SSD is your best bet against HDD, at least 256G SSD, and 8GIG RAM.
Be consistent and do more of project.
Happy learning

aluko360:
Studied Agric Economics.

Payroll Admin (9-5 job)

Data Engineer (remote job)

For data engineering, which tools you most used or work with?

Victorakay1:
almost completing courses on Microsoft excel,power bi and SQL...please link me to jobs as a data analyst

Ok boss. Most jobs are through refferals and some gigs, majority on web scraping though..

akaahs:

I want to learn data analysis, what ll be ur advice?

It's a good field bro, you have to learn to standout because we have many people into the loop already. Be grounded in SQL and communication skills. Learn Excel, Visualisation tool, one cloud skill and if possible Python because it will help when going further and expanding.

Good luck brother

Industrial Chemistry!

Environmentalist (full time)
Side hustle - Data Analyst
- Web scraping

Please do you have HP15 battery? What's the price? Follow come or new?

YoungCabal:
Lol! We are cool, man.

Sure, it can work on every page, it depends on how much time you are willing to invest in coding it, there are selenium libraries for some languages which you can integrate with a crawler you build and use regex pattern matching to determine which page is which, that's why I was against tagging it simple as you did since we both don't know OP 's full intention

I feel you bro, the script I developed won't work for all the pages cos it was for personal project. I said simple for the fact that I have seen many tough websites to scrape compared to the easier jiji type. Thanks brother.

YoungCabal:
if it's not that complex, why don't you just paste the source code here for him or the full instruction on how to do it ? admit it, it's something that demands quality attention not just something you can run over.

Nigga calm down, just tell me you need it. If it demands quality attention then it will not be free,otherwise he should do a personal search and learn how to do it if he can't pay for it. Besides, do you think the script is going to work for all the pages in jiji.. Abeg move.

YoungCabal:
It's not even the infinite scrolling alone, you have to click on some data to unhide them, beautiful soup is not the right tool, even with selenium, it won't be an easy task because you either go category by category or build a mini js enabled crawler to index the site

I laughed when I saw someone comment he can show OP how to do it with beautiful soup,

Impossible for Bs4 alone, but selenium will work for sure, the clicking of buttons etc., depending on what you want to scrape from the site..not that complex..

I have done a project on it before, beautifulsoup will not handle the jiji site due to the infinite scrolling pattern of the website. You have to use selenium + Bs4 + page rendering to render the javascript before scraping.

Umarfarouksalau:
I find it more comfortable and consistent having to call sql from jupyter, cleaning and visualizing in same notebook. Really works well for me though.

Well, it is about getting results and comfortability. But most time, you need to have interactive and share-able dashboard to the management or colleagues at work, no be everybody understands jupyter or coding.

airsaylongcome:
Felixitie,
Great job! Very insightful analysis. Especially the time of day to get the most views. Tableau is such a powerful tool for Dashboards. And the relative ease of getting these graphs makes it such a useful tool. Na bootleg version you dey use? Laid my hands on a 2019 portable version and I've been dragging my foot on learning it. With my second job finally coming to an end I may just dive into learning it for the rest of the year and leverage the knowledge in 2023. Do you do any automations? I've been learning Microsoft Power Automate and Unipath to automate the process of data scrapping. Do you use any of those?

Would also be nice if NL could share data of locations and devices that view threads.

Thanks for the comments boss man. Tbh I just started learning Tableau, though I have experience using Power BI. Right now, I don't have full flexibility of automating due to the limited functionality of the version I use, and I don't use MS Power Automate.. but you can schedule the scraper and refresh the dashboard.

fnep2smooth:
Nice one bro... These are the simple features that Nairaland supposed to add.

Absolutely.. a lot of them . Thanks.

folake4u:
This is beautiful and well detailed.

Thanks thanks

silento:
Omor op drop u contact make credit some airtime this you analysis and presentation is clean and professional

thanks man!

Deasegun19:
beautiful insights op

Many thanks!

Umarfarouksalau:
Great job there…. So what’s the advantage of Tableau over matplotlib in .py?

Interactivity and flexibility,especially when you have to create and design a dashboard without hard coding., and imagine if your data is coming from a database in real time!

RepoMan007:
Hmmm. Nairaland is an entertainment / leisure forum.

Programming, technology, career, etc are guest sections.

You are right sir, but why more than half of the topics on front page are about politics? guess most members are political fanatics

Thanks boss!!

qtguru:


This is beautiful OP

These sections got the lowest topics on the front page, less than 10% of the total topic (1344) and Programming section got None.

What were the hours of the day those topics were posted? Data no dey lie ! Posters should focus more at posting btw the hour of 1-7pm for more coverage.

INTROVERT:
Bigot filled political section.

These sections got the lowest topics on the front page, less than 10% of the total topic (1344) and Programming section got None.