DataMina:
Hello everyone. I am a data analyst who enjoys webscraping. Nairaland has been my go to place to keep up with trending news asides Twitter, and being a lover of this platform I decided to build a webscrapper with Selenium and Python.

This scraper is designed to extract the thread topics the count of views, users, and guests from Nairaland.

The Scraped data can be used to develop a power bi report that can be refreshed each time Scraping is done. The published dashboard can be used to morning threads that generate the highest and lowest views. You can check out the project in my GitHub repository:https://github.com/StephDAnalyst/Nairaland

DataMina:
I use a different trick to do it. If you need the service you can let me know

BlackhatMentor:
A scrapper that scrapes emails, phone numbers with their usernames will be better.

This one u did isn't very useful

airsaylongcome:


How isn't this useful? A social media management team would use the report from this scrape to target threads and topics they should be driving contentand engagement with. Some of you need to think laterally

BlackhatMentor:


I don't just comment for commenting sake.

It's not useful...

Nairaland already provides that data so what's the point creating another one

airsaylongcome:
OP,

Is it possible to do the "impossible"? Scrape the entire site's content and dump in an LLM

airsaylongcome:


Ehhhh...what's the point? How about the OP learning? How about having a visual Dashboard instead of Nairaland bland numbers only without any visual perception of how those numbers compare? I don't get up and make random comments. But to say the OP's work isn't useful is absolutely not correct. It is useful. I find it useful and I'm sure there are loads and loads of non-data science or data engineering folks that will find it very useful

Babangidapikin:

Okay by the way can you visualize data

DataMina:
Hello everyone. I am a data analyst who enjoys webscraping. Nairaland has been my go to place to keep up with trending news asides Twitter, and being a lover of this platform I decided to build a webscrapper with Selenium and Python.

This scraper is designed to extract the thread topics the count of views, users, and guests from Nairaland.

The Scraped data can be used to develop a power bi report that can be refreshed each time Scraping is done. The published dashboard can be used to morning threads that generate the highest and lowest views. You can check out the project in my GitHub repository:https://github.com/StephDAnalyst/Nairaland

Cheryl463337:
why can't you use normal requests and BeautifulSoup which i believe will be faster than selenium?

BeLookingIDIOT:
You're doing something illegal while announcing it on the very platform

BlackhatMentor:


It's useful for learning purpose only.

I don't see any problem it offers a solution to sha.

DataMina:

I was just experimenting with it because when I tried using Octoparse (a badass no code tool) to scrap nairaland website, it couldn't work because the site didn't appear structured. So I decided to experiment with Selenium and it worked

DataMina:

I was just experimenting with it because when I tried using Octoparse (a badass no code tool) to scrap nairaland website, it couldn't work because the site didn't appear structured. So I decided to experiment with Selenium and it worked

airsaylongcome:


Illegal is pushing it a bit. Unethical, yes. But if NL doesn’t expose APIs for devs to legally consume data, then people have no option than to scrape shege from it.

airsaylongcome:


Think of it as first alpha. They definitely would refine it until they get to v1

Felixitie:


Tho, it seems the page loads dynamically making Bs4 hard to easily get the data out, selenium can load the page and render the javascript, then you may now use Bs4 to soup it and get the stuff (combination of Sele&Bs4). Scrappy works too easily.

You can as well grab all the front page topic links first and then loop through it using Bs4 to get all the data points , to improve the speed.

You have done so well.

Can we work on a portfolio project together using scrapy with splash or scrapy with playwright to generate leads, then we dump it into a database plus scheduling using airflow?

landiqa:
Can you develop one that scrape for phone numbers and validate the phone numbers for Whatsapp.

DataMina:

I can scrap Jiji for phone numbers but the validation on Whatsapp is what I don't know about

airsaylongcome:
OP,

Is it possible to do the "impossible"? Scrape the entire site's content and dump in an LLM

BlackhatMentor:
A scrapper that scrapes emails, phone numbers with their usernames will be better.

This one u did isn't very useful

DyingFetus:
I did something but not web scrapping just extraction of useful threads and posts by certain monikers using requests