Yeah,I was trying to check my proficiency in data analysis, I decided to use the first 10 pages on Nairaland front page topics.

Web scraping( selemium, Bs4 , HTMLsession, Scrapy) , Python Pandas, PowerBI , Excel, SQL

I Web scraped the data using python Beautifulsoup (Bs4). This was done by getting all the front page topic links, lopped through each one of them to get ; the title of the topic, poster of the topic, time and date it was posted.

The output was then saved in csv file format.

I could have cleaned the raw data using python Pandas, but I prefer to used powerbi since I planned using it to create a simple dashboard for the analysis. Therefore PowerBI was used for visualization.

More of descriptive analysis for the month of February ;

Who posted most topic?
Time of the with most topic?
How does the Daily Topic trends look?

Nice one. Would like to follow your work. Please check your DM.

Nice one bro. I would like to get in contact with you

To further the analysis, I want to see the sections that produces top most topics on the front page for the current month of March This should give us the section that interest members most or the ones the NL management wants people to see. Based on the last 10 front page topics ( March 4th, today to the last 10 pages.

Enskynelson:
Nice one. Would like to follow your work. Please check your DM.

OKAY SeniorMan

Samboizz:
Nice one bro. I would like to get in contact with you

Responded to your pm!

Felixitie:
[b]To further the analysis, I want to see the sections that produces top most topics on the front page [/b]for the current month of March This should give us the section that interest members most or the ones the NL management wants people to see.

I swear I thought he would have included that in his analysis

At least 60% of those topics were posted between Monday to Thursday of the week. You still have some 40% of the topics left to grab over the weekend if you were busy going through the mid-week topics lol

Felixitie:
To further the analysis, I want to see the sections that produces top most topics on the front page for the current month of March This should give us the section that interest members most or the ones the NL management wants people to see. Based on the last 10 front page topics ( March 4th, today to the last 10 pages.

It is obvious that the political section got highest topic moved to the front page..

Even the programming section is missing, is it that people aren't interested in programming or IT related field?

Nice initiative

Nice work. Keep it up

Oga u too much. NL should just employ you.

Nice!

You are finishing werk.

LikeAking:
Oga u too much. NL should just employ you.

Nice!

You are finishing werk.

LOL. We are just trying to cruise with the little skills we learn. Thanks boss man.

How did you get the data? Does nairaland have an API or you manually read the lines and reg expressions to get dates and all?

Chuukwudi:
How did you get the data? Does nairaland have an API or you manually read the lines and reg expressions to get dates and all?

NL does not have an API. You have to scrape and parse whatever data you need using libraries like scrapy/beautifulsoup/regex.

Wow good work op
On my way too
Lalasticlala

extendo:

NL does not have an API. You have to scrape and parse whatever data you need using libraries like scrapy/beautifulsoup/regex.

Nice.
If the op Felixitie would let me have the csv file, I might be able to predict the most likely section subsequent front page topics will emerge from.

This can Infact be used to score the admins of all the sections and classify them in a way that reflects what section Nairalanders are most interested in, the job of the mods, etc.


I am sure I will find high correlation between snake threads and lalasticlala.

I wonder if I've got time for that. But it would be a nice idea for anyone trying to learn some Machine Learning.

Chuukwudi:
How did you get the data? Does nairaland have an API or you manually read the lines and reg expressions to get dates and all?

Web scraping bro, with Request & Beautifulsoup..

Chuukwudi:


Nice.
If the op Felixitie would let me have the csv file, I might be able to predict the most likely section subsequent front page topics will emerge from.

This can Infact be used to score the admins of all the sections and classify them in a way that reflects what section Nairalanders are most interested in, the job of the mods, etc.


I am sure I will find high correlation between snake threads and lalasticlala.

I wonder if I've got time for that. But it would be a nice idea for anyone trying to learn some Machine Learning.

This idea is cool bro, if you are still willing to have the csv file and work on it, I can forward it to you. Machine Learning no be beans ooo..

Felixitie:



It is obvious that the political section got highest topic moved to the front page..

Even the programming section is missing, is it that people aren't interested in programming or IT related field?

What percentage of FP topics were from Programming sub? Programming threads have started hitting fp now a bit more than before. Cam you extend your data harvesting to see percentage increase of Programming threads?

airsaylongcome:


What percentage of FP topics were from Programming sub? Programming threads have started hitting fp now a bit more than before. Cam you extend your data harvesting to see percentage increase of Programming threads?

Yes sure I can,apparently it is a nice insight to look at. BTW,do u mean percentage difference in month on month? And from which month specifically do u want me to look at ? Jan - February or December - January etc. I hope you understand!

Felixitie:


Yes sure I can,apparently it is a nice insight to look at. BTW,do u mean percentage difference in month on month? And from which month specifically do u want me to look at ? Jan - February or December - January etc. I hope you understand!

December 2021 to January 2022 would be interesting. Just to see the percentage difference in Programming Section thread that had hit FP.

I'm a data lover and with the data you have to many thoughts have come to my head. Which section noticed a decline in threads hitting FP?

airsaylongcome:


December 2021 to January 2022 would be interesting. Just to see the percentage difference in Programming Section thread that had hit FP.

I'm a data lover and with the data you have to many thoughts have come to my head. Which section noticed a decline in threads hitting FP?

Definitely I should be able to come up with that information. I will extend my scraping to December. Obviously we have a lot of insights to derive from the dataset.. but for what purpose actually? NL no send me message.. Lol

Kudos brother..

Checking the url of each topic, I noticed they fellow different pattern( unlike some web where you can easily loop through multiple pages with range of page numbers to perform the scraping )...

My question is, do you manually copied all the url link you scraped or there is a way you go about it?

Wow

I also used web scraping to redesign Nairaland into a new app.
I used PHP / Laravel and React js

This topic of yours will look like this ( check the pictures)

Link at : https://nairaland-v2.netlify.app/topic/7013550/nairalands-february-front-page-data

Gentleman001:
Kudos brother..

Checking the url of each topic, I noticed they fellow different pattern( unlike some web where you can easily loop through multiple pages with range of page numbers to perform the scraping )...

My question is, do you manually copied all the url link you scraped or there is a way you go about it?

This is how it works, with beautifulsoup.
You get or scrap the link for all the topics you need in form of a list and have it assigned a variable, then you loop through each of them. Those links have the same html setup or tags..

Felixitie:


This is how it works, with beautifulsoup.
You get or scrap the link for all the topics you need in form of a list and have it assigned a variable, then you loop through each of them. Those links have the same html setup or tags..

Nice one. Just curious, can request and beautifulsoup bypass Amazon's anti-scraping mechanism.

Scraping Nairaland with php also went smoothly since they didn't prevent scraping.

mustyx001:



Nice one. Just curious, can request and beautifulsoup bypass Amazon's anti-scraping mechanism.

Scraping Nairaland with php also went smoothly since they didn't prevent scraping.

Selenium works, then pass it to beautifulsoup

[i][/i]Thank you

Felixitie:


This idea is cool bro, if you are still willing to have the csv file and work on it, I can forward it to you. Machine Learning no be beans ooo..

mustyx001:
Wow

I also used web scraping to redesign Nairaland into a new app.
I used PHP / Laravel and React js

This topic of yours will look like this ( check the pictures)

Link at : https://nairaland-v2.netlify.app/topic/7013550/nairalands-february-front-page-data

it takes time to load lol..

Felixitie:



Selenium works, then pass it to beautifulsoup

do you run it on cloud or locally

Anon4:
do you run it on cloud or locally

On my local machine!