Dum20:


Okay great.

1. Firstly how did you start?

I started with self learning. But now am taking some courses online (edX, dataquest and Datacamp)

2. What are the courses you took to get to this stage of competence.

Not a lot. I don't even think am so competent just yet. Still work in progress. But most of the courses I have taken have been by myself, online.
Scraping Nairaland was with BeautifulSoup, and I just used the documentation.

3. For how long have been in Data Science

Six months, give or take. I have been learning python for 9 mnths.

My background i am taking a Bussiness Intelligence course on Udemy. I have taken the statistics, SQL, Tableau parts of the course. I am just started the python section. But i still feel inadequate.
I have also taken courses on Excel Power tools and Power BI.

Try to get into the act of doing, not only while you are watching the tutorials, but also at other times.
Doing makes you learn a lot. Start with something that interests you. If you like Nairaland, you may actually start with that nairaland dataset.

N.B. Can you give a step by step guide on how you got to final charts above.
You do not need to go into details. Something like:
1. You used XYZ to extract data
2. You used ABC to clean the data
3. Used MNOP to visualise the data

-Use jupyter notebook. Most of the libraries are pre-installed
-Import the required libraries
-Scraping was with beautifulsoup and you can consult the documentation online
-used pandas to clean the data. It was very dirty.
-Used seaborn to visualize the data. There are other options, including plotly.

There are a lot of courses to study. I am wondering if i should stop for now and practice real life examples of the subjects i have learnt.

Make sure to practice real-life examples. That's something you should always do. You can stop studying for now. Practice, and when you have challenges, go back to refer to your notes or to the videos.

Zabiboy:

Nice one @ cochtrane
I'll use Tableau/Power bi to analyse mine..
Pandas needs more logic...not to talk of matplotlib ...
Although i'll still use them ( pandas nd mpl) later

Cool. Share the results with us when you are done.

Dum20:


Bro this is fantastic and awesome.

I have plenty questions for you later

Thanks! Just drop a message here if you've got a question.

And how frequently has a topic containing the "Buhari" keyword appeared on the frontpage? Comprises about 5% of all topics in the past year.
You would think this is not such a big number, until you see that the frontpage topics that contain the keyword "Tinubu" or "Osibanjo" were only 0.63% and 0.0069% respectively. Crazy, right? Nigerians do talk a lot about the number one man.

I feel the fact that politics dominate discussion on Nairaland is some evidence that the site is becoming more like a news website than a social network. It's probably firstly a news website, then it is a social network.

What more can we discover from this data?
-A time analysis of post frequency? using cumulative sums
-A time analysis of front page posts especially for the top posters? using cumulative sums
-A bag of words model for frontpage titles over the past years. Would be nice to see the most frequent words which make frontpage
-The shortest intervals between when a post is created and when it makes frontpage
-The last time some infrequent sections made frontpage
-The sections with the most frontpage material per month (probably politics, but need to confirm)
-How often have you or your friend or someone you know made frontpage? -you really may want to find this out, if you think you've been on the frontpage before

I leave the rest to you!

What are the longest years between the creation of a post and when it made front page? (first figure).

And has lalasticlala posted anything associated with snakes in the past year? Apparently not. (second figure).
The data shows that hasn't been the case.

As a budding data scientist who visits NL often, it's not surprising that you start to get more than interested in the some of the topics making front page and how frequent topics from individual sections reach the top. I have been looking into this for a while and thought it would be nice to do some investigation in this regard. For example, which section makes front page most often? How often do we see programming topics get to the front page? Who posts more often on the front page? Is it really lalasticlala, as is frequently supposed, or is it someone else? What exactly has been the relationship between lalasticalala and snakes over the past year? Some people think he loves to push snake topics to the frontpage more often than other topics. What else can we learn from the topics making frontpage? Like for example, are they mostly about Buhari or something else?

To this end, I scrapped the front page data and obtained more than 28,000 records. You can download this data set I obtained here on my github. If you are a data science enthusiast who also likes Nairaland, this may be good motivation to dig into a topic that interests you. You will find a metadata file in the sublink as well and can investigate what the attributes are about. You've got titles, links, sections and time that posts made front page. It's a year of data from 31st May 2019 till date. It turns out to get the whole frontpage information may need more than 230,000 records! That's huge, and probably not so wise to collect for a quick, lazy analysis. Except, of course, you have business motives

For me, I was interested in a few topics.
First, from which section did we get the most frontpage material over the past year? Apparently, it is "Politics". It trumps everything. "Celebrities" come a close second. Not surprising, right? What with the volume of Bobrisky posts and co. And then "Crime" comes third. Does this point to a high frequency of crime in Nigeria? I leave that question to you. "Programming"? Didn't even make bank one time!
The fact that politics make frontpage more often clearly shows that top on our discourse as Nigerians is probably politics, if Nairaland reflects a microcosm of the Nigerian environment, which I feel it does.

Who posts more often on the frontpage? Not lalasticlala like you might think. It's a person called dre11; at least over the last year. Maybe you know him, may you don't. Lalasticlala is not even in the top three.

One quirky thing I found, however, was that the time it takes for a post to get to frontpage has a heavily right-skewed distribution. Before plotting this, I lazily thought it might be dome-shaped, cos...well, a lot of things in life are and it shouldn't be unusual to have this behave the same; few make front page early, few late, and most are in between. Right? On the contrary, the reality is skewed. I feel the heavy skewness and bimodal attribute probably points to deliberate human intervention. Most posts make front page early, not late. They are created and in little time pushed to the front page. Evidently in a deliberate fashion. Else the data should be dome-shaped, don't you think? Anyways, that's what my data shows. Maybe, better insight could be derived though if one scraped randomly over the past several years in order to obtain a truly random sample.

And there were a few threads which made front page late. Very late! In the past year, we have had threads from 8 years ago make frontpage. Yes, 8 years ago! Thats's 2012. And then there are those that were initially posted 5 years ago before they made front page. Perhaps you can find more if you looked into the data set?

Anyways, getting your hands dirty with a data set is always a good way to learn data analysis. If you need help with navigating this, you can buzz me.

einsteino:


You have a very good point but in the last week or so, the streets have been busy due to summer. Not many people are adhering to the sit at home order anymore.Hopefully, after the lockdown I would retake the pic so we could compare.

In any case, one can see from the picture that some effort was dedicated to urban planning, unlike Lagos.

Of course! More effort is dedicated to urban planning.
I like the pic from the roof top.

einsteino:
I took this one from the rooftop of a 32 storey building. At least it gives a good view of the "over crowded and chaotic" Toronto lol . This area is Weston at Lawrence.

You are forgetting the city/province is still under lockdown. We can't make anything of the "overcrowding or chaos" at this time.

CsRockefeller:
After over 25 years, I wish I wasnt an introvert.

Men!!....... I'm tired and exhausted. Even when I try to go out of my way to make friends or meet people it never ends well.

Dont wish to be one ever again.

All of this self-pity is unwarranted. You need to start thinking of your qualities more from a strong or work-in-progress standpoint than from a weak one. No one is immune to being tired or exhausted or lacking energy. Studies have shown repeatedly that that has little to do with your personality type than it does with your self-motivation, inner drive or determination to succeed. To think that not being introvert (or even extrovert) is the way out is a highly erroneous route to take. Maybe if you are not human, then that's your way out; but I'm digressing.

There is a lot out there in the world to face regardless of one's personality type. Everyone handles it differently. Those who are talkative talk to calm their nerves, even when they feel anxious. Those who aren't talkative on the other hand, stay quiet to calm their nerves. Different people, different reactions to the stimulus called environment. Learn to handle your own reaction well and you shall thrive.

Habbeys:
THE SHOCK OF DIFFERENCE (Musings of a Nigerian).
I am not one who is liberal in the usage of my tongue for the purposes of cursing, but I may have been tempted to so do lately. What has prompted this temptation you may want to ask, well, quite a few things. For one, the difference in the...

I tried so hard, but I honestly can't relate to this. Doesn't reflect my experience as a Nigerian

Jbelieve:
Hello all complement of the season. Has anyone invested in REITS on this platform in Canada? If yes, please share you knowledge on it especially the risk. I'm working towards building a diversified investment portfolio.

If you also know other safe and risk free investment please share your thoughts. Thanks.

You should look into TFSA with Questrade. That way you can put your money in index funds like Vanguard.

remiaig:
Diary of a new working Canadian Permanent Resident . Hope someone finds this encouraging and useful

Nice observations. Good luck on your new role.

Brilliant!

Is anyone here going to GTA, Ryerson, UofT for January 2020? Looking for fellow students who may want to form connections or share accommodation.

Wait o. Are you guys saying it's almost impossible to earn six figures in Canada?

Enculer2:


Congrats but it was not magic.

You made it because you met the criteria not through any miracle.

Just what I thought too

Jarus:


Dana's first engine failed about 30 minutes mid-air. Closer to Lagos than returning to Abuja.

He lost the the other engine later.

You are wrong. That's not what the report said.

bily:


Funny thing is, he may not be a native of Canada. His grand parents or parents may have come in to Canada as immigrants.

He/she sounds like it. The English is not particularly stellar.

DOMINGOPAPI:

Please I am fighting through frustration and depression with my last breathe. I need job, support and admonition before it's too late. Life has never treated me this cruelly and unfairly and I am at my wits end. I have tried and exhausted the only thing I was hanging on to (HOPE), The future seem so bleak and I am scared of tomorrow. My number is. I need Good spirited, mature forum members to speak to...

Take it easy bro. No matter the situation you find yourself, just know things could be worse. Always have that at the back of your mind. You feel okay when you tell yourself things are or will be okay.

chanceux:
Good evening family. Pls I have 2 issues as regards my reference letter. During my nysc, I worked in an intelligence agency and when they provided a reference letter, they didn't include a phone number or email address (don't know if it's for security reasons). So I'm wondering how IRCC can contact them for verification if need be

Secondly, when writing the date I worked there, they didn't spell out the day, they only wrote from month this to month that with the years.

Pls what can I do? Can I go ahead and submit the letter like that? Do I raise a LOE?

This is a tricky one. Your letter apparently doesn't meet the requirements for a work reference letter. You'd need to find a way to have them put the contacts of the HR person on that letter or at least obtain his/her business card and attach that to the letter. There's got to be a way people reach them, right?

Dfavouredone:
Please Fam, Just a quick question. I have gotten my ECA and IELTS and my current CRS is 456. 

My marriage is next month. Is it advisable to wait after my marriage before submitting my profile or to submit it now? If I submit my profile now and I receive ITA before my marriage next month, can i still edit my marital status after ITA, before AOR and would that affect my CRS score?

Read the links under the first question in this summary. You should find satisfactory answers there:

mosbiod:
Been following this thread for a while since I landed in January, but the hustle has been real that I hardly come to nairaland again like I used to when my application was ongoing. I hope y'all enjoying Canada. My boss @Cochtrane I hail oo. Long time

How far, boss? I'm sure the hustle is going great.

ChiefVal:
Hello all. Please does anybody have any experience in going to Canada with their pet. Maybe sending the pet later or going with the pet as you travelled. Please share the experience.
Thanks

This is gonna be a whole lot of effort, I can only imagine. You should look through canada.ca and see what the requirements are for bringing a pet into Canada. There's detailed info there on every kind of pet, from birds to reptiles, dogs to fish, etc.

olufemikayode:
Good afternoon everyone, please I have a question regarding work experience; I did my POP December 19 2018 and I immediately started another job on the 20th. My one year experience ends November 19, 2019 right? Somebody Please Help Me.

The maths doesn't add up well if your 1-year experience ends on November 19, 2019. It should be December 18, 2019, shouldn't it?