₦airaland Forum

KlausMichaelson:
Lols Is I never said df.describe() gives ANOVA Sir. if it gives ANOVA why then should I still be bothered about ANOVA this or ANOVA that.

What I said was that df.describe()
brings up a sort of table showing some statistical analysis(Mean, Std, etc) of any data you provide for it and not ANOVA sir.

Anyways thanks for your advice.

As a matter of fact, df.describe() just gives you an idea of the data you are dealing with. Like a statistical summary. Although, there could be more than a way to carry out in-depth inferential analytics, scipy library in conjunction with researchpy and pandas should always give you what you want.

Modified:
Steps in determining what tests to use in a statistical analysis;
1. What type of variables are you dealing with? Numerical or categorical?

2. What type of analysis are you doing?
a. comparison (mean,median,proportion); this is where you use t-tests, ANOVA, etc.. depending on the distribution of the data(parametric or not)
b. Relationship between two variables (say gender and smartness); this is where you use correlation (Pearson or Spearman rank)
c. Predicting one variable from another (say exposure to smoke predicts the risk of lung cancer); this is a Regression task

3. Number of groups involved (say the effect of Chloroquine in Covid 19 and non Covid 19 patients - 2 groups)

4. Distribution of your data (Normal or not); there are different ways to determine this.

I believe you now understand the point I was talking from when I mentioned theoretical knowledge of statistics.

KlausMichaelson:
Honestly, the reason why I'm very concerned about the use of ANOVA is simply because our Lecturers are always talking about it. I can actually do the calculations by the use of the various formula to get the required values for the ANOVA table. That's fine but what if I have a very large data?

The truth of the matter is that they wouldn't give me a straight forward answer as to the real life application of the ANOVA design neither will they tell me when to use it specifically whenever I ask them that question. They just beat around the bush.

ANOVA actually stands for Analysis of Variance and to my understanding, it is used to analyze Data that comes with a number of replications for a given Sample. It helps determine an experiment's Standard error, and some other parameters which I can't remember. My school books are far away from me at the moment.


Anyways, the truth of the matter is that you're right about the fact that one needs to be grounded in not only the Practical but the theoretical part of Data Analysis. Why I felt that ANOVA table could be gotten from any of these libraries or tools is simply because of a code I came across on pandas **df.describe( )**. It brings up a table showing some statistical analysis of any data you provide for it.

Maybe there is no ANOVA design from any of the tools or libraries but there is absolutely no problem. I'll work towards my other targets.

You don't need any lecturer to tell you the type of statistics to use, your data should point you towards that. You really need to learn more about inferential statistics. There are many statistical tests available and a lot of factors need to be considered before choosing the right one. As for pandas, I wonder if you know what being an intermediate implies. df.describe() does not give you ANOVA. You still have a lot of homework to do.

KlausMichaelson:
I'm so happy right now

I mean it's already a month and 6days in this journey. Before now, around 2019 or so, before school resumption I watched some videos on python but I didn't take it too serious. And yes I have a good foundation in Excel since my 300lvl cos I use it to plot assignments on graphs(A bit complex ones) and I make lot of money from it . I also understand the syntaxes and it's operations but I would still love to know more.

By June 20th 2020(last month) I saw this thread and I became very serious to know more about Data analysis. So serious that I had to Deactivate my Whatsapp cos it's always taking my precious time away.

Now here I am today and here are the things I have learnt after staying away from Social media for a month and 6days.
(1). Python (Intermediate tho )
(2). Pandas (Intermediate) although I can do almost anything with it but I won't still call myself a pro. Pandas is much better than excel as it could run many cells operations.
(3). Matplotlib (Same as pandas, my wonderful friend)
(4). Numpy ( still a learner cos I haven't seen much of it's use). And lastly
(5). Pyplot (my very good friend also)
Currently I'm leaning Seaborn and Powerbi.


Honestly speaking I know there are much more to learn . But seeing that I'm still a student, I feel I need to focus more on the ones that are more important to me as a student. So my targets are:

(1). Master all these tools in Data Analysis I've learnt so far. Later on after my studies, I'll learn Sql, Tableau and others.

(2). Find out which tool is best for ANOVA design table.

(3) Do Data Analysis for your fellow final year students undergraduate report and get paid
(4). Conduct Seminars for Students in Data Analysis and get paid

Please do well to profer any other library or tool to help meet my targets especially in the ANOVA design.

Thank you @Ejiod for the wonderful thread and to everyone on this platform

Any reason why ANOVA is specifically mentioned?Do you even know when to use it in a statistical analysis?
Are you not supposed to focus on all the concepts of descriptive and inferential analytics if your aim is academics?

See, data analysis is more than just mastering the tools, the theoretical knowledge and application of statistics are equally important.

BelieverDE:
I have been learning Excel for close to 4 months. Everything you described above can be done with Excel.

If you load your data to the Data Model or convert the table to an Excel Table, you can perform calculations to a column.

You can also design the graph to your preferences described above.

Flexibility.....

Hi there! I'm organizing a free virtual class on Python fundamentals. If you'll like to be part of this class, quickly fill out the form below as the tutorial commences soon. https:///30Kr1Nn


I don't know why the link gets truncated, you can just insert bit dot ly after https://or send me a mail

Semtu:
Programming Gurus.... Am currently working on a project, I've succesfully created half of it which is building an accelerometer based hand-gesture controlled robot.

Now I want to add video surveillance to it by using a webcam and pir sensor for motion detection, please has anyone done a project like this before ?

I don't really know much about embedded system but I think you should be able to achieve this using OpenCV (python or C++)Raspberry pi and webcam.

Chuukwudi:
Hallo guys.

It's so disappointing that a Microsoft surface pro laptop that cost me about $1000 to acquire can't even do some of the things I want it to.

Right now, I need to train my neural networks and consolidate some knowledge but can't, simply because the laptop doesn't have an NVIDIA CUDA capable GPU but an Intel(R) HD 520 GPU.

Have you a walkaround for this? The online course I'm taking requires it. I can't even run some of the codes without it.

I hope I don't have to suspend learning Deep Learning until I'm able to buy another laptop.

While I still train DL models on my Elitebook 2170p Core i5 4gb RAM....

You can try Google colab.... things should work perfectly there.

Chuukwudi:
I know Andrew Ng. Please, The other two names... Who are they? Are they nice, are their courses free too.?

They are all great professors of computer science who have contributed significantly to the field of DS.

stanliwise:
This algorithms would be automatically flawed as most topics and post are constructed in a way to fit multiple categories. Tagging will be a good suggestion in my opinion.
The problem of Nairaland forum is the integrity of user within it. Users are verified usually by mail only and new user have same power as old and trusted users. This means there is no rule specific to user credibility and everyone is anyone aside admin. Which has a good and bad side.

Are you saying it is possible for a topic to be found in more than one section? While tagging is similar to topic categorization, they do not serve the same purpose. More so, I think Nairaland already implements some sort of tagging by giving you topics related to the current one you are viewing.

donproject:
Have you done Machine learning, Deep learning, Artificial intelligent, Big data analytics.
2 days you say, you must be a comedian. Even learning ordinary Numpy by heart takes a lot of efforts...you have pandas, matplotlib viz seaborn, plotly, power bi. Don't let me even mention tools for ML. Have you learnt art of web scrapping.

Can you do simple analysis if I give you raw data? Sit down there and be deceiving yourself. Something that some people spend reasonable amount of years getting accustomed to, you claimed you've seen it all in 2 days.

Can you even define data science to start with? If I tear you slap ehn.

Leave the guy alone. He seems frustrated. If people like Andrew Ng, Yoshua Bengio, Sridhar Mahadevan can say they are still learning as far as data science is concerned, then I wonder who the OP is to tell us there's nothing in DS.

blife2:
well i feel data science is hard than web dev but its easier than learning JavaScript off heart what i mean is started data science 2 days a go and it doesn't meet the hype sometimes i think is this what ill be paid to do

Quick question
Can you tell us what you've learnt in those two days?

Nairaland remains Africa's largest forum with over 2M members. As such, several topics belonging to different sections hit the front page per day with posters having to choose the right section their posts should go to. If you are a regular user of this forum, you must have seen one or two occasions where posts end up in a wrong section. In order to tackle this issue, I feel it would be better if Nairaland can embrace state of the art approach by embedding some ML/NLP algorithms that automatically classify topics into the right sections, rather than leaving posters to choose the section for their posts.

What do you think guys?

gbolly1151:
I stay in ilorin and want to off load most stuff i have learned in python so far to people...if you are interested pm on 07060435624

Paid service but friendly

You can also check my thread on "common python programing practice"

Boss so you stay in Ilorin...that's really nice. I'll like to connect with you via whatsapp.

lokoventurex1:
Hello guys,

I'm an aspiring data scientist with no science foundation. In fact, I studied accountancy and finance. However, I'm interested to be become a data scientist.

I've found a school that's ready to train me for four months though, but they want me to complete a task in R and Phyton which I've no idea about for now.

I need a God's sent individual to help me look into this task. My success at the task given determines my admission for the data science course proper.

It looks very weird looking at the task .

I'll be very grateful to get some help here....

Can you post the task here?

KlausMichaelson:
Good evening House. I've been following this thread for a while now. I'm a lover of Data analysis.

I'm in my final year, an Engineering student. Sooner than later when this pandemic is over, I'll be analysing my project report. I'll be using some statistical analysis like ANOVA table and others to analyse my Data.

Now I will consider myself an intermediate in Python and in Excel. Also I'm a beginner in Pandas. I'm yet to learn Sql, MySql, Matplotlib, seaborne, etc. I only have gotten an intro into Pandas and it's really cool and interesting. I'll soon be done with it.


My Questions are
(1). Which among these; Pandas, Sql, Mysql, Matplotlib, seaborne among others, should I master so as to have a smooth ride in analyzing my project results. Which one also applies ANOVA table??

(2). Which of them aside Pandas (cos I've already started learning Pandas), should I start learning immediately I master Pandas? I'm very conscious of time. I'll really love to use this little time we have to master the more important ones for final year result Analysis. I want to make money from it when school resumes. I want to Master the ones I can so that I can help provide Analysis on other students project results and get paid for it.


Pls I need your Candid responses. Thank you in advance

If you are comfortable using python, you can achieve most of these tasks with it (just learn the relevant libraries)... However, the deeper you go into data analysis/science, the more you discover different tools and what they are best good at. My advice is that whatever tool you are learning, learn it well and practice with it.

For data manipulation, pandas is a must. For visualization, you need matplotlib and/seaborn. I highly recommend plotly for fanciful charts. I don't think you'll be needing SQL for now since you are not going to be interacting with DB.

Heylonmusk:
Good morning, fellow data scientist, I'm a Python programmer with machine learning knowledge but since I'm an undergraduate I can't apply for jobs now but I need projects I can do to sharpen my knowledge for now.





PS: I'm in my second year and I also participate in competitions on kaggle.

Download datasets and get your hands dirty with codes....

rastaxarm:
I prefer Flask over Django due to total control and flexibility.

Currently using it to develop a voting system for universities/polytechnics elections

Test - Still under development
https://digitavote.pythonanywhere.com/

No regret so far and the plugins/extensions are good too.

Be good with Python and everything will be ok. Though flexibility comes with great responsibilities

Nice work boss! I recently deployed an AI based API with Flask and it was really fun playing with the framework.

getmykeyword.herokuapp.com/

I'm learning more about the framework because I intend on building on the app to offer more features....

EngtTemmy:
Go for Django

It's a better option than flask, i was learning flask before, i did some research online, i got to understand that flask is somehow limited, so i switched to Django

With companies like Netflix, Reddit, Airbnb etc.. using Flask?

Does anyone know how to permanently suppress __pycache__ in pycharm?

cochtrane:
The code is written in R, and uses the tm package

Alright! Thanks.

cochtrane:
Following up on this dataset, I started wondering, can one define a machine learning question with this data set given its limited number of features? Apparently yes!

In this second part, I examine procedures for fitting a model with this data set. The research question takes the form: given a post title, can one tell which section it is from? For example, given a title "COVID-19: Governor Ikpeazu's Two Aides Test Positive", can our model tell that it is from the Health section?

Using NLP procedures, one can design a machine learning model which takes some part of this data set and fits a model to it, so that with the test data, we can ask questions of the remainder titles. This is a typical unsupervised model design known as classification. This particular task is multi-class classification with about 37 classes (all sections on Nairaland). This is a little harder than binary classification which has just two labels, because there are many more labels and the chances of being right for any one prediction is quite low (1/37 in this case, if we consider independence).

Before fitting, I generated a wordcloud to see which words are the most prominent. Apparently, "buhari" has been a prominent word over the past year on Nairaland's front page. Little wonder it chances of occurrence was quite high in the initial analysis I did. "lagos" is also prominent. And unsurprisingly, "coronavirus" is also.

SVM with a linear kernel was used for the classification task, and worked quite well. Ended up with an overall accuracy of about 69%. For some specific keywords, the accuracy was even higher. For example, for the keyword "buhari", the model placed the frontpage topic in "Politics" all the time and was correct for 96% of the time. For the keyword "rape", it had a choice of three different sections and still managed an accuracy of 84%. For the keyword "coronavirus", it didn't do so well. Managed only an accuracy of 69%. In any case, it shows that some of these predictions are possible. One can probably improve this model by training it on more features such as number of posts, post author, time of post. More features should improve its accuracy. I may get around to that if I've got more time. If you are able to do it, drop a message.

Man you are on another level....
Seun should employ you.
I'll like to ask what you use for NLP; spacy, NLTK or Textblob?

Graspad:
Do you mean from scratch??

Not necessarily..

Graspad:
Is it a bad idea to go into deep learning after this??

What significant model have you built with those things you learnt? If you are able to answer the question then I think you are good to go.

Graspad:
This thread is created for people to come and challenge themselves on what they plan to learn this month.

The rules:
You list what you want to learn and people who have learnt the same thing can encourage you while people who wish to learn the same thing can get va chance to meet others.

Please, no advertisement ��

Mine:
I want to finish my machine learning course this month.

Guys join up, let's make the thread lively.

Pytorch for DL
OpenCV for Computer vision
Tkinter for GUI

4ckz:
I am in the medical field, but I learnt html, css and javascript.

I am just confused as to what to do with the knowledge, and what more languages should I learn?

Please someone should advice me

It doesn't mean anything. I study Optometry and vision science and I'm currently exploring AI and Blockchain.

trigar12:
Boss, can you please help me with the code you used in getting the required info from the table in ncdc site?

I came up with one but it is giving me all the table data alongside the code in a column..

Can you layout the code for me?

Thank you

I encountered something similar to that while scraping the website but I later found my way around it. Quote me with your email, I'll send the code. Alternatively, you can use power BI to get the data. Just put the URL under source.

Chukwudaalu:
Download "video panda compress and save" on Google playstore. It helps a lot.

Thanks! The app is not helping much

Gcool2:
Guys,I have an Idea,can we create a WhatsApp group where we can easily reach each other,or if there is any existing group for data science pls kindly signify to add members...

Creating a Whatsapp group would defeat the purpose of creating this thread. This is a thread that has a lot of resources embedded in it, that anyone can come across it on the web and benefit from it (as long as Nairaland exists).

jiggyniga:
Hi everyone I am wondering if there is any need to learn the plotly library as I can use matplotlib and seaborn for data visualization

Whereas matplotlib and seaborn give quick visualization, plotly is much more elegant and good for reporting your analysis. I'll always go for plotly most especially if I'm dealing with any analysis that has to do with academics. It is a matter of knowing which tool(s) to use for what.

Still on the matter...... Power BI dashboard

mcemmy0z:
Use python to melt it down

Good morning bro, did you webscrape your data with request and bs4? If yes, can I have access to your script?

Never mind again, the issue I'm encountering is solved. Thanks!