Question One
Perform data processing on the data set, as required. Give evidence whether there are problems with data quality, duplicate data or missing data. Note: With respect to predictor variables, missing categorical values should be replaced by the mode value by gender for that predictor variable. e.g. if the voice_mail flag is missing for a record that has gender of male, the replacement value should be the most popular voice_mail flag value for males. Missing numeric values should be replaced by the median value for the variable. Comment on your findings as well as the actions you carried out.

Note:[/b]Before starting your analysis, please familiarise yourself with the churn dataset provided. You can assume the sample was randomly selected from a population that is normally distributed. Please refer to Appendix 1 for description of the predictor variables and response (target) variable.
As part of an appendix to your report provide clearly commented supporting R code you used to carry out your tasks).

[b]Solution

From the data processing, it is evident that there are 10 missing values and 12 empty/blank cells. Below is the breakdown of the predictors with missing values and blank cells:
CUST_MOS (The number of continuous months the Customer is with the provider) had 3 missing values
MINUTES_3_MONTHS_AGO (Number of phone minutes used in the previous 3 months) had 3 missing values
TOT_MINUTES_USAGE (The total number of minutes used to date) had 4 missing values

While,

PHONE_PLAN (The phone plan the customer has signed up for) had 4 empty cells
EDUCATION (Highest Level of education attainment the account holder has achieved) had 8 empty cells

With this, it is evident that there are missing values and blank cells in the dataset. We use the R command below to import the dataset into R and replace all blank cells with NA

# R command
eurocom <- read.csv("eurocom.csv", header = TRUE, na.strings=c("", "NA")
attach(eurocom)

We use the command below to count the number of NA’s in the dataset which was 22

table(is.na(eurocom))
FALSE TRUE
37256 22

The command below tells us the predictors with NA’s

sapply(eurocom, function(eurocom) sum(is.na(eurocom)))

We also use the R command below to determine the customer with missing values

eurocom[!complete.cases(eurocom),]

For the predictors that are numeric in nature (CUST_MOS, MINUTES_3_MONTHS_AGO, TOT_MINUTES_USAGE), we replace the missing values (NA’s) with the median value using the R command below

#Replaces NA's with median value of Number of minutes 3 months ago
eurocom$MINUTES_3MONTHS_AGO[is.na(eurocom$MINUTES_3MONTHS_AGO)]=median(eurocom$MINUTES_3MONTHS_AGO, na.rm=TRUE)

#Replaces NA's with median value of months the Customer is with the provider
eurocom$CUST_MOS[is.na(eurocom$CUST_MOS)]=median(eurocom$CUST_MOS, na.rm=TRUE)

#Replaces NA's with median value The total number of minutes used to date
eurocom$TOT_MINUTES_USAGE[is.na(eurocom$TOT_MINUTES_USAGE)]=median(eurocom$TOT_MINUTES_USAGE, na.rm=TRUE)

For the predictors that are categorical (PHONE_PLAN and EDUCATION), we replace the missing values by the mode value by gender for that predictor variable. To do this, we use the commands below to view the mode value by gender for phone plan and education.

table(PHONE_PLAN, GENDER)
GENDER
PHONE_PLAN F M
Euro-Zone 20 39
International 449 618
National 261 410
Promo_plan 0 270

table(EDUCATION, GENDER)
GENDER
EDUCATION F M
Bachelors 270 130
High School 1 2
Masters 0 330
PhD 190 220
Post Primary 266 563
Primary 1 90

It is shown from the results above that the most used phone plan for both gender was “International”. The highest education level attained for most males was Post Primary education while for females it was Bachelors’ degree. So, we replace these according to the customer with blank cell. Recall, we use the sapply(eurocom, function(eurocom) sum(is.na(eurocom))) command to view the predictor variables with missing values (see appendix) and it was shown that customers’ 6, 10, 109, 233, 283, 339, 673, 979, 1191, 1366, 1389 and 1465 have missing values for PHONE_PLAN and EDUCATION [recall we have replace the rest by their median values since they were numeric variables but we can’t do that for this as it is categorical]. To do this, we use the R command below to store the new dataset (i.e. the one we removed the missing values by its median) and make the adjustment.

write.csv(eurocom, "Neweurocom.csv"

Now, the issue of missing values has been resolved, the next we check for is duplicate data, we use the R command below to do this

# REMOVING DUPLICATES
duplicated(eurocom)
which(duplicated(eurocom))
integer(0)

This tells us there is no duplicated data.


QUESTION TWO

Discretise the Income predictor variable as follows;
Income >= 88,000 -> High Income
Income < 88,000 && Income >= 38,000 -> Medium Income
Income < 38,000 -> Low Income

Solution

We use the R commands below to discretize the Income into groups

Income <- data.frame(eurocom)
HighIncome <- subset(Income, INCOME>=88000)
MediumIncome <- subset(Income, INCOME<88,000 && Income >= 38,000)
LowIncome <- subset(Income, INCOME<38,000)

Please, if you want me to continue, you all have to cut short your sabbatical leave and drop your comments oh, ehn ehn!

Hello guys,

Since this thread has died, I guess i have to make it alive again. How? You may ask, I would review the jobs i receive from different clients with you and i hope someone somewhere will learn one or two things; what's the essence of living a life without impacting on anyone's life anyways?

Way to go

I would start with easy ones first before moving to more advanced ones, I hope it's okay by you. Lest i forget, our 'gurus' in the house, don't be offended oh [biko! I'm on my knees] i know this looks easy to you but guess what? You don't know who might be interested in these little things.

Yeah! Now that I've gotten your approval let's get the party started! Young John 'the wicked producer' please give me the beat, thank you sir!

The Problem

EuroCom provides new fixed line, mobile phone and broadband services to customers in the Euro-zone. It uses the Euro-zone telecommunications network and competes for customers against other major telecommunications companies in the market. Currently the company has a 50,000 customer base and there are roughly 250,000 calls per day.

One of the biggest problems facing EuroCom is customer retention. Customers are free to move between telecommunications service providers and some regularly change service providers, a phenomenon known as churn in the telecommunications industry. Some companies have churn rates as high as 20% of their customers changing service providers per annum. EuroCom has placed a high emphasis on churn and are always trying to find new ways of reducing it. The directors of EuroCom are looking for the answer to the main question of how we detect customers who are going to churn

Overall, the directors are looking for answers to these types of questions.

1. What is it that makes a customer churn?
2. Are some customers more likely to churn than others?
3. How can we identify these customers before they churn?

_______________________________________________________________________________________________________________________

The data was collected from 2071 subscribers, below is the screenshot

Thank you for sharing your experience OP. Life is not a bed of roses, you either choose to lie on it or stuck to your noses.

I also read the post bro, I concur with all you have said. I was in a similar position sometimes ago (though not arrogant nor insultive), I had to fast and pray about this thing called "self"; though it wasn't easy but at the end 'GodWin'. You're right about the claim that there are many smart people on nairaland and I've been opportune to learn from them.

Anyways, he who has ears let him ear.

"None of self but All of thee"

... and that was how they derailed the thread.

I thank God for giving my mother the grace to raise me in a way a responsible man should be. All these pride, insults, curses, etc. haba!

alex81:




yeah he is. Jarus is a 1st class graduate from OAU Economics dept!

loss/pain kwa what is ds one saying are u ok

jarus is an entrepreneur, a speaker,a writer, a consultant, a source of inspiration to many unemployed youths organising job seminars upandan both online and 'offline'. Also works with a leading oil company here in Lagos. I'm a beneficiary of one of dz seminars.

what else do u want a man to be?



just Google jarushub and stop saying what u don't know.

cc:

other haters

I'm forever grateful to him for changing my life by reading the articles on his blog in 2015. I hope to write a 'Thank you' message to him this week or next. A Yoruba adage says "A man we rendered help who do not appreciate is synonymous to a robber who stole our goods".

OluDare01:
Thanks alot. I'm sending you a pm right away

Replied sir!

OluDare01:
Please who knows how to use "R" very well
Any data scientist in the house?

Yes sir, I'm good with R. My undergraduate degree is in statistics. Any issue with it? I hope I can help.

OluDare01:
Please who knows how to use "R" very well
Any data scientist in the house?

Yes sir, I'm a data scientist.

Easier said than done.

Introduction

There is huge amount of data available in Information Industry. This data is of no use until converted into useful information. Analysing this huge amount of data and extracting useful information from it is necessary.

The extraction of information is not the only process we need to perform, it also involves other processes such as Data Cleaning, Data Integration, Data Transformation, Data Mining, Pattern Evaluation and Data Presentation. Once all these processes are over, we are now position to use this information in many applications such as Fraud Detection, Market Analysis, Production Control, Science Exploration etc.

What is Data Mining

Data Mining is defined as extracting the information from the huge set of data. In other words we can say that data mining is mining the knowledge from data. This information can be used for any of the following applications:

Market Analysis

Fraud Detection

Customer Retention

Production Control

Science Exploration

Need of Data Mining

Here are the reasons listed below:

In field of Information technology we have huge amount of data available that need to be turned into useful information.

This information further can be used for various applications such as market analysis, fraud detection, customer retention, production control, science exploration etc.

Data Mining Applications

Here is the list of applications of Data Mining:

Market Analysis and Management

Corporate Analysis & Risk Management

Fraud Detection

Other Applications

Market Analysis and Management

Following are the various fields of market where data mining is used:

Customer Profiling - Data Mining helps to determine what kind of people buy what kind of products.

Identifying Customer Requirements - Data Mining helps in identifying the best products for different customers. It uses prediction to find the factors that may attract new customers.

Cross Market Analysis - Data Mining performs Association/correlations between product sales.

Target Marketing - Data Mining helps to find clusters of model customers who share the same characteristics such as interest, spending habits, income etc.

Determining Customer purchasing pattern - Data mining helps in determining customer purchasing pattern.

Providing Summary Information - Data Mining provide us various multidimensional summary reports

Corporate Analysis & Risk Management

Following are the various fields of Corporate Sector where data mining is used:

Finance Planning and Asset Evaluation - It involves cash flow analysis and prediction, contingent claim analysis to evaluate assets.

Resource Planning - Resource Planning It involves summarizing and comparing the resources and spending.

Competition - It involves monitoring competitors and market directions.

Fraud Detection

Data Mining is also used in fields of credit card services and telecommunication to detect fraud. In fraud telephone call it helps to find destination of call, duration of call, time of day or week. It also analyse the patterns that deviate from an expected norms.

Other Applications

Data Mining also used in other fields such as sports, astrology and Internet Web Surf-Aid.

Artificial intelligence, or AI, is the field that studies the synthesis and analysis of computational agents that act intelligently. Let us examine each part of this definition. An agent is something that acts in an environment – it does something. Agents include worms, dogs, thermostats, airplanes, robots, humans, companies, and countries. We are interested in what an agent does; that is, how it acts. We judge an agent by its actions. An agent acts intelligently when what it does is appropriate for its circumstances and its goals,it is flexible to changing environments and changing goals,it learns from experience, andit makes appropriate choices given its perceptual and computational limitations.

An agent typically cannot observe the state of the world directly; it has only a finite memory and it does not have unlimited time to act. A computational agent is an agent whose decisions about its actions can be explained in terms of computation. That is, the decision can be broken down into primitive operation that can be implemented in a physical device. This computation can take many forms. In humans this computation is carried out in “wetware”; in computers it is carried out in “hardware.” Although there are some agents that are arguably not computational, such as the wind and rain eroding a landscape, it is an open question whether all intelligent agents are computational. The central scientific goal of AI is to understand the principles that make intelligent behavior possible in natural or artificial systems. This is done by
the analysis of natural and artificial agents;formulating and testing hypotheses about what it takes to construct intelligent agents; anddesigning, building, and experimenting with computational systems that perform tasks commonly viewed as requiring intelligence.

As part of science, researchers build empirical systems to test hypotheses or to explore the space of possibilities. These are quite distinct from applications that are built to be useful for an application domain. Note that the definition is not for intelligent thought.We are only interested in thinking intelligently insofar as it leads to better performance. The role of thought is to affect action. The central engineering goal of AI is the design and synthesis of useful, intelligent artifacts.We actually want to build agents that act intelligently. Such agents are useful in many applications.

Since this thread is getting dry, do you mind if I spice it up with some talks on machine learning using R?

Gaara101:

I'm very much aware of the importance of math in machine learning. I'm currently studying a linear algebra textbook and I want to study statistics in general as well.
I will definitely read those articles.
Thanks for offering your help. I will definitely ask if I come across something difficult.
Perhaps it was how I phrased the question? I was asking of the sector you apply ML. Be it health, finance etc.

I apply ML across many sectors including but not limited to: finance, health, engineering, education. It depends on the problem I'm faced with. I do analysis where I use R week in and out, I do hope to leave my comfort zone (R) and move to Python, SAS (I've tried to download the university free edition all to no avail).

Parting word: You are an homo sapien with unquantifiable ability, only ants in ants' colony micro-specialise. Don't limit yourself to a particular sector.

My two cents....

Gaara101:

That's great.
What aspect of machine learning are you focused on?
I just downloaded the book you listed. Do you need to have a statistical background before delving in.
I skimmed through the chapter and it seems kind of easy but the author said the reader should have taken at least a statistics class.

To be proficient with machine learning, basic knowledge of statistics and mathematics is required. You will need a bit of mathematics to understand topics such as SVM, dimension reduction techniques, and others. I took the Stanford's online course "Introduction to Statistical learning" I think in 2014 during my third year, but I scored 41%, I had to retake in 2015 which I scored (if I can remember vividly) 65%. The course is taken yearly, the next class will likely hold in March or thereabout.

By the way, I don't understand your question about what aspect of machine learning I'm focused on. I hope to get involved in Deep learning, artificial intelligence, pattern recognition and other related areas. For now, I can only say I work on supervised and unsupervised problems. Bro, read articles by Brendan Connor, Andrew Gelman, Nathan Yau of flowing data; there is so much to learn, I'm also trying to get my feet on the ground as well. If you are have any problem with the book, I will be glad to help.

Hi! I'm a graduate of Statistics with a decent grade. I'm a data analyst and machine learning enthusiast; I'm pretty good with machine learning algorithms.
After taking a course on Data mining at school, the book that changed my life was "Introduction to Statistical Learning Using R" by Hastie, Written, Tibshirani. I would love to share ideas with other machine learners. Anyways I'm proficient with R, STATA, and Minitab, I'd love to collaborate with any one with valuable data.

Favlex10:


Done!!!

Please add me

Oya add me oh.....

...

Accept God’s Timing

God gives us hopes and dreams for certain things to happen in our lives, but He doesn’t always allow us to see the exact timing of His plan. Although frustrating, not knowing the exact timing is often what keeps us in the program. There are times when we might give up if we knew how long it was going to take, but when we accept God’s timing, we can learn to live in hope and enjoy our lives while God is working on our problems. We know that God’s plan for our lives is good, and when we entrust ourselves to Him, we can experience total peace and happiness.

The book of Genesis tells the story of Joseph, who waited many years for the fulfillment of the dream God had given him. He was falsely accused and imprisoned before the time came for him to do what God had shown him he was to do. Exodus 13:17-18 tells us that God led the Israelites the longer, harder way on their journey to the Promised Land because He knew they were not yet ready to go in. There had to be time for their training, and they had to go through some very trying situations. They wasted a lot of time wondering about God’s timing, but God never failed to take care of them and show them what He wanted them to do. The same is true in our lives.

Abeg .... I've accepted God's will for me. God has perfect timing; never early, never late. It takes a little patience and faith, but it's worth the wait.

The guy above me sha...
Nigerians and scam be like bread and beans...

I gave up on Nigeria a long time ago but Buhari just hypnotize me sha..
I no dey vote till 2023, enough of heartbreaks abeg

gunther6:

Lmao
Baba long time now

Baba, I don PM you tire...

baum1:


Hi,

I already responded immediately.

I just sent another.

Thank you.....

baum1:


Wow, impressive, please send me an email. The above opportunity is not for you, but I will work with you and by God's grace we will get something. Let's be quick about this.

I just sent you a mail sir...

baum1:
Hello people, if you are qualified or you know anyone who is, send this to them.

Good afternoon sir,
Many thanks for your magnanimity, may God continue to bless the work of your hands.
I have a degree in Statistics
(BSc. Statistics). I'll be anticipating your assistance sir.

@sirRiddy

Please I'm having problem sending it to your mail (it's not delivering). I sent it to your username@yahoo. com, I'm I wrong?

@sirRiddy I'll send it right away, many thanks sir...

sirRiddy:
Nice job!
Are you part of the GRE Facebook group?
Lots of veterans there. I met some Indians there who helped me
improve my score. They post from disparate sources which helped me
learn from different material in one place.
Secondly, you want to try this site: kaomanfen.com. It does not get better
than that for practice tests, maybe not as great as magoosh. I found out about
this pretty late.

If you make around 320 and have around 3.5/4 GPA, you can apply to K-state as a
safe school. Education here is awesome, the engineering college is a place to be.
Elect is just next to my department in the same complex. The funding is good and the
research is renowned too. Enjoy the experience, share the experience with us and good luck!

NB: Read more about K-State Elect here: http://www.ece.k-state.edu//

I would be grateful if you can help in reviewing my personal statement. Your criticism would be wholly welcome.

happyday:

For a start, you need the Official Guide to the GRE. To get a feel of what's being tested across sections.

Your confidence level afterwards will then help you choose materials suitable for you.

Baba, please I would be grateful if you can help me in reviewing my personal statement.
Many thanks bro...