https://www.youtube.com/watch?v=RzG8LQ4Ntuw

Data Science Nigeria Opens First Nigerian Artificial Intelligence HUB in UNILAG

Data Science Nigeria in furtherance of its drive to boost research and innovation in Nigeria, opened the first ever
Nigerian Artificial Intelligence Hub in the University of Lagos.

4kings:

If you're dealing with comparing two documents like two news articles.
Then the importance similarities is not really high. Because if tf-idf is based on words and a news articles about "Israel and trump" would have similar words that tf-idf can use to rank similarities.
However for a better result consider applying stemming and stopwords to reduce noise.

But i just realised you were talking about document titles, well if that's the case you can use word embeddings to vectorize words to find meaning, however the approach towards word embeddings would tend to classify only contextual words and not in an antonyms-synoynms structure like "hot" and "cold" would have the same vector point.

Python has NLTK, TextBlob and Spacy which are NLP packages and they can be used easily to connect to wordnet to find words similarity, so i'm sure java would have similar packages or better still write one that connects to wordnet yourself.
This approach might not be effective because it will require making http requests on every word and you might not be able to afford to store all words and synonyms somewhere.

So if you are finding similarities in general plain text then that's difficult, word embeddings is the best choice i can think of right now and you could get already trained model online for your task but i doubt it will perform well for just general plain text unless your task is singular domain related then you can train the word embedding yourself.

The best way to approach this is to analyse the document itself and not just the title, with tf-idf to get a good result.

I finally implemented the tf-idf thing, it appears to be grossly inadequate for what i am looking to achieve.
My goal was to detect news with same message from various news vendor or media houses. I have tried matching them by titles, excerpt and full body but no luck so far. The stop words in tf-idf are really a headache, like u suggested lemmatization should help but i strongly suspect it may not solve the challenge of reducing stop words to acceptable level.

I am still looking for the solution to it just in case.
I felt since the WordPress similar post feature is so common I should be able to get this done in another language, java, but that's not the story so far.

pls anybody with embedded programming knowledge

tollyboy5:
pls anybody with embedded programming knowledge

Kindly state your problem so everyone makes an input

4kings:

If you're dealing with comparing two documents like two news articles.
Then the importance similarities is not really high. Because if tf-idf is based on words and a news articles about "Israel and trump" would have similar words that tf-idf can use to rank similarities.
However for a better result consider applying stemming and stopwords to reduce noise.

But i just realised you were talking about document titles, well if that's the case you can use word embeddings to vectorize words to find meaning, however the approach towards word embeddings would tend to classify only contextual words and not in an antonyms-synoynms structure like "hot" and "cold" would have the same vector point.

Python has NLTK, TextBlob and Spacy which are NLP packages and they can be used easily to connect to wordnet to find words similarity, so i'm sure java would have similar packages or better still write one that connects to wordnet yourself.
This approach might not be effective because it will require making http requests on every word and you might not be able to afford to store all words and synonyms somewhere.

So if you are finding similarities in general plain text then that's difficult, word embeddings is the best choice i can think of right now and you could get already trained model online for your task but i doubt it will perform well for just general plain text unless your task is singular domain related then you can train the word embedding yourself.

The best way to approach this is to analyse the document itself and not just the title, with tf-idf to get a good result.

OK. I made some progress. I was able to obtain the body of the news. This means I now have the title, the excerpt and the body text in full.

So far tf-idf is giving me unpleasant matches. for example the news about the psychic pig prediction super eagles semi final adventure appears along other news related to super eagle like in the same cluster. with the new about "super eagle team song by olamide and phyno" and the other one by simi et al.

My next target is to reduce stop words but I am not using a list of words directly. I am using POSTagger (Parts Of Speech Tagger) to eliminate the injunctions, modifiers and other category stop words generally fall into. With this approach words like for, from, a, to, with, am, when, soon, & am get eliminated because of the category POSTagger is placing them and not because I specified those words directly.


Something tells me you know what I need but I haven't been able to communicate it in the right words to you yet. As I google more I am learning new things. I know what lemmatization, tokenization, clustering is now.


Modified:
What I am looking for is clustering of news covering same issue, event or happening. If given 10 articles from a number of news outlets (e.g vanguard, punch, tribune and cable) and five of those are reporting Buhari's expose of budget padding by senate and HOR , Then I need to be able to cluster those five together and nothing more. I hope this explains it to you.

Desyner:
OK. I made some progress. I was able to obtain the body of the news. This means I now have the title, the excerpt and the body text in full.

So far tf-idf is giving me unpleasant matches. for example the news about the psychic pig prediction super eagles semi final adventure appears along other news related to super eagle like in the same cluster. with the new about "super eagle team song by olamide and phyno" and the other one by simi et al.

My next target is to reduce stop words but I am not using a list of words directly. I am using POSTagger (Parts Of Speech Tagger) to eliminate the injunctions, modifiers and other category stop words generally fall into. With this approach words like for, from, a, to, with, am, when, soon, & am get eliminated because of the category POSTagger is placing them and not because I specified those words directly.


Something tells me you know what I need but I haven't been able to communicate it in the right words to you yet. As I google more I am learning new things. I know what lemmatization, tokenization, clustering is now.


Modified:
What I am looking for is clustering of news covering same issue, event or happening. If given 10 articles from a number of news outlets (e.g vanguard, punch, tribune and cable) and five of those are reporting Buhari's expose of budget padding by senate and HOR , Then I need to be able to cluster those five together and nothing more. I hope this explains it to you.

Hmm, interesting.
You've not used word embeddings yet? That tends to improve performance.
And also u could perform some semantic analysis tricks on subjects of the articles.

I'm not with my system at the moment, I would work on this and get back to you when I get home.

Horus:

https://www.youtube.com/watch?v=RzG8LQ4Ntuw

Data Science Nigeria Opens First Nigerian Artificial Intelligence HUB in UNILAG

Data Science Nigeria in furtherance of its drive to boost research and innovation in Nigeria, opened the first ever
Nigerian Artificial Intelligence Hub in the University of Lagos.

Good development!!!

Sorry I have been offline guys. I have been writing my PhD thesis(still writing).

Sad... This thread was promising.
Life happens. oops.
Watsup with SoftEng?

4kings:
Sad... This thread was promising.
Life happens. oops.
Watsup with SoftEng?

4kings and all
I apologise for my absence (Yes, life happens ).
I have been mostly busy with different things.

Meanwhile, there are communities that have been very active in the AI space.
Data Science Nigeria and AI saturday Lagos have been really doing well. AI saturday abuja also have some activities.
Also an AI event called IndabaXNigeria will be happening in Univeristy of Lagos in May.

I'm not sure whether I can be as active on this thread as I used to be. However, this should NOT dissuade the purpose of the thread. At bare minimum, if anyone needs help or has a question, I can assist.

Thanks 4kings, you've been very helpful and you have been a good friend on this thread.

SoftEng:


4kings and all
I apologise for my absence (Yes, life happens ).
I have been mostly busy with different things.

Meanwhile, there are communities that have been very active in the AI space.
Data Science Nigeria and AI saturday Lagos have been really doing well. AI saturday abuja also have some activities.
Also an AI event called IndabaXNigeria will be happening in Univeristy of Lagos in May.

I'm not sure whether I can be as active on this thread as I used to be. However, this should NOT dissuade the purpose of the thread. At bare minimum, if anyone needs help or has a question, I can assist.

Thanks 4kings, you've been very helpful and you have been a good friend on this thread.

nice one bro.

I have been on learning path since last year November, I started with learning python basic and move over to Python libraries. I started ML month ago but have not really seem to understand it. Actually I understand the theoretical aspect of it. I am doing self learning, as a beginner I need a place in Lagos that I can always go to meet people on the same path so I can learn physically and have more practical experience. self learning is not really working for me. Thanks

SoftEng
4kings
Darivie04

Avast:
I have been on learning path since last year November, I started with learning python basic and move over to Python libraries. I started ML month ago but have not really seem to understand it. Actually I understand the theoretical aspect of it. I am doing self learning, as a beginner I need a place in Lagos that I can always go to meet people on the same path so I can learn physically and have more practical experience. self learning is not really working for me. Thanks

SoftEng
4kings
Darivie04

browse AI6 Lagos

odizeey:
browse AI6 Lagos

Thanks so much for this.

Pdsco:


Thanks so much for this.

welcome. You joined the last cohort?

odizeey:
welcome. You joined the last cohort?

Tried joining but error on the page.

Pdsco:


Tried joining but error on the page.

ok. It just ended, another will be coming up.

Avast:
I have been on learning path since last year November, I started with learning python basic and move over to Python libraries. I started ML month ago but have not really seem to understand it. Actually I understand the theoretical aspect of it. I am doing self learning, as a beginner I need a place in Lagos that I can always go to meet people on the same path so I can learn physically and have more practical experience. self learning is not really working for me. Thanks

SoftEng
4kings
Darivie04

Learning on one's own can be very discouraging since you have few people to talk and share ideas with. There are some online communities but they don't compare to physical communication with other people.

I don't live in Lagos so I can't suggest places in Lagos but I'm sure some others can.

Darivie04:

Learning on one's own can be very discouraging since you have few people to talk and share ideas with. There are some online communities but they don't compare to physical communication with other people.

I don't live in Lagos so I can't suggest places in Lagos but I'm sure some others can.

It is not easy

Dive into the top 10 Artificial Intelligence Startups in Africa....Some of these tech companies can put money in your pocket by providing services to people around you, some can help improve crop yields and early disease detection to farmers etc. There's a lot to learn from these AI startups.

Watch below...


https://www.youtube.com/watch?v=VHLdsL2xE9w

odizeey:
welcome. You joined the last cohort?

Sorry if this question is irrelevant but I presently do not have a functioning laptop, it's my phone that I've been using for programming classes (I only know very basic Python). I have a Python IDE app and I also use Jupyter Notebooks on Microsoft Azure.

Can I still join the classes?

darkhorizon:


Sorry if this question is irrelevant but I presently do not have a functioning laptop, it's my phone that I've been using for programming classes (I only know very basic Python). I have a Python IDE app and I also use Jupyter Notebooks on Microsoft Azure.

Can I still join the classes?

I don't know. Just register

odizeey:

I don't know. Just register

Registration for cohorts closed.

arduino:

Registration for cohorts closed.

Register for the next one.

Good afternoon All....I have been told that Data Science/Machine Learning, Deep Learning, Computer vision are only for those who are expert in this field, that is If I want to work in google/facebook/amazon etc it has scope but outside that it is pretty hard to get a job
without years of work experience. This discouraging to me because I am planning to go for a masters degree in Germany with focus in these areas and a plan to do personal studies along the line...

Can someone who is working in this field explain better to me?

What's my prospect of getting employed after my masters?

I currently have a bachelors degree in Electrical Electronics with 4.06 cgpa and my background in mathematics is very good but I don't want to go into this field now and regret later

SOLARPOWER1:
Good afternoon All....I have been told that Data Science/Machine Learning, Deep Learning, Computer vision are only for those who are expert in this field, that is If I want to work in google/facebook/amazon etc it has scope but outside that it is pretty hard to get a job
without years of work experience. This discouraging to me because I am planning to go for a masters degree in Germany with focus in these areas and a plan to do personal studies along the line...

Can someone who is working in this field explain better to me?

What's my prospect of getting employed after my masters?

I currently have a bachelors degree in Electrical Electronics with 4.06 cgpa and my background in mathematics is very good but I don't want to go into this field now and regret later

Anyone who told you data science is only meant for expert that want to work for Facebook and google has given you the biggest lie of the century.

Secondly you're not here to acquire relevant knowledge and degrees only to work for the fulfilment of another persons dream in order to recieve salary. Think outside the box and liberate yourself from the social norm.

Moreover, the internet is now a global village and any skills including data science, machine learning, deep learning can be acquired on YouTube, Udemy...

Irrespective of your previous or intending qualifications, the best job out there is the one you're willing to create through the application of your acquired knowledge.

The field Artificial intelligence is opportunity driven but certainly not for the one who wants to land a "dream job" with fat salary. What works here is "creative works with human impact".

Few weeks ago,i pick up deep learning and it seem easy to me, please it is that easy or am getting it wrong?

So far,i have been able to understand

>>>weight
>>>perceptron
>>>activation function
>>>multi-perceptron
>>>feed forward
>>>back propagation
>>>loss (cost function)
>>>optimizer

Below is the link to my posts about what i have learnt so far

https://www.nairaland.com/5869408

With all these,it was easy to go back to ML and start picking the algorithm.

I have been able to build a model and tune the hyperparameter.

Is there still something am getting wrong,it seem easy than the hype

Abcruz:


Anyone who told you data science is only meant for expert that want to work for Facebook and google has given you the biggest lie of the century.

Secondly you're not here to acquire relevant knowledge and degrees only to work for the fulfilment of another persons dream in order to recieve salary. Think outside the box and liberate yourself from the social norm.

Moreover, the internet is now a global village and any skills including data science, machine learning, deep learning can be acquired on YouTube, Udemy...

Irrespective of your previous or intending qualifications, the best job out there is the one you're willing to create through the application of your acquired knowledge.

The field Artificial intelligence is opportunity driven but certainly not for the one who wants to land a "dream job" with fat salary. What works here is "creative works with human impact".

So, what should someone who wants to land a dream job with fat salary look at?

BoleAndFish:

So, what should someone who wants to land a dream job with fat salary look at?

Explore your creative reserve and find solutions to man's insatiable want and all other benefits including fat salary will be added unto you.