4kings:

Interesting challenges. Thanks for keeping us updated.
I sure would love to collaborate with you, though i will only be able to fully meet requirment 2 after the next three weeks. Got very tight schedule... Damn

I understand.
However, all the requirements are necessary.
Maybe some other time.

Thanks for your interest.

odizeey:

My own view.
You need an understanding of statistics and programming.
If you don't have enough money download NPTEL data analytics video on YouTube to start with. If you have money, take edureka data science course or Udemy.
Zenva also has a relatively cheap package comprising of 12 course.

thanks

amulet:
thanks

welcome

SoftEng:


I'll be trying my hands on one of the challenges. I'm not yet sure which challenge I'll work on.

Anybody here interested in collaborating, please let me know. However, you have be:
1. really serious minded.
2. create time out of your busy schedule to work on the challenge.
3. ready to read research papers.
4. ready to learn quickly required background/related techniques.
5. already fluent in at least one programming language (preferably python)

Please only those who meet the above requirement should indicate interest.

Thank you.

Some other time, I will like to collaborate with you but please what is your definition of 'fluent' in Python? Make I know if I fluent or not.

SoftEng:


I don't think this job is genuine. I should have commented about it earlier. I'm sorry.

This advert is old. I think it earlier appeared in March 2017 (see link below). Apart from being old it may also be fake. A company willing to pay 300-400k for a fresh graduate coder (which is a bit high) should at least have a professional email for application or an application system.

Thanks for sharing 4kings. It shows you have good interest at heart for the group member here.

To all,
Moving ahead, try to validate job adverts before sharing.

Thanks.

Earlier link for the job post
https://www.nairaland.com/3707935/vacancy

Na wa for naija scammers.
Thanks for the confirmation.

SoftEng:



I understand.
However, all the requirements are necessary.
Maybe some other time.

Thanks for your interest.

Have you started already?
I'm done with all my major works and meetings/travels?

Can't foresee any other job or meeting that will take out much time like it did in the past two weeks. That's if you've not gone ahead already.

SoftEng:


I'll be trying my hands on one of the challenges. I'm not yet sure which challenge I'll work on.

Anybody here interested in collaborating, please let me know. However, you have be:
1. really serious minded.
2. create time out of your busy schedule to work on the challenge.
3. ready to read research papers.
4. ready to learn quickly required background/related techniques.
5. already fluent in at least one programming language (preferably python)

Please only those who meet the above requirement should indicate interest.

Thank you.

Bro I am interested. I am fluent in Python and have lots of time this period. I also have an OK background in Machine Learning, most importantly willing to learn

SoftEng:


I'll be trying my hands on one of the challenges. I'm not yet sure which challenge I'll work on.

Anybody here interested in collaborating, please let me know. However, you have be:
1. really serious minded.
2. create time out of your busy schedule to work on the challenge.
3. ready to read research papers.
4. ready to learn quickly required background/related techniques.
5. already fluent in at least one programming language (preferably python)

Please only those who meet the above requirement should indicate interest.

Thank you.

Let me get your mail so we can talk better.

Nice one guys, I've been hearing bout AI/ML, cool to see our people are into it and using Big Big grammer, I would love to learn and be part of it... lemme go and do my homework, will be back one day to start contributing...

@4kings & softEng, I read pages 1 & 7(this page) and am impressed by efforts here.
I happen to apply ML some weeks back without even realizing what I was applying was ML. I used OpenNLP (Java) cos Standford NLP guides weren't showing up in my searches as expected.



I like to know what's the best approach to help read meaning into two document title and determine they are addressing same topic or subject?


My Effort:
I tokenized and POS tagged the sentences then tried comparing the key words nouns (NN, NNP, NNS etc) but the key words sometimes change examples are IGP/IG, Buhari/The President, NFF/Football Federation.

Desyner:
@4kings & softEng, I read pages 1 & 7(this page) and am impressed by efforts here.
I happen to apply ML some weeks back without even realizing what I was applying was ML. I used OpenNLP (Java) cos Standford NLP guides weren't showing up in my searches as expected.



I like to know what's the best approach to help read meaning into two document title and determine they are addressing same topic or subject?


My Effort:
I tokenized and POS tagged the sentences then tried comparing the key words nouns (NN, NNP, NNS etc) but the key words sometimes change examples are IGP/IG, Buhari/The President, NFF/Football Federation.

Hey, did you read about the tf-idf(term frequency-inverse document frequency) i told you about in the other thread were you first raised this issue?
If you did, you would have solved your problem by now.

POS tagging may find similarity in sentence structure; for example; "'Buhari is the best president Nigeria ever had'" and "Trump is the best player China never had" would give the same part-of-speech structure but there aren't similar.
Well, pos tagging can only go far, but not far enough.

The purpose of TF-IDF is just to represent words as numbers(or vectorisation) for easy computation. From the name you would know this works by getting frequecny of words(normalized though) * total number of times words appear in a document(well this explanation is just me making it easy ).

This link should explain it simply for you and the code is written in java; It's been long i wrote Java code, but tf-idf should be easy to implement yourself once you understand it and i just skim through that link but i think it's explained simply enough.

After getting tf-idf of words in document, we can find similarity between two documents by computing the similarity between the two values(note that this values are represented as vectors for example 1,2,3 can be represented as (0,1), (0,2), (0,3) and (0,4))

And from mathematics we can find similarities between points in a graph or vector space with varieties of formulas including Euclidean distance, mahattan distance, jaccard similarity, cosine similarity and so on.

The best algorithm for this kind of situation is cosine similarity. I don't want to go into mathematical details of why it is, if you are an engineering student then it will be obvious why cosine similarity works best for one dimensional data(sparse vectors), if you are not, don't worry just memorise the formula for now, till you go deeper in machine learning.

I don't know how to write mathematical formulas well on a webpage last time i explained something similar to someone i used image and python code to explain, but seems the image is not showing again.

The image below shows the formula for cosine similarity.
Note that the tf-idf value you get for two documents represent A and B.

So the lower the value you get from calculating the similarity implies how dissimilar the documents are and vice-versa.
Just put a threshold value of say 0.6 to be comfortable or test for cross validation.

However, the link i gave also computes cosine similarity.

The Data Science Nigeria Challenge is over and i didn't have time to participate(thought it will be my first kaggle challenge).
Anyways checked their website for the upcoming one.

---

Na wa ooo, the last post was febuary 18 and not even from an old member of the group.
Seems this is a busy period for everyone.

4kings:

Hey, did you read about the tf-idf(term frequency-inverse document frequency) i told you about in the other thread were you first raised this issue?
If you did, you would have solved your problem by now.

POS tagging may find similarity in sentence structure; for example; "'Buhari is the best president Nigeria ever had'" and "Trump is the best player China never had" would give the same part-of-speech structure but there aren't similar.
Well, pos tagging can only go far, but not far enough.

The purpose of TF-IDF is just to represent words as numbers(or vectorisation) for easy computation. From the name you would know this works by getting frequecny of words(normalized though) * total number of times words appear in a document(well this explanation is just me making it easy ).

This link should explain it simply for you and the code is written in java; It's been long i wrote Java code, but tf-idf should be easy to implement yourself once you understand it and i just skim through that link but i think it's explained simply enough.

After getting tf-idf of words in document, we can find similarity between two documents by computing the similarity between the two values(note that this values are represented as vectors for example 1,2,3 can be represented as (0,1), (0,2), (0,3) and (0,4))

And from mathematics we can find similarities between points in a graph or vector space with varieties of formulas including Euclidean distance, mahattan distance, jaccard similarity, cosine similarity and so on.

The best algorithm for this kind of situation is cosine similarity. I don't want to go into mathematical details of why it is, if you are an engineering student then it will be obvious why cosine similarity works best for one dimensional data(sparse vectors), if you are not, don't worry just memorise the formula for now, till you go deeper in machine learning.

I don't know how to write mathematical formulas well on a webpage last time i explained something similar to someone i used image and python code to explain, but seems the image is not showing again.

The image below shows the formula for cosine similarity.
Note that the tf-idf value you get for two documents represent A and B.

So the lower the value you get from calculating the similarity implies how dissimilar the documents are and vice-versa.
Just put a threshold value of say 0.6 to be comfortable or test for cross validation.

However, the link i gave also computes cosine similarity.

i have gone through your reply. I found that tf-idf concept too detailed for my task initially but after this reply I have to go research it further. The method I adapted gave almost 100% result when the keywords or important tokens weren't changing. I hope the tf-idf approach solves the challenge of interpreting synonyms in their context correctly.

Desyner:
i have gone through your reply. I found that tf-idf concept too detailed for my task initially but after this reply I have to go research it further. The method I adapted gave almost 100% result when the keywords or important tokens weren't changing. I hope the tf-idf approach solves the challenge of interpreting synonyms in their context correctly.

If you're dealing with comparing two documents like two news articles.
Then the importance similarities is not really high. Because if tf-idf is based on words and a news articles about "Israel and trump" would have similar words that tf-idf can use to rank similarities.
However for a better result consider applying stemming and stopwords to reduce noise.

But i just realised you were talking about document titles, well if that's the case you can use word embeddings to vectorize words to find meaning, however the approach towards word embeddings would tend to classify only contextual words and not in an antonyms-synoynms structure like "hot" and "cold" would have the same vector point.

Python has NLTK, TextBlob and Spacy which are NLP packages and they can be used easily to connect to wordnet to find words similarity, so i'm sure java would have similar packages or better still write one that connects to wordnet yourself.
This approach might not be effective because it will require making http requests on every word and you might not be able to afford to store all words and synonyms somewhere.

So if you are finding similarities in general plain text then that's difficult, word embeddings is the best choice i can think of right now and you could get already trained model online for your task but i doubt it will perform well for just general plain text unless your task is singular domain related then you can train the word embedding yourself.

The best way to approach this is to analyse the document itself and not just the title, with tf-idf to get a good result.

Hi everyone,

I apologise for being away so long.
I've recently had very busy schedule that is still ongoing.

4kings
Well done for your help and contributions.
With respect to my earlier message about the research competition I was planning to work on, I did not continue (partly due to current busy schedule). I only read through some Google blogpost about the Learned Image Compression challenge. However, I did some work on the Data Science Nigeria kaggle challenge I earlier shared (It was my first real work outside anything that involved using strictly neural networks. In fact, it was my first real-world attempt at Machine Learning for structured data). I did not do so well.

I'll post links to some new Google challenge that is currently on Kaggle.
Regardless of my schedule, I will try to be more active here.

Lastly, for other members that earlier indicated interest in the formation of the group, please try to be active. Communities derive much more value when everyone participates.

Thanks all.

Hello Everyone,

I know i have not contributed lately but that doesn't mean am not carrying out some analysis on some data-sets from kaggle, just that studies and clients projects have more priority these days but once i have some free time after exams would finish up some of the analysis and then upload for you all to review.

Best Regards.

raymod170:
Hello Everyone,

I know i have not contributed lately but that doesn't mean am not carrying out some analysis on some data-sets from kaggle, just that studies and clients projects have more priority these days but once i have some free time after exams would finish up some of the analysis and then upload for you all to review.

Best Regards.

Thanks.
All the best with you studies and projects.

SoftEng:
Hi everyone,

I apologise for being away so long.
I've recently had very busy schedule that is still ongoing.

4kings
Well done for your help and contributions.
With respect to my earlier message about the research competition I was planning to work on, I did not continue (partly due to current busy schedule). I only read through some Google blogpost about the Learned Image Compression challenge. However, I did some work on the Data Science Nigeria kaggle challenge I earlier shared (It was my first real work outside anything that involved using strictly neural networks. In fact, it was my first real-world attempt at Machine Learning for structured data). I did not do so well.

I'll post links to some new Google challenge that is currently on Kaggle.
Regardless of my schedule, I will try to be more active here.

Lastly, for other members that earlier indicated interest in the formation of the group, please try to be active. Communities derive much more value when everyone participates.

Thanks all.

Dear all,

Below are the links to the Google challenge on Kaggle (currently ongoing)

1. Google Landmark Retrieval Challenge
Short Description: Given an image, can you find all of the same landmarks in a dataset?
Prize: $2,500
https://www.kaggle.com/c/landmark-retrieval-challenge

2. Google Landmark Recognition Challenge
Short Description: Label famous (and not-so-famous) landmarks in images
Prize: $2,500
https://www.kaggle.com/c/landmark-recognition-challenge

Have fun.
Thanks all.

Please can someone create a whatsapp group for this

SoftEng:
Hi everyone,

I apologise for being away so long.
I've recently had very busy schedule that is still ongoing.

4kings
Well done for your help and contributions.
With respect to my earlier message about the research competition I was planning to work on, I did not continue (partly due to current busy schedule). I only read through some Google blogpost about the Learned Image Compression challenge. However, I did some work on the Data Science Nigeria kaggle challenge I earlier shared (It was my first real work outside anything that involved using strictly neural networks. In fact, it was my first real-world attempt at Machine Learning for structured data). I did not do so well.

Couldn't participate in the competition also, i was very busy. (Got the data though, noticed some null values when i did my analysis, i just vex postpone am )

---

I've been researching on some existing compression techniques.
Dynamic markov compression has been interesting so far.

Maybe we should lighten up the group by solving some existing compression algorithms starting with simple JPEG lossy compression and so on, maybe an idea could spark up from someone here.

realmundi:
Please can someone create a whatsapp group for this

Hmmm, are whatsapp groups good?
I think concepts can be well explained here in details than whatsapp.
And new members can see and be referred to existing discussions as pointed out by SoftEng.

Just that we've not been very active lately.

I need suggestions on transferring my machine learning models to the web and mobile apps. I use TensorFlow and sometimes Keras, So I deal with protobufs and HDF5 formats.

I've tried deploying with TensorFlow serving and kubernetes on docker but I hate the Remote Procedure Call concept. Any other alternative solution would be much appreciated.

Is it possible for me to get some1 to teach me data mining programming here... Help needed asap

zamie:
I need suggestions on transferring my machine learning models to the web and mobile apps. I use TensorFlow and sometimes Keras, So I deal with protobufs and HDF5 formats.

I've tried deploying with TensorFlow serving and kubernetes on docker but I hate the Remote Procedure Call concept. Any other alternative solution would be much appreciated.

Wawu...
I don't use tensorflow and keras...
Never used protobufs or HDF5 format.

SoftEng osarenomaspecial lum1 and co here you go...

Jaaypeee:
Is it possible for me to get some1 to teach me data mining programming here... Help needed asap

Have you started learning already?
You can share your progress here... let's discuss...

zamie:
I need suggestions on transferring my machine learning models to the web and mobile apps. I use TensorFlow and sometimes Keras, So I deal with protobufs and HDF5 formats.

I've tried deploying with TensorFlow serving and kubernetes on docker but I hate the Remote Procedure Call concept. Any other alternative solution would be much appreciated.

I haven't dealt with protobuf. Also, I haven't deployed ML models to web/mobile apps. However, I have saved/loaded models saved in HDF5 format.

In terms of deploying to mobile apps (not web apps), I suggest you check out On-device Machine Learning. Tensorflow has on-device ML platform called Tensorflow Lite (see URL below). Tensorflow Lite is used in mobile and embedded system, but I think it's more geared towards Android devices.

I think a second On-device ML platform worth mentioning is CoreML by Apple (see URL below). I'm not 100% sure that it's On-device ML as I have not looked into it in-depth, but I guess it is. CoreML specifically targets Apple devices. It also has the feature of converting a trained model from a third-party framework (e.g. Caffe and Keras) to its own model format.

In terms of deploying ML for webapps, I have very little to contribute. I guess you always have to make a Remote call to the server were your ML model is deployed. But you can also check out DeepLearn JS to see whether you can deploy your model in the web browser (using javascript). I'm not sure if this is possible, but you can find out more about the library. I think I have also read about some interplay between Keras and Deep Learn JS but I'm not sure how it works.

Conclusion
I only have some abstract ideas about these various platforms, nothing concrete. However, I believe they can guide you in solving some of your current challenges.

Links
1. Tensorflow Lite:
https://www.tensorflow.org/mobile/tflite/

2. CoreML
https://developer.apple.com/documentation/coreml
https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml

3. Deep Learn JS
https://deeplearnjs.org/

Thanks 4kings.
osarenomaspecial lum1 and the general house, please contribute to this question in order to provide more suggestions

Thanks all.

4kings:

Hmmm, are whatsapp groups good?
I think concepts can be well explained here in details than whatsapp.
And new members can see and be referred to existing discussions as pointed out by SoftEng.

Just that we've not been very active lately.

Yeah. Thanks for this.

We keep the group here. Open for any interested person to simply join the coversation and be part of the group.
An added benefit is the fact that the thread may show in search results for "AI/ML community in Nigeria", thus allowing more people to easily discover the thread/community.

4kings:

Couldn't participate in the competition also, i was very busy. (Got the data though, noticed some null values when i did my analysis, i just vex postpone am )

---

I've been researching on some existing compression techniques.
Dynamic markov compression has been interesting so far.

Maybe we should lighten up the group by solving some existing compression algorithms starting with simple JPEG lossy compression and so on, maybe an idea could spark up from someone here.

... noticed some null values when i did my analysis, i just vex postpone am ...
Yeah. In fact, the last row (example) in the CSV file was NULL throughout.

I've been researching on some existing compression techniques
Nice

...starting with simple JPEG lossy compression and so on...
I'm still kind of busy sha, but this would be nice. I suggest it should be thrown open for all interested person in the group to tackle.

Thanks.

SoftEng:


I haven't dealt with protobuf. Also, I haven't deployed ML models to web/mobile apps. However, I have saved/loaded models saved in HDF5 format.

In terms of deploying to mobile apps (not web apps), I suggest you check out On-device Machine Learning. Tensorflow has on-device ML platform called Tensorflow Lite (see URL below). Tensorflow Lite is used in mobile and embedded system, but I think it's more geared towards Android devices.

I think a second On-device ML platform worth mentioning is CoreML by Apple (see URL below). I'm not 100% sure that it's On-device ML as I have not looked into it in-depth, but I guess it is. CoreML specifically targets Apple devices. It also has the feature of converting a trained model from a third-party framework (e.g. Caffe and Keras) to its own model format.

In terms of deploying ML for webapps, I have very little to contribute. I guess you always have to make a Remote call to the server were your ML model is deployed. But you can also check out DeepLearn JS to see whether you can deploy your model in the web browser (using javascript). I'm not sure if this is possible, but you can find out more about the library. I think I have also read about some interplay between Keras and Deep Learn JS but I'm not sure how it works.

Conclusion
I only have some abstract ideas about these various platforms, nothing concrete. However, I believe they can guide you in solving some of your current challenges.

Links
1. Tensorflow Lite:
https://www.tensorflow.org/mobile/tflite/

2. CoreML
https://developer.apple.com/documentation/coreml
https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml

3. Deep Learn JS
https://deeplearnjs.org/

Thanks 4kings.
osarenomaspecial lum1 and the general house, please contribute to this question in order to provide more suggestions

Thanks all.

Thanks SoftEng

I'm currently trying out TensorFlow Lite. It seems promising. I've not delved into ios development yet and I'm also not planning to work with Deeplearn js yet (hate Javascript).

Thanks again, TensorFlow Lite is awesome.

4kings:

Have you started learning already?
You can share your progress here... let's discuss...

I'm starting with d basics... I think I'm working with fraud detection in telecommunications companies... I've gotten some resources from a telecommunications company, I'm.having issues with converting the algorithm to codes. I was advised to learn python for such programming. That's the delay I'm having for now

Jaaypeee:

I'm starting with d basics... I think I'm working with fraud detection in telecommunications companies... I've gotten some resources from a telecommunications company, I'm.having issues with converting the algorithm to codes. I was advised to learn python for such programming. That's the delay I'm having for now

What are the algorithms? We can help out
And are you permitted to share the data? If not just tell the structure we'd help out either ways.

Python is pretty simple. You can cover the basics with this one video (44 minuttes):

https://www.youtube.com/watch?v=N4mEzFDjqtA


Then we continue discussing here

plz add me up 08160457776