@blenyo11 great work. What server language/framework did you use?

^^^Hmn

asalimpo:


I'm not.
this is on page 170 - none of the the bots could find it.
https://www.nairaland.com/18002/visual-fortran

this crawler does however:
http://crawler.ademuanthony.com/

fincoApps doesnt. It's results are not even filtered (lots of irrelevant results)
try "programming competition" and you see a page full of results without "programming" or "competition"
in them!

One other thing I think you should consider is the posts/comments not only the title.

If its just crawling the title of topics then its way easier than crawling and indexing the topic contents (including comments) as well, which is why it seems my bots are slower.

jidez007:


One other thing I think you should consider is the posts/comments not only the title.

If its just crawling the title of topics then its way easier than crawling and indexing the topic contents (including comments) as well, which is why it seems my bots are slower.

decent crawlers produce relevant results.
A crawler that can't fetch something as prominent as the title is wacked.
other crawlers caught the title.

asalimpo:


decent crawlers produce relevant results.
A crawler that can't fetch something as prominent as the title is wacked.
other crawlers caught the title.

Ofcourse it fetches the title. I'm saying a decent crawler should crawl everything on a page not just the title.

I have used Google and Nairaland search, they search topic contents as well not just the title

jes came in here and saw this, forgot to provide github link, been busy, thanks for the rating, and i know am at fault for not concentrating on the competition

Eeeewo! I have been missing sha lol

jidez007:


Ofcourse it fetches the title. I'm saying a decent crawler should crawl everything on a page not just the title.

I have used Google and Nairaland search, they search topic contents as well not just the title

What are you saying! One crawler, I strongly suspect it was yours, didnt even catch the title.
I entered "programming competition" expecting to see this thread listed and it didnt.
It produced a page full of all kinds of results - junk results.
c'mon guy.

Fantastic job guys! Keep it up!

jidez007:
@blenyo11 great work. What server language/framework did you use?

Thanks for your commendation
It is a dot net application

You actually tried as well

sisqology:
Eeeewo! I have been missing sha lol

Been toasting girls all over the globe abi? Well, I am not surprised. . .

asalimpo:

What are you saying! One crawler, I strongly suspect it was yours, didnt even catch the title.
I entered "programming competition" expecting to see this thread listed and it didnt.
It produced a page full of all kinds of results - junk results.
c'mon guy.

The challenge said crawl the programming section not the title of the topics. What I'm now saying is my bot tries to crawl and index everything in this section including topic titles and their comments. It is obvious that a bot that only crawls the topic title will definitely finish faster than 1 that also tries to visit the topic and index all the posts underneath.

So my database contains both the topics and the comments, you should know that it's merely a line of code that is needed to be changed to effect title searches instead of posts.

So all you need to do is type anything from the topic content not the title.

I think that's also how nairaland search works.

asalimpo:

What are you saying! One crawler, I strongly suspect it was yours, didnt even catch the title.
I entered "programming competition" expecting to see this thread listed and it didnt.
It produced a page full of all kinds of results - junk results.
c'mon guy.

Please throw some points to Blenyo, if you think his crawler is better. If you can't allocate separate points the way i did, just post a score over 100 and when it's 10:30pm i'd count everything and pick the winner.

[size=22pt]What i've got to say[/size]

Currently Jidez is on top and Blenyo is in second place, I must commend Blenyo because i didn't know he'd be a major contender (I was expecting FincoApps to be the second) it was after i calculated the scores i saw Blenyo in second place, was surprised myself. Blenyo was sitting at third place till he got the 26pts for speed, he attracted the lowest negative points for code redundancy, I must say, you really did well with the coding..all variables were defined, no unnecessary functions or repetitions, but Jidez is too good to ignore, he came fully prepared and i appreciate that effort. A lesson Jidez can learn from this competition is that, apart from User Interface and material design, performance and speed should also be key. I think you and Blenyo can share some tips and tricks as you both will make a super team.

Sleeping Demon, oh my God, the Demon was not asleep at all..your crawler's speed can literally be compared to google's as i tested it from various countries (I even tested it more from a distant country's server and got a very fast ping back), but do not be carried away you could have had a spot at the top three if the crawler was really functional, it is that fast because it isn't doing much work or crawling properly, this crawler is what Asalimpo should constructively criticize. I believe you can do better.

As for ChinenYen, I think if he posts a link with the finished program, he has a chance to squeeze in between Jidez and Blenyo and take over the second spot because his code is just natural, not depending much on frameworks or external libraries. ChinenYen deserves much more credit i think, but let's wait and see if his link will come up before 10:30pm.

FincoApps, I expected you to either come up on top or be a serious head ache to other competitors, but your crawler not being functional enough and absence of some button functionality gave your seat out to Blenyo. For instance: if i search a term first, using the Nairaland option..everything works well but if i decide to swith to Stack Overflow by clicking the radio button, the preloader comes up but does not fetch data till i push the enter key. Improvements could be made on the results accuracy too, and this goes to every other competitor too..even Jidez, It can always be improved.


To those that used pre-loaders, please know that they can reduce the speed of the site and even make it very slow if it has been set to delay for some seconds. Blenyo's program did well in this aspect.


Overall i think Jidez's GUI and all round balance gave him an edge over every other program.

DanielTheGeek :


Sleeping Demon, oh my God, the Demon was not asleep at all..your crawler's speed can literally be compared to google's as i tested it from various countries (I even tested it more from a distant country's server and got a very fast ping back), but do not be carried away you could have had a spot at the top three if the crawler was
.

let me start by saying thanks for the comment, while i agree that the crawler wasnt doing much work, i just wanted to put something up to compete for the first time on nairaland as office work wouldnt give me much time to work on it, so i jes did something....even my codes werent hosted on github as i kept procrastinating as to when i would do that , all in all, i say thank you for taking out your time to review it

@Danielthegeek you are right. But I had to do everything I could to speed the development time as we were having exams then. I was able to squeeze little time and the challenge was too enticing to resist.

sleepingdemon:


let me start by saying thanks for the comment, while i agree that the crawler wasnt doing much work, i just wanted to put something up to compete for the first time on nairaland as office work wouldnt give me much time to work on it, so i jes did something....even my codes werent hosted on github as i kept procrastinating as to when i would do that , all in all, i say thank you for taking out your time to review it

You are welcome.

jidez007:
@Danielthegeek you are right. But I had to do everything I could to speed the development time as we were having exams then. I was able to squeeze little time and the challenge was too enticing to resist.

You did well.
I respect every single competitor, as far as the programming section is concerned, you have shown your worth. Thumbs up everyone..let the best man win by 10:30pm.

@Dhtml18, @Okonofuas, @Yinkeys, @Guru01, @Seun, @Olumyco, @Javanian, @Booyakasha, @Airsaylongcon and the rest, please drop your points.

DanielTheGeek:


Please throw some points to Blenyo, if you think his crawler is better. If you can't allocate separate points the way i did, just post a score over 100 and when it's 10:30pm i'd count everything and pick the winner.

[size=22pt]What i've got to say[/size]

Currently Jidez is on top and Blenyo is in second place, I must commend Blenyo because i didn't know he'd be a major contender (I was expecting FincoApps to be the second) it was after i calculated the scores i saw Blenyo in second place, was surprised myself. Blenyo was sitting at third place till he got the 26pts for speed, he attracted the lowest negative points for code redundancy, I must say, you really did well with the coding..all variables were defined, no unnecessary functions or repetitions, but Jidez is too good to ignore, he came fully prepared and i appreciate that effort. A lesson Jidez can learn from this competition is that, apart from User Interface and material design, performance and speed should also be key. I think you and Blenyo can share some tips and tricks as you both will make a super team.

Sleeping Demon, oh my God, the Demon was not asleep at all..your crawler's speed can literally be compared to google's as i tested it from various countries (I even tested it more from a distant country's server and got a very fast ping back), but do not be carried away you could have had a spot at the top three if the crawler was really functional, it is that fast because it isn't doing much work or crawling properly, this crawler is what Asalimpo should constructively criticize. I believe you can do better.

As for ChinenYen, I think if he posts a link with the finished program, he has a chance to squeeze in between Jidez and Blenyo and take over the second spot because his code is just natural, not depending much on frameworks or external libraries. ChinenYen deserves much more credit i think, but let's wait and see if his link will come up before 10:30pm.

FincoApps, I expected you to either come up on top or be a serious head ache to other competitors, but your crawler not being functional enough and absence of some button functionality gave your seat out to Blenyo. For instance: if i search a term first, using the Nairaland option..everything works well but if i decide to swith to Stack Overflow by clicking the radio button, the preloader comes up but does not fetch data till i push the enter key. Improvements could be made on the results accuracy too, and this goes to every other competitor too..even Jidez, It can always be improved.


To those that used pre-loaders, please know that they can reduce the speed of the site and even make it very slow if it has been set to delay for some seconds. Blenyo's program did well in this aspect.


Overall i think Jidez's GUI and all round balance gave him an edge over every other program.

in terms of searching i'll put blenyo's crawler as the best.
searches to the last page on NL.

In terms of gui, I think jidez or finco (i may be wrong) -
the guy that separates nairaland query from stackoverflow.
The rest just bundled them together. I think it's jidez.

But I think blenyo relied heavily on somebody's else's code/framework to acheive that feat.
I strongly think so.

so search wise - 98/100 (searchability) blenyo (for its deep searching)
@daniel the geek (interprete my input as you will ).

Also most folks dont have their code where anyone can see them!! - that should be a minus.
chinenyen, blenyo - have their code on github . I think only those two are visible.

I think one or two contestants differentiated between results from NL vs stackoverflow
while the others just provided a universal search box.
the interfaces that distinguished were the more usable one.

asalimpo:


in terms of searching i'll put blenyo's crawler as the best.
searches to the last page on NL.

In terms of gui, I think jidez or finco (i may be wrong) -
the guy that separates nairaland query from stackoverflow.
The rest just bundled them together. I think it's jidez.

But I think blenyo relied heavily on somebody's else's code/framework to acheive that feat.
I strongly think so.

so search wise - 98/100 (searchability) blenyo (for its deep searching)
@daniel the geek (interprete my input as you will ).

Also most folks dont have their code where anyone can see them!! - that should be a minus.
chinenyen, blenyo - have their code on github . I think only those two are visible.

I think one or two contestants differentiated between results from NL vs stackoverflow
while the others just provided a universal search box.
the interfaces that distinguished were the more usable one.

Jidez also has a repo where his codes lie..

Your points have been accepted, how do you rate others too over hundred?

@Danielthegeek
This challenge could have been solved with a desktop program that polls NL, parses the html and presents the data. Search wise it would probably be slower but it would probably be the most original and hardest
to write: html parser, indexer, network interface, gui etc.
What if a competitor took this approach-he'd lose out. Besides such code wouldnt need a server to run!!
just download and run from a desktop with internet access.

i just hope this helps, i dint put up my code on github, but this zip file attached here contains the small code i wrote

I can't continue to say the same thing. But can the judges allow me to make a modification. I will just be deleting a block of code, then I guarantee I will have all the topics here in less than 1 hour.

If not then we need one more judge to balance

asalimpo:


But I think blenyo relied heavily on somebody's else's code/framework to acheive that feat.
I strongly think so.

Thanks for your rating. But i did not relied heavily on somebody's else's code/framework to achieve that feat.

I will take out time to explain all I did there after the rain.

For the user interface; I am sorry i neglected that as I think to competition will be more about solving the problem (getting the functionality in place).

GitHub readme file: there was no time and i didn't wanted to modify the project after submission.

my judgement:

gui
-----
finco crawler (8.5/10) - distinquishes between target options. Nairaland logo is cool.

djade.net (9/10) - the green them is more appealing.
Boxed results is more beautiful.
also distinguishes between NL and stackoverflow
defaultly, searches both NL and S.O if no
option is selected(that's a plus). Nairaland logo is cool.

ademuanthony (6.5/10) - doesnt distinquish between target sites.


search functionality
-----------------------
finco crawler - (6/10) - hangs. doesnt respond even after an entry has been made sometimes.
clicking and unclicking a target site,often fails to trigger a crawl.

djade.net - (7/10) - is responsive. the most responsive i think , in terms to crawling while you are typing.
minus. irrelevant search results. crawling not deep.
e.g couldnt find this thread after i entered "programming competition"
djade.net , performs content crawling, which is a plus.

ademuanthony (9.3/10) - significantly fast crawling. deep crawling. But not instant (as you type rather after i finish with the input it crawls. If i pause for a brief second , it will crawl my input but if i type fast it waits. djade.net is the best so far in crawling as you type .i.e instantness).


repositories
--------------------
djade - (9/10) - githup repo with read me, detailed and comprehensive installation notes.
ademuanthony - (7/10) - github repo.
chinenyen - (7/10) - github repo.


@judges,
adjust the names - i used the site names to avoid confusing myself.
adjust the scores to your rating system.

The composition did not asked us to separate the target site. I is supposed to be a crawler for searching NL programming section and stactover flow. But in my search result you will see source wihich is eaither NL or stackoverflow

blenyo11:
The composition did not asked us to separate the target site. I is supposed to be a crawler for searching NL programming section and stactover flow. But in my search result you will see source wihich is eaither NL or stackoverflow

Then the marks go to them for creativity.
Their sites are more usable, because at a glance, the partitions tell where results are coming from.

@asalimpo this may not count again but Check my crawler. I have made the modifications you were looking for. And like I promised, I commented out the loop that tries to get all the comments then made sure you are searching for topics title.

http://search.djade.net

^^^ skipped the topic comments. The bots have now indexed all the topics in this section. @asalimpo can confirm.

Sorry, everyone. I actually work as an accountant and I've been busy over the past week so I had not the time to finish my amendments and make the search engine live. Nonetheless, I liked this challenge. I liked seeing what everyone else came up with. Most especially, I liked the fact that I actually learned something, and this new knowledge is definitely something I can use in the future.

In summary

Chinenyre--75/100
Blenyo- 80/100
Djade-90/100
Finco-70/100

Lo and behold, the winner of this competition is Jidez007 (Djade.net), followed by Blenyo11 (Ademuanthony) in second place and FincoApps in third place



[size=19pt]Djade.net ----196pts [WINNER] [/size]
[size=19pt]Ademuanthony ----175.8pts [FIRST RUNNER UP] [/size]
[size=19pt]FincoCrawler ----130.5pts [SECOND RUNNER UP] [/size]


To the competitors,
Thank you for participating, it was nice having you all in the competition.

To the Judges
Thank you for taking time out to allocate points.

To the onlookers that didn't say anything:
Thank you for looking, i hope you have learnt one, two or three things from this competition, it was nice having you on the benches. Hopefully next time you'd participate.

@Dhtml18, it's time to award the winners.

Blenyo11, Jidez007 and FincoApps, please send me a mail to daniel@rangostudios.com.

Santa maria! This is great.

Dudes, I meant winners, please communicate with me via the Nairaland pm system so that I can have your accounts (preferrably gtb accounts).

However, there is going to be a slight delay in the awards because my ATM card got stolen, and I should get a new one this week (not sure when). So all my online transfers are blocked at the moment sadly.

DanielTheGeek:

Lo and behold, the winner of this competition is Jidez007 (Djade.net), followed by Blenyo11 (Ademuanthony) in second place and FincoApps in third place



[size=19pt]Djade.net ----196pts [WINNER] [/size]
[size=19pt]Ademuanthony ----175.8pts [FIRST RUNNER UP] [/size]
[size=19pt]FincoCrawler ----130.5pts [SECOND RUNNER UP] [/size]


To the competitors,
Thank you for participating, it was nice having you all in the competition.

To the Judges
Thank you for taking time out to allocate points.

To the onlookers that didn't say anything:
Thank you for looking, i hope you have learnt one, two or three things from this competition, it was nice having you on the benches. Hopefully next time you'd participate.

@Dhtml18, it's time to award the winners.

Blenyo11, Jidez007 and FincoApps, please send me a mail to daniel@rangostudios.com.

well deserved... Djade.net definitely took it... but everything wasn't intricate as i expected..