Welcome, Guest: Register On Nairaland / LOGIN! / Trending / Recent / New
Stats: 3,143,484 members, 7,781,450 topics. Date: Friday, 29 March 2024 at 02:34 PM

A New Search Engine - Programming (3) - Nairaland

Nairaland Forum / Science/Technology / Programming / A New Search Engine (7545 Views)

Meet The 19-year-old Who Built A New Search Engine To Rival Google And Youtube / Do You Think You Can Develop A Search Engine From Scratch? / Programming Competition: Search Engine Task. Rewards up for grabs. (2) (3) (4)

(1) (2) (3) (4) (Reply) (Go Down)

Re: A New Search Engine by Nobody: 3:43pm On May 17, 2016
guru01:

How do you know that Php is not powerful enough for this task?
Even me that I used to say that PHP is my strongest language (with all my bragado) cannot answer this your question - oya over to the goland and java boys.
Re: A New Search Engine by sleepingdemon: 4:00pm On May 17, 2016
dhtml18:

Even me that I used to say that PHP is my strongest language (with all my bragado) cannot answer this your question - oya over to the goland and java boys.

Using php and mysql isnt the best choice really because:
To scale PHP on a relational database, you need to shard your data. This means splitting the data by some reasonable key. This might mean East Coast customers go on one RDBMS, Midwest customers go on another, and the West Coast on a third. This is a lot of complexity to swallow when you chose PHP because it was "simple" and "free."
But like i said, till your app grows to when scaling is most needed, its not a bad idea using php alongside a no-sql database not a RDBMS database.
Re: A New Search Engine by Nobody: 4:36pm On May 17, 2016
I use many technologies, and i am sure when next i do a review of my skills - MEANSTACK/JAVA should probably top
Re: A New Search Engine by sleepingdemon: 4:43pm On May 17, 2016
dhtml18:
I use many technologies, and i am sure when next i do a review of my skills - MEANSTACK/JAVA should probably top
i'm just here waiting for google to move on to swift so i dont have to hear the name JAVA all the time. i slightly hate that language.
Re: A New Search Engine by Olumyco(m): 5:30pm On May 17, 2016
sleepingdemon:

i'm just here waiting for google to move on to swift so i dont have to hear the name JAVA all the time. i slightly hate that language.

I can't laugh oooooo.
Re: A New Search Engine by micodon(m): 7:41pm On May 17, 2016
sleepingdemon:

i'm just here waiting for google to move on to swift so i dont have to hear the name JAVA all the time. i slightly hate that language.

on Android alone, you mean?
Re: A New Search Engine by micodon(m): 8:10pm On May 17, 2016
guru01:

How do you know that PHP is not powerful enough for this task?

The very first rule in developing an app is: CHOOSE THE RIGHT TOOL(S) FOR THE JOB. In choosing "THE RIGHT TOOLS", you look at performance, scalabilty, ease of development and others

PHP has been my first choice language for close to a decade now, but TBT its only REALLY good at making websites and some web apps.
Last Year i wrote an eCommerce application with my HIGHLY DECOUPLED PHP framework (a "HELLO WORLD" benchmark gives about 1,300 rps). Benchmarks showed that it ran at about 250rps (Opencart runs at about 60rps). But it starts dropping requests when there are above 500 concurrent users.

I rewrote the same application with GOLANG and it handles over 5 million concurrent requests with breeze at about 6,500 requests per second with less than 15MB or RAM.

The Point? In a PERFORMANCE CRITICAL Application, PHP will fall your hands....It was designed for just short-lived webpages. Here are some reasons why PHP is not suitable for PERFORMANCE CRITICAL APPLICATIONS

1. Blocking I/O
2. No Parallel Processing out of the box
3. No Multi-threading out of the box.
4. Not Suitable for background Tasks (Remember your crawler)
5. Most probably used with Apache which spins up a thread for every request
6. Dies after every request
7. Not good for Computationally Intensive tasks (i.e Search engine analytic, ranking system, etc)
8. Not Memory Efficient
9. Extremely Poor Concurrency Model

In designing and developing your application, NEVER EVER LET THE DEVELOPMENT LANGUAGE IMPLEMENTATION BE YOUR BOTTLENECK...THAT PROBLEM SHOULD BE YOUR DB IMPLEMENTATION.

Should you use PHP now until you have performance or scalability issues? ABSOLUTELY NOT. By the time you get to that problem, you'll have had a huge PHP code-base which would make transitioning to another language NEAR IMPOSSIBLE. If you think i'm lying, ASK FACEBOOK.

JAVA is good (except for the JVM being resource-hungry)
GOLANG has its Limitations but its good
RUST is a new kid on the block, but is EXCEPTIONAL
JULIA,
SCALA (JVM Problem again)
HASKELL
C/C++

One of these could be used in combination with any of the following:
Python
Nodejs (Not a language, but an execution environment)
LUA, etc

1 Like

Re: A New Search Engine by sleepingdemon: 10:20pm On May 17, 2016
micodon:


on Android alone, you mean?
yeah yeah, but we all know most people this days learn java only for andriod development.
Re: A New Search Engine by sleepingdemon: 10:22pm On May 17, 2016
micodon:


The very first rule in developing an app is: CHOOSE THE RIGHT TOOL(S) FOR THE JOB. In choosing "THE RIGHT TOOLS", you look at performance, scalabilty, ease of development and others

PHP has been my first choice language for close to a decade now, but TBT its only REALLY good at making websites and some web apps.
Last Year i wrote an eCommerce application with my HIGHLY DECOUPLED PHP framework (a "HELLO WORLD" benchmark gives about 1,300 rps). Benchmarks showed that it ran at about 250rps (Opencart runs at about 60rps). But it starts dropping requests when there are above 500 concurrent users.

I rewrote the same application with GOLANG and it handles over 5 million concurrent requests with breeze at about 6,500 requests per second with less than 15MB or RAM.

The Point? In a PERFORMANCE CRITICAL Application, PHP will fall your hands....It was designed for just short-lived webpages. Here are some reasons why PHP is not suitable for PERFORMANCE CRITICAL APPLICATIONS

1. Blocking I/O
2. No Parallel Processing out of the box
3. No Multi-threading out of the box.
4. Not Suitable for background Tasks (Remember your crawler)
5. Most probably used with Apache which spins up a thread for every request
6. Dies after every request
7. Not good for Computationally Intensive tasks (i.e Search engine analytic, ranking system, etc)
8. Not Memory Efficient
9. Extremely Poor Concurrency Model

In designing and developing your application, NEVER EVER LET THE DEVELOPMENT LANGUAGE IMPLEMENTATION BE YOUR BOTTLENECK...THAT PROBLEM SHOULD BE YOUR DB IMPLEMENTATION.

Should you use PHP now until you have performance or scalability issues? ABSOLUTELY NOT. By the time you get to that problem, you'll have had a huge PHP code-base which would make transitioning to another language NEAR IMPOSSIBLE. If you think i'm lying, ASK FACEBOOK.

JAVA is good (except for the JVM being resource-hungry)
GOLANG has its Limitations but its good
RUST is a new kid on the block, but is EXCEPTIONAL
JULIA,
SCALA (JVM Problem again)
HASKELL
C/C++

One of these could be used in combination with any of the following:
Python
Nodejs (Not a language, but an execution environment)
LUA, etc
op, just in case my oga here has killed you with too much technicals here, the pin point of what he is saying is php dosent handle concurrency very well, and its also having blocking i.o's and for a search engine on the other hand, concurrency is key.
Re: A New Search Engine by guru01(m): 10:29pm On May 17, 2016
micodon:


The very first rule in developing an app is: CHOOSE THE RIGHT TOOL(S) FOR THE JOB. In choosing "THE RIGHT TOOLS", you look at performance, scalabilty, ease of development and others

PHP has been my first choice language for close to a decade now, but TBT its only REALLY good at making websites and some web apps.
Last Year i wrote an eCommerce application with my HIGHLY DECOUPLED PHP framework (a "HELLO WORLD" benchmark gives about 1,300 rps). Benchmarks showed that it ran at about 250rps (Opencart runs at about 60rps). But it starts dropping requests when there are above 500 concurrent users.

I rewrote the same application with GOLANG and it handles over 5 million concurrent requests with breeze at about 6,500 requests per second with less than 15MB or RAM.

The Point? In a PERFORMANCE CRITICAL Application, PHP will fall your hands....It was designed for just short-lived webpages. Here are some reasons why PHP is not suitable for PERFORMANCE CRITICAL APPLICATIONS

1. Blocking I/O
2. No Parallel Processing out of the box
3. No Multi-threading out of the box.
4. Not Suitable for background Tasks (Remember your crawler)
5. Most probably used with Apache which spins up a thread for every request
6. Dies after every request
7. Not good for Computationally Intensive tasks (i.e Search engine analytic, ranking system, etc)
8. Not Memory Efficient
9. Extremely Poor Concurrency Model

In designing and developing your application, NEVER EVER LET THE DEVELOPMENT LANGUAGE IMPLEMENTATION BE YOUR BOTTLENECK...THAT PROBLEM SHOULD BE YOUR DB IMPLEMENTATION.

Should you use PHP now until you have performance or scalability issues? ABSOLUTELY NOT. By the time you get to that problem, you'll have had a huge PHP code-base which would make transitioning to another language NEAR IMPOSSIBLE. If you think i'm lying, ASK FACEBOOK.

JAVA is good (except for the JVM being resource-hungry)
GOLANG has its Limitations but its good
RUST is a new kid on the block, but is EXCEPTIONAL
JULIA,
SCALA (JVM Problem again)
HASKELL
C/C++

One of these could be used in combination with any of the following:
Python
Nodejs (Not a language, but an execution environment)
LUA, etc
My brother leave all these English for British.
The only reason your Php app is slow is due to your logic, and also because Php is not a strict language.
If you write Php the way you write your c++ you will notice improvement.
Beside this is just a start, there is always room for other languages.

3 Likes

Re: A New Search Engine by guru01(m): 10:36pm On May 17, 2016
sleepingdemon:


Using php and mysql isnt the best choice really because:
To scale PHP on a relational database, you need to shard your data. This means splitting the data by some reasonable key. This might mean East Coast customers go on one RDBMS, Midwest customers go on another, and the West Coast on a third. This is a lot of complexity to swallow when you chose PHP because it was "simple" and "free."
But like i said, till your app grows to when scaling is most needed, its not a bad idea using php alongside a no-sql database not a RDBMS database.
To my understanding, mysql can handle millions of data and is very much scalable.
The only limitation is the app logic, how you code.

2 Likes

Re: A New Search Engine by micodon(m): 10:50pm On May 17, 2016
guru01:

My brother leave all these English for British.
The only reason your Php app is slow is due to your logic, and also because Php is not a strict language.
If you write Php the way you write your c++ you will notice improvement.
Beside this is just a start, there is always room for other languages.

You see, this is one problem with some people who know only 1 or 2 languages of the same type. They never listen. I was like this about 7 years ago.

The best PHP (or an interpreted language) code of an app will ALWAYS RUN TONS OF MAGNITUDE SLOWER than the best c++ (or any compiled language).

If you read my post you'd notice that my PHP framework clocks at about 1,300 rps which makes it one of the fastest PHP frameworks ever.

Why are Facebook transpiling their PHP. codes to intermediate byte code if running PHP is good enough.


Please write a simple application that prints the first 10000 numbers of the fibonacci series in PHP and any of the languages I listed before say JAVA and run a benchmark. The PHP script may even run out of memory .

1 Like

Re: A New Search Engine by sleepingdemon: 10:51pm On May 17, 2016
guru01:

To my understanding, mysql can handle millions of data and is very much scalable.
The only limitation is the app logic, how you code.

mysql uses one connection pool, so if you have 10 users launching your app at the same time, and the 10 isers need to run 100 queries each, mysql lines up the 1000 queries from the 10 users and executes them one after the other.
that said, the 10th user gets his data last and slower. now apply that logic to 1 million users running a query that searches a mysql database of over 40tb of data....
why do you think even mysql enginners wont tell you its scalable, rather they would optimize indexes for faster search time and at the worst, they master - slave replication, which is almost the same thing as data sharding which i mentioned.
i hope this gives u a bit more insight to your understanding.
Re: A New Search Engine by micodon(m): 10:54pm On May 17, 2016
sleepingdemon:

mysql uses one connection pool, so if you have 10 users launching your app at the same time, and the 10 isers need to run 100 queries each, mysql lines up the 1000 queries from the 10 users and executes them one after the other.
that said, the 10th user gets his data last and slower. now apply that logic to 1 million users running a query that searches a mysql database of over 40tb of data....
why do you think even mysql enginners wont tell you its scalable, rather they would optimize indexes for faster search time and at the worst, they master - slave replication, which is almost the same thing as data sharding which i mentioned.
i hope this gives u a bit more insight to your understanding.

Bros, leave him. I don't think he wants to learn. Let him do his PHP and MYSQL. even Rasmus will chuckle at him
Re: A New Search Engine by guru01(m): 10:56pm On May 17, 2016
micodon:


You see, this is one problem with some people who know only 1 or 2 languages of the same type. They never listen. I was like this about 7 years ago.

The best PHP (or an interpreted language) code of an app will ALWAYS RUN TONS OF MAGNITUDE SLOWER than the best c++ (or any compiled language).

If you read my post you'd notice that my PHP framework clocks at about 1,300 rps which makes it one of the fastest PHP frameworks ever.

Why are Facebook transpiling their PHP. codes to intermediate byte code if running PHP is good enough.


Please write a simple application that prints the first 10000 numbers of the fibonacci series in PHP and any of the languages I listed before say JAVA and run a benchmark. The PHP script may even run out of memory .

You may be right according to your understanding, but I will like to get to that bridge where Php can not sustain the app.

1 Like

Re: A New Search Engine by guru01(m): 11:00pm On May 17, 2016
sleepingdemon:

mysql uses one connection pool, so if you have 10 users launching your app at the same time, and the 10 isers need to run 100 queries each, mysql lines up the 1000 queries from the 10 users and executes them one after the other.
that said, the 10th user gets his data last and slower. now apply that logic to 1 million users running a query that searches a mysql database of over 40tb of data....
why do you think even mysql enginners wont tell you its scalable, rather they would optimize indexes for faster search time and at the worst, they master - slave replication, which is almost the same thing as data sharding which i mentioned.
i hope this gives u a bit more insight to your understanding.
This is a fat lie, mysql can handle multiple connection to same data. Please read about mysql properly.
http://stackoverflow.com/questions/14901508/maximum-concurrent-connections-to-mysql
Re: A New Search Engine by guru01(m): 11:04pm On May 17, 2016
Please make una leave me let me use want am comfortable with, and you guys can you what you wish.
Like I stated early, the search engine is slow not because Php or mysql is bad or can handle the task, but due to error on my part.
I am coming out with a fix soon.

3 Likes

Re: A New Search Engine by micodon(m): 11:07pm On May 17, 2016
guru01:

This is a fat lie, mysql can handle multiple connection to same data. Please read about mysql properly.

Did he say anything against that? This guy sef... He only told you how the multiple connections are handled... IN A DARN CONNECTION POOL
Re: A New Search Engine by sleepingdemon: 11:13pm On May 17, 2016
i officially give up on this thread. any other thing i would do here is to read comments, troll and sip the new holandia youghurt...yummy something.
Re: A New Search Engine by guru01(m): 11:14pm On May 17, 2016
micodon:


Did he say anything against that? This guy sef... He only told you how the multiple connections are handled... IN A DARN CONNECTION POOL
Show my the doc that states that mysql uses one connection pool.
Re: A New Search Engine by micodon(m): 11:18pm On May 17, 2016
guru01:

Show my the doc that states that mysql uses one connection pool.

If u understand connection pooling, you wouldn't even ask that question. I also give up on the thread. Go on with php and mysql
Re: A New Search Engine by FincoApps(m): 12:27am On May 18, 2016
I really understand the OP's point.... when it get's to that stage, he'll change technology.

For example, 1 of my apps has a chat room, for the chat room I used a 3rd party cat lib which allows up to 20 simultaneous messages/sec for free users. I didn't say because of this limitation, I won't implement the chat room until I write my own chat script/ have a dedicated server.

Right now, I'm approaching that limit , but already have what I need to upgrade ready... so guys leave the OP let him do what works for him.... after all, with the current status of his engine, he shouldn't experience issues
Re: A New Search Engine by worldbest(m): 6:42am On May 18, 2016
sleepingdemon:
in all of this, that guy that asked you to use golang, i need to flog him very hard. ehen, that advice on java is still fair, on the other hand, what do i know, just in case you're more comfy with php, you can use mongodb as your database instead, when the app outgrows php, you can move to another lang. or if ure a javascript person, mean stack is actually useful.

What is the issue with golang? A project like this needs languages with superior concurrency models and golang and scala are perfect for that. Your crawlers and parsers can be written in golang, scala/java while your client application can be written preferrably in Node.js or PHP .

1 Like

Re: A New Search Engine by losprince(m): 2:42pm On May 18, 2016
@op yours is the worst search engine i have ever stumbled upon undecided
Re: A New Search Engine by Nobody: 2:44pm On May 18, 2016
^^^maka why? it is still in active development
Re: A New Search Engine by losprince(m): 3:22pm On May 18, 2016
undecided
Re: A New Search Engine by Nobody: 3:55pm On May 18, 2016
^^^wait,what kind of search be that now? All me i want is for the search to understand some keywords like:
- goat (top priority e.g. our president is a goat)
- boko haram
- fulani herdsman
- snake
- tiwa savage
after that, then we can talk business!
Re: A New Search Engine by sleepingdemon: 5:49pm On May 18, 2016
dhtml18:
^^^wait,what kind of search be that now? All me i want is for the search to understand some keywords like:
- goat (top priority e.g. our president is a goat)
- boko haram
- fulani herdsman
- snake
- tiwa savage
after that, then we can talk business!

snake, you must be lala's brother
Re: A New Search Engine by Nobody: 6:14pm On May 18, 2016
Coughs, you can also add crocodile to the list.
Re: A New Search Engine by Pastev1(m): 2:17pm On May 19, 2016
Congrats guys, nice job; but i think u could use something meaningful such as searchng (huntng) instead of sachng.
But trust me u try! kip on.

1 Like

Re: A New Search Engine by Nmeri17: 7:38am On May 20, 2016
Constructive arguments. This section don dey sweet again. hallelujah!
Re: A New Search Engine by guru01(m): 3:09pm On May 24, 2016
Hey guys, Sachng.com is back and better.
Thanks to all those that drop their little dislike and like.
@dhtml18 you goat search is now activated.
CHeers.

1 Like

(1) (2) (3) (4) (Reply)

Download All Phone Number Generator, Imei Tweaking And Bb Imei Gen. For Pc / *~ Omo-to-dun Voted Programming Section Poster Of The Year *~ Congratulations!! / No Java Required: Write Android Apps In C#

(Go Up)

Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health
religion celebs tv-movies music-radio literature webmasters programming techmarket

Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 63
Disclaimer: Every Nairaland member is solely responsible for anything that he/she posts or uploads on Nairaland.