Anyone know why it is so much faster than it was initially?

I suspect database indexing, but am not sure.

Seun is an avowed fan of database indexing. Nairalist, his first public Python app used database indexing extensively and am sure the lessons from that were transferred back to Nairaland. Besides, there might be some kind of primitive (as in first draft. not a competence assessment) cacheing behind the scenes.

kodewrita: ... caching behind the scenes.

^^^ agree much, I noticed nginx is being used, which is a great cache/proxy, if run in conjunction with Apache. Per the header, my guess is, it's more like a proxy, see the attachments...

I don't think he used indexing when he first rolled it out. It was really, really slow.

I don‘t really understand grammar easily in mathematics and programming class, if anyone will show me example of a dirty coding and example of the corresponding clean code.[img]http://www.dubaa.info/iPad.gif[/img]

I don't think he used indexing when he first rolled it out[img]http://www.keyforex.info/hmp.gif[/img]

I always create indexes when designing databases. It's inconceivable that I'd launch an important website without database indexes.

Seun: I always create indexes when designing databases. It's inconceivable that I'd launch an important website without database indexes.

What's powering the back-end?

Then how did it go from molasses slow to reasonably fast?

Kudos at seun. U get mouth

This is more like a cross btw mobile browser mod and desktop Mod.
ie the visual interface looks much like a mobile browser hook/mod. Thus making it faster to load etc on pc and mobile browsers, but with limited functionality.

ekt_bear: Anyone know why it is so much faster than it was initially?

I noticed this today nairaland, facebook and stackoverflow seem to be much faster than most other sites. You would notice it if you are on a slow Internet connection...

Seun: I always create indexes when designing databases. It's inconceivable that I'd launch an important website without database indexes.

But you haven't answered the question... Please we all want to learn...

here are my own suggestions:

He is obviously using database indexing esp with his primary keys and those columns he is performing join queries on.

I don't know waht database he is using but I think he will be using mysql (that is what the previous platform was based on anyway). He could also be using sqlite. This database is very fast.

Also I think seun is doing a lot of caching at the page level. Using tools like memcached or varnish to cache entire pages not just database results. U will notice this when u log out of nairaland, the topics u see on the front page are not as up to date as wot u saw when u were logged in. Also if u are logged in, general sites statistics like users birthdays, online users, topic stats, headers and footers are also cached. They do not need to be updated at every user request.

I don't think nairaland uses nay form of loadbalancing or high availablity cos if it did we will not be seeing "error bad gateway" when nigeria is playing a match

Kudos to seun and crew.

Nairaland's initial slowness was due to locks and deadlocks as a result of our decision to use InnoDB's strictest transaction isolation mode to protect the integrity of our database. We mitigated the problem a bit by breaking up long transactions, and tried to improve performance further with caching, but we still had to deal with occasional slow transactions locking up the forum a few times daily. So we switched to a system that uses the InnoDB's faster default isolation mode for hot, read-only pages. Writers block each other, but readers never block. This approach solved our problems and has scaled well with our increasing traffic over the past 11 months.

sarutobi: I don't know waht database he is using but I think he will be using mysql (that is what the previous platform was based on anyway). He could also be using sqlite. This database is very fast.

We use MySQL/InnoDB. It is much faster than SQLite and handles concurrency infinitely better. The query planner is not smart, but you can work around that.

Also I think seun is doing a lot of caching at the page level. Using tools like memcached or varnish to cache entire pages not just database results.

We don't do page-level caching. You can't do that on a forum like Nairaland. We do cache some hot queries, slow python functions, and page fragments. The idea of caching pages for an arbitrary amount of time (e.g. 5 minutes) is unacceptable to us. When people visit a page on a web forum, they expect to get the latest version of the page.

U will notice this when u log out of nairaland, the topics u see on the front page are not as up to date as wot u saw when u were logged in.

In our system, cached content is immediately invalidated when the underlying data is changed. We no not display stale content. When the contents of a page are changed, all cached items related to that change are instantly invalidated. If you notice this problem, it may be an issue with your ISP. Are you on the Etisalat network?

Seun: Nairaland's initial slowness was due to locks and deadlocks as a result of our decision to use InnoDB's strictest transaction isolation mode to protect the integrity of our database. We mitigated the problem a bit by breaking up long transactions, and tried to improve performance further with caching, but we still had to deal with occasional slow transactions locking up the forum a few times daily. So we switched to a system that uses the InnoDB's faster default isolation mode for hot, read-only pages. Writers block each other, but readers never block. This approach solved our problems and has scaled well with our increasing traffic over the past 11 months.


We use MySQL/InnoDB. It is much faster than SQLite and handles concurrency infinitely better. The query planner is not smart, but you can work around that.


We don't do page-level caching. You can't do that on a forum like Nairaland. We do cache some hot queries, slow python functions, and page fragments. The idea of caching pages for an arbitrary amount of time (e.g. 5 minutes) is unacceptable to us. When people visit a page on a web forum, they expect to get the latest version of the page.


In our system, cached content is immediately invalidated when the underlying data is changed. We no not display stale content. When the contents of a page are changed, all cached items related to that change are instantly invalidated. If you notice this problem, it may be an issue with your ISP. Are you on the Etisalat network?

Etisalat ke? nah. i use Ipnx.

Nice move on isolating the queries for read-only pages considering the fact that u will probably have 10x more reads than write queries / user request, readers will work concurrently without blocking each other. this will handle most of the traffic coming from the search engines.

I have always been wary of mysql and scale. For instance, my current project (jobler.me) makes use of Hbase. I started having problems when my recordset hit around 12million with mysql hence i just denormalized everything and then switched. I have always doubted mysql when data records starts to get into tens of millions. But in your case, u will still need the powerful transactions, joins etc hence Mysql is the only way.

Though judging from the current stats on nairaland and the rate at which it grows, i will suggest you start to consider other forms of structured data storage as u push the upper limits of a single dedicated box with mysql. Mysql does not scale well horizontally so sharding and replication might not do much good.

I see u have really hacked Mysql (query planner, optimizer, maybe innoDB itself) . Afterall why did they make them anyways

Cheers.