Updates pushed to my repo http://code.google.com/p/kadeer-jambpy/.

Zip file download, a few quirks I had seen have been fixed.

Would have wanted to create an installer for the project, but I think I have learnt enough Python for 4 days.

Lemme know what you all think.

Phew!.

I am willing to update and improve the code based upon suggestions new and later. Feel free to ask me any questions.

I do VB (not so much anymore), C#, Ruby, Ruby (lately), Javascript.

techytom: Finally... That is some cool strategy @kadeerna
Sharding+ caching, really cool

@techytom Thanks man.

Seun: @kadeerna: Wow, last day ninja entry. Oh my.

Hehe. *evil laugh*

Repo is at http://code.google.com/p/kadeer-jambpy/

This is my stab at an implementation of a relatively scalable implementation of a read heavy app which is the case of JAMB's examination results checking website. Even though I don't support the idea of limiting the dozens of applicable tools to the problem, I have come out on the other side with 1 more language under my belt, Python.

Before the replies begin to refer to this one, I want you to understand that this implementation / solution refers to the JAMB's candidate accessible database for the SOLE purpose of checking their results. That means JAMB can have multiple versions of their databases. A few used internally, but most importantly 1 candidates check their results against.

I was prompted to enter this challenge mainly by the many arguments I have seen on the thread with majority examining the problem of JAMB's catastrophic failure in 2012 to provide read access to results for ~ 1.5million between day 0 and 2 of releasing results, as an issue simply requiring MORE servers or hosting in the cloud. Yes that could help, yes that helps, but still a far off shot from all the hundred or thousands of other things architects & designers should work on before going for that "all reliable" solution of throwing more hardware at the problem. A few examples of such an optimization are the selection of proper database table field types, use of indexes on the tables, normalization OR denormalizaion, sharding and partitioning, caching and others I will talk about in detail given time. Just a bit about proper field type now.

Using chars in place of varchars in my implementation, varchar(50) and char(50) can hold up to 50 charachters of text, but the difference is that char only uses as much space as there are characters for on a row-by-row basis. Variable row sizing might sound good at first - after all, why leave unused space to waste? The problem lies in the fact that, becasue varchars can vary in size, mysql need to calculate the length of each varchar field in a row so it knows how far to jump to get the next record. Compare that to using char as the data type, where mysql can just add 50 (in my above example) to get to the end of one field and start of another.

Another optimization is the collation type and character set used for fields. The JAMB examination result checking website is a Nigerian+English only database. Probably to vague to comprende but my point is the database stores records and charachters comprising of english alphabet letters [A-Za-z] and numbers [0-9] a few other characters [.=]. If the designer of the database knows this, why use UTF, LATIN or other convoluted character set when ASCII can do the job. Comparing an ASCII record ex "left" = "right" will be faster than comparing Unicode or some other ISO encoded string of the same u"left" == u"right". Storing ASCII "left" COULD also save more space than storing u"left".

Normalization of such a READ heavy database like JAMB's results database I feel is simple illogical and unnecessary. Why even try to do a join? Just because I can do one is not necessarily the right approach to designing large apps. As much as possible, JOINS should be eliminated because they are evil (thinking Illuminati?). Better denormalize and partition or shard against a unique consistent field key. With normalization, single tables, single databases, single instances will grow large. Many rows in tables, logs, connection and threads to manage in the case of instances are there to manage. I will try to expand on this point and few others in a subsequent reply (if need be).

The source code is also dotted with suggestions and ideas as to how the a few things will change in a real implementation.

So, this is a result of 3 days of learning and coding Python. Depends on Bottles 0.11.dev, CherryPy 3.2.2., python3_memcached-1.44-py3.2, MySQLDb, Jinja2. CherryPy might need to be patched with the latest changes to the wsgiserver3.py source. The latest changes are can be applied by pasting wsgiserver3.py in the root of the repo in the "cherrypy/wsgiserver" folder in the "site-packages".

Begin by
- Running "Create DBs" under the "db-Ops" menu on the home page,
- Generating say any number of results and cards in using the "Number of results to generate" & "Number of cards to generate" forms at the top of the home page
- Running the "Load MySQL Memory Results DB" & "Load Memcache Results cache" to load the results data into MySQL memory tables and memcache cache.

Every time you restart you mysql server, make sure to run "Load MySQL Memory Results DB".

Oh, and last word. "OFF COURSE IT'S NOT COMPLETE"

Update:
Almost forgot, uncomment 1 of lines 51, 52, or 53 of result_controller.py to select a data store repo strategy. MySQL Shards, MySQL Memory or Memcache. Need I say restart the server.

Update
Crap, had pushed the wrong db setup script. Fixed now.