We are starting a windows application development in python.
This application needs on a remote database server to function, a lot of polling needs to be done especially to check for updates to the remote database, these updates are done via any of mobile, web, external api responses.
We have decided to use sockets to connect to the server from our application as this allows the remote server to notify our app the moment updates become available
this should definitely increase response time also as a connection has already been established. (becomes boring when you have to wait longer for responses to every clicks)

We are putting scalability into consideration, and thats why we arent looking at replicating the remote data to a local database (imagine data size growing into gigabytes over time and new installation means downloading 5GB before initializing)

the windows app will run on up to 100 computers from different locations.
From experience, how will you approach this.

use sockets, poll for changes or ?.
Thanks.

I will strongly advise you use sockets,do not ask me why but it is preferable in this situation of yours?

why sockets? to expensive to manage ...

I would simply request version info(serial) from the server and compare with the mine(client).
Polling is fine, but it has to be handled intelligently.
e.g. Have fixed time during the day to poll and download(the way most anti virus handle their updates).
Update the database when there's a functional need.

But your 5GB also throws up questions. More info about the nature of your app and the master data might be insightful.

Thanks for your contributions.

The master data will reside on a remote server, reason is to make it accessible via webpages even when systems are shutdown and everyone is asleep.

customers should be able to make changes to own account and same changes accessible to coy staff when customer calls in.
meaning, we(i.e customer and company staff) want to be messing with up-to-date data at every point in time.

The 5GB i quoted is an example.
lets say in 6 months of intensive usage, database size surges to 5 or 10GB.
and a new staff joins the company and needs the software installed on her system.
replicating remote databases to local will pose many serious problems.
e.g downloaded data will be out of date as the come in
a single update by a staff needs be updated across all staff systems nationwide
etc.
for that we rule out replication.


Now we have polling and socket
can you expand on your "expensive to manage"

An image file gives you 24 bits per pixel and 256 options to play with, all in an extremely compact space.
Dude, if you have numeric id fields, encode changed record ids in a tiny bitmap or gif file that you can either request or read on the server only when you need, instead of continously polling. That will save you tonnes of bandwidth and greatly simplify your architecture.

Beaf: An image file gives you 24 bits per pixel and 256 options to play with, all in an extremely compact space.
Dude, if you have numeric id fields, encode changed record ids in a tiny bitmap or gif file that you can either request or read on the server only when you need, instead of continously polling. That will save you tonnes of bandwidth and greatly simply your architecture.

that will be a hack, too early to start looking in this direction.
Thanks very much for your input.

It's very difficult to contribute without knowing the kind of application you're writing.
All the solutions discussed could work but the right one to use depends on the actual problem.

What do you mean by:

these updates are done via any of mobile, web, external api responses.

?

deleted

Seun: It's very difficult to contribute without knowing the kind of application you're writing.
All the solutions discussed could work but the right one to use depends on the actual problem.

skydancer: What do you mean by:

?

i mean user can access the database via a web page, maybe from their laptop
via phone, eg blackberry, iphone
via external api e.g we will be pulling data from another server's web service and updating our master database.

We have made a decision but i will like to still have your input.

dude, use socket. Iocal database just ain't right. Besides your updates should be to your app's libraries and not replace the whole app...

Depending on the kind of data that is being pulled/pushed -- generally I'd think of a local map-cache (the GMail model)

-- create a pointer of the DB records on a local map (generally this prevents user's from banging the DB all the time), this "validates" its states with the "master" DB, remember, it's just a pointer, so it's relatively cheap.
-- fetch the most recent change 'on-demand' [asynchronously?] (and that kinda solves "having to wait"

Actually, I think there is a name for this kind of thing: "publish/subscribe" (but again, it depends on the kind of data we're talking about here)