For example, A site like nairaland that has no API and I really want to make a client for it. So I decide since HTML is just markup, that if I parse the HTML and get whatever data I want from the markup and put into XML then use it for any evil stuff I want to do with it. What restrictions am I likely to encounter?

Before u process nairaland with an xml parser, u have 2 b sure dat nairaland is xhtml compliant, or else ur parser will keep throwing exception

Chimanet: Before u process nairaland with an xml parser, u have 2 b sure dat nairaland is xhtml compliant, or else ur parser will keep throwing exception

assuming I was able to parse it successfully

Maybe I decide to let the xml converter software reside on a proxy server.... I make requests to it, it makes the request to the site in question strips it of useless data then wraps it in xml and passes it to my client... I can always use any user agent even a mobile browser's user-agent to reduce size of data to parse...
Example request for a user's profile on NL assuming I have access to all the headers...
GET http:\\nairaland.com\user HTTP 1.1
accept-encoding: *\*
Host: www.nairaland.com
user-agent: mozilla.....

Will this work?

with nairaland undergoing continuous updates you can bet your code will break every 2/3 days
expect this as long as seun breaths.

webdezzi: with nairaland undergoing continuous updates you can bet your code will break every 2/3 days
expect this as long as seun breaths.

so the only problem I'll have is if the webmaster updates the site?

webdezzi: with nairaland undergoing continuous updates you can bet your code will break every 2/3 days
expect this as long as seun breaths.

so the only problem I'll have is if the webmaster updates the site? But then he will most likely be adding new features to his site than to change the name of the <div> that contains posts' data..

Is quite an unrealizable project my dear, if is feasible nairaland would have brought out an api since easily, nairaland was not designed as a restful webservice from scratch, its more of a usual website design architecture.

Chimanet: Is quite an unrealizable project my dear, if is feasible nairaland would have brought out an api since easily, nairaland was not designed as a restful webservice from scratch, its more of a usual website design architecture.

so you understand what I mean and now you wonder why it hasn't been done... I don't have to care about the underlying architecture as all my client needs is to know how the data is marked up..

What you want to do is screen scraping and its very possible. I recently scraped nl's homepage to confirm a users session In one of my projects. One of the problem with this approach is the markup changing. I don't think that would happen often though. But then, if seun finds out that you are eating up his bandwidth, he might just ban your crawlers.

worldbest: What you want to do is screen scraping and its very possible. I recently scraped nl's homepage to confirm a users session In one of my projects. One of the problem with this approach is the markup changing. I don't think that would happen often though. But then, if seun finds out that you are eating up his bandwidth, he might just ban your crawlers.

scraping.. Hmmm.. Quite a befitting name. How can the webmaster find out cos for all the server knows am browsing with my mozilla... NL is just an example I don't plan on using it on NL

How the server finds out depends on how often your crawlers visit. Even if you use a mozilla user-agent, the admin might suspect you if they see that an ip is visiting too fast/often and they may be able check how much bandwidth your crawler's ip has used up. Some websites do this automatically, if your crawler is irresponsible enough to crawl pages without adequate delays, they may block it by redirecting to a captcha page. Nothing beats an API.

worldbest: How the server finds out depends on how often your crawlers visit. Even if you use a mozilla user-agent, the admin might suspect you if they see that an ip is visiting too fast/often and they may be able check how much bandwidth your crawler's ip has used up. Some websites do this automatically, if your crawler is irresponsible enough to crawl pages without adequate delays, they may block it by redirecting to a captcha page. Nothing beats an API.

okay. This makes sense. But the webmaster is aware... So I don't have to worry.

Since the webmaster is aware, that's fine. Just run a test on the page regularly to be sure the markup hasn't changed.

worldbest: Since the webmaster is aware, that's fine. Just run a test on the page regularly to be sure the markup hasn't changed.

okay bosss. Thanks

Well

It will be pretty annoying unless the website itself provides one. You'll have to replicate lots of NL functionality, write a scraper, etc.

ekt_bear: It will be pretty annoying unless the website itself provides one. You'll have to replicate lots of NL functionality, write a scraper, etc.

you mean like going through links and figure out how the requests are constructed?

hmm learning

lordZOUGA:
you mean like going through links and figure out how the requests are constructed?

What actions do you want your API to allow? If read only operations, then you can do it. But write sort of operations...more difficult.