₦airaland Forum

Kodejuice:
Download link (Download nl-pageparse.zip), includes tests to see how it works and a manual.

Thanks so much!

Darangi007:
have u heard of kimono labs?

Never heard of it.
I'll check it out right now.
Thanks!

Kodejuice:
If the websites use RSS, you could just grab the url and use an XMLParser to conver it to object and arrays, XMLParser are found in all web prog-languages, or a plugin for parsing XML will certainly exist for the language u wish to use.

Or you could go hardcore, i actually built a Nairaland App for android using phonegap(screenshots included below), my popsi seized my Laptop untill i write my neco finish, after then i could push the app to Android app stores and probably build the ios version.

You know nairaland provides no API for getting it posts,it has RSS but it only includes 12 post from the FP (very annoying), so i used JavaScript to parse the pages fire for fire,

first of all i got the source code of each sections in nairaland, and i inspected the HTML, got the elements that holds the posts, comments, topics and stuffs like that, i then used jQuery to get the contents of these elements and stored them in an array, i did this for all sections apart from the FP, that one is quite different, different element for its posts and stuffs like that.

Acts much like an API but its just a hack, not saying u should follow this route, because its a very cumbersome task and a dangerous one any slight change in nairalands html may cause the app to fail, the app allows you to do everything, i couldnt include things like (login) so i added an option 'Nairaland web', gives u the nairaland webview inside the app, allowing you to do everything!, will be releasing the app after my Neco exam.

If you need, i could give you the script i wrote for parsing NL pages, it doesnt parse the (search, edit profile, new) and other irrelevant pages.

The screenshot i included is an old one, updated the app with more options b4 my Lapi got seized, options like (Trending, Recent, ...)

So, best of luck with the hacks!.

Thanks a lot for your help. A lot of the websites do use RSS, so I'll just extract XML content from those websites. Could you please send the script you used, I would really love to check it out and do let me know when you release the app.

elfico:
It depends on what you want to achieve. There is a firefox extension called Scrapbook that can save an entire website including links to other pages. To target some other form of data, you can try import.io

Essentially, I just want to extract the title text and the content text from the websites so that I can save it on to my program for later offline viewing. I want it to be fully automated daily, so I wanted to know if there are any data APIs out there for these websites.
Thanks for your help!

I want to be able to extract data from online news sites like punchng, naij, and nairaland for later offline viewing and just wanted to know if you guys know any APIs or tools that can help me extracting data with ease.
Thanks

I was wondering if there is a way to extract information/content from news articles like the trending posts from www.naij.com, www.punchng.com.
Are there any APIs available to simply extract data from those websites?