We use cookies to keep our site relevant and easy to use, your continued use of this site is consent that we may set several cookies (see our Privacy & Cookie Policy), click to always allow cookies from our site (and not see this notifcation on your next visit) or read more.Allow Cookies

EU legislation requires that all websites clearly specify if cookies are being used and their purpose, You can read more about how we use cookies (and which cookies we use) in our Privacy and Cookie Policy.

You will see this notification the first time you visit our website unless you accept cookies (in which case we'll set a cookie to remember thay you're happy for us to to set cookies!).

Posts Tagged crawler

XULRunner and Crowbar – Crawling of sorts?

This was going to be a tutorial on getting these two things running to achieve everything I want, sadly I can’t work out how to get the last step working, which is to navigate the returned Ajax page to allow me to extract different information. As such this is more a guide on getting the two things installed and working – if you have any more luck than I do on getting navigating Ajax working then let me know!! XULRunner First things first, I downloaded the Windows version of XULRunner from (look in the runtimes directory!): http://releases.mozilla.org/pub/mozilla.org/xulrunner/releases/ (Unpacking takes a while the 8.23MB download contained 302 items totalling 18.8MB!) Crowbar Not such a simple download for the uninitiated. It’s not… Continue reading »

by Keiron on November 30th, 2008 | Programming | 6 Comments » |

Ajax Crawling with PHP and Curl?

I apologise now if I put off some of the more general readers with this post, but I’ve struck upon a bit a problem! I have some automated code that I wrote using PHP and Curl, that retrieves a mountain of information from a website, does some statistical analysis on it and then presents me with a nice little report (having inserted the data into a MySQL database). It’s wonderful – to do the process manually would take maybe 2-3 hours every day, as it is I wake up to a nice report sat in my inbox everyday with all the information in it. Now I have a problem, the website that I crawl to get this information is converting… Continue reading »

by Keiron on November 29th, 2008 | PHP | 10 Comments » |