I don’t know if this exists or not, so I’m either making a call for a reference or I’m giving away a business idea (or possibly sounding really stupid)…
…but in our top-secret development of BBX, it’s come up that a web indexing power would be great. The thing is, we’re not going to pump a dime into developing a web crawling application, because that’s super expensive (Jimmy Wales says so!), and we’re just looking into it as a feature, but not a central one.
So this brings me to my idea/request… a crawler for HIRE.
If I could call up Google or Yahoo or any other of the giants going around, indexing the net, collecting data and say, “Hey! Would you mind looking for this specific data for me while you’re out there?” then I would certainly pay for that data harvesting capacity.
Of course I’m sure there some way to do this now, but what I’m looking for specifically is a transparent, scalable, service provider, not some fly-by-night hack-job.
You've found Nate Westheimer's blog. Nate wears many hats. He's the Entrepreneur in Residence at 
2 responses so far ↓
1 epc // Apr 10, 2007 at 7:43 pm
Can you use the Yahoo! search services? They return data in a variety of formats and let you programmatically hit Yahoo’s search database (I’m using it instead of building my own crawler for a project). Amazon’s a9 had something similar but it’s unclear if it’s still available since they’ve cut back on a9 development.
2 Benjamin Stein // Apr 11, 2007 at 2:56 pm
Try Alexa’s web crawl:
“The Alexa Web Search Platform provides public access to the vast web crawl collected by Alexa Internet. Users can search and process billions of documents — even create their own search engines — using Alexa’s search and publication tools. Alexa provides compute and storage resources that allow users to quickly process and store large amounts of web data. Users can view the results of their processes interactively, transfer the results to their home machine, or publish them as a new web service.”
https://websearch.alexa.com/welcome.html
Leave a Comment