I need this: For Hire Web Crawler
I don't know if this exists or not, so I'm either making a call for a reference or I'm giving away a business idea (or possibly sounding really stupid)...
...but in our top-secret development of BBX, it's come up that a web indexing power would be great. The thing is, we're not going to pump a dime into developing a web crawling application, because that's super expensive (Jimmy Wales says so!), and we're just looking into it as a feature, but not a central one.
So this brings me to my idea/request... a crawler for HIRE.
If I could call up Google or Yahoo or any other of the giants going around, indexing the net, collecting data and say, "Hey! Would you mind looking for this specific data for me while you're out there?" then I would certainly pay for that data harvesting capacity.
Of course I'm sure there some way to do this now, but what I'm looking for specifically is a transparent, scalable, service provider, not some fly-by-night hack-job.