Another idea down the drain

The most talked about news item for the past day or so has been the release of Yahoo! Pipes. The guys over at the O'Reilly radar blog do a pretty good job of giving a rundown of what the new service is (here and here). Apparently it's brand spanking new out of a new initiative put in place by Yahoo! called the Brickhouse (it's their "innovation incubator" for want of a better term) which is being driven by Caterina Fake of Flickr fame (according to this article).

Why am I mentioning this? Because I've been meaning to get cracking on prototyping something like this for awhile now (I can't prove it, so you'll have to take my word)! Like the title of this post says, another idea down the drain...no pun intended.

You may have gathered I know IBM's Tivoli Security suite of products very well (Disclaimer: I still work for IBM for at least another week but I'm not trying to sell any product here, hence I've deliberately left out linking to it - you want to read about it, go find the links yourself). One of the products in the suite is something called IBM Tivoli Directory Integrator (ITDI). It's essentially a data integration tool, but with lots of drag and drop functionality, scripting capabilities (for some business logic) and out of the box connectors into many data sources (databases, LDAPs, filesystems, Web Services etc.). In other words, you don't need to be a hard core programmer to use it and build integration solutions. So to explain a simple use case, you could grab some data from one or more sources, do something with that data (combine it, use it as lookup inputs etc.) and output results along the way into other sources...and still grab data at any point to help with the tasks at hand. By sources, I mean anything that a connector can get to ("But hey", you say, "that's kinda what Pipes does". "Yes, I know").

I've been a believer in this product because it makes life easier for those of us that need to integrate data...especially identity data because of the very nature of having to figure out relationships between disparate data sets. In fact, I gave a presentation on what the product could do at a Tivoli conference last year. I used an example where I had ITDI listening for a HTTP request from a user (via a web browser) asking for specific types of events occurring in a location during a specified time period. ITDI would then fetch information from an online event database (via a REST web service), match each returned event with its top Google search result and the location coordinates for each event, "mash" it all together and return the result to the user's browser in the form of KML data which could be opened in Google Earth to view the location and details of each event returned. For those interested (I'd imagine this to be only the ones that use ITDI - the rest of you can ignore this part), the presentation I used can be found here and the actual ITDI solution package can be found here.

The stuff I've just mentioned is basically to illustrate an example use case where ITDI is useful. Yahoo! Pipes uses pretty much the same concept. Get data from various places, mash them all together and get something more useful from it at the end. At the moment, I think Pipes focuses on RSS feeds, but it is not inconceivable that they'll extend it to other sources. Just build more "connectors". I've always felt it was only a matter of time before someone released something like Yahoo! Pipes for use on the web by the masses. I've known for awhile how powerful the concepts behind Yahoo! Pipes and ITDI are first hand. The excitement in the online community over it just reinforces my thinking. I just didn't get off my butt to do anything about it. Procrastination's been my enemy as usual (I already whinged about this in an earlier post).

Well, I shall chalk this one up as another "Gosh darnit, hot diggity dang!" moment.

Now excuse me while I go code something up...anything!