Earlier last year the data produced by NASA’s Stardust spacecraft was made available for “crowdsource” style analysis by UC Berkeley’s Stardust@home project. Interestingly, Berkeley partnered with another private sector innovator - Amazon - to use Amazon’s S3 web services based storage to store the millions of images resulting from the Stardust project.

These are excellent examples of Federal agencies tapping the expertise and innovation of private sector companies. It will be interesting to see how other projects might take advantage of the technology and services being developed in projects like this. The Large Hadron Collider, for example, is set to go online later this year. I wonder if we will be able to Google that data!

(CeC regularly posts to the TeraGrid Whiteboard.)

Thousand years ago: science was empirical, describing natural phenomena

Last few hundred years: theoretical branch, using models, generalizations

Last few decades: a computational branch, simulating complex phenomena

Today: data exploration (eScience)–unify theory, experiment, and simulation. (Data captured by instruments, or generated by simulator; processed by software; information/knowledge stored in computer; scientist analyzes database/files, using data management and statistics.)

Jim’s equating of “eScience” with “data exploration” seems a little too narrow. (John Taylor, who coined the term, had a somewhat broader definition: “e-Science will refer to the large scale science that will increasingly be carried out through distributed global collaborations enabled by the Internet.” ) However, the growing importance of data can hardly be overstated, and Jim’s perspectives are worthy of careful consideration, especially by those who think of “computation and science” as being entirely about simulation.

[From http://ianfoster.typepad.com/blog/2006/12/the_nature_of_e.html] _uacct = “UA-686198-1″; urchinTracker();

Now Karim Lakhani of Harvard Business School is looking into whether such techniques can be applied to scientific problems. From a recent article (and interview):

In a perfect world, scientists share problems and work together on solutions for the good of society. In the real world, however, that’s usually not the case. The main obstacles: competition for publication and intellectual property protection.

…

What [Karim Lakhani] and his coauthors discovered: “broadcasting” or introducing problems to outsiders yields effective solutions. Indeed, it was outsiders—those with expertise at the periphery of a problem’s field—who were most likely to find answers and do so quickly.

He cites a few intriguing examples, including both more traditional “competitions” (with prizes) that have yielded novel solutions, and also more novel approaches such as MathWorks’ collaborative programming contest. I am also reminded of NASA’s involvement of the general public in analyzing image data from its comet return mission (StarDust@home), and of the nice work at CMU on enlisting people (under the guise of a game) to tag photos. I suspect that there is much much more to be done here.

(From http://ianfoster.typepad.com/blog/2006/12/open_source_pro.html.)