Tapping Private Sector Innovation

by catlett on January 12th, 2007

NASA, with as strong a history of technical innovation as any Federal agency, has been making the news recently initiating partnerships with Google, Inc. . Originally announced late last year without specific details, one of their first joint projects was detailed this week - applying Google search technology to help scientists process, organize, and analyze the large-scale streams of data coming from the Large Synoptic Survey Telescope (LSST), located in Chile.

Earlier last year the data produced by NASA’s Stardust spacecraft was made available for “crowdsource” style analysis by UC Berkeley’s Stardust@home project. Interestingly, Berkeley partnered with another private sector innovator - Amazon - to use Amazon’s S3 web services based storage to store the millions of images resulting from the Stardust project.

These are excellent examples of Federal agencies tapping the expertise and innovation of private sector companies. It will be interesting to see how other projects might take advantage of the technology and services being developed in projects like this. The Large Hadron Collider, for example, is set to go online later this year. I wonder if we will be able to Google that data!

(CeC regularly posts to the TeraGrid Whiteboard.)

The Nature of eScience

by Ian Foster on December 8th, 2006

A talk by Tiejien Luo at CANS reminded me of Jim Gray’s nice formulation of the evolution of science methodologies:

Thousand years ago: science was empirical, describing natural phenomena

Last few hundred years: theoretical branch, using models, generalizations

Last few decades: a computational branch, simulating complex phenomena

Today: data exploration (eScience)–unify theory, experiment, and simulation. (Data captured by instruments, or generated by simulator; processed by software; information/knowledge stored in computer; scientist analyzes database/files, using data management and statistics.)

Jim’s equating of “eScience” with “data exploration” seems a little too narrow. (John Taylor, who coined the term, had a somewhat broader definition: “e-Science will refer to the large scale science that will increasingly be carried out through distributed global collaborations enabled by the Internet.” ) However, the growing importance of data can hardly be overstated, and Jim’s perspectives are worthy of careful consideration, especially by those who think of “computation and science” as being entirely about simulation.

[From http://ianfoster.typepad.com/blog/2006/12/the_nature_of_e.html] _uacct = “UA-686198-1″; urchinTracker();

Open Source Problem Solving in Science

by Ian Foster on December 3rd, 2006

Linus’ Law according to Eric S. Raymond: “given enough eyeballs, all bugs are shallow.” In other words, if a large enough community of users and developers has access to (and is using) your source code, even subtle problems will be identified and resolved quickly.The use of the Internet to create a “massively parallel human problem-solving system” is a powerful concept, as evidenced by such phenomena as the blogger as a source of news, wikipedia as a source of information, and advertising campaigns that solicit user-generated spots. (For more examples, see Jeff Howe’s writings on crowdsourcing.)

Now Karim Lakhani of Harvard Business School is looking into whether such techniques can be applied to scientific problems. From a recent article (and interview):

In a perfect world, scientists share problems and work together on solutions for the good of society. In the real world, however, that’s usually not the case. The main obstacles: competition for publication and intellectual property protection.

…

What [Karim Lakhani] and his coauthors discovered: “broadcasting” or introducing problems to outsiders yields effective solutions. Indeed, it was outsiders—those with expertise at the periphery of a problem’s field—who were most likely to find answers and do so quickly.

He cites a few intriguing examples, including both more traditional “competitions” (with prizes) that have yielded novel solutions, and also more novel approaches such as MathWorks’ collaborative programming contest. I am also reminded of NASA’s involvement of the general public in analyzing image data from its comet return mission (StarDust@home), and of the nice work at CMU on enlisting people (under the guise of a game) to tag photos. I suspect that there is much much more to be done here.

(From http://ianfoster.typepad.com/blog/2006/12/open_source_pro.html.)