It’s a growing problem in the information age: How do you sort through ever-expanding troves of data to find meaningful patterns and needles in the digital haystack – especially when traditional analysis methods can’t keep pace with the explosion of available data? Bill Perrizo, a North Dakota State University computer science professor and researcher, has spent years developing a solution – one promising enough to attract an entrepreneur who turned the technology into a new business.

Businesses and government agencies are producing and storing more data than ever before, Perrizo said – from hospital records to photos of crop fields to images from defense satellites. But as the information piles up, traditional methods for analyzing data become increasingly ineffective because there’s too much data to manage.

Perrizo’s software essentially turns the problem on its head. Instead of combing through a spreadsheet cell by cell and row by row – a “horizontal” approach – it turns the data “sideways” and treats it as a series of vertical slices, shaving down search time and improving scalability.

Perrizo’s system was good enough to win the 2006 Knowledge Discovery in Data Cup, an annual completion that challenges participants to solve large-scale data-mining challenges. In that competition, Perrizo’s software was able to predict pulmonary embolisms from a huge set of human lung images about twice as effectively as the next competitor.

Perrizo says that kind of application for data mining and prediction has critical real-world consequences: If doctors confuse an embolism with an aneurysm or other condition because they misinterpret an image, the wrong treatment can be deadly.

Last year, representatives from NDSU’s Research Foundation, which is charged with helping university innovations find commercial outlets, got in touch with Mark Silverman, an East Coast technology entrepreneur. Silverman said the software stood out “as the right technology at the right time in the market.”

Silverman and NDSU recently announced a deal to license the technology to Silverman’s new company, Maryland-based Treeminer Inc. Silverman said he plans to bring the product to market later this year. He said government agencies are the biggest potential customer right now. Defense and intelligence, large-scale climate study and modeling, and agricultural modeling are all big potential applications, he said. “Wherever there’s a lot of data, there’s a lot of need,” he said.

Original article by Marino Eccher, INFORUM, reprinted with permission