Helping to Tame Big Data

NJIT graduate student Shreya Mittapalli presenting the results of her work involving the challenges of "big data" at the National Center for Atmospheric Research.

The world is awash in data increasingly important for progress in science and technology. Often referred to collectively as “big data,” this massive fund of information can pose significant challenges when it comes to using it efficiently and not overwhelming the computing resources available to researchers.

But there are ways to process the huge volumes of data now gathered from numerous sources that can facilitate widely useful analysis and application. This was the focus of NJIT graduate student Shreya Mittapalli’s 2015 summer internship at the National Center for Atmospheric Research (NCAR) in Boulder, Colorado.

A master’s candidate in the Department of Mathematical Sciences, Mittapalli pursued investigation of how a technique known as lossy compression could be innovatively applied to the processing problems presented by big data sets in fields such as atmospheric research. Lossy compression is also used in multimedia consumer applications that include images, video and music. As its name implies, this technique does not restore data to 100 percent of its original size upon decompression.

Sponsored primarily by the National Science Foundation, NCAR is dedicated to service, research and education in the atmospheric and related sciences. Its mission is to understand the behavior of the atmosphere and related physical, biological and social systems in support of helping the public and private sectors use the knowledge gained to improve the quality of life on our planet.

During her internship at NCAR, the data that Mittapalli worked with included complex information related to the major hurricanes Katrina and Sandy. Investigating optimal parameters for compressing numerical simulation data, she did help to confirm that there are promising ways to use lossy compression to minimize distortion for a desired output-file size while constraining computation cost for compressing and decompressing the data.

Mittapalli, who anticipates completing her master’s in applied mathematics by the end of the fall 2015 semester, says that the reasons she chose NJIT for graduate study include the university’s reputation for teaching excellence, and outstanding on-campus employment opportunities and career development services.

“I hope to become a data scientist or math lecturer,” she says of her post-graduation plans. It’s a future that may very well continue to engage her with the big challenges of big data.

By Dean Maskevich

Photo copyright (c) 2015 University Corporation for Atmospheric Research