Earth science data analytics: Definitions, techniques and skills

Full article title	Earth science data analytics: Definitions, techniques and skills
Journal	Data Science Journal
Author(s)	Kempler, Steve; Mathews, Tiffany
Author affiliation(s)	NASA Goddard Space Flight Center, NASA Langley Research Center
Primary contact	Email: gulliver2100 at verizon dot net
Year published	2017
Volume and issue	16
Page(s)	6
DOI	10.5334/dsj-2017-006
ISSN	1683-1470
Distribution license	Creative Commons Attribution 4.0 International
Website	http://datascience.codata.org/articles/10.5334/dsj-2017-006/
Download	http://datascience.codata.org/articles/10.5334/dsj-2017-006/galley/632/download/ (PDF)

Abstract

The continuous evolution of data management systems affords great opportunities for the enhancement of knowledge and advancement of science research. To capitalize on these opportunities, it is essential to understand and develop methods that enable data relationships to be examined and information to be manipulated. Earth science data analytics (ESDA) comprises the techniques and skills needed to holistically extract information and knowledge from all sources of available, often heterogeneous, data sets. This paper reports on the ground-breaking efforts of the Earth Science Information Partners' (ESIP) ESDA Cluster in defining ESDA and identifying ESDA methodologies. As a result of the void of earth science data analytics in the literature, the ESIP ESDA definition and goals serve as an initial framework for a common understanding of techniques and skills that are available, as well as those still needed to support ESDA. Through the acquisition of earth science research use cases and categorization of ESDA result oriented research goals, ESDA techniques/skills have been assembled. The resulting ESDA techniques/skills provide the community with a definition for ESDA that is useful in articulating data management and research needs, as well as a working list of techniques and skills relevant to the different types of ESDA.

Keywords: data science, analytics, techniques, skills, information, knowledge

Introduction

The continuous evolution of data management systems affords great opportunities to the enhancement of knowledge and advancement of earth science research. With the growing need and desire to leverage information from various sources to better understand our environment, it becomes evident — through community experience and foresight — that this can be maximized by accepting new ways to cross-examine this information. As excerpts from Hey, Tansley, and Tolle's The Fourth Paradigm explain^[1]:

We have to do better at producing tools to support the whole research cycle—from data capture and data curation to data analysis and data visualization. (pg. xvii)

Clearly, data-intensive science … must move beyond data warehouses and closed systems, striving instead to allow access to data to those outside the main project teams, allow for greater integration of sources, and provide interfaces to those who are expert scientists but not experts in data administration and computation. (pg. 147)

We are already seeing some attempts to infer knowledge based on the world’s information. (pg. 167)

To capitalize on these opportunities, it is essential to develop a data analytics framework in which we scope the scientific, technical, and methodological components that contribute to advancing science research. Through this framework, we can categorize discussions among individuals of like component interests, instead of attempting to draw specific direction from a set of starting points that may greatly vary.

Data analytics is the process of examining large amounts of data of a variety of types to reveal hidden patterns, unknown correlations, and other useful information, key to facilitating Earth science research opportunities. Thus, the research presented here is motivated by the need to determine and categorize available data analytics techniques and skills and to identify the gaps for where they are still needed.

Today (and well into the foreseeable future) there is a rapid growth in the amount of earth science data and value-added heterogeneous information that many earth science researchers have not yet holistically leveraged. It is important to realize that this rate of data growth is new and challenging. It is new in that information technology is just beginning to provide the tools for advancing the analysis of heterogeneous datasets in a "big" way to provide opportunities to discover non-obvious scientific relationships, previously invisible to the science eye. The challenge is it takes individuals, or teams of individuals, with just the right combination of skills to understand the data and develop the methods to glean knowledge from data and information.

The ability to apply information technology, tools, and services necessary to facilitate the advancement of earth science research is becoming more obvious and necessary at a rate that is accelerating. That is, if data manipulation (subsetting, data transformation, format conversion, etc.) extract information from data, then data analytics techniques and skills glean knowledge from information.^[2]

The objectives of the Earth Sciences Information Partners (ESIP)^[3] Federation Earth Science Data Analytics (ESDA) Cluster is to understand, define, and facilitate the implementation of earth science data analytics. As a result of Cluster efforts, an ESIP-adopted definition of ESDA has been generated, along with 10 ESDA goals, to set the framework for a common understanding for advancing earth science research. In addition, the ESIP ESDA Cluster performed an exhaustive search identifying ESDA types, techniques, and skills used in performing data analytics.

In this paper, we present the ESIP-derived definition of ESDA and differentiate it from other publicized definitions of data analytics. We then describe different types of ESDA and their driving goals. This is followed by an exhaustive survey of current techniques and skills for performing ESDA, made available for the benefit of data scientists exploring new earth science data analytics methodologies, and their potential use.

Literature review

The advancement of information use resulting from evolving technologies, newly developed techniques, and refined skills has become the purview of the data scientist performing data analytics. Data analytics got its start in the business world, which is why most literature and developed tools reflect back on business as the primary application. In the literature, we find that data analytics is comprised of five types: descriptive, diagnostic, discoverative, predictive, and prescriptive. When the ESIP ESDA Cluster attempted to categorize earth science research use cases into these data analytics types, the use cases did not fit: categorizing was ambiguous and/or they fit in more than one type category. Where business data analytics types reflect looking for patterns, and predicting (and prescribing) actions, earth science data analytics also include assessing, validating, calibrating, and applying techniques required to prepare raw datasets for co-use. In addition, characteristics of earth science data introduces data analytics challenges such as dealing with differing formats, differing spatial and temporal data resolutions, inconsistent data acquisition techniques and units for the same measurement, noise, and biasing, to mention a few. This led to the need for a data analytics definition directed specifically at earth science research goals.

In addition, insights by Hey et al., like "[r]esearchers in science must work with colleagues in computer science and informatics to develop field-specific requirement"^[1] (pg. 151) and "[t]he process of combining information from existing scientific knowledge … including the specific methodologies that were followed to produce conclusions, should be automatic and implicitly supported"^[1] (pp. 170-171), are the genesis for melding the expanding manipulation of information and associated technologies with physical science.

A significant aspect of ESDA is information literacy, the ability to "recognize when information is needed and have the ability to locate, evaluate, and use information effectively."^[4] Now that we are well into the information age, ensuring the responsible and appropriate use of data and information has become extremely important and a key skill for ESDA. The Framework for Information Literacy for Higher Education^[5] provides groundwork for learners to understand and fully appreciate the value of information used when performing ESDA.

As seen in the literature, there is no shortage of data analytics definitions, and descriptions of individuals who performs data analytics, the data scientist. The Booz Allen Hamilton (BAH) report The Field Guide to Data Science reminds us that "the term Data Science appeared in the computer science literature throughout the 1960s-1980. It was not until the late 1990s however, that the field, began to emerge from the statistics and data mining communities."^[6] The BAH report further states that "[p]erforming Data Science requires the extraction of timely, actionable information from diverse data sources to drive data products."^[6]

The National Institute of Standards and Technology provides the following definitions^[7]:

• Data science is the extraction of actionable knowledge directly from data through a process of discovery, or hypothesis formulation and hypothesis testing.

• A data scientist is a practitioner who has sufficient knowledge in the overlapping regimes of business needs, domain knowledge, analytical skills, and software and systems engineering to manage the end-to-end data processes in the data life cycle.

• The analytics process is the synthesis of knowledge from information.

The article "8 Skills You Need to Be a Data Scientist"^[8] defines the skills as follows in Table 1.

Table 1. Skills of a data scientist[8] Skills of a data scientist Basic tools Data munging Basic statistics Data visualization and communication Machine learning Software engineering Multivariable calculus and linear algebra Thinking like a data scientist

The website Master's in Data Science^[9] summarizes the technical skills and tools needed of a data scientist to perform data analytics as follows in Table 2.

Table 2. Technical skills and tools of a data scientist[9] Technical skills and tools of a data scientist Math (e.g., linear algebra, calculus, and probability) Statistics (e.g., hypothesis testing and summary statistics) Machine learning tools and techniques (e.g., k-nearest neighbors, random forests, ensemble methods, etc.) Software engineering skills (e.g., distributed computing, algorithms, and data structures) Data mining Data cleaning and munging Data visualization (e.g. ggplot and d3.js) and reporting techniques Unstructured data techniques R and/or SAS languages SQL databases and database querying languages Python (most common), C/C++, Java, Perl Big data platforms like Hadoop, Hive, and Pig Cloud tools like Amazon S3

In addition, the McKinsey Global Institute^[10], National Research Council^[11], and the ongoing Quoara question "What is the difference between Data Analytics, Data Analysis, Data Mining, Data Science, Machine Learning, and Big Data?"^[12] provide further insights into the current thinking that defines data analytics.

O'Neil and Schutt's Doing Data Science found that most data scientist job descriptions ask for expertise in "computer science, statistics, communication, data visualization, and to have extensive domain expertise."^[13]

Much information regarding data analytics techniques and skills has been found in presentations given at informatics forums such as the American Geophysical Union's (AGU) Earth and Space Science Informatics (ESSI) Focus Group, ESIP, and SciDataCon. In addition to forum sessions being rich with experienced individuals who describe techniques and skills that facilitate data analytics, the authors of this paper have also taken the opportunity, at AGU, to visit science poster presentations to better understand research methodologies (analytics) utilized. At these meetings, research that involved co-analysis of multiple datasets were sought out. After scanning hundreds of presentations for methodologies used, 31 atmospheric science-focused (study of gases) and 12 hydrologic science-focused (study of liquid) presentations were targeted in which research techniques were identified (Table 3).

Table 3. Sampling of science research techniques being used Science research technologies In atmospheric research In hydrology research Correlation analysis and bias correlation Spectral analysis Linear regression Regression analysis and bivariant regression Temporal trending and trend analysis Monte Carlo Decision tree Spatial interpolation Darcy equation Machine learning Revised averaging scheme Poisson regression Data mining Forward modeling and inverse modeling Multi-variate time series analysis Data fusion Radiative transfer model BUDYKO formula Computational tools Baysian synthesis inversion Smoothing (Gaussian) Constrained variational analysis Temporal stability Filtering (Destriping) Model simulations Gaussian distribution MESH model Ratios Exponential differentiation Time series analysis

This small study opens our eyes to the data analytics techniques pertaining specifically to data analysis. Yet many of these techniques also find homes in performing data preparation and data reduction analytics.

Defining earth science data analytics

For clarity, data analytics activities fall within the scope and expertise of the data scientist. Data scientists study and develop methods for analyzing, storing, and presenting data. When they practice their skills on specific problems, they are performing data analytics, applying tools and techniques to co-analyze heterogeneous data. Data scientists — as researchers, developers, or data analytics practitioners — require similar skill sets.

Through literature research, analysis of earth science research use cases, and integration of the science research methods, the ESIP Federation – a collaborative organization of over 170 information-centric partners – has defined and adopted the following definition of ESDA^[14]:

The process of examining, preparing, reducing, and analyzing large amounts of spatial (multi-dimensional), temporal, or spectral data, encompassing varieties of data types. ESDA enables people to uncover patterns, correlations, and other information, to better understand our Earth.

This includes:

• Data preparation – Preparing heterogeneous data so that they can be jointly analyzed
• Data reduction – Correcting, ordering, and simplifying data in support of analytic objectives
• Data analysis – Applying techniques/methods to derive results

Data preparation includes the methods and techniques that uncover, discover, and extract data of greatest interest. This can involve filtering, mining, format conversion, smoothing, visualization, etc. Data reduction addresses very large amounts of heterogeneous data that face earth science research. Several methods for the purpose of data reduction can be applied, with the goal of making data transfer, computation, and analysis easier and/or more focused. Analytics to perform data analysis is not as clear cut. Science, by its nature, often utilizes technologies that are not decided upon until data/information is initially looked at and better understood. It is then that researchers experiment with existing and/or novel analytics techniques. In our paradigm, data analysis analytics includes all aspects of science research: hypothesis- and data-discovery-driven methods, as well as goal-driven decisions, outcomes, and impacts. Data analysis analytics aspects can be categorized separately, if desired.

ESDA is categorized by the goals of the analytics performed (Table 4).

Table 4. Earth science data analytics goals ESDA goals To calibrate data To validate data (note it does not have to be via data intercomparison) To assess data quality To perform coarse data preparation (e.g., subsetting data, mining data, transforming data, recovering data) To intercompare datasets (i.e., any data intercomparison; could be used to better define validation/quality) To tease out information from data To glean knowledge from data and information To forecast/predict/model phenomena (i.e., special kind of conclusion) To derive conclusions (i.e., that do not easily fall into another type) To derive new analytics tools

With a definition of ESDA, we now have an initial framework for which to speak the same "language"; discuss ESDA scope, infrastructure, and methodologies with a common focus; and to grow upon.

Earth science data analytics techniques and skills

ESDA techniques are considered to be computational methods. Repeatedly, individuals seeking to perform earth science express their need to utilize mathematics, numerical modeling, statistics, software engineering, and the ability to integrate data from across multiple domains. Also, there is a need for expertise in techniques, such as rule learning, classification, cluster analysis, data fusion, machine learning, neural networks, anomaly detection, modeling, time series analysis, and visualization.

In addition, many other computational methods have been identified as potential techniques to be used in performing ESDA. Table 5 provides a broad scope of the possibilities associated with the three types of ESIP ESDA: data preparation, data reduction, and data analysis.

Table 5. Earth science data analytics techniques Earth science data analytics techniques Data preparation Data reduction Data analysis Bias correction Aggregation Anomaly detection Coordinate transformation Anomaly detection Bayesian techniques Data engineering Cluster analysis Bivariant regression Data mining Data engineering Classification Data munging Data fusion Correlation/regression analysis Database management Factor analysis Factor analysis Exponential differentiation Filtering Fourier analysis Filtering Neural networks Gaussian distribution Format conversion Outlier removal Graphics analysis Imputation Ratios Imputation Normalization/Transformation Revised averaging scheme Linear/Non-linear regression Outlier removal Rule learning Machine learning/Decision tree Ratios Time series Mathematics/Calculus Rule learning Visualization Modeling Sensitivity analysis Monte Carlo method Smoothing Multi-variate time series Spatial interpolation Normalization Time series Pattern recognition Visualization Principal component analysis Revised averaging scheme Rule learning Signal processing Spectral analysis Statistics Temporal trend analysis Time series Visualization

ESDA skills are considered to be the ability to apply techniques. In regards to earth science, this refers to applying ESDA techniques to earth science domains being studied. Thus, ESDA skills include knowledge in particular earth science domains where data analytics can advance the understanding of science.

ESDA skills also refer to the ability to facilitate making data useful. This includes understanding the relevance of the data lifecycle, data structures, metadata, data integration, and data interpretation.

Table 6 summarizes skills that are useful in practicing ESDA.

Table 6. Earth science data analytics skills ESDA skills Ability to integrate data across multiple domains Support domain scientists with data & computational knowledge Communicate across domains Knowledge of data cycle Software engineering Software programming Data Engineering Decision science

In short, ESDA techniques and skills need to be interdisciplinary from the start. One needs to know what domain specific information is available, where to get it, how it is generated, as well as statistical, mathematical, and computational methods to manipulate it.

Conclusions

Although data analytics definitions and types that are oriented at business are well documented, data analytics to facilitate the inter-analysis of large heterogeneous earth science datasets has only begun to be addressed methodically. The significance of the development of a set of definitions, types, goals, techniques, and skills that target ESDA specifically provides earth scientists the opportunity to better articulate the techniques and skills they employ in furthering their science research. In particular, now communications can be performed in terms that engage information technologists who can provide support by implementing responsive tools. With a categorization of known techniques and skills associated with data preparation, data reduction, and data analysis analytics, we know what techniques and skills are presently available, what tools have been implemented that perform these techniques, and what tool gaps need to be filled as science research methodologies evolve.

Next steps include engaging the earth science research community, to better understand their research methodologies and share information technologies that may be useful; engaging scientists to acquire additional use cases to further validate and update our knowledge of known ESDA techniques and skills; continuing to refine our understanding of the skills needed to perform ESDA; promoting the development of ESDA techniques and skills through university curriculum, and; addressing the "moving" gap analysis.

Acknowledgements

The authors would like to thank the ESIP Federation and, in particular, the ESDA Cluster members for their support and insights, as well as Peter Fox and Erin Robinson for their wisdom and encouragement.

Competing interests

The authors have no competing interests to declare.

References

Notes

This presentation is faithful to the original, with only a few minor changes to presentation and grammar. In some cases important information was missing from the references, and that information was added. The original article had citations listed alphabetically; they are listed in the order they appear here due to the way the wiki works.