Python, netCDF, and Model Results Visualization
Intro to Python
Jupyter Notebook, netCDF, and Model Results
We have an ongoing project to develop a collection of Jupyter Notebooks that provide discussion, examples, and best practices for plotting various kinds of model results from netCDF files. There are code examples in the notebooks and also examples of the use of functions from the SalishSeaTools Package.
ERDDAP and xarray
From late-2013 until early-2016 we used the netCDF4-python library to open locally stored files. The notebooks above describe that way of working. In early-2016 we set up an ERDDAP server to provide public access to our model results. The netCDF4-python library can open datasets from ERDDAP URLs just as easily as it can open them from local files. So, here is a reworking of the Exploring netCDF Files.ipynb notebook using ERDDAP:
One reason that you might want to use ERDDAP to access our model results is if you don’t have access to our results files stored on the UBC EOAS Ocean cluster. Our ERDDAP server is public.
Another reason to use ERDDAP is that it provides access to the daily model results as continuous data streams, hiding the fact that they are stored in per-day files. ERDDAP makes it much easier to work with a dataset that spans multiple days because it removes the task of opening each day’s file(s) and splicing the variable values into arrays. You can just ask for a slice of the dataset in time and space and ERDDAP takes care of the slicing and splicing (provided that the resulting dataset is less than 2Gb in size).
Another new development is the xarray package. Quoting from the introduction to its documentation:
xarray … is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures.
Our goal is to provide a pandas-like and pandas-compatible toolkit for analytics on multi-dimensional arrays, rather than the tabular data for which pandas excels. Our approach adopts the Common Data Model for self-describing scientific data in widespread use in the Earth sciences:
xarray.Datasetis an in-memory representation of a netCDF file.
Here is a reworking of the Exploring netCDF Files.ipynb notebook using xarray:
xarray uses the netCDF4-python library so it is capable of accessing netCDF datasets from either local files or from ERDDAP servers.
xarray.Dataset object hides many of the low level details of the
netcdf4.Dataset objects to provide a more Pythonic interface to the dataset that is heavily inspired by pandas. Like panada variables,
xarray variables have a
plot() method that makes quick visualization of datasets very easy.
xarray provides sophisticated handling of the time coordinate of datasets. In combination with ERDDAP that feature makes accessing arbitrary length time slices from the daily Salish Sea Nowcast system results collection very easy.
In summary, you can think of ERDDAP as a higher level abstraction for storage of our model results, and xarray as a higher level abstraction for working with the results as Python objects. The ERDDAP abstraction hides some of the discrete daily runs storage details, and the xarray abstraction hides some of the netCDF4 file structure details.