HDF5 and NETCDF

Global Mapper has definitions for over 80 different types of HDF5 file formats from various scientific projects including: 

  • ASTER GED (Global Emissivity Database) Files (GDEM, emissivity, temperature, etc.)
  • CMIP5 HDF5 data sets with sea surface temperatures
  • NDVI (MODIS Normalized Difference Vegetation Index) HDF5 Files
  • CORTAD (Coral Reef Temperature Anomaly) data
  • Digital Bathymetric Database Variable Resolution (DBDB-V) files
  • Cosmo-SkyMed SCS Radar Data
  • VIIRS data
  • CMIP5 HDF5
  • Landsat Imagery, ESA CCI Soil Moisture, and Gravity Grids in HDF5/NetCDF
  • NSIDC Ice Velocity Extraction (GoLIVE) data
  • GMT 5 grid files
  • NASA Greenland ice thickness data in HDF5/NetCDF
  • National Snowfall Analysis HDF5 data grid
  • Aquaveo mesh definition and simulation results HDF5 format files
  • Copernicus Global Land Service (Valid data sets include LAI, FCOVER, FAPAR, LCCS, NDVI, SSM, GDMP, DMP, LST, LSWT)
  • HEC-RAS plan HDF5 files (*.Pxx.HDF)to show 'Depth' and 'Water Surface' data
  • GEBCO including HDF5 GEBCO 2019 and NETCDF GEBCO 2020

Additional HDF5 file formats can be custom added using the HDF5 Definition tool.

Import HDF5 File

If the definition of the HDF5 file matches the tables of one of the built in a definitions or a custom added one, the HDF5 file will load directly.

When multiple z values within the file have been defined, there will be a prompt to specify which value to load from the file:

Data with a 3rd Dimension

If the selected layer has multiple time slices in a 3rd dimension, there may also be a selection of the time slice, layer, or resolution to load:

Check individual time slices, or highlight multiple with the SHIFT key and then check one to check the highlighted times.

Undefined HDF5 Files

When the file does not match an existing HDF5 definition, the following dialog will appear:

Press Yes to bring up the HDF Definition Editor described below to create a new HDF5 definition.

Edit HDF5 Definition

The HDF5 Creator tool is used to create a definition for an HDF5 file so that Global Mapper will know how to open it. HDF5 is a standard for storing large amounts of data into a compressed set of tables. The structure of an HDF5 file is determined by the organization that generates the file. An organization may design the representation and storage of geolocated scientific data in many different ways, and within a single HDF5 file, there may be multiple data tables to choose from. The HDF5 Creator tool lets the user specify which tables to use. For more information see Requirements.

Open the HDF5 Creator from the Tools menu option HDF5 Definition Manager.... Or, when attempting to open an unrecognized HDF5 file, answer Yes to the prompt:
Unsupported HDF5 data set. Would you like to try to create a definition for this file?

To create a new HDF5 definition, go to Tools > HDF5 Definition Manager.... Press the New... button to create a new HDF5 definition. In the open dialog, point to a reference HDF5 file to use as a template.

The HDF5 Definition Editor dialog will show a list of tables contained in the Template file on the left side of the dialog.

  1. In the HDF5 table list, select the table representing the longitude (X coordinate). Either click on the table name and drag to the Longitude box on the right or click on the > button. Do the same for the latitude (Y coordinate), copying the table to the Latitude box on the right.

  2. In the table list, select the table representing the data (Z coordinate). Either click on the table name and drag to the Data box on the right or click on the > button. If the data table has a 3rd dimension (for example data sets at different time values), enter the value for that dimension ranging from 1 to the value of the third dimension shown next the Data. The data table shown in the example only has a single 3rd dimension and so it is not an option.
  3. Optionally specify a Scale Factor, Offset or custom Units description.
  4. Enter a Definition Description in the top right box.
  5. If there are no more Data Z value to reference in the file, press Ok and the new defintion will be saved. If you would like to create additional definitions from this file, press the Save button. Then update the Data (Z) value, specify a new Definition Description, update Scale Factor, Offset or Units if appropriate, and press Save again to save a second definition.

Press the New...button to create a new HDF5 definition. The first step is to load a relevant file to use as a template to create the definition.

An existing definition can be edited by selecting the Edit... button and selecting from a list of user- created HDF5 definitions. Editing an existing definition without changing the description overwrites the previous definition, whereas a new description creates a new definition. This would be useful, for example, if the geolocation tables remained the same and only the data table was changed.

Use the Delete button to remove highlighted existing custom definitions.

Requirements

There are many ways to relate and store data using the HDF5 format, however, for Global Mapper, the file must have two grid-based geolocation (latitude and longitude) tables and one or more data tables representing a value at each location.

The dimensions of the geolocation tables must match the dimensions of the data table. Latitude and longitude tables can be either 1 or 2 dimensions. In the example above the data table (Soil_Moisture_Retrieval_Data/soil_moisture) is 1624 rows x 3856 columns, matching the size of the geographic tables. Table dimensions do not have to be square.

Latitude and longitude tables can be either 1 or 2 dimensions. One dimensional geographic tables imply that each row of data values corresponds to a single longitude, and each column to a single latitude. Two dimensional geographic tables imply that each data value corresponds to a single longitude and latitude pair. Data tables must be at least 2 dimensions.

Note: Global Mapper does not currently support EASE 2.0 type gridded data. EASE-2 type data will fail to create a definition, showing an error message: Data table must contain at least 2 dimensions

Template and Tables

The Template field will list the name of the loaded template file. The list on the left shows all tables in the loaded template file.

Use the Filter text field to limit the tables and groups shown to those containing the text entered. The filter is case sensitive. This is useful when an HDF5 file contains many tables and groups, to limit the list to just those of interest.

The Show Groups checkbox will show or hide groups in the file. The structure of an HDF file is hierarchical with a structure like a computer filing system. Groups are similar to computer file folders. In an HDF5 file there is sometimes metadata attached to a group that may be of interest. While viewing groups and their metadata might be useful, it can also clutter up the list of tables, so the listing of groups can be toggled on or off. When shown, group names are displayed inside angle brackets.

The metadata and table Dimensions for the selected table will display on the bottom left. Metadata are additional information about the selected table. Dimensions are shown as number of rows by number of columns.

Definition

Definition Description is description of the HDF5 definition. It will show as an option when loading a file matching multiple data definitions, so typically help identify the Data (z) value being loaded.

The user selected table names for Latitude, Longitude, and Datashould be populated by dragging corresponding table names, or highlighting them in the list on the right and using the arrow buttons.

The Data Layer is an additional option when the z Data contains a 3rd Dimension (for data tables containing more than one third dimension). This is most often used to store data over various time slices. Select which layer to load, or uncheck the load layer option to provide a selection prompt each time the data is loaded. See Data with a 3rd Dimension for an example.

The Scale Factor and Offsetvalues will be applied to the data. These values are usually be found by looking at the data tables metadata. When imported into Global Mapper, data values are multiplied by the Scale Factor and the Offset is added to it.

Buttons to Save the current and Open an existing definition.

Attempting to open the HDF5 file Global Mapper will check if the file contains all three tables and if so open them and read and display the data.

Format Updates

Below is a summary of recent changes to HDF5 format support:

Version Description
22.1 Added support for NETCDF gebco 2020 files, and HDF5 library was updated to the Format Updates [Item #GM-11020]
21 Added support for importing NetCDF (HDF5) files from GEBCO[Item #GM-9734]
21 Imports HECRAS plan HDF5 files (*.Pxx.HDF)to show 'Depth' and 'Water Surface' data [Item #26065]
21 Added a tool, HDF5 Definition Creator, to let the user define the data to extract from an HDF5 type file not in GM internal database. Works with HDF5 files containing defined latitude, longitude, and data tables with the same dimensions.[Item #19182/26210]
21 Added support for HDF5 files from Copernicus Global Land ServiceValid data sets include LAI, FCOVER, FAPAR, LCCS, NDVI, SSM, GDMP, DMP, LST, LSWT(see https://land.copernicus.eu/global/ for details of these types of data sets)[Item #GM-6693]
20.1 Added support to read Aquaveo mesh definition and simulation results HDF5 format files. User will be prompted to pick the data set and the time slice.[Item #25785]
20.1 Added support for loading National Snowfall Analysis HDF5 data grid.[Item #25132]
20 Added support for NASA Greenland ice thickness data in HDF5/NetCDF format.[Item #23171]
20 Added support for loading GMT 5 grid files in HDF5 format.[Item #23678]
19 Added support for importing NSIDC Ice Velocity Extraction (GoLIVE) data from an HDF5 file as a velocity grid and optional quiver plot. [Item #20296]
17.1 Added support for loading ASTER GED data (GDEM, emissivity, temperature, etc.) from HDF5 files. [Item #16742]
17.1 No longer show prompt to confirm load of file when a layer based on the same filename is already loaded for file types that provide a list of layers to the user to select from, like HDF5 or NetCDF format files. [Item #16742]
17.1 Added support for loading ASTER GED data (GDEM, emissivity, temperature, etc.) from HDF5 files. [Item #16742]
17 Added support for loading NetCDF4 SPEI (Standardized Precipitation-Evapotranspiration Index) files. [Item #16272]
16.2 Added support for any type of NetCDF file with recognized 2D grids in any variables, including grids with more than 3 data dimensions, 2 of which are the data grid. You can also load multiple grids at once from the NetCDF files rather than needing a separate load for each grid to extract. Data with recognized U/V vector components at a series of XY locations are also automatically loaded as a quiver plot, useful for things like ocean currents with velocity or directional wind speeds.
16.1 Support Landsat Imagery, ESA CCI Soil Moisture, and Gravity Grids in HDF5/NetCDF format.
15.2 Added support for additional HDF5 elevation grid data formats.
15.1 Allow loading NetCDF files directly from .grd.gz files.
15 Added support for loading ANUGA Rainfall and Greenland Terrain NetCDF format files.
15 Added support for CMIP5 HDF5 data sets with sea surface temperatures.
14.2 Added support for loading NetCDF files with time-dependent variables
14 Added support for loading Cosmo-SkyMed SCS Radar Data in HDF5 Format.
14 Added support for additional bands of VIIRS data in HDF5 Format.
13.2 Added support for additional bands of VIIRS data in HDF5 Format.
13.2 Added support for VIIRS data in HDF5 format.
13.1 Added support for Digital Bathymetric Database Variable Resolution (DBDB-V) files in HDF5 format.
13.1 Added support for loading CORTAD (Coral Reef Temperature Anomaly) data in HDF5 format.