Note
Go to the end to download the full example code.
Basic Functionality
The GS standard organizes datasets and metadata within a Data Tree. In GSPy, this is implemented through accessors into Xarray DataTrees, Datasets, and DataArrays. This example demonstrates basic xarray functionality for exploring the data and metadata for each class type.
This example uses the TEMPEST AEM survey as a basis for demonstration.
Source Reference: Minsley, B.J., James, S.R., Bedrosian, P.A., Pace, M.D., Hoogenboom, B.E., and Burton, B.L., 2021, Airborne electromagnetic, magnetic, and radiometric survey of the Mississippi Alluvial Plain, November 2019 - March 2020: U.S. Geological Survey data release, https://doi.org/10.5066/P9E44CTQ.
import matplotlib.pyplot as plt
from os.path import join, isfile
import gspy
from gspy import Survey
from pprint import pprint
First open the netcdf GS standard file, generate it if doesn’t already exist
input_file = "..//data_files//tempest_aseg//Tempest.nc"
if not isfile(input_file):
import subprocess
import sys
subprocess.run([sys.executable, "..//Creating_GS_Files//plot_aseg_tempest.py"])
survey = gspy.open_datatree(input_file)['survey']
Accessing the groups within the tree
Survey
the survey object here is a DataTree, printing it will show the entire contents
of the DataTree
print('Survey:\n')
print(survey)
Survey:
<xarray.DataTree 'survey'>
Group: /survey
│ Dimensions: ()
│ Coordinates:
│ spatial_ref float64 8B ...
│ Data variables:
│ survey_information float64 8B ...
│ survey_units float64 8B ...
│ flightline_information float64 8B ...
│ survey_equipment float64 8B ...
│ Attributes:
│ type: survey
│ title: Example Tempest Airborne Electromagnetic (AEM) Dataset
│ institution: USGS Geology, Geophysics, & Geochemistry Science Center
│ source: Contractor provided ASEG-GDF2 formatted data
│ history: This example dataset includes the raw AEM data and gridded...
│ references: Minsley, B.J., James, S.R., Bedrosian, P.A., Pace, M.D., H...
│ comment: This dataset is incomplete and has been downsampled for th...
│ content: Tempest AEM Survey from the Mississippi Alluvial Plain /su...
│ gspy_version: 2.2.4
│ conventions: GS-2.0, CF-1.13
├── Group: /survey/data
│ │ Dimensions: ()
│ │ Data variables:
│ │ spatial_ref float64 8B ...
│ │ Attributes:
│ │ content: raw data
│ │ type: container
│ └── Group: /survey/data/raw_data
│ │ Dimensions: (index: 2001, gate_times: 15)
│ │ Coordinates:
│ │ * index (index) float64 16kB 0.0 1.0 2.0 ... 1.999e+03 2e+03
│ │ * gate_times (gate_times) float64 120B 1.085e-05 3.255e-05 ... 0.01338
│ │ spatial_ref float64 8B ...
│ │ x (index) float64 16kB ...
│ │ y (index) float64 16kB ...
│ │ z (index) float64 16kB ...
│ │ Data variables: (12/58)
│ │ line (index) float64 16kB ...
│ │ flight (index) float64 16kB ...
│ │ fiducial (index) float64 16kB ...
│ │ proj_cgg (index) float64 16kB ...
│ │ proj_client (index) float64 16kB ...
│ │ date (index) float64 16kB ...
│ │ ... ...
│ │ z_primaryfield (index) float64 16kB ...
│ │ z_vlf1 (index) float64 16kB ...
│ │ z_vlf2 (index) float64 16kB ...
│ │ z_vlf3 (index) float64 16kB ...
│ │ z_vlf4 (index) float64 16kB ...
│ │ z_geofact (index) float64 16kB ...
│ │ Attributes:
│ │ content: raw AEM data
│ │ comment: This dataset includes minimally processed (raw) AEM data
│ │ type: data
│ │ mode: airborne
│ │ method: ['magnetic', 'radiometric', 'electromagnetic, time domain']
│ │ instrument: 30Hz Tempest
│ │ structure: tabular
│ └── Group: /survey/data/raw_data/tempest_system
│ Dimensions: (gate_times: 15, nv: 2, n_transmitter: 1,
│ waveform_time: 7, n_receiver: 2,
│ n_couplet: 2, dim_0: 1)
│ Coordinates:
│ * nv (nv) float64 16B 0.0 1.0
│ * n_transmitter (n_transmitter) float64 8B 0.0
│ * waveform_time (waveform_time) float64 56B -0.01667 ... 0....
│ * n_receiver (n_receiver) float64 16B 0.0 1.0
│ * n_couplet (n_couplet) float64 16B 0.0 1.0
│ Dimensions without coordinates: dim_0
│ Data variables: (12/26)
│ gate_times_bnds (gate_times, nv) float64 240B ...
│ transmitter_label (n_transmitter) <U1 4B ...
│ transmitter_area (n_transmitter) int64 8B ...
│ transmitter_waveform_type (n_transmitter) <U6 24B ...
│ transmitter_waveform_current (waveform_time) float64 56B ...
│ transmitter_scale_factor (n_transmitter) float64 8B ...
│ ... ...
│ couplet_txrx_dz (n_couplet) int64 16B ...
│ data_normalized (dim_0) bool 1B ...
│ output_data_type <U1 4B ...
│ reference_frame <U24 96B ...
│ output_sample_frequency (dim_0) int64 8B ...
│ digitization_frequency (dim_0) int64 8B ...
│ Attributes:
│ type: system
│ mode: airborne
│ method: electromagnetic, time domain
│ instrument: 30Hz Tempest
│ name: tempest_system
├── Group: /survey/models
│ │ Dimensions: ()
│ │ Data variables:
│ │ spatial_ref float64 8B ...
│ │ Attributes:
│ │ content: inverted 1-D electrical resistivity models
│ │ type: container
│ └── Group: /survey/models/inverted_models
│ │ Dimensions: (layer_depth: 30, nv: 2, gate_times: 15,
│ │ index: 2001)
│ │ Coordinates:
│ │ * layer_depth (layer_depth) float64 240B 1.5 4.65 ... 424.2 467.5
│ │ * nv (nv) float64 16B 0.0 1.0
│ │ * gate_times (gate_times) float64 120B 1.085e-05 ... 0.01338
│ │ * index (index) float64 16kB 0.0 1.0 ... 1.999e+03 2e+03
│ │ spatial_ref float64 8B ...
│ │ x (index) float64 16kB ...
│ │ y (index) float64 16kB ...
│ │ z (index) float64 16kB ...
│ │ Data variables: (12/46)
│ │ layer_depth_bnds (layer_depth, nv) float64 480B ...
│ │ gate_times_bnds (gate_times, nv) float64 240B ...
│ │ uniqueid (index) float64 16kB ...
│ │ survey (index) float64 16kB ...
│ │ date (index) float64 16kB ...
│ │ flight (index) float64 16kB ...
│ │ ... ...
│ │ phic (index) float64 16kB ...
│ │ phit (index) float64 16kB ...
│ │ phig (index) float64 16kB ...
│ │ phis (index) float64 16kB ...
│ │ lambda (index) float64 16kB ...
│ │ iterations (index) float64 16kB ...
│ │ Attributes:
│ │ content: inverted resistivity models
│ │ comment: This dataset includes inverted resistivity models derived fr...
│ │ type: model
│ │ method: electromagnetic, time domain
│ │ instrument: 30Hz Tempest
│ │ mode: airborne
│ │ property: electrical conductivity
│ │ structure: tabular
│ ├── Group: /survey/models/inverted_models/tempest_system
│ │ Dimensions: (gate_times: 15, nv: 2, n_transmitter: 1,
│ │ waveform_time: 7, n_receiver: 2,
│ │ n_couplet: 2, dim_0: 1)
│ │ Coordinates:
│ │ * n_transmitter (n_transmitter) float64 8B 0.0
│ │ * waveform_time (waveform_time) float64 56B -0.01667 ... 0....
│ │ * n_receiver (n_receiver) float64 16B 0.0 1.0
│ │ * n_couplet (n_couplet) float64 16B 0.0 1.0
│ │ Dimensions without coordinates: dim_0
│ │ Data variables: (12/26)
│ │ gate_times_bnds (gate_times, nv) float64 240B ...
│ │ transmitter_label (n_transmitter) <U1 4B ...
│ │ transmitter_area (n_transmitter) int64 8B ...
│ │ transmitter_waveform_type (n_transmitter) <U6 24B ...
│ │ transmitter_waveform_current (waveform_time) float64 56B ...
│ │ transmitter_scale_factor (n_transmitter) float64 8B ...
│ │ ... ...
│ │ couplet_txrx_dz (n_couplet) int64 16B ...
│ │ data_normalized (dim_0) bool 1B ...
│ │ output_data_type <U1 4B ...
│ │ reference_frame <U24 96B ...
│ │ output_sample_frequency (dim_0) int64 8B ...
│ │ digitization_frequency (dim_0) int64 8B ...
│ │ Attributes:
│ │ type: system
│ │ mode: airborne
│ │ method: electromagnetic, time domain
│ │ instrument: 30Hz Tempest
│ │ name: tempest_system
│ └── Group: /survey/models/inverted_models/inversion_parameters
│ Dimensions: ()
│ Data variables:
│ software <U61 244B ...
│ software_reference <U298 1kB ...
│ description <U1402 6kB ...
│ doi_calculation <U592 2kB ...
│ phid_cut <U256 1kB ...
│ Attributes:
│ type: parameters
│ method: electromagnetic, time domain
│ instrument: 30Hz Tempest
│ mode: airborne
│ property: electrical conductivity
│ name: inversion_parameters
└── Group: /survey/derived_maps
│ Dimensions: ()
│ Data variables:
│ spatial_ref float64 8B ...
│ Attributes:
│ content: derived maps
│ type: container
└── Group: /survey/derived_maps/maps
│ Dimensions: (x: 599, nv: 2, y: 1212)
│ Coordinates:
│ * x (x) float64 5kB 2.928e+05 2.934e+05 ... 6.51e+05 6.516e+05
│ * nv (nv) float64 16B 0.0 1.0
│ * y (y) float64 10kB 1.607e+06 1.606e+06 ... 8.808e+05 8.802e+05
│ spatial_ref float64 8B ...
│ Data variables:
│ x_bnds (x, nv) float64 10kB ...
│ y_bnds (y, nv) float64 19kB ...
│ magnetic_tmi (y, x) float64 6MB ...
│ Attributes:
│ comment: contractor-derived product
│ content: gridded map of total magnetic intensity
│ type: data
│ mode: airborne
│ method: magnetic
│ instrument: Scintrex CS-3 cesium vapor magnetometer
│ structure: raster
└── Group: /survey/derived_maps/maps/magnetic_system
Dimensions: (base_mag_locations: 6, nv: 2,
n_transmitter: 1, n_receiver: 1,
n_couplet: 1, n_base_magnetometer: 1)
Coordinates:
* base_mag_locations (base_mag_locations) float64 48B 1.0 ...
* n_transmitter (n_transmitter) float64 8B 0.0
* n_receiver (n_receiver) float64 8B 0.0
* n_couplet (n_couplet) float64 8B 0.0
* n_base_magnetometer (n_base_magnetometer) float64 8B 0.0
Data variables: (12/35)
base_mag_locations_bnds (base_mag_locations, nv) float64 96B ...
transmitter_label (n_transmitter) <U7 28B ...
transmitter_description (n_transmitter) <U107 428B ...
receiver_label (n_receiver) <U19 76B ...
receiver_sensor_type (n_receiver) <U12 48B ...
receiver_sensor_model (n_receiver) <U4 16B ...
... ...
diurnal_correction <U142 568B ...
igrf_model_date <U10 40B ...
igrf_model_height <U7 28B ...
igrf_removed_model_epoch <U6 24B ...
tieline_levelling <U137 548B ...
deliverables <U63 252B ...
Attributes:
type: system
mode: airborne
method: magnetic
instrument: Scintrex CS-3 c cesium-vapor magnetometer
name: magnetic_system
The DataSet object at the location /survey can be isolated through two options:
Option 1) directly form the DataTree object survey
print('\n\nOption 1:\n')
print(survey.dataset)
Option 1:
<xarray.DatasetView> Size: 40B
Dimensions: ()
Coordinates:
spatial_ref float64 8B ...
Data variables:
survey_information float64 8B ...
survey_units float64 8B ...
flightline_information float64 8B ...
survey_equipment float64 8B ...
Attributes:
type: survey
title: Example Tempest Airborne Electromagnetic (AEM) Dataset
institution: USGS Geology, Geophysics, & Geochemistry Science Center
source: Contractor provided ASEG-GDF2 formatted data
history: This example dataset includes the raw AEM data and gridded...
references: Minsley, B.J., James, S.R., Bedrosian, P.A., Pace, M.D., H...
comment: This dataset is incomplete and has been downsampled for th...
content: Tempest AEM Survey from the Mississippi Alluvial Plain /su...
gspy_version: 2.2.4
conventions: GS-2.0, CF-1.13
or Option 2) use the path to the group and then retrieve the DataSet
print('\n\nOption 2:\n')
print(survey['/survey'].dataset)
Option 2:
<xarray.DatasetView> Size: 40B
Dimensions: ()
Coordinates:
spatial_ref float64 8B ...
Data variables:
survey_information float64 8B ...
survey_units float64 8B ...
flightline_information float64 8B ...
survey_equipment float64 8B ...
Attributes:
type: survey
title: Example Tempest Airborne Electromagnetic (AEM) Dataset
institution: USGS Geology, Geophysics, & Geochemistry Science Center
source: Contractor provided ASEG-GDF2 formatted data
history: This example dataset includes the raw AEM data and gridded...
references: Minsley, B.J., James, S.R., Bedrosian, P.A., Pace, M.D., H...
comment: This dataset is incomplete and has been downsampled for th...
content: Tempest AEM Survey from the Mississippi Alluvial Plain /su...
gspy_version: 2.2.4
conventions: GS-2.0, CF-1.13
To look just at the attributes of the Survey, once again it can be accessed either directly or by using the group path
Survey Attributes Option 1:
pprint(survey.attrs)
{'comment': 'This dataset is incomplete and has been downsampled for the '
'purposes of this example.',
'content': 'Tempest AEM Survey from the Mississippi Alluvial Plain /survey; '
'raw data /survey/data; inverted 1-D electrical resistivity models '
'/survey/models; derived maps /survey/derived_maps; raw AEM data '
'/survey/data/raw_data; inverted resistivity models '
'/survey/models/inverted_models; gridded map of total magnetic '
'intensity /survey/derived_maps/maps; '
'/survey/data/raw_data/tempest_system; '
'/survey/models/inverted_models/tempest_system; '
'/survey/models/inverted_models/inversion_parameters; '
'/survey/derived_maps/maps/magnetic_system; ',
'conventions': 'GS-2.0, CF-1.13',
'gspy_version': '2.2.4',
'history': 'This example dataset includes the raw AEM data and gridded '
'magnetic data as provided by the contractor, CGG Canada Services, '
'Ltd, as well as 1-D resistivity models inverted by the USGS using '
'the GALEISBSTDEM time-domain deterministic inversion software '
'(Brodie 2015, Geoscience Australia, Release-20160606).',
'institution': 'USGS Geology, Geophysics, & Geochemistry Science Center',
'references': 'Minsley, B.J., James, S.R., Bedrosian, P.A., Pace, M.D., '
'Hoogenboom, B.E., and Burton, B.L., 2021, Airborne '
'electromagnetic, magnetic, and radiometric survey of the '
'Mississippi Alluvial Plain, November 2019 - March 2020: U.S. '
'Geological Survey data release, '
'https://doi.org/10.5066/P9E44CTQ.',
'source': 'Contractor provided ASEG-GDF2 formatted data',
'title': 'Example Tempest Airborne Electromagnetic (AEM) Dataset',
'type': 'survey'}
Survey Attributes Option 2:
pprint(survey['/survey'].attrs)
{'comment': 'This dataset is incomplete and has been downsampled for the '
'purposes of this example.',
'content': 'Tempest AEM Survey from the Mississippi Alluvial Plain /survey; '
'raw data /survey/data; inverted 1-D electrical resistivity models '
'/survey/models; derived maps /survey/derived_maps; raw AEM data '
'/survey/data/raw_data; inverted resistivity models '
'/survey/models/inverted_models; gridded map of total magnetic '
'intensity /survey/derived_maps/maps; '
'/survey/data/raw_data/tempest_system; '
'/survey/models/inverted_models/tempest_system; '
'/survey/models/inverted_models/inversion_parameters; '
'/survey/derived_maps/maps/magnetic_system; ',
'conventions': 'GS-2.0, CF-1.13',
'gspy_version': '2.2.4',
'history': 'This example dataset includes the raw AEM data and gridded '
'magnetic data as provided by the contractor, CGG Canada Services, '
'Ltd, as well as 1-D resistivity models inverted by the USGS using '
'the GALEISBSTDEM time-domain deterministic inversion software '
'(Brodie 2015, Geoscience Australia, Release-20160606).',
'institution': 'USGS Geology, Geophysics, & Geochemistry Science Center',
'references': 'Minsley, B.J., James, S.R., Bedrosian, P.A., Pace, M.D., '
'Hoogenboom, B.E., and Burton, B.L., 2021, Airborne '
'electromagnetic, magnetic, and radiometric survey of the '
'Mississippi Alluvial Plain, November 2019 - March 2020: U.S. '
'Geological Survey data release, '
'https://doi.org/10.5066/P9E44CTQ.',
'source': 'Contractor provided ASEG-GDF2 formatted data',
'title': 'Example Tempest Airborne Electromagnetic (AEM) Dataset',
'type': 'survey'}
Similarly, to expand a specific variable of the survey, we can access it directly from the survey object, i.e. no path needed first
print('\n\nSurvey Information:\n')
print(survey['survey_information'])
Survey Information:
<xarray.DataArray 'survey_information' ()> Size: 8B
[1 values with dtype=float64]
Coordinates:
spatial_ref float64 8B ...
Attributes:
contractor_project_number: 603756FWA
contractor: CGG Canada Services Ltd.
client: U.S. Geological Survey
survey_type: electromagneticmagneticradiometric
survey_area_name: Mississippi Alluvial Plain (MAP)
state: MO,AR,TN,MS,LA,IL,KY
country: USA
acquisition_start: 20191120
acquisition_end: 20200307
dataset_created: 20200420
DataTree Items
The method “items” on a DataTree returns the variables within the top-level Dataset (in this case /survey) and all children. Note this does not return grandchildren! So in this case we only see the container branches immediately beneath the survey
print('\n\nItems of the Survey DataTree\n')
for name, item in survey.items():
# do something with the item if desired
print(name)
Items of the Survey DataTree
survey_information
survey_units
flightline_information
survey_equipment
data
models
derived_maps
Datasets are attached to the DataTree and can be isolated by their path. If a Dataset has children beneath it then technically it is still a DataTree object and printing it will show the rest of that branch of the tree.
# Show the data branch
print('\n\nData Branch:\n')
print(survey['data'])
Data Branch:
<xarray.DataTree 'data'>
Group: /survey/data
│ Dimensions: ()
│ Data variables:
│ spatial_ref float64 8B ...
│ Attributes:
│ content: raw data
│ type: container
└── Group: /survey/data/raw_data
│ Dimensions: (index: 2001, gate_times: 15)
│ Coordinates:
│ * index (index) float64 16kB 0.0 1.0 2.0 ... 1.999e+03 2e+03
│ * gate_times (gate_times) float64 120B 1.085e-05 3.255e-05 ... 0.01338
│ spatial_ref float64 8B ...
│ x (index) float64 16kB ...
│ y (index) float64 16kB ...
│ z (index) float64 16kB ...
│ Data variables: (12/58)
│ line (index) float64 16kB ...
│ flight (index) float64 16kB ...
│ fiducial (index) float64 16kB ...
│ proj_cgg (index) float64 16kB ...
│ proj_client (index) float64 16kB ...
│ date (index) float64 16kB ...
│ ... ...
│ z_primaryfield (index) float64 16kB ...
│ z_vlf1 (index) float64 16kB ...
│ z_vlf2 (index) float64 16kB ...
│ z_vlf3 (index) float64 16kB ...
│ z_vlf4 (index) float64 16kB ...
│ z_geofact (index) float64 16kB ...
│ Attributes:
│ content: raw AEM data
│ comment: This dataset includes minimally processed (raw) AEM data
│ type: data
│ mode: airborne
│ method: ['magnetic', 'radiometric', 'electromagnetic, time domain']
│ instrument: 30Hz Tempest
│ structure: tabular
└── Group: /survey/data/raw_data/tempest_system
Dimensions: (gate_times: 15, nv: 2, n_transmitter: 1,
waveform_time: 7, n_receiver: 2,
n_couplet: 2, dim_0: 1)
Coordinates:
* nv (nv) float64 16B 0.0 1.0
* n_transmitter (n_transmitter) float64 8B 0.0
* waveform_time (waveform_time) float64 56B -0.01667 ... 0....
* n_receiver (n_receiver) float64 16B 0.0 1.0
* n_couplet (n_couplet) float64 16B 0.0 1.0
Dimensions without coordinates: dim_0
Data variables: (12/26)
gate_times_bnds (gate_times, nv) float64 240B ...
transmitter_label (n_transmitter) <U1 4B ...
transmitter_area (n_transmitter) int64 8B ...
transmitter_waveform_type (n_transmitter) <U6 24B ...
transmitter_waveform_current (waveform_time) float64 56B ...
transmitter_scale_factor (n_transmitter) float64 8B ...
... ...
couplet_txrx_dz (n_couplet) int64 16B ...
data_normalized (dim_0) bool 1B ...
output_data_type <U1 4B ...
reference_frame <U24 96B ...
output_sample_frequency (dim_0) int64 8B ...
digitization_frequency (dim_0) int64 8B ...
Attributes:
type: system
mode: airborne
method: electromagnetic, time domain
instrument: 30Hz Tempest
name: tempest_system
Zoom in to the lowest level of the tree, notice it is still technically a DataTree object but it has no children.
print('\n\nSystem Leaflet at the bottom of the Tree:\n')
print(survey['data/raw_data/tempest_system'])
System Leaflet at the bottom of the Tree:
<xarray.DataTree 'tempest_system'>
Group: /survey/data/raw_data/tempest_system
Dimensions: (index: 2001, gate_times: 15, nv: 2,
n_transmitter: 1, waveform_time: 7,
n_receiver: 2, n_couplet: 2, dim_0: 1)
Coordinates:
* nv (nv) float64 16B 0.0 1.0
* n_transmitter (n_transmitter) float64 8B 0.0
* waveform_time (waveform_time) float64 56B -0.01667 ... 0....
* n_receiver (n_receiver) float64 16B 0.0 1.0
* n_couplet (n_couplet) float64 16B 0.0 1.0
Inherited coordinates:
* index (index) float64 16kB 0.0 1.0 ... 2e+03
* gate_times (gate_times) float64 120B 1.085e-05 ... 0.0...
Dimensions without coordinates: dim_0
Data variables: (12/26)
gate_times_bnds (gate_times, nv) float64 240B ...
transmitter_label (n_transmitter) <U1 4B ...
transmitter_area (n_transmitter) int64 8B ...
transmitter_waveform_type (n_transmitter) <U6 24B ...
transmitter_waveform_current (waveform_time) float64 56B ...
transmitter_scale_factor (n_transmitter) float64 8B ...
... ...
couplet_txrx_dz (n_couplet) int64 16B ...
data_normalized (dim_0) bool 1B ...
output_data_type <U1 4B ...
reference_frame <U24 96B ...
output_sample_frequency (dim_0) int64 8B ...
digitization_frequency (dim_0) int64 8B ...
Attributes:
type: system
mode: airborne
method: electromagnetic, time domain
instrument: 30Hz Tempest
name: tempest_system
Use the .to_dataset method to isolate the Dataset (convert from DataTree to Dataset), can optionally create a new variable with just that Dataset:
tempest_system = survey['data/raw_data/tempest_system'].to_dataset()
print('\n\nSystem Leaflet, as a Dataset:\n')
print(tempest_system)
System Leaflet, as a Dataset:
<xarray.Dataset> Size: 17kB
Dimensions: (gate_times: 15, nv: 2, n_transmitter: 1,
waveform_time: 7, n_receiver: 2,
n_couplet: 2, dim_0: 1, index: 2001)
Coordinates:
* gate_times (gate_times) float64 120B 1.085e-05 ... 0.0...
* nv (nv) float64 16B 0.0 1.0
* n_transmitter (n_transmitter) float64 8B 0.0
* waveform_time (waveform_time) float64 56B -0.01667 ... 0....
* n_receiver (n_receiver) float64 16B 0.0 1.0
* n_couplet (n_couplet) float64 16B 0.0 1.0
* index (index) float64 16kB 0.0 1.0 ... 2e+03
Dimensions without coordinates: dim_0
Data variables: (12/26)
gate_times_bnds (gate_times, nv) float64 240B ...
transmitter_label (n_transmitter) <U1 4B ...
transmitter_area (n_transmitter) int64 8B ...
transmitter_waveform_type (n_transmitter) <U6 24B ...
transmitter_waveform_current (waveform_time) float64 56B ...
transmitter_scale_factor (n_transmitter) float64 8B ...
... ...
couplet_txrx_dz (n_couplet) int64 16B ...
data_normalized (dim_0) bool 1B ...
output_data_type <U1 4B ...
reference_frame <U24 96B ...
output_sample_frequency (dim_0) int64 8B ...
digitization_frequency (dim_0) int64 8B ...
Attributes:
type: system
mode: airborne
method: electromagnetic, time domain
instrument: 30Hz Tempest
name: tempest_system
!!!! Important !!!! This returns an copy from the DataTree, i.e. any changes to tempest_system does not change the source. For example:
tempest_system.attrs['aaaaaaa'] = 'adding a new attribute'
print('\n\nAltered attributes on the DataSet:')
pprint(tempest_system.attrs)
print('\n\nSource group attributes do not see the new attribute:')
pprint(survey['data/raw_data/tempest_system'].attrs)
Altered attributes on the DataSet:
{'aaaaaaa': 'adding a new attribute',
'instrument': '30Hz Tempest',
'method': 'electromagnetic, time domain',
'mode': 'airborne',
'name': 'tempest_system',
'type': 'system'}
Source group attributes do not see the new attribute:
{'instrument': '30Hz Tempest',
'method': 'electromagnetic, time domain',
'mode': 'airborne',
'name': 'tempest_system',
'type': 'system'}
Coordinates, Dimensions, and Attributes
Dimensions
Dimensions are simply name: length pairs corresponding to the dimension
coordinate variables represented in the specific Dataset group being examined
# use the method "sizes" to see a list of all the dimensions of a group
print('\n\nDimensions:\n')
print(survey['models/inverted_models'].sizes)
Dimensions:
Frozen(ChainMap({'layer_depth': 30, 'nv': 2, 'gate_times': 15, 'index': 2001}, {}, {}, {}))
Isolate an individual dimension coordinate for further examination
Tabular data are typically 1-D or 2-D variables with the primary dimension
being index, which often corresponds to the rows of the input text
file representing individual measurements.
print('\n\nLooking at the Index dimension coordinate:\n')
print(survey['models/inverted_models']['index'])
Looking at the Index dimension coordinate:
<xarray.DataArray 'index' (index: 2001)> Size: 16kB
array([0.000e+00, 1.000e+00, 2.000e+00, ..., 1.998e+03, 1.999e+03, 2.000e+03],
shape=(2001,))
Coordinates:
* index (index) float64 16kB 0.0 1.0 2.0 ... 1.998e+03 1.999e+03 2e+03
spatial_ref float64 8B ...
x (index) float64 16kB ...
y (index) float64 16kB ...
z (index) float64 16kB ...
Attributes:
standard_name: index
long_name: Index of individual data points
units: not_defined
valid_range: [ 0. 2000.]
grid_mapping: spatial_ref
If a dimension is not discrete, meaning it represents ranges (such as depth layers), then the bounds on each dimension value also need to be defined, and are linked to the dimension through the “bounds” attribute.
print('\n\nExample of a non-discrete dimension:\n')
print(survey['models/inverted_models']['layer_depth'])
Example of a non-discrete dimension:
<xarray.DataArray 'layer_depth' (layer_depth: 30)> Size: 240B
array([ 1.5 , 4.65 , 8.115, 11.925, 16.115, 20.725, 25.795, 31.375,
37.515, 44.265, 51.69 , 59.86 , 68.85 , 78.74 , 89.615, 101.575,
114.73 , 129.2 , 145.12 , 162.63 , 181.89 , 203.08 , 226.39 , 252.03 ,
280.235, 311.26 , 345.385, 382.925, 424.22 , 467.48 ])
Coordinates:
* layer_depth (layer_depth) float64 240B 1.5 4.65 8.115 ... 382.9 424.2 467.5
spatial_ref float64 8B ...
Attributes:
standard_name: layer_depth
long_name: inverted model layer depth
units: meters
valid_range: [ 1.5 467.48]
grid_mapping: spatial_ref
bounds: layer_depth_bnds
Notice that the bounds variable is 2-D [index, nv] where nv = number of vertices, in this case of length 2:
print('\n\nCorresponding bounds on this non-discrete dimension:\n')
print(survey['models/inverted_models']['layer_depth_bnds'])
Corresponding bounds on this non-discrete dimension:
<xarray.DataArray 'layer_depth_bnds' (layer_depth: 30, nv: 2)> Size: 480B
[60 values with dtype=float64]
Coordinates:
* layer_depth (layer_depth) float64 240B 1.5 4.65 8.115 ... 382.9 424.2 467.5
* nv (nv) float64 16B 0.0 1.0
spatial_ref float64 8B ...
Attributes:
standard_name: layer_depth_bounds
long_name: inverted model layer depth cell boundaries
valid_range: [ 0. 489.11]
grid_mapping: spatial_ref
print('\n\nSee the bounds:\n')
print(survey['models/inverted_models']['layer_depth_bnds'].values)
See the bounds:
[[ 0. 3. ]
[ 3. 6.3 ]
[ 6.3 9.93]
[ 9.93 13.92]
[ 13.92 18.31]
[ 18.31 23.14]
[ 23.14 28.45]
[ 28.45 34.3 ]
[ 34.3 40.73]
[ 40.73 47.8 ]
[ 47.8 55.58]
[ 55.58 64.14]
[ 64.14 73.56]
[ 73.56 83.92]
[ 83.92 95.31]
[ 95.31 107.84]
[107.84 121.62]
[121.62 136.78]
[136.78 153.46]
[153.46 171.8 ]
[171.8 191.98]
[191.98 214.18]
[214.18 238.6 ]
[238.6 265.46]
[265.46 295.01]
[295.01 327.51]
[327.51 363.26]
[363.26 402.59]
[402.59 445.85]
[445.85 489.11]]
Coordinates
Coordinates define the spatial and temporal positioning of the data (X Y Z T).
Additionally, all dimensions are linked to dimension coordinate variables that
house the coordinates (i.e. values) of that dimension.
This means a dataset can have both dimensional and non-dimensional coordinates.
Dimensional coordinates are noted with a * (or bold text) in printed output of
the xarray, such as index and gate_times in this example.
print('\n\nInspect the coordinates of a tabular dataset:\n')
print(survey['data/raw_data'].dataset.coords)
Inspect the coordinates of a tabular dataset:
Coordinates:
* index (index) float64 16kB 0.0 1.0 2.0 ... 1.998e+03 1.999e+03 2e+03
* gate_times (gate_times) float64 120B 1.085e-05 3.255e-05 ... 0.01338
spatial_ref float64 8B ...
x (index) float64 16kB ...
y (index) float64 16kB ...
z (index) float64 16kB ...
Tabular Coordinates
In Tabular data, coordinates are typically non-dimensional, since the primary
dataset dimension is index. By default, we define the spatial coordinates,
x and y, based on the longitude and latitude (or easting/northing)
data variables. If relevant, z and t coordinate variables can also be
defined, representing the vertical and temporal coordinates of the data points.
Per CF conventions, these spatiotemporal coordinates have extra attributes
required such as “axis” and strict requirements on “standard_names” to make
the datasets recognizable to GIS software. GSPy handles automatically handles
these requirements for users.
print('\n\nInspecting a specific coordinate variable:\n')
print(survey['data/raw_data']['x'])
Inspecting a specific coordinate variable:
<xarray.DataArray 'x' (index: 2001)> Size: 16kB
[2001 values with dtype=float64]
Coordinates:
* index (index) float64 16kB 0.0 1.0 2.0 ... 1.998e+03 1.999e+03 2e+03
spatial_ref float64 8B ...
x (index) float64 16kB ...
y (index) float64 16kB ...
z (index) float64 16kB ...
Attributes:
standard_name: projection_x_coordinate
long_name: Easting_Albers:PROJECTION=WGS84/Albers
units: m
format: f13.2
valid_range: [357875.5 463737.46]
grid_mapping: spatial_ref
axis: X
Note: All coordinates must match the coordinate reference system defined in the Survey.
Raster Coordinates
Raster data are gridded, typically representing maps or multi-dimensional
models.Therefore, Raster data almost always have dimensional coordinates, i.e.,
the data dimensions correspond directly to either spatial or temporal
coordinates (x, y, z, t).
print('\n\nInspect the coordinates of a raster dataset:\n')
print(survey['derived_maps/maps'].coords)
Inspect the coordinates of a raster dataset:
Coordinates:
* x (x) float64 5kB 2.928e+05 2.934e+05 ... 6.51e+05 6.516e+05
* nv (nv) float64 16B 0.0 1.0
* y (y) float64 10kB 1.607e+06 1.606e+06 ... 8.808e+05 8.802e+05
spatial_ref float64 8B ...
The Spatial Reference Coordinate
the spatial_ref coordinate variable is a non-dimensional coordinate that
contains information on the coordinate reference system. All groups within the
DataTree inherit the same spatial_ref. For more information, see Coordinate Reference Systems.
print('\n\nCRS spatial_ref variable:\n')
print(survey['derived_maps/maps']['spatial_ref'])
CRS spatial_ref variable:
<xarray.DataArray 'spatial_ref' ()> Size: 8B
[1 values with dtype=float64]
Coordinates:
spatial_ref float64 8B ...
Attributes: (12/19)
crs_wkt: PROJCRS["NAD83 / Conus Albers",BASEGEOGCR...
semi_major_axis: 6378137.0
semi_minor_axis: 6356752.314140356
inverse_flattening: 298.257222101
reference_ellipsoid_name: GRS 1980
longitude_of_prime_meridian: 0.0
... ...
longitude_of_central_meridian: -96.0
false_easting: 0.0
false_northing: 0.0
authority: EPSG
wkid: 5070
GeoTransform: [ 2.9250e+05 6.0000e+02 0.0000e+00 1.6...
Attributes
Both datasets and data variables have attributes (metadata fields). Certain attributes are required, see our documentation on Metadata Requirements for more details.
Dataset attributes
Dataset attributes provide users a way to document and describe supplementary
information about a dataset group as a whole, such as model inversion parameters
or other processing descriptions. At a minimum, a content attribute should
contain a brief summary of the contents of the dataset.
print('\n\nDataset Attributes:\n')
pprint(survey['models/inverted_models'].attrs)
Dataset Attributes:
{'comment': 'This dataset includes inverted resistivity models derived from '
'processed AEM data produced by USGS',
'content': 'inverted resistivity models',
'instrument': '30Hz Tempest',
'method': 'electromagnetic, time domain',
'mode': 'airborne',
'property': 'electrical conductivity',
'structure': 'tabular',
'type': 'model'}
Variable attributes
Each data variable must contain attributes detailing the metadata of that individual variable. These follow the Climate and Forecast (CF) metadata conventions.
print('\n\nDataArray (variable) Attributes:\n')
pprint(survey['models/inverted_models']['conductivity'].attrs)
DataArray (variable) Attributes:
{'format': '30e15.6',
'grid_mapping': 'spatial_ref',
'long_name': 'Layer conductivity',
'standard_name': 'conductivity',
'units': 'S/m',
'valid_range': array([1.456164e-04, 1.000000e+01])}
Filtering & Searching
The required keys can be used to search through a tree and find groups of interest
Here’s a simply function using native netCDF and xarray methods to search through a file and identify groups based on their attributes
import xarray as xr
from netCDF4 import Dataset
def find_groups_by_attrs(nc_path, find_attr='type', find_value=None):
with Dataset(nc_path, mode="r") as ds:
def walk(group, path=""):
for attr in group.ncattrs():
if attr == find_attr:
if find_value:
cur_value = group.getncattr(attr)
if isinstance(cur_value, list):
for ind_value in cur_value:
if ind_value == find_value:
print(f"Group: {path}")
print(f"\t{attr} = {group.getncattr(attr)!r}")
else:
if cur_value == find_value:
print(f"Group: {path}")
print(f"\t{attr} = {group.getncattr(attr)!r}")
else:
print(f"Group: {path}")
print(f"\t{attr} = {group.getncattr(attr)!r}")
# Recurse into subgroups
for gname, subg in group.groups.items():
walk(subg, path + "/" + gname)
walk(ds)
see all the groups and their types
find_groups_by_attrs(input_file, find_attr='type')
Group: /survey
type = 'survey'
Group: /survey/data
type = 'container'
Group: /survey/data/raw_data
type = 'data'
Group: /survey/data/raw_data/tempest_system
type = 'system'
Group: /survey/models
type = 'container'
Group: /survey/models/inverted_models
type = 'model'
Group: /survey/models/inverted_models/tempest_system
type = 'system'
Group: /survey/models/inverted_models/inversion_parameters
type = 'parameters'
Group: /survey/derived_maps
type = 'container'
Group: /survey/derived_maps/maps
type = 'data'
Group: /survey/derived_maps/maps/magnetic_system
type = 'system'
find just the system type groups
find_groups_by_attrs(input_file, find_attr='type', find_value='system')
Group: /survey/data/raw_data/tempest_system
type = 'system'
Group: /survey/models/inverted_models/tempest_system
type = 'system'
Group: /survey/derived_maps/maps/magnetic_system
type = 'system'
find just the model type groups
find_groups_by_attrs(input_file, find_attr='type', find_value='model')
Group: /survey/models/inverted_models
type = 'model'
find every group related to the ‘magnetic’ method
find_groups_by_attrs(input_file, find_attr='method', find_value='magnetic')
Group: /survey/data/raw_data
method = ['magnetic', 'radiometric', 'electromagnetic, time domain']
Group: /survey/derived_maps/maps
method = 'magnetic'
Group: /survey/derived_maps/maps/magnetic_system
method = 'magnetic'
Then if you want to open one of these groups now that you see the path, without GSPy just native xarray. Notice this opens just that single group, you do not have access to the rest of the datatree here:
ds = xr.open_dataset(input_file, group='/survey/derived_maps/maps')
print(ds)
<xarray.Dataset> Size: 6MB
Dimensions: (x: 599, nv: 2, y: 1212)
Coordinates:
* x (x) float64 5kB 2.928e+05 2.934e+05 ... 6.51e+05 6.516e+05
* nv (nv) float64 16B 0.0 1.0
* y (y) float64 10kB 1.607e+06 1.606e+06 ... 8.808e+05 8.802e+05
spatial_ref float64 8B ...
Data variables:
x_bnds (x, nv) float64 10kB ...
y_bnds (y, nv) float64 19kB ...
magnetic_tmi (y, x) float64 6MB ...
Attributes:
comment: contractor-derived product
content: gridded map of total magnetic intensity
type: data
mode: airborne
method: magnetic
instrument: Scintrex CS-3 cesium vapor magnetometer
structure: raster
If you want to open a datatree, point the group to where you want to start the tree, can be ‘/survey’ for the entire tree, or in this case we load just the models branch:
dt = xr.open_datatree(input_file, group='/survey/models',
decode_timedelta=True)
print(dt)
<xarray.DataTree>
Group: /
│ Dimensions: ()
│ Data variables:
│ spatial_ref float64 8B ...
│ Attributes:
│ content: inverted 1-D electrical resistivity models
│ type: container
└── Group: /inverted_models
│ Dimensions: (layer_depth: 30, nv: 2, gate_times: 15,
│ index: 2001)
│ Coordinates:
│ * layer_depth (layer_depth) float64 240B 1.5 4.65 ... 424.2 467.5
│ * nv (nv) float64 16B 0.0 1.0
│ * gate_times (gate_times) timedelta64[ns] 120B 00:00:00.00001...
│ * index (index) float64 16kB 0.0 1.0 ... 1.999e+03 2e+03
│ spatial_ref float64 8B ...
│ x (index) float64 16kB ...
│ y (index) float64 16kB ...
│ z (index) float64 16kB ...
│ Data variables: (12/46)
│ layer_depth_bnds (layer_depth, nv) float64 480B ...
│ gate_times_bnds (gate_times, nv) float64 240B ...
│ uniqueid (index) float64 16kB ...
│ survey (index) float64 16kB ...
│ date (index) float64 16kB ...
│ flight (index) float64 16kB ...
│ ... ...
│ phic (index) float64 16kB ...
│ phit (index) float64 16kB ...
│ phig (index) float64 16kB ...
│ phis (index) float64 16kB ...
│ lambda (index) float64 16kB ...
│ iterations (index) float64 16kB ...
│ Attributes:
│ content: inverted resistivity models
│ comment: This dataset includes inverted resistivity models derived fr...
│ type: model
│ method: electromagnetic, time domain
│ instrument: 30Hz Tempest
│ mode: airborne
│ property: electrical conductivity
│ structure: tabular
├── Group: /inverted_models/tempest_system
│ Dimensions: (gate_times: 15, nv: 2, n_transmitter: 1,
│ waveform_time: 7, n_receiver: 2,
│ n_couplet: 2, dim_0: 1)
│ Coordinates:
│ * n_transmitter (n_transmitter) float64 8B 0.0
│ * waveform_time (waveform_time) float64 56B -0.01667 ... 0....
│ * n_receiver (n_receiver) float64 16B 0.0 1.0
│ * n_couplet (n_couplet) float64 16B 0.0 1.0
│ Dimensions without coordinates: dim_0
│ Data variables: (12/26)
│ gate_times_bnds (gate_times, nv) timedelta64[ns] 240B ...
│ transmitter_label (n_transmitter) <U1 4B ...
│ transmitter_area (n_transmitter) int64 8B ...
│ transmitter_waveform_type (n_transmitter) <U6 24B ...
│ transmitter_waveform_current (waveform_time) float64 56B ...
│ transmitter_scale_factor (n_transmitter) float64 8B ...
│ ... ...
│ couplet_txrx_dz (n_couplet) int64 16B ...
│ data_normalized (dim_0) bool 1B ...
│ output_data_type <U1 4B ...
│ reference_frame <U24 96B ...
│ output_sample_frequency (dim_0) int64 8B ...
│ digitization_frequency (dim_0) int64 8B ...
│ Attributes:
│ type: system
│ mode: airborne
│ method: electromagnetic, time domain
│ instrument: 30Hz Tempest
│ name: tempest_system
└── Group: /inverted_models/inversion_parameters
Dimensions: ()
Data variables:
software <U61 244B ...
software_reference <U298 1kB ...
description <U1402 6kB ...
doi_calculation <U592 2kB ...
phid_cut <U256 1kB ...
Attributes:
type: parameters
method: electromagnetic, time domain
instrument: 30Hz Tempest
mode: airborne
property: electrical conductivity
name: inversion_parameters
Total running time of the script: (0 minutes 0.604 seconds)