Examples¶
Reading and plotting ASCAT H25 data from netCDF format¶
This Example script reads and plots ASCAT H25 SSM data with different masking options and also converts the data to absolute values using the included porosity data. It can be found in the /examples folder of the pytesmo package under the name read_ASCAT_H25.py
If the standard file names assumed by the script have changed this can be specified during initialization of the AscatH25_SSM object. Please see the documentation of pytesmo.io.sat.ascat.AscatH25_SSM
In[1]:
import matplotlib.pyplot as plt
import pytesmo.io.sat.ascat as ascat
import os
ascat_folder = os.path.join('R:\\','Datapool_processed','WARP','WARP5.5',
'ASCAT_WARP5.5_R1.2','080_ssm','netcdf')
ascat_grid_folder = os.path.join('R:\\','Datapool_processed','WARP','ancillary','warp5_grid')
#init the ASCAT_SSM reader with the paths
#ascat_folder is the path in which the cell files are located e.g. TUW_METOP_ASCAT_WARP55R12_0600.nc
#ascat_grid_folder is the path in which the file TUW_WARP5_grid_info_2_1.nc is located
#let's not include the orbit direction since it is saved as 'A'
#or 'D' it can not be plotted
# the AscatH25_SSM class automatically detects the version of data that you have in your
# ascat_folder. Please do not mix files of different versions in one folder
ascat_SSM_reader = ascat.AscatH25_SSM(ascat_folder,ascat_grid_folder,
include_in_df=['sm', 'sm_noise', 'ssf', 'proc_flag'])
gpi = 2329253
ascat_ts = ascat_SSM_reader.read_ssm(gpi)
ascat_ts.plot()
plt.show()

In[2]:
#the ASCATTimeSeries object also contains metadata
print "Topographic complexity", ascat_ts.topo_complex
print "Wetland fraction", ascat_ts.wetland_frac
print "Porosity from GLDAS model", ascat_ts.porosity_gldas
print "Porosity from Harmonized World Soil Database", ascat_ts.porosity_hwsd
Topographic complexity 14
Wetland fraction 0
Porosity from GLDAS model 0.542222
Porosity from Harmonized World Soil Database 0.430234
In[3]:
#It is also possible to automatically convert the soil moisture to absolute values using
ascat_ts_absolute = ascat_SSM_reader.read_ssm(gpi, absolute_values=True)
#this introduces 4 new columns in the returned data
#scaled sm and sm_noise with porosity_gldas
#scaled sm and sm_noise with porosity_hwsd
print ascat_ts_absolute.data
#select relevant columns and plot
ascat_ts_absolute.data = ascat_ts_absolute.data[['sm_por_gldas','sm_por_hwsd']]
ascat_ts_absolute.plot()
plt.show()
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 2292 entries, 2007-01-01 09:39:39 to 2013-07-12 20:39:08
Data columns (total 10 columns):
proc_flag 2292 non-null values
sm 2285 non-null values
sm_noise 2285 non-null values
ssf 2292 non-null values
sm_por_gldas 2285 non-null values
sm_noise_por_gldas 2285 non-null values
sm_por_hwsd 2285 non-null values
sm_noise_por_hwsd 2285 non-null values
frozen_prob 2285 non-null values
snow_prob 2285 non-null values
dtypes: float64(8), int16(1), int8(1)

In[4]:
#We can also automatically mask the data during reading
#In this example all measurements where the Surface State Flag
#shows frozen or where the frozen or snow probabilities are more
#than 10 percent are removed from the time series
ascat_ts = ascat_SSM_reader.read_ssm(gpi, mask_ssf=True,
mask_frozen_prob=10,
mask_snow_pro=10)
ascat_ts.plot()
plt.show()

Reading and plotting H-SAF images¶
H-SAF provides three different image products:
- SM OBS 1 - H07 - Large scale surface soil moisture by radar scatterometer in BUFR format over Europe
- SM OBS 2 - H08 - Small scale surface soil moisture by radar scatterometer in BUFR format over Europe
- SM DAS 2 - H14 - Profile index in the roots region by scatterometer data assimilation in GRIB format, global
The following example will show how to read and plot each of them. It can be found in the /examples folder of the pytesmo package under the name Read_H_SAF_images.py
Reading H-SAF images¶
In this Example we will read and plot images of the H-SAF products H07, H08 and H14.
import pytesmo.io.sat.h_saf as h_saf
import os
import datetime
#set the paths to the image files
#I'm using the test images included in the pytesmo package
test_data_path = os.path.join(os.sep+'home','pydev','pytesmo','tests','test_sat','test_data','h_saf')
h07_path = os.path.join(test_data_path, 'h07')
h08_path = os.path.join(test_data_path, 'h08')
h14_path = os.path.join(test_data_path, 'h14')
#initialize the readers with the path
h07_reader = h_saf.H07img(h07_path)
h08_reader = h_saf.H08img(h08_path)
h14_reader = h_saf.H14img(h14_path)
Reading the H07 product¶
pytesmo includes one h07 image with the timestamp 2010-05-01 08:33:01 We can either read this image alone if we know the timestamp or iterate over all images on 2010-05-01.
#the reader returns not only the data but also metadata and the longitudes and latitudes
h07_data, metadata, timestamp, lons, lats, time_var = h07_reader.read_img(datetime.datetime(2010,5,1,8,33,1))
let’s inspect the data
print type(h07_data)
# the data is a dictionary, each dictionary key contains the array of one variable
print "The following variables are in this image", h07_data.keys()
<type 'dict'>
The following variables are in this image ['snow_cover', 'topo_complex', 'ssm', 'jd', 'ssm_noise', 'frozen_prob']
let’s inspect surface soil moisture (ssm)
print h07_data['ssm'].shape
# it is only a 1D array to plot the data we also need latitude and logitude information
print lons.shape
print lats.shape
(1017,)
(1017,)
(1017,)
let’s resample and plot the ssm data H07 data is not that easy to plot because it comes in orbit geometry and not on a fixed grid.
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
import pytesmo.colormaps.load_cmap as smcolormaps
import pytesmo.grid.resample as resample
import numpy as np
#lets resample to a 0.1 degree grid
#define the grid points in latitude and logitude
lats_dim = np.arange(25,75,0.1)
lons_dim = np.arange(-25,45,0.1)
#make 2d grid out the 1D grid spacings
lons_grid, lats_grid = np.meshgrid(lons_dim,lats_dim)
resampled_data = resample.resample_to_grid(h07_data, lons, lats,
lons_grid, lats_grid)
fig = plt.figure(figsize=(10,10))
ax = fig.add_axes([0.1,0.1,0.8,0.8])
# setup of basemap for europe
# simple mercator projection
m = Basemap(llcrnrlon=-25.0,llcrnrlat=25.0,urcrnrlon=45.0,urcrnrlat=75.0,\
resolution='l',area_thresh=1000.,projection='merc',\
lat_ts=50.,ax=ax)
# make a pseudocolor plot using the ASCAT SWI colormap
im = m.pcolormesh(lons_grid, lats_grid, resampled_data['ssm'], latlon=True,
cmap=smcolormaps.load('SWI_ASCAT'))
m.drawcoastlines()
m.drawcountries()
# draw parallels and meridians.
# label on left and bottom of map.
parallels = np.arange(20,80,10.)
m.drawparallels(parallels,labels=[1,0,0,0])
meridians = np.arange(-60,50,10.)
m.drawmeridians(meridians,labels=[0,0,0,1])
# add colorbar
cb = m.colorbar(im,"right", size="5%", pad='2%')
ax.set_title('H07 Soil Moisture in %')
plt.show()

Reading the H08 product¶
H08 data has a much higher resolution and comes on a 0.00416 degree grid.
The sample data included in pytesmo was observed on the same time as the included H07 product.
Instead of read_img you can also use the daily_images iterator.
You just specify a day and it will read all the images that are in your folder for this day.
This also works for the other H07 and H14 reader.
#the reader returns not only the data but also metadata and the longitudes and latitudes
for h08_data, metadata, timestamp, lons, lats, time_var in h08_reader.daily_images(datetime.datetime(2010,5,1)):
# this tells you the exact timestamp of the read image
print timestamp.isoformat()
print type(h08_data)
# the data is a dictionary, each dictionary key contains the array of one variable
print "The following variables are in this image", h08_data.keys()
print h08_data['ssm'].shape
print lons.shape
print lats.shape
2010-05-01T08:33:01
<type 'dict'>
The following variables are in this image ['ssm', 'proc_flag', 'ssm_noise', 'corr_flag']
(3120, 7680)
(3120, 7680)
(3120, 7680)
In our case only one image is in the folder so the loop exits after this image is read.
The data has higher resolution but it already comes as a 2D image.
Let’s plot it.
fig = plt.figure(figsize=(10,10))
ax = fig.add_axes([0.1,0.1,0.8,0.8])
# setup of basemap for europe but zoomed in little bit
# simple mercator projection
m = Basemap(llcrnrlon=10,llcrnrlat=50.0,urcrnrlon=45.0,urcrnrlat=75.0,\
resolution='l',area_thresh=1000.,projection='merc',\
lat_ts=50.,ax=ax)
# make a pseudocolor plot using the ASCAT SWI colormap
# latitudes and data have to be flipped upside down because the latitudes
# have to be in ascending order to be plotted correctly
# mask values > 100 so that they are not plotted
im = m.pcolormesh(lons, np.flipud(lats), np.ma.masked_greater(np.flipud(h08_data['ssm']),100), latlon=True,
vmin=0, vmax=100,cmap=smcolormaps.load('SWI_ASCAT'))
m.drawcoastlines()
m.drawcountries()
# draw parallels and meridians.
# label on left and bottom of map.
parallels = np.arange(20,80,10.)
m.drawparallels(parallels,labels=[1,0,0,0])
meridians = np.arange(-60,50,10.)
m.drawmeridians(meridians,labels=[0,0,0,1])
# add colorbar
cb = m.colorbar(im,"right", size="5%", pad='2%')
ax.set_title('H08 Soil Moisture in %')
plt.show()

Reading only area of interest¶
H08 has a very high resolution, so most people will only want to read it for their area of interest. This can be done using the lat_lon_bbox keyword
#the reader returns not only the data but also metadata and the longitudes and latitudes
h08_roi, metadata, timestamp, lons, lats, time_var = h08_reader.read_img(datetime.datetime(2010,5,1,8,33,1),
lat_lon_bbox=[60,70,15,25])
fig = plt.figure(figsize=(10,10))
ax = fig.add_axes([0.1,0.1,0.8,0.8])
# setup of basemap for europe but zoomed in little bit
# simple mercator projection
m = Basemap(llcrnrlon=10,llcrnrlat=50.0,urcrnrlon=45.0,urcrnrlat=75.0,\
resolution='l',area_thresh=1000.,projection='merc',\
lat_ts=50.,ax=ax)
# make a pseudocolor plot using the ASCAT SWI colormap
# latitudes and data have to be flipped upside down because the latitudes
# have to be in ascending order to be plotted correctly
# mask values > 100 so that they are not plotted
im = m.pcolormesh(lons, np.flipud(lats), np.ma.masked_greater(np.flipud(h08_roi['ssm']),100), latlon=True,
vmin=0, vmax=100,cmap=smcolormaps.load('SWI_ASCAT'))
m.drawcoastlines()
m.drawcountries()
# draw parallels and meridians.
# label on left and bottom of map.
parallels = np.arange(20,80,10.)
m.drawparallels(parallels,labels=[1,0,0,0])
meridians = np.arange(-60,50,10.)
m.drawmeridians(meridians,labels=[0,0,0,1])
# add colorbar
cb = m.colorbar(im,"right", size="5%", pad='2%')
ax.set_title('H08 Soil Moisture in %')
plt.show()

Reading the H14 product¶
The H14 product is a global product on a reduced gaussian grid with a resolution of approx. 25km.
#the reader returns not only the data but also metadata and the longitudes and latitudes
h14_data, metadata, timestamp, lons, lats, time_var = h14_reader.read_img(datetime.datetime(2014, 5, 15))
print type(h14_data)
# the data is a dictionary, each dictionary key contains the array of one variable
print "The following variables are in this image", h14_data.keys()
<type 'dict'>
The following variables are in this image ['SM_layer1_0-7cm', 'SM_layer2_7-28cm', 'SM_layer3_28-100cm', 'SM_layer4_100-289cm']
print h14_data['SM_layer1_0-7cm'].shape
print lons.shape
print lats.shape
(800, 1600)
(800, 1600)
(800, 1600)
The data comes as a 2D array. If the keyword expand_grid is set to False during reader initialization then only 1D arrays would be returned.
This can be good for working with the data but for plotting the expanded grid is easier to handle.
h14_reader_1d = h_saf.H14img(h14_path, expand_grid=False)
#the reader returns not only the data but also metadata and the longitudes and latitudes
h14_data_1d, metadata, timestamp, lons_1d, lats_1d, time_var = h14_reader_1d.read_img(datetime.datetime(2014, 5, 15))
print h14_data_1d['SM_layer1_0-7cm'].shape
print lons_1d.shape
print lats_1d.shape
(843490,)
(843490,)
(843490,)
Let’s plot all layers in the H14 image
for layer in h14_data:
fig = plt.figure(figsize=(10,10))
ax = fig.add_axes([0.1,0.1,0.8,0.8])
# setup of basemap for the world
# Robinson projection
m = Basemap(projection='robin',lon_0=0,resolution='c',ax=ax)
# make a pseudocolor plot using the ASCAT SWI colormap
# latitudes and data have to be flipped upside down because the latitudes
# have to be in ascending order to be plotted correctly
# mask values > 100 so that they are not plotted
im = m.pcolormesh(lons, lats, h14_data[layer], latlon=True,
vmin=0, vmax=1,cmap=smcolormaps.load('SWI_ASCAT'))
m.drawcoastlines()
m.drawcountries()
# draw parallels and meridians.
# label on left and bottom of map.
parallels = np.arange(-90,90,30.)
m.drawparallels(parallels,labels=[1,0,0,0])
meridians = np.arange(-180,180,40.)
m.drawmeridians(meridians,labels=[0,0,0,1])
# add colorbar
cb = m.colorbar(im,"right", size="5%", pad='2%')
ax.set_title('H14 {:}'.format(layer))
plt.show()




Reading and plotting data from the ISMN¶
This example program chooses a random Network and Station and plots the first variable,depth,sensor combination. To see how to get data for a variable from all stations see the next example.
It can be found in the /examples folder of the pytesmo package under the name plot_ISMN_data.py.
In[1]:
import pytesmo.io.ismn.interface as ismn
import os
import matplotlib.pyplot as plt
import random
In[2]:
#path unzipped file downloaded from the ISMN web portal
#on windows the first string has to be your drive letter
#like 'C:\\'
path_to_ismn_data = os.path.join('D:\\','small_projects','cpa_2013_07_ISMN_userformat_reader',
'header_values_parser_test')
In[3]:
#initialize interface, this can take up to a few minutes the first
#time, since all metadata has to be collected
ISMN_reader = ismn.ISMN_Interface(path_to_ismn_data)
#plot available station on a map
ISMN_reader.plot_station_locations()

In[4]:
#select random network and station to plot
networks = ISMN_reader.list_networks()
print "Available Networks:"
print networks
Available Networks:
['OZNET']
In[5]:
network = random.choice(networks)
stations = ISMN_reader.list_stations(network = network)
print "Available Stations in Network %s"%network
print stations
Available Stations in Network OZNET
['Alabama' 'Balranald-Bolton_Park' 'Banandra' 'Benwerrin' 'Bundure'
'Canberra_Airport' 'Cheverelis' 'Cooma_Airfield' 'Cootamundra_Aerodrome'
'Cox' 'Crawford' 'Dry_Lake' 'Eulo' 'Evergreen' 'Ginninderra_K4'
'Ginninderra_K5' 'Griffith_Aerodrome' 'Hay_AWS' 'Keenan' 'Kyeamba_Downs'
'Kyeamba_Mouth' 'Kyeamba_Station' 'Rochedale' 'S_Coleambally' 'Samarra'
'Silver_Springs' 'Spring_Bank' 'Strathvale' 'Uri_Park' 'Waitara'
'Weeroona' 'West_Wyalong_Airfield' 'Widgiewa' 'Wollumbi' 'Wynella'
'Yamma_Road' 'Yammacoona' 'Yanco_Research_Station']
In[6]:
station = random.choice(stations)
station_obj = ISMN_reader.get_station(station)
print "Available Variables at Station %s"%station
#get the variables that this station measures
variables = station_obj.get_variables()
print variables
Available Variables at Station Evergreen
['precipitation' 'soil moisture' 'soil temperature']
In[7]:
#to make sure the selected variable is not measured
#by different sensors at the same depths
#we also select the first depth and the first sensor
#even if there is only one
depths_from,depths_to = station_obj.get_depths(variables[0])
sensors = station_obj.get_sensors(variables[0],depths_from[0],depths_to[0])
#read the data of the variable, depth, sensor combination
time_series = station_obj.read_variable(variables[0],depth_from=depths_from[0],depth_to=depths_to[0],sensor=sensors[0])
#print information about the selected time series
print "Selected time series is:"
print time_series
Selected time series is:
OZNET Evergreen -0.50 m - -0.50 m precipitation measured with TB4-0.2-mm-tipping-bucket-raingauge
In[8]:
#plot the data
time_series.plot()
#with pandas 0.12 time_series.plot() also works
plt.legend()
plt.show()

In[9]:
#we also want to see soil moisture
sm_depht_from,sm_depht_to = station_obj.get_depths('soil moisture')
print sm_depht_from,sm_depht_to
[ 0. 0. 0.3 0.6] [ 0.05 0.3 0.6 0.9 ]
In[10]:
#read sm data measured in first layer 0-0.05m
sm = station_obj.read_variable('soil moisture',depth_from=0,depth_to=0.05)
sm.plot()
plt.show()

Calculating anomalies and climatologies¶
This Example script reads and plots ASCAT H25 SSM data. The pytesmo.time_series.anomaly module is then used to calculate anomalies and climatologies of the time series. It can be found in the /examples folder of the pytesmo package under the name anomalies.py
import pytesmo.io.sat.ascat as ascat
import pytesmo.time_series as ts
import os
import matplotlib.pyplot as plt
ascat_folder = os.path.join('R:\\','Datapool_processed','WARP','WARP5.5',
'ASCAT_WARP5.5_R1.2','080_ssm','netcdf')
ascat_grid_folder = os.path.join('R:\\','Datapool_processed','WARP','ancillary','warp5_grid')
#init the ASCAT_SSM reader with the paths
ascat_SSM_reader = ascat.AscatH25_SSM(ascat_folder,ascat_grid_folder)
ascat_ts = ascat_SSM_reader.read_ssm(45,0)
#plot soil moisture
ascat_ts.data['sm'].plot()
<matplotlib.axes.AxesSubplot at 0x22ee3550>

#calculate anomaly based on moving +- 17 day window
anomaly = ts.anomaly.calc_anomaly(ascat_ts.data['sm'], window_size=35)
anomaly.plot()
<matplotlib.axes.AxesSubplot at 0x269109e8>

#calculate climatology
climatology = ts.anomaly.calc_climatology(ascat_ts.data['sm'])
climatology.plot()
<matplotlib.axes.AxesSubplot at 0x1bc54ef0>

#calculate anomaly based on climatology
anomaly_clim = ts.anomaly.calc_anomaly(ascat_ts.data['sm'], climatology=climatology)
anomaly_clim.plot()
<matplotlib.axes.AxesSubplot at 0x1bc76860>

Calculation of the Soil Water Index¶
The Soil Water Index which is a method to estimate root zone soil moisture can be calculated from Surface Soil Moisture using an exponential Filter. For more details see the publication of C.Abergel et.al. The following example shows how to calculate the SWI for two T values from ASCAT H25 SSM.
import os
import matplotlib.pyplot as plt
from pytesmo.time_series.filters import exp_filter
import pytesmo.io.sat.ascat as ascat
ascat_folder = os.path.join('/media', 'sf_R', 'Datapool_processed',
'WARP', 'WARP5.5', 'IRMA0_WARP5.5_P2',
'R1', '080_ssm', 'netcdf')
ascat_grid_folder = os.path.join('/media', 'sf_R',
'Datapool_processed', 'WARP',
'ancillary', 'warp5_grid')
# init the ASCAT_SSM reader with the paths
# ascat_folder is the path in which the cell files are
# located e.g. TUW_METOP_ASCAT_WARP55R12_0600.nc
# ascat_grid_folder is the path in which the file
# TUW_WARP5_grid_info_2_1.nc is located
# let's not include the orbit direction since it is saved as 'A'
# or 'D' it can not be plotted
# the AscatH25_SSM class automatically detects the version of data
# that you have in your ascat_folder. Please do not mix files of
# different versions in one folder
ascat_SSM_reader = ascat.AscatH25_SSM(ascat_folder, ascat_grid_folder,
include_in_df=['sm', 'sm_noise',
'ssf', 'proc_flag'])
ascat_ts = ascat_SSM_reader.read_ssm(gpi, mask_ssf=True, mask_frozen_prob=10,
mask_snow_prob=10)
ascat_ts.plot()

# Drop NA measurements
ascat_sm_ts = ascat_ts.data[['sm', 'sm_noise']].dropna()
# Get julian dates of time series
jd = ascat_sm_ts.index.to_julian_date().get_values()
# Calculate SWI T=10
ascat_sm_ts['swi_t10'] = exp_filter(ascat_sm_ts['sm'].values, jd, ctime=10)
ascat_sm_ts['swi_t50'] = exp_filter(ascat_sm_ts['sm'].values, jd, ctime=50)
fig, ax = plt.subplots(1, 1, figsize=(15, 5))
ascat_sm_ts['sm'].plot(ax=ax, alpha=0.4, marker='o',color='#00bfff', label='SSM')
ascat_sm_ts['swi_t10'].plot(ax=ax, lw=2,label='SWI T=10')
ascat_sm_ts['swi_t50'].plot(ax=ax, lw=2,label='SWI T=50')
plt.legend()

Comparing ASCAT and insitu data from the ISMN¶
This example program loops through all insitu stations that measure soil moisture with a depth between 0 and 0.1m it then finds the nearest ASCAT grid point and reads the ASCAT data. After temporal matching and scaling using linear CDF matching it computes several metrics, like the correlation coefficients(Pearson’s, Spearman’s and Kendall’s), Bias, RMSD as well as the Nash–Sutcliffe model efficiency coefficient.
It also shows the usage of the pytesmo.df_metrics module.
It is stopped after 2 stations to not take to long to run and produce a lot of plots
It can be found in the /examples folder of the pytesmo package under the name compare_ISMN_ASCAT.py.
import pytesmo.io.ismn.interface as ismn
import pytesmo.io.sat.ascat as ascat
import pytesmo.temporal_matching as temp_match
import pytesmo.scaling as scaling
import pytesmo.df_metrics as df_metrics
import pytesmo.metrics as metrics
import os
import matplotlib.pyplot as plt
ascat_folder = os.path.join('R:\\','Datapool_processed','WARP','WARP5.5',
'ASCAT_WARP5.5_R1.2','080_ssm','netcdf')
ascat_grid_folder = os.path.join('R:\\','Datapool_processed','WARP','ancillary','warp5_grid')
#init the ASCAT_SSM reader with the paths
#let's not include the orbit direction since it is saved as 'A'
#or 'D' it can not be plotted
ascat_SSM_reader = ascat.AscatH25_SSM(ascat_folder,ascat_grid_folder,
include_in_df=['sm', 'sm_noise', 'ssf', 'proc_flag'])
#set path to ISMN data
path_to_ismn_data =os.path.join('D:\\','small_projects','cpa_2013_07_ISMN_userformat_reader','header_values_parser_test')
#Initialize reader
ISMN_reader = ismn.ISMN_Interface(path_to_ismn_data)
i = 0
label_ascat='sm'
label_insitu='insitu_sm'
#this loops through all stations that measure soil moisture
for station in ISMN_reader.stations_that_measure('soil moisture'):
#this loops through all time series of this station that measure soil moisture
#between 0 and 0.1 meters
for ISMN_time_series in station.data_for_variable('soil moisture',min_depth=0,max_depth=0.1):
ascat_time_series = ascat_SSM_reader.read_ssm(ISMN_time_series.longitude,
ISMN_time_series.latitude,
mask_ssf=True,
mask_frozen_prob = 5,
mask_snow_prob = 5)
#drop nan values before doing any matching
ascat_time_series.data = ascat_time_series.data.dropna()
ISMN_time_series.data = ISMN_time_series.data.dropna()
#rename the soil moisture column in ISMN_time_series.data to insitu_sm
#to clearly differentiate the time series when they are plotted together
ISMN_time_series.data.rename(columns={'soil moisture':label_insitu},inplace=True)
#get ISMN data that was observerd within +- 1 hour(1/24. day) of the ASCAT observation
#do not include those indexes where no observation was found
matched_data = temp_match.matching(ascat_time_series.data,ISMN_time_series.data,
window=1/24.)
#matched ISMN data is now a dataframe with the same datetime index
#as ascat_time_series.data and the nearest insitu observation
#continue only with relevant columns
matched_data = matched_data[[label_ascat,label_insitu]]
#the plot shows that ISMN and ASCAT are observed in different units
matched_data.plot(figsize=(15,5),secondary_y=[label_ascat],
title='temporally merged data')
plt.show()
#this takes the matched_data DataFrame and scales all columns to the
#column with the given reference_index, in this case in situ
scaled_data = scaling.scale(matched_data, method='lin_cdf_match',
reference_index=1)
#now the scaled ascat data and insitu_sm are in the same space
scaled_data.plot(figsize=(15,5), title='scaled data')
plt.show()
plt.scatter(scaled_data[label_ascat].values,scaled_data[label_insitu].values)
plt.xlabel(label_ascat)
plt.ylabel(label_insitu)
plt.show()
#calculate correlation coefficients, RMSD, bias, Nash Sutcliffe
x, y = scaled_data[label_ascat].values, scaled_data[label_insitu].values
print "ISMN time series:",ISMN_time_series
print "compared to"
print ascat_time_series
print "Results:"
#df_metrics takes a DataFrame as input and automatically
#calculates the metric on all combinations of columns
#returns a named tuple for easy printing
print df_metrics.pearsonr(scaled_data)
print "Spearman's (rho,p_value)", metrics.spearmanr(x, y)
print "Kendalls's (tau,p_value)", metrics.kendalltau(x, y)
print df_metrics.kendalltau(scaled_data)
print df_metrics.rmsd(scaled_data)
print "Bias", metrics.bias(x, y)
print "Nash Sutcliffe", metrics.nash_sutcliffe(x, y)
i += 1
#only show the first 2 stations, otherwise this program would run a long time
#and produce a lot of plots
if i >= 2:
break



ISMN time series: OZNET Alabama 0.00 m - 0.05 m soil moisture measured with Stevens-Hydra-Probe
compared to
ASCAT time series gpi:1884359 lat:-35.342 lon:147.541
Results:
(Pearsons_r(sm_and_insitu_sm=0.61607679781575175), p_value(sm_and_insitu_sm=3.1170801211098453e-65))
Spearman's (rho,p_value) (0.64651747115098912, 1.0057610194056589e-73)
Kendalls's (tau,p_value) (0.4685441550995097, 2.4676437876515864e-67)
(Kendall_tau(sm_and_insitu_sm=0.4685441550995097), p_value(sm_and_insitu_sm=2.4676437876515864e-67))
rmsd(sm_and_insitu_sm=0.078018684719599857)
Bias 0.00168114697282
Nash Sutcliffe 0.246416864767



ISMN time series: OZNET Balranald-Bolton_Park 0.00 m - 0.08 m soil moisture measured with CS615
compared to
ASCAT time series gpi:1821003 lat:-33.990 lon:146.381
Results:
(Pearsons_r(sm_and_insitu_sm=0.66000287576696759), p_value(sm_and_insitu_sm=1.3332742454781072e-126))
Spearman's (rho,p_value) (0.65889275747696652, 4.890533231776912e-126)
Kendalls's (tau,p_value) (0.48653686844813893, 6.6517671082477896e-118)
(Kendall_tau(sm_and_insitu_sm=0.48653686844813893), p_value(sm_and_insitu_sm=6.6517671082477896e-118))
rmsd(sm_and_insitu_sm=0.028314835540753237)
Bias 4.56170862568e-05
Nash Sutcliffe 0.316925662899
Reading and plotting ASCAT data from binary format¶
This example program reads and plots ASCAT SSM and SWI data with different masking options. It can be found in the /examples folder of the pytesmo package under the name plot_ASCAT_data.py.
In[1]:
import pytesmo.io.sat.ascat as ascat
import os
import matplotlib.pyplot as plt
In[2]:
#I've downloaded my ASCAT data to a folder on my D drive
path_to_ascat_ssm_data = os.path.join('D:\\','small_projects','cpa_2013_07_userformat_reader',
'data','ASCAT_SSM_25km_ts_WARP5.5_R0.1','data')
path_to_ascat_swi_data = os.path.join('D:\\','small_projects','cpa_2013_07_userformat_reader',
'data','ASCAT_SWI_25km_ts_WARP5.5_R0.1','data')
#path to grid definition file, default name TUW_W54_01_lonlat-ld-land.txt
path_to_grid_definition = os.path.join('D:\\','small_projects','cpa_2013_07_userformat_reader',
'data','auxiliary_data','grid_info')
#path to advisory flags from FTP Server
path_to_adv_flags = os.path.join('D:\\','small_projects','cpa_2013_07_userformat_reader',
'data','auxiliary_data','advisory_flags')
In[3]:
#init the ASCAT_SSM reader with the paths
ascat_SSM_reader = ascat.Ascat_SSM(path_to_ascat_ssm_data,path_to_grid_definition,
advisory_flags_path = path_to_adv_flags)
In[4]:
lon, lat = 16, 48
#reads ssm data nearest to this lon,lat coordinates
ssm_data_raw = ascat_SSM_reader.read_ssm(lon,lat)
#plot the data using pandas builtin plot functionality
ssm_data_raw.plot()
plt.show()

In[5]:
#read the same data but mask observations where the SSF shows frozen
#and where frozen and snow probabilty are greater than 20%
ssm_data_masked = ascat_SSM_reader.read_ssm(lon,lat,mask_ssf=True,mask_frozen_prob=20,mask_snow_prob=20)
#plot the data using pandas builtin plot functionality
#this time using a subplot for each variable in the DataFrame
ssm_data_masked.plot(subplots=True)
plt.show()

In[6]:
#plot raw and masked SSM data in one plot to compare them
ssm_data_raw.data['SSM'].plot(label='raw SSM data')
ssm_data_masked.data['SSM'].plot(label='masked SSM data')
plt.legend()
plt.show()

In[7]:
ascat_SWI_reader = ascat.Ascat_SWI(path_to_ascat_swi_data,path_to_grid_definition,
advisory_flags_path = path_to_adv_flags)
#reads swi data nearest to this lon,lat coordinates
#without any additional keywords all unmasked T values and
#Quality flags will be read
swi_data_raw = ascat_SWI_reader.read_swi(lon,lat)
#plot the data using pandas builtin plot functionality
swi_data_raw.plot()
plt.show()

In[8]:
#read the same data but this time only SWI with a T value
#of 20 is returned
swi_data_T_20 = ascat_SWI_reader.read_swi(lon,lat,T=20)
#plot the data using pandas builtin plot functionality
#this time using a subplot for each variable in the DataFrame
swi_data_T_20.plot(subplots=True)
plt.show()

In[9]:
#you can also mask manually if you prefer
swi_data_T_20.data = swi_data_T_20.data[swi_data_T_20.data['frozen_prob'] < 10]
swi_data_T_20.plot(subplots=True)
plt.show()
