pytesmo.io.sat package¶
Submodules¶
pytesmo.io.sat.ascat module¶
Created on Jul 29, 2013
@author: Christoph Paulik christoph.paulik@geo.tuwien.ac.at
- exception pytesmo.io.sat.ascat.ASCATReaderException[source]¶
Bases: exceptions.Exception
- class pytesmo.io.sat.ascat.ASCATTimeSeries(gpi, lon, lat, cell, data, topo_complex=None, wetland_frac=None, porosity_gldas=None, porosity_hwsd=None)[source]¶
Bases: object
Container class for ASCAT time series
Parameters: gpi : int
grid point index
lon : float
longitude of grid point
lat : float
latitude of grid point
cell : int
cell number of grid point
data : pandas.DataFrame
DataFrame which contains the data
topo_complex : int, optional
topographic complexity at the grid point
wetland_frac : int, optional
wetland fraction at the grid point
porosity_gldas : float, optional
porosity taken from GLDAS model
porosity_hwsd : float, optional
porosity calculated from Harmonised World Soil Database
Attributes
gpi (int) grid point index longitude (float) longitude of grid point latitude (float) latitude of grid point cell (int) cell number of grid point data (pandas.DataFrame) DataFrame which contains the data topo_complex (int) topographic complexity at the grid point wetland_frac (int) wetland fraction at the grid point porosity_gldas (float) porosity taken from GLDAS model porosity_hwsd (float) porosity calculated from Harmonised World Soil Database Methods
plot(*args, **kwargs) wrapper for pandas.DataFrame.plot which adds title to plot
- class pytesmo.io.sat.ascat.AscatH25_SSM(path, grid_path, grid_info_filename='TUW_WARP5_grid_info_2_1.nc', topo_threshold=50, wetland_threshold=50, include_in_df=['sm', 'sm_noise', 'ssf', 'proc_flag', 'orbit_dir'])[source]¶
Bases: pytesmo.io.sat.ascat.AscatNetcdf
class for reading ASCAT SSM data. It extends AscatNetcdf and provides the information necessary for reading SSM data
Parameters: path : string
path to data folder which contains the netCDF files from the FTP server
grid_path : string
path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info netCDF file in grid_path default ‘TUW_WARP5_grid_info_2_1.nc’
advisory_flags_path : string, optional
path to advisory flags .dat files, if not provided they will not be used
topo_threshold : int, optional
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int, optional
if wetland fraction of read grid point is above this threshold a warning is output during reading
include_in_df : list, optional
list of variables which should be included in the returned DataFrame. Default is all variables [‘sm’, ‘sm_noise’, ‘ssf’, ‘proc_flag’, ‘orbit_dir’]
Attributes
include_in_df (list) list of variables in the netcdf file that should be returned to the user after reading Methods
read_ssm(*args,**kwargs) read surface soil moisture - read_ssm(*args, **kwargs)[source]¶
function to read SSM takes either 1 or 2 arguments. It can be called as read_ssm(gpi,**kwargs) or read_ssm(lon,lat,**kwargs)
Parameters: gpi : int
grid point index
lon : float
longitude of point
lat : float
latitude of point
mask_ssf : boolean, optional
default False, if True only SSF values of 1 will be allowed, all others are removed
mask_frozen_prob : int,optional
if included in kwargs then all observations taken when frozen probability > mask_frozen_prob are removed from the result
mask_snow_prob : int,optional
if included in kwargs then all observations taken when snow probability > mask_snow_prob are removed from the result
absolute_values : boolean, optional
if True soil porosities from HWSD and GLDAS will be used to derive absolute values which will be available in the pandas.DataFrame in the columns ‘sm_por_gldas’,’sm_noise_por_gldas’, ‘sm_por_hwsd’,’sm_noise_por_hwsd’
Returns: ASCATTimeSeries : object
pytesmo.io.sat.ascat.ASCATTimeSeries instance
- class pytesmo.io.sat.ascat.AscatNetcdf(path, grid_path, grid_info_filename='TUW_WARP5_grid_info_2_1.nc', topo_threshold=50, wetland_threshold=50, netcdftemplate='TUW_METOP_ASCAT_WARP55R12_%04d.nc', loc_id='gpi', obs_var='row_size', topo_var='topo', wetland_var='wetland', snow_var='snow', frozen_var='frozen', read_bulk=False)[source]¶
Bases: object
Class that provides access to ASCAT data stored in netCDF format which is downloadable from the HSAF website.
Parameters: path : string
path to data folder which contains the zip files from the FTP server
grid_path : string
path to grid_info folder which contains a netcdf file with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info netCDF file in grid_path default ‘TUW_WARP5_grid_info_2_1.nc’
topo_threshold : int, optional
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int, optional
if wetland fraction of read grid point is above this threshold a warning is output during reading
netcdftemplate : string, optional
string template for the netCDF filename. This specifies where the cell number is in the netCDF filename. Standard value is ‘TUW_METOP_ASCAT_WARP55R12_%04d.nc’ in which %04d will be substituded for the cell number during reading of the data
loc_id : string, optional
name of the location id in the netCDF file
obs_var : string, optional
observation variable that provides the lookup between observation number and the location id
topo_var : string, optional
name of topographic complexity variable in netCDF file
wetland_var : string, optional
name of wetland fraction variable in netCDF file
snow_var : string, optional
name of snow probability variable in netCDF file
frozen_var : string, optional
name of frozen probability variable in netCDF file
Attributes
———-
path : string
path to data folder which contains the zip files from the FTP server
grid_path : string
path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info netCDF file in grid_path default ‘TUW_WARP5_grid_info_2_1.nc’
topo_threshold : int
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int
if wetland fraction of read grid point is above this threshold a warning is output during reading
grid_info_loaded : boolean
true if the grid information has already been loaded
grid : grids.CellGrid object
CellGrid object, which provides nearest neighbor search and other features
advisory_flags_path : string
path to advisory flags .dat files, if not provided they will not be used
include_advflags : boolean
True if advisory flags are available
- class pytesmo.io.sat.ascat.Ascat_SSM(*args, **kwargs)[source]¶
Bases: pytesmo.io.sat.ascat.Ascat_data
class for reading ASCAT SSM data. It extends Ascat_data and provides the information necessary for reading SSM data
Parameters: path : string
path to data folder which contains the zip files from the FTP server
grid_path : string
path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info txt file in grid_path
advisory_flags_path : string, optional
path to advisory flags .dat files, if not provided they will not be used
topo_threshold : int, optional
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int, optional
if wetland fraction of read grid point is above this threshold a warning is output during reading
Attributes
gp_filename_template (string) defines how the gpi is put into the template string to make the filename gp_filestruct (numpy.dtype) structure template of the SSM .dat file scale_factor (dict) factor by which to multiply the raw data to get the correct values for each field in the gp_filestruct include_in_df (list) list of fields that should be returned to the user after reading nan_values (dict) nan value saved in the file which will be replaced by numpy.nan values during reading datatype (dict) datatype of the fields that the return data should have Methods
read_ssm(*args,**kwargs) read surface soil moisture - read_ssm(*args, **kwargs)[source]¶
function to read SSM takes either 1 or 2 arguments. It can be called as read_ssm(gpi,**kwargs) or read_ssm(lon,lat,**kwargs)
Parameters: gpi : int
grid point index
lon : float
longitude of point
lat : float
latitude of point
mask_ssf : boolean, optional
default False, if True only SSF values of 1 will be allowed, all others are removed
mask_frozen_prob : int,optional
if included in kwargs then all observations taken when frozen probability > mask_frozen_prob are removed from the result
mask_snow_prob : int,optional
if included in kwargs then all observations taken when snow probability > mask_snow_prob are removed from the result
Returns: ASCATTimeSeries : object
pytesmo.io.sat.ascat.ASCATTimeSeries instance
- class pytesmo.io.sat.ascat.Ascat_SWI(*args, **kwargs)[source]¶
Bases: pytesmo.io.sat.ascat.Ascat_data
class for reading ASCAT SWI data. It extends Ascat_data and provides the information necessary for reading SWI data
Parameters: path : string
path to data folder which contains the zip files from the FTP server
grid_path : string
path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info txt file in grid_path
advisory_flags_path : string, optional
path to advisory flags .dat files, if not provided they will not be used
topo_threshold : int, optional
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int, optional
if wetland fraction of read grid point is above this threshold a warning is output during reading
Attributes
gp_filename_template (string) defines how the gpi is put into the template string to make the filename gp_filestruct (numpy.dtype) structure template of the SSM .dat file scale_factor (dict) factor by which to multiply the raw data to get the correct values for each field in the gp_filestruct include_in_df (list) list of fields that should be returned to the user after reading nan_values (dict) nan value saved in the file which will be replaced by numpy.nan values during reading datatype (dict) datatype of the fields that the return data should have T_SWI (dict) information about which numerical T-Value maps to which entry in the datastructure T_QFLAG (dict) information about which numerical T-Value maps to which entry in the datastructure Methods
read_swi(*args,**kwargs) read soil water index - read_swi(*args, **kwargs)[source]¶
function to read SWI takes either 1 or 2 arguments being. It can be called as read_swi(gpi,**kwargs) or read_swi(lon,lat,**kwargs)
Parameters: gpi : int
grid point index
lon : float
longitude of point
lat : float
latitude of point
T : int, optional
if set only the SWI and QFLAG of this T-Value will be returned
mask_qf : int, optional
if set, SWI values with a QFLAG value lower than the mask_qf value will be masked. This is done for each T value independently
mask_frozen_prob : int,optional
if included in kwargs then all observations taken when frozen probability > mask_frozen_prob are removed from the result
mask_snow_prob : int,optional
if included in kwargs then all observations taken when snow probability > mask_snow_prob are removed from the result
Returns: df : pandas.DataFrame
containing all fields in self.include_in_df plus frozen_prob and snow_prob if advisory_flags_path was set. If T was set then only SWI and QFLAG values for the selected T value are included plut frozen_prob and snow_prob if applicable
- class pytesmo.io.sat.ascat.Ascat_data(path, grid_path, grid_info_filename='TUW_W54_01_lonlat-ld-land.txt', advisory_flags_path=None, topo_threshold=50, wetland_threshold=50)[source]¶
Bases: object
Class that provides access to ASCAT data stored in userformat which is downloadable from the TU Wien FTP Server after registration at http://rs.geo.tuwien.ac.at .
Parameters: path : string
path to data folder which contains the zip files from the FTP server
grid_path : string
path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info txt file in grid_path
advisory_flags_path : string, optional
path to advisory flags .dat files, if not provided they will not be used
topo_threshold : int, optional
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int, optional
if wetland fraction of read grid point is above this threshold a warning is output during reading
Attributes
path (string) path to data folder which contains the zip files from the FTP server grid_path (string) path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell grid_info_filename (string) name of the grid info txt file in grid_path grid_info_np_filename (string) name of the numpy save file to the grid information topo_threshold (int) if topographic complexity of read grid point is above this threshold a warning is output during reading wetland_threshold (int) if wetland fraction of read grid point is above this threshold a warning is output during reading grid_info_loaded (boolean) true if the grid information has already been loaded grid (pytesmo.grid.grids.CellGrid object) CellGrid object, which provides nearest neighbor search and other features advisory_flags_path (string) path to advisory flags .dat files, if not provided they will not be used include_advflags (boolean) True if advisory flags are available Methods
unzip_cell(cell) unzips zipped grid point files into subdirectory read_advisory_flags(gpi) reads the advisory flags for a given grid point index - read_advisory_flags(gpi)[source]¶
Read the advisory flags located in the self.advisory_flags_path Advisory flags include frozen probability, snow cover probability topographic complexity and wetland fraction.
Parameters: gpi : long
grid point index
Returns: df : pandas.DataFrame
containing the columns frozen_prob and snow_prob. lenght 366 with one entry for every day of the year, including February 29th
topo : numpy.uint8
topographic complexity ranging from 0-100
wetland : numpy.uint8
wetland fraction of pixel in percent
pytesmo.io.sat.ers module¶
Created on Oct 22, 2013
@author: Christoph Paulik christoph.paulik@geo.tuwien.ac.at
- class pytesmo.io.sat.ers.ERSTimeSeries(gpi, lon, lat, cell, data, topo_complex=None, wetland_frac=None, porosity_gldas=None, porosity_hwsd=None)[source]¶
Bases: pytesmo.io.sat.ascat.ASCATTimeSeries
Extends pytesmo.io.sat.ascat.ASCATTimeSeries and provides correct string representation for ERS data
Methods
plot(*args, **kwargs) wrapper for pandas.DataFrame.plot which adds title to plot
- class pytesmo.io.sat.ers.ERS_SSM(path, grid_path, grid_info_filename='TUW_WARP5_grid_info_2_1.nc', topo_threshold=50, wetland_threshold=50, netcdftemplate='TUW_ERS_AMI_SSM_WARP55R11_%04d.nc', include_in_df=['sm', 'sm_noise', 'proc_flag', 'orbit_dir'])[source]¶
Bases: pytesmo.io.sat.ascat.AscatNetcdf
class for reading ERS SSM data. It extends pytesmo.io.sat.ascat.AscatNetcdf instance and provides the information necessary for reading SSM data
Parameters: path : string
path to data folder which contains the netCDF files from the FTP server
grid_path : string
path to grid_info folder which contains txt files with information about grid point index,latitude, longitude and cell
grid_info_filename : string, optional
name of the grid info netCDF file in grid_path default ‘TUW_WARP5_grid_info_2_1.nc’
advisory_flags_path : string, optional
path to advisory flags .dat files, if not provided they will not be used
topo_threshold : int, optional
if topographic complexity of read grid point is above this threshold a warning is output during reading
wetland_threshold : int, optional
if wetland fraction of read grid point is above this threshold a warning is output during reading
netcdftemplate : string, optional
string template for the netCDF filename. This specifies where the cell number is in the netCDF filename. Standard value is ‘TUW_ERS_AMI_SSM_WARP55R11_%04d.nc’ in which %04d will be substituded for the cell number during reading of the data
include_in_df : list, optional
list of variables which should be included in the returned DataFrame. Default is all variables [‘sm’, ‘sm_noise’, ‘proc_flag’, ‘orbit_dir’]
Attributes
include_in_df (list) list of variables in the netcdf file that should be returned to the user after reading Methods
read_ssm(*args,**kwargs) read surface soil moisture - read_ssm(*args, **kwargs)[source]¶
function to read SSM takes either 1 or 2 arguments. It can be called as read_ssm(gpi,**kwargs) or read_ssm(lon,lat,**kwargs)
Parameters: gpi : int
grid point index
lon : float
longitude of point
lat : float
latitude of point
mask_frozen_prob : int,optional
if included in kwargs then all observations taken when frozen probability > mask_frozen_prob are removed from the result
mask_snow_prob : int,optional
if included in kwargs then all observations taken when snow probability > mask_snow_prob are removed from the result
absolute_values : boolean, optional
if True soil porosities from HWSD and GLDAS will be used to derive absolute values which will be available in the pandas.DataFrame in the columns ‘sm_por_gldas’,’sm_noise_por_gldas’, ‘sm_por_hwsd’,’sm_noise_por_hwsd’
Returns: ERSTimeSeries : object
pytesmo.io.sat.ers.ERSTimeSeries instance
pytesmo.io.sat.h_saf module¶
Created on May 21, 2014
@author: Christoph Paulik christoph.paulik@geo.tuwien.ac.at
- class pytesmo.io.sat.h_saf.H07img(path, month_path_str='h07_%Y%m_buf', day_search_str='h07_%Y%m%d_*.buf', file_search_str='h07_%Y%m%d_%H%M%S*.buf', filename_datetime_format=(4, 19, '%Y%m%d_%H%M%S'))[source]¶
Bases: pytesmo.io.dataset_base.DatasetImgBase
Class for reading HSAF H07 SM OBS 1 images in bufr format. The images have the same structure as the ASCAT 3 minute pdu files and these 2 readers could be merged in the future The images have to be uncompressed in the following folder structure path -
month_path_str (default ‘h07_%Y%m_buf’)For example if path is set to /home/user/hsaf07 and month_path_str is left to the default ‘h07_%Y%m_buf’ then the images for March 2012 have to be in the folder /home/user/hsaf07/h07_201203_buf/
Parameters: path: string
path where the data is stored
month_path_str: string, optional
if the files are stored in folders by month as is the standard on the HSAF FTP Server then please specify the string that should be used in datetime.datetime.strftime Default: ‘h07_%Y%m_buf’
day_search_str: string, optional
to provide an iterator over all images of a day the method _get_possible_timestamps looks for all available images on a day on the harddisk. This string is used in datetime.datetime.strftime and in glob.glob to search for all files on a day. Default : ‘h07_%Y%m%d_*.buf’
file_search_str: string, optional
this string is used in datetime.datetime.strftime and glob.glob to find a 3 minute bufr file by the exact date. Default: ‘h07_%Y%m%d_%H%M%S*.buf’
Methods
daily_images(day, **kwargs) Yield all images for a day. iter_images(start_date, end_date, **kwargs) Yield all images for a given date range. read_img(timestamp, **kwargs) Return an image if a specific datetime is given. tstamps_for_daterange(startdate, enddate) Get the timestamps as datetime array that are possible for the - tstamps_for_daterange(startdate, enddate)[source]¶
Get the timestamps as datetime array that are possible for the given day, if the timestamps are
For this product it is not fixed but has to be looked up from the hard disk since bufr files are not regular spaced and only europe is in this product. For a global product a 3 minute spacing could be used as a fist approximation
Parameters: start_date : datetime.date or datetime.datetime
start date
end_date : datetime.date or datetime.datetime
end date
Returns: dates : list
list of datetimes
- class pytesmo.io.sat.h_saf.H08img(path, month_path_str='h08_%Y%m_buf', day_search_str='h08_%Y%m%d_*.buf', file_search_str='h08_%Y%m%d_%H%M%S*.buf', filename_datetime_format=(4, 19, '%Y%m%d_%H%M%S'))[source]¶
Bases: pytesmo.io.dataset_base.DatasetImgBase
Reads HSAF H08 images. The images have to be uncompressed in the following folder structure path -
month_path_str (default ‘h08_%Y%m_buf’)For example if path is set to /home/user/hsaf08 and month_path_str is left to the default ‘h08_%Y%m_buf’ then the images for March 2012 have to be in the folder /home/user/hsaf08/h08_201203_buf/
Parameters: path: string
path where the data is stored
month_path_str: string, optional
if the files are stored in folders by month as is the standard on the HSAF FTP Server then please specify the string that should be used in datetime.datetime.strftime Default: ‘h08_%Y%m_buf’
day_search_str: string, optional
to provide an iterator over all images of a day the method _get_possible_timestamps looks for all available images on a day on the harddisk. This string is used in datetime.datetime.strftime and in glob.glob to search for all files on a day. Default : ‘h08_%Y%m%d_*.buf’
file_search_str: string, optional
this string is used in datetime.datetime.strftime and glob.glob to find a 3 minute bufr file by the exact date. Default: ‘h08_%Y%m%d_%H%M%S*.buf’
Methods
daily_images(day, **kwargs) Yield all images for a day. iter_images(start_date, end_date, **kwargs) Yield all images for a given date range. read_img(timestamp, **kwargs) Return an image if a specific datetime is given. tstamps_for_daterange(startdate, enddate) Get the timestamps as datetime array that are possible for the - tstamps_for_daterange(startdate, enddate)[source]¶
Get the timestamps as datetime array that are possible for the given day, if the timestamps are
For this product it is not fixed but has to be looked up from the hard disk since bufr files are not regular spaced and only europe is in this product. For a global product a 3 minute spacing could be used as a fist approximation
Parameters: start_date : datetime.date or datetime.datetime
start date
end_date : datetime.date or datetime.datetime
end date
Returns: dates : list
list of datetimes
- class pytesmo.io.sat.h_saf.H14img(path, month_path_str='h14_%Y%m_grib', file_str='H14_%Y%m%d00.grib', expand_grid=True)[source]¶
Bases: pytesmo.io.dataset_base.DatasetImgBase
Class for reading HSAF H14 SM DAS 2 products in grib format The images have to be uncompressed in the following folder structure path -
month_path_str (default ‘h14_%Y%m_grib’)For example if path is set to /home/user/hsaf14 and month_path_str is left to the default ‘h14_%Y%m_grib’ then the images for March 2012 have to be in the folder /home/user/hsaf14/h14_201203_grib/
Parameters: path: string
path where the data is stored
month_path_str: string, optional
if the files are stored in folders by month as is the standard on the HSAF FTP Server then please specify the string that should be used in datetime.datetime.strftime Default: ‘h14_%Y%m_grib’
file_str: string, optional
this string is used in datetime.datetime.strftime to get the filename of a H14 daily grib file Default: ‘H14_%Y%m%d00.grib’
expand_grid : boolean, optional
if set the images will be expanded to a 2D image during reading if false the images will be returned as 1D arrays on the reduced gaussian grid Default: True
Methods
daily_images(day, **kwargs) Yield all images for a day. iter_images(start_date, end_date, **kwargs) Yield all images for a given date range. read_img(timestamp, **kwargs) Return an image if a specific datetime is given. tstamps_for_daterange(start_date, end_date) Return all valid timestamps in a given date range.