Common submodule

This module provides routines that are used throughout the library.

rainforest.common.io_data : read various MCH data formats

rainforest.common.retrieve_data : retrieve MCH files (COSMO, polar, Cartesian) from storage

rainforest.common.radarprocessing : radar processing routines

rainforest.common.constants : define constants used throughout the code

rainforest.common.graphics : creating plots for QPE realization and evaluation

rainforest.common.lookup : computing and reading lookup tables that are used for interpolation and geolocalization of data

rainforest.common.lookup : various small utilities used throughout the code

rainforest.common.io_data module

Functions to read MeteoSwiss products

Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019

rainforest.common.io_data.read_cart(cart_file)

Generic function that reads a Cartesian radar file, either in gif or binary format (ELDES or as generated by the qpe module of this library) and converts its data to floating numbers

Parameters

cart_file (str) – Full path of the file to read as a a string

Returns

Return type

The cartesian data in a numpy array

rainforest.common.io_data.read_gif(gif_file)

Reads a Cartesian radar file in gif format

Parameters

gif_file (str) – Full path of the file to read as a a string

Returns

Return type

The cartesian data in a numpy array

rainforest.common.io_data.read_polar(polar_files, physic_value=True)

Reads a polar radar metranet file as a pyart radar instance, if multiple files corresponding to multiple elevations (sweeps) are provided they are merged into a single pyart instance

Parameters

polar_files (str or list of strings) –

Can be either: - Full path of the file to read as a a string - string with wildcard pointing to the files to read,

e.g. …../MLA192711055.*

  • a list of full filepaths

Returns

  • sweepnumbers (list) – list of sweeps numbers corresponding to all files that were read

  • merged (a pyart Radar instance)

rainforest.common.io_data.read_station_data(gauge_file)

Reads gauge data as given by the Climap software

Parameters

gauge_file (str) – Full path of the file to read as a a string

Returns

Return type

The cartesian data in a numpy array

rainforest.common.io_data.read_status(status_file, add_wet_radome=False)

Reads a radar xml status file

Parameters
  • fname (str) – Full path of the status xml file to be read

  • add_wet_Radome (boolean (optional)) – For older files, there is not information about the wet radome. If this is true, the script will estimate the wet radome precipitation as a 3 x 3 mean of the RZC product at the given time (as is done for more recent files)

Returns

Return type

The status as a python dict

rainforest.common.io_data.read_vpr(xml_file, radar=None)

Reads a vpr xml file and returns an interpolator that can be used to obtain the vpr correction at any altitude

Parameters

vpr_file (str) –

Full path of the file to read radar : char (optional)

the radar is required to know the vpr reference height, it needs to be either ‘A’,’D’,’L’,’P’ or ‘W’, if not specified, it will be inferred from the vpr filename

Returns

  • An interpolator that returns the vpr correction as a function of

  • the height, the correction is a multiplicative factor

rainforest.common.io_data.read_xls(xls_file)

Reads an excel file such as those used for CPC vlaidation

Parameters

fname (str) – Full path of the excel file to be read

Returns

Return type

The excel file as a Pandas dataframe

rainforest.common.io_data.save_gif(gif_file, precip)

Saves a precipitation map in float to a gif file (AQC format)

Parameters
  • gif_file (str) – Full path of the file to write as a a string

  • precip (2D array) – 2D array containing the precipitation intensities

rainforest.common.retrieve_data module

Functions to retrieve MeteoSwiss products from the archives

Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019

rainforest.common.retrieve_data.get_COSMO_T(time, sweeps=None, radar=None)

Retrieves COSMO temperature data from the CSCS repository, and interpolates them to the radar gates, using precomputed lookup tables

Parameters
  • time (datetime.datetime instance) – the time at which to get the COSMO data in datetime format

  • sweeps (list of integers) – specify which sweeps (elevations) need to be retrieved in the form of a list, if not specified, all 20 will be retrieved

  • radar (list of chars) – list of radars for which to retrieve COSMO data, if not specified all 5 radars will be used (‘A’,’L’,’D’,’W’,’P’)

Returns

T_at_radar – A dict containing the temperature at the radar gates, in the following form: dict[radar][‘T’][sweep_number]

Return type

dict

rainforest.common.retrieve_data.get_COSMO_variables(time, variables, sweeps=None, radar=None, tmp_folder='/tmp/', cleanup=True)

Retrieves COSMO data from the CSCS repository, and interpolates them to the radar gates, using precomputed lookup tables This is a more generic but much slower function than the previous one, as it reads all COSMO variables directly from the GRIB files

Parameters
  • time (datetime.datetime instance) – the time at which to get the COSMO data in datetime format

  • variables (list of strings) – List of COSMO variables to retrieve, ex. P, T, QV, QR, RH, etc…

  • sweeps (list of integers (optional)) – specify which sweeps (elevations) need to be retrieved in the form of a list, if not specified, all 20 will be retrieved

  • = list of chars (optional) (radar) – list of radars for which to retrieve COSMO data, if not specified all 5 radars will be used (‘A’,’L’,’D’,’W’,’P’)

  • = str (optional) (tmp_folder) – Directory where to store the extracted files

  • = boolean (optional) (cleanup) – If true all extracted files will be deleted before returning the output (recommended)

Returns

  • A dict containing the COSMO variables at the radar gates, in the following

  • form (dict[radar][variables][sweep_number])

rainforest.common.retrieve_data.retrieve_CPCCV(time, stations)

Retrieves cross-validation CPC data for a set of stations from the xls files prepared by Yanni

Parameters
  • time (datetime.datetime instance) – starting time of the time range

  • stations (list of str) – list of weather stations at which to retrieve the CPC.CV data

Returns

  • A numpy array corresponding at the CPC.CV estimations at every specified

  • station

rainforest.common.retrieve_data.retrieve_prod(folder_out, start_time, end_time, product_name, pattern=None, pattern_type='shell', sweeps=None)

Retrieves radar data from the CSCS repository for a specified time range, unzips them and places them in a specified folder

Parameters
  • folder_out (str) – directory where to store the unzipped files

  • start_time (datetime.datetime instance) – starting time of the time range

  • end_time (datetime.datetime instance) – end time of the time range

  • product_name (str) – name of the product, as stored on CSCS, e.g. RZC, CPCH, MZC, BZC…

  • pattern (str) – pattern constraint on file names, can be used for products which contain multiple filetypes, f.ex CPCH folders contain both rda and gif files, if only gifs are wanted : file_type = ‘*.gif’

  • pattern_type (either 'shell' or 'regex' (optional)) – use ‘shell’ for standard shell patterns, which use * as wildcard use ‘regex’ for more advanced regex patterns

  • sweeps (list of int (optional)) – For polar products, specifies which sweeps (elevations) must be retrieved, if not specified all available sweeps will be retrieved

Returns

Return type

A list containing all the filepaths of the retrieved files

rainforest.common.radarprocessing module

Created on Mon Dec 16 17:38:52 2019

@author: wolfensb

class rainforest.common.radarprocessing.Radar(radname, polfiles, statusfile=None, vprfile=None)

Bases: object

A class that contains polar radar data and performs some pre-processing before adding that data to the database or computing the QPE product The different elevations are stored in a dictionary rather as in a single pyart radar instance as this was found to be faster in practice

Creates an Radar class instance

Parameters
  • radname (char) – Name of the radar, either ‘A’,’D’,’L’,’P’ or ‘W’

  • polfiles (list of str) – List of full filepaths of the radar files for a given timestemp, one file for every elevation, typically obtained with the retrieve_prod function in the common submodule

  • statusfile (str (optional)) – Full path of the status file that corresponds to this particular radar and timestep, used to compute noise estimates

  • vprfile (str(optional)) – Full path of the vpr xml file that corresponds to this particular radar and timestep, used to compute VPR correction

add_cosmo_data(cosmo_data)

Adds COSMO data to the radar instance

Parameters

cosmo_data (dict) –

dict of COSMO data at polar coordinates obtained from the get_COSMO_variables function in the common submodule this dict must have the following format

dic[variable][sweep]

compute_hydro()

Computes the hydrometeor classification using Nikola Besic’ algorithm, all necessary fields ZH, ZDR, RHOHV, KDP, T (COSMO) must be available

compute_kdp(dscfg)

Computes KDP using the simple moving least-square algorithm

Parameters

dscfg (dict) – dictionary containing the following fields RMIN: RMAX: RWIND: ZMIN: ZMAX:

compute_noise()

Computes a noise estimate from a status file

correct_attenuation()

Corrects for attenuation using the ZPHI algorithm (Testud et al.) using the COSMO temperature to identify liquid precipitation

get_field(sweep, field_name)

Gets a radar variable at given elevation (sweep)

Parameters
  • sweep (int) – Sweep number from 1 to 20

  • field_name (str) – name of the variable, e.g. ZH, ZDR, RHOHV, SW, …

snr_mask(snr_threshold)

Masks the radar data at low SNR

Parameters

snr_threshold (float) – Minimal SNR to consider in dB

visib_mask(min_visib, max_visib_corr)

Masks the radar data at low visibility and corrects the reflectivity for visibility

Parameters
  • min_visib (int) – Minimal visibility below which the data is masked

  • max_visib_corr (float) – Maximum visibility correction factor, the visibility correction is 100/VISIB (with VISIB in %) and can be thresholded with this parameter. This is usually set to 2 at MeteoSwiss

rainforest.common.radarprocessing.hydroClass_single(radars, zh, zdr, kdp, rhohv, temp, weights=array([1., 1., 1., 0.75, 0.5 ]))

Computes the hydrometeor classes for columnar data, note that all input arrays except weights must have the same length

Parameters
  • radars (ndarray of char) – Array of radar IDs, (‘A’,’D’,’L’,’P’,’W’)

  • zh (ndarray) – Array of radar reflectivity in dBZ

  • zdr (ndarray) – Array of diff. reflectivity in dB

  • kdp (ndarray) – Array of specific phase shift on propagation in deg / km

  • rhohv (ndarray) – Array of copolar correlation coefficient

  • temp (ndarray) – Array of radar temperature in Celcius

  • weights (ndarray (optional)) – The weight of every input feature, zh, zdr, kdp, rhohv, temp in the hydrometeor classification

Returns

  • The hydrometeor classes as ndarray with values from 0 to 8, corresponding to

  • the classes – 0 : no data 1 : aggregates (AG) 2 : light rain (LR) 3 : ice crystals (CR) 4 : rimed particles (RP) 5 : rain (RN) 6 : vertically aligned ice (VI) 7 : wet snow (WS) 8 : melting hail (MH) 9: dry hail / high density graupel (IH/HDG)

rainforest.common.constants module

Set of constants regarding MeteoSwiss radars and QPE

rainforest.common.constants.MODE(x)

rainforest.common.graphics module

Set of functions to display QPE precipitation data and verification scores

class rainforest.common.graphics.MidpointNormalize(vmin=None, vmax=None, transition=None, clip=False)

Bases: matplotlib.colors.Normalize

Normalizing that is linear up to a certain transition value, logarithmic afterwards

If vmin or vmax is not given, they are initialized from the minimum and maximum value respectively of the first input processed. That is, __call__(A) calls autoscale_None(A). If clip is True and the given value falls outside the range, the returned value will be 0 or 1, whichever is closer. Returns 0 if:

vmin==vmax

Works with scalars or arrays, including masked arrays. If clip is True, masked values are set to 1; otherwise they remain masked. Clipping silently defeats the purpose of setting the over, under, and masked colors in the colormap, so it is likely to lead to surprises; therefore the default is clip = False.

class rainforest.common.graphics.QPE_cmap

Bases: matplotlib.colors.LinearSegmentedColormap

Colormap that uses purple tones for small values, and then a transition from blue to red for values above the transition

Create color map from linear mapping segments

segmentdata argument is a dictionary with a red, green and blue entries. Each entry should be a list of x, y0, y1 tuples, forming rows in a table. Entries for alpha are optional.

Example: suppose you want red to increase from 0 to 1 over the bottom half, green to do the same over the middle half, and blue over the top half. Then you would use:

cdict = {'red':   [(0.0,  0.0, 0.0),
                   (0.5,  1.0, 1.0),
                   (1.0,  1.0, 1.0)],

         'green': [(0.0,  0.0, 0.0),
                   (0.25, 0.0, 0.0),
                   (0.75, 1.0, 1.0),
                   (1.0,  1.0, 1.0)],

         'blue':  [(0.0,  0.0, 0.0),
                   (0.5,  0.0, 0.0),
                   (1.0,  1.0, 1.0)]}

Each row in the table for a given color is a sequence of x, y0, y1 tuples. In each sequence, x must increase monotonically from 0 to 1. For any input value z falling between x[i] and x[i+1], the output value of a given color will be linearly interpolated between y1[i] and y0[i+1]:

row i:   x  y0  y1
               /
              /
row i+1: x  y0  y1

Hence y0 in the first row and y1 in the last row are never used.

See also

LinearSegmentedColormap.from_list

Static method; factory function for generating a smoothly-varying LinearSegmentedColormap.

makeMappingArray

For information about making a mapping array.

rainforest.common.graphics.plot_crossval_stats(stats, output_folder)

Plots the results of a crossvalidation intercomparion as performed in the rf.py module

Parameters
  • stats (dict) – dictionary containing the result statistics as obtained in the rf.py:model_intercomparison function

  • output_folder (str) – where to store the plots

rainforest.common.graphics.qpe_plot(data, subplots=None, figsize=None, vmin=0.04, vmax=120, transition=10, ch_border=True, xlim=None, ylim=None, cbar_orientation='horizontal', **kwargs)

Plots one or multiple QPE realizations using a special colormap, that shows a clear transition between low and high precipitation intensities, for low precipitation it is linear whereas for high precipitation it is logarithmic

If multiple QPE realizations are given, they will be displayed as subplots

Parameters
  • data (list of numpy arrays or numpy array) – the set of QPE realizations to display

  • subplots (2-element tuple (optional)) – Tuple indicating the number of subplots in each direction, the product of its elements must be equal to the number of QPE realizations If not provided, the default will be (1,n) where n is the number of realizations

  • figsize (2-element tuple (optional)) – Tuple indicating the size of the figure in inches in both directions (w,h)

  • vmin (float (optional)) – Minimum value of precipitation to display, values below will be blank

  • vmax (float (optional)) – Maximum value of precipitation to display, values below above will be shown with the color corresponding to vmax

  • transition (float (optional)) – The transition value from which to change colormap and switch from linear to logarithmic scale

  • ch_border (bool (optiona)) – Whether to overlay the shapefile of the Swiss borders

  • xlim (2 element tuple (optional)) – limits of the plots in the west-east direction (in Swiss coordinates)

  • ylim (2 element tuple (optional)) – limits of the plots in the south-north direction (in Swiss coordinates)

  • cbar_orientation (str (optional)) – colorbar orientation, either ‘horizontal’ or ‘vertical’

  • **kwargs – All additional arguments that can be passed to imshow

Returns

Return type

Nothing

rainforest.common.graphics.qpe_scatterplot(qpe_est, ref, title_prefix='', figsize=(10, 7.5))

Plots the results of multiple QPE models as a function of the reference gauge measurement

Parameters
  • qpe_est (dict of arrays) – Every value in the dictionary is a set of QPE estimates, every key is a model

  • ref (np.ndarray) – contains the reference observations (gauge), must have the same shape as any element in qpe_est

title_prefix: str (optional)

a prefix for the suptitle (global titl

figsize: 2-element tuple (optional)

Tuple indicating the size of the figure in inches in both directions (w,h)

Returns

Return type

Nothing

rainforest.common.graphics.score_plot(scores, title_prefix='', figsize=(10, 5))

Plots a series of QPE verification scores in the form of stacked barplots, for different ranges of precipitation

IMPORTANT: the scores dictionary must have the following structure

scores[model][precip_range][score]

for example

scores[‘RF_dualpol’][‘0.0-2.0’][‘ME’] = -0.27

you can get such a dictionary with the perfscores function in common.utils i.e. scores[‘RZC’] = perfscores(…)

Parameters
  • scores (dict of dict of dict of scores) – the set of scores to display

  • title_prefix (str (optional)) – a prefix for the suptitle (global title)

  • figsize (2-element tuple (optional)) – Tuple indicating the size of the figure in inches in both directions (w,h)

Returns

Return type

Nothing

rainforest.common.lookup module

Functions to read and compute lookup tables

The available lookup tables are

qpebias_station : dict with the bias correction of RZC at every station

cosmo1_to_rad : dict which maps COSMO grid to polar radar grid it has keys [sweep][coord_type], coord_type is ‘idx_0’ : first index of COSMO grids, ‘idx_1’: second, ‘idx_3’: third, mask’ is 1 for points that fall outside of COSMO domain. This lookup table is valid only for COSMO data stored in /store/s83/owm/COSMO-1/

cosmo2_to_rad : same thing but for COSMO 2 data

cosmo1T_to_rad : same thing but for netCDF files of COSMO1 temperature extracted for MDR and stored in /store/s83/owm/COSMO-1/ORDERS/MDR/

cosmo1T_to_rad : same thing but for netCDF files of COSMO2 temperature extracted for MDR and stored in /store/msrad/cosmo/cosmo2/data/

station_to_rad : maps the SMN stations to radar coordinates, it is an extraction of the more generic but less convenient qpegrid_to_rad table It is list of 3 elements, first element is a dict with keys [station][sweep][ncode] and gives the polar gates (azimuth_idx, range_idx) that fall within a Cartesian pixel at a given radar elevation and for a given station neighbour (00 = station location, -1-1 = one to the south-west, 22 = two km to north and 2 km to east) second element is a dict giving distance from every station to the radar third element is a dict of keys [station][sweep] giving the height above ground of the radar observations above that station at a given elevation (sweep) MISSING KEYS IMPLY NO RADAR VISIBILITY

cartcoords_rad : gives the Cartesian (Swiss LV03) coordinates of all polar gates. It is a dict that gives for every sweep a 3D array of shape ncoords x nazimuth x nrange, ncoords is 3, first slice is Swiss Y coordinate (west to east), second is Swiss X-coordinate (south to north) and last is Swiss Z coordinate (altitude)

qpegrid_to_rad : maps the radar polar data to any gridpoint of the Swiss QPE grid. It is simply a 2D array with 5 columns | sweep | azimuth_idx | range_idx | Swiss Y coord | Swiss X coord|

station_to_qpegrid : maps every station to the corresponding QPE gridpoint it is a dict of keys [station][ncode] and gives the index of every neighbour of every station in the QPE 2D grid 640 x 710 pixels

visibility_rad : gives the (static) visibility of every polar gate for a given sweep number in the form of a 2D field of size nazimuth x nrange

rainforest.common.lookup.calc_lookup(lookup_type, radar=None)

Calculates a lookup table and stores it in the /data/lookup_data folder

Parameters
  • lookup_type (str) –

    the lookup table type, must be one of

    • qpebias_station

    • cosmo1_to_rad

    • cosmo2_to_rad

    • cosmo1T_to_rad

    • cosmo2T_to_rad

    • station_to_rad

    • cartcoords_rad

    • qpegrid_to_rad

    • station_to_qpegrid

    • visibility_rad

  • radar (char or list of chars (optional)) – the radar for which to retrieve the lookup table, needed only if the lookup_type contains the term ‘rad’, must be either ‘A’, ‘D’, ‘L’, ‘W’ or ‘P’

rainforest.common.lookup.get_lookup(lookup_type, radar=None)

Read a lookup table from the /data/lookup_data folder

Parameters
  • lookup_type (str) –

    the lookup table type, must be one of

    • qpebias_station

    • cosmo1_to_rad

    • cosmo2_to_rad

    • cosmo1T_to_rad

    • cosmo2T_to_rad

    • station_to_rad

    • cartcoords_rad

    • qpegrid_to_rad

    • station_to_qpegrid

    • visibility_rad

  • radar (char or list of chars (optional)) – the radar for which to retrieve the lookup table, needed only if the lookup_type contains the term ‘rad’, must be either ‘A’, ‘D’, ‘L’, ‘W’ or ‘P’

Returns

lut – The lookup table in the form of a python dict

Return type

dict

rainforest.common.utils module

Set of functions that can be useful

Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019

rainforest.common.utils.LV03toWGS84(east, north, heights)

Converts a set of WGS84, lat/lon/heights to Swiss CH1903 coordinates

Parameters
  • east (ndarray) – Easterly Swiss coordinates (CHY)

  • north (ndarray) – northerly Swiss coordinates (CHX)

  • heights (ndarray) – heights a.s.l in WGS84 coordinates

Returns

Return type

lat, lon and height coordinates in WGS84

rainforest.common.utils.aggregate_multi(array_3d, agg_operators)

Aggregates a 3D numpy array alongs its first axis, using different aggregation operators

Parameters
  • array_3d (ndarray) – 3D numpy array, of shape (N x M x L)

  • agg_operators (ndarray of integers) – array of aggregation operators as defined in the constants.py file, must have the same length as the first dimension of array_3D (N)

Returns

Return type

An aggregated array of size M x L

rainforest.common.utils.chunks(l, n)

Divides a list l into n sublists of similar sizes

rainforest.common.utils.dict_flatten(mydict)

Flattens a nested dictionary

rainforest.common.utils.envyaml(filename)

Reads a yaml configuration file while parsing environment variables. Environment variables must be written as ${NAME_OF_VAR} in the yaml file

Parameters

filename (str) – path of the input yaml file

Returns

the yaml content in the form of a python dict

Return type

dict

rainforest.common.utils.get_qpe_files(input_folder, t0=None, t1=None, time_agg=None, list_models=None)

Gets the list of all qpe files in a folder (as saved by qpe_compute) and separates them by qpe type and timestep

Parameters
  • input_folder (str) – main directory where the qpe files are saved, it contains one subfolder for every qpe model (type) that was used

  • t0 (datetime (optional)) – Starting time of the period to retrieve, will be used to filter files, if not provided starting time will be time of first file

  • t1 (datetime (optional)) – End time of the period to retrieve, will be used to filter files, if not provided end time will be time of last file

  • time_agg (minutes (optional)) – Will aggregate all files to a reference time in minutes (e.g. use 10 to put together all files that correspond to a single gauge measurement)

  • list_models ((optional)) – List of qpe types to retrieve , if not provided all folders in input_folder will be used

Returns

  • A dictionary where every key is a QPE model and every value is a list

  • with all files in chronological order

rainforest.common.utils.hex_to_rgb(value)
rainforest.common.utils.idx_cart(x, y)

Returns the Cartesian index of a set of coordinates x and y

rainforest.common.utils.nanadd_at(a, indices, b)

Replaces nans by zero in call to np.add.at

rainforest.common.utils.nearest_time(dt, reference)

Gets the nearest earlier reference timestep to a given datetime, for ex. if dt = 1 Jan 2020 10:12, and reference is 10 it will return

1 Jan 2020 10:10, or if dt = 1 Jan 2020 10:59 and reference is 60 it will return 1 Jan 2020 10:00

Parameters
  • dt (datetime) – The datetime to check

  • reference (int) – The reference timestep in minutes

Returns

Return type

The closest earlier timestep in datetime format

rainforest.common.utils.nested_dict_gen(d)

The generator for the previous function

rainforest.common.utils.nested_dict_values(d)

Extracts all values from a nested dictionary

rainforest.common.utils.perfscores(est_data, ref_data, bounds=None, array=False)

Computes a set of precipitation performance scores, on different data ranges. The scores are

  • scatter: 0.5 * (Qw84(x) - Qw16(x)), where Qw is a quantile weighted by ref_data / sum(ref_data) and x is est_data / ref_data in dB scale

  • RMSE: root mean square error (linear error)

  • bias: (ME/mean(ref_data) + 1) in dB

  • ED: the energy distance which is a measure of the distance between two distributions (https://en.wikipedia.org/wiki/Energy_distance)

Parameters
  • est_data (ndarray) – array of estimates (ex. precip from QPE)

  • ref_data (ndarray) – array of reference (ex. precip from gauge)

  • bounds (list (optional)) – list of bounds on ref_data for which to compute the error metrics, by default all data will be used (unbounded), note that even if you prescribe bounds the scores for the overall data will always be added in the output

  • array (boolean (optional)) – Whether or not to convert the output dict to a numpy array

Returns

all_metrics – a dictionary containing all the scores, organized in the following way all_metrics[bound][score]

Return type

dict or ndarray

rainforest.common.utils.quantile(data, weights, quantile)

Weighted quantile of an array with respect to the last axis.

Parameters
  • data (ndarray) – Input array.

  • weights (ndarray) – Array with the weights. It must have the same size of the last axis of data.

  • quantile (float) – Quantile to compute. It must have a value between 0 and 1.

Returns

quantile – The output value.

Return type

float

rainforest.common.utils.quantile_1D(data, weights, quantile)

Compute the weighted quantile of a 1D numpy array.

Parameters
  • data (ndarray) – Input array (one dimension).

  • weights (ndarray) – Array with the weights of the same size of data.

  • quantile (float) – Quantile to compute. It must have a value between 0 and 1.

Returns

quantile_1D – The output value.

Return type

float

rainforest.common.utils.read_df(pattern, dbsystem='dask', sqlContext=None)

Reads a set of data contained in a folder as a spark or dask DataFrame

Parameters
  • pattern (str) – Unix style wildcard pattern pointing to the files, for example /store/msrad/folder/*.csv will read all csv files in that folder

  • dbsystem (str) – Either “dask” if you want a Dask DataFrame or “spark” if you want a spark dataframe

  • sqlContext (sqlContext instance) – sqlContext to use, required only if dbystem = ‘spark’

Returns

Return type

A spark or dask DataFrame instance

rainforest.common.utils.read_task_file(task_file)

Reads a database processing task file

rainforest.common.utils.rename_fields(data)

Rename pyart fields from pyrad names to simpler names, according to the dictionary PYART_NAMES_MAPPING in the constants.py module

rainforest.common.utils.round_to_hour(dt)

Returns the sweep number of a polar file based on its name

rainforest.common.utils.split_by_time(files_rad)

Separate a list of files by their timestamp

rainforest.common.utils.stack_uneven(arrays, fill_value=nan)

Fits mmltiple into a single numpy array, even if they are different sizes, assigning a fill_value to parts with no data

Parameters
  • arrays (list of np arrays) – list of numpy array to stack, they can have different dimensions

  • fill_value ((float, optional)) – the fill value with which to fill the missing pixels

Returns

  • a np.ndarray with size N x M, where N is the sum of the number of

  • rows of all arrays and M is the maximal number of columns in all arrays

rainforest.common.utils.sweepnumber_fromfile(fname)

Returns the sweep number of a polar file based on its name

rainforest.common.utils.timefromfilename(fname)

Returns the datetime of a file based on its name

rainforest.common.utils.timestamp_from_datestr(datestr)
rainforest.common.utils.timestamp_from_datetime(dt)
rainforest.common.utils.wgs84toCH1903(lat, lon, heights)

Converts a set of WGS84, lat/lon/heights to Swiss CH1903 coordinates, east, north and height

Parameters
  • lat (ndarray) – latitudes in decimal format (degrees)

  • lon (ndarray) – longitudes in decimal format (degrees)

  • heights (ndarray) – heights a.s.l in WGS84 coordinates

Returns

Return type

east, north and height coordinates in CHLV190

rainforest.common.wgs84_ch1903 module

class rainforest.common.wgs84_ch1903.GPSConverter

Bases: object

GPS Converter class which is able to perform convertions between the CH1903 and WGS84 system.

CHtoWGSheight(y, x, h)
CHtoWGSlat(y, x)
CHtoWGSlng(y, x)
DecToSexAngle(dec)
LV03toWGS84(east, north, height)

Convert LV03 to WGS84 Return a array of double that contain lat, long, and height

SexAngleToSeconds(dms)
SexToDecAngle(dms)
WGS84toLV03(latitude, longitude, ellHeight)

Convert WGS84 to LV03 Return an array of double that contaign east, north, and height

WGStoCHh(lat, lng, h)
WGStoCHx(lat, lng)
WGStoCHy(lat, lng)