Remote access using python

The easiest way to use the Madrigal python remote data access API is to simply let the web interface write the script you need for you. Just choose the Access data pull-down menu and choose Create a command to download multiple exps. Then follow the instructions, and you will have the command you need to download whatever you want from Madrigal. Be sure to select python as the language you want to create the command with. You can choose to download files as they are in Madrigal in either column-delimited ascii, Hdf5, or netCDF4 formats, or you can choose the parameters yourself (including derived parameters), and optionally include filters on the data you get back.

This web interface will generate python commands using one of the following two Python scripts: globalDownload.py and globalIsprint.py. Use globalDownload.py if you want data as it is in Madrigal. Use globalIsprint.py to choose parameters and/or filters. These two scripts are documented below, for those who do not want to use the web interface to generate the needed arguments. There are also some plotting scripts that require matplotlib to be installed, one for scatter plots, the other for pcolor plots.

globalDownload.py globalIsprint.py
madrigalPColor.py madrigalScatter.py

globalDownload.py

 Usage:
 globalDownload.py --url=<Madrigal url> --outputDir=<output directory> \
   --user_fullname=<user fullname> --user_email=<user email> \
   --user_affiliation=<user affiliation> --format=<ascii,hdf5> [options]
 where:
 --url=<Madrigal url> - url to homepage of site to be searched
   (ie, http://madrigal.haystack.mit.edu/madrigal/)
   This is required.
 --outputDir=<output directory> - the output directory to store all files in.  Default is to store
   all files in the same directory, and a number is added to the filename if a file might be overwritten.  Set
   --tree flag to store all files in the same directory structure they appear in Madrigal.  This
   allows all files to keep their original names. 
 --user_fullname=<user fullname> - the full user name (probably in quotes unless your name is
   Sting or Madonna)
 --user_email=<user email>
 --user_affiliation=<user affiliation> - user affiliation.  Use quotes if it contains spaces.
   
   --format=<ascii or hdf5>
 and options are:
 --startDate=<MM/DD/YYYY> - start date to filter experiments before.  Defaults to allow all experiments.
 --endDate=<MM/DD/YYYY> - end date to filter experiments after.  Defaults to allow all experiments.
 --inst=<instrument list> - comma separated list of instrument codes or names.  See Madrigal documentation
   for this list.  Defaults to allow all instruments. If names are given, the
   argument must be enclosed in double quotes.  An asterick will perform matching as
   in glob.  For example:
   
   --inst=10,30
   
   --inst="Jicamarca IS Radar,Arecibo*"
 --expName  - filter experiments by the experiment name.  Give all or part of the experiment name. Matching
   is case insensitive.  Default is no filtering by experiment name.
   
   --fileDesc - filter files using input file Description string and case-insensitive fnmatch
 --kindat=<kind of data list> - comma separated list of kind of data codes.  See Madrigal documentation
   for this list.  Defaults to allow all kinds of data.  If names are given, the
   argument must be enclosed in double quotes.  An asterick will perform matching as
   in glob. For example:
   
   --kindat=3001,13201
   
   --kindat="INSCAL Basic Derived Parameters,*efwind*,2001"
 --seasonalStartDate=<MM/DD> - seasonal start date to filter experiments before.  Use this to select only part of the
   year to collect data.  Defaults to Jan 1.  Example: 
   
   --seasonalStartDate=07/01 would only allow experiments after July 1st from each year.
 
   --seasonalEndDate=<MM/DD> - seasonal end date to filter experiments after.  Use this to select only part of the
   year to collect data.  Defaults to Dec 31.  Example: 
   
   --seasonalEndDate=10/31 would only allow experiments before Oct 31 of each year.
   
   --tree - add if you want to store the downloaded files in the same hierarchy as in Madrigal: 
   <YYYY/<instCode>/<experimentDir>.  Without --tree, stores all downloaded files in one directory.
   
   --includeNonDefault - if given, include realtime files when there are no default.  Default is to search only default files.
 --verbose - if given, print each file processed info to stdout.  Default is to run silently.
   
   Example:
   
   globalDownload.py --url=http://madrigal.haystack.mit.edu/madrigal --outputDir=/tmp --user_fullname="Bill Rideout" 
   --user_email=brideout@haystack.mit.edu --user_affiliation=MIT --startDate=01/01/1998 
   --endDate=-01/30/1998 --inst=30

globalIsprint.py

 Usage:
 globalIsprint.py --url=<Madrigal url> --parms=<Madrigal parms> --output=<output file> \
   --user_fullname=<user fullname> --user_email=<user email> \
   --user_affiliation=<user affiliation> [options]
 where:
 --url=<Madrigal url> - url to homepage of site to be searched
   (ie, http://madrigal.haystack.mit.edu/madrigal/)
   This is required.
 --parms=<Madrigal parms> - a comma delimited string listing the desired Madrigal parameters
   in mnemonic form.  (Example: gdalt,dte,te).  Data will be returned
   in the same order as given in this string. See
   http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata and
   choose "Parameter code table" for all possible parameters
 --output=<output file or directory name> - the file or directory name to store the resulting data.
   If you give a file, all output will be stored in a single
   ascii file that you specify.
   Use a directory name if you want data stored as individual
   files, in either ascii, Hdf5, or netCDF4 formats.  To use
   this option, you must set a format in the optional format
   argument. File names will be based on file names in Madrigal.
   Hdf5 or netCDF4 formats only available from Madrigal 3.0 or 
   higher sites.
 --user_fullname=<user fullname> - the full user name (probably in quotes unless your name is
   Sting or Madonna)
 --user_email=<user email>
 --user_affiliation=<user affiliation> - user affiliation.  Use quotes if it contains spaces.
 and options are:
 --startDate=<MM/DD/YYY> - start date to filter experiments before.  Defaults to allow all experiments.
 --endDate=<MM/DD/YYY> - end date to filter experiments after.  Defaults to allow all experiments.
 --inst=<instrument list> - comma separated list of instrument codes or names.  See Madrigal documentation
   for this list.  Defaults to allow all instruments. If names are given, the
   argument must be enclosed in double quotes.  An asterick will perform matching as
   in glob.  Examples: (--inst=10,30 or --inst="Jicamarca IS Radar,Arecibo*")
   
   --format=<Hdf5 or netCDF4 or ascii> - format must be specified if output is a directory so that data is stored
   in individual files, one for each Madrigal file. Hdf5 or netCDF4 formats only 
   available from Madrigal 3.0 or higher sites.
 --expName  - filter experiments by the experiment name.  Give all or part of the experiment name. Matching
   is case insensitive and fnmatch characters * and ? are allowed.  Default is no filtering by 
   experiment name.
   
   --fileDesc - filter files by their file description string. Give all or part of the file description string. Matching
   is case insensitive and fnmatch characters * and ? are allowed.  Default is no filtering by 
   file description.
 --kindat=<kind of data list> - comma separated list of kind of data codes.  See Madrigal documentation
   for this list.  Defaults to allow all kinds of data.  If names are given, the
   argument must be enclosed in double quotes.  An asterick will perform matching as
   in glob. Examples: (--kindat=3001,13201 or 
   --kindat="INSCAL Basic Derived Parameters,*efwind*,2001")
 --filter=<[mnemonic] or [mnemonic1,[+-*/]mnemonic2]>,<lower limit1>,<upper limit1>[or<lower limit2>,<upper limit2>...]
   a filter using any measured or derived Madrigal parameter, or two Madrigal parameters either added,
   subtracted, multiplied or divided.  Each filter has one or more allowed ranges.  The filter accepts
   data that is in any allowed range.  If the Madrigal parameter value is missing, the filter will always
   reject that data.  Multiple filter arguments are allowed on the command line.  To skip either a lower
   limit or an upper limit, leave it blank.  Examples: (--filter=ti,500,1000  (Accept when 500 <= Ti <= 1000)
   or --filter=gdalt,-,sdwht,0,  (Accept when gdalt > shadowheight - that is, point in direct sunlight)
   or  --filter=gdalt,200,300or1000,1200 (Accept when 200 <= gdalt <= 300 OR 1000 <= gdalt <= 1200))
 --seasonalStartDate=<MM/DD> - seasonal start date to filter experiments before.  Use this to select only part of the
   year to collect data.  Defaults to Jan 1.  Example:
   (--seasonalStartDate=07/01) would only allow experiments after July 1st from each year.
 
   --seasonalEndDate=<MM/DD> - seasonal end date to filter experiments after.  Use this to select only part of the
   year to collect data.  Defaults to Dec 31.  Example: 
   (--seasonalEndDate=10/31) would only allow experiments before Oct 31 of each year.
 --showFiles - if given, show file names.  Default is to not show file names. Not used if format in <Hdf5, netCDF4).
 --showSummary - if given, summarize all arguments at the beginning.  Not used if format in <Hdf5, netCDF4).
   Default is to not show summary.
   
   --includeNonDefault - if given, include realtime files when there are no default.  Not used if format in <Hdf5, netCDF4).
   Default is to search only default files.
 --missing=<missing string> (defaults to "missing"). Not used if format in <Hdf5, netCDF4).
 --assumed=<assumed string> (defaults to "assumed"). Not used if format in <Hdf5, netCDF4).
 --knownbad=<knownbad string> (defaults to "knownbad"). Not used if format in <Hdf5, netCDF4).
 --verbose - if given, print each file processed info to stdout.  Default is to run silently.

madrigalPColor.py

madrigalPColor.py is a remote command-line application that creates PColor plots from Madrigal data

Usage:

 python madrigalPColor.py  --url=madrigalUrl --file=filename --parm=parameter
   --output=outputFile --name=userName --email=userEmail --affiliation=userAffilitation
   [--filter=filterString --title=plotTitle --startHour=startHour --endHour=endHour
   --minAlt=minimum_altitude --maxAlt=maximum_altitude --minParm=minimum_parm_value 
   --maxParm=maximum_parameter_value]

See http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html#isprint for details of how filters work. Optional startHour and endHour are UT hours, with 0 being midnight UT on the day the experiment started. Default is entire experiment.

Example:

 python madrigalPColor.py  --url=http://madrigal.haystack.mit.edu/madrigal \
   --file=/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.002 --parm=nel \
   --output=/tmp/mlh_20jan98.png --name="Bill Rideout" --email=brideout@haystack.mit.edu \
   --affiliation=MIT --filter="filter=elm,80,90 filter=gdalt,,500"

This script requires Matplotlib be installed.


madrigalScatter.py

madrigalScatter.py is a remote command-line application that creates scatter plots from Madrigal data

Usage:

 python madrigalScatter.py  --url=madrigalUrl --file=filename --parm=parameter
   --output=outputFile --name=userName --email=userEmail --affiliation=userAffilitation
   [--filter=filterString --title=plotTitle --startHour=startHour --endHour=endHour
   --yMin=y_minimum --yMax=y_maximum]

For details of how filters work, go to any Madrigal 3 site and choose Documentation -> Administrator's Guide -> Using isprint for file quick looks. Optional startHour and endHour are UT hours, with 0 being midnight UT on the day the experiment started. Default is entire experiment.

Example:

 python madrigalScatter.py  --url=http://madrigal.haystack.mit.edu/madrigal \
   --file=/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.002 --parm=systmp \
   --output=/tmp/mlh_20jan98.png --name="Bill Rideout" --email=brideout@haystack.mit.edu \
   --affiliation=MIT --filter="filter=elm,80,90"

Requires Matplotlib be installed