Source code for pywws.toservice

#!/usr/bin/env python

"""Post weather update to services such as Weather Underground
::

%s

Introduction
------------

Several organisations allow weather stations to upload data using a
simple HTTP 'POST' or 'GET' request, with the data encoded as a
sequence of ``key=value`` pairs separated by ``&`` characters.

This module enables pywws to upload readings to these organisations.
It is highly customisable using configuration files. Each 'service'
requires a configuration file and two templates in ``pywws/services``
(that should not need to be edited by the user) and a section in
``weather.ini`` containing user specific data such as your site ID and
password.

There are currently six services for which configuration files have
been written.

+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+
| organisation                                                          | service name          | config file                                                |
+=======================================================================+=======================+============================================================+
| `UK Met Office <http://wow.metoffice.gov.uk/>`_                       | ``metoffice``         | :download:`../../code/pywws/services/metoffice.ini`        |
+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+
| `Open Weather Map <http://openweathermap.org/>`_                      | ``openweathermap``    | :download:`../../code/pywws/services/openweathermap.ini`   |
+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+
| `Stacja Pogody <http://stacjapogody.waw.pl/index.php?id=mapastacji>`_ | ``stacjapogodywawpl`` | :download:`../../code/pywws/services/stacjapogodywawpl.ini`|
+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+
| `temperatur.nu <http://www.temperatur.nu/>`_                          | ``temperaturnu``      | :download:`../../code/pywws/services/temperaturnu.ini`     |
+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+
| `Weather Underground <http://www.wunderground.com/>`_                 | ``underground``       | :download:`../../code/pywws/services/underground.ini`      |
+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+
| `wetter.com <http://www.wetter.com/community/>`_                      | ``wetterarchivde``    | :download:`../../code/pywws/services/wetterarchivde.ini`   |
+-----------------------------------------------------------------------+-----------------------+------------------------------------------------------------+

Configuration
-------------

If you haven't already done so, visit the organisation's web site and
create an account for your weather station. Make a note of any site ID
and password details you are given.

Stop any pywws software that is running and then run ``toservice.py``
to create a section in ``weather.ini``::

    python pywws/toservice.py data_dir service_name

``service_name`` is a single word service name, such as ``metoffice``,
``data_dir`` is your weather data directory, as usual.

Edit ``weather.ini`` and find the section corresponding to the service
name, e.g. ``[underground]``. Copy your site details into this
section, for example::

    [underground]
    password = secret
    station = ABCDEFG1A

Now you can test your configuration::

    python pywws/toservice.py -vvv data_dir service_name

This should show you the data string that is uploaded. Any failure
should generate an error message.

Upload old data
---------------

Now you can upload your last 7 days' data, if the service supports it.
Edit your ``weather.ini`` file and remove the ``last update`` line
from the appropriate section, then run ``toservice.py`` with the
catchup option::

    python pywws/toservice.py -cvv data_dir service_name

This may take 20 minutes or more, depending on how much data you have.

Add service(s) upload to regular tasks
--------------------------------------

Edit your ``weather.ini`` again, and add a list of services to the
``[live]``, ``[logged]``, ``[hourly]``, ``[12 hourly]`` or ``[daily]``
section, depending on how often you want to send data. For example::

    [live]
    twitter = []
    plot = []
    text = []
    services = ['underground']

    [logged]
    twitter = []
    plot = []
    text = []
    services = ['metoffice', 'stacjapogodywawpl']

    [hourly]
    twitter = []
    plot = []
    text = []
    services = ['underground']

Note that the ``[live]`` section is only used when running
``LiveLog.py``. It is a good idea to repeat any service selected in
``[live]`` in the ``[logged]`` or ``[hourly]`` section in case you
switch to running :mod:`Hourly`.

Restart your regular pywws program (:mod:`Hourly` or :mod:`LiveLog`)
and visit the appropriate web site to see regular updates from your
weather station.

Notes on the services
---------------------

UK Met Office
=============

* Create account: https://register.metoffice.gov.uk/WaveRegistrationClient/public/register.do?service=weatherobservations
* API: http://wow.metoffice.gov.uk/support?category=dataformats#automatic
* Example ``weather.ini`` section::

    [metoffice]
    site id = 12345678
    aws pin = 987654

Open Weather Map
================

* Create account: http://openweathermap.org/login
* API: http://openweathermap.org/API
* Example ``weather.ini`` section::

    [openweathermap]
    lat = 51.501
    long = -0.142
    alt = 10
    user = Elizabeth Windsor
    password = corgi
    id = Buck House

The default behaviour is to use your user name to identify the weather
station. However, it's possible for a user to have more than one
weather station, so there is an undocumented ``name`` parameter in the
API that can be used to identify the station. This appears as ``id``
in ``weather.ini``. Make sure you don't choose a name that is already
in use.

Weather Underground
===================

* Create account: http://www.wunderground.com/members/signup.asp
* API: http://wiki.wunderground.com/index.php/PWS_-_Upload_Protocol
* Example ``weather.ini`` section::

    [underground]
    station = ABCDEFGH1
    password = xxxxxxx

API
---

"""

__docformat__ = "restructuredtext en"
__usage__ = """
 usage: python RunModule.py toservice [options] data_dir service_name
 options are:
  -h or --help     display this help
  -c or --catchup  upload all data since last upload
  -v or --verbose  increase amount of reassuring messages
 data_dir is the root directory of the weather data
 service_name is the service to upload to, e.g. underground
"""
__doc__ %= __usage__
__usage__ = __doc__.split('\n')[0] + __usage__

from ConfigParser import SafeConfigParser
import getopt
import logging
import os
import re
import socket
import sys
import urllib
import urllib2
from datetime import datetime, timedelta

from . import DataStore
from .Logger import ApplicationLogger
from . import Template

FIVE_MINS = timedelta(minutes=5)

[docs]class ToService(object): """Upload weather data to weather services such as Weather Underground. """ def __init__(self, params, calib_data, service_name=None): """ :param params: pywws configuration. :type params: :class:`pywws.DataStore.params` :param calib_data: 'calibrated' data. :type calib_data: :class:`pywws.DataStore.calib_store` :keyword service_name: name of service to upload to. :type service_name: string """ if service_name: self.config_section = service_name self.logger = logging.getLogger( 'pywws.ToService(%s)' % service_name) else: self.logger = logging.getLogger( 'pywws.%s' % self.__class__.__name__) self.params = params self.data = calib_data self.old_response = None self.old_ex = None # set default socket timeout, so urlopen calls don't hang forever socket.setdefaulttimeout(30) # open params file service_params = SafeConfigParser() service_params.optionxform = str service_params.readfp(open(os.path.join( os.path.dirname(__file__), 'services', '%s.ini' % (self.config_section)))) # get URL self.server = service_params.get('config', 'url') # get fixed part of upload data self.fixed_data = dict() for name, value in service_params.items('fixed'): if value[0] == '*': value = self.params.get( self.config_section, value[1:], 'unknown') self.fixed_data[name] = value # create templater self.templater = Template.Template( self.params, self.data, self.data, None, None, use_locale=False) self.template_file = os.path.join( os.path.dirname(__file__), 'services', '%s_template_%s.txt' % (service_name, self.params.get('fixed', 'ws type'))) # get other parameters self.catchup = eval(service_params.get('config', 'catchup')) self.use_get = eval(service_params.get('config', 'use get')) rapid_fire = eval(service_params.get('config', 'rapidfire')) if rapid_fire: self.server_rf = service_params.get('config', 'url-rf') self.fixed_data_rf = dict(self.fixed_data) for name, value in service_params.items('fixed-rf'): self.fixed_data_rf[name] = value else: self.server_rf = self.server self.fixed_data_rf = self.fixed_data self.expected_result = eval(service_params.get('config', 'result'))
[docs] def translate_data(self, current, fixed_data): """Convert a weather data record to upload format. The :obj:`current` parameter contains the data to be uploaded. It should be a 'calibrated' data record, as stored in :class:`pywws.DataStore.calib_store`. The :obj:`fixed_data` parameter contains unvarying data that is site dependent, for example an ID code and authentication data. :param current: the weather data record. :type current: dict :param fixed_data: unvarying upload data. :type fixed_data: dict :return: converted data, or :obj:`None` if invalid data. :rtype: dict(string) """ # check we have enough data if current['temp_out'] is None or current['hum_out'] is None: return None # convert data result = dict(fixed_data) template_data = self.templater.make_text(self.template_file, current) result.update(eval(template_data)) return result
[docs] def send_data(self, data, server, fixed_data): """Upload a weather data record. The :obj:`data` parameter contains the data to be uploaded. It should be a 'calibrated' data record, as stored in :class:`pywws.DataStore.calib_store`. The :obj:`fixed_data` parameter contains unvarying data that is site dependent, for example an ID code and authentication data. :param data: the weather data record. :type data: dict :param server: web address to upload to. :type server: string :param fixed_data: unvarying upload data. :type fixed_data: dict :return: success status :rtype: bool """ coded_data = self.translate_data(data, fixed_data) if not coded_data: return True self.logger.debug(coded_data) coded_data = urllib.urlencode(coded_data) # have three tries before giving up for n in range(3): try: if sys.hexversion <= 0x020406ff: wudata = urllib.urlopen('%s?%s' % (server, coded_data)) else: try: if self.use_get: wudata = urllib2.urlopen( '%s?%s' % (server, coded_data)) else: wudata = urllib2.urlopen(server, coded_data) except urllib2.HTTPError, ex: if ex.code != 400: raise wudata = ex response = wudata.readlines() wudata.close() if len(response) == len(self.expected_result): for actual, expected in zip(response, self.expected_result): if not re.match(expected, actual): break else: self.old_response = response return True if response != self.old_response: for line in response: self.logger.error(line.strip()) self.old_response = response except Exception, ex: e = str(ex) if e != self.old_ex: self.logger.error(e) self.old_ex = e return False
[docs] def Upload(self, catchup): """Upload one or more weather data records. This method uploads either the most recent weather data record, or all records since the last upload (up to 7 days), according to the value of :obj:`catchup`. It sets the ``last update`` configuration value to the time stamp of the most recent record successfully uploaded. :param catchup: upload all data since last upload. :type catchup: bool :return: success status :rtype: bool """ if catchup and self.catchup > 0: start = datetime.utcnow() - timedelta(days=self.catchup) last_update = self.params.get_datetime( self.config_section, 'last update') if last_update: start = max(start, last_update + timedelta(minutes=1)) count = 0 for data in self.data[start:]: if not self.send_data(data, self.server, self.fixed_data): return False self.params.set( self.config_section, 'last update', data['idx'].isoformat(' ')) count += 1 if count: self.logger.info('%d records sent', count) else: # upload most recent data last_update = self.data.before(datetime.max) if not self.send_data( self.data[last_update], self.server, self.fixed_data): return False self.params.set( self.config_section, 'last update', last_update.isoformat(' ')) return True
[docs] def RapidFire(self, data, catchup): """Upload a 'Rapid Fire' weather data record. This method uploads either a single data record (typically one obtained during 'live' logging), or all records since the last upload (up to 7 days), according to the value of :obj:`catchup`. It sets the ``last update`` configuration value to the time stamp of the most recent record successfully uploaded. The :obj:`data` parameter contains the data to be uploaded. It should be a 'calibrated' data record, as stored in :class:`pywws.DataStore.calib_store`. :param data: the weather data record. :type data: dict :param catchup: upload all data since last upload. :type catchup: bool :return: success status :rtype: bool """ last_log = self.data.before(datetime.max) if not last_log or last_log < data['idx'] - FIVE_MINS: # logged data is not (yet) up to date return True if catchup and self.catchup > 0: last_update = self.params.get_datetime( self.config_section, 'last update') if not last_update: last_update = datetime.min if last_update <= last_log - FIVE_MINS: # last update was well before last logged data if not self.Upload(True): return False if not self.send_data(data, self.server_rf, self.fixed_data_rf): return False self.params.set( self.config_section, 'last update', data['idx'].isoformat(' ')) return True
[docs]def main(argv=None): if argv is None: argv = sys.argv try: opts, args = getopt.getopt( argv[1:], "hcv", ['help', 'catchup', 'verbose']) except getopt.error, msg: print >>sys.stderr, 'Error: %s\n' % msg print >>sys.stderr, __usage__.strip() return 1 # process options catchup = False verbose = 0 for o, a in opts: if o == '-h' or o == '--help': print __usage__.strip() return 0 elif o == '-c' or o == '--catchup': catchup = True elif o == '-v' or o == '--verbose': verbose += 1 # check arguments if len(args) != 2: print >>sys.stderr, "Error: 2 arguments required" print >>sys.stderr, __usage__.strip() return 2 logger = ApplicationLogger(verbose) return ToService( DataStore.params(args[0]), DataStore.calib_store(args[0]), service_name=args[1] ).Upload(catchup)
if __name__ == "__main__": sys.exit(main())