Metering Package Indroduction¶
The 'metering' package is intended to automate many of the routine processes encountered when handling data collected from meters. It allows for further analysis to be preformed in Python, or for data to be written to a .csv file and analyzed in a different program, like Excel, Matlab, or R. If you haven't installed Python, you can download it here by selecting the 64-bit version of Python 2.7.
The code included below can be run from the console window or copied into a script. The 'In [1]' indicates the input at line 1, and if an output is returned, it will appear next to 'Out [1]'.
Installing metering¶
Use pip to install the metering package from pypi.org.
!pip install metering
If metering is already installed, you can use pip to upgrade the version.
!pip install metering -U
Importing metering¶
Now that metering is installed, modules can be imported from python.
import metering
metering
Inside of metering are several modules that can be accessed using dot notation. We'll be working with the 'exports' module.
import metering.exports
metering.exports
The exports module contains classes for reading data files that have been exported from various applications. Here we'll look at .csv files exported from HOBOWare. Accessing these classes similarly uses dot notation. The top level class in the exports module is called 'Export,' which contains attirbutes and methods that apply generically, and below it are classes specific to a certain application, including the 'HOBOWare' class.
from metering.exports import Export
Export
import sys
path = str()
for s in sys.path:
if s.split('\\')[-1] == 'site-packages':
path = s
path
Directory Setup¶
When a file is read by the Export, it is essential for the module to know what application that file came from because this will determine how it is processed. Currently (and this can change) I use the directory the data is stored in to indicate to Export the type of file that is being passed. This process requires directories have specific names and that the data contained in them all comes from the same application. For files exported form HOBOWare, the directory name needs to be 'HOBOWare.'
In the Windows Explorer address bar, you can see the path to the five files we'll be working with, where the open directory is 'HOBOWare.' These five files include data from U30s, plug load meters, state loggers, and Temp/RH loggers.
from PIL import Image
Image.open(path + '\\metering\\data\\Images\\Directory.png')
Processing Data¶
To process these five export files, we'll pass their path to the method Export.make() with a second argument to specify the sampling frequency we'd like to use for later analysis. The file path is a string the indicates the location of the file and is make up of the directories the file is in and the file name. Here each path will begin with 'F:\metering demo\Exports\HOBOWare\' and end with the name of the file, including the '.csv' at the end. This path depends on where you've stored your data.
A quick way to compile these paths is to use python's os.listdir method, which takes a directory path as an argument and returnes the files it contains. Appending these file names to the directory path, we can create a list of file paths that can be passed to Export.make().
import os
dir_path = path + '\\metering\\data\\Exports\\HOBOWare'
l = list()
for i in os.listdir(dir_path):
l.append(dir_path + '\\' + i)
l
A note on file names: The files here all follow a convention Cadmus has used in the past, which is to seperate by a hyphen a site identifier, a last name, a meter serial number, and in some cases the date and time. This convention has been useful, but won't be necessary for every project. Currently, the HOBOWare export class does require at least a site identifier followed by a hyphen.
We can now call the Export.make() method with one of the file paths above and the additional argument '1Min' (to indicate we'd like the data stored in 1 minute intervals), and this call will return a HOBOWareExport object that we'll store as the variable 'e1.'
file_path = l[0]
w = '1Min'
e1 = Export.make_Export(file_path, w)
e1
What's going on behind the Export.make() method are all the processes required for further analysis, including reading in the data, creating and localizing a timestamp index, clean out unused columns or rows, resampling, interpolating, and storing identifiers so the data can later be mapped to some piece of equipiment or quantity. We can look at some of the data attributes stored in an export object.
# Identifier associated with an export
e1.idy
# A portion of the data from the export file
e1.data.head()
# The application the data was exported form
e1.app
Another attribute of any export object is a list of 'Measurements.' Measurement is a class used to represent specific measurement like temperature or power, as opposed to Export, which represents a file exported from an applicaiton. Since the export file we read in before has four columns, the measurement list attribute of the export object will have four elements.
e1.meas_list
The important attributes of Measurement objects are the data and the measurement descriptions. Measurement description is meant to be a unique identifier so that when multiple exports are used to meter the same quantity (say you have two file each measuring the same temperature during two different periods of time) so the they can later be combine into a single time series. Below we'll assign the first Measurement object in the exports measurement list and access it's attributes.
m = e1.meas_list[0]
m.meas_des
m.data.head()
We can process the rest of export files from before (takes a few seconds).
e2 = Export.make_Export(l[1], w)
e3 = Export.make_Export(l[2], w)
e4 = Export.make_Export(l[3], w)
e5 = Export.make_Export(l[4], w)
If we wanted to group the data from these exports together, say because they all came from the same site, we can use the Location class. The function this class serves is to combine measurements across multiple exports and compile everything into a single data frame that can be written to a .csv file. This process is accomplished by calling the Location.make_Location() method with the arguments of the exports you'd like to combine as a list. Here we'll input exports e1, e2, and e3 because they were all collected at the same site.
from metering.exports import Location
export_list = [e1, e2, e3]
site_id = 'Test Site'
loc = Location.make_Location(export_list, site_id)
If we look at the measurment lists of the individual exports inputted into the MeasurementGroup.make() method and those of the MeasurementGroup object mg, we see that four of the Measurement objects were combine across exports, this was because the represented the same measurements taken at a site but that occured during different periods of time.
e1.meas_list
e2.meas_list
e3.meas_list
loc.meas_list
Plotting¶
Using the matplotlib package we can plot some of this processed data.
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
m = loc.meas_list[2]
fig = plt.figure(figsize=(10, 8))
plt.plot(m.data.index, m.data)
plt.ylim(bottom=0, top=100)
plt.grid()
plt.title(m.meas_des)
fig.autofmt_xdate()
plt.show()
plt.close('all')
