mdf4reader module documentation¶
Measured Data Format file reader module for version 4.x.
Platform and python version¶
With Unix and Windows for python 2.6+ and 3.2+
Author: | Aymeric Rateau |
---|
Created on Thu Dec 10 12:57:28 2013
Dependencies¶
- Python >2.6, >3.2 <http://www.python.org>
- Numpy >1.6 <http://numpy.scipy.org>
- bitarray to parse bits in not aligned bytes
- Sympy to convert channels with formula if needed
- zlib to uncompress data block if needed
Attributes¶
- PythonVersion : float
- Python version currently running, needed for compatibility of both python 2.6+ and 3.2+
mdf4reader module¶
-
class
mdf4reader.
DATA
(fid, pointer)¶ Bases:
dict
DATA class is organizing record classes itself made of recordchannel. This class inherits from dict. Keys are corresponding to channel group recordID A DATA class corresponds to a data block, a dict of record classes (one per channel group) Each record class contains a list of recordchannel class representing the structure of channel record.
Attributes
fid (io.open) file identifier pointerToData (int) position of Data block in mdf file type (str) ‘sorted’ or ‘unsorted’ data block Methods
addRecord(record) Adds a new record in DATA class dict read(channelList, zip=None) Reads data block load(record, zip=None, nameList=None) Reads sorted data block from record definition readRecord(recordID, buf, channelList=None): read record from a buffer -
addRecord
(record)¶ Adds a new record in DATA class dict.
Parameters: record class
channel group definition listing record channel classes
-
load
(record, zip=None, nameList=None, sortedFlag=True)¶ Reads data block from record definition
Parameters: record class
channel group definition listing record channel classes
zip : bool, optional
flag to track if data block is compressed
nameList : list of str, optional
list of channel names
Returns: numpy recarray of data
-
read
(channelList, zip=None)¶ Reads data block
Parameters: channelList : list of str
list of channel names
zip : bool, optional
flag to track if data block is compressed
-
readRecord
(recordID, buf, channelList=None)¶ read record from a buffer
Parameters: recordID : int
record identifier
buf : str
buffer of data from file to be converted to channel raw data
channelList : list of str
list of channel names to be read
-
-
mdf4reader.
DATABlock
(record, parent_block, channelList=None, sortedFlag=True)¶ DATABlock converts raw data into arrays
Parameters: record : class
record class instance describing a channel group record
parent_block : class
MDFBlock class containing at least parent block header
channelList : list of str, optional
defines list of channels to only read, can be slow but saves memory, for big files
sortedFlag : bool, optional
flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading
Returns: a recarray containing the channels data
Notes
This function will read DTBlock, RDBlock, DZBlock (compressed), RDBlock (VLSD), sorted or unsorted
-
mdf4reader.
append_field
(rec, name, arr, numpy_dtype=None)¶ append new field in a recarray
Parameters: rec : numpy recarray
name : str
name of field to be appended
arr : numpy array to be appended
numpy_dtype : numpy dtype, optional
apply same dtype as arr by default but can be modified
Returns: numpy recarray appended
-
mdf4reader.
arrayformat4
(signalDataType, numberOfBits)¶ function returning numpy style string from channel data type and number of bits
Parameters: signalDataType : int
channel data type according to specification
numberOfBits : int
number of bits taken by channel data in a record
Returns: dataType : str
numpy dtype format used by numpy.core.records to read channel raw data
-
mdf4reader.
bits_to_bytes
(nBits)¶ Converts number of bits into number of aligned bytes
Parameters: nBits : int
number of bits
Returns: number of equivalent bytes
-
mdf4reader.
change_field_name
(arr, old_name, new_name)¶ modifies name of field in a recarray
Parameters: arr : numpy recarray
old_name : str
old field
new_name : str
new field
Returns: numpy recarray with modified field name
-
mdf4reader.
convertChannelData4
(channel, channelName, convert_tables, multiProc=False, Q=None)¶ converts specific channel from raw to physical data according to CCBlock information
Parameters: channelName : dict
channel dict containing keys like ‘data’, ‘unit’, ‘comment’ and potentially ‘conversion’ dict
channelName : str
name of channel
convert_tables : bool
activates computation intensive loops for conversion with tables. Default is False
multiProc : bool, default False
flag to put data in multiprocess queue
Q : Queue class, default None
Queue used for multiprocessing
Returns: dict
returns dict with channelName key containing numpy array converted to physical values according to conversion type
-
mdf4reader.
convertName
(channelName)¶ Adds ‘_title’ to channel name for numpy.core.records methods.
-
mdf4reader.
datatypeformat4
(signalDataType, numberOfBits)¶ function returning C format string from channel data type and number of bits
Parameters: signalDataType : int
channel data type according to specification
numberOfBits : int
number of bits taken by channel data in a record
Returns: dataType : str
C format used by fread to read channel raw data
-
mdf4reader.
decompress_datablock
(block, zip_type, zip_parameter, org_data_length)¶ decompress datablock.
Parameters: block : bytes
raw data compressed
zip_type : int
0 for non transposed, 1 for transposed data
zip_parameter : int
first dimension of matrix to be transposed
org_data_length : int
uncompressed data length
Returns: uncompressed raw data
-
mdf4reader.
equalizeStringLength
(buf)¶ Makes all strings in a list having same length by appending spaces strings.
Parameters: buf : list of str Returns: list of str elements all having same length
-
mdf4reader.
formulaConv
(vect, formula)¶ apply formula conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
class
mdf4reader.
invalid_bytes
(info, dataGroup, channelGroup, recordIDsize, byte_aligned=True)¶ invalid_bytes class to handle invalid bytes in record if existing
Attributes
name (str) Name of channel unit (str, default empty string) channel unit desc (str) channel description conversion (info class) conversion dictionnary signalDataType (int) signal type according to specification bitCount (int) number of bits used to store channel record nBytes (int) number of bytes (1 byte = 8 bits) taken by channel record dataFormat (str) numpy dtype as string Format : C format understood by fread CFormat (struct class instance) struct instance to convert from C Format byteOffset (int) position of channel record in complete record in bytes bitOffset (int) bit position of channel value inside byte in case of channel having bit count below 8 RecordFormat (list of str) dtype format used for numpy.core.records functions ((name,name_title),str_stype) channelType (int) channel type posByteBeg (int) start position in number of bit of channel record in complete record posByteEnd (int) end position in number of bit of channel record in complete record posBitBeg (int) start position in number of bit of channel record in complete record posBitEnd (int) end position in number of bit of channel record in complete record VLSD_CG_Flag (bool) flag when Channel Group VLSD is used data (int) pointer to data block linked to a channel (VLSD, MLSD) Methods
__init__(info, dataGroup, channelGroup, recordIDsize) constructor channel_validity(channelName) returns channel validity bit array -
validity_channel
(channelName)¶ extract channel validity bits
Parameters: channelName : str
channel name
-
-
mdf4reader.
linearConv
(vect, cc_val)¶ apply linear conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
class
mdf4reader.
mdf4
(fileName=None, channelList=None, convertAfterRead=True, filterChannelNames=False)¶ Bases:
mdf.mdf_skeleton
mdf file reader class from version 4.0 to 4.1
Attributes
fileName (str) file name MDFVersionNumber (int) mdf file version number masterChannelList (dict) Represents data structure: a key per master channel with corresponding value containing a list of channels One key or master channel represents then a data group having same sampling interval. multiProc (bool) Flag to request channel conversion multi processed for performance improvement. One thread per data group. convertAfterRead (bool) flag to convert raw data to physical just after read filterChannelNames (bool) flag to filter long channel names from its module names separated by ‘.’ file_metadata (dict) file metadata with minimum keys : author, organisation, project, subject, comment, time, date Methods
read4( fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True) Reads mdf 4.x file data and stores it in dict _getChannelData4(channelName) Returns channel numpy array _convertChannel4(channelName) converts specific channel from raw to physical data according to CCBlock information _convertAllChannel4() Converts all channels from raw data to converted data according to CCBlock information -
read4
(fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True, filterChannelNames=False)¶ Reads mdf 4.x file data and stores it in dict
Parameters: fileName : str, optional
file name
info : mdfinfo4.info4 class
info3 class containing all MDF Blocks
multiProc : bool
flag to activate multiprocessing of channel data conversion
channelList : list of str, optional
list of channel names to be read If you use channelList, reading might be much slower but it will save you memory. Can be used to read big files
convertAfterRead : bool, optional
flag to convert channel after read, True by default If you use convertAfterRead by setting it to false, all data from channels will be kept raw, no conversion applied. If many float are stored in file, you can gain from 3 to 4 times memory footprint To calculate value from channel, you can then use method .getChannelData()
-
-
mdf4reader.
rationalConv
(vect, cc_val)¶ apply rational conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
class
mdf4reader.
record
(dataGroup, channelGroup)¶ Bases:
list
record class lists recordchannel classes, it is representing a channel group
Attributes
CGrecordLength (int) length of record corresponding of channel group in Byte CG Block information recordLength (int) length of record as understood by program based on C datatypes numberOfRecords (int) number of records in data block recordID (int) recordID corresponding to channel group recordIDsize (int) size of recordID recordIDCFormat (str) record identifier C format string as understood by fread dataGroup (int:) data group number channelGroup (int) channel group number numpyDataRecordFormat (list) list of numpy (dtype) for each channel dataRecordName (list) list of channel names used for recarray attribute definition master (dict) define name and number of master channel recordToChannelMatching (dict) helps to identify nested bits in byte channelNames (list) channel names to be stored, useful for low memory consumption but slow Flags (bool) channel flags as from specification VLSD_CG (dict) dict of Channel Group VLSD, key being recordID VLSD (list of recordChannel) list of recordChannel being VLSD MLSD (dict) copy from info[‘MLSD’] if existing byte_aligned (Bool) flag for byte aligned record invalid_channel (Default None) invalid_byte class if existing in record otherwise None Methods
addChannel(info, channelNumber) loadInfo(info) readSortedRecord(fid, pointer, channelList=None) readRecordBuf(buf, channelList=None) readBitarray(bita, channelList=None) -
addChannel
(info, channelNumber)¶ add a channel in class
Parameters: info : mdfinfo4.info4 class
channelNumber : int
channel number in mdfinfo4.info4 class
-
loadInfo
(info)¶ gathers records related from info class
Parameters: info : mdfinfo4.info4 class
-
readBitarray
(bita, channelList=None)¶ reads stream of record bytes using bitarray module needed for not byte aligned data
Parameters: bita : stream
stream of bytes
channelList : List of str, optional
list of channel to read
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
-
readRecordBuf
(buf, channelList=None)¶ read stream of record bytes
Parameters: buf : stream
stream of bytes read in file
channelList : list of str, optional
list of channel to read
Returns: rec : dict
returns dictionary of channel with its corresponding values
-
readSortedRecord
(fid, pointer, channelList=None)¶ reads record, only one channel group per datagroup
Parameters: fid : float
file identifier
pointer
position in file of data block beginning
channelList : list of str, optional
list of channel to read
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
Notes
If channelList is None, read data using numpy.core.records.fromfile that is rather quick. However, in case of large file, you can use channelList to load only interesting channels or only one channel on demand, but be aware it might be much slower.
-
-
class
mdf4reader.
recordChannel
(info, dataGroup, channelGroup, channelNumber, recordIDsize)¶ recordChannel class gathers all about channel structure in a record
Attributes
name (str) Name of channel unit (str, default empty string) channel unit desc (str) channel description conversion (info class) conversion dictionnary channelNumber (int) channel number corresponding to mdfinfo3.info3 class channelGroup (int) channel group number corresponding to mdfinfo3.info3 class dataGroup (int) data group number corresponding to mdfinfo3.info3 class signalDataType (int) signal type according to specification bitCount (int) number of bits used to store channel record nBytes (int) number of bytes (1 byte = 8 bits) taken by channel record dataFormat (str) numpy dtype as string Format : C format understood by fread CFormat (struct class instance) struct instance to convert from C Format byteOffset (int) position of channel record in complete record in bytes bitOffset (int) bit position of channel value inside byte in case of channel having bit count below 8 RecordFormat (list of str) dtype format used for numpy.core.records functions ((name,name_title),str_stype) channelType (int) channel type ; 0 fixed length data, 1 VLSD, 2 master, 3 virtual master, 4 sync, 5 MLSD, 6 virtual data channelSyncType (int) channel synchronisation type ; 0 None, 1 Time, 2 Angle, 3 Distance, 4 Index posByteBeg (int) start position in number of byte of channel record in complete record posByteEnd (int) end position in number of byte of channel record in complete record posBitBeg (int) start position in number of bit of channel record in complete record posBitEnd (int) end position in number of bit of channel record in complete record VLSD_CG_Flag (bool) flag when Channel Group VLSD is used data (int) pointer to data block linked to a channel (VLSD, MLSD) Methods
__init__(info, dataGroup, channelGroup, channelNumber, recordIDsize) constructor __str__() to print class attributes -
attachment
(fid, info)¶
-
-
mdf4reader.
textToTextConv
(vect, cc_ref)¶ apply text to text conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdf4reader.
textToValueConv
(vect, cc_val, cc_ref)¶ apply text to value conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdf4reader.
valueRangeToTextConv
(vect, cc_val, cc_ref)¶ apply value range to text conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdf4reader.
valueRangeToValueTableConv
(vect, cc_val)¶ apply value range to value table conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdf4reader.
valueToTextConv
(vect, cc_val, cc_ref)¶ apply value to text conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdf4reader.
valueToValueTableWInterpConv
(vect, cc_val)¶ apply value to value table with interpolation conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdf4reader.
valueToValueTableWOInterpConv
(vect, cc_val)¶ apply value to value table without interpolation conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value