mdf4reader module documentation

Measured Data Format file reader module for version 4.x.

Platform and python version

With Unix and Windows for python 2.6+ and 3.2+

Author:Aymeric Rateau

Created on Thu Dec 10 12:57:28 2013

Dependencies

Attributes

PythonVersion : float
Python version currently running, needed for compatibility of both python 2.6+ and 3.2+

mdf4reader module

class mdf4reader.DATA(fid, pointer)

Bases: dict

DATA class is organizing record classes itself made of recordchannel. This class inherits from dict. Keys are corresponding to channel group recordID A DATA class corresponds to a data block, a dict of record classes (one per channel group) Each record class contains a list of recordchannel class representing the structure of channel record.

Attributes

fid (io.open) file identifier
pointerToData (int) position of Data block in mdf file
type (str) ‘sorted’ or ‘unsorted’ data block

Methods

addRecord(record) Adds a new record in DATA class dict
read(channelList, zip=None) Reads data block
load(record, zip=None, nameList=None) Reads sorted data block from record definition
readRecord(recordID, buf, channelList=None): read record from a buffer
addRecord(record)

Adds a new record in DATA class dict.

Parameters:

record class

channel group definition listing record channel classes

load(record, zip=None, nameList=None, sortedFlag=True)

Reads data block from record definition

Parameters:

record class

channel group definition listing record channel classes

zip : bool, optional

flag to track if data block is compressed

nameList : list of str, optional

list of channel names

Returns:

numpy recarray of data

read(channelList, zip=None)

Reads data block

Parameters:

channelList : list of str

list of channel names

zip : bool, optional

flag to track if data block is compressed

readRecord(recordID, buf, channelList=None)

read record from a buffer

Parameters:

recordID : int

record identifier

buf : str

buffer of data from file to be converted to channel raw data

channelList : list of str

list of channel names to be read

mdf4reader.DATABlock(record, parent_block, channelList=None, sortedFlag=True)

DATABlock converts raw data into arrays

Parameters:

record : class

record class instance describing a channel group record

parent_block : class

MDFBlock class containing at least parent block header

channelList : list of str, optional

defines list of channels to only read, can be slow but saves memory, for big files

sortedFlag : bool, optional

flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading

Returns:

a recarray containing the channels data

Notes

This function will read DTBlock, RDBlock, DZBlock (compressed), RDBlock (VLSD), sorted or unsorted

mdf4reader.append_field(rec, name, arr, numpy_dtype=None)

append new field in a recarray

Parameters:

rec : numpy recarray

name : str

name of field to be appended

arr : numpy array to be appended

numpy_dtype : numpy dtype, optional

apply same dtype as arr by default but can be modified

Returns:

numpy recarray appended

mdf4reader.arrayformat4(signalDataType, numberOfBits)

function returning numpy style string from channel data type and number of bits

Parameters:

signalDataType : int

channel data type according to specification

numberOfBits : int

number of bits taken by channel data in a record

Returns:

dataType : str

numpy dtype format used by numpy.core.records to read channel raw data

mdf4reader.bits_to_bytes(nBits)

Converts number of bits into number of aligned bytes

Parameters:

nBits : int

number of bits

Returns:

number of equivalent bytes

mdf4reader.change_field_name(arr, old_name, new_name)

modifies name of field in a recarray

Parameters:

arr : numpy recarray

old_name : str

old field

new_name : str

new field

Returns:

numpy recarray with modified field name

mdf4reader.convertChannelData4(channel, channelName, convert_tables, multiProc=False, Q=None)

converts specific channel from raw to physical data according to CCBlock information

Parameters:

channelName : dict

channel dict containing keys like ‘data’, ‘unit’, ‘comment’ and potentially ‘conversion’ dict

channelName : str

name of channel

convert_tables : bool

activates computation intensive loops for conversion with tables. Default is False

multiProc : bool, default False

flag to put data in multiprocess queue

Q : Queue class, default None

Queue used for multiprocessing

Returns:

dict

returns dict with channelName key containing numpy array converted to physical values according to conversion type

mdf4reader.convertName(channelName)

Adds ‘_title’ to channel name for numpy.core.records methods.

mdf4reader.datatypeformat4(signalDataType, numberOfBits)

function returning C format string from channel data type and number of bits

Parameters:

signalDataType : int

channel data type according to specification

numberOfBits : int

number of bits taken by channel data in a record

Returns:

dataType : str

C format used by fread to read channel raw data

mdf4reader.decompress_datablock(block, zip_type, zip_parameter, org_data_length)

decompress datablock.

Parameters:

block : bytes

raw data compressed

zip_type : int

0 for non transposed, 1 for transposed data

zip_parameter : int

first dimension of matrix to be transposed

org_data_length : int

uncompressed data length

Returns:

uncompressed raw data

mdf4reader.equalizeStringLength(buf)

Makes all strings in a list having same length by appending spaces strings.

Parameters:buf : list of str
Returns:list of str elements all having same length
mdf4reader.formulaConv(vect, formula)

apply formula conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

class mdf4reader.invalid_bytes(info, dataGroup, channelGroup, recordIDsize, byte_aligned=True)

invalid_bytes class to handle invalid bytes in record if existing

Attributes

name (str) Name of channel
unit (str, default empty string) channel unit
desc (str) channel description
conversion (info class) conversion dictionnary
signalDataType (int) signal type according to specification
bitCount (int) number of bits used to store channel record
nBytes (int) number of bytes (1 byte = 8 bits) taken by channel record
dataFormat (str) numpy dtype as string
Format : C format understood by fread
CFormat (struct class instance) struct instance to convert from C Format
byteOffset (int) position of channel record in complete record in bytes
bitOffset (int) bit position of channel value inside byte in case of channel having bit count below 8
RecordFormat (list of str) dtype format used for numpy.core.records functions ((name,name_title),str_stype)
channelType (int) channel type
posByteBeg (int) start position in number of bit of channel record in complete record
posByteEnd (int) end position in number of bit of channel record in complete record
posBitBeg (int) start position in number of bit of channel record in complete record
posBitEnd (int) end position in number of bit of channel record in complete record
VLSD_CG_Flag (bool) flag when Channel Group VLSD is used
data (int) pointer to data block linked to a channel (VLSD, MLSD)

Methods

__init__(info, dataGroup, channelGroup, recordIDsize) constructor
channel_validity(channelName) returns channel validity bit array
validity_channel(channelName)

extract channel validity bits

Parameters:

channelName : str

channel name

mdf4reader.linearConv(vect, cc_val)

apply linear conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

class mdf4reader.mdf4(fileName=None, channelList=None, convertAfterRead=True, filterChannelNames=False)

Bases: mdf.mdf_skeleton

mdf file reader class from version 4.0 to 4.1

Attributes

fileName (str) file name
MDFVersionNumber (int) mdf file version number
masterChannelList (dict) Represents data structure: a key per master channel with corresponding value containing a list of channels One key or master channel represents then a data group having same sampling interval.
multiProc (bool) Flag to request channel conversion multi processed for performance improvement. One thread per data group.
convertAfterRead (bool) flag to convert raw data to physical just after read
filterChannelNames (bool) flag to filter long channel names from its module names separated by ‘.’
file_metadata (dict) file metadata with minimum keys : author, organisation, project, subject, comment, time, date

Methods

read4( fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True) Reads mdf 4.x file data and stores it in dict
_getChannelData4(channelName) Returns channel numpy array
_convertChannel4(channelName) converts specific channel from raw to physical data according to CCBlock information
_convertAllChannel4() Converts all channels from raw data to converted data according to CCBlock information
read4(fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True, filterChannelNames=False)

Reads mdf 4.x file data and stores it in dict

Parameters:

fileName : str, optional

file name

info : mdfinfo4.info4 class

info3 class containing all MDF Blocks

multiProc : bool

flag to activate multiprocessing of channel data conversion

channelList : list of str, optional

list of channel names to be read If you use channelList, reading might be much slower but it will save you memory. Can be used to read big files

convertAfterRead : bool, optional

flag to convert channel after read, True by default If you use convertAfterRead by setting it to false, all data from channels will be kept raw, no conversion applied. If many float are stored in file, you can gain from 3 to 4 times memory footprint To calculate value from channel, you can then use method .getChannelData()

mdf4reader.rationalConv(vect, cc_val)

apply rational conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

class mdf4reader.record(dataGroup, channelGroup)

Bases: list

record class lists recordchannel classes, it is representing a channel group

Attributes

CGrecordLength (int) length of record corresponding of channel group in Byte CG Block information
recordLength (int) length of record as understood by program based on C datatypes
numberOfRecords (int) number of records in data block
recordID (int) recordID corresponding to channel group
recordIDsize (int) size of recordID
recordIDCFormat (str) record identifier C format string as understood by fread
dataGroup (int:) data group number
channelGroup (int) channel group number
numpyDataRecordFormat (list) list of numpy (dtype) for each channel
dataRecordName (list) list of channel names used for recarray attribute definition
master (dict) define name and number of master channel
recordToChannelMatching (dict) helps to identify nested bits in byte
channelNames (list) channel names to be stored, useful for low memory consumption but slow
Flags (bool) channel flags as from specification
VLSD_CG (dict) dict of Channel Group VLSD, key being recordID
VLSD (list of recordChannel) list of recordChannel being VLSD
MLSD (dict) copy from info[‘MLSD’] if existing
byte_aligned (Bool) flag for byte aligned record
invalid_channel (Default None) invalid_byte class if existing in record otherwise None

Methods

addChannel(info, channelNumber)  
loadInfo(info)  
readSortedRecord(fid, pointer, channelList=None)  
readRecordBuf(buf, channelList=None)  
readBitarray(bita, channelList=None)  
addChannel(info, channelNumber)

add a channel in class

Parameters:

info : mdfinfo4.info4 class

channelNumber : int

channel number in mdfinfo4.info4 class

loadInfo(info)

gathers records related from info class

Parameters:info : mdfinfo4.info4 class
readBitarray(bita, channelList=None)

reads stream of record bytes using bitarray module needed for not byte aligned data

Parameters:

bita : stream

stream of bytes

channelList : List of str, optional

list of channel to read

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

readRecordBuf(buf, channelList=None)

read stream of record bytes

Parameters:

buf : stream

stream of bytes read in file

channelList : list of str, optional

list of channel to read

Returns:

rec : dict

returns dictionary of channel with its corresponding values

readSortedRecord(fid, pointer, channelList=None)

reads record, only one channel group per datagroup

Parameters:

fid : float

file identifier

pointer

position in file of data block beginning

channelList : list of str, optional

list of channel to read

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

Notes

If channelList is None, read data using numpy.core.records.fromfile that is rather quick. However, in case of large file, you can use channelList to load only interesting channels or only one channel on demand, but be aware it might be much slower.

class mdf4reader.recordChannel(info, dataGroup, channelGroup, channelNumber, recordIDsize)

recordChannel class gathers all about channel structure in a record

Attributes

name (str) Name of channel
unit (str, default empty string) channel unit
desc (str) channel description
conversion (info class) conversion dictionnary
channelNumber (int) channel number corresponding to mdfinfo3.info3 class
channelGroup (int) channel group number corresponding to mdfinfo3.info3 class
dataGroup (int) data group number corresponding to mdfinfo3.info3 class
signalDataType (int) signal type according to specification
bitCount (int) number of bits used to store channel record
nBytes (int) number of bytes (1 byte = 8 bits) taken by channel record
dataFormat (str) numpy dtype as string
Format : C format understood by fread
CFormat (struct class instance) struct instance to convert from C Format
byteOffset (int) position of channel record in complete record in bytes
bitOffset (int) bit position of channel value inside byte in case of channel having bit count below 8
RecordFormat (list of str) dtype format used for numpy.core.records functions ((name,name_title),str_stype)
channelType (int) channel type ; 0 fixed length data, 1 VLSD, 2 master, 3 virtual master, 4 sync, 5 MLSD, 6 virtual data
channelSyncType (int) channel synchronisation type ; 0 None, 1 Time, 2 Angle, 3 Distance, 4 Index
posByteBeg (int) start position in number of byte of channel record in complete record
posByteEnd (int) end position in number of byte of channel record in complete record
posBitBeg (int) start position in number of bit of channel record in complete record
posBitEnd (int) end position in number of bit of channel record in complete record
VLSD_CG_Flag (bool) flag when Channel Group VLSD is used
data (int) pointer to data block linked to a channel (VLSD, MLSD)

Methods

__init__(info, dataGroup, channelGroup, channelNumber, recordIDsize) constructor
__str__() to print class attributes
attachment(fid, info)
mdf4reader.textToTextConv(vect, cc_ref)

apply text to text conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdf4reader.textToValueConv(vect, cc_val, cc_ref)

apply text to value conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdf4reader.valueRangeToTextConv(vect, cc_val, cc_ref)

apply value range to text conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdf4reader.valueRangeToValueTableConv(vect, cc_val)

apply value range to value table conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdf4reader.valueToTextConv(vect, cc_val, cc_ref)

apply value to text conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdf4reader.valueToValueTableWInterpConv(vect, cc_val)

apply value to value table with interpolation conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdf4reader.valueToValueTableWOInterpConv(vect, cc_val)

apply value to value table without interpolation conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value