dosertools.data_processing package — dosertools 2022 documentation

# dosertools.data_processing package

## Submodules

## dosertools.data_processing.array module

dosertools.data_processing.array.closest_index_for_value(dataset: pandas.core.frame.DataFrame, column: str, value: float) → int

  Finds the closest value for a given value in a column and returns its index.

  Parameters

    - dataset (pandas.DataFrame) – the dataframe that contains at least the column “column” column must be numeric (int or float)

    - column (str) – name of column to look the closest value in

    - value (float) – numeric value to compare entries in dataset[column] to

  Returns

    closest_index_for_value (int) – Closest index in dataset[column] for given value

  Examples

  Given dataframe ‘df’ with column ‘a’ with values [-1,0,1,2] and ‘b’ with values [‘c’,1,1,1.2], the following would result from use of the function. closest_index_for_value(df,’a’,1.1) = 2 closest_index_for_value(df,’a’,1.9) = 3 closest_index_for_value(df,’b’,1.1) –> TypeError

dosertools.data_processing.array.continuous_nonzero(array: numpy.ndarray) → numpy.ndarray

  Returns array with index pairs indicating blocks of nonzero in given array.

  Returns array with shape (m, 2), where m is the number of “blocks” of non-zeros. The first column is the index of the first non-zero, the second is the index of the first zero following the blocks. If the block reaches the end of the array, the second index will be the size of the array + 1. Follows convention of numpy where array(a,a+n) yields the values of indices a through a+n-1.

  Parameters

    array (np.ndarray) – array to look for nonzero blocks within array must be numeric (integer or float)

  Returns

    continuous_nonzero (np.ndarray) – (m, 2) array where m is the number of “blocks” of non-zeros. The first column is the index of the first non-zero, the second is the index of the first zero following the block.

  Examples

  array continuous_nonzero(array) [1,1,1,1,0,0,1,1,0] [[0,4],[6,8]] [0,0,-1,1,-1,1] [[2,6]]

dosertools.data_processing.array.continuous_zero(array: numpy.ndarray) → numpy.ndarray

  Returns array with index pairs indicating blocks of zero in given array.

  Returns array with shape (m, 2), where m is the number of “blocks” of zeros. The first column is the index of the first zero, the second is the index of the first non-zero following the block. If the block reaches the end of the array, the second index will be the size of the array + 1. Follows convention of numpy where array(a,a+n) yields the values of indices a through a+n-1.

  Parameters

    array (np.ndarray) – array to look for zero runs within array must be numeric (integer or float)

  Returns

    nonzero_runs (np.ndarray) – (m, 2) array where m is the number of “blocks” of zeros. The first column is the index of the first zero, the second is the index of the first non-zero following the block.

  Examples

  array continuous_zero(array) [1,1,1,1,0,0,1,1,0] [[4,6],[8,9]] [0,0,-1,1,-1,1] [[0,2]]

dosertools.data_processing.array.is_array_numeric(array: numpy.ndarray) → bool

  Return True if array is float or int (numeric), otherwise False

  Parameters

    array (np.ndarray) – array to check if numeric

  Returns

    is_array_numeric (bool) – True if array is float or signed/unsigned int, otherwise False

  Examples

  is_array_numeric([0,1,2,3]) = True is_array_numeric([1.1,1.2,1.5]) = True is_array_numeric([‘a’,’b’,’c’]) = False is_array_numeric([True,False,False]) = False

dosertools.data_processing.array.is_dataframe_column_numeric(dataset: pandas.core.frame.DataFrame, column: str) → bool

  Return True if column in dataset is float or int (numeric), otherwise False

  Parameters

    - dataset (pandas.DataFrame) – the dataframe that contains at least the column “column”

    - column (str) – name of column to check if numeric

  Returns

    is_dataframe_column_numeric (bool) – True if column in dataset is float or int, otherwise False

  Examples

  Given dataframe ‘df’ with column ‘a’ with values [-1,0,1,2] and ‘b’ with values [‘c’,1,1,1.2], the following would result from use of the function. is_dataframe_column_numeric(df,’a’) = True is_dataframe_column_numeric(df,’b’) = False is_dataframe_column_numeric(df,’c’) –> KeyError

## dosertools.data_processing.csv module

dosertools.data_processing.csv.csv_to_dataframe(csv: str, fname_format: str, sampleinfo_format: str, optional_settings: dict = {}) → pandas.core.frame.DataFrame

  Reads in a csv into a dataframe with sample parameters.

  Parameters

    - csv (string) – Path to csv file to import.

    - tc_bounds (np.array) – Two value array containing the upper and lower bounds in “D/D0” where tc will be found in between.

    - fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split. ex. “date_sampleinfo_fps_run”

    - sampleinfo_format (str) – The format of the sampleinfo section of the fname, separated by the deliminator specified by sample_split.

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - fname_split (string) – The deliminator for splitting folder/file names, used in fname_format. Default is “_”.

    - sample_split (string) – The deliminator for splitting sampleinfo tag in folder/file names, used in sampleinfo_format. Default is “-“.

  Returns

    csv_to_dataframe (pd.DataFrame) – dataframe with data from csv, sample information from filename, strain rate and critical time calculated

dosertools.data_processing.csv.generate_df(csv_location: Union[str, bytes, os.PathLike], fname_format: str, sampleinfo_format: str, optional_settings: dict = {}) → pandas.core.frame.DataFrame

  Reads in all csvs and process them into a dataframe.

  Reads in data from all csvs in csv_location, process each, adding strain rate, critical time, diameter at critical time, and parameters from the filename, and put all data into one dataframe. Loops csv_to_dataframe for all csvs in folder.

  Parameters

    - csv_location (path-like) – folder in which csvs to process are stored

    - tc_bounds (np.array) – two value array containing the upper and lower bounds in “D/D0” where tc will be found in between

    - fname_format (str) – the format of the fname with parameter names separated by the deliminator specified by fname_split ex. “date_sampleinfo_fps_run”

    - sampleinfo_format (str) – the format of the sampleinfo section of the fname separated by the deliminator specified by sample_split

    - optional_settings (dict) – A dictionary of optional settings.

  Returns

    generate_df (pd.DataFrame) – dataframe containing data from all csvs in csv_location

  Optional Settings and Defaults

    verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

dosertools.data_processing.csv.get_csvs(csv_location: Union[str, bytes, os.PathLike]) → list

  Returns list of csvs in csv_location.

  Parameters

    csv_location (path-like) – path to a location containing desired csvs

  Returns

    get_csvs (list) – sorted list of csvs in csv_location as path strings

## dosertools.data_processing.extension module

dosertools.data_processing.extension.add_critical_time(dataset: pandas.core.frame.DataFrame, optional_settings: dict = {}) → pandas.core.frame.DataFrame

  Finds critical time from maximum in strain rate, adds relevant columns.

  Finds the critical time from the maximum in the strain rate within the bounds in di specified by tc_bounds. Adds the columns “tc (s)” (critical time), “t-tc (s)” (time past critical time), and “Dtc/D0” (diameter at critical time divided by initial diameter) to the dataset.

  Parameters

    - dataset (pandas.DataFrame) – dataset to which to add the “tc (s)”, “t-tc (s)”, and “Dtc/D0” columns must contain “D/D0”, “time (s)”, and “strain rate (1/s)” columns

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    tc_bounds (2 element list of floats) – [start, end] The D/D0 to bound the start and end for finding the critical time. Default is [0.3,0.07].

  Returns

    add_critical_time (pd.DataFrame) – dataset with “tc”, “t - tc (s)”, and “Dtc/D0” columns added

dosertools.data_processing.extension.add_strain_rate(dataset: pandas.core.frame.DataFrame) → pandas.core.frame.DataFrame

  Calculates strain rate from D/D0 and time(s) data and adds it to dataset

  Using the formula -2(d(D/D0)/dt)/(D/D0) for the strain rate, calculates the strain rate at each point in dataset using np.gradient for the derivative. Removes rows where the strain rate is infinite/NaN from the dataset. Returns a dataframe with all existing columns and the new strain rate (1/s) column.

  Parameters

    dataset (pandas.DataFrame) – dataset to which to add the “strain rate (1/s)” column must contain “D/D0” and “time (s)” columns

  Returns

    add_strain_rate (pandas.DataFrame) – dataset with strain rate (1/s) column added and all rows with infinite/NaN removed

dosertools.data_processing.extension.truncate_data(dataset: pandas.core.frame.DataFrame, before: bool = True) → pandas.core.frame.DataFrame

  Truncates a dataset before/after the longest block of continuous zeroes.

  Given a dataset, truncates the dataset before/after (depending on the True/False value of before) the longest block of continuous zeroes in the “D/D0” column. The longest block of zeroes should occur after the liquid bridge breaks and the readout is no longer accurate.

  Parameters

    - dataset (pd.DataFrame) – Dataframe containing data to truncate. Dataframe must contain “D/D0” column.

    - before (bool, optional) – True if truncation should occur at the last nonzero value before the longest block of zeroes. (default) False if truncation should occur at the last zero in the longest block of zeroes.

  Returns

    truncate_data (pd.DataFrame) – Dataframe with truncated data.

  Examples

  DataFrame ex1: time D/D0 0 1 0.1 0.9 0.2 0.8 0.3 0 0.4 0.6 0.5 0.4 0.6 0 0.7 0 0.8 0 0.9 0 1.0 0.1 1.1 0.2 1.2 0 1.3 0 1.4 0.1

  truncate_data(ex1) time D/D0 0 1 0.1 0.9 0.2 0.8 0.3 0 0.4 0.6 0.5 0.4

  truncate_data(ex1, False) time D/D0 0 1 0.1 0.9 0.2 0.8 0.3 0 0.4 0.6 0.5 0.4 0.6 0 0.7 0 0.8 0 0.9 0

  DataFrame ex2: time 0 0.1

  truncate_data(ex2) –> KeyError

## dosertools.data_processing.figures module

dosertools.data_processing.figures.layout_time_csvs(df: pandas.core.frame.DataFrame, plot_normalized: bool) → holoviews.element.geom.Points

  Plots a time vs D/D0 graph of all samples and runs in df.

  Plots with raw (time) or normalized (t - t_c) depending on the value of plot_normalized.

  Parameters

    - df (pd.DataFrame) – Dataframe of D/D0 and time data with sample and run information

    - plot_normalized (bool) – True to normalize time by t_c, the critical time, and plot t - tc on the x-axis False to plot raw time on the x-axis

  Returns

    hv_layout (hv.Points) – Set of plots for each run and sample included in df

dosertools.data_processing.figures.layout_viscosity_csvs(df: pandas.core.frame.DataFrame) → holoviews.element.geom.Points

  Plots a strain vs (elongational viscosity / surface tension) graph of all samples and runs in df.

  Parameters

    df (pd.DataFrame) – Dataframe with (elongational viscosity / surface tension) and strain with sample and run information

  Returns

    hv_layout (hv.Points) – Set of plots for each run and sample included in df

dosertools.data_processing.figures.save_figure(figure: holoviews.element.geom.Points, figure_name: str, summary_folder: Union[str, bytes, os.PathLike], optional_settings: dict = {}) → None

  Saves the figure as an .html file which enables interactivity

  Parameters

    - figure (hv.Points) – Set of plots for each run and sample included in df

    - figure_name (string) – Filename with which to save the figure

    - summary_folder (Path) – Location to save the figure

    - optional_settings (dict) – Dictionary of optional settings.

  Optional Settings and Defaults

    verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

  Returns

    None, file saved to disk

## dosertools.data_processing.fitting module

dosertools.data_processing.fitting.annotate_summary_df(fitting_results_list: list, header_params: dict) → pandas.core.frame.DataFrame

  Do we want to bring other columns with us like ion, polymer identity, etc? How to code that?

  Parameters

    - fitting_results_list (list) – generated by find_EC_slope

    - header_params (dict) – Contains the information profiled in the samplename

  Returns

    lambdaE_df (pd.DataFrame) – dataframe containing lambdaE relaxation time for each run from the input df

dosertools.data_processing.fitting.calculate_elongational_visc(df: pandas.core.frame.DataFrame, summary_df: pandas.core.frame.DataFrame, optional_settings: dict = {}) → pandas.core.frame.DataFrame

  Calculates the quantity (elongational viscosity / surface tension) for each moment in the DOS dataset.

  Parameters

    - df (pd.DataFrame) – Contains D/D0, time, t - tc, strain rate, D(tc)/D0, etc for multiple runs and samples generated from data_processing.csv.generate_df

    - summary_df (pd.DataFrame) – Contains relaxation time, D(t_c)/D0, and sample info for all runs and samples generated from data_procescing.fitting.make_summary_dataframe

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    needle_diameter_mm (float) – The needle outer diameter in millimeters. Default is 0.7176 mm (22G needle).

  Returns

    dataset_w_visc (pd.DataFrame) – Dataset with the additional values calculated by this function: strain and (elongational viscosity / surface tension)

dosertools.data_processing.fitting.derivative_EC_fit(Dtc_D0: float, lambdaE: float, time: float, tc: float) → float

  Calculates the derivative of the elasto-capillary region.

  D(t)/D0 = D(tc)/D0 * (exp(-(t - tc)/(3*LambdaE))) D’(t)/D0 = (-1/(3*LambdaE)) * D(tc)/D0 * (exp(-(t - tc)/(3*LambdaE)))

  Parameters

    - Dtc_D0 (float) – The normalized diameter at which the transition to EC behavior occurs

    - LambdaE (float) – The relaxation time of the polymer solution

    - time (float) – the moment in time to evaluate the derivative

    - tc (float) – the critical time for the experiment, this number is purely emperical (it depends entirely on when in the process the video starts)

  Returns

    D’(t)/D0 (float)

dosertools.data_processing.fitting.find_EC_slope(run_dataset: pandas.core.frame.DataFrame, start: float, end: float) → Tuple[float, float, float]

  Finds the exponential decay of the EC region for a single dataset. Also returns the intercept and r value of the fit

  Parameters

    - run_dataset (pd.DataFrame) – dataset of a single DOS run containing at least D/D0, time,

    - start (float) – value of D/D0 to start fitting the EC region from

    - end (float) – value of D/D0 to end fitting the EC region with

  Returns

    slope, intercept, r_value (floats) – slope is the slope of the semilog-transformed version of the dataset. It corresponds to the decaying exponential in the linear-linear version of the data

dosertools.data_processing.fitting.make_summary_dataframe(df: pandas.core.frame.DataFrame, sampleinfo_format: str, optional_settings: dict = {}) → pandas.core.frame.DataFrame

  Condenses a DOS run into an extensional relaxation time by fitting the EC region (t > tc) to a decaying exponential

  Parameters

    - df (pd.DataFrame) – Contains D/D0, time, t - tc, strain rate, D(tc)/D0, etc for multiple runs and samples generated from data_processing.csv.generate_df

    - sampleinfo_format (str) – the format of the sampleinfo section of the filename separated by the deliminator specified by sample_split

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    fitting_bounds (2 element list of floats) – [start, end] The D/D0 to bound the start and end of fitting of EC region. Default is [0.1, 0.045].

  Returns

    summary_df (pd.DataFrame) – dataframe containing lambdaE (relaxation time) and R(t_c)/R_0 for each run from the input df, along with their sample info

dosertools.data_processing.fitting.save_processed_df(df: pandas.core.frame.DataFrame, save_location: Union[str, bytes, os.PathLike], optional_settings: dict = {})

  Saves the processed dataset from a batch of videos.

  The summary dataset includes the fluid properties, the ‘raw’ diameter vs time data, and the sampleinfo from the filename.

  Parameters

    - df (pd.DataFrame) – Raw dataset with calculated columns for strain rate, strain, etc, and parsed sampleinfo columns.

    - save_location (path-like) – Path to folder in which to save the csv.

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - skip_existing (bool) – Determines the behavior when a file already appears exists when a function would generate it. True to skip any existing files. False to overwrite (or delete and then write, where overwriting would generate an error). Default is True.

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - summary_filename (string) – The base filename (no extension) for saving the summary csvs. If not provided, will be generated automatically based on the current date and time. Default is “” to trigger automatic generation.

  Returns

    - filename_string (string) – Filename at which the annotated dataframe was saved.

    - Saves file to disk.

dosertools.data_processing.fitting.save_summary_df(summary_df: pandas.core.frame.DataFrame, save_location: Union[str, bytes, os.PathLike], optional_settings: dict = {})

  Saves the summary dataset from a large processed batch of videos.

  The summary dataset includes only the fluid properties, not the ‘raw’ diameter vs time data.

  Parameters

    - summary_df (pd.DataFrame) – Contains LambdaE, D(tc)/D0, and sample info for multiple runs and samples

    - save_location (path-like) – path to folder in which to save the csv

    - filename (string) – Currently not used, but this will be the name to give to the summary dataset csv

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - skip_existing (bool) – Determines the behavior when a file already appears exists when a function would generate it. True to skip any existing files. False to overwrite (or delete and then write, where overwriting would generate an error). Default is True.

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - summary_filename (string) – The base filename (no extension) for saving the summary csvs. If not provided, will be generated automatically based on the current date and time. Default is “” to trigger automatic generation.

  Returns

    - filename_string (string) – Filename at which the summary dataframe was saved.

    - Saves file to disk.

## dosertools.data_processing.integration module

dosertools.data_processing.integration.binaries_to_csvs(images_folder: Union[str, bytes, os.PathLike], csv_folder: Union[str, bytes, os.PathLike], summary_folder: Union[str, bytes, os.PathLike], short_fname_format: str, sampleinfo_format: str, optional_settings: dict = {})

  Converts binary image folders into csvs of D/D0 vs. time.

  Given a folder of folders of binary images, converts each set of binary images into a csv of D/D0 vs. time, retaining information in the filename.

  Parameters

    - images_folder (path-like) – Path to a folder in which the results of image processing were saved (i.e. the folders of binary images).

    - csv_folder (path-like) – Path to a folder in which to save the csv containing D/D0 vs. time.

    - short_fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split with only tags present in the names of the folders in images_folder. Should have “vtype” and “remove” tags removed compared to videos_to_binaries. Must contain “fps” tag. ex. “date_sampleinfo_fps_run”

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - fname_split (string) – The deliminator for splitting folder/file names, used in fname_format. Default is “_”.

dosertools.data_processing.integration.csvs_to_summaries(csv_folder: Union[str, bytes, os.PathLike], summary_folder: Union[str, bytes, os.PathLike], short_fname_format: str, sampleinfo_format: str, optional_settings: dict = {})

  Processes the raw csvs and determines elongational relaxation time, D(tc)/D0, and elongational viscosity.

  Parameters

    - csv_folder (path-like) – Path to a folder in which to find the csv containing D/D0 vs. time.

    - summary_folder (path-like) – Path to a folder in which to save the csv of the summary and the annotated datatset

    - short_fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split. Format should not have “vtype” and “remove” tags–csvs will not have those formatting tags still attached. ex. “date_sampleinfo_fps_run”

    - sampleinfo_format (str) – The format of the sampleinfo section of the fname separated by the deliminator specified by sample_split.

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - fname_split (string) – The deliminator for splitting folder/file names, used in fname_format. Default is “_”.

    - sample_split (string) – The deliminator for splitting sampleinfo tag in folder/file names, used in sampleinfo_format. Default is “-“.

    - fitting_bounds (2 element list of floats) – [start, end] The D/D0 to bound the start and end of fitting of EC region. Default is [0.1, 0.045].

    - tc_bounds (2 element list of floats) – [start, end] The D/D0 to bound the start and end for finding the critical time. Default is [0.3,0.07].

dosertools.data_processing.integration.multiprocess_binaries_to_csvs(subfolder_index: int, subfolders: list, images_folder: Union[str, bytes, os.PathLike], csv_folder: Union[str, bytes, os.PathLike], short_fname_format: str, tic: float, optional_settings: dict = {}) → None

  Converts binary image folders into csvs of D/D0 vs. time.

  Given a folder of folders of binary images, converts each set of binary images into a csv of D/D0 vs. time, retaining information in the filename.

  For multiprocessing to work properly, the function that invokes the pool of processors and the function that uses them need to be defined separately. Thus, this function is here, and is called in binaries_to_csvs.

  Parameters

    - subfolder_index (int) – Index to keep track of which folder we are currently processing

    - subfolders (list of folders) – List of folders that contain the binaries that this function is reading to produce Diameter and time data

    - images_folder (path-like) – Path to a folder in which the results of image processing were saved (i.e. the folders of binary images).

    - csv_folder (path-like) – Path to a folder in which to save the csv containing D/D0 vs. time.

    - short_fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split with only tags present in the names of the folders in images_folder. Should have “vtype” and “remove” tags removed compared to videos_to_binaries. Must contain “fps” tag. ex. “date_sampleinfo_fps_run”

    - tic (float) – Stores the time that the processing began at. Used in verbose mode to determine how long processing takes

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - fname_split (string) – The deliminator for splitting folder/file names, used in fname_format. Default is “_”.

dosertools.data_processing.integration.multiprocess_vid_to_bin(file_number: int, fnames: list, exp_videos: list, bg_videos: list, images_folder: Union[str, bytes, os.PathLike], tic: float, optional_settings: dict = {}) → None

  Converts videos in given folder into binary images.

  Matches videos in videos_folder into experimental and background pairs, and converts those paired videos into background-subtracted binaries.

  For multiprocessing to work properly, the function that invokes the pool of processors and the function that uses them need to be defined separately. Thus, this function is here, and is called in videos_to_binaries.

  Parameters

    - file_number (int) – Index to keep track of which folder or video we are processing

    - fnames (list of strings) – List of base folder names for each matched pair of experimental and background folders.

    - exp_videos (list of paths) – List of paths to experimental video folders that were matched with backgrounds.

    - bg_videos (list of paths) – List of paths to background video folders matched with exp_videos.

    - images_folder (path-like) – Path to a folder in which to save the results of image processing, binaries and optional cropped and background-subtracted images.

    - tic (float) – Stores the time that the processing began at. Used in verbose mode to determine how long processing takes.

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - experiment_tag (string) – The tag for identifying experimental videos. May be empty (“”). Default is “exp”.

    - background_tag (string) – The tag for identifying background videos. May not be empty. Default is “bg”.

    - one_background (bool) – True to use one background for a group of experiments only differing by run number. False to pair backgrounds and experiments 1:1. Default is False.

    - save_crop (bool) – True to save intermediate cropped images (i.e. experimental video images cropped but not background-subtracted or binarized). Default is False.

    - save_bg_sub (bool) – True to save background-subtracted images (i.e. experimental video images cropped and background-subtracted but not binarized). Default is False.

    - skip_existing (bool) – Determines the behavior when a file already appears exists when a function would generate it. True to skip any existing files. False to overwrite (or delete and then write, where overwriting would generate an error). Default is True.

    - image_extension (string) – The extension for images in the video folder. TIFF recommended. Default is “tif”. Do not include “.”.

dosertools.data_processing.integration.set_defaults(optional_settings: dict = {}) → dict

  Sets default values for unset kets in optional_settings.

  Parameters

    optional_settings (dict) – Dictionary of optional settings.

  Returns

    settings (dict) – Dictionary with optional_settings and default values, prioritizing optional_settings values.

  Optional Settings and Defaults

    - nozzle_row (int) – Row to use for determining the nozzle diameter. Default is 1.

    - crop_width_coefficient (float) – Multiplied by the calculated nozzle_diameter to determine the buffer on either side of the observed nozzle edges to include in the cropped image. Default is 0.02

    - crop_height_coefficient (float) – Multiplied by the calculated nozzle_diameter to determine the bottom row that will be included in the cropped image. Default is 2.

    - crop_nozzle_coefficient (float) – Multiplied by the calculated nozzle_diameter to determine the top row of the cropped image. Default is 0.15.

    - fname_split (string) – The deliminator for splitting folder/file names, used in fname_format. Default is “_”.

    - sample_split (string) – The deliminator for splitting sampleinfo tag in folder/file names, used in sampleinfo_format. Default is “-“.

    - experiment_tag (string) – The tag for identifying experimental videos. May be empty (“”). Default is “exp”.

    - background_tag (string) – The tag for identifying background videos. May not be empty. Default is “bg”.

    - one_background (bool) – True to use one background for a group of experiments only differing by run number. False to pair backgrounds and experiments 1:1. Default is False.

    - bg_drop_removal (bool) – True to remove the background drop from the background that is subtracted from the image before binarization. False to not alter the background. Default is False.

    - save_crop (bool) – True to save intermediate cropped images (i.e. experimental video images cropped but not background-subtracted or binarized). Default is False.

    - save_bg_sub (bool) – True to save background-subtracted images (i.e. experimental video images cropped and background-subtracted but not binarized). Default is False.

    - fitting_bounds (2 element list of floats) – [start, end] The D/D0 to bound the start and end of fitting of EC region. Default is [0.1, 0.045].

    - tc_bounds (2 element list of floats) – [start, end] The D/D0 to bound the start and end for finding the critical time. Default is [0.3,0.07].

    - needle_diameter_mm (float) – The needle outer diameter in millimeters. Default is 0.7176 mm (22G needle).

    - skip_existing (bool) – Determines the behavior when a file already appears exists when a function would generate it. True to skip any existing files. False to overwrite (or delete and then write, where overwriting would generate an error). Default is True.

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - image_extension (string) – The extension for images in the video folder. TIFF recommended. Default is “tif”. Do not include “.”.

    - summary_filename (string) – The base filename (no extension) for saving the summary csvs. If not provided, will be generated automatically based on the current date and time. Default is “” to trigger automatic generation.

    - cpu_count (int) – How many cores to use for multithreading/multiprocessing. If nothing provided, default will be the maximum number of cores returned from os.cpu_count()

dosertools.data_processing.integration.videos_to_binaries(videos_folder: Union[str, bytes, os.PathLike], images_folder: Union[str, bytes, os.PathLike], fname_format: str, optional_settings: dict = {})

  Converts videos in given folder into binary images.

  Matches videos in videos_folder into experimental and background pairs, and converts those paired videos into background-subtracted binaries.

  Parameters

    - videos_folder (path-like) – Path to a folder of experimental and background video folders.

    - images_folder (path-like) – Path to a folder in which to save the results of image processing, binaries and optional cropped and background-subtracted images.

    - fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split. Must contain the “vtype” tag corresponding to experiment vs. background. Can contain “remove” to remove information that is not relevant or is different between the experimental and background video names and would prevent matching. ex. “date_sampleinfo_fps_run_vtype_remove_remove”

    - sampleinfo_format (str) – The format of the sampleinfo section of the fname separated by the deliminator specified by sample_split.

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - cpu_count (int) – How many cores to use for multithreading/multiprocessing. If nothing provided, default will be the maximum number of cores returned from os.cpu_count()

    - experiment_tag (string) – The tag for identifying experimental videos. May be empty (“”). Default is “exp”.

    - background_tag (string) – The tag for identifying background videos. May not be empty. Default is “bg”.

    - one_background (bool) – True to use one background for a group of experiments only differing by run number. False to pair backgrounds and experiments 1:1. Default is False.

    - save_crop (bool) – True to save intermediate cropped images (i.e. experimental video images cropped but not background-subtracted or binarized). Default is False.

    - save_bg_sub (bool) – True to save background-subtracted images (i.e. experimental video images cropped and background-subtracted but not binarized). Default is False.

    - skip_existing (bool) – Determines the behavior when a file already appears exists when a function would generate it. True to skip any existing files. False to overwrite (or delete and then write, where overwriting would generate an error). Default is True.

    - image_extension (string) – The extension for images in the video folder. TIFF recommended. Default is “tif”. Do not include “.”.

dosertools.data_processing.integration.videos_to_csvs(videos_folder: Union[str, bytes, os.PathLike], images_folder: Union[str, bytes, os.PathLike], csv_folder: Union[str, bytes, os.PathLike], summary_folder: Union[str, bytes, os.PathLike], fname_format: str, sampleinfo_format: str, optional_settings: dict = {})

  Converts videos in given folder into csvs of D/D0 vs. time.

  Matches videos in videos_folder into experimental and background pairs, converts those paired videos into background-subtracted binaries, analyzes the resulting binaries to extract D/D0 vs. time, and saves the results to csvs.

  Parameters

    - videos_folder (path-like) – Path to a folder of experimental and background video folders.

    - images_folder (path-like) – Path to a folder in which to save the results of image processing, binaries and optional cropped and background-subtracted images.

    - csv_folder (path-like) – Path to a folder in which to save the csv containing D/D0 vs. time.

    - fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split. Must contain the “vtype” tag corresponding to experiment vs. background. Can contain “remove” to remove information that is not relevant or is different between the experimental and background video names and would prevent matching. Must contain “fps” tag. ex. “date_sampleinfo_fps_run_vtype_remove_remove”

    - optional_settings (dict) – A dictionary of optional settings.

  Optional Settings and Defaults

    - fname_split (string) – The deliminator for splitting folder/file names, used in fname_format. Default is “_”.

    - experiment_tag (string) – The tag for identifying experimental videos. May be empty (“”). Default is “exp”.

    - background_tag (string) – The tag for identifying background videos. May not be empty. Default is “bg”.

    - one_background (bool) – True to use one background for a group of experiments only differing by run number. False to pair backgrounds and experiments 1:1. Default is False.

    - save_crop (bool) – True to save intermediate cropped images (i.e. experimental video images cropped but not background-subtracted or binarized). Default is False.

    - save_bg_sub (bool) – True to save background-subtracted images (i.e. experimental video images cropped and background-subtracted but not binarized). Default is False.

    - skip_existing (bool) – Determines the behavior when a file already appears exists when a function would generate it. True to skip any existing files. False to overwrite (or delete and then write, where overwriting would generate an error). Default is True.

    - verbose (bool) – Determines whether processing functions print statements as they progress through major steps. True to see print statements, False to hide non-errors/warnings. Default is False.

    - image_extension (string) – The extension for images in the video folder. TIFF recommended. Default is “tif”. Do not include “.”.

dosertools.data_processing.integration.videos_to_summaries(videos_folder: Union[str, bytes, os.PathLike], images_folder: Union[str, bytes, os.PathLike], csv_folder: Union[str, bytes, os.PathLike], summary_folder: Union[str, bytes, os.PathLike], fname_format: str, sampleinfo_format: str, optional_settings: dict = {})

  Full integrating function: converts from videos to csv files

  Parameters

    - videos_folder (path-like) – Path to a folder of experimental and background video folders.

    - images_folder (path-like) – Path to a folder in which to save the results of image processing, binaries and optional cropped and background-subtracted images.

    - csv_folder (path-like) – Path to a folder in which to save the csv containing D/D0 vs. time.

    - summary_save_location (path-like) – Path to a folder in which to save the csv of the summary and the annotated datatset

    - fname_format (str) – The format of the fname with parameter names separated by the deliminator specified by fname_split. Must contain the “vtype” tag corresponding to experiment vs. background. Can contain “remove” to remove information that is not relevant or is different between the experimental and background video names and would prevent matching. ex. “date_sampleinfo_fps_run_vtype_remove_remove”

    - sampleinfo_format (str) – The format of the sampleinfo section of the fname separated by the deliminator specified by sample_split.

    - optional_settings (dict) – A dictionary of optional settings.

## Module contents