Metadata-Version: 2.4
Name: ka_uts_prc
Version: 2.1.1.250415
Summary: Processing Utilities
Author-email: Bernd Stroehle <bernd.stroehle@gmail.com>
Maintainer-email: Bernd Stroehle <bernd.stroehle@gmail.com>
Project-URL: Source Code, https://github.com/bs29/ka_uts_prc/tree/master
Project-URL: Homepage, https://kosakya.de/
Project-URL: Documentation, https://ka-apc.readthedocs.io/en/latest
Project-URL: Apache-2.0 License, https://apache.org/licenses/LICENSE-2.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Natural Language :: English
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
Requires-Dist: joblib>=1.4.2
Requires-Dist: ka_uts_com>=1.0.1.250131
Dynamic: license-file

##########
ka_uts_prc
##########

Overview
********

.. start short_desc

**Processing Utilities**

.. end short_desc

Installation
************
.. start installation

The package ``ka_uts_prc`` can be installed from PyPI or Anaconda.

To install with ``pip``:

.. code-block:: shell

	$ python -m pip install ka_uts_prc

To install with ``conda``:

.. code-block:: shell

	$ conda install -c conda-forge ka_uts_prc

.. end installation

This requires that the ``readme`` extra is installed:

.. code-block:: bash

	$ python -m pip install ka_uts_prc[readme]

Modules
*******

The Package ``ka_uts_prc`` contains the subsequent modules.

  .. Modules-of-package-ka_uts_prc-label:
  .. table:: Modules of package ka_uts_prc*

   +---------------------------------------------+
   |Module                                       |
   +--------+------------------------------------+
   |Name    |Description                         |
   +========+====================================+
   |prc.py  |Management of "array of arrays".    |
   +--------+------------------------------------+
   |run.py  |Array of Dictionaries               |
   +--------+------------------------------------+

  .. processing-module-with-classes-label:
  .. table:: *Processing Module with classes*

   +-----------------------------------+----------------------------------------------+
   |Module                             |Static Classes                                |
   +-------------------+---------------+----+-----------------------------------------+
   |Name               |Type           |Name|Description                              |
   +===================+===============+====+=========================================+
   |prc                |Task runner    |Ra  |Process Task decorated by the ray remote |
   |                   |for iterator   |    |decorator for all iterator items.        |
   |                   |               +----+-----------------------------------------+
   |                   |               |Co  |Process Task using concurrent futures    |
   |                   |               |    |with process- or thread-pool.            |
   |                   |               +----+-----------------------------------------+
   |                   |               |Mp  |Process Tasks using multiprocessing with |
   |                   |               |    |process- or thread-pool                  |
   |                   |               +----+-----------------------------------------+
   |                   |               |Sp  |Process Task sequentially for all        |
   |                   |               |    |iterator items.                          |
   |                   |               +----+-----------------------------------------+
   |                   |               |Prc |Create iterator for path-templates       |
   |                   |               |    |optionally modified by transformer       |
   |                   |               |    |functions (if defined).                  |
   |                   |               |    +-----------------------------------------+
   |                   |               |    |Process Tasks by selecting the run-,     |
   |                   |               |    |process- or thread-function of one of the|
   |                   |               |    |Processing classes Ra, Co, Mp, Sp with   |
   |                   |               |    |the run_method using the function table  |
   |                   |               |    |of run methods: d_run_method_2_fnc.      |
   +-------------------+---------------+----+-----------------------------------------+

  .. special-modules-without-classes-label:
  .. table:: *Special-Modules without classes*

   +-----------+----------------+------------------------------------------------+
   |Module     |Type            |Statements                                      |
   +===========+================+================================================+
   |__init__   |Package         |Module with no Statements; the module is used to|
   |           |directory       |mark the directory it contains as a package. A  | 
   |           |marker          |dummy Module enforces explicit imports and thus |
   |           |                |clear namespace use and call them with the dot  |
   |           |                |dot notation                                    |
   +-----------+----------------+------------------------------------------------+
   |__version__|Version         |Assignment Statements for Version System        |
   |           |                |Variables                                       |
   +-----------+----------------+------------------------------------------------+

Processing Modul: prc_by_dir
============================

The Modul ``prc_by_dir`` contains the subsequent static classes with static variables and methods

  .. Static-classes-of_module-prc_by_dir-label:
  .. table:: *Classes of module prc_by_dir*

   +---------------------------------------------+
   |Static Classes                               |
   +-----+---------------------------------------+
   |Name |Description                            |
   +=====+=======================================+
   |Ra   |Execute tasks using Ray                |
   +-----+---------------------------------------+
   |Co   |Execute tasks using Concurrent futures |
   +-----+---------------------------------------+
   |Mp   |Execute tasks using multiprocessing    |
   +-----+---------------------------------------+
   |Sp   |Execute tasks using single processing  |
   +-----+---------------------------------------+
   |Prc  |Run tasks with selected method         |
   +-----+---------------------------------------+

static Class: Ra
----------------

static Methods
^^^^^^^^^^^^^^

  .. ra-static-methods-of-prc-label:
  .. table:: *Ra static Methods of prc*

   +-----------------------------------------+-----------------------------------+
   |Static Method                            |Parameter\: P / Return Value\: RV  |
   +-------+---------------------------------+------+-----+----+-----------------+
   |Name   |Description                      |Name  |Type |P/RV|Description      |
   +=======+=================================+======+=====+====+=================+
   |process|Run tasks in parallel by         |cls   |class|P   |current class    |
   |       |executing the task decorated with+------+-----+----+-----------------+
   |       |with the ray remote decorator for|kwargs|T_Dic|P   |keyword arguments|
   |       |every iterator item              +------+-----+----+-----------------+
   |       |                                 |      |None |RV  |no return value  |
   +-------+---------------------------------+------+-----+----+-----------------+

static Class: Co
----------------

static Methods
^^^^^^^^^^^^^^

  .. co-static-methods-of-prc-label:
  .. table:: *Co static Methods of prc*

   +-----------------------------------------+-----------------------------------+
   |Static Method                            |Parameter\: P / Return Value\: RV  |
   +-------+---------------------------------+------+-----+----+-----------------+
   |Name   |Description                      |Name  |Type |P/RV|Description      |
   +=======+=================================+======+=====+====+=================+
   |process|Run tasks in parallel by applying|cls   |class|P   |current class    |
   |       |the map method of the concurrent +------+-----+----+-----------------+
   |       |futures process pool executor to |kwargs|T_Dic|P   |keyword arguments|
   |       |the task and the iterator.       +------+-----+----+-----------------+
   |       |                                 |      |None |RV  |no return value  |
   +-------+---------------------------------+------+-----+----+-----------------+
   |thread |Run tasks in parallel by applying|cls   |class|P   |current class    |
   |       |the map method of the concurrent +------+-----+----+-----------------+
   |       |futures thread pool executor to  |kwargs|T_Dic|P   |keyword arguments|
   |       |the task and the iterator.       +------+-----+----+-----------------+
   |       |                                 |      |None |RV  |no return value  |
   +-------+---------------------------------+------+-----+----+-----------------+

static Class: Mp
----------------

static Methods
^^^^^^^^^^^^^^

  .. mp-static-methods-of-prc_label:
  .. table:: *Mp static Methods of prc*

   +-----------------------------------------+-----------------------------------+
   |Static Method                            |Parameter\: P / Return Value\: RV  |
   +-------+---------------------------------+------+-----+----+-----------------+
   |Name   |Description                      |Name  |Type |P/RV|Description      |
   +=======+=================================+======+=====+====+=================+
   |process|Run tasks in parallel by applying|cls   |class|P   |current class    |
   |       |the map method of the            +------+-----+----+-----------------+
   |       |multiprocessing process pool to  |kwargs|T_Dic|P   |keyword arguments|
   |       |the task and the iterator.       +------+-----+----+-----------------+
   |       |                                 |      |None |RV  |no return value  |
   +-------+---------------------------------+------+-----+----+-----------------+
   |thread |Run tasks in parallel by applying|cls   |class|P   |current class    |
   |       |the map method of the            +------+-----+----+-----------------+
   |       |multiprocessing thread pool to   |kwargs|T_Dic|P   |keyword arguments|
   |       |the task and the iterator.       +------+-----+----+-----------------+
   |       |                                 |      |None |RV  |no return value  |
   +-------+---------------------------------+------+-----+----+-----------------+

static Class: Sp
----------------

static Methods
^^^^^^^^^^^^^^

  .. sp-static-methods-of-prc-label:
  .. table:: *Sp static Methods of prc*

   +-----------------------------------------+-----------------------------------+
   |Static Method                            |Parameter\: P / Return Value\: RV  |
   +-------+---------------------------------+------+-----+----+-----------------+
   |Name   |Description                      |Name  |Type |P/RV|Description      |
   +=======+=================================+======+=====+====+=================+
   |run    |Run tasks sequentially for all   |cls   |class|P   |current class    |
   |       |items of the iterator            +------+-----+----+-----------------+
   |       |                                 |kwargs|T_Dic|P   |keyword arguments|
   |       |                                 +------+-----+----+-----------------+
   |       |                                 |      |None |RV  |no return value  |
   +-------+---------------------------------+------+-----+----+-----------------+

static Class: Prc
-----------------

static Variables
^^^^^^^^^^^^^^^^

  .. prc-static-variables-of-prc-label:
  .. table:: *Prc static Variables of prc*


   +-----------------------------------------------------------+
   |Static Variable                                            |
   +------------------+------+---------------------------------+
   |Name              |Type  |Description                      |
   +========+=========+======+=================================+
   |d_run_method_2_fnc|T_Dic |Dictionary to translate run      |
   |                  |      |methods into run functions       |
   +------------------+------+---------------------------------+

static Methods
^^^^^^^^^^^^^^

  .. prc-static-methods-of-prc-label:
  .. table:: *Prc static Methods of prc*

   +-------------------------------------+-----------------------------------+
   |Static Method                        |Parameter\: P / Return Value\: RV  |
   +-----+-------------------------------+------+-----+----+-----------------+
   |Name |Description                    |Name  |Type |P/RV|Description      |
   +=====+===============================+======+=====+====+=================+
   |do   |Select executer with run method|cls   |class|P   |current class    |
   |     |and run method function table. |      |     |    |                 |
   |     +-------------------------------+------+-----+----+-----------------+
   |     |Create iterator using the Task |kwargs|T_Dic|P   |keyword arguments|
   |     |function: yield_dl_template.   |      |     |    |                 |
   |     +-------------------------------+------+-----+----+-----------------+
   |     |Run the Task function: task    |      |None |RV  |no return value  |
   |     |with the executor.             |      |     |    |                 |
   +-----+-------------------------------+------+-----+----+-----------------+

Special Modul: __version__
==========================

The Modul ``__version__`` contains no classes, but assignment statements for
system variables used by versioning.

  .. modul-__version__-system-variables:
  .. table:: *__version_ System Variables*

   +---------------+-----------------------------------------+
   |System Variable|Example                                  |
   +===============+=========================================+
   |__title__      |'ka_uts_prc'                             |
   +---------------+-----------------------------------------+
   |__description__|'Processing Utilities.'                  |
   +---------------+-----------------------------------------+
   |__url__        |'https://ka-com.readthedocs.io/en/latest'|
   +---------------+-----------------------------------------+
   |__version___   |'1.0.0.24.08.24'                         |
   +---------------+-----------------------------------------+
   |__build__      |0x022200                                 |
   +---------------+-----------------------------------------+
   |__author_email_|'Bernd Stroehle'                         |
   +---------------+-----------------------------------------+
   |__license__    |'Apache-2.0'                             |
   +---------------+-----------------------------------------+
   |__copyright__  |'Copyright 2024 Bernd Stroehle           |
   +---------------+-----------------------------------------+
   |__cake__       |u'\u2728 \U0001f370 \u2728'              |
   +---------------+-----------------------------------------+

Module Data
===========

   +-------------------+-------------------------------------+
   |System Variable    |Description                          |
   +===================+=====================================+
   |log.main.tenant.yml|'ka_uts_prc'                         |
   |__copyright__      |'Copyright 2023 bs29'                |
   +-------------------+-------------------------------------+

Appendix
********

Package Logging
===============

Description
-----------

The Standard or user specifig logging is carried out by the log.py module of the logging
package ka_uts_log using the configuration files **ka_std_log.yml** or **ka_usr_log.yml**
in the configuration directory **cfg** of the logging package **ka_uts_log**.
The Logging configuration of the logging package could be overriden by yaml files with
the same names in the configuration directory **cfg** of the application packages.

Log message types
-----------------

Logging defines log file path names for the following log message types: .

#. *debug*
#. *info*
#. *warning*
#. *error*
#. *critical*

Application parameter for logging
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  .. Application-parameter-used-in-log-naming-label:
  .. table:: *Application parameter used in log naming*

   +-----------------+---------------------------+----------+------------+
   |Name             |Decription                 |Values    |Example     |
   +=================+===========================+==========+============+
   |dir_dat          |Application data directory |          |/otev/data  |
   +-----------------+---------------------------+----------+------------+
   |tenant           |Application tenant name    |          |UMH         |
   +-----------------+---------------------------+----------+------------+
   |package          |Application package name   |          |otev_xls_srr|
   +-----------------+---------------------------+----------+------------+
   |cmd              |Application command        |          |evupreg     |
   +-----------------+---------------------------+----------+------------+
   |pid              |Process ID                 |          |æevupreg    |
   +-----------------+---------------------------+----------+------------+
   |log_ts_type      |Timestamp type used in     |ts,       |ts          |
   |                 |logging files|ts, dt       |dt        |            |
   +-----------------+---------------------------+----------+------------+
   |log_sw_single_dir|Enable single log directory|True,     |True        |
   |                 |or multiple log directories|False     |            |
   +-----------------+---------------------------+----------+------------+

Log type and Log directories
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Single or multiple Application log directories can be used for each message type:

  .. Log-types-and-Log-directories-label:
  .. table:: *Log types and directoriesg*

   +--------------+---------------+
   |Log type      |Log directory  |
   +--------+-----+--------+------+
   |long    |short|multiple|single|
   +========+=====+========+======+
   |debug   |dbqs |dbqs    |logs  |
   +--------+-----+--------+------+
   |info    |infs |infs    |logs  |
   +--------+-----+--------+------+
   |warning |wrns |wrns    |logs  |
   +--------+-----+--------+------+
   |error   |errs |errs    |logs  |
   +--------+-----+--------+------+
   |critical|crts |crts    |logs  |
   +--------+-----+--------+------+

Log files naming
^^^^^^^^^^^^^^^^

Conventions
"""""""""""

  .. Naming-conventions-for-logging-file-paths-label:
  .. table:: *Naming conventions for logging file paths*

   +--------+-------------------------------------------------------+-------------------------+
   |Type    |Directory                                              |File                     |
   +========+=======================================================+=========================+
   |debug   |/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>|<Log type>_<ts>_<pid>.log|
   +--------+-------------------------------------------------------+-------------------------+
   |info    |/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>|<Log type>_<ts>_<pid>.log|
   +--------+-------------------------------------------------------+-------------------------+
   |warning |/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>|<Log type>_<ts>_<pid>.log|
   +--------+-------------------------------------------------------+-------------------------+
   |error   |/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>|<Log type>_<ts>_<pid>.log|
   +--------+-------------------------------------------------------+-------------------------+
   |critical|/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>|<Log type>_<ts>_<pid>.log|
   +--------+-------------------------------------------------------+-------------------------+

Examples
""""""""

  .. Naming-examples-for-logging-file-paths-label:
  .. table:: *Naming examples for logging file paths*

   +--------+--------------------------------------------+------------------------+
   |Type    |Directory                                   |File                    |
   +========+============================================+========================+
   |debug   |/data/otev/umh/RUN/otev_xls_srr/evupreg/logs|debs_1737118199_9470.log|
   +--------+--------------------------------------------+------------------------+
   |info    |/data/otev/umh/RUN/otev_xls_srr/evupreg/logs|infs_1737118199_9470.log|
   +--------+--------------------------------------------+------------------------+
   |warning |/data/otev/umh/RUN/otev_xls_srr/evupreg/logs|wrns_1737118199_9470.log|
   +--------+--------------------------------------------+------------------------+
   |error   |/data/otev/umh/RUN/otev_xls_srr/evupreg/logs|errs_1737118199_9470.log|
   +--------+--------------------------------------------+------------------------+
   |critical|/data/otev/umh/RUN/otev_xls_srr/evupreg/logs|crts_1737118199_9470.log|
   +--------+--------------------------------------------+------------------------+

Python Terminology
==================

Python package
--------------

Overview
^^^^^^^^

  .. Python package-label:
  .. table:: *Python package*

   +--------------+-----------------------------------------------------------------+
   |Name          |Definition                                                       |
   +==============+==========+======================================================+
   |Python package|Python packages are directories that contains the special module |
   |              |``__init__.py`` and other modules, packages files or directories.|
   +--------------+-----------------------------------------------------------------+
   |Python        |Python sub-packages are python packages which are contained in   |
   |sub-package   |another pyhon package.                                           |
   +--------------+-----------------------------------------------------------------+

Python package sub-directories
------------------------------

Overview
^^^^^^^^

  .. Python package sub-direcories-label:
  .. table:: *Python package sub-directories*

   +--------------+-----------------------------------------+
   |Name          |Definition                               |
   +==============+==========+==============================+
   |Python package|Python packages sub-directories are      |
   |sub-directory |directories contained in python packages.|
   +--------------+-----------------------------------------+
   |Special Python|Special Python package sub-directories   |
   |package       |are python package sub-directories with  |
   |sub-directory |with a special meaning                   |
   +--------------+-----------------------------------------+

Special python package sub-directories
--------------------------------------

Overview
^^^^^^^^

  .. Special-python-package-sub-directories-label:
  .. table:: *Special python sun-directories*

   +----+------------------------------------------+
   |Name|Description                               |
   +====+==========================================+
   |data|Directory for package data files.         |
   +----+------------------------------------------+
   |cfg |Directory for package configuration files.|
   +----+------------------------------------------+

Python package files
--------------------

Overview
^^^^^^^^

  .. Python-package-files-label:
  .. table:: *Python package files*

   +--------------+--------------------------------------------------------------------+
   |Name          |Definition                                                          |
   +==============+==========+=========================================================+
   |Python        |Python packages are files within a python package.                  |
   |package files |                                                                    |
   +--------------+--------------------------------------------------------------------+
   |Special python|Special python package files are package files which are not modules|
   |package files |and used as python marker files like ``__init__.py``                |
   +--------------+--------------------------------------------------------------------+
   |Python package|Python modules are files with suffix ``.py``; they could be empty or|
   |module        |contain python code; other modules can be imported into a module.   |
   +--------------+--------------------------------------------------------------------+
   |Special python|Special python modules like ``__init__.py`` or ``main.py`` are      |
   |package module|python modules with special names and functionality.                |
   +--------------+--------------------------------------------------------------------+

Special python package files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Overview
°°°°°°°°

  .. Special-python-package-files-label:
  .. table:: *Special python package files*

   +--------+--------+---------------------------------------------------------------+
   |Name    |Type    |Description                                                    |
   +========+========+===============================================================+
   |py.typed|Type    |The ``py.typed`` file is a marker file used in Python packages |
   |        |checking|to indicate that the package supports type checking. This is a |
   |        |marker  |part of the PEP 561 standard, which provides a standardized way|
   |        |file    |to package and distribute type information in Python.          |
   +--------+--------+---------------------------------------------------------------+

Special python package modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Overview
°°°°°°°°

  .. Special-Python-package-modules-label:
  .. table:: *Special Python package modules*

   +--------------+-----------+-----------------------------------------------------------------+
   |Name          |Type       |Description                                                      |
   +==============+===========+=================================================================+
   |__init__.py   |Package    |The dunder (double underscore) module ``__init__.py`` is used to |
   |              |directory  |execute initialisation code or mark the directory it contains as |
   |              |marker     |a package. The Module enforces explicit imports and thus clear   |
   |              |file       |namespace use and call them with the dot notation.               |
   +--------------+-----------+-----------------------------------------------------------------+
   |__main__.py   |entry point|The dunder module ``__main__.py`` serves as an entry point for   |
   |              |for the    |the package. The module is executed when the package is called by|
   |              |package    |the interpreter with the command **python -m <package name>**.   |
   +--------------+-----------+-----------------------------------------------------------------+
   |__version__.py|Version    |The dunder module ``__version__.py`` consist of assignment       |
   |              |file       |statements used in Versioning.                                   |
   +--------------+-----------+-----------------------------------------------------------------+

Python elements
---------------

Overview
°°°°°°°°

  .. Python elements-label:
  .. table:: *Python elements*

   +-------------+--------------------------------------------------------------+
   |Python method|Python methods are python functions defined in python modules.|
   +-------------+--------------------------------------------------------------+
   |Special      |Special python methods are python functions with special names|
   |python method|and functionalities.                                          |
   +-------------+--------------------------------------------------------------+
   |Python class |Python classes are defined in python modules.                 |
   +-------------+--------------------------------------------------------------+
   |Python class |Python class methods are python methods defined python        |
   |method       |classes.                                                      |
   +-------------+--------------------------------------------------------------+

Special python methods
^^^^^^^^^^^^^^^^^^^^^^

Overview
°°°°°°°°

  .. Special-python-methods-label:
  .. table:: *Special python methods*

   +--------+------------+----------------------------------------------------------+
   |Name    |Type        |Description                                               |
   +========+============+==========================================================+
   |__init__|class object|The special method ``__init__`` is called when an instance|
   |        |constructor |(object) of a class is created; instance attributes can be|
   |        |method      |defined and initalized in the method.                     |
   +--------+------------+----------------------------------------------------------+

Table of Contents
=================

.. contents:: **Table of Content**
