Utilities for Process and Thread Management
The package ka_uts_prc can be installed from PyPI or Anaconda.
To install with pip:
$ python -m pip install ka_uts_prc
To install with conda:
$ conda install -c conda-forge ka_uts_prc
This requires that the readme extra is installed:
$ python -m pip install ka_uts_prc[readme]
The Package ka_uts_prc contains the subsequent modules.
Modules of package ka_uts_prc* Module Name Description prc.py Management of "array of arrays". run.py Array of Dictionaries
Processing Module with classes Module Static Classes Name Type Name Description prc Task runner for iterator Ra Process Task decorated by the ray remote decorator for all iterator items. Co Process Task using concurrent futures with process- or thread-pool. Mp Process Tasks using multiprocessing with process- or thread-pool Sp Process Task sequentially for all iterator items. Prc Create iterator for path-templates optionally modified by transformer functions (if defined). Process Tasks by selecting the run-, process- or thread-function of one of the Processing classes Ra, Co, Mp, Sp with the run_method using the function table of run methods: d_run_method_2_fnc.
Special-Modules without classes Module Type Statements __init__ Package directory marker Module with no Statements; the module is used to mark the directory it contains as a package. A dummy Module enforces explicit imports and thus clear namespace use and call them with the dot dot notation __version__ Version Assignment Statements for Version System Variables
The Modul prc_by_dir contains the subsequent static classes with static variables and methods
Classes of module prc_by_dir Static Classes Name Description Ra Execute tasks using Ray Co Execute tasks using Concurrent futures Mp Execute tasks using multiprocessing Sp Execute tasks using single processing Prc Run tasks with selected method
Ra static Methods of prc Static Method Parameter: P / Return Value: RV Name Description Name Type P/RV Description process Run tasks in parallel by executing the task decorated with with the ray remote decorator for every iterator item cls class P current class kwargs T_Dic P keyword arguments None RV no return value
Co static Methods of prc Static Method Parameter: P / Return Value: RV Name Description Name Type P/RV Description process Run tasks in parallel by applying the map method of the concurrent futures process pool executor to the task and the iterator. cls class P current class kwargs T_Dic P keyword arguments None RV no return value thread Run tasks in parallel by applying the map method of the concurrent futures thread pool executor to the task and the iterator. cls class P current class kwargs T_Dic P keyword arguments None RV no return value
Mp static Methods of prc Static Method Parameter: P / Return Value: RV Name Description Name Type P/RV Description process Run tasks in parallel by applying the map method of the multiprocessing process pool to the task and the iterator. cls class P current class kwargs T_Dic P keyword arguments None RV no return value thread Run tasks in parallel by applying the map method of the multiprocessing thread pool to the task and the iterator. cls class P current class kwargs T_Dic P keyword arguments None RV no return value
Sp static Methods of prc Static Method Parameter: P / Return Value: RV Name Description Name Type P/RV Description run Run tasks sequentially for all items of the iterator cls class P current class kwargs T_Dic P keyword arguments None RV no return value
Prc static Variables of prc Static Variable Name Type Description d_run_method_2_fnc T_Dic Dictionary to translate run methods into run functions
Prc static Methods of prc Static Method Parameter: P / Return Value: RV Name Description Name Type P/RV Description do Select executer with run method and run method function table. cls class P current class Create iterator using the Task function: yield_dl_template. kwargs T_Dic P keyword arguments Run the Task function: task with the executor. None RV no return value
The Modul __version__ contains no classes, but assignment statements for system variables used by versioning.
__version_ System Variables System Variable Example __title__ 'ka_uts_prc' __description__ 'Processing Utilities.' __url__ 'https://ka-com.readthedocs.io/en/latest' __version___ '1.0.0.24.08.24' __build__ 0x022200 __author_email_ 'Bernd Stroehle' __license__ 'Apache-2.0' __copyright__ 'Copyright 2024 Bernd Stroehle __cake__ u'u2728 U0001f370 u2728'
System Variable Description log.main.tenant.yml __copyright__ 'ka_uts_prc' 'Copyright 2023 bs29'
The Standard or user specifig logging is carried out by the log.py module of the logging package ka_uts_log using the standard- or user-configuration files in the logging package configuration directory:
The Logging configuration of the logging package could be overriden by yaml files with the same names in the application package- or application data-configuration directories:
Logging defines log file path names for the following log message types: .
Single or multiple Application log directories can be used for each message type:
Log types and directoriesg Log type Log directory long short multiple single debug dbqs dbqs logs info infs infs logs warning wrns wrns logs error errs errs logs critical crts crts logs
Application parameter used in log naming Name Decription Values Example dir_dat Application data directory /otev/data tenant Application tenant name UMH package Application package name otev_xls_srr cmd Application command evupreg pid Process ID 681025 log_ts_type Timestamp type used in logging files|ts, dt ts, dt' ts log_sw_single_dir Enable single log directory or multiple log directories True, False True
Naming conventions for logging file paths Type Directory File debug /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> <Log type>_<ts>_<pid>.log info /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> <Log type>_<ts>_<pid>.log warning /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> <Log type>_<ts>_<pid>.log error /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> <Log type>_<ts>_<pid>.log critical /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> <Log type>_<ts>_<pid>.log
Naming examples for logging file paths Type Directory File debug /data/otev/umh/RUN/otev_xls_srr/evupreg/logs debs_1737118199_9470.log info /data/otev/umh/RUN/otev_xls_srr/evupreg/logs infs_1737118199_9470.log warning /data/otev/umh/RUN/otev_xls_srr/evupreg/logs wrns_1737118199_9470.log error /data/otev/umh/RUN/otev_xls_srr/evupreg/logs errs_1737118199_9470.log critical /data/otev/umh/RUN/otev_xls_srr/evupreg/logs crts_1737118199_9470.log
Python Packages Overview Name Definition Python package Python packages are directories that contains the special module __init__.py and other modules, packages files or directories. Python sub-package Python sub-packages are python packages which are contained in another pyhon package. Python package sub-directory directory contained in a python package. Python package special sub-directory Python package sub-directories with a special meaning like data or cfg
Python Package sub-directory-Examples Name Description bin Directory for package scripts. cfg Directory for package configuration files. data Directory for package data files. service Directory for systemd service scripts.
Python package overview files Name Definition Python package files Files within a python package. Special python package files Package files which are not modules and used as python and used as python marker files like __init__.py. Python package module Files with suffix .py; they could be empty or contain python code; other modules can be imported into a module. Special python package module Modules like __init__.py or main.py with special names and functionality.
Python package examples files Name Type Description py.typed Type checking marker file The py.typed file is a marker file used in Python packages to indicate that the package supports type checking. This is a part of the PEP 561 standard, which provides a standardized way to package and distribute type information in Python. __init__.py Package directory marker file The dunder (double underscore) module __init__.py is used to execute initialisation code or mark the directory it contains as a package. The Module enforces explicit imports and thus clear namespace use and call them with the dot notation. __main__.py entry point for the package The dunder module __main__.py serves as an entry point for the package. The module is executed when the package is called by the interpreter with the command python -m <package name>. __version__.py Version file The dunder module __version__.py consist of assignment statements used in Versioning.
Python methods overview Name Description Python method Python functions defined in python modules. Special python method Python functions with special names and functionalities. Python class Classes defined in python modules. Python class method Python methods defined in python classes
Python methods examples Name Type Description __init__ class object constructor method The special method __init__ is called when an instance (object) of a class is created; instance attributes can be defined and initalized in the method.