Metadata-Version: 2.4
Name: pymemtrace
Version: 0.4.0
Summary: Python memory tracing.
Home-page: https://github.com/paulross/pymemtrace
Author: Paul Ross
Author-email: apaulross@gmail.com
License: MIT license
Keywords: pymemtrace
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: C
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Software Development
Description-Content-Type: text/x-rst
License-File: LICENSE
License-File: AUTHORS.rst
Requires-Dist: psutil
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

*******************
Introduction
*******************


``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different
granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools
======================

The tools provided by ``pymemtrace`` are:

``process_tree``
----------------

A command line tool that shows the total memory usage of a process *and* its child processes
at regular time intervals.
It can log this data to a JSON file for later analysis.
See `some process_tree examples <https://pymemtrace.readthedocs.io/en/latest/examples/process_tree.html>`_

``process``
-----------

A very lightweight way of logging the total memory usage of a single process at regular time intervals.
It can plot memory over time with plotting programs such as ``gnuplot``.
See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_

``cPyMemTrace``
---------------

This module, written in C, provides real time logging of Python and C actions:

- ``pymemtrace.cPyMemTrace.Profile`` is a memory tracer written in C that can report total memory usage
  for every function call/return for both C and Python sections.
  This is more suitable for logging C code, for example Python's C extensions.
- ``pymemtrace.cPyMemTrace.Trace`` is a memory tracer written in C that can report total memory usage
  for every function call/return/line for Python sections.
  This is more suitable for logging pure Python code.
- ``pymemtrace.cPyMemTrace.ReferenceTracing`` can report every object allocation
  and de-allocation with
  `Reference Tracing <https://docs.python.org/3/c-api/profiling.html#reference-tracing>`_.
  This is quite invasive but the API allows this to filter out most of the noise or target specific types
  of interest.
  (Python 3.13+ only).

See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_
and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.

DTrace
------

There are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and
report how much memory was allocated and by whom.
See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_
and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.

``trace_malloc``
----------------

A convenience wrapper around the Python standard library `tracemalloc` module.
This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.
This can take memory snapshots before and after code blocks and show the change on memory caused by that code.
See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_

``debug_malloc_stats``
----------------------

Awrapper around the ``sys._debugmallocstats`` function that can take snapshots of
memory before and after code execution and report the significant differences of the Python small object allocator.
See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_

Tool Characteristics
======================

Each tool can be characterised by:

- *Memory Granularity*: In how much detail is a memory change is observed.
  An example of *coarse* memory granularity is measuring the
  `Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ (RSS) which is normally in
  chunks of 4096 bytes.
  An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.
- *Execution Granularity*: In how much code detail is the memory change observed.
  An example of *coarse* execution granularity is measuring the memory usage every second.
  An example of *fine* execution granularity is recording the memory usage every Python line.
- *Memory Cost*: How much extra memory the tool needs.
- *Execution Cost*: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

Firstly granularity:

.. list-table:: **Tool Granularity**
   :widths: 40 30 30
   :header-rows: 1

   * - Tool
     - Memory Granularity
     - Execution Granularity
   * - ``process_tree``
     - RSS.
     - Regular time intervals.
   * - ``process``
     - RSS.
     - Regular time intervals.
   * - ``cPyMemTrace.Profile``
     - RSS.
     - Per Python line, Python function and return. C function call and return.
   * - ``cPyMemTrace.Trace``
     - RSS.
     - Per Python line, Python function and return. Python Opcode and exception.
   * - ``cPyMemTrace.ReferenceTracing``
     - RSS.
     - Every object allocation/de-allocation.
   * - DTrace
     - Every ``malloc()`` and ``free()``.
     - Per function call and return.
   * - ``trace_malloc``
     - Every Python object.
     - Per Python line, per function call.
   * - ``debug_malloc_stats``
     - Python memory pool.
     - Snapshots the CPython memory pool either side of a block of code.

Secondly cost:

.. list-table:: **Tool Cost**
   :widths: 40 30 30
   :header-rows: 1

   * - Tool
     - Memory Cost
     - Execution Cost
   * - ``process_tree``
     - Near zero.
     - Near zero.
   * - ``process``
     - Near zero.
     - Near zero.
   * - ``cPyMemTrace.Profile``
     - Near zero.
     - 10x to 40x.
   * - ``cPyMemTrace.Trace``
     - Near zero.
     - 20x to 60x.
   * - ``cPyMemTrace.ReferenceTracing``
     - Near zero.
     - 2x to 80x.
   * - DTrace
     - Minimal.
     - 90x to 100x.
   * - ``trace_malloc``
     - Significant but compensated.
     - 900x for small objects, 6x for large objects.
   * - ``debug_malloc_stats``
     - Minimal.
     - +2000x for small objects, 12x for large objects.

Installation
============

To install pymemtrace, run this command in your terminal:

.. code-block:: console

    $ pip install pymemtrace


Licence
=======

Python memory tracing.

* Free software: MIT license
* Documentation: https://pymemtrace.readthedocs.io.
* Project: https://github.com/paulross/pymemtrace.

Credits
=======

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different
implementation, of ``cPyMemTrace`` in pure Python.

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage



History
=======

0.4.0 (2026-04-18)
---------------------

* cPyMemTrace:
    * User filtering of Reference Tracing events to include/exclude specific events.
    * Reference Tracing runs the Garbage Collector on exit to make the log more accurate.
    * Much more reliable processing of Reference Tracing events.
    * Fix the issue with Reference Tracing where handling type "frame" and "code"
      was causing pytest and CPython runtime assert failures.
    * Write profile/trace/reference tracing context switches to the appropriate log file.
    * Add cPyMemTrace.ReferenceTracingSimple as an example. Includes documentation and test code.
    * Add suspend() and resume() methods for Reference Tracing.
    * Document pytest issues with Reference Tracing (now historical information).
* General documentation improvements, now around 140 PDF A4 pages.
* Add Doxygen documentation of the C source.
* Supported Python versions are: 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14, 3.15
* Development Status :: 5 - Production/Stable

0.3.1 (2026-03-23)
---------------------

* pymemtrace:
    * Add decorator for pymemtrace.process.
    * pymemtrace.process can now summarise JSON in the log to stdout.
* cPyMemTrace:
    * Add decorators for Python functions for Profile, Trace and ReferenceTracing.
    * Fix a SIGSEGV when bad keyword arguments were passed to cPyMemTrace.ReferenceTracing.

0.3.0 (2026-03-19)
---------------------

* Add ``process-tree.py`` for logging a process and its children.
* cPyMemTrace:
    * Add Reference Tracing (Python 3.13+) that can record every object allocation or de-allocation.
    * Add an option to log to a specific file.
    * Add an API ``write_message_to_log()`` to inject text into the log file.
    * Better structure of the log file format.
    * Define the log file format.
    * Add debug exploration code with ``debug_cPyMemtrace()``.
    * Fix stacking ``pop()`` issue with trace/profile functions with linked list of ``tTraceFileWrapperLinkedList``.
* Add support for Python 3.14
* Remove support for Python 3.7
* Supported Python versions are: 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14
* Development Status :: 5 - Production/Stable

0.2.0 (2024-11-17)
---------------------

* cPyMemTrace:
    * Add P/T, stack depth and python version to log file name, example:
      ``"20241107_195847_62264_P_0_PY3.13.0b3.log"``
    * Add stacking of trace/profile functions with linked list of ``tTraceFileWrapperLinkedList``.
    * Add an option to log to a specific file.
    * Add an API ``write_to_log()`` to inject text into the log file.
    * Add an optional message to the log file in ``cPyMemTrace``.
    * Add Python API to get log file being written to by ``cPyMemTrace``.
    * Bug fixes in ``cPyMemTrace.c``
    * Safety fix for file path name lengths.
    * Fix for log files where ``'#'`` was being concatenated.

0.1.7 (2024-09-12)
------------------

* Minor fix for a single test.

0.1.6 (2024-09-11)
------------------

* Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.

0.1.5 (2023-06-21)
------------------

* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.

0.1.4 (2022-03-19)
------------------

* Fix Linux build.

0.1.3 (2022-03-17)
------------------

* Fix some tests.

0.1.2 (2022-03-17)
------------------

* Fix source distribution that had missing headers.

0.1.1 (2020-11-17)
------------------

* Add ``cPyMemTrace`` the C level profiler.
* Add DTrace scripts for low level tracing.
* Add ``debug_malloc_stats`` the wrapper around ``sys._debugmallocstats``.
* Add ``process.py`` from the TotalDepth project.
* Add redirect_stdout for ``debug_malloc_stats``.
* Add ``trace_malloc``, a wrapper around the ``tracemalloc`` module.
* Includes extensive documentation and performance measurement.
* First release on PyPI.

0.1.0 (2017-12-04)
------------------

* Initial idea and implementation, never released.
