| | |
- builtins.object
-
- BackgroundIOPriority
- threading.Thread(builtins.object)
-
- BackgroundWriteProcess
class BackgroundIOPriority(builtins.object) |
| |
BackgroundIOPriority - Priority Profile for doing background writes.
See __init__ for fields
|
| |
Methods defined here:
- __getitem__(self, key)
- __init__(self, chainPollTime, defaultChunkSize, priorityPct, charityRate=1.85, charityTime=0.0003)
- __init__ - Create a BackgroundIOPriority.
Some terms: throughput - Bandwidth out (Megs per second)
interactivity - CPU time available for other tasks (calculations, other I/O, etc)
@param chainPollTime - float > 0, When chaining, this is the sleep time between checking if prior is finished.
Too low and the polling takes up CPU time, too high and you'll lose a little time in between chained writes, while gaining interactivity elsewhere.
@param defaultChunkSize - integer > 0, When providing a straight string/bytes to bgwrite (instead of chunking yourself, or using bgwrite_chunk) this will
be used as the max size of each chunk. Each chunk is written and a flush is issued (if the stream supports it).
Increasing this increases throughput while decreasing interactivity
@param priorityPct - integer > 0, generally 0-100. When this number is high, throughput for the operation will be higher. When it is lower,
interactivity is higher, e.x. if you have a calculation going and a write going, the lower this number the longer the write will take, but the more
calculations will be performed during that period.
@param charityRate - float >= 0, Every couple of blocks written, the current throughput is checked and if things have been going swiftly
a short sleep will be incurred. Increasing this number causes that check to happen more often.
This number is related to both the number of blocks and the priorityPct. The default, should be fine, but you may find it better
as a different value in certain cases. Increasing or decreasing could either increase or decrease interactivity, depending on those other factors.
Generally, however, increasing this increases interactivity and ability to write in parallel, at the cost of throughput.
@param charityTime - float >= 0 - Used to calculate the time to sleep when the charity period hits. The equation is:
sleepTime = charityTime * ((dataWritten / delta) / ( (dataWritten / delta) * priorityPctDec))
Where dataWritten = number of bytes written already, delta = total time spent writing (not including charity time sleeping)
and priorityPctDec = priorityPct / 100.
Increasing this can increase interactivity and allow more parallel operations at the cost of throughput.
The default should be fine for the majority of cases, but it is tunable.
An "interactivity score" is defined to be (number of calculations) / (time to write data).
- __setitem__(self, key, value)
Data descriptors defined here:
- chainPollTime
- charityRate
- charityTime
- defaultChunkSize
- priorityPct
|
class BackgroundWriteProcess(threading.Thread) |
| |
BackgroundWriteProcess - A thread and data store representing a background write task. You should probably use one of the bgwrite* methods and not this directly.
Attributes:
remainingData <deque> - A queue representing the data yet to be written
startedWriting <bool> - Starts False, changes to True when writing has started (thread has started and any pending prior chain has completed)
finished <bool> - Starts False, changes to True after writing has completed, and if closeWhenFinished is True the handle is also closed.
|
| |
- Method resolution order:
- BackgroundWriteProcess
- threading.Thread
- builtins.object
Methods defined here:
- __init__(self, fileObj, dataBlocks, closeWhenFinished=False, chainAfter=None, ioPrio=4)
- __init__ - Create the BackgroundWriteProcess thread. You should probably use bgwrite or bgwrite_chunk instead of calling this directly.
@param fileObj <stream> - A stream, like a file, to write into. Hopefully it supports flushing, but it is not a requirement.
@param dataBlocks <bytes/str/list<bytes/str>> - If a list of bytes/str, those are treated as the data blocks, written in order with heuristics for interactivity in between blocks. If bytes/str are provided not in a list form, they will be split based on the rules of the associated #ioPrio
@param closeWhenFinished <bool> - Default False. If True, the fileObj will be closed after writing has completed.
@param chainAfter <None/BackgroundWriteProcess> - If provided, will hold off writing until the provided BackgroundWriteProcess has completed (used for queueing multiple writes whilst retaining order)
@param ioPrio <int/BackgroundIOPriority> - If an integer (1-10), a predefined BackgroundIOPriority will be used. 1 is highest throughput, 10 is most interactivity. You can also pass in your own BackgroundIOPriority object if you want to define a custom profile.
@raises ValueError - If ioPrio is neither a BackgroundIOPriority nor integer 1-10 inclusive
- If chainAfter is not a BackgroundWriteProcess or None
- run(self)
- run - Starts the thread. bgwrite and bgwrite_chunk automatically start the thread.
| |