Metadata-Version: 2.5
Name: iransystem
Version: 0.4.0
Summary: IranSystem encoding for Python
Project-URL: Source, https://git.sr.ht/~gurban/python-iransystem
Author-email: "Ghorban M. Tavakoly" <gmt3141@gmail.com>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: codec,dos,encoding,farsi,iransystem,persian
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Persian
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# python-iransystem

Nearly all old Iranian programs (especially programs from the golden DOS
era) and a great number of new enterprise programs use the IranSystem
encoding. Here is a Python module for easy reading and writing
IranSystem encoded files.

## Installing

The module is installed from its repository. In a project managed with
[uv](https://docs.astral.sh/uv/), add it as a dependency:

```sh
uv add git+https://git.sr.ht/~gurban/python-iransystem
```

To install it into the current environment instead, use `uv pip` or
pip:

```sh
uv pip install git+https://git.sr.ht/~gurban/python-iransystem
pip install git+https://git.sr.ht/~gurban/python-iransystem
```

To try it without installing anything, let uv run a script with the
module available:

```sh
uv run --with git+https://git.sr.ht/~gurban/python-iransystem script.py
```

## Using the module

The module needs Python 3.10 or newer. Importing it registers the
`iransystem` and `iransystem_visual` codecs (`iran_system`,
`iran-system`, `iransystem-visual` and `iran_system_visual` work too):

```python
import iransystem

with open("document.txt", encoding="iransystem") as f:
    text = f.read()
```

IranSystem stores text in visual order (each line laid out left to right
as it appears on screen), so Persian text is stored backwards. The
`iransystem` codec converts each line to logical order while decoding
(and back while encoding). It keeps Latin words and numbers in their
original order, along with email addresses, URLs, DOS paths and words
such as `C++` or `50%`, and mirrors brackets. Punctuation that ends a
sentence stays with the Persian text. If you need the raw visual order,
use the `iransystem_visual` codec.

The same reordering is available as `iransystem.flip()`, for text you
already have in visual order (or want in visual order). It converts
either way, so `flip(flip(text)) == text`:

```python
from iransystem import flip

flip("Workbench 2.0 مالس")  # "سلام Workbench 2.0"
flip("سلام Workbench 2.0")  # "Workbench 2.0 مالس"
```

IranSystem also stores each letter in its contextual form (initial,
medial, final or isolated). Where those forms show two letters are not
joined but Unicode would join them, the `iransystem` codec adds a ZERO
WIDTH NON-JOINER (as in «می‌کند» or «خانه‌ها»). When encoding, it
picks the form of each letter from its neighbours and respects ZERO
WIDTH NON-JOINER and ZERO WIDTH JOINER.

IranSystem has no codes for the two joiners themselves. When encoding,
they only choose the forms of the letters around them and are then
dropped, so `"ب‍"` encodes as a BEH that joins the next letter, and
a joiner on its own encodes to nothing. When decoding, the `iransystem`
codec adds a ZERO WIDTH NON-JOINER back only where the stored forms show
a break the renderer wouldn't make, so a non-joiner that changed no form
(after ALEF, for example) is lost, and so is every ZERO WIDTH JOINER.

### Text is converted line by line

Both codecs (and `flip()`) work on one line at a time. A line ends at LF
or CR, so CR LF ends one line; no other character (such as form feed or
U+2028 LINE SEPARATOR) ends a line. Each line is reordered on its own,
and the form of a letter depends only on its neighbours in that line.
What this means depends on how the text is converted:

- `str.encode()` and `bytes.decode()` see the whole text at once, so
  they are always correct.
- Reading with `iransystem` (through `open()`, an incremental decoder
  or a `StreamReader`) holds back an incomplete last line until the rest
  of it arrives or the input ends, so the bytes can come in any pieces.
  Reading with `iransystem_visual` maps each byte on its own, so the
  same is true.
- Writing (through `open()`, an incremental encoder or a
  `StreamWriter`) converts each piece on its own, so each `write()` must
  hold whole lines (see [Write whole lines](#write-whole-lines)).

## Examples

### Example 1: Read a text file

```python
import iransystem

with open("document.txt", encoding="iransystem") as f:
    for line in f:
        print(line, end="")
```

### Example 2: Write a text file

IranSystem files usually come from DOS, so write DOS line endings:

```python
import iransystem

with open("document.txt", "w", encoding="iransystem", newline="\r\n") as f:
    f.write("سلام دنیا\n")
```

### Example 3: Read a Paradox DBF file

```python
import dbfread
import iransystem

# Read an Iranian Tamin Ejtema-ee's Paradox DBF file
table = dbfread.DBF("DSKWOR00.dbf", encoding="iransystem")

for record in table:
    print(f"{record['DSW_FNAME']} {record['DSW_LNAME']}")
```

## Problems

### Write whole lines

When writing with the `iransystem` or `iransystem_visual` codec
(through `open()`, `codecs.open()` or a `StreamWriter`), each `write()`
call is encoded separately: the `iransystem` codec converts it to
visual order on its own, and both codecs pick letter forms without
seeing the text of the next call. So write whole lines at a time. For
example, `print("سلام", "دنیا", file=f)` writes its arguments one by one
and stores them in the wrong order, and `f.write("سل"); f.write("ام")`
stores LAM and ALEF as separate letters instead of the LAM-ALEF
ligature of «سلام». Join the text first: `f.write("سلام دنیا\n")`.

## TODO

- [ ] Report encoding errors at their position in the input. Errors are
  raised after the text is shaped and reordered, so
  `"سلام €".encode("iransystem")` reports position 0, not 5, and error
  handlers see the reordered text.
- [ ] Return the input length as consumed when encoding, not the length
  of the shaped text (`"لا\u200c"` reports 1 instead of 3).
- [ ] Reject private use characters U+F78D to U+F7FE in the input. They
  are used internally for shaped letters, so `"\uf792"` encodes silently
  as BEH.
- [ ] Add ZERO WIDTH NON-JOINERs when decoding with `iransystem_visual`
  too. Without them it loses which letters are shown apart, and encoding
  the text again joins them.
- [ ] Keep symbols between a Latin and a Persian word with the Persian
  word: in «سلام GNU/لینوکس» the slash is stored on the GNU side.
- [ ] Encode across `write()` calls, so text doesn't have to be written
  in whole lines (see [Write whole lines](#write-whole-lines)).
- [ ] Bump the version for the next release (still 0.3.0, although
  encoding is new).

## Development

The project uses [uv](https://docs.astral.sh/uv/). `uv sync` creates a
virtual environment in `.venv` with the module and the development
tools (ruff), as pinned in `uv.lock`; `uv run` runs a command in it,
syncing it first if needed. From the repository root:

```sh
uv sync                         # set up the environment
uv run python -m unittest       # run the tests
uv run ruff check .             # lint
uv run ruff format .            # format the code
uv run samples/decode-text.py   # decode the sample file
uv build                        # build the wheel and sdist into dist/
```

The version is `__version__` in `iransystem.py`; `pyproject.toml` reads
it from there.

## License

Copyright © 2020  [Ghorban M. Tavakoly](mailto:gmt3141@gmail.com)

This program is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation, either version 3 of the License, or (at your
option) any later version.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.
