Metadata-Version: 2.4
Name: ckdoffline
Version: 0.1.0
Summary: Offline ISPF-style editor for datasets on 3390 volume images: z/OS for ZD&T, MVS 3.8j on Hercules
Author: Henri / Mainframe Society
License-Expression: MIT
Keywords: zdt,zd&t,z/os,mvs,mvs 3.8j,hercules,mainframe,dasd,ckd,cckd,3390,ispf,pds
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Emulators
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# CKD-Offline

Edit datasets on 3390 volumes without IPLing the system that owns them: z/OS volumes for
IBM ZD&T, or MVS 3.8j volumes on Hercules. It opens the volume's image file (plain CKD, CCKD
or CCKD64) and gives you an ISPF-style DSLIST, member list, EDIT, BROWSE and a hex editor in
a browser tab. Saves go straight into the image,
including the VTOC, the VTOC index, PDS directories and ISPF statistics.

Typical uses: fix a PARMLIB or PROCLIB member, patch some JCL, edit a sequential file, zap a
load module, then put the volume back into service.

Python 3.10 or newer, standard library only (no dependencies).

## Install

```sh
pip install ckdoffline
```

Or from a checkout: `pip install .` (add `-e` for development).

## Quick start

Work on a copy. On APFS, `cp -c` makes an instant clone:

```sh
cp -c MYVOL.CCKD work.cckd
ckdoffline work.cckd
```

Your default browser opens at `http://localhost:3270/`. Without an image argument you start on
the Open panel, where you can type a path or pick a file. `python3 -m ckdoffline` works too.

Options:

| Option | Meaning |
|---|---|
| `--readonly` | open the image read-only |
| `--no-backup` | skip the `.bak-*` copy before the first write |
| `--codepage 1047\|037` | EBCDIC code page (default 1047; also on the Open panel) |
| `--userid ID` | user ID written into ISPF statistics (default: your login name) |
| `--port N` | HTTP port (default 3270) |
| `--no-browser` | don't open a browser |

The server listens on 127.0.0.1 only. It is for one person on one machine.

## What you can do

- **DSLIST**: every dataset on the volume with DSORG, RECFM, LRECL, BLKSIZE, tracks and
  extents. Line commands `E` edit, `B` browse, `M` member list, `I` info (full format-1/3
  DSCB decode), `Z` compress. Primary command `L prefix`.
- **Member list**: ISPF statistics, aliases. Line commands `S`, `E`, `B`, `D` delete,
  `R` rename. Primary commands `S name` (create a new member, as in ISPF) and `L name`.
- **EDIT** (PS and PDS members, RECFM F/FB/FBS/FBA, V/VB/VBA): line commands `I`, `D`, `R`,
  `C`/`M` with `A`/`B`, and `DD`, `CC`, `MM`, `RR`. Primary commands `SAVE`, `END`,
  `CANCEL`, `FIND`/`RFIND`, `CHANGE`/`RCHANGE`, `RESET`, `CAPS ON|OFF`, `NUM ON|OFF`,
  `HEX ON|OFF`, `L n`. Lines you don't touch are written back byte for byte, including bytes
  that have no character in the code page.
- **BROWSE**: read-only view of any PS or PO text, `HEX ON` available.
- **Hex editor** (RECFM U): overtype in place, for a member of a load library or a whole
  sequential (PS) RECFM U dataset (`E` on it in DSLIST). Block lengths never change; all
  changed blocks are written in one transaction.
- **Compress** a PDS in place, like IEBCOPY: members (and their aliases and note lists) are
  moved down, the directory TTRs are relocated, and the freed tracks stay allocated.
- **Extend**: when a save no longer fits, the dataset gets a new extent from its secondary
  allocation, as z/OS would: format-1/3 DSCBs, format 4, and the VTOC index (or format 5 on a
  non-indexed VTOC) are updated. Up to 16 extents on the volume. Format 5 maintenance
  (non-indexed VTOCs) is tested only on synthetic volumes; the real volume used in testing has an indexed VTOC.
  If the free-space map disagrees with the extents, the volume still opens and in-place
  edits work; DSLIST shows why the free space is unknown and only extends are refused.

Every save sets DS1DSIND "changed since backup" and DS1REFDT to today.

## Keys

Laid out so that a Mac keyboard without an F-key row works too. The legend is on every
screen.

| Key | ISPF | Action |
|---|---|---|
| Enter | Enter | process the command line and line commands |
| F3 or Esc | PF3 | End (in EDIT: save and leave) |
| F5 | PF5 | Repeat find |
| F7 or PgUp | PF7 | Scroll up |
| F8 or PgDn | PF8 | Scroll down |
| F10 or Shift+← | PF10 | Scroll left |
| F11 or Shift+→ | PF11 | Scroll right |
| F12 or Ctrl+Shift+X | PF12 | Cancel (in EDIT: leave without saving) |
| Home | Home | cursor to the command line |
| Tab | Tab | next field |
| Ctrl+I or Insert | Insert | toggle insert mode |

## Safety model

- **Stop the emulator first** (ZD&T or Hercules), or detach the device. CKD-Offline can't
  rely on a file lock, and writing to an image the emulator has open will corrupt it. Don't
  run the emulator and CKD-Offline on the same image at once, and after the emulator has used
  the image, close and reopen it in CKD-Offline.
- **External changes are detected.** If anything else writes the image while it is open
  (the emulator, another CKD-Offline), the next save is refused with "image changed on disk
  since it was opened; reopen it", so nothing is written from stale copies. Reopen the image
  (Enter on the Open panel) and redo the change. An "opened" flag that appears after the
  open needs confirming again.
- **One volume per server.** Opening an image (or changing the code page) in one browser tab
  makes the other tabs' next request fail with "Volume changed in another tab": they go back
  to the Open panel instead of saving into the wrong volume.
- **Opened-flag warning.** If a CCKD/CCKD64 image has its "opened" flag set, an emulator is
  running with it, or crashed. You get a warning and choose to continue read-write (only if
  the emulator is really stopped) or reopen read-only. Plain CKD has no such flag, so the Open
  panel always reminds you.
- **Backup.** Before the first write of a session the image is copied to
  `<image>.bak-YYYYMMDD-HHMMSS` (an APFS clone, so it's instant and takes no space until the
  files diverge). An existing backup is never overwritten. `--no-backup` turns this off.
- **Undo journal.** Every save is one transaction. The tool works out every byte it will
  change, writes the original bytes to `<image>.journal` and fsyncs it, applies the change,
  fsyncs, and deletes the journal. A refused save (line too long, directory full, no space)
  writes nothing.
- **Recovery.** If the tool was killed mid-save (kill -9, power loss), the journal is left
  behind. The next read-write open rolls the image back to how it was before that save, and
  says so. A read-only open refuses and offers to reopen read-write to recover.
- **Read-only mode** (`--readonly` or the Open panel field) never writes the image.
- **Refusals.** It won't touch a VTOC marked as needing recovery (DIRF), an index it can't
  read, a dataset with no secondary allocation, or anything past the end of the volume.

## Out of scope

- Creating, deleting or renaming datasets (that needs catalog entries, which aren't on the
  volume).
- Anything multi-volume.
- PDSE, HFS, VSAM, DA/IS and every DSORG other than PS and PO. These are listed, not opened.
- RECFM VBS is browse only. New RECFM U members can't be created (`S name` is refused on a
  load library); existing ones can be patched, renamed and deleted.
- Devices other than 3390, FBA, EAV volumes (format 8/9 DSCBs), Hercules shadow files and
  split plain-CKD images.
- Getting data in or out other than by typing: no upload/download, XMIT or IEBCOPY unload.
- More than one user, remote access, a real 3270 data stream.

## Tests

The tests check every write with Hyperion's own utilities (`dasdls`, `dasdcat`, `dasdseq`,
`dasdload`, `dasdcopy`, `cckdcdsk`) as an independent oracle. Using the tool doesn't need
any of this. Only running the tests does.

### The `.oracle/` directory

Everything the oracle needs lives in `.oracle/` in the repo root. It is git-ignored and
never committed: it holds source checkouts, build output and test volumes, all of which you
build locally.

| Path | What |
|---|---|
| `.oracle/src/hyperion` | SDL Hyperion source (or set `HYPERION_SRC` to an existing checkout) |
| `.oracle/ext/<pkg>` | sources of the Hercules external packages |
| `.oracle/extpkgs` | those packages, built and installed |
| `.oracle/hyperion` | the build copy of Hyperion with the utilities |
| `.oracle/fixtures` | small synthetic 3390 test volumes, made on first test run |

### Setting it up

Tested on macOS on Apple Silicon. Needs a C toolchain, CMake and Homebrew's `libtool`
(for `ltdl.h`). On Intel or Linux, drop `-m aarch64` below and adjust the library path in
`tools/build_oracle.sh`.

```sh
E=$PWD/.oracle
git clone https://github.com/SDL-Hercules-390/hyperion.git $E/src/hyperion
for p in crypto decNumber SoftFloat telnet; do
  git clone https://github.com/SDL-Hercules-390/$p.git $E/ext/$p
  mkdir -p $E/work/$p $E/extpkgs
  (cd $E/work/$p && $E/ext/$p/build -d $E/ext/$p -n $p -m aarch64 -a 64 -c Release -i $E/extpkgs)
done
tools/build_oracle.sh
```

The external packages are needed because Hyperion's repository ships them prebuilt for
x86 only. `tools/build_oracle.sh` copies the Hyperion source into `.oracle/hyperion` and
builds the utilities there; the source checkout itself is left untouched. The synthetic test
volumes are built by `tools/make_fixtures.sh` on the first test run.

### Tests against real volumes

`tests/local/` is git-ignored and meant for tests against your own z/OS volumes, which
can't be published. Put the image (a `*.cckd`/`*.ckd` file, also git-ignored) and a
`conftest.py` providing a fixture that clones it into `tmp_path` there; the tests should
only ever touch the clone. pytest picks the directory up automatically.

During development the tool was also run end to end against a real indexed 3390-9 volume:
member edits, adds, deletes and renames; a VB dataset grown by 50,000 records into four
extents with a format-3 DSCB; a load module patch; PDS compress; and saves through the HTTP
API. After each step `cckdcdsk`, `dasdls` and `dasdcat` agreed with the tool and the VTOC
index matched the extents.

### Running

Needs pytest; the JS tests need Node:

```sh
pip install -e '.[dev]'
python3 -m pytest -q -m "not slow"   # fast
python3 -m pytest -q                 # full, including tests/local/ if present
node --test tests/js/                # front-end editing model
```

## Notes

- **Hyperion's DIRF constant is out of date.** `dasdblks.h` names X'40' in DS4VTOCI as
  DS4VTOCI_DIRF. In current IBM documentation DS4DIRF is X'04', and X'40' is DS4DVTOC. This
  tool uses X'04', and refuses to write if either bit is set (see the comment in
  `ckdoffline/vtoc/space.py`).

## Final check is yours

The tests prove Hyperion's DASD utilities read the result cleanly. The last word belongs to
the system that uses the volume: attach an edited copy to ZD&T for z/OS, or to Hercules for
MVS 3.8j, IPL or vary the volume online, and browse the changed members. If you like, run
`IEHLIST LISTVTOC FORMAT` against it, and on z/OS also `ICKDSF ANALYZE` or `BUILDIX`.

## License

MIT. See [LICENSE](LICENSE). CKD-Offline is an independent rewrite: SDL Hyperion (Hercules)
was used as a format reference and its utilities as a test oracle, but no Hercules code is
included.
