Metadata-Version: 2.5
Name: 3dsem
Version: 0.2.8
Summary: Classify point clouds with pretrained 3D semantic segmentation models, from one command
Project-URL: Source, https://github.com/gcsgeospatial/Inference_TUI
Project-URL: Issues, https://github.com/gcsgeospatial/Inference_TUI/issues
License: MIT
License-File: LICENSE
Requires-Python: >=3.9
Provides-Extra: modal
Requires-Dist: modal>=1.0; extra == 'modal'
Description-Content-Type: text/markdown

# 3dsem

Classify point clouds with pretrained 3D semantic segmentation models, on
your own machine, from one command.

```
pip install 3dsem

sem models
sem install [whichever model looks good to you]
sem output (to set a permanent output directory)
```

The classified `.laz` appears next to your input file, with per-point
classification, confidence, and every original dimension carried over.

## What you need

- An NVIDIA GPU with a current driver (Windows 527.41+, Linux 525.60.13+)
- Python 3.9 or newer

`sem install` shows the download size, the license, and a GPU check, then
asks once before any bytes move. It is safe to interrupt and resumes where it
stopped. After install, inference runs fully offline.

## Models

`dales-utonia` is trained on [DALES](https://arxiv.org/abs/2004.11985) aerial
LiDAR and predicts 7 classes: ground, vegetation, vehicle, powerline, fence,
pole, building.

`dales-hag-utonia` predicts the same 7 classes from the same data, with
height above ground added as an input. It scores higher on held-out DALES
scenes (0.86 mIoU against 0.98 overall accuracy) and is the one to reach for
first.

`ieee-utonia` is for low density airborne LiDAR and predicts 5 classes:
ground, vegetation, building, water, bridge deck. Alongside geometry it reads
return number, and the run it was trained from recorded that channel by column
position, so its name (`column_4`) says nothing about what it holds. sem binds
it for you off the return number your file names, which a LAS or LAZ always
does. A `.ply` or `.pcd` that names it differently needs the binding by hand:

```
sem infer ieee-utonia scan.ply --input-field column_4:return_number
```

`h3d-utonia` is for coloured photogrammetric clouds and predicts 8 classes:
ground, vehicle, urban furniture, roof, wall, low veg, tree, chimneys. It is
the one model that consumes colour, so it needs `--rgb-max` naming the
full-scale value your file uses:

```
sem infer h3d-utonia tile.laz --rgb-max 255
```

Every model is licensed CC-BY-NC 4.0 (non-commercial)

`sem models` lists what is available and installed. Bare `sem` opens an
interactive picker.

## How do I make it more accurate?

Use a preset. Each one turns on more of the same three ideas: predict the
scene from several augmented views and vote (test-time augmentation), tile
the scene a second time at a half offset so no point sits on a tile edge
(overlapped voting), and clean up the labels afterwards (smoothing, island
removal, geometry rules).

- default: one pass with light smoothing
- `--med`: 4 voting views, island removal, single tiling pass. Roughly 4x.
- `--high`: 6 views including flips and rotations, stronger smoothing, and
  the overlapped second pass. Roughly 24x, because the views and the second
  pass multiply.
- `--ultra`: 9 views, the strongest smoothing, overlapped. Roughly 36x.

## How to evaluate accuracy on labeled dataset?

```
sem eval tile_pred.laz tile_truth.laz
```

No inference runs. All it does is reads the `Classification` column of both files and
prints, per class, IoU, precision, recall and F1 with point counts, then
overall accuracy, mIoU, macro F1, Cohen's kappa, and a confusion matrix.
Files written by `sem infer` carry their own class legend, whatever codes
that run actually used, so rows read `2 ground` rather than bare numbers.
A file without a legend shows plain codes, and `--names 1=ground,4=water`
names any code by hand.
The files must be the same points in the same order, which is what
`sem infer`'s own export gives you. `--pred-field` and `--truth-field`
name a different column on either side.

When the truth file uses different codes than the model, `--map` translates
them first, and `--ignore` keeps codes out of the scoring:

```
sem eval tile_pred.laz tile_truth.laz --map 3=5,4=5 --ignore 0,1
```

folds the truth's low and medium vegetation (depicted as 3 and 4 in the classification field) into the model's single
vegetation class and skips points that were never classified.

To see whether the cleanup passes help or hurt on your data, run the
inference with `--keep-raw`: it also exports a copy from before
post-processing into a `raw/` folder inside the job, under the same names.
Score each against the same truth and compare:

```
sem eval tile_dales-hag-utonia_predictions/ truth/
sem eval tile_dales-hag-utonia_predictions/raw/ truth/
```

A whole batch works the same way with two folders: files pair by name
(`tile_pred.laz` matches `tile.laz`), each tile gets a summary line, and
the totals pool every point before computing IoU, so a big tile counts
for more than a small one. A predicted file with no truth partner stops
the run and is named. A folder is evaluated against a folder, a file
against a file.

## How do I make it faster?

`--low` is a single pass with no cleanup. If a preset is
mostly what you want, `--no-overlap` drops its second tiling pass, which is
about half its extra inference time.

## Big buildings come out patchy or cut through. Why?

The scene is processed in square tiles, by default 50 m on a side. An object
bigger than one tile is predicted in pieces, and the pieces can disagree.
Two fixes that combine well:

- `--chunk-xy 100` makes the tiles bigger, so a large building fits in one.
  Costs GPU memory.
- `--overlap` (on automatically with `--high` and up) predicts a second pass
  at a half offset and votes, which removes most seam artifacts. It costs
  about 4x on its own.

## It ran out of GPU memory

Lower `--chunk-xy`, try 35 and then 25. Smaller tiles need less VRAM, and
the extra seams they create are what `--overlap` is for.

## Poles or powerlines are disappearing

Presets from `--med` up turn on island removal, which absorbs clusters
smaller than 10 points into their surroundings. Thin objects are exactly
small clusters. Keep the filter but make it gentler with
`--sieve-min-pts 5`, or turn it off with `--no-sieve`.

## Can I hide the model's low-confidence guesses?

`--unclass 0.6` exports every point below 60% confidence as unclassified
instead of its best guess. Bare `--unclass` uses 0.5.
To see where the model is unsure, `--extra-dims` adds per-point diagnostic
fields: `entropy` (0..1 uncertainty), `margin` (gap between the top two
classes), `probs` (one probability field per class, grows the file), and
`ood` (the raw scores behind the gates below). Combine them freely:
`--extra-dims entropy,margin`.

## It confidently labels things it has never seen

A crane or a boat has no class to land in, so the model puts it in the
nearest one it knows, often with high confidence. Confidence alone will not
catch that, because the model is confident and wrong. Two other scores catch
it instead.

`--unclass-gmm` compares each point against how the training classes actually
looked to the model and unclassifies anything too far from all of them. Bare,
it uses the threshold the model was shipped with; give it a number to
override. `--unclass-maxlogit 4.0` catches the opposite case, points where no
class drew much evidence. They find different mistakes, so using both is
normal.

To choose your own numbers, run once with `--extra-dims ood`, which writes
the raw scores into the output file so you can see where your data sits, then
set the thresholds and re-export. Re-exporting does not re-run the model.

## Part of my file is already classified correctly

`--input-field preserve:Classification:2,9` keeps the codes your file already
carries for those points and writes the model's prediction everywhere else.
They are read from the source file at export, so inference, the unclassified
gates and the cleanup passes never touch them.

Kept codes are written as your file spells them. When a kept code means
something different in the model's output (your water 9 has no model
class), add `--preserve-map 9=2` to write it as ground instead. The export
warns whenever a kept code is not one the model uses.

The export reports how often the model disagreed with what you preserved,
which is a quick check on both.

You can also do it afterwards, on a file that is already classified:

```
sem preserve tile_pred.laz tile.las --keep Classification:2,9
```

That writes `tile_pred_preserved.laz` beside it. Same overlay, decided after
you have seen the result rather than before, with the same `--map 9=2`
option for codes that need translating. The two files have to be the same
points in the same order, which is checked on counts and coordinates. The
predicted file needs to be on ASPRS codes for this: the codes you keep come
from your own file's scheme, so overlaying them onto raw model indices would
put two vocabularies in one column. Export it with `--asprs`, which preserving
during inference turns on for you.

## Can I combine models?

`sem infer dales-utonia+dales-hag-utonia tile.las` runs every model in the
chain and merges their predictions with a vote: each model's per-class
probabilities are averaged, and the strongest combined evidence wins. Exact
ties go to the model you listed first, so lead with your strongest.

The models do not have to predict the same classes. Classes spelled the same
way are one class that every model votes on; a class only one model has is
that model's word, and the models that never saw it do not get to vote it
down. So

```
sem infer dales-hag-utonia+ieee-utonia tile.laz
```

gives you nine classes: DALES' seven, with `water` and `bridge deck` added
from `ieee-utonia`. The run prints the combined legend and who contributed to
each class before any model starts, so a mix that is not what you meant costs
nothing to abandon.

The result carries an `agreement` field (what fraction of the models entitled
to name that class agreed) and an `ens_member` field (which model drove each
label). Each member's own predictions sit in `members/` inside the job, so
one run leaves one folder and you can still compare them individually.

## The models call the same thing by different names

Classes only merge when they are spelled alike, so `dales-hag-utonia`'s
`vegetation` and `h3d-utonia`'s `tree` stay separate and compete. `--merge`
folds them into one class:

```
sem infer dales-hag-utonia+h3d-utonia tile.laz --rgb-max 255 \
    --merge vegetation="tree,low veg" \
    --merge building="roof,wall,chimneys"
```

Repeat it per class. Everything you do not mention still merges by name, so
you only write down the parts sem cannot know.

`--merge` names a class of the result and the member classes that feed it. It
only goes that way round: several member classes can become one, but one
cannot be split into several, because a model that never learned the
difference has no opinion about which side a point falls on.

Classes you leave out of a merge are still classes. Nothing is dropped.

## Can I run my own trained model?

If a training run left you a folder holding `final_model.pth` and
`run.json`, register it:

```
sem local-model C:\runs\my_run --name myrun
```

It appears in `sem models` and runs like any other: `sem infer myrun
tile.las`. Nothing is copied; the folder is used in place, and
`sem local-model --forget myrun` unregisters it without touching the
files.

Inference borrows the installed env of a catalog model with the same
backbone, so install one that matches first. Everything else the run needs
(grid, features, classes) is read from the `run.json` beside the weights.

## My file has no coordinate system

It still runs. sem carries the CRS through to the classified output when your
file has one and writes none when it does not. Coordinates are used as stored;
nothing is reprojected.

The unit is taken on trust. Tile size, voxel grid and neighbourhoods are
lengths in metres, so a file in metres is correct and a file in feet is off by
that factor. Most aerial LiDAR is already metre-projected. If yours is not,
convert it first, or state the lengths in your file's own unit with `--grid`
and `--chunk-xy`.

`--epsg 26917` sets the CRS written to the output (use your own zone's code).
It does not transform anything.

Coordinates in degrees are worth watching. A 50 m tile covers a fraction of a
degree several times over, so the whole scene lands in one tile. sem says so
and continues; reproject to a metric CRS first if the result looks wrong.

## My file's coordinate system is wrong

A LAS header can be absent, or present and wrong. Exports that record the
datum and drop the projection are common, and the file then claims degrees
while holding metres.

This does not affect the classification, since the CRS only labels the
output. `--epsg` overrides whatever the header says.

## My file is not a LAS. How does sem know what its columns mean?

It does not. LAS and LAZ name their dimensions in the format spec, so
intensity, return number and classification are read straight off a LAS with
no help from you. `.ply` and `.pcd` leave the meaning of a property up to
whoever wrote the file, so sem asks you:

```
sem infer dales-utonia scan.ply --input-field intensity:scalar_Intensity
```

`--input-field KEY:COLUMN[:VALUE]` says where one thing lives in your file,
repeated per entry. Columns are names or 0-based numbers.

- `xyz:A,B,C` and `rgb:R,G,B` take three columns
- `intensity:C`, `return-number:C` and `hag:C` take one
- `ground:C:V,...` and `preserve:C:V,...` add the value(s) in that column
  that mean it, since the column alone does not say which value counts

A model that wants a channel you did not name stops and lists the fields your
file actually has.

Nothing is inferred from a column's position or from a name that looks
familiar, because a file with `id,x,y,z` in that order and a file with
`x,y,z,id` are indistinguishable to anything except you.

## My colours come out black, or sem asks for --rgb-max

Point clouds store colour as 8, 10, 12 or 16 bit, and no format records which.
`--rgb-max` states the full-scale value: `255`, `1023`, `4095`, `65535`, or `1`
for float 0 to 1 colour. It is only needed when the model actually consumes
colour; a model that runs on intensity ignores the colour in your file and
never asks. Of the shipped models that is `h3d-utonia` alone.

On a las or laz you do not have to supply it. sem reads the colour before the
run and takes the smallest of those values that covers the brightest point it
saw, printing what it picked:

```
[portal] color: values up to 4032 -> --rgb-max 4095
```

Read that line, because it comes from the first million points rather than the
whole file. A tile whose bright points all sit late in it reads low, and a
12-bit cloud scored as 10-bit comes out dark. Passing `--rgb-max` yourself
overrides the probe. Every other format has no colour sem can probe this way,
so it stops and asks.

## Height above ground

Models trained with a HAG channel compute one at staging, using a ground
raster whose cell size comes from the linear unit your file's CRS declares: 2
metres, or whatever length equals 2 metres in your file's own unit. A cloud in
US survey feet gets a 6.56 foot cell, which is the same ground resolution.

A cloud that declares no projected CRS has no unit to read, so the cell stays
2, which is right if the coordinates are metres and wrong by that factor if
they are not. `--epsg` supplies a CRS to read the unit from, and `--hag-cell`
states the cell outright and skips the question.

`--hag-cell` also overrides the derived cell when you want a different
resolution. The height error it costs is roughly the cell size times the local
slope, so a smaller cell buys accuracy on steep ground and costs memory.

`--ground-method` picks where the ground comes from (`smrf`, `csf`, `zmin`, or
`labels` when your file already marks ground). `--csf-rigidness` (1 steep, 2 moderate,
3 flat/urban), `--smrf-window` and `--smrf-cut` tune the ground filters. If
your file marks ground with a class code, say so in one entry:
`--input-field ground:Classification:2`. The column alone is not enough,
because it does not say which value in it means ground. More than one code
can count as ground: `ground:Classification:2,9` also folds water under
the ground surface, which keeps HAG flat across lakes and rivers.

## The cleanup rules are wrong for my data

The geometry rules only run when you ask for them and tell them what your
classes mean: `--rules --roles veg=vegetation,building=building`. sem no
longer decides that a class is vegetation because its name contains "tree".
Their thresholds are lengths in your scene's vertical unit and unitless
ratios, all settable through `--rule-set`: `ground_hag` (0.1), `lowveg_hag`
(0.35), `high_hag` (2.0), `planar_min` (0.55), `scatter_min` (0.4). For
example `--rule-set high_hag=3,planar_min=0.6`.

## Controlling the exported class codes

By default each class exports as its own model index: 0, 1, 2 and so on, in the
order the model lists them. On the shipped DALES models that is ground 0,
vegetation 1, vehicle 2, powerline 3, fence 4, pole 5, building 6, and gated
points 7.

Those indices are not ASPRS codes. In the spec 2 is Ground, so a viewer reading
your file per the spec sees the vehicles as ground. Exported las and laz carry
a legend recording what each code meant, and `sem eval` reads it, but nothing
outside sem does. Any file leaving sem wants the next flag.

`--asprs` exports spec codes instead: ground 2, vegetation 5, building 6,
powerline 14, pole 15, and user-definable codes from 64 for classes the spec
has no name for, which on these models is vehicle 64 and fence 65. A class the
table does not name takes the next free code at or above 64.

`--asprs-map ground=2,building=6` overrides any of them by name and implies
`--asprs`. Two classes landing on one code stops the run rather than merging
them silently. Preserving codes from your own file implies `--asprs` too, since
those codes already speak it.

`--unclassified-code` sets the code gated points receive. It defaults to the
class count, or to 1 under `--asprs`, which is ASPRS "Unclassified" and no
class maps onto. Export stops rather than letting that code collide with a real
class.

## I don't have a GPU

`sem` classifies on your own machine and needs an NVIDIA GPU. Conversion and
export are CPU work; only the model needs one.

If you do not have one, `--modal` rents one for the length of a run:

```
pip install "3dsem[modal]"
modal token new

sem infer dales-hag-utonia tile.laz --modal
```

`modal token new` opens a browser and links your Modal account. It is asked
once. After that, `--modal` on any `sem infer` uploads your file, runs the
same command on a cloud GPU, and writes the results where a local run would
have put them: next to your input, or under `sem output` if you set one.

Every flag works. Presets, `--asprs`, `--unclass`, `--input-field`, model
chains, a whole folder of tiles.

The first run installs the model on Modal, which takes a few minutes. Later
runs reuse that install and start in about a minute. Modal bills for run time
only, so an idle account between jobs costs nothing.

`SEM_MODAL_GPU` picks the GPU (default `A10G`) and `SEM_MODAL_TIMEOUT_HOURS`
caps one run (default 4).

Your own weights go up too. `sem infer myrun tile.laz --modal` uploads the
folder you registered with `sem local-model` and runs it against a catalog
model's environment, so a machine with no GPU can run a model it could not
install.

One limit: `--modal` needs Python 3.10 or newer, where the rest of sem runs
on 3.9. Your Python and the rented machine's have to match.

## Is my data safe on Modal?

Your file is uploaded to a private Modal volume in your own account, and the
uploaded copy is deleted once the results come back. Results stay on the
volume until you remove them. Nothing is shared with anyone, and nothing is
sent anywhere else.

If that is not acceptable for your data, classify locally. The models are the
same either way.

## Where do Modal jobs go?

`sem output-modal` shows what is in use:

```
sem output-modal            show the volumes and the GPU
sem output-modal site-a     send jobs to the site-a volume
sem output-modal off        back to sem-outputs
```

This is separate from `sem output`. One names a Modal volume, the other a
folder on your disk, and both apply to a `--modal` run: the job is written to
the volume and a copy comes back to the folder.

Two volumes are used, both created on first use in your own account.
`sem-outputs` (or whatever you named) holds the jobs. `sem-envs` holds the
installed models, which is why the first run is the slow one.
`modal volume delete sem-envs` clears them and the next run reinstalls;
`modal volume ls sem-outputs` lists what jobs are still up there.

## A Modal run failed partway. What now?

Re-run it. Nothing is left in a state that needs cleaning up first.

When a run fails, whatever it managed to write stays on the volume under the
timestamp printed in the error, and your uploaded input stays with it, since
sem only clears the upload after results come back safely. Both are visible
with `modal volume ls`. Re-running uploads afresh under a new timestamp and
does not touch the old one, so `modal volume rm sem-outputs <timestamp> -r`
is how a failed attempt goes away once you no longer want it.

## Can I rerun with different settings without reconverting?

Yes, if you pass `--keep-npz` on the first run.

The output folder is a self-contained job, named after your file and the
model (`tile_dales-utonia_predictions/`). The staged `.npz` inside it hold
the converted channels, and they are what a re-run reuses. By default they
are deleted once the export is written, because most runs only want the
classified file. `--keep-npz` retains them.

With them kept: quality options (presets, TTA, cleanup, export) never
reconvert. Conversion options (`--epsg`, `--ground-method`, `--hag`, ...)
and changes to the input file itself reconvert automatically; identical
settings reuse the staged files. Every model keeps its own job folder, so
switching models never mixes results.

## Can I just name the output file?

Yes. An output ending in `.laz`, `.las`, `.txt`, `.csv` or `.ply` is the file
itself, and its extension picks the format, so `--format` becomes redundant:

```
sem infer dales-utonia tile.las out/pc.laz
```

The job folder is created beside it (`out/pc_dales-utonia_predictions/`), so
everything the run makes stays where you pointed. This is for a single input
file; a folder of tiles produces one result per scene, so give it a folder.

If you have set a default with `sem output`, a bare filename lands there:
`sem infer dales-utonia tile.las pc.laz` writes `<your dir>/pc.laz`. Put a
folder in the path and it goes exactly there instead.

## Where does everything live?

Downloads go to `~/.trainer` (set `TRAINER_HOME` to move them). Your data
and results never go there: each job is a folder next to your input, or
under a default you set with `sem output <dir>` and undo with
`sem output off`. `sem clean dales-utonia` removes one model;
`sem clean --all` removes everything sem ever downloaded.

## Can I look at the intermediate files?

Run with `--keep-npz`, or the intermediates are gone once the export is
written. Then `sem tolaz job/tile_input.npz` writes `tile_input.laz`
beside it. The xyz becomes the cloud, rgb the color, and every other
per-point channel a named field you can shade by in CloudCompare. That
works on the staged input as well as on predictions, so it is how you
see the features a model actually received, not just what it predicted.

## Can I build my own preset?

`--pick` gives you the six knobs the presets are made of, on arrow keys:

```
sem infer dales-hag-utonia tile.laz --pick
```

```
your settings
  tta            -
  overlap        default
  no-postproc    off
  sieve          default
  knn-k          -
  knn-radius     -
  start the run
```

Set the ones you want and leave the rest. A preset is a name for one
combination of these; here you choose your own.

The command builds itself in the footer as you go, and is printed before the
run, so this is also a quick way to learn the flags:

```
sem infer dales-hag-utonia tile.laz --tta full --sieve --knn-k 20
```

You can start from a preset. `sem infer m tile.laz --med --pick` opens with
the knobs showing what `--med` means, and anything you change is added to it:

```
sem infer m tile.laz --med --knn-k 20
```

Anything else you pass alongside `--pick` is carried through untouched, so
`sem infer m tile.laz --asprs --pick` keeps `--asprs`.

## Every option

`sem infer --help` documents all of it, and `sem infer --help all` shows
every flag in one table.

## Licensing

The `sem` tool is MIT licensed. Each model ships a `NOTICE.md` stating its
architecture credits and license terms; some models carry a non-commercial
restriction inherited from their pretrained components, shown before you
install.
