Package catmaid_publish
catmaid_publish
For the latest version, see here: https://github.com/clbarnes/catmaid_publish ; for docs see here: https://clbarnes.github.io/catmaid_publish
Scripts for publishing data from CATMAID. Also useful for taking a snapshot of a particular set of data for further reproducible analysis (but be careful not to mix exported data with live data from the server).
Originally created using cookiecutter and clbarnes/python-template-sci.
Installation
First, ensure you're working in a virtual environment:
# create a virtual environment if you don't have one
python -m venv --prompt catmaid_publish venv
# activate it
source venv/bin/activate
Then install the package, using one of:
# from github
pip install git+https://github.com/clbarnes/catmaid_publish.git
# a local copy of the repo, from within the parent directory
pip install -e .
Usage
catmaid_publish
fetches data from a CATMAID instance based on a configuration file.
If the instance requires authentication, credentials can be passed with environment variables or a separate TOML file.
The workflow looks like this:
# Empty config files will be created at these paths
catmaid_publish_init my_config.toml --toml-credentials my_credentials.toml
# Edit my_config.toml for your export needs
# Edit my_credentials.toml with your login details (do not share or version control this file!)
catmaid_publish my_config.toml my_export/ my_credentials.toml
# optionally, compress export into a single zip file for transfer
zip -r my_export.zip my_export
catmaid_publish_init
Use this to initialise an empty config and optional credentials files.
If you have already set your credentials using environment variables starting with CATMAID_
, the credentials files will be filled in.
usage: catmaid_publish_init [-h] [--toml-credentials TOML_CREDENTIALS]
[--env-credentials ENV_CREDENTIALS] [--ignore-env]
[--no-http-basic] [--no-token]
config
Write an empty config file and, optionally, credentials files.
positional arguments:
config Path to write TOML config
options:
-h, --help show this help message and exit
--toml-credentials TOML_CREDENTIALS, -t TOML_CREDENTIALS
Path to write TOML file for credentials. Will be
populated by CATMAID_* environment variables if set.
--env-credentials ENV_CREDENTIALS, -e ENV_CREDENTIALS
Path to write env file for credentials. Will be
populated by CATMAID_* environment variables if set.
--ignore-env, -i Ignore CATMAID_* environment variables when writing
credential files.
--no-http-basic, -H Omit HTTP basic auth from credentials file.
--no-token, -T Omit CATMAID API token from credentials file.
If you would prefer to write the config and credentials files yourself, see the examples here.
Configuration
Fill in the config file using TOML formatting.
Citation information will be included with the export. Project information will be used to connect to CATMAID (but sensitive credentials should be stored elsewhere).
For other data types, all = true
means export all data of that type.
Note that this can take a very long time for common data types (e.g. neurons) in large projects.
If all = false
, you can list the names of specific objects to be exported.
You can also rename specific objects by mapping the old name to the new one (objects to be renamed will be added to the list of objects to export).
Some objects can be annotated.
In this case, you can instead list annotations for which annotated objects will be exported.
Indirectly annotated ("sub-annotated") objects, e.g. the relationship between A and C in annotation "A" -> annotation "B" -> neuron "C"
will also be exported.
All exported data have a pre-written README.md
file detailing the data format and structure.
You can add additional information to the README using the readme_footer
key.
This string will have leading and trailing whitespace stripped, and, if still non-empty, will be appended to the default README below a thematic break.
Authentication
If your CATMAID instance requires authentication (with a CATMAID account and/or HTTP Basic authentication), fill in these details in a separate TOML file, or as environment variables (which can be loaded from a shell script file).
Passwords, API tokens etc. MUST NOT be tracked with git.
A credentials file simply looks like this:
# If your instance requires login to browse
api_token = "y0urc47ma1d70k3n"
# If your instance uses HTTP Basic authentication to access
http_user = "myuser"
http_password = "mypassword"
Or use environment variables CATMAID_API_TOKEN
, CATMAID_HTTP_USER
, and CATMAID_HTTP_PASSWORD
.
catmaid_publish
Once you have filled in the config file, use the catmaid_publish
command to fetch and write the data, e.g.
# leave out the credentials path if you are using environment variables
catmaid_publish path/to/config.toml path/to/output_dir path/to/credentials.toml
Full usage details are here:
usage: catmaid_publish [-h] config out [credentials]
Export data from CATMAID in plaintext formats with simple configuration.
positional arguments:
config Path to TOML config file.
out Path to output directory. Must not exist.
credentials Optional path to TOML file containing CATMAID credentials
(http_user, http_password, api_token as necessary).
Alternatively, use environment variables with the same names
upper-cased and prefixed with CATMAID_.
options:
-h, --help show this help message and exit
Output
README files in the output directory hierarchy describe the formats of the included data. All data are sorted deterministically and in plain text, and are highly compressible.
Reading
As detailed in the top-level README of the exported data, this package contains a utility for reading an export into common python data structures for neuronal analysis.
For example:
from catmaid_publish import DataReader, ReadSpec, Location
import networkx as nx
import navis
reader = DataReader("path/to/exported/data")
annotation_graph: nx.DiGraph = reader.annotations.get_graph()
neuron: navis.TreeNeuron = reader.neurons.get_by_name(
"my neuron",
ReadSpec(nodes=True, connectors=False, tags=True),
)
landmark_locations: list[Location] = list(reader.landmarks.get_all())
volume: navis.Volume = reader.volumes.get_by_name("my volume")
Tips
In general, it's most robust to use the CATMAID UI to make an annotation specifically for your export; ideally namespaced and timestamped (e.g. cbarnes_export_2023-02-15
).
Later exports can be a superset of this one.
Publication
Consider running the export once to find which objects are exported, and determine whether any objects need renaming. Then update your configuration with these renames.
Analysis snapshot
In large CATMAID projects, there are relatively few landmarks, volumes, and annotations compared to neurons.
As these are all helpful for mining data, consider exporting with all = true
for everything except neurons.
Use the CATMAID UI (e.g. connectivity widget, graph widget, volume intersection) to annotate a superset of your neurons of interest for the export.
It is easier to bounce between local analysis and use of the CATMAID UI if you do not rename any objects in this case.
Containerisation
This project can be containerised with apptainer (formerly called Singularity) (bundling it with a python environment and full OS) on linux, so that it can be run on any system with apptainer installed.
Just run make container
(requires sudo).
The python files are installed in the container at /project
.
Depending on where your config and credentials files are stored, they may be accessible to the container by default. Otherwise, you can manually bind mount the containing directories inside the container at runtime:
# Find the data path your environment is using, defaulting to the local ./data
DATA_PATH="$(pwd)/data"
CREDS_PATH="$(pwd)/credentials"
# Execute the command `/bin/bash` (i.e. get a terminal inside the container),
# mounting the data directory and credentials you're already using.
# Container file (.sif) must already be built
apptainer exec \
--bind "$DATA_PATH:/data" \
--bind "$CREDS_PATH:/credentials" \
catmaid_publish.sif /bin/bash
# Now you're inside the container...
>>> catmaid_publish_init /data/config.toml -t /credentials/my_instance.toml
>>> catmaid_publish /data/config.toml /data/my_export /credentials/my_instance.toml
Expand source code
# isort: skip_file
import sys
from .version import version as __version__ # noqa: F401
from .version import version_tuple as __version_info__ # noqa: F401
from .io_helpers import hash_toml
from .main import publish_from_config
from .reader import (
DataReader,
SkeletonReader,
LandmarkReader,
VolumeReader,
AnnotationReader,
)
from .skeletons import ReadSpec
from .landmarks import Location
from .volumes import AnnotatedVolume
if sys.version_info >= (3, 10):
from importlib.resources import files
else:
from importlib_resources import files
__doc__ = files("catmaid_publish.package_data").joinpath("README.md").read_text()
__all__ = [
"publish_from_config",
"DataReader",
"hash_toml",
"ReadSpec",
"Location",
"SkeletonReader",
"LandmarkReader",
"VolumeReader",
"AnnotatedVolume",
"AnnotationReader",
]
Sub-modules
catmaid_publish.annotations
catmaid_publish.constants
catmaid_publish.initialise
-
Write an empty config file and, optionally, credentials files.
catmaid_publish.io_helpers
catmaid_publish.landmarks
catmaid_publish.main
-
Export data from CATMAID in plaintext formats with simple configuration.
catmaid_publish.reader
catmaid_publish.skeletons
catmaid_publish.utils
catmaid_publish.version
catmaid_publish.volumes
Functions
def hash_toml(fpath) ‑> str
-
Expand source code
def hash_toml(fpath) -> str: orig = read_toml(fpath) hashable = hashable_toml_dict(orig) return hex(hash(hashable))[2:]
def publish_from_config(config_path: pathlib.Path, out_dir: pathlib.Path, creds_path: Optional[pathlib.Path] = None)
-
Expand source code
def publish_from_config( config_path: Path, out_dir: Path, creds_path: Optional[Path] = None ): timestamp = dt.datetime.utcnow().replace(tzinfo=ZoneInfo("UTC")) out_dir.mkdir(parents=True) config = Config.from_toml(config_path) config_hash = config.hex_digest() if creds_path is not None: creds = read_toml(creds_path) else: creds = None project = config.get("project", default={}, as_config=False) catmaid_info = { "server": config.get("project", "server_url"), "project_id": config.get("project", "project_id"), } _ = get_catmaid_instance( catmaid_info, creds, ) with tqdm(total=4) as pbar: _, ann_renames = publish_annotations(config, out_dir, pbar) _ = publish_volumes(config, out_dir, ann_renames, pbar) _ = publish_skeletons(config, out_dir, ann_renames, pbar) _ = publish_landmarks(config, out_dir, pbar) meta = { "units": project["units"], "export": { "timestamp": timestamp, "config_hash": config_hash, "package": { "name": "catmaid_publish", "url": "https://github.com/clbarnes/catmaid_publish", "version": f"{__version__}", }, }, } cit = config.get("citation", default=dict(), as_config=False) ref = dict() if url := cit.get("url", "").strip(): ref["url"] = url if doi := cit.get("doi", "").strip(): ref["doi"] = f"https://doi.org/{doi}" if biblatex := cit.get("biblatex", "").strip(): multiline_strings = True ref["biblatex"] = biblatex else: multiline_strings = False if ref: meta["reference"] = ref with open(out_dir / "metadata.toml", "wb") as f: tomli_w.dump(meta, f, multiline_strings=multiline_strings) with open(out_dir / "README.md", "w") as f: readme = join_markdown( README, project.get(README_FOOTER_KEY), ) f.write(readme)
Classes
class AnnotatedVolume (vertices: Union[list, numpy.ndarray], faces: Union[list, numpy.ndarray] = None, name: Optional[str] = None, color: Union[str, collections.abc.Sequence[Union[int, float]]] = (0.85, 0.85, 0.85, 0.2), id: Optional[int] = None, annotations: Optional[set[str]] = None, **kwargs)
-
Mesh consisting of vertices and faces.
Subclass of
trimesh.Trimesh
with a few additional methods.Parameters
vertices
:list | array | mesh-like
(N, 3)
vertices coordinates or an object that has.vertices
and.faces
attributes in which casefaces
parameter will be ignored.faces
:list | array
(M, 3)
array of indexed triangle faces.name
:str
, optional- A name for the volume.
color
:tuple
, optional- RGB(A) color.
id
:int
, optional- If not provided, neuron will be assigned a random UUID as
.id
. **kwargs
- Keyword arguments passed through to
trimesh.Trimesh
See Also
:func:
~navis.example_volumeLoads example volume(s).
A Trimesh object contains a triangular 3D mesh.
Parameters
vertices
:(n, 3) float
- Array of vertex locations
faces
:(m, 3)
or(m, 4) int
- Array of triangular or quad faces (triangulated on load)
face_normals
:(m, 3) float
- Array of normal vectors corresponding to faces
vertex_normals
:(n, 3) float
- Array of normal vectors for vertices
metadata
:dict
- Any metadata about the mesh
process
:bool
- if True, Nan and Inf values will be removed
- immediately and vertices will be merged
validate
:bool
- If True, degenerate and duplicate faces will be
- removed immediately, and some functions will alter
- the mesh to ensure consistent results.
use_embree
:bool
- If True try to use pyembree raytracer.
- If pyembree is not available it will automatically fall
- back to a much slower rtree/numpy implementation
initial_cache
:dict
- A way to pass things to the cache in case expensive
- things were calculated before creating the mesh object.
visual
:ColorVisuals
orTextureVisuals
Assigned to self.visual
Expand source code
class AnnotatedVolume(navis.Volume): def __init__( self, vertices: Union[list, np.ndarray], faces: Union[list, np.ndarray] = None, name: Optional[str] = None, color: Union[str, Sequence[Union[int, float]]] = (0.85, 0.85, 0.85, 0.2), id: Optional[int] = None, annotations: Optional[set[str]] = None, **kwargs, ): super().__init__(vertices, faces, name, color, id, **kwargs) self.annotations = set() if not annotations else set(annotations)
Ancestors
- navis.core.volumes.Volume
- trimesh.base.Trimesh
- trimesh.parent.Geometry3D
- trimesh.parent.Geometry
- abc.ABC
class AnnotationReader (dpath: pathlib.Path)
-
Class for reading exported annotation data.
Parameters
dpath
:Path
- Directory in which the annotation data is saved.
Expand source code
class AnnotationReader: """Class for reading exported annotation data.""" def __init__(self, dpath: Path) -> None: """ Parameters ---------- dpath : Path Directory in which the annotation data is saved. """ self.dpath = dpath @copy_cache() def get_graph(self) -> nx.DiGraph: """Return the saved graph of text annotations. Returns ------- nx.DiGraph Directed graph of text annotations, where an edge denotes the source annotating the target. All nodes have attributes ``type="annotation``; all edges have attributes ``meta_annotation=True``. """ with open(self.dpath / "annotation_graph.json") as f: d = json.load(f) g = nx.DiGraph() for u, vs in d.items(): for v in vs: g.add_edge(u, v, meta_annotation=True) for _, d in g.nodes(data=True): d["type"] = "annotation" return g
Methods
def get_graph(self) ‑> networkx.classes.digraph.DiGraph
-
Return the saved graph of text annotations.
Returns
nx.DiGraph
- Directed graph of text annotations,
where an edge denotes the source annotating the target.
All nodes have attributes
type="annotation
; all edges have attributesmeta_annotation=True
.
Expand source code
@copy_cache() def get_graph(self) -> nx.DiGraph: """Return the saved graph of text annotations. Returns ------- nx.DiGraph Directed graph of text annotations, where an edge denotes the source annotating the target. All nodes have attributes ``type="annotation``; all edges have attributes ``meta_annotation=True``. """ with open(self.dpath / "annotation_graph.json") as f: d = json.load(f) g = nx.DiGraph() for u, vs in d.items(): for v in vs: g.add_edge(u, v, meta_annotation=True) for _, d in g.nodes(data=True): d["type"] = "annotation" return g
class DataReader (dpath: pathlib.Path)
-
Class for reading exported data.
Attributes
metadata
:Optional[dict[str, Any]]
- Various metadata of export, if present.
volumes
:Optional[VolumeReader]
- Reader for volume data, if present.
landmarks
:Optional[LandmarkReader]
- Reader for landmark data, if present.
neurons
:Optional[SkeletonReader]
- Reader for neuronal/ skeleton data, if present.
annotations
:Optional[AnnotationReader]
- Reader for annotation data, if present.
Parameters
dpath
:Path
- Directory in which all data is saved.
Expand source code
class DataReader: """Class for reading exported data. Attributes ---------- metadata : Optional[dict[str, Any]] Various metadata of export, if present. volumes : Optional[VolumeReader] Reader for volume data, if present. landmarks : Optional[LandmarkReader] Reader for landmark data, if present. neurons : Optional[SkeletonReader] Reader for neuronal/ skeleton data, if present. annotations : Optional[AnnotationReader] Reader for annotation data, if present. """ def __init__(self, dpath: Path) -> None: """ Parameters ---------- dpath : Path Directory in which all data is saved. """ self.dpath = Path(dpath) meta_path = self.dpath / "metadata.toml" if meta_path.is_file(): self.metadata = read_toml(meta_path) else: self.metadata = None self.volumes = ( VolumeReader(dpath / "volumes") if (dpath / "volumes").is_dir() else None ) self.landmarks = ( LandmarkReader(dpath / "landmarks") if (dpath / "landmarks").is_dir() else None ) self.neurons = ( SkeletonReader( dpath / "neurons", self.metadata.get("units") if self.metadata else None, ) if (dpath / "neurons").is_dir() else None ) self.annotations = ( AnnotationReader(dpath / "annotations") if (dpath / "annotations").is_dir() else None ) def get_metadata(self, *keys: str, default=NO_DEFAULT): """Get values from nested metadata dict. e.g. to retrieve possibly-absent ``myvalue`` from metadata like ``{"A": {"B": myvalue}}``, use ``my_reader.get_metadata("A", "B", default=None)``. Parameters ---------- *keys : str String keys for accessing nested values. default : any, optional Value to return if key is not present, at any level. By default, raises a ``KeyError``. Returns ------- Any Raises ------ KeyError Key does not exist and no default given. """ if self.metadata is None: if default is not NO_DEFAULT: return default elif keys: raise KeyError(keys[0]) else: return None d = self.metadata for k in keys: try: d = d[k] except KeyError as e: if default is NO_DEFAULT: raise e return default return d def get_full_annotation_graph(self) -> nx.DiGraph: """Get annotation graph including meta-annotations and neurons. Returns ------- nx.DiGraph Edges are from annotation name to annotation, neuronm or volume name. Nodes have attribute ``"type"``, which is either ``"annotation"``, ``"neuron"``, or ``"volume"``. Edges have a boolean attribute ``"meta_annotation"`` (whether the target is an annotation). """ g = nx.DiGraph() if self.annotations: g.update(self.annotations.get_graph()) if self.neurons: g.update(self.neurons.get_annotation_graph()) if self.volumes: g.update(self.volumes.get_annotation_graph()) return g
Methods
def get_full_annotation_graph(self) ‑> networkx.classes.digraph.DiGraph
-
Get annotation graph including meta-annotations and neurons.
Returns
nx.DiGraph
- Edges are from annotation name to annotation, neuronm or volume name.
Nodes have attribute
"type"
, which is either"annotation"
,"neuron"
, or"volume"
. Edges have a boolean attribute"meta_annotation"
(whether the target is an annotation).
Expand source code
def get_full_annotation_graph(self) -> nx.DiGraph: """Get annotation graph including meta-annotations and neurons. Returns ------- nx.DiGraph Edges are from annotation name to annotation, neuronm or volume name. Nodes have attribute ``"type"``, which is either ``"annotation"``, ``"neuron"``, or ``"volume"``. Edges have a boolean attribute ``"meta_annotation"`` (whether the target is an annotation). """ g = nx.DiGraph() if self.annotations: g.update(self.annotations.get_graph()) if self.neurons: g.update(self.neurons.get_annotation_graph()) if self.volumes: g.update(self.volumes.get_annotation_graph()) return g
def get_metadata(self, *keys: str, default=<object object>)
-
Get values from nested metadata dict.
e.g. to retrieve possibly-absent
myvalue
from metadata like{"A": {"B": myvalue}}
, usemy_reader.get_metadata("A", "B", default=None)
.Parameters
*keys
:str
- String keys for accessing nested values.
default
:any
, optional- Value to return if key is not present, at any level.
By default, raises a
KeyError
.
Returns
Any
Raises
KeyError
- Key does not exist and no default given.
Expand source code
def get_metadata(self, *keys: str, default=NO_DEFAULT): """Get values from nested metadata dict. e.g. to retrieve possibly-absent ``myvalue`` from metadata like ``{"A": {"B": myvalue}}``, use ``my_reader.get_metadata("A", "B", default=None)``. Parameters ---------- *keys : str String keys for accessing nested values. default : any, optional Value to return if key is not present, at any level. By default, raises a ``KeyError``. Returns ------- Any Raises ------ KeyError Key does not exist and no default given. """ if self.metadata is None: if default is not NO_DEFAULT: return default elif keys: raise KeyError(keys[0]) else: return None d = self.metadata for k in keys: try: d = d[k] except KeyError as e: if default is NO_DEFAULT: raise e return default return d
class LandmarkReader (dpath: Path)
-
Class for reading exported landmark data.
Parameters
dpath
:Path
- Directory in which landmark data is saved.
Expand source code
class LandmarkReader: """Class for reading exported landmark data.""" def __init__(self, dpath: Path) -> None: """ Parameters ---------- dpath : Path Directory in which landmark data is saved. """ self.dpath = dpath self.fpath = dpath / "locations.json" @copy_cache() def _locations(self): with open(self.fpath) as f: d = json.load(f) return [Location.from_jso(loc) for loc in d] def get_all(self) -> Iterable[Location]: """Lazily iterate through landmark locations. Yields ------ Location """ yield from self._locations() def get_group_names(self) -> set[str]: """Return all groups with locations in the dataset. Returns ------- set[str] Set of group names. """ out = set() for loc in self._locations(): out.update(loc.groups) return out def get_landmark_names(self) -> set[str]: """Return all landmarks with locations in the dataset. Returns ------- set[str] Set of landmark names. """ out = set() for loc in self._locations(): out.update(loc.landmarks) return out def get_group(self, *group: str) -> Iterable[Location]: """Lazily iterate through all locations from any of the given groups. Parameters ---------- group : str Group name (can give multiple as *args). Yields ------ Location """ groupset = set(group) for loc in self._locations(): if not loc.groups.isdisjoint(groupset): yield loc def get_landmark(self, *landmark: str) -> Iterable[Location]: """Lazily iterate through all locations from any of the given landmarks. Parameters ---------- landmark : str Landmark name (can give multiple as *args) Yields ------ Location """ lmarkset = set(landmark) for loc in self._locations(): if not loc.landmarks.isdisjoint(lmarkset): yield loc def get_paired_locations( self, group1: str, group2: str ) -> Iterable[tuple[Location, Location]]: """Iterate through paired locations. Locations are paired when both belong to the same landmark, and each location is the only one of that landmark to exist in that group, and they are not the same location. This is useful for creating transformations between two spaces (as landmark groups) by shared features (as landmarks). Parameters ---------- group1 : str Group name group2 : str Group name Yields ------ tuple[Location, Location] """ la_lo1: dict[str, list[Location]] = dict() la_lo2: dict[str, list[Location]] = dict() for loc in self._locations(): if group1 in loc.groups: if group2 in loc.groups: continue for landmark in loc.landmarks: la_lo1.setdefault(landmark, []).append(loc) elif group2 in loc.groups: for landmark in loc.landmarks: la_lo2.setdefault(landmark, []).append(loc) landmarks = sorted(set(la_lo1).intersection(la_lo2)) for la in landmarks: lo1 = la_lo1[la] if len(lo1) != 1: continue lo2 = la_lo2[la] if len(lo2) != 1: continue yield lo1[0], lo2[0]
Methods
def get_all(self) ‑> Iterable[Location]
-
Expand source code
def get_all(self) -> Iterable[Location]: """Lazily iterate through landmark locations. Yields ------ Location """ yield from self._locations()
def get_group(self, *group: str) ‑> Iterable[Location]
-
Lazily iterate through all locations from any of the given groups.
Parameters
group
:str
- Group name (can give multiple as *args).
Yields
Expand source code
def get_group(self, *group: str) -> Iterable[Location]: """Lazily iterate through all locations from any of the given groups. Parameters ---------- group : str Group name (can give multiple as *args). Yields ------ Location """ groupset = set(group) for loc in self._locations(): if not loc.groups.isdisjoint(groupset): yield loc
def get_group_names(self) ‑> set[str]
-
Return all groups with locations in the dataset.
Returns
set[str]
- Set of group names.
Expand source code
def get_group_names(self) -> set[str]: """Return all groups with locations in the dataset. Returns ------- set[str] Set of group names. """ out = set() for loc in self._locations(): out.update(loc.groups) return out
def get_landmark(self, *landmark: str) ‑> Iterable[Location]
-
Lazily iterate through all locations from any of the given landmarks.
Parameters
landmark
:str
- Landmark name (can give multiple as *args)
Yields
Expand source code
def get_landmark(self, *landmark: str) -> Iterable[Location]: """Lazily iterate through all locations from any of the given landmarks. Parameters ---------- landmark : str Landmark name (can give multiple as *args) Yields ------ Location """ lmarkset = set(landmark) for loc in self._locations(): if not loc.landmarks.isdisjoint(lmarkset): yield loc
def get_landmark_names(self) ‑> set[str]
-
Return all landmarks with locations in the dataset.
Returns
set[str]
- Set of landmark names.
Expand source code
def get_landmark_names(self) -> set[str]: """Return all landmarks with locations in the dataset. Returns ------- set[str] Set of landmark names. """ out = set() for loc in self._locations(): out.update(loc.landmarks) return out
def get_paired_locations(self, group1: str, group2: str) ‑> Iterable[tuple[Location, Location]]
-
Iterate through paired locations.
Locations are paired when both belong to the same landmark, and each location is the only one of that landmark to exist in that group, and they are not the same location.
This is useful for creating transformations between two spaces (as landmark groups) by shared features (as landmarks).
Parameters
group1
:str
- Group name
group2
:str
- Group name
Yields
Expand source code
def get_paired_locations( self, group1: str, group2: str ) -> Iterable[tuple[Location, Location]]: """Iterate through paired locations. Locations are paired when both belong to the same landmark, and each location is the only one of that landmark to exist in that group, and they are not the same location. This is useful for creating transformations between two spaces (as landmark groups) by shared features (as landmarks). Parameters ---------- group1 : str Group name group2 : str Group name Yields ------ tuple[Location, Location] """ la_lo1: dict[str, list[Location]] = dict() la_lo2: dict[str, list[Location]] = dict() for loc in self._locations(): if group1 in loc.groups: if group2 in loc.groups: continue for landmark in loc.landmarks: la_lo1.setdefault(landmark, []).append(loc) elif group2 in loc.groups: for landmark in loc.landmarks: la_lo2.setdefault(landmark, []).append(loc) landmarks = sorted(set(la_lo1).intersection(la_lo2)) for la in landmarks: lo1 = la_lo1[la] if len(lo1) != 1: continue lo2 = la_lo2[la] if len(lo2) != 1: continue yield lo1[0], lo2[0]
class Location (xyz: tuple[float, float, float], groups: set[str] = <factory>, landmarks: set[str] = <factory>)
-
Location of importance to landmarks and groups.
Attributes
xyz
:tuple[float, float, float]
- Coordinates of location
groups
:set[str]
- Set of landmark groups this location belongs to.
landmarks
:set[str]
- Set of landmarks this location belongs to.
Expand source code
@dataclass class Location: """Location of importance to landmarks and groups. Attributes ---------- xyz : tuple[float, float, float] Coordinates of location groups : set[str] Set of landmark groups this location belongs to. landmarks : set[str] Set of landmarks this location belongs to. """ xyz: tuple[float, float, float] groups: set[str] = field(default_factory=set) landmarks: set[str] = field(default_factory=set) def to_jso(self) -> dict[str, Any]: """Convert to JSON-serialisable object. Returns ------- dict[str, Any] """ d = asdict(self) d["xyz"] = list(d["xyz"]) d["groups"] = sorted(d["groups"]) d["landmarks"] = sorted(d["landmarks"]) return d @classmethod def from_jso(cls, jso: dict[str, Any]) -> Location: """Instantiate from JSON-like dict. Parameters ---------- jso : dict[str, Any] Keys ``"xyz"`` (3-length list of float), ``"groups"`` (list of str), ``"landmarks"`` (list of str) Returns ------- Location """ return cls( tuple(jso["xyz"]), set(jso["groups"]), set(jso["landmarks"]), )
Class variables
var groups : set[str]
var landmarks : set[str]
var xyz : tuple[float, float, float]
Static methods
def from_jso(jso: dict[str, Any]) ‑> Location
-
Instantiate from JSON-like dict.
Parameters
jso
:dict[str, Any]
- Keys
"xyz"
(3-length list of float),"groups"
(list of str),"landmarks"
(list of str)
Returns
Expand source code
@classmethod def from_jso(cls, jso: dict[str, Any]) -> Location: """Instantiate from JSON-like dict. Parameters ---------- jso : dict[str, Any] Keys ``"xyz"`` (3-length list of float), ``"groups"`` (list of str), ``"landmarks"`` (list of str) Returns ------- Location """ return cls( tuple(jso["xyz"]), set(jso["groups"]), set(jso["landmarks"]), )
Methods
def to_jso(self) ‑> dict[str, typing.Any]
-
Convert to JSON-serialisable object.
Returns
dict[str, Any]
Expand source code
def to_jso(self) -> dict[str, Any]: """Convert to JSON-serialisable object. Returns ------- dict[str, Any] """ d = asdict(self) d["xyz"] = list(d["xyz"]) d["groups"] = sorted(d["groups"]) d["landmarks"] = sorted(d["landmarks"]) return d
class ReadSpec (nodes: bool = True, connectors: bool = True, tags: bool = True)
-
Specify a subset of a skeleton's data to read.
Expand source code
class ReadSpec(NamedTuple): """Specify a subset of a skeleton's data to read.""" nodes: bool = True connectors: bool = True tags: bool = True def copy(self, nodes=None, connectors=None, tags=None): return type(self)( nodes=nodes if nodes is not None else self.nodes, connectors=connectors if connectors is not None else self.connectors, tags=tags if tags is not None else self.tags, )
Ancestors
- builtins.tuple
Instance variables
var connectors : bool
-
Alias for field number 1
var nodes : bool
-
Alias for field number 0
-
Alias for field number 2
Methods
def copy(self, nodes=None, connectors=None, tags=None)
-
Expand source code
def copy(self, nodes=None, connectors=None, tags=None): return type(self)( nodes=nodes if nodes is not None else self.nodes, connectors=connectors if connectors is not None else self.connectors, tags=tags if tags is not None else self.tags, )
class SkeletonReader (dpath: pathlib.Path, units=None, read_spec=ReadSpec(nodes=True, connectors=True, tags=True))
-
Class for reading exported skeletonised neuron data.
Most "get_" methods take a
read_spec
argument. This is a 3-(named)tuple of bools representing whether to populate the nodes, connectors, and tags fields of the returned TreeNeuron objects. By default, all will be populated. Metadata is always populated.Parameters
dpath
:Path
- Directory in which the neuron data is saved.
Expand source code
class SkeletonReader: """Class for reading exported skeletonised neuron data. Most "get_" methods take a ``read_spec`` argument. This is a 3-(named)tuple of bools representing whether to populate the nodes, connectors, and tags fields of the returned TreeNeuron objects. By default, all will be populated. Metadata is always populated. """ def __init__(self, dpath: Path, units=None, read_spec=ReadSpec()) -> None: """ Parameters ---------- dpath : Path Directory in which the neuron data is saved. """ self.dpath = dpath self.units = units self.default_read_spec = ReadSpec(*read_spec) @copy_cache(maxsize=CACHE_SIZE) def _read_meta(self, dpath): return json.loads((dpath / "metadata.json").read_text()) @copy_cache(maxsize=CACHE_SIZE) def _read_nodes(self, dpath): return pd.read_csv(dpath / "nodes.tsv", sep="\t") @copy_cache(maxsize=CACHE_SIZE) def _read_tags(self, dpath): return json.loads((dpath / "tags.json").read_text()) @copy_cache(maxsize=CACHE_SIZE) def _read_connectors(self, dpath): conns = pd.read_csv(dpath / "connectors.tsv", sep="\t") conns.rename(columns={"is_input": "type"}, inplace=True) return conns def parse_read_spec(self, read_spec: Optional[Sequence[bool]] = None) -> ReadSpec: if read_spec is None: read_spec = self.default_read_spec else: read_spec = ReadSpec(*read_spec) return read_spec def _construct_neuron(self, meta, nodes=None, tags=None, connectors=None): nrn = navis.TreeNeuron(nodes, self.units, annotations=meta["annotations"]) nrn.id = meta["id"] nrn.name = meta["name"] nrn.soma = meta["soma_id"] nrn.tags = tags nrn.connectors = connectors return nrn def _read_neuron( self, dpath, read_spec: Optional[ReadSpec] = None ) -> navis.TreeNeuron: read_spec = self.parse_read_spec(read_spec) meta = self._read_meta(dpath) if read_spec.nodes: nodes = self._read_nodes(dpath) else: nodes = None if read_spec.tags: tags = self._read_tags(dpath) else: tags = None if read_spec.connectors: connectors = self._read_connectors(dpath) else: connectors = None return self._construct_neuron(meta, nodes, tags, connectors) def get_by_id( self, skeleton_id: int, read_spec: Optional[ReadSpec] = None ) -> navis.TreeNeuron: """Read neuron with the given skeleton ID. Parameters ---------- skeleton_id : int Returns ------- navis.TreeNeuron """ return self._read_neuron(self.dpath / str(skeleton_id), read_spec) def _iter_dirs(self): for path in self.dpath.iterdir(): if path.is_dir(): yield path @lru_cache def name_to_id(self) -> dict[str, int]: """Mapping from neuron name to skeleton ID. Returns ------- dict[str, int] """ out = dict() for dpath in self._iter_dirs(): meta = self._read_meta(dpath) out[meta["name"]] = meta["id"] return out @lru_cache def annotation_to_ids(self) -> dict[str, list[int]]: """Which skeletons string annotations are applied to. Returns ------- dict[str, list[int]] Mapping from annotation name to list of skeleton IDs. """ out: dict[str, list[int]] = dict() for dpath in self._iter_dirs(): meta = self._read_meta(dpath) for ann in meta["annotations"]: out.setdefault(ann, []).append(meta["id"]) return out def get_by_name( self, name: str, read_spec: Optional[ReadSpec] = None ) -> navis.TreeNeuron: """Read neuron with the given name. Parameters ---------- name : str Exact neuron name. Returns ------- navis.TreeNeuron """ d = self.name_to_id() return self.get_by_id(d[name], read_spec) def get_by_annotation( self, annotation: str, read_spec: Optional[ReadSpec] = None ) -> Iterable[navis.TreeNeuron]: """Lazily iterate through neurons with the given annotation. Parameters ---------- annotation : str Exact annotation. Yields ------ navis.TreeNeuron """ d = self.annotation_to_ids() for skid in d[annotation]: yield self.get_by_id(skid, read_spec) def get_annotation_names(self) -> set[str]: """Return all annotations represented in the dataset. Returns ------- set[str] Set of annotation names. """ d = self.annotation_to_ids() return set(d) def get_annotation_graph(self) -> nx.DiGraph: """Return graph of neuron annotations. Returns ------- nx.DiGraph Edges are from annotations to neuron names. All nodes have attribute ``"type"``, which is either ``"neuron"`` or ``"annotation"``. All edges have attribute ``"meta_annotation"=False``. """ g = nx.DiGraph() anns = set() neurons = set() for dpath in self._iter_dirs(): meta = self._read_meta(dpath) name = meta["name"] neurons.add(name) for ann in meta["annotations"]: anns.add(ann) g.add_edge(ann, name, meta_annotation=False) ann_data = {"type": "annotation"} for ann in anns: g.nodes[ann].update(ann_data) return g def get_all( self, read_spec: Optional[ReadSpec] = None ) -> Iterable[navis.TreeNeuron]: """Lazily iterate through neurons in arbitrary order. Can be used for filtering neurons based on some metadata, e.g. lefts = [] for nrn in my_reader.get_all(ReadSpec(False, False, False)): if "left" in nrn.name: lefts.append(my_reader.get_by_id(nrn.id))) Yields ------ navis.TreeNeuron """ for dpath in self._iter_dirs(): yield self._read_neuron(dpath, read_spec)
Methods
def annotation_to_ids(self) ‑> dict[str, list[int]]
-
Which skeletons string annotations are applied to.
Returns
dict[str, list[int]]
- Mapping from annotation name to list of skeleton IDs.
Expand source code
@lru_cache def annotation_to_ids(self) -> dict[str, list[int]]: """Which skeletons string annotations are applied to. Returns ------- dict[str, list[int]] Mapping from annotation name to list of skeleton IDs. """ out: dict[str, list[int]] = dict() for dpath in self._iter_dirs(): meta = self._read_meta(dpath) for ann in meta["annotations"]: out.setdefault(ann, []).append(meta["id"]) return out
def get_all(self, read_spec: Optional[ReadSpec] = None) ‑> collections.abc.Iterable[navis.core.skeleton.TreeNeuron]
-
Lazily iterate through neurons in arbitrary order.
Can be used for filtering neurons based on some metadata, e.g.
lefts = [] for nrn in my_reader.get_all(ReadSpec(False, False, False)): if "left" in nrn.name: lefts.append(my_reader.get_by_id(nrn.id)))
Yields
navis.TreeNeuron
Expand source code
def get_all( self, read_spec: Optional[ReadSpec] = None ) -> Iterable[navis.TreeNeuron]: """Lazily iterate through neurons in arbitrary order. Can be used for filtering neurons based on some metadata, e.g. lefts = [] for nrn in my_reader.get_all(ReadSpec(False, False, False)): if "left" in nrn.name: lefts.append(my_reader.get_by_id(nrn.id))) Yields ------ navis.TreeNeuron """ for dpath in self._iter_dirs(): yield self._read_neuron(dpath, read_spec)
def get_annotation_graph(self) ‑> networkx.classes.digraph.DiGraph
-
Return graph of neuron annotations.
Returns
nx.DiGraph
- Edges are from annotations to neuron names.
All nodes have attribute
"type"
, which is either"neuron"
or"annotation"
. All edges have attribute"meta_annotation"=False
.
Expand source code
def get_annotation_graph(self) -> nx.DiGraph: """Return graph of neuron annotations. Returns ------- nx.DiGraph Edges are from annotations to neuron names. All nodes have attribute ``"type"``, which is either ``"neuron"`` or ``"annotation"``. All edges have attribute ``"meta_annotation"=False``. """ g = nx.DiGraph() anns = set() neurons = set() for dpath in self._iter_dirs(): meta = self._read_meta(dpath) name = meta["name"] neurons.add(name) for ann in meta["annotations"]: anns.add(ann) g.add_edge(ann, name, meta_annotation=False) ann_data = {"type": "annotation"} for ann in anns: g.nodes[ann].update(ann_data) return g
def get_annotation_names(self) ‑> set[str]
-
Return all annotations represented in the dataset.
Returns
set[str]
- Set of annotation names.
Expand source code
def get_annotation_names(self) -> set[str]: """Return all annotations represented in the dataset. Returns ------- set[str] Set of annotation names. """ d = self.annotation_to_ids() return set(d)
def get_by_annotation(self, annotation: str, read_spec: Optional[ReadSpec] = None) ‑> collections.abc.Iterable[navis.core.skeleton.TreeNeuron]
-
Lazily iterate through neurons with the given annotation.
Parameters
annotation
:str
- Exact annotation.
Yields
navis.TreeNeuron
Expand source code
def get_by_annotation( self, annotation: str, read_spec: Optional[ReadSpec] = None ) -> Iterable[navis.TreeNeuron]: """Lazily iterate through neurons with the given annotation. Parameters ---------- annotation : str Exact annotation. Yields ------ navis.TreeNeuron """ d = self.annotation_to_ids() for skid in d[annotation]: yield self.get_by_id(skid, read_spec)
def get_by_id(self, skeleton_id: int, read_spec: Optional[ReadSpec] = None) ‑> navis.core.skeleton.TreeNeuron
-
Read neuron with the given skeleton ID.
Parameters
skeleton_id
:int
Returns
navis.TreeNeuron
Expand source code
def get_by_id( self, skeleton_id: int, read_spec: Optional[ReadSpec] = None ) -> navis.TreeNeuron: """Read neuron with the given skeleton ID. Parameters ---------- skeleton_id : int Returns ------- navis.TreeNeuron """ return self._read_neuron(self.dpath / str(skeleton_id), read_spec)
def get_by_name(self, name: str, read_spec: Optional[ReadSpec] = None) ‑> navis.core.skeleton.TreeNeuron
-
Read neuron with the given name.
Parameters
name
:str
- Exact neuron name.
Returns
navis.TreeNeuron
Expand source code
def get_by_name( self, name: str, read_spec: Optional[ReadSpec] = None ) -> navis.TreeNeuron: """Read neuron with the given name. Parameters ---------- name : str Exact neuron name. Returns ------- navis.TreeNeuron """ d = self.name_to_id() return self.get_by_id(d[name], read_spec)
def name_to_id(self) ‑> dict[str, int]
-
Mapping from neuron name to skeleton ID.
Returns
dict[str, int]
Expand source code
@lru_cache def name_to_id(self) -> dict[str, int]: """Mapping from neuron name to skeleton ID. Returns ------- dict[str, int] """ out = dict() for dpath in self._iter_dirs(): meta = self._read_meta(dpath) out[meta["name"]] = meta["id"] return out
def parse_read_spec(self, read_spec: Optional[Sequence[bool]] = None) ‑> ReadSpec
-
Expand source code
def parse_read_spec(self, read_spec: Optional[Sequence[bool]] = None) -> ReadSpec: if read_spec is None: read_spec = self.default_read_spec else: read_spec = ReadSpec(*read_spec) return read_spec
class VolumeReader (dpath: pathlib.Path)
-
Class for reading exported volume data.
Parameters
dpath
:Path
- Path to directory in which the volume data is saved.
Expand source code
class VolumeReader: """Class for reading exported volume data.""" def __init__(self, dpath: Path) -> None: """ Parameters ---------- dpath : Path Path to directory in which the volume data is saved. """ self.dpath = dpath self._names_df = None @property def names_df(self) -> pd.DataFrame: """Dataframe representing ``names.tsv``. Returns ------- pd.DataFrame Columns ``filename``, ``volume_name`` """ if self._names_df is None: self._names_df = pd.read_csv( self.dpath / "names.tsv", sep="\t", ) return self._names_df @lru_cache def _dict(self, keys, values): return df_to_dict(self.names_df, keys, values) @copy_cache(maxsize=None) def _get_annotations(self) -> dict[str, set[str]]: """Map annotation names to volume names. Returns ------- dict[str, set[str]] """ d = json.loads((self.dpath / "annotations.json").read_text()) return {k: set(v) for k, v in d.items()} def get_annotation_graph(self) -> nx.DiGraph: """Get graph of annotations to volumes. Returns ------- networkx.DiGraph """ g = nx.DiGraph() for k, vs in self._get_annotations().items(): g.add_node(k, type="annotation") for v in vs: if v not in g.nodes: g.add_node(v, type="volume") g.add_edge(k, v, meta_annotation=False) return g def _annotations_for_volume(self, name: str): d = self._get_annotations() return {a for a, names in d.items() if name in names} @copy_cache(maxsize=CACHE_SIZE) def _read_vol( self, fpath: Path, name: Optional[str], volume_id: Optional[int] ) -> AnnotatedVolume: vol = AnnotatedVolume.from_file(fpath) if name is not None: d = self._dict("filename", "volume_name") name = d[fpath.name] vol.name = name vol.annotations.update(self._annotations_for_volume(name)) if volume_id is None: volume_id = int(fpath.stem) vol.id = volume_id return vol def get_by_id(self, volume_id: int) -> AnnotatedVolume: """Read a volume with a given (arbitrary) ID. Parameters ---------- volume_id : int Returns ------- AnnotatedVolume """ return self._read_vol( self.dpath / f"{volume_id}.stl", None, volume_id, ) def get_by_name(self, volume_name: str) -> AnnotatedVolume: """Read a volume with a given name. Parameters ---------- volume_name : str Returns ------- AnnotatedVolume """ d = self._dict("volume_name", "filename") fname = d[volume_name] path = self.dpath / fname return self._read_vol(path, volume_name, None) def get_by_annotation(self, annotation: str) -> Iterable[AnnotatedVolume]: """Lazily iterate through all volumes with the given annotation. Parameters ---------- annotation : str Annotation name. Yields ------ Iterable[AnnotatedVolume] """ d = self._get_annotations() for vol_name in d[annotation]: yield self.get_by_name(vol_name) def get_all(self) -> Iterable[AnnotatedVolume]: """Lazily iterate through all available volumes. Iteration is in the order used by ``names.tsv``. Yields ------ AnnotatedVolume """ for fname, name in self._dict("filename", "volume_name").items(): fpath = self.dpath / fname yield self._read_vol(fpath, name, None)
Instance variables
var names_df : pandas.core.frame.DataFrame
-
Dataframe representing
names.tsv
.Returns
pd.DataFrame
- Columns
filename
,volume_name
Expand source code
@property def names_df(self) -> pd.DataFrame: """Dataframe representing ``names.tsv``. Returns ------- pd.DataFrame Columns ``filename``, ``volume_name`` """ if self._names_df is None: self._names_df = pd.read_csv( self.dpath / "names.tsv", sep="\t", ) return self._names_df
Methods
def get_all(self) ‑> Iterable[AnnotatedVolume]
-
Lazily iterate through all available volumes.
Iteration is in the order used by
names.tsv
.Yields
Expand source code
def get_all(self) -> Iterable[AnnotatedVolume]: """Lazily iterate through all available volumes. Iteration is in the order used by ``names.tsv``. Yields ------ AnnotatedVolume """ for fname, name in self._dict("filename", "volume_name").items(): fpath = self.dpath / fname yield self._read_vol(fpath, name, None)
def get_annotation_graph(self) ‑> networkx.classes.digraph.DiGraph
-
Get graph of annotations to volumes.
Returns
networkx.DiGraph
Expand source code
def get_annotation_graph(self) -> nx.DiGraph: """Get graph of annotations to volumes. Returns ------- networkx.DiGraph """ g = nx.DiGraph() for k, vs in self._get_annotations().items(): g.add_node(k, type="annotation") for v in vs: if v not in g.nodes: g.add_node(v, type="volume") g.add_edge(k, v, meta_annotation=False) return g
def get_by_annotation(self, annotation: str) ‑> Iterable[AnnotatedVolume]
-
Lazily iterate through all volumes with the given annotation.
Parameters
annotation
:str
- Annotation name.
Yields
Iterable[AnnotatedVolume]
Expand source code
def get_by_annotation(self, annotation: str) -> Iterable[AnnotatedVolume]: """Lazily iterate through all volumes with the given annotation. Parameters ---------- annotation : str Annotation name. Yields ------ Iterable[AnnotatedVolume] """ d = self._get_annotations() for vol_name in d[annotation]: yield self.get_by_name(vol_name)
def get_by_id(self, volume_id: int) ‑> AnnotatedVolume
-
Expand source code
def get_by_id(self, volume_id: int) -> AnnotatedVolume: """Read a volume with a given (arbitrary) ID. Parameters ---------- volume_id : int Returns ------- AnnotatedVolume """ return self._read_vol( self.dpath / f"{volume_id}.stl", None, volume_id, )
def get_by_name(self, volume_name: str) ‑> AnnotatedVolume
-
Expand source code
def get_by_name(self, volume_name: str) -> AnnotatedVolume: """Read a volume with a given name. Parameters ---------- volume_name : str Returns ------- AnnotatedVolume """ d = self._dict("volume_name", "filename") fname = d[volume_name] path = self.dpath / fname return self._read_vol(path, volume_name, None)