Metadata-Version: 2.4
Name: file_golem
Version: 0.1.15
Summary: A file i/o utility to handle complex projects with multiple datasets, project locations and calls to 3rd party codebases.
Home-page: https://github.com/CameronBraunstein/file_golem
Author: Cameron Braunstein
Author-email: your.email@example.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: omegaconf
Requires-Dist: filelock
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

This package handles file I/O for larger projects, with the aim to simplify/standardize working with many datasets and 3rd party codebases across different naming conventions, data types, and storage systems. 

To run unit tests, run 

```
$python -m unittest discover -s tests


$for dev, it is better to run pip install -e .
```

## Project set up

## System Identity File
Create a file named `.system_identity.txt` at the top level project directory. Decide on a name for your local system, and add it as a single line to the the file as:

<my local system name>


In conf/system_configs/system_conf.yaml, input the following:

## System config

Create a config file in conf/system_configs/system_conf.yaml. This holds information file_golem needs to navigate your file systems: 
```
filelock_timeout: <number of seconds to wait for filelock (default: wait indefinitely)>

system_paths:
  <my local system name>:
    base_data_directory: <project directory (output of pwd())>

    base_conda_name: <path to conda environment when calling conda env list>
    external_calls:
      <name of external codebase used in code>:
        entry_point: <absolute path of top project directory>
        conda_path: <path to external environment when calling conda env list>
        python_entry_point: <Name of the python file (from top level project directory sans .py)>


    (optional) data_directory_name: <name of subdirectory where untracked data is stored > (default: data)

    system_transfers:
        <non local mounted system name>:
            mount_point: <absolute path of the mount point on your local system>
            data_directory_extension: <path to the root of the data directory on the mounted system> (file_folem will look in mount_point/data_directory_extension) 
```
base_data_directory can be customized to an alternative location if desired.


## Intializing a FileGolem object

It is best practice to have a saved system config file. From there, the syntax for initializing a FileGolem object is

```
data_io = FileGolem(
    system_config_path = <absolute system config path>,
    is_debug = <True or False if debug messages are desired>
)

```

If you need to quickly initialize a FileGolem without a saved config file, the syntax is:

```
system_config = {
    (See above)
}
data_io = FileGolem(
    system_config = system_config,
    is_debug = <True or False if debug messages are desired>
)

```



## Building a file path

FilePathEntries.CUSTOM_LOGIC : '<my_static_function>'

Will call my_static_function(data_args)

which will include data_io and data_type


## Dealing with h5py files

H5 files efficiently store data at scale. file_golem standardizes i/o for these files. First, create a class that inherits from H5PYAbstractDatatype:

```
from file_golem import H5PYAbstractDatatype
class <my h5py datatype>(H5PYAbstractDatatype):
    RELATIVE_PATH_TUPLE = H5PYAbstractDatatype.RELATIVE_PATH_TUPLE + (
        <path extension 1>, <path extension 2>, ...
        )

    FILE_STRUCTURE = {
        <node name a>: {
            H5PYAbstractDatatype.DATASET_DATATYPE: <file_golem datatype of <node name a> or H5PYAbstractDatatype.ATTRITBUTE>
            }
        <group name 0>: {
            <node name a>: {
                <file_golem datatype of <node name a> >
                },
            },
        <group name 1>: {
            <node name b>: {
                <file_golem datatype of <node name b> >,
                }
            },
        <group name 2>: {
            <group name 3>: {
                ...
            }
            <node name c>: {
                <file_golem datatype of <node name c> >
                },
            },
        }
```

To perform i/o, one can call the typical file_golem calls, 

```
data_io.load_data(<my h5py datatype>, data_args={
    <my h5py datatype>.DATA: <torch or numpy tensor to save>
    <my h5py datatype>.DATASET_ADDRESS: [<group name 2>,<group name 3>,...]
    <my h5py datatype>.DATASET_SIZE: <shape of the node tensor>
    <my h5py datatype>.H5PY_IDX: <single index or tensor of indices of the full tensor>
    ...

    }
)
```
