kedro.io.AbstractDataSet¶
-
class
kedro.io.
AbstractDataSet
[source]¶ Bases:
abc.ABC
AbstractDataSet
is the base class for all data set implementations. All data set implementations should extend this abstract class and implement the methods marked as abstract.Example:
from pathlib import Path, PurePosixPath import pandas as pd from kedro.io import AbstractDataSet class MyOwnDataSet(AbstractDataSet): def __init__(self, filepath, param1, param2=True): self._filepath = PurePosixPath(filepath) self._param1 = param1 self._param2 = param2 def _load(self) -> pd.DataFrame: return pd.read_csv(self._filepath) def _save(self, df: pd.DataFrame) -> None: df.to_csv(str(self._filepath)) def _exists(self) -> bool: return Path(self._filepath.as_posix()).exists() def _describe(self): return dict(param1=self._param1, param2=self._param2)
Example catalog.yml specification:
my_dataset: type: <path-to-my-own-dataset>.MyOwnDataSet filepath: data/01_raw/my_data.csv param1: <param1-value> # param1 is a required argument # param2 will be True by default
Attributes
Methods
Checks whether a data set’s output already exists by calling the provided _exists() method.
AbstractDataSet.from_config
(name, config[, …])Create a data set instance using the configuration provided.
Loads data by delegation to the provided load method.
Release any cached data.
AbstractDataSet.save
(data)Saves data by delegation to the provided save method.
-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
exists
()[source]¶ Checks whether a data set’s output already exists by calling the provided _exists() method.
- Return type
bool
- Returns
Flag indicating whether the output already exists.
- Raises
DataSetError – when underlying exists method raises error.
-
classmethod
from_config
(name, config, load_version=None, save_version=None)[source]¶ Create a data set instance using the configuration provided.
- Parameters
name (
str
) – Data set name.config (
Dict
[str
,Any
]) – Data set config dictionary.load_version (
Optional
[str
]) – Version string to be used forload
operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.save_version (
Optional
[str
]) – Version string to be used forsave
operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
- Return type
AbstractDataSet
- Returns
An instance of an
AbstractDataSet
subclass.- Raises
DataSetError – When the function fails to create the data set from its config.
-
load
()[source]¶ Loads data by delegation to the provided load method.
- Return type
Any
- Returns
Data returned by the provided load method.
- Raises
DataSetError – When underlying load method raises error.
-
release
()[source]¶ Release any cached data.
- Raises
DataSetError – when underlying release method raises error.
- Return type
None
-
save
(data)[source]¶ Saves data by delegation to the provided save method.
- Parameters
data (
Any
) – the value to be saved by provided save method.- Raises
DataSetError – when underlying save method raises error.
- Return type
None
-