kedro.extras.datasets.pandas.SQLTableDataSet¶
-
class
kedro.extras.datasets.pandas.
SQLTableDataSet
(table_name, credentials, load_args=None, save_args=None)[source]¶ Bases:
kedro.io.core.AbstractDataSet
SQLTableDataSet
loads data from a SQL table and saves a pandas dataframe to a table. It usespandas.DataFrame
internally, so it supports all allowed pandas options onread_sql_table
andto_sql
methods. Since Pandas uses SQLAlchemy behind the scenes, when instantiatingSQLTableDataSet
one needs to pass a compatible connection string either incredentials
(see the example code snippet below) or inload_args
andsave_args
. Connection string formats supported by SQLAlchemy can be found here: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urlsSQLTableDataSet
modifies the save parameters and stores the data with no index. This is designed to make load and save methods symmetric.Example:
from kedro.extras.datasets.pandas import SQLTableDataSet import pandas as pd data = pd.DataFrame({"col1": [1, 2], "col2": [4, 5], "col3": [5, 6]}) table_name = "table_a" credentials = { "con": "postgresql://scott:tiger@localhost/test" } data_set = SQLTableDataSet(table_name=table_name, credentials=credentials) data_set.save(data) reloaded = data_set.load() assert data.equals(reloaded)
Attributes
Methods
Checks whether a data set’s output already exists by calling the provided _exists() method.
SQLTableDataSet.from_config
(name, config[, …])Create a data set instance using the configuration provided.
Loads data by delegation to the provided load method.
Release any cached data.
SQLTableDataSet.save
(data)Saves data by delegation to the provided save method.
-
DEFAULT_LOAD_ARGS
: Dict[str, Any] = {}¶
-
DEFAULT_SAVE_ARGS
: Dict[str, Any] = {'index': False}¶
-
__init__
(table_name, credentials, load_args=None, save_args=None)[source]¶ Creates a new
SQLTableDataSet
.- Parameters
table_name (
str
) – The table name to load or save data to. It overwrites name insave_args
andtable_name
parameters inload_args
.credentials (
Dict
[str
,Any
]) – A dictionary with aSQLAlchemy
connection string. Users are supposed to provide the connection string ‘con’ through credentials. It overwrites con parameter inload_args
andsave_args
in case it is provided. To find all supported connection string formats, see here: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urlsload_args (
Optional
[Dict
[str
,Any
]]) – Provided to underlying pandasread_sql_table
function along with the connection string. To find all supported arguments, see here: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html To find all supported connection string formats, see here: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urlssave_args (
Optional
[Dict
[str
,Any
]]) – Provided to underlying pandasto_sql
function along with the connection string. To find all supported arguments, see here: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html To find all supported connection string formats, see here: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls It hasindex=False
in the default parameters.
- Raises
DataSetError – When either
table_name
orcon
is empty.
-
exists
()¶ Checks whether a data set’s output already exists by calling the provided _exists() method.
- Return type
bool
- Returns
Flag indicating whether the output already exists.
- Raises
DataSetError – when underlying exists method raises error.
-
classmethod
from_config
(name, config, load_version=None, save_version=None)¶ Create a data set instance using the configuration provided.
- Parameters
name (
str
) – Data set name.config (
Dict
[str
,Any
]) – Data set config dictionary.load_version (
Optional
[str
]) – Version string to be used forload
operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.save_version (
Optional
[str
]) – Version string to be used forsave
operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
- Return type
AbstractDataSet
- Returns
An instance of an
AbstractDataSet
subclass.- Raises
DataSetError – When the function fails to create the data set from its config.
-
load
()¶ Loads data by delegation to the provided load method.
- Return type
Any
- Returns
Data returned by the provided load method.
- Raises
DataSetError – When underlying load method raises error.
-
release
()¶ Release any cached data.
- Raises
DataSetError – when underlying release method raises error.
- Return type
None
-
save
(data)¶ Saves data by delegation to the provided save method.
- Parameters
data (
Any
) – the value to be saved by provided save method.- Raises
DataSetError – when underlying save method raises error.
- Return type
None
-