Metadata-Version: 2.4
Name: trueloss
Version: 0.2.1
Summary: A Keras utility for measuring training loss and metrics in evaluation mode for clearer comparison with validation performance.
Home-page: https://github.com/trueloss/trueloss
Author: Siddique Abusaleh
Author-email: trueloss.py@gmail.com
License: MIT
Project-URL: Documentation, https://github.com/trueloss/trueloss#readme
Project-URL: Source Code, https://github.com/trueloss/trueloss
Project-URL: Issue Tracker, https://github.com/trueloss/trueloss/issues
Keywords: keras,tensorflow,machine learning,deep learning,training loss,validation loss,regularization,model evaluation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Natural Language :: English
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: tensorflow>=2.16
Requires-Dist: keras>=2.0
Requires-Dist: matplotlib>=3.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# trueloss

`trueloss` is a Python library for measuring and plotting a Keras model's loss and metrics on the training data under evaluation conditions.

It adds these values to the normal Keras training history, making training-data performance more directly comparable with validation performance.

## Why trueloss?

Training loss and validation loss are not always measured under identical conditions.

During training:

* the model operates in training mode;
* dropout and other training-only behaviour may be active;
* model weights change between batches;
* the epoch loss is accumulated throughout optimization.

Validation is measured in evaluation mode using the model weights available after the epoch.

`trueloss` evaluates the model on the training data after every epoch using:

```python
model.evaluate(...)
```

The results are added to the Keras history with the `base_` prefix:

```text
loss
=
loss reported during training

base_loss
=
loss evaluated on the training data after the epoch

val_loss
=
loss evaluated on the validation data
```

Because `base_loss` and `val_loss` are both measured under evaluation conditions, they are generally more directly comparable.

However, `base_loss` is not guaranteed to contain only the data loss. Keras evaluation may still include regularizers or losses added through `add_loss()`, depending on the model.

`trueloss` does not replace the normal Keras training loss. It provides an additional view of training-data performance.

## Features

* Supports classification and regression models.
* Adds `base_loss` and `base_<metric_name>` values to the Keras history.
* Supports multiple compiled metrics.
* Automatically creates separate plots for loss and each detected metric.
* Supports arrays, TensorFlow datasets, and other Keras-compatible inputs.
* Returns the same Keras `History` object produced by `model.fit()`.
* Supports TensorFlow Keras and Keras 3-based TensorFlow installations.

## Installation

```bash
pip install trueloss
```

## Basic Usage

```python
import tensorflow as tf
from trueloss import trueloss
```

Create and compile the model normally:

```python
model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(784,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)
```

Create a `trueloss` instance and use its `fit()` method:

```python
true_loss_instance = trueloss(
    model=model,
    plot=True,
)

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)
```

Instead of:

```python
history = model.fit(...)
```

use:

```python
history = true_loss_instance.fit(...)
```

The returned object is the normal Keras `History` object.

## Using Multiple Metrics

You can compile the model with multiple compatible metrics:

```python
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=[
        "accuracy",
        tf.keras.metrics.SparseTopKCategoricalAccuracy(
            k=3,
            name="top_3_accuracy",
        ),
    ],
)
```

`trueloss` detects the available metric names automatically. No metric name is hard-coded into the plotting function.

## Recommended Parameter Style

Positional arguments are supported:

```python
history = true_loss_instance.fit(
    x_train,
    y_train,
    epochs=10,
)
```

However, keyword arguments are recommended, especially when using several `fit()` parameters:

```python
history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)
```

If parameters do not behave as expected through the wrapper, use explicit keyword arguments instead of passing many optional parameters positionally.

### Callbacks

When callbacks are required, pass them as a list:

```python
history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    epochs=10,
    callbacks=[
        early_stopping,
        model_checkpoint,
    ],
)
```

When callbacks are not required, omit the parameter.

Do not pass:

```python
callbacks=None
```

with the current version of `trueloss`.

## Training History

For a model compiled with accuracy, the history may contain:

```python
history.history["loss"]
history.history["accuracy"]

history.history["val_loss"]
history.history["val_accuracy"]

history.history["base_loss"]
history.history["base_accuracy"]
```

For multiple metrics, corresponding `base_` values are added:

```python
history.history["base_loss"]
history.history["base_accuracy"]
history.history["base_top_3_accuracy"]
```

For regression models, the history may contain:

```python
history.history["base_mean_absolute_error"]
history.history["base_mean_squared_error"]
```

View all available values with:

```python
print(history.history.keys())
```

## Automatic Plotting

Set `plot=True` to display the plots automatically:

```python
true_loss_instance = trueloss(
    model=model,
    plot=True,
)
```

The plotting function:

* creates one plot for loss;
* detects every additional metric;
* creates a separate subplot for each metric;
* plots training, validation, and `base_` values when available.

For example, a model using accuracy and top-3 accuracy can produce:

```text
Loss
Accuracy
Top 3 Accuracy
```

## Manual Plotting

For greater control, disable automatic plotting:

```python
true_loss_instance = trueloss(
    model=model,
    plot=False,
)
```

Train the model and inspect the history:

```python
history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
)

print(history.history.keys())
```

You can then create a custom plot:

```python
import matplotlib.pyplot as plt

epochs = range(1, len(history.history["loss"]) + 1)

plt.plot(
    epochs,
    history.history["loss"],
    label="Training Loss",
)

plt.plot(
    epochs,
    history.history["base_loss"],
    label="Training-Data Evaluation Loss",
)

if "val_loss" in history.history:
    plt.plot(
        epochs,
        history.history["val_loss"],
        label="Validation Loss",
    )

plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.legend()
plt.show()
```

This gives full control over the selected values, labels, colours, and layout.

## Important Notes

### Prefer Explicit Validation Data

Using explicit validation data is recommended:

```python
history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
)
```

Be careful when using:

```python
validation_split=0.2
```

Keras creates this split internally, while the current `trueloss` callback receives the original input arrays. The `base_` values may therefore be evaluated on the complete original data rather than only the portion used for training.

### Dataset Inputs

A TensorFlow dataset can be passed directly:

```python
train_dataset = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train)
).batch(32)

validation_dataset = tf.data.Dataset.from_tensor_slices(
    (x_val, y_val)
).batch(32)

history = true_loss_instance.fit(
    x=train_dataset,
    validation_data=validation_dataset,
    epochs=10,
)
```

The training dataset is evaluated again after every epoch, so it should be finite and reusable.

## Regression Usage

Regression models use the same workflow:

```python
regression_model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(20,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(1),
])

regression_model.compile(
    optimizer="adam",
    loss="mean_squared_error",
    metrics=[
        "mean_absolute_error",
        "mean_squared_error",
    ],
)

true_loss_instance = trueloss(
    model=regression_model,
    plot=True,
)

history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=20,
    batch_size=32,
)
```

The history may include:

```python
history.history["loss"]
history.history["val_loss"]
history.history["base_loss"]

history.history["mean_absolute_error"]
history.history["val_mean_absolute_error"]
history.history["base_mean_absolute_error"]

history.history["mean_squared_error"]
history.history["val_mean_squared_error"]
history.history["base_mean_squared_error"]
```

The plotting function automatically creates separate plots for loss, mean absolute error, and mean squared error.

This can also be used with regression outputs in more complex models, including bounding-box coordinate regression, provided the model works normally with `model.evaluate()`.

## Plotting an Existing History

An existing compatible Keras history can be plotted directly:

```python
true_loss_instance = trueloss()

true_loss_instance.plot_fn(history)
```

The function reads the available values from:

```python
history.history
```

## Complete Classification Example

```python
import tensorflow as tf
from trueloss import trueloss


# Load and preprocess the data
(x_train, y_train), (x_val, y_val) = (
    tf.keras.datasets.mnist.load_data()
)

x_train = x_train.astype("float32") / 255.0
x_val = x_val.astype("float32") / 255.0


# Create the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(28, 28)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation="softmax"),
])


# Compile the model
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=[
        "accuracy",
        tf.keras.metrics.SparseTopKCategoricalAccuracy(
            k=3,
            name="top_3_accuracy",
        ),
    ],
)


# Initialize trueloss
true_loss_instance = trueloss(
    model=model,
    plot=True,
)


# Train the model
history = true_loss_instance.fit(
    x=x_train,
    y=y_train,
    validation_data=(x_val, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
)


# View available values
print(history.history.keys())


# Access the additional values
print(history.history["base_loss"])
print(history.history["base_accuracy"])
print(history.history["base_top_3_accuracy"])
```

View the [example notebook](https://github.com/trueloss/trueloss/blob/main/tests/example.ipynb).

## Project Links

* [PyPI](https://pypi.org/project/trueloss/)
* [Source code](https://github.com/trueloss/trueloss)
* [Issue tracker](https://github.com/trueloss/trueloss/issues)
* [Example notebook](https://github.com/trueloss/trueloss/blob/main/tests/example.ipynb)

## Contributing

Contributions, compatibility reports, bug reports, and pull requests are welcome through the GitHub repository.

## License

This project is licensed under the MIT License. See the [LICENSE](https://github.com/trueloss/trueloss/blob/main/LICENSE) file for details.

## Contact

For questions or support, use the [GitHub issue tracker](https://github.com/trueloss/trueloss/issues) or contact Siddique Abusaleh at `trueloss.py@gmail.com`.

## Citation

```bibtex
@misc{trueloss,
  author       = {Siddique Abusaleh},
  title        = {trueloss: Comparable Training Loss and Metrics for Keras Models},
  year         = {2026},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/trueloss/trueloss}}
}
```
