============================= test session starts =============================
platform win32 -- Python 3.11.0, pytest-9.0.3, pluggy-1.6.0
rootdir: C:\Users\GuillaumePELLUET\Documents\Codes\weightslab
configfile: pyproject.toml
plugins: anyio-4.12.1, langsmith-0.7.14
collected 15 items

weightslab\tests\components\test_checkpoint_workflow.py ..F.F.FFFFFF...  [100%]

================================== FAILURES ===================================
_____________ CheckpointSystemTests.test_02_train_B_model_change ______________
weightslab\tests\components\test_checkpoint_workflow.py:552: in test_02_train_B_model_change
    model.operate(-2, {}, 3)  # Freeze fc1 layer
    ^^^^^^^^^^^^^^^^^^^^^^^^
weightslab\models\model_with_ops.py:208: in operate
    self._operate(
weightslab\models\model_with_ops.py:277: in _operate
    raise ValueError(
E   ValueError: [NetworkWithOps.operate] No module with id -2
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST B: Modify Model Architecture
================================================================================

Modifying model architecture...
04/06/2026-23:13:46.451 DEBUG:weightslab.models.model_with_ops:_operate: Operate currently on neurons: {} of layer id: -2 with op_type: 3
04/06/2026-23:13:46.452 DEBUG:weightslab.models.model_with_ops:_operate: [DEBUG OPERATE] Called with layer_id=-2
04/06/2026-23:13:46.452 DEBUG:weightslab.models.model_with_ops:_operate: [DEBUG OPERATE] After _reverse_indexing, layer_id=-2
------------------------------ Captured log call ------------------------------
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:260 Operate currently on neurons: {} of layer id: -2 with op_type: 3
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:273 [DEBUG OPERATE] Called with layer_id=-2
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:275 [DEBUG OPERATE] After _reverse_indexing, layer_id=-2
-------------------------- Captured stdout teardown ---------------------------
04/06/2026-23:13:46.454 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:46.646 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:46.649 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:46.649] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:46.650 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Completed non-blocking flush check. Pending count after flush: 0.
______________ CheckpointSystemTests.test_04_train_D_data_change ______________
weightslab\tests\components\test_checkpoint_workflow.py:739: in test_04_train_D_data_change
    self.state['model_c1_neurons'] = model.layers[0].out_neurons
                                     ^^^^^^^^^^^^^^^
E   IndexError: list index out of range
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST D: Change Data State (Tags and Discard)
================================================================================

Modifying data...
04/06/2026-23:13:51.066 DEBUG:weightslab.data.dataframe_manager:upsert_df: [LedgeredDataFrameManager] Global DataFrame updated: 10 rows, 12 columns. Index: ['sample_id', 'annotation_id']
  Added 'ugly' tag to 20 samples
  Discarded 20 samples
04/06/2026-23:13:51.072 INFO:weightslab.components.experiment_hash:has_changed: Experiment configuration changed: {'data'}
04/06/2026-23:13:51.073 INFO:weightslab.components.experiment_hash:generate_hash: Generated experiment hash: 4c107d095ee5e995cd6a6ae5- (HP: 4c107d09, Model: 5ee5e995, Data: cd6a6ae5)
04/06/2026-23:13:51.073 DEBUG:weightslab.components.experiment_hash:generate_hash:   HP hash: 4c107d09
04/06/2026-23:13:51.073 DEBUG:weightslab.components.experiment_hash:generate_hash:   Model hash: 5ee5e995
04/06/2026-23:13:51.074 DEBUG:weightslab.components.experiment_hash:generate_hash:   Data hash: cd6a6ae5
04/06/2026-23:13:51.074 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: New experiment hash: 4c107d09-5ee5e995-cd6a6ae5 (previous: 4c107d09-5ee5e995-bb73166f)
04/06/2026-23:13:51.074 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: Changed components: {'data'}
04/06/2026-23:13:51.074 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: Changes pending (not dumped yet): {'data'}

[OK] New experiment hash D: 4c107d095ee5e995cd6a6ae5
[OK] Changed components: {'data'}

Resuming training for 11 epochs...
04/06/2026-23:13:51.075 INFO:weightslab.components.global_monitoring:resume: 
Attempting to resume training...
04/06/2026-23:13:51.077 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:51.078 INFO:weightslab.components.checkpoint_manager:save_pending_changes: Dumping pending changes: {'data'}
04/06/2026-23:13:51.080 DEBUG:weightslab.components.checkpoint_manager:_create_exp_hash_directories: Created checkpoint directories for 4c107d095ee5e995cd6a6ae5: hp_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\HP\4c107d09, model_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\models\5ee5e995, data_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\cd6a6ae5
04/06/2026-23:13:51.131 INFO:weightslab.components.checkpoint_manager:_save_changes: Dumping data snapshot...
04/06/2026-23:13:51.138 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Flushing 2 buffered records to DataFrame (non-blocking).
04/06/2026-23:13:51.221 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=2: name 'get_mask' is not defined
04/06/2026-23:13:51.234 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=2: name 'get_mask' is not defined
04/06/2026-23:13:51.243 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=3: name 'get_mask' is not defined
04/06/2026-23:13:51.259 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=3: name 'get_mask' is not defined
04/06/2026-23:13:51.272 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Applied 2 buffered records to DataFrame (non-blocking).
04/06/2026-23:13:51.284 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Checking if flush to H5 is needed (non-blocking). Pending count: 6.
04/06/2026-23:13:51.291 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:51.291] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:51.317 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:51.323 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:51.323] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:51.324 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:51.336 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:51.511 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 6 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:51.515 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:51.515] [LedgeredDataFrameManager] Flushed 6 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:51.516 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Completed non-blocking flush check. Pending count after flush: 0.
04/06/2026-23:13:51.520 DEBUG:weightslab.components.checkpoint_manager:save_data_snapshot: Captured dataloader iteration states: {'main': {'samples_yielded': 22, 'batch_size': 2}}
04/06/2026-23:13:51.535 INFO:weightslab.components.checkpoint_manager:save_data_snapshot: Saved data snapshot: cd6a6ae5_data_snapshot.json (10 rows) with RNG state
04/06/2026-23:13:51.537 INFO:weightslab.components.checkpoint_manager:_save_changes: Saving model weights checkpoint with component changes...
04/06/2026-23:13:51.538 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 22, 'batch_size': 2}}
04/06/2026-23:13:51.554 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 4c107d095ee5e995cd6a6ae5_step_000022.pt
04/06/2026-23:13:51.581 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 4c107d095ee5e995cd6a6ae5_step_000022.pt (step 22)
04/06/2026-23:13:51.620 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:13:51.659 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:13:51.660 INFO:weightslab.components.global_monitoring:resume: Hashes by module: ['4c107d09', '5ee5e995', 'cd6a6ae5']
04/06/2026-23:13:51.661 INFO:weightslab.components.global_monitoring:resume: Resuming training now...
04/06/2026-23:13:51.661 INFO:weightslab.components.global_monitoring:resume: Hashes by module on resume: ['4c107d09', '5ee5e995', 'cd6a6ae5']
04/06/2026-23:13:51.661 INFO:weightslab.components.global_monitoring:resume: 
Training resumed as modules hashes have been computed: ['4c107d09', '5ee5e995', 'cd6a6ae5'].
04/06/2026-23:13:51.663 DEBUG:weightslab.backend.dataloader_interface:__next__: [DataLoaderInterface] iter invalidated by DOWN_ONLY change; resetting workers
04/06/2026-23:13:51.663 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:13:51.926 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Checking if flush to H5 is needed (non-blocking). Pending count: 0.
04/06/2026-23:13:51.927 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Completed non-blocking flush check. Pending count after flush: 0.
04/06/2026-23:13:51.929 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=3)
04/06/2026-23:13:51.935 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:51.935 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:51.949 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:51.949 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:51.949 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:52.028 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:52.028 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:52.028] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:52.040 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:52.040] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:52.040 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:52.040 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:52.045 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:52.045] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:52.050 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:52.050 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:52.053 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:52.066 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 25, 'batch_size': 2}}
04/06/2026-23:13:52.079 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 4c107d095ee5e995cd6a6ae5_step_000025.pt
04/06/2026-23:13:52.082 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:52.129 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 4c107d095ee5e995cd6a6ae5_step_000025.pt (step 25)
04/06/2026-23:13:52.280 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:13:52.317 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:52.321 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:52.322 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:52.328 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:52.329 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:52.329] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:52.329 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:52.429 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:52.430 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:52.430 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:52.430] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:52.445 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:52.445] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:52.445 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:52.445 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:52.448 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:52.448] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:52.475 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:52.552 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:52.566 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:52.596 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:52.697 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:13:52.961 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:52.961 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=3)
04/06/2026-23:13:52.965 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:52.965] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:52.965 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:52.969 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:52.969 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:52.969 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:53.066 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:53.066 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:53.066] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:53.074 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:53.082 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:53.082] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:53.083 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:53.083 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:53.083 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:53.086 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:53.086] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:53.097 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:53.115 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:53.115 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:53.115 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:53.115 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:53.217 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:53.324 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:53.329 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:53.331 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:53.330] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:53.331 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:53.431 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:53.431 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:53.431 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:53.431] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:53.450 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:53.450] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:53.450 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:53.450 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:53.452 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:53.452] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:53.478 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:53.543 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:53.569 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:53.601 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:53.713 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:53.717 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:53.717] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:53.717 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:53.723 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:53.725 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:53.725 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:53.818 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:53.818 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:53.818] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:53.832 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:53.833] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:53.832 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:53.832 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:53.832 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:53.832 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:53.835 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:53.835] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:53.844 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:53.852 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:13:54.182 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=3)
04/06/2026-23:13:54.185 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:54.222 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:54.230 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:54.234 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:54.347 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:54.419 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:54.427 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:54.426] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:54.427 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:54.449 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:54.528 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:54.529 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:54.529] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:54.545 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:54.545] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:54.545 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:54.545 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:54.549 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:54.549] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:54.551 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:54.556 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:54.558 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:54.579 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 30, 'batch_size': 2}}
04/06/2026-23:13:54.587 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:54.590 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 4c107d095ee5e995cd6a6ae5_step_000030.pt
04/06/2026-23:13:54.633 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 4c107d095ee5e995cd6a6ae5_step_000030.pt (step 30)
04/06/2026-23:13:54.801 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:13:54.867 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:54.877 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:54.878 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:54.878 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:54.878] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:54.879 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:54.879 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:54.980 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:54.981 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:54.981 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:54.981] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:54.981 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:55.003 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:55.002] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:55.003 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:55.003 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:55.010 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:55.010] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:55.019 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:55.036 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:55.038 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:55.038 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:55.041 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:55.140 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:55.246 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:55.264 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:55.267 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:55.267] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:55.267 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:55.347 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:55.369 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:55.369 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:55.369] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:55.386 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:55.386] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:55.386 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:55.386 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:55.392 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:55.391] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:55.413 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:55.450 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:55.459 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:55.569 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:55.656 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:13:55.938 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=3)
04/06/2026-23:13:55.975 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:55.981 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:55.980] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:55.981 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:55.987 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:55.988 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:55.988 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:56.083 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:56.083 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:56.083] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:56.102 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:56.102] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:56.102 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:56.103 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:56.103 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:56.103 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:56.108 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:56.108] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:56.118 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:13:56.137 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:13:56.138 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:56.138 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:13:56.139 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:56.240 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:56.349 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:56.359 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:13:56.365 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:56.365] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:56.365 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:13:56.451 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:13:56.466 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:13:56.466 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:56.466] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:13:56.488 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:13:56.488] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:13:56.489 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:13:56.489 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:13:56.496 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:56.496] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:13:56.523 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:13:56.553 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:13:56.563 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:13:56.660 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
Trained on ['2', '3', '5', '6', '7', '8', '2', '3', '5', '6', '7', '8', '2', '3', '5', '6', '7', '8', '2', '3', '5', '6'].
04/06/2026-23:13:56.769 INFO:weightslab.components.global_monitoring:pause: 
Training paused.

Training completed.
04/06/2026-23:13:56.789 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
[OK] Found 9 weight checkpoint files
---------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\n\rTraining:   0%|          | 0/11 [00:00<?, ?it/s]\rTraining:   9%|\u2589         | 1/11 [00:00<00:02,  3.54it/s]\rTraining:  18%|\u2588\u258a        | 2/11 [00:00<00:01,  5.46it/s]\rTraining:  27%|\u2588\u2588\u258b       | 3/11 [00:01<00:03,  2.56it/s]\rTraining:  36%|\u2588\u2588\u2588\u258b      | 4/11 [00:01<00:02,  2.52it/s]\rTraining:  45%|\u2588\u2588\u2588\u2588\u258c     | 5/11 [00:02<00:02,  2.13it/s]\rTraining:  55%|\u2588\u2588\u2588\u2588\u2588\u258d    | 6/11 [00:02<00:01,  2.77it/s]\rTraining:  64%|\u2588\u2588\u2588\u2588\u2588\u2588\u258e   | 7/11 [00:02<00:01,  2.10it/s]\rTraining:  73%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e  | 8/11 [00:03<00:01,  2.12it/s]\rTraining:  82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 9/11 [00:03<00:01,  1.92it/s]\rTraining:  91%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 10/11 [00:04<00:00,  1.98it/s]\rTraining: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:05<00:00,  1.84it/s]\rTraining: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:05<00:00,  2.16it/s]
------------------------------ Captured log call ------------------------------
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:830 [LedgeredDataFrameManager] Global DataFrame updated: 10 rows, 12 columns. Index: ['sample_id', 'annotation_id']
INFO     weightslab.components.experiment_hash:experiment_hash.py:165 Experiment configuration changed: {'data'}
INFO     weightslab.components.experiment_hash:experiment_hash.py:94 Generated experiment hash: 4c107d095ee5e995cd6a6ae5- (HP: 4c107d09, Model: 5ee5e995, Data: cd6a6ae5)
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:95   HP hash: 4c107d09
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:96   Model hash: 5ee5e995
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:97   Data hash: cd6a6ae5
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:639 New experiment hash: 4c107d09-5ee5e995-cd6a6ae5 (previous: 4c107d09-5ee5e995-bb73166f)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:642 Changed components: {'data'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:690 Changes pending (not dumped yet): {'data'}
INFO     weightslab.components.global_monitoring:global_monitoring.py:142 
Attempting to resume training...
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1340 Dumping pending changes: {'data'}
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:292 Created checkpoint directories for 4c107d095ee5e995cd6a6ae5: hp_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\HP\4c107d09, model_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\models\5ee5e995, data_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\cd6a6ae5
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:856 Dumping data snapshot...
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2563 Flushing 2 buffered records to DataFrame (non-blocking).
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=2: name 'get_mask' is not defined
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=2: name 'get_mask' is not defined
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=3: name 'get_mask' is not defined
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=3: name 'get_mask' is not defined
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2565 Applied 2 buffered records to DataFrame (non-blocking).
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2568 Checking if flush to H5 is needed (non-blocking). Pending count: 6.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:51.291] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:51.323] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 6 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:51.515] [LedgeredDataFrameManager] Flushed 6 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2570 Completed non-blocking flush check. Pending count after flush: 0.
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1200 Captured dataloader iteration states: {'main': {'samples_yielded': 22, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1212 Saved data snapshot: cd6a6ae5_data_snapshot.json (10 rows) with RNG state
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:873 Saving model weights checkpoint with component changes...
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 22, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 4c107d095ee5e995cd6a6ae5_step_000022.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 4c107d095ee5e995cd6a6ae5_step_000022.pt (step 22)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.components.global_monitoring:global_monitoring.py:153 Hashes by module: ['4c107d09', '5ee5e995', 'cd6a6ae5']
INFO     weightslab.components.global_monitoring:global_monitoring.py:157 Resuming training now...
INFO     weightslab.components.global_monitoring:global_monitoring.py:159 Hashes by module on resume: ['4c107d09', '5ee5e995', 'cd6a6ae5']
INFO     weightslab.components.global_monitoring:global_monitoring.py:160 
Training resumed as modules hashes have been computed: ['4c107d09', '5ee5e995', 'cd6a6ae5'].
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:936 [DataLoaderInterface] iter invalidated by DOWN_ONLY change; resetting workers
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2568 Checking if flush to H5 is needed (non-blocking). Pending count: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2570 Completed non-blocking flush check. Pending count after flush: 0.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=3)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:52.028] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:52.040] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:52.045] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 25, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 4c107d095ee5e995cd6a6ae5_step_000025.pt
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 4c107d095ee5e995cd6a6ae5_step_000025.pt (step 25)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:52.329] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:52.430] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:52.445] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:52.448] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=3)
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:52.965] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:53.066] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:53.082] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:53.086] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:53.330] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:53.431] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:53.450] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:53.452] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:53.717] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:53.818] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:53.833] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:53.835] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=3)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:54.426] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:54.529] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:54.545] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:54.549] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 30, 'batch_size': 2}}
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 4c107d095ee5e995cd6a6ae5_step_000030.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 4c107d095ee5e995cd6a6ae5_step_000030.pt (step 30)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:54.878] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:54.981] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:55.002] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:55.010] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:55.267] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:55.369] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:55.386] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:55.391] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=3)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:55.980] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:56.083] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:56.102] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:56.108] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:13:56.365] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:13:56.466] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:13:56.488] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:13:56.496] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.components.global_monitoring:global_monitoring.py:134 
Training paused.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
-------------------------- Captured stdout teardown ---------------------------
04/06/2026-23:13:56.795 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:13:56.795] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:13:56.795 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
__________ CheckpointSystemTests.test_06_reload_before_model_change ___________
weightslab\tests\components\test_checkpoint_workflow.py:919: in test_06_reload_before_model_change
    model.operate(-2, {}, 3)
weightslab\models\model_with_ops.py:208: in operate
    self._operate(
weightslab\models\model_with_ops.py:277: in _operate
    raise ValueError(
E   ValueError: [NetworkWithOps.operate] No module with id -2
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST 06: Reload Before Model Change - Fix Conv Size with RNG State
================================================================================

Reloading state A (before model change) for verification: 01cbbef25ee5e995...
04/06/2026-23:14:09.017 INFO:weightslab.components.checkpoint_manager:load_state: 
============================================================
04/06/2026-23:14:09.017 INFO:weightslab.components.checkpoint_manager:load_state: Loading and applying state: 01cbbef25ee5e995...
04/06/2026-23:14:09.017 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================
04/06/2026-23:14:09.022 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loading checkpoint 01cbbef25ee5e995...
04/06/2026-23:14:09.022 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Target: HP=01cbbef2 MODEL=5ee5e995 DATA=bb73166f
04/06/2026-23:14:09.022 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Current: HP=fdc268da MODEL=808a8c02 DATA=6176a561
04/06/2026-23:14:09.390 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded model architecture from hash 5ee5e995
04/06/2026-23:14:09.391 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   Using weight checkpoint from manifest: 01cbbef25ee5e995bb73166f_step_000010.pt
04/06/2026-23:14:09.400 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded weights from step 10 with RNG state
04/06/2026-23:14:09.400 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 10, 'batch_size': 2}}
04/06/2026-23:14:09.402 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded config (hash changed)
04/06/2026-23:14:09.405 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded data snapshot (10 rows) with RNG state
04/06/2026-23:14:09.405 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loaded components: {'config', 'data', 'weights', 'model'}
04/06/2026-23:14:09.405 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied model architecture and weights (step 10)
04/06/2026-23:14:09.405 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied weights to existing model (step 10)
04/06/2026-23:14:09.405 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied hyperparameters config
04/06/2026-23:14:09.405 DEBUG:weightslab.data.dataframe_manager:upsert_df: [LedgeredDataFrameManager] Global DataFrame updated: 10 rows, 12 columns. Index: ['sample_id', 'annotation_id']
04/06/2026-23:14:09.405 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied data snapshot (10 rows)
04/06/2026-23:14:09.405 DEBUG:weightslab.utils.tools:restore_rng_state: Restored Python random state
04/06/2026-23:14:09.405 DEBUG:weightslab.utils.tools:restore_rng_state: Restored NumPy random state
04/06/2026-23:14:09.405 DEBUG:weightslab.utils.tools:restore_rng_state: Restored PyTorch RNG state
04/06/2026-23:14:09.405 DEBUG:weightslab.utils.tools:restore_rng_state: Restored CUDA RNG state
04/06/2026-23:14:09.405 DEBUG:weightslab.utils.tools:restore_rng_state: Successfully restored RNG states
04/06/2026-23:14:09.405 DEBUG:weightslab.components.checkpoint_manager:load_state: Restored RNG state from checkpoint
04/06/2026-23:14:09.416 DEBUG:weightslab.components.experiment_hash:restore_hashes: Restored hashes hp=01cbbef2, model=5ee5e995, data=bb73166f, combined=01cbbef25ee5e995bb73166f
04/06/2026-23:14:09.416 INFO:weightslab.components.checkpoint_manager:load_state: 
[OK] Successfully loaded and applied state: 01cbbef25ee5e995
04/06/2026-23:14:09.416 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================

[OK] HP batch_size: 2
[OK] Data state verified from state A
[OK] RNG state restored for reproducible batching

Training with original model from state A (11 epochs)...
04/06/2026-23:14:09.416 INFO:weightslab.components.global_monitoring:resume: 
Attempting to resume training...
04/06/2026-23:14:09.419 INFO:weightslab.components.experiment_hash:has_changed: Experiment configuration changed: {'model'}
04/06/2026-23:14:09.419 INFO:weightslab.components.experiment_hash:generate_hash: Generated experiment hash: 01cbbef27d311ca8bb73166f- (HP: 01cbbef2, Model: 7d311ca8, Data: bb73166f)
04/06/2026-23:14:09.419 DEBUG:weightslab.components.experiment_hash:generate_hash:   HP hash: 01cbbef2
04/06/2026-23:14:09.420 DEBUG:weightslab.components.experiment_hash:generate_hash:   Model hash: 7d311ca8
04/06/2026-23:14:09.420 DEBUG:weightslab.components.experiment_hash:generate_hash:   Data hash: bb73166f
04/06/2026-23:14:09.420 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: New experiment hash: 01cbbef2-7d311ca8-bb73166f (previous: 01cbbef2-5ee5e995-bb73166f)
04/06/2026-23:14:09.421 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: Changed components: {'model'}
04/06/2026-23:14:09.421 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: Changes pending (not dumped yet): {'model'}
04/06/2026-23:14:09.422 INFO:weightslab.components.checkpoint_manager:save_pending_changes: Dumping pending changes: {'model'}
04/06/2026-23:14:09.422 DEBUG:weightslab.components.checkpoint_manager:_create_exp_hash_directories: Created checkpoint directories for 01cbbef27d311ca8bb73166f: hp_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\HP\01cbbef2, model_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\models\7d311ca8, data_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\bb73166f
04/06/2026-23:14:09.428 INFO:weightslab.components.checkpoint_manager:_save_changes: Dumping model architecture...
04/06/2026-23:14:09.438 INFO:weightslab.components.checkpoint_manager:save_model_architecture: Saved model architecture: 7d311ca8_architecture.pkl
04/06/2026-23:14:09.438 INFO:weightslab.components.checkpoint_manager:_save_changes: Saving model weights checkpoint with component changes...
04/06/2026-23:14:09.438 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 55, 'batch_size': 2}}
04/06/2026-23:14:09.438 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 01cbbef27d311ca8bb73166f_step_000010.pt
04/06/2026-23:14:09.455 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 01cbbef27d311ca8bb73166f_step_000010.pt (step 10)
04/06/2026-23:14:09.476 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:09.489 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:09.489 INFO:weightslab.components.global_monitoring:resume: Hashes by module: ['01cbbef2', '7d311ca8', 'bb73166f']
04/06/2026-23:14:09.489 INFO:weightslab.components.global_monitoring:resume: Resuming training now...
04/06/2026-23:14:09.489 INFO:weightslab.components.global_monitoring:resume: Hashes by module on resume: ['01cbbef2', '7d311ca8', 'bb73166f']
04/06/2026-23:14:09.489 INFO:weightslab.components.global_monitoring:resume: 
Training resumed as modules hashes have been computed: ['01cbbef2', '7d311ca8', 'bb73166f'].
04/06/2026-23:14:09.489 DEBUG:weightslab.backend.dataloader_interface:__next__: [DataLoaderInterface] iter invalidated by DOWN_ONLY change; resetting workers
04/06/2026-23:14:09.489 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:14:09.632 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=5)
04/06/2026-23:14:09.636 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:09.636 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:09.636 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:09.732 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:09.732 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:09.732] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:09.739 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:09.739] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:09.739 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:09.739 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:09.739 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:09.739 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 10.
04/06/2026-23:14:09.739 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:09.739] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:09.755 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:09.756 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:09.756 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:09.756 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:09.756 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:09.866 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:09.897 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 10 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:09.897 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:09.897] [LedgeredDataFrameManager] Flushed 10 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:09.897 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:09.967 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:10.004 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:10.004 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.004] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.015 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.015] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.015 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:10.015 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:10.017 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.017] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:10.021 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:10.074 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:10.086 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:10.111 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:10.171 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:10.171 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.171] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:10.171 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:10.181 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:10.181 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:10.181 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:10.279 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:10.279 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.279] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.286 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.286] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.286 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:10.286 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:10.286 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:10.286 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:10.286 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.286] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:10.291 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:10.301 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:10.301 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:10.301 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:10.301 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:10.404 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:10.452 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:10.452 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.452] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:10.452 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:10.513 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:10.557 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:10.557 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.557] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.578 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.578] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.579 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:10.579 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:10.580 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.580] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:10.593 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:10.613 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:10.617 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:10.664 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:10.733 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:10.735 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.735] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:10.736 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:10.738 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 60, 'batch_size': 2}}
04/06/2026-23:14:10.743 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 01cbbef27d311ca8bb73166f_step_000015.pt
04/06/2026-23:14:10.759 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 01cbbef27d311ca8bb73166f_step_000015.pt (step 15)
04/06/2026-23:14:10.783 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:10.787 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:10.787 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:10.787 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:10.836 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:10.836 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.836] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.844 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:10.844] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:10.844 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:10.844 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:10.845 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:10.845] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:10.858 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:10.888 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:10.891 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:10.931 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:11.013 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:14:11.162 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=5)
04/06/2026-23:14:11.167 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:11.170 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.170] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:11.171 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:11.177 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:11.177 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:11.177 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.272 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:11.272 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:11.272] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:11.277 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:11.285 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:11.285] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:11.285 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:11.285 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:11.285 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:11.287 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.287] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:11.292 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:11.300 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:11.301 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:11.301 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.301 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:11.403 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.437 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:11.440 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.440] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:11.440 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:11.505 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.541 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:11.541 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:11.541] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:11.547 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:11.547] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:11.547 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:11.548 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:11.550 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.550] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:11.563 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:11.607 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:11.623 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:11.662 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:11.706 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:11.710 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.709] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:11.710 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:11.715 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:11.715 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:11.715 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.810 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:11.810 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:11.810] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:11.816 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:11.816] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:11.816 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:11.816 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:11.816 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:11.821 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.821] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:11.821 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:11.826 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:11.833 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:11.833 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:11.833 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:11.833 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.937 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:11.985 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:11.987 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:11.987] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:11.987 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:12.038 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:12.088 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:12.088 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:12.088] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:12.103 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:12.103] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:12.103 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:12.103 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:12.103 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.103] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:12.126 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:12.139 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:12.142 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:12.188 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:12.267 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:12.269 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.269] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:12.269 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:12.272 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 65, 'batch_size': 2}}
04/06/2026-23:14:12.278 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 01cbbef27d311ca8bb73166f_step_000020.pt
04/06/2026-23:14:12.289 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 01cbbef27d311ca8bb73166f_step_000020.pt (step 20)
04/06/2026-23:14:12.305 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:12.319 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:12.319 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:12.321 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:12.370 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:12.370 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:12.370] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:12.377 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:12.377] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:12.377 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:12.377 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:12.378 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.378] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:12.390 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:12.422 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:12.433 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:12.439 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:12.517 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:14:12.639 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:12.641 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=5)
04/06/2026-23:14:12.641 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.641] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:12.641 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:12.650 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:12.650 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:12.650 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:12.744 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:12.744 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:12.744] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:12.755 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:12.755] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:12.755 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:12.755 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:12.755 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:12.755 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:12.757 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.757] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:12.762 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
Trained on ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1'].
04/06/2026-23:14:12.766 INFO:weightslab.components.global_monitoring:pause: 
Training paused.

Reloading state A again (to reset RNG for fair comparison) and modifying model architecture...
04/06/2026-23:14:12.766 INFO:weightslab.components.checkpoint_manager:load_state: 
============================================================
04/06/2026-23:14:12.766 INFO:weightslab.components.checkpoint_manager:load_state: Loading and applying state: 01cbbef25ee5e995...
04/06/2026-23:14:12.766 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================
04/06/2026-23:14:12.770 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:12.789 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loading checkpoint 01cbbef25ee5e995...
04/06/2026-23:14:12.792 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Target: HP=01cbbef2 MODEL=5ee5e995 DATA=bb73166f
04/06/2026-23:14:12.794 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Current: HP=01cbbef2 MODEL=7d311ca8 DATA=bb73166f
04/06/2026-23:14:12.862 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded model architecture from hash 5ee5e995
04/06/2026-23:14:12.880 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   Using weight checkpoint from manifest: 01cbbef25ee5e995bb73166f_step_000010.pt
04/06/2026-23:14:12.922 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:12.927 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.927] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:12.927 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:12.927 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Flushing 2 buffered records to DataFrame (non-blocking).
04/06/2026-23:14:12.938 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=0: name 'get_mask' is not defined
04/06/2026-23:14:12.938 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=0: name 'get_mask' is not defined
04/06/2026-23:14:12.938 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=1: name 'get_mask' is not defined
04/06/2026-23:14:12.938 DEBUG:weightslab.data.dataframe_manager:_normalize_arrays_for_storage: [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=1: name 'get_mask' is not defined
04/06/2026-23:14:12.938 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded weights from step 10 with RNG state
04/06/2026-23:14:12.954 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Applied 2 buffered records to DataFrame (non-blocking).
04/06/2026-23:14:12.954 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 10, 'batch_size': 2}}
04/06/2026-23:14:12.954 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Checking if flush to H5 is needed (non-blocking). Pending count: 2.
04/06/2026-23:14:12.955 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [-] Config unchanged, using current config
04/06/2026-23:14:12.957 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:12.957] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:12.967 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded RNG state for reproducibility (data unchanged)
04/06/2026-23:14:12.967 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loaded components: {'weights', 'model'}
04/06/2026-23:14:12.968 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied model architecture and weights (step 10)
04/06/2026-23:14:12.971 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied weights to existing model (step 10)
04/06/2026-23:14:12.972 DEBUG:weightslab.utils.tools:restore_rng_state: Restored Python random state
04/06/2026-23:14:12.973 DEBUG:weightslab.utils.tools:restore_rng_state: Restored NumPy random state
04/06/2026-23:14:12.973 DEBUG:weightslab.utils.tools:restore_rng_state: Restored PyTorch RNG state
04/06/2026-23:14:12.974 DEBUG:weightslab.utils.tools:restore_rng_state: Restored CUDA RNG state
04/06/2026-23:14:12.974 DEBUG:weightslab.utils.tools:restore_rng_state: Successfully restored RNG states
04/06/2026-23:14:12.974 DEBUG:weightslab.components.checkpoint_manager:load_state: Restored RNG state from checkpoint
04/06/2026-23:14:12.978 DEBUG:weightslab.components.experiment_hash:restore_hashes: Restored hashes hp=01cbbef2, model=5ee5e995, data=bb73166f, combined=01cbbef25ee5e995bb73166f
04/06/2026-23:14:12.979 INFO:weightslab.components.checkpoint_manager:load_state: 
[OK] Successfully loaded and applied state: 01cbbef25ee5e995
04/06/2026-23:14:12.979 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================


Fixing model architecture...
04/06/2026-23:14:12.979 DEBUG:weightslab.models.model_with_ops:_operate: Operate currently on neurons: {} of layer id: -2 with op_type: 3
04/06/2026-23:14:12.980 DEBUG:weightslab.models.model_with_ops:_operate: [DEBUG OPERATE] Called with layer_id=-2
04/06/2026-23:14:12.980 DEBUG:weightslab.models.model_with_ops:_operate: [DEBUG OPERATE] After _reverse_indexing, layer_id=-2
---------------------------- Captured stderr call -----------------------------
\rTraining:   0%|          | 0/11 [00:00<?, ?it/s]\rTraining:   9%|\u2589         | 1/11 [00:00<00:02,  3.74it/s]\rTraining:  18%|\u2588\u258a        | 2/11 [00:00<00:03,  2.82it/s]\rTraining:  27%|\u2588\u2588\u258b       | 3/11 [00:00<00:01,  4.04it/s]\rTraining:  36%|\u2588\u2588\u2588\u258b      | 4/11 [00:01<00:02,  3.15it/s]\rTraining:  45%|\u2588\u2588\u2588\u2588\u258c     | 5/11 [00:01<00:01,  3.23it/s]\rTraining:  55%|\u2588\u2588\u2588\u2588\u2588\u258d    | 6/11 [00:01<00:01,  3.32it/s]\rTraining:  64%|\u2588\u2588\u2588\u2588\u2588\u2588\u258e   | 7/11 [00:02<00:01,  2.97it/s]\rTraining:  73%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e  | 8/11 [00:02<00:00,  3.72it/s]\rTraining:  82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 9/11 [00:02<00:00,  3.20it/s]\rTraining:  91%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 10/11 [00:03<00:00,  3.31it/s]\rTraining: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:03<00:00,  3.50it/s]\rTraining: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:03<00:00,  3.36it/s]\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined\nTraceback (most recent call last):\n  File "C:\\Users\\GuillaumePELLUET\\Documents\\Codes\\weightslab\\weightslab\\data\\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage\n    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)\n               ^^^^^^^^\nNameError: name 'get_mask' is not defined
------------------------------ Captured log call ------------------------------
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1854 
============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1855 Loading and applying state: 01cbbef25ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1856 ============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1646 Loading checkpoint 01cbbef25ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1647   Target: HP=01cbbef2 MODEL=5ee5e995 DATA=bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1648   Current: HP=fdc268da MODEL=808a8c02 DATA=6176a561
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1681   [OK] Loaded model architecture from hash 5ee5e995
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1724   Using weight checkpoint from manifest: 01cbbef25ee5e995bb73166f_step_000010.pt
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1745   [OK] Loaded weights from step 10 with RNG state
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1761   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 10, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1779   [OK] Loaded config (hash changed)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1810   [OK] Loaded data snapshot (10 rows) with RNG state
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1824 Loaded components: {'config', 'data', 'weights', 'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1923 [OK] Applied model architecture and weights (step 10)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1942 [OK] Applied weights to existing model (step 10)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1998 [OK] Applied hyperparameters config
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:830 [LedgeredDataFrameManager] Global DataFrame updated: 10 rows, 12 columns. Index: ['sample_id', 'annotation_id']
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2020 [OK] Applied data snapshot (10 rows)
DEBUG    weightslab.utils.tools:tools.py:181 Restored Python random state
DEBUG    weightslab.utils.tools:tools.py:194 Restored NumPy random state
DEBUG    weightslab.utils.tools:tools.py:210 Restored PyTorch RNG state
DEBUG    weightslab.utils.tools:tools.py:224 Restored CUDA RNG state
DEBUG    weightslab.utils.tools:tools.py:228 Successfully restored RNG states
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:2030 Restored RNG state from checkpoint
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:344 Restored hashes hp=01cbbef2, model=5ee5e995, data=bb73166f, combined=01cbbef25ee5e995bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2122 
[OK] Successfully loaded and applied state: 01cbbef25ee5e995
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2126 ============================================================

INFO     weightslab.components.global_monitoring:global_monitoring.py:142 
Attempting to resume training...
INFO     weightslab.components.experiment_hash:experiment_hash.py:165 Experiment configuration changed: {'model'}
INFO     weightslab.components.experiment_hash:experiment_hash.py:94 Generated experiment hash: 01cbbef27d311ca8bb73166f- (HP: 01cbbef2, Model: 7d311ca8, Data: bb73166f)
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:95   HP hash: 01cbbef2
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:96   Model hash: 7d311ca8
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:97   Data hash: bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:639 New experiment hash: 01cbbef2-7d311ca8-bb73166f (previous: 01cbbef2-5ee5e995-bb73166f)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:642 Changed components: {'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:690 Changes pending (not dumped yet): {'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1340 Dumping pending changes: {'model'}
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:292 Created checkpoint directories for 01cbbef27d311ca8bb73166f: hp_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\HP\01cbbef2, model_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\models\7d311ca8, data_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:845 Dumping model architecture...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1051 Saved model architecture: 7d311ca8_architecture.pkl
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:873 Saving model weights checkpoint with component changes...
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 55, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 01cbbef27d311ca8bb73166f_step_000010.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 01cbbef27d311ca8bb73166f_step_000010.pt (step 10)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.components.global_monitoring:global_monitoring.py:153 Hashes by module: ['01cbbef2', '7d311ca8', 'bb73166f']
INFO     weightslab.components.global_monitoring:global_monitoring.py:157 Resuming training now...
INFO     weightslab.components.global_monitoring:global_monitoring.py:159 Hashes by module on resume: ['01cbbef2', '7d311ca8', 'bb73166f']
INFO     weightslab.components.global_monitoring:global_monitoring.py:160 
Training resumed as modules hashes have been computed: ['01cbbef2', '7d311ca8', 'bb73166f'].
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:936 [DataLoaderInterface] iter invalidated by DOWN_ONLY change; resetting workers
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=5)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:09.732] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:09.739] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 10.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:09.739] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 10 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:09.897] [LedgeredDataFrameManager] Flushed 10 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:10.004] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:10.015] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:10.017] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:10.171] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:10.279] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:10.286] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:10.286] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:10.452] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:10.557] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:10.578] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:10.580] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:10.735] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 60, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 01cbbef27d311ca8bb73166f_step_000015.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 01cbbef27d311ca8bb73166f_step_000015.pt (step 15)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:10.836] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:10.844] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:10.845] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=5)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:11.170] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:11.272] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:11.285] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:11.287] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:11.440] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:11.541] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:11.547] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:11.550] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:11.709] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:11.810] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:11.816] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:11.821] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:11.987] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:12.088] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:12.103] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:12.103] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:12.269] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 65, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 01cbbef27d311ca8bb73166f_step_000020.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 01cbbef27d311ca8bb73166f_step_000020.pt (step 20)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:12.370] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:12.377] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:12.378] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=5)
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:12.641] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:12.744] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:12.755] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:12.757] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.components.global_monitoring:global_monitoring.py:134 
Training paused.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1854 
============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1855 Loading and applying state: 01cbbef25ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1856 ============================================================
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1646 Loading checkpoint 01cbbef25ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1647   Target: HP=01cbbef2 MODEL=5ee5e995 DATA=bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1648   Current: HP=01cbbef2 MODEL=7d311ca8 DATA=bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1681   [OK] Loaded model architecture from hash 5ee5e995
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1724   Using weight checkpoint from manifest: 01cbbef25ee5e995bb73166f_step_000010.pt
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:12.927] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2563 Flushing 2 buffered records to DataFrame (non-blocking).
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=0: name 'get_mask' is not defined
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=0: name 'get_mask' is not defined
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=1: name 'get_mask' is not defined
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1665 [_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=1: name 'get_mask' is not defined
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1745   [OK] Loaded weights from step 10 with RNG state
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2565 Applied 2 buffered records to DataFrame (non-blocking).
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1761   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 10, 'batch_size': 2}}
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2568 Checking if flush to H5 is needed (non-blocking). Pending count: 2.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1785   [-] Config unchanged, using current config
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:12.957] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1816   [OK] Loaded RNG state for reproducibility (data unchanged)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1824 Loaded components: {'weights', 'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1923 [OK] Applied model architecture and weights (step 10)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1942 [OK] Applied weights to existing model (step 10)
DEBUG    weightslab.utils.tools:tools.py:181 Restored Python random state
DEBUG    weightslab.utils.tools:tools.py:194 Restored NumPy random state
DEBUG    weightslab.utils.tools:tools.py:210 Restored PyTorch RNG state
DEBUG    weightslab.utils.tools:tools.py:224 Restored CUDA RNG state
DEBUG    weightslab.utils.tools:tools.py:228 Successfully restored RNG states
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:2030 Restored RNG state from checkpoint
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:344 Restored hashes hp=01cbbef2, model=5ee5e995, data=bb73166f, combined=01cbbef25ee5e995bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2122 
[OK] Successfully loaded and applied state: 01cbbef25ee5e995
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2126 ============================================================

DEBUG    weightslab.models.model_with_ops:model_with_ops.py:260 Operate currently on neurons: {} of layer id: -2 with op_type: 3
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:273 [DEBUG OPERATE] Called with layer_id=-2
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:275 [DEBUG OPERATE] After _reverse_indexing, layer_id=-2
-------------------------- Captured stdout teardown ---------------------------
04/06/2026-23:14:13.004 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
____________ CheckpointSystemTests.test_07_change_data_from_test06 ____________
weightslab\tests\components\test_checkpoint_workflow.py:967: in test_07_change_data_from_test06
    print(f"Starting from state H: {hash_H[:16]}...")
                                    ^^^^^^^^^^^
E   TypeError: 'NoneType' object is not subscriptable
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST 07: Change Data from Test 06 - Discard More Data
================================================================================

__ CheckpointSystemTests.test_08_reload_before_data_change_verify_and_modify __
weightslab\tests\components\test_checkpoint_workflow.py:1052: in test_08_reload_before_data_change_verify_and_modify
    model.operate(-2, {}, 3)
weightslab\models\model_with_ops.py:208: in operate
    self._operate(
weightslab\models\model_with_ops.py:277: in _operate
    raise ValueError(
E   ValueError: [NetworkWithOps.operate] No module with id -2
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST 08: Reload Before Data Change - Verify and Modify Model
================================================================================

Part A: Reloading state C and verifying training reproducibility...
Reloading state C: 4c107d095ee5e995...
04/06/2026-23:14:13.101 INFO:weightslab.components.checkpoint_manager:load_state: 
============================================================
04/06/2026-23:14:13.104 INFO:weightslab.components.checkpoint_manager:load_state: Loading and applying state: 4c107d095ee5e995...
04/06/2026-23:14:13.107 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================
04/06/2026-23:14:13.135 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loading checkpoint 4c107d095ee5e995...
04/06/2026-23:14:13.135 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Target: HP=4c107d09 MODEL=5ee5e995 DATA=bb73166f
04/06/2026-23:14:13.135 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Current: HP=01cbbef2 MODEL=5ee5e995 DATA=bb73166f
04/06/2026-23:14:13.198 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 2 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:13.200 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:13.200] [LedgeredDataFrameManager] Flushed 2 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:13.200 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Completed non-blocking flush check. Pending count after flush: 0.
04/06/2026-23:14:13.206 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded model RNG state for reproducibility (model unchanged)
04/06/2026-23:14:13.206 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   Using weight checkpoint from manifest: 4c107d095ee5e995bb73166f_step_000020.pt
04/06/2026-23:14:13.211 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded weights from step 20 with RNG state
04/06/2026-23:14:13.211 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 20, 'batch_size': 2}}
04/06/2026-23:14:13.222 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded config (hash changed)
04/06/2026-23:14:13.223 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded RNG state for reproducibility (data unchanged)
04/06/2026-23:14:13.223 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loaded components: {'config', 'weights'}
04/06/2026-23:14:13.224 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied weights to existing model (step 20)
04/06/2026-23:14:13.226 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied hyperparameters config
04/06/2026-23:14:13.226 DEBUG:weightslab.utils.tools:restore_rng_state: Restored Python random state
04/06/2026-23:14:13.226 DEBUG:weightslab.utils.tools:restore_rng_state: Restored NumPy random state
04/06/2026-23:14:13.226 DEBUG:weightslab.utils.tools:restore_rng_state: Restored PyTorch RNG state
04/06/2026-23:14:13.227 DEBUG:weightslab.utils.tools:restore_rng_state: Restored CUDA RNG state
04/06/2026-23:14:13.227 DEBUG:weightslab.utils.tools:restore_rng_state: Successfully restored RNG states
04/06/2026-23:14:13.227 DEBUG:weightslab.components.checkpoint_manager:load_state: Restored RNG state from checkpoint
04/06/2026-23:14:13.230 DEBUG:weightslab.components.experiment_hash:restore_hashes: Restored hashes hp=4c107d09, model=5ee5e995, data=bb73166f, combined=4c107d095ee5e995bb73166f
04/06/2026-23:14:13.231 INFO:weightslab.components.checkpoint_manager:load_state: 
[OK] Successfully loaded and applied state: 4c107d095ee5e995
04/06/2026-23:14:13.231 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================


Training for 11 epochs to verify reproducibility...
04/06/2026-23:14:13.231 INFO:weightslab.components.global_monitoring:resume: 
Attempting to resume training...
04/06/2026-23:14:13.232 INFO:weightslab.components.experiment_hash:has_changed: Experiment configuration changed: {'model'}
04/06/2026-23:14:13.232 INFO:weightslab.components.experiment_hash:generate_hash: Generated experiment hash: 4c107d099f96ab56bb73166f- (HP: 4c107d09, Model: 9f96ab56, Data: bb73166f)
04/06/2026-23:14:13.232 DEBUG:weightslab.components.experiment_hash:generate_hash:   HP hash: 4c107d09
04/06/2026-23:14:13.232 DEBUG:weightslab.components.experiment_hash:generate_hash:   Model hash: 9f96ab56
04/06/2026-23:14:13.233 DEBUG:weightslab.components.experiment_hash:generate_hash:   Data hash: bb73166f
04/06/2026-23:14:13.233 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: New experiment hash: 4c107d09-9f96ab56-bb73166f (previous: 4c107d09-5ee5e995-bb73166f)
04/06/2026-23:14:13.233 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: Changed components: {'model'}
04/06/2026-23:14:13.233 INFO:weightslab.components.checkpoint_manager:update_experiment_hash: Changes pending (not dumped yet): {'model'}
04/06/2026-23:14:13.233 INFO:weightslab.components.checkpoint_manager:save_pending_changes: Dumping pending changes: {'model'}
04/06/2026-23:14:13.234 DEBUG:weightslab.components.checkpoint_manager:_create_exp_hash_directories: Created checkpoint directories for 4c107d099f96ab56bb73166f: hp_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\HP\4c107d09, model_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\models\9f96ab56, data_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\bb73166f
04/06/2026-23:14:13.240 INFO:weightslab.components.checkpoint_manager:_save_changes: Dumping model architecture...
04/06/2026-23:14:13.240 INFO:weightslab.components.checkpoint_manager:save_model_architecture: Saved model architecture: 9f96ab56_architecture.pkl
04/06/2026-23:14:13.240 INFO:weightslab.components.checkpoint_manager:_save_changes: Saving model weights checkpoint with component changes...
04/06/2026-23:14:13.240 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 66, 'batch_size': 2}}
04/06/2026-23:14:13.258 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 4c107d099f96ab56bb73166f_step_000020.pt
04/06/2026-23:14:13.273 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 4c107d099f96ab56bb73166f_step_000020.pt (step 20)
04/06/2026-23:14:13.295 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:13.304 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:13.304 INFO:weightslab.components.global_monitoring:resume: Hashes by module: ['4c107d09', '9f96ab56', 'bb73166f']
04/06/2026-23:14:13.304 INFO:weightslab.components.global_monitoring:resume: Resuming training now...
04/06/2026-23:14:13.304 INFO:weightslab.components.global_monitoring:resume: Hashes by module on resume: ['4c107d09', '9f96ab56', 'bb73166f']
04/06/2026-23:14:13.304 INFO:weightslab.components.global_monitoring:resume: 
Training resumed as modules hashes have been computed: ['4c107d09', '9f96ab56', 'bb73166f'].
04/06/2026-23:14:13.304 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:13.304 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:13.321 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:13.321 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:13.321 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:13.404 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:13.404 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:13.404] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:13.423 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:13.423] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:13.423 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:13.423 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:13.424 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:13.424] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:13.427 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:13.427 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:13.430 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:13.438 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:13.438 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:13.438 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:13.438 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:13.537 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:13.539 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:13.539 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:13.539] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:13.539 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:13.640 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:13.640 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:13.640 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:13.640] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:13.648 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:13.648] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:13.648 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:13.648 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:13.649 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:13.649] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:13.657 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:13.741 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:13.741 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:13.780 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:13.817 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:13.820 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:13.820] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:13.820 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:13.824 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:13.824 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:13.824 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:13.921 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:13.921 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:13.921] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:13.928 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:13.928] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:13.929 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:13.929 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:13.929 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:13.930 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:13.931 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:13.931] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:13.939 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:13.943 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:14:14.062 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=5)
04/06/2026-23:14:14.065 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:14.069 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 71, 'batch_size': 2}}
04/06/2026-23:14:14.090 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 4c107d099f96ab56bb73166f_step_000025.pt
04/06/2026-23:14:14.133 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 4c107d099f96ab56bb73166f_step_000025.pt (step 25)
04/06/2026-23:14:14.199 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:14.200 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.200] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:14.201 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:14.201 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:14.203 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:14.203 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:14.203 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.302 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:14.302 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:14.302] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:14.306 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:14.313 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:14.313] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:14.313 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:14.313 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:14.314 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:14.315 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.315] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:14.320 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:14.329 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:14.329 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:14.329 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:14.329 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.433 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.440 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:14.455 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.455] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:14.455 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:14.535 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.557 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:14.558 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:14.557] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:14.574 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:14.574] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:14.574 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:14.574 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:14.574 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.574] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:14.589 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:14.636 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:14.641 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:14.673 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:14.729 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:14.729 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.729] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:14.729 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:14.736 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:14.736 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:14.736 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.833 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:14.833 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:14.833] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:14.836 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:14.836] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:14.836 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:14.836 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:14.836 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:14.836 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:14.836 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.836] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:14.836 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:14.850 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:14.850 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:14.850 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.850 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:14.955 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:14.968 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:14.970 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:14.970] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:14.971 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:15.056 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:15.071 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:15.071 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:15.071] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:15.084 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:15.083] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:15.084 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:15.084 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:15.085 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:15.085] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:15.100 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:15.158 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:15.163 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:15.210 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:15.245 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:15.249 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:15.249] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:15.249 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:15.249 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:15.249 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:15.249 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:15.350 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:15.350 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:15.350] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:15.355 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:15.360 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:15.360] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:15.360 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:15.360 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:15.360 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:15.360 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:15.360] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:15.374 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:15.378 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Deleted old iterator for cleanup
04/06/2026-23:14:15.490 DEBUG:weightslab.backend.dataloader_interface:_reset_iterator: Created new iterator (num_workers=0, sampler_len=5)
04/06/2026-23:14:15.490 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:15.518 DEBUG:weightslab.components.checkpoint_manager:save_model_checkpoint: Captured dataloader iteration states: {'main': {'samples_yielded': 76, 'batch_size': 2}}
04/06/2026-23:14:15.546 INFO:weightslab.components.checkpoint_manager:save_model_checkpoint: Saved model checkpoint: 4c107d099f96ab56bb73166f_step_000030.pt
04/06/2026-23:14:15.613 DEBUG:weightslab.components.checkpoint_manager:_update_manifest_weight_checkpoint: Updated manifest with weight checkpoint: 4c107d099f96ab56bb73166f_step_000030.pt (step 30)
04/06/2026-23:14:15.684 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:15.686 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:15.686] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:15.687 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:15.687 INFO:weightslab.components.checkpoint_manager:save_logger_snapshot: Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
04/06/2026-23:14:15.689 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:15.689 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:15.689 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:15.787 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:15.787 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:15.787] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:15.795 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:15.795] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:15.795 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:15.795 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:15.797 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:15.796] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:15.797 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:15.800 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:15.803 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:15.810 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:15.814 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 4.
04/06/2026-23:14:15.814 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
04/06/2026-23:14:15.814 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:15.915 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:15.940 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
04/06/2026-23:14:15.942 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:15.942] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:15.942 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:16.019 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
04/06/2026-23:14:16.043 INFO:weightslab.data.dataframe_manager:flush: Flushing 4 buffered records to DataFrame.
04/06/2026-23:14:16.043 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:16.043] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
04/06/2026-23:14:16.053 DEBUG:weightslab.data.dataframe_manager:_apply_buffer_records: [23:14:16.053] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
04/06/2026-23:14:16.053 INFO:weightslab.data.dataframe_manager:flush: Applied 4 buffered records to DataFrame.
04/06/2026-23:14:16.053 INFO:weightslab.data.dataframe_manager:flush: Checking if flush to H5 is needed. Pending count: 4.
04/06/2026-23:14:16.056 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:16.056] [LedgeredDataFrameManager] flushing to H5 store.
04/06/2026-23:14:16.070 DEBUG:weightslab.data.h5_dataframe_store:_create_backup: [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
04/06/2026-23:14:16.124 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
04/06/2026-23:14:16.128 DEBUG:weightslab.data.dataframe_manager:flush_async: [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
04/06/2026-23:14:16.159 INFO:weightslab.data.dataframe_manager:enqueue_batch: Enqueued 2 records to buffer. Buffer size is now 2.
04/06/2026-23:14:16.193 DEBUG:weightslab.data.h5_dataframe_store:upsert: [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
Trained on ['2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3'].
04/06/2026-23:14:16.195 DEBUG:weightslab.data.dataframe_manager:_flush_snapshot_to_h5: [23:14:16.195] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
04/06/2026-23:14:16.195 INFO:weightslab.data.dataframe_manager:flush: Completed flush. Pending count after flush: 0.
04/06/2026-23:14:16.195 INFO:weightslab.components.global_monitoring:pause: 
Training paused.

Part B: Modifying model from reloaded state C...
04/06/2026-23:14:16.195 INFO:weightslab.components.checkpoint_manager:load_state: 
============================================================
04/06/2026-23:14:16.195 INFO:weightslab.components.checkpoint_manager:load_state: Loading and applying state: 4c107d095ee5e995...
04/06/2026-23:14:16.195 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================
04/06/2026-23:14:16.208 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loading checkpoint 4c107d095ee5e995...
04/06/2026-23:14:16.208 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Target: HP=4c107d09 MODEL=5ee5e995 DATA=bb73166f
04/06/2026-23:14:16.208 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   Current: HP=4c107d09 MODEL=9f96ab56 DATA=bb73166f
04/06/2026-23:14:16.214 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded model architecture from hash 5ee5e995
04/06/2026-23:14:16.214 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   Using weight checkpoint from manifest: 4c107d095ee5e995bb73166f_step_000020.pt
04/06/2026-23:14:16.221 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded weights from step 20 with RNG state
04/06/2026-23:14:16.221 DEBUG:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 20, 'batch_size': 2}}
04/06/2026-23:14:16.221 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [-] Config unchanged, using current config
04/06/2026-23:14:16.222 INFO:weightslab.components.checkpoint_manager:load_checkpoint:   [OK] Loaded RNG state for reproducibility (data unchanged)
04/06/2026-23:14:16.222 INFO:weightslab.components.checkpoint_manager:load_checkpoint: Loaded components: {'weights', 'model'}
04/06/2026-23:14:16.223 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied model architecture and weights (step 20)
04/06/2026-23:14:16.224 INFO:weightslab.components.checkpoint_manager:load_state: [OK] Applied weights to existing model (step 20)
04/06/2026-23:14:16.224 DEBUG:weightslab.utils.tools:restore_rng_state: Restored Python random state
04/06/2026-23:14:16.224 DEBUG:weightslab.utils.tools:restore_rng_state: Restored NumPy random state
04/06/2026-23:14:16.224 DEBUG:weightslab.utils.tools:restore_rng_state: Restored PyTorch RNG state
04/06/2026-23:14:16.225 DEBUG:weightslab.utils.tools:restore_rng_state: Restored CUDA RNG state
04/06/2026-23:14:16.225 DEBUG:weightslab.utils.tools:restore_rng_state: Successfully restored RNG states
04/06/2026-23:14:16.225 DEBUG:weightslab.components.checkpoint_manager:load_state: Restored RNG state from checkpoint
04/06/2026-23:14:16.229 DEBUG:weightslab.components.experiment_hash:restore_hashes: Restored hashes hp=4c107d09, model=5ee5e995, data=bb73166f, combined=4c107d095ee5e995bb73166f
04/06/2026-23:14:16.231 INFO:weightslab.components.checkpoint_manager:load_state: 
[OK] Successfully loaded and applied state: 4c107d095ee5e995
04/06/2026-23:14:16.231 INFO:weightslab.components.checkpoint_manager:load_state: ============================================================


Modifying model architecture...
04/06/2026-23:14:16.231 DEBUG:weightslab.models.model_with_ops:_operate: Operate currently on neurons: {} of layer id: -2 with op_type: 3
04/06/2026-23:14:16.231 DEBUG:weightslab.models.model_with_ops:_operate: [DEBUG OPERATE] Called with layer_id=-2
04/06/2026-23:14:16.231 DEBUG:weightslab.models.model_with_ops:_operate: [DEBUG OPERATE] After _reverse_indexing, layer_id=-2
---------------------------- Captured stderr call -----------------------------
\rTraining:   0%|          | 0/11 [00:00<?, ?it/s]\rTraining:  18%|\u2588\u258a        | 2/11 [00:00<00:00, 15.50it/s]\rTraining:  36%|\u2588\u2588\u2588\u258b      | 4/11 [00:00<00:01,  5.67it/s]\rTraining:  45%|\u2588\u2588\u2588\u2588\u258c     | 5/11 [00:01<00:01,  4.22it/s]\rTraining:  55%|\u2588\u2588\u2588\u2588\u2588\u258d    | 6/11 [00:01<00:01,  3.50it/s]\rTraining:  64%|\u2588\u2588\u2588\u2588\u2588\u2588\u258e   | 7/11 [00:01<00:00,  4.20it/s]\rTraining:  73%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e  | 8/11 [00:01<00:00,  3.50it/s]\rTraining:  82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 9/11 [00:02<00:00,  4.18it/s]\rTraining:  91%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 10/11 [00:02<00:00,  3.38it/s]\rTraining: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:02<00:00,  3.09it/s]\rTraining: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:02<00:00,  3.81it/s]
------------------------------ Captured log call ------------------------------
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1854 
============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1855 Loading and applying state: 4c107d095ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1856 ============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1646 Loading checkpoint 4c107d095ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1647   Target: HP=4c107d09 MODEL=5ee5e995 DATA=bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1648   Current: HP=01cbbef2 MODEL=5ee5e995 DATA=bb73166f
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 2 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:13.200] [LedgeredDataFrameManager] Flushed 2 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2570 Completed non-blocking flush check. Pending count after flush: 0.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1702   [OK] Loaded model RNG state for reproducibility (model unchanged)
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1724   Using weight checkpoint from manifest: 4c107d095ee5e995bb73166f_step_000020.pt
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1745   [OK] Loaded weights from step 20 with RNG state
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1761   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 20, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1779   [OK] Loaded config (hash changed)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1816   [OK] Loaded RNG state for reproducibility (data unchanged)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1824 Loaded components: {'config', 'weights'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1942 [OK] Applied weights to existing model (step 20)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1998 [OK] Applied hyperparameters config
DEBUG    weightslab.utils.tools:tools.py:181 Restored Python random state
DEBUG    weightslab.utils.tools:tools.py:194 Restored NumPy random state
DEBUG    weightslab.utils.tools:tools.py:210 Restored PyTorch RNG state
DEBUG    weightslab.utils.tools:tools.py:224 Restored CUDA RNG state
DEBUG    weightslab.utils.tools:tools.py:228 Successfully restored RNG states
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:2030 Restored RNG state from checkpoint
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:344 Restored hashes hp=4c107d09, model=5ee5e995, data=bb73166f, combined=4c107d095ee5e995bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2122 
[OK] Successfully loaded and applied state: 4c107d095ee5e995
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2126 ============================================================

INFO     weightslab.components.global_monitoring:global_monitoring.py:142 
Attempting to resume training...
INFO     weightslab.components.experiment_hash:experiment_hash.py:165 Experiment configuration changed: {'model'}
INFO     weightslab.components.experiment_hash:experiment_hash.py:94 Generated experiment hash: 4c107d099f96ab56bb73166f- (HP: 4c107d09, Model: 9f96ab56, Data: bb73166f)
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:95   HP hash: 4c107d09
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:96   Model hash: 9f96ab56
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:97   Data hash: bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:639 New experiment hash: 4c107d09-9f96ab56-bb73166f (previous: 4c107d09-5ee5e995-bb73166f)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:642 Changed components: {'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:690 Changes pending (not dumped yet): {'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1340 Dumping pending changes: {'model'}
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:292 Created checkpoint directories for 4c107d099f96ab56bb73166f: hp_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\HP\4c107d09, model_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\models\9f96ab56, data_dir=C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:845 Dumping model architecture...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1051 Saved model architecture: 9f96ab56_architecture.pkl
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:873 Saving model weights checkpoint with component changes...
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 66, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 4c107d099f96ab56bb73166f_step_000020.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 4c107d099f96ab56bb73166f_step_000020.pt (step 20)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.components.global_monitoring:global_monitoring.py:153 Hashes by module: ['4c107d09', '9f96ab56', 'bb73166f']
INFO     weightslab.components.global_monitoring:global_monitoring.py:157 Resuming training now...
INFO     weightslab.components.global_monitoring:global_monitoring.py:159 Hashes by module on resume: ['4c107d09', '9f96ab56', 'bb73166f']
INFO     weightslab.components.global_monitoring:global_monitoring.py:160 
Training resumed as modules hashes have been computed: ['4c107d09', '9f96ab56', 'bb73166f'].
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:13.404] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:13.423] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:13.424] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:13.539] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:13.640] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:13.648] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:13.649] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:13.820] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:13.921] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:13.928] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:13.931] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=5)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 71, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 4c107d099f96ab56bb73166f_step_000025.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 4c107d099f96ab56bb73166f_step_000025.pt (step 25)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:14.200] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:14.302] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:14.313] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:14.315] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:14.455] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:14.557] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:14.574] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:14.574] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:14.729] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:14.833] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:14.836] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:14.836] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:14.970] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:15.071] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:15.083] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:15.085] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:15.249] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:15.350] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:15.360] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:15.360] [LedgeredDataFrameManager] flushing to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1089 Deleted old iterator for cleanup
DEBUG    weightslab.backend.dataloader_interface:dataloader_interface.py:1120 Created new iterator (num_workers=0, sampler_len=5)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:949 Captured dataloader iteration states: {'main': {'samples_yielded': 76, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:980 Saved model checkpoint: 4c107d099f96ab56bb73166f_step_000030.pt
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:338 Updated manifest with weight checkpoint: 4c107d099f96ab56bb73166f_step_000030.pt (step 30)
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:15.686] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1313 Saved logger snapshot: C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\loggers\loggers.manifest.json (1 chunks)
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:15.787] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:15.795] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:15.796] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2545 [LedgeredDataFrameManager] Waiting for buffer to drain. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:15.942] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 4.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2586 Flushing 4 buffered records to DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1863 [23:14:16.043] [LedgeredDataFrameManager] Applying 4 buffered records to Global DataFrame.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:1883 [23:14:16.053] [LedgeredDataFrameManager] Applied 4 buffered records to Global DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2588 Applied 4 buffered records to DataFrame.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2591 Checking if flush to H5 is needed. Pending count: 4.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2066 [23:14:16.056] [LedgeredDataFrameManager] flushing to H5 store.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:491 [H5DataFrameStore] Created backup at C:\Users\GUILLA~1\AppData\Local\Temp\claude\checkpoint_v3_test__205_rkn\experiments\checkpoints\data\data.h5.backup
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2548 [LedgeredDataFrameManager] Acquiring buffer lock for flush_async check. Buffer size: 0.
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2550 [LedgeredDataFrameManager] Buffer drained, proceeding. Buffer size: 0.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:1161 Enqueued 2 records to buffer. Buffer size is now 2.
DEBUG    weightslab.data.h5_dataframe_store:h5_dataframe_store.py:744 [H5DataFrameStore] Successfully upserted 4 rows for unknown_taggabledataset_2269881410768
DEBUG    weightslab.data.dataframe_manager:dataframe_manager.py:2071 [23:14:16.195] [LedgeredDataFrameManager] Flushed 4 rows (origin=unknown_taggabledataset_2269881410768) to H5 store.
INFO     weightslab.data.dataframe_manager:dataframe_manager.py:2593 Completed flush. Pending count after flush: 0.
INFO     weightslab.components.global_monitoring:global_monitoring.py:134 
Training paused.
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1854 
============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1855 Loading and applying state: 4c107d095ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1856 ============================================================
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1646 Loading checkpoint 4c107d095ee5e995...
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1647   Target: HP=4c107d09 MODEL=5ee5e995 DATA=bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1648   Current: HP=4c107d09 MODEL=9f96ab56 DATA=bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1681   [OK] Loaded model architecture from hash 5ee5e995
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1724   Using weight checkpoint from manifest: 4c107d095ee5e995bb73166f_step_000020.pt
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1745   [OK] Loaded weights from step 20 with RNG state
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:1761   [OK] Found dataloader iteration state(s): {'main': {'samples_yielded': 20, 'batch_size': 2}}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1785   [-] Config unchanged, using current config
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1816   [OK] Loaded RNG state for reproducibility (data unchanged)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1824 Loaded components: {'weights', 'model'}
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1923 [OK] Applied model architecture and weights (step 20)
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:1942 [OK] Applied weights to existing model (step 20)
DEBUG    weightslab.utils.tools:tools.py:181 Restored Python random state
DEBUG    weightslab.utils.tools:tools.py:194 Restored NumPy random state
DEBUG    weightslab.utils.tools:tools.py:210 Restored PyTorch RNG state
DEBUG    weightslab.utils.tools:tools.py:224 Restored CUDA RNG state
DEBUG    weightslab.utils.tools:tools.py:228 Successfully restored RNG states
DEBUG    weightslab.components.checkpoint_manager:checkpoint_manager.py:2030 Restored RNG state from checkpoint
DEBUG    weightslab.components.experiment_hash:experiment_hash.py:344 Restored hashes hp=4c107d09, model=5ee5e995, data=bb73166f, combined=4c107d095ee5e995bb73166f
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2122 
[OK] Successfully loaded and applied state: 4c107d095ee5e995
INFO     weightslab.components.checkpoint_manager:checkpoint_manager.py:2126 ============================================================

DEBUG    weightslab.models.model_with_ops:model_with_ops.py:260 Operate currently on neurons: {} of layer id: -2 with op_type: 3
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:273 [DEBUG OPERATE] Called with layer_id=-2
DEBUG    weightslab.models.model_with_ops:model_with_ops.py:275 [DEBUG OPERATE] After _reverse_indexing, layer_id=-204/06/2026-23:14:18.282 INFO:weightslab.data.dataframe_manager:flush_if_needed_nonblocking: Flushing 2 buffered records to DataFrame (non-blocking).
Traceback (most recent call last):
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage
    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)
               ^^^^^^^^
NameError: name 'get_mask' is not defined
--- Logging error ---

___ CheckpointSystemTests.test_09_reload_before_hp_change_verify_and_modify ___
weightslab\tests\components\test_checkpoint_workflow.py:1088: in test_09_reload_before_hp_change_verify_and_modify
    print(f"Reloading state B: {hash_b[:16]}...")
                                ^^^^^^^^^^^
E   TypeError: 'NoneType' object is not subscriptable
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST 09: Reload Before HP Change - Verify and Fix Everything
================================================================================

Part A: Reloading state B and verifying training reproducibility...
____ CheckpointSystemTests.test_10_reload_branch_j_verify_reproducibility _____
weightslab\tests\components\test_checkpoint_workflow.py:1174: in test_10_reload_branch_j_verify_reproducibility
    print(f"Reloading branch J: {hash_j[:16]}...")
                                 ^^^^^^^^^^^
E   TypeError: 'NoneType' object is not subscriptable
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST 10: Reload Branch J - Verify Training Reproducibility
================================================================================

_ CheckpointSystemTests.test_11_restart_from_scratch_to_hash_d_and_verify_reproducibility _
weightslab\tests\components\test_checkpoint_workflow.py:1223: in test_11_restart_from_scratch_to_hash_d_and_verify_reproducibility
    self.chkpt_manager = CheckpointManager(root_log_dir=self.config.get('root_log_dir'))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weightslab\components\checkpoint_manager.py:91: in __init__
    self.root_log_dir = Path(root_log_dir).absolute()
                        ^^^^^^^^^^^^^^^^^^
..\..\..\AppData\Local\Programs\Python\Python311\Lib\pathlib.py:871: in __new__
    self = cls._from_parts(args)
           ^^^^^^^^^^^^^^^^^^^^^
..\..\..\AppData\Local\Programs\Python\Python311\Lib\pathlib.py:509: in _from_parts
    drv, root, parts = self._parse_args(args)
                       ^^^^^^^^^^^^^^^^^^^^^^
..\..\..\AppData\Local\Programs\Python\Python311\Lib\pathlib.py:493: in _parse_args
    a = os.fspath(a)
        ^^^^^^^^^^^^
weightslab\backend\ledgers.py:148: in __fspath__
    raise TypeError("ValueProxy resolved to None")
E   TypeError: ValueProxy resolved to None
---------------------------- Captured stdout call -----------------------------

================================================================================
TEST 11: Restart Experiment from Config - Verify Full Reproducibility
================================================================================

Simulating fresh restart: loading everything from config...
Target state: 4c107d095ee5e995 (branch_d)
[OK] Cleared existing ledger entries
============================== warnings summary ===============================
weightslab\art.py:42
  C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\art.py:42: DeprecationWarning: invalid escape sequence '\ '
    _BANNER = f"""

weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_01_train_A
  C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py:1799: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '[array([1.], dtype=float32) array([0.], dtype=float32)]' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
    self._df.loc[df_updates.index, df_updates.columns] = (

weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_01_train_A
  C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py:1799: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '[0 0]' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
    self._df.loc[df_updates.index, df_updates.columns] = (

weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_01_train_A
  C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\h5_dataframe_store.py:708: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '['[8]' '[4]']' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
    existing.loc[common_idx, col] = df_norm.loc[common_idx, col]

weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_01_train_A
  C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\h5_dataframe_store.py:708: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '['[0.0]' '[1.0]']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
    existing.loc[common_idx, col] = df_norm.loc[common_idx, col]

weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_04_train_D_data_change
  C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\h5_dataframe_store.py:708: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '[True False False True True True]' has dtype incompatible with bool, please explicitly cast to a compatible dtype first.
    existing.loc[common_idx, col] = df_norm.loc[common_idx, col]

weightslab/tests/components/test_checkpoint_workflow.py: 51 warnings
  C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\tables\path.py:146: NaturalNameWarning: object name is not a valid Python identifier: 'tag:ugly'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
    check_attribute_name(name)

weightslab/tests/components/test_checkpoint_workflow.py: 51 warnings
  C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\tables\attributeset.py:508: NaturalNameWarning: object name is not a valid Python identifier: 'tag:ugly_kind'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
    check_attribute_name(name)

weightslab/tests/components/test_checkpoint_workflow.py: 51 warnings
  C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\tables\attributeset.py:508: NaturalNameWarning: object name is not a valid Python identifier: 'tag:ugly_meta'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
    check_attribute_name(name)

weightslab/tests/components/test_checkpoint_workflow.py: 51 warnings
  C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\tables\attributeset.py:508: NaturalNameWarning: object name is not a valid Python identifier: 'tag:ugly_dtype'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
    check_attribute_name(name)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ===========================
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_02_train_B_model_change
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_04_train_D_data_change
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_06_reload_before_model_change
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_07_change_data_from_test06
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_08_reload_before_data_change_verify_and_modify
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_09_reload_before_hp_change_verify_and_modify
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_10_reload_branch_j_verify_reproducibility
FAILED weightslab/tests/components/test_checkpoint_workflow.py::CheckpointSystemTests::test_11_restart_from_scratch_to_hash_d_and_verify_reproducibility
================= 8 failed, 7 passed, 210 warnings in 54.55s ==================
Traceback (most recent call last):
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage
    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)
               ^^^^^^^^
NameError: name 'get_mask' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\logging\__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 2241, in _worker
    self.flush_if_needed_nonblocking(force=True)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 2564, in flush_if_needed_nonblocking
    self._apply_buffer_records_nonblocking(buffered)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1926, in _apply_buffer_records_nonblocking
    normalized_rows = self._df.loc[applied_index].apply(
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py", line 10374, in apply
    return op.apply().__finalize__(self, method="apply")
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\apply.py", line 916, in apply
    return self.apply_standard()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\apply.py", line 1063, in apply_standard
    results, res_index = self.apply_series_generator()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\apply.py", line 1081, in apply_series_generator
    results[i] = self.func(v, *self.args, **self.kwargs)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1927, in <lambda>
    lambda row: self._normalize_arrays_for_storage(row),
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1665, in _normalize_arrays_for_storage
    logger.debug(f"[_normalize_arrays_for_storage] Failed to normalize array for column={col}, sample_id={sample_id}: {e}")
Message: "[_normalize_arrays_for_storage] Failed to normalize array for column=prediction, sample_id=2: name 'get_mask' is not defined"
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage
    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)
               ^^^^^^^^
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\logging\__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
NameError: name 'get_mask' is not defined
ValueError: I/O operation on closed file.
Call stack:
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\utils\logs.py", line 62, in _print_log_location
    print(f"\n{'='*60}\nWeightsLab session log saved to:\n{_LOG_FILE_PATH}\n{'='*60}", flush=True)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\utils\logs.py", line 218, in print
    logging.info(log_message)
Message: '\n============================================================\nWeightsLab session log saved to:\nC:\\Users\\GUILLA~1\\AppData\\Local\\Temp\\claude\\checkpoint_v3_test__205_rkn\\experiments\\weightslab_20260604_231333.log\n============================================================'
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1662, in _normalize_arrays_for_storage
    row[col] = get_mask(value, dataset=dataset, dataset_index=dataset_index)
               ^^^^^^^^
NameError: name 'get_mask' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\logging\__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 2241, in _worker
    self.flush_if_needed_nonblocking(force=True)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 2564, in flush_if_needed_nonblocking
    self._apply_buffer_records_nonblocking(buffered)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1926, in _apply_buffer_records_nonblocking
    normalized_rows = self._df.loc[applied_index].apply(
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py", line 10374, in apply
    return op.apply().__finalize__(self, method="apply")
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\apply.py", line 916, in apply
    return self.apply_standard()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\apply.py", line 1063, in apply_standard
    results, res_index = self.apply_series_generator()
  File "C:\Users\GuillaumePELLUET\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\apply.py", line 1081, in apply_series_generator
    results[i] = self.func(v, *self.args, **self.kwargs)
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1927, in <lambda>
    lambda row: self._normalize_arrays_for_storage(row),
  File "C:\Users\GuillaumePELLUET\Documents\Codes\weightslab\weightslab\data\dataframe_manager.py", line 1665, in _normalize_arrays_for_storage
    logger.debug(f"[_normalize_arrays_for_storage] Failed to normalize array for column={col}, sample_id={sample_id}: {e}")
Message: "[_normalize_arrays_for_storage] Failed to normalize array for column=target, sample_id=2: name 'get_mask' is not defined"
Arguments: ()
