Events disabled
Loading tokenizer from "/home/marcel/projects/tokers/.claude/worktrees/agent-a5ba8c6f81c499ac8/data/gpt2_tokenizer.json"...
Reading "/home/marcel/data/owt_train.txt"...
Read 1.00 GB in 0.2s
Encoding (single-threaded)...
Events enabled
pass 0: 228107519 tokens in 1.24s — 0.80 GB/s (804 MB/s)
cache: short 1315258 entries (cap 2097152), long 51476 (cap 57344, 1157749 key bytes), arena 918268 tokens (cap 1835008)

 Performance counter stats for 'env ENCODE_MB=1000 ENCODE_PASSES=1 ./target/release/deps/encode_st-57c4b9fdb93c41d6':

     5,417,746,484      cycles:u                                                              
    23,706,791,860      instructions:u                                                        

       1.066140098 seconds time elapsed

       1.331699000 seconds user
       0.134272000 seconds sys


