Downloading numpy (15.9MiB)
Downloading pandas (10.3MiB)
Downloading hf-xet (4.3MiB)
Downloading pygments (1.2MiB)
Downloading aiohttp (1.7MiB)
Downloading pyarrow (47.8MiB)
 Downloading hf-xet
 Downloading aiohttp
 Downloading pygments
 Downloading numpy
 Downloading pyarrow
 Downloading pandas
Installed 40 packages in 846ms
============================= test session starts ==============================
platform linux -- Python 3.13.7, pytest-8.4.1, pluggy-1.6.0
rootdir: /tests
plugins: anyio-4.15.1, json-ctrf-0.3.5
collected 1 item

../tests/test_outputs.py F                                               [100%]

=================================== FAILURES ===================================
______________________ test_compress_decompress_workflow _______________________

generate_test_data = {'c4-mini-00000-of-10000.jsonl': '755f90ce91ae322c2643712308e12d120ca079180bf0d5180accfe7342ec2ee3', 'c4-mini-00001-of...3db97db8f138', 'c4-mini-00003-of-10000.jsonl': 'a7e4c98cecbca03fe6bcb719cfa252c6629c613863058da907cda1311f53342b', ...}

    def test_compress_decompress_workflow(generate_test_data):
        """Test the full compress -> decompress workflow is successful."""
        original_hashes = generate_test_data
    
        # Part 1: Test compression
        # Clean up output directory
        if os.path.exists(TEST_OUTPUT_DIR):
            shutil.rmtree(TEST_OUTPUT_DIR)
    
        # Install dependencies first to avoid snapshot contamination
        print("Installing dependencies with uv sync...")
        sync_result = subprocess.run(["uv", "sync"], capture_output=True, text=True)
        if sync_result.returncode != 0:
            print(f"uv sync failed: {sync_result.stderr}")
    
        # Run the compress script
        try:
            print(
                f"Running compress script: {COMPRESS_SCRIPT} "
                f"{TEST_INPUT_DIR} {TEST_OUTPUT_DIR}"
            )
            result = subprocess.run(
                ["uv", "run", COMPRESS_SCRIPT, TEST_INPUT_DIR, TEST_OUTPUT_DIR],
                timeout=600,
                capture_output=True,
                text=True,
            )
            if result.returncode != 0:
                print(f"Compress script failed with return code {result.returncode}")
                print(f"stdout: {result.stdout}")
                print(f"stderr: {result.stderr}")
                assert False, f"Compress script failed: {result.stderr}"
        except subprocess.TimeoutExpired:
            assert False, "The compress script timed out!"
    
        # Verify output directory exists
        assert os.path.exists(TEST_OUTPUT_DIR), "Output directory was not created"
    
        # Check that the maximum file size in TEST_OUTPUT_DIR is MAX_SIZE_PER_FILE
        for root, dirs, files in os.walk(TEST_OUTPUT_DIR):
            for file in files:
                file_path = os.path.join(root, file)
                file_size = os.path.getsize(file_path)
                assert file_size <= MAX_SIZE_PER_FILE, (
                    f"File {file_path} is {file_size} bytes, exceeds {MAX_SIZE_PER_FILE}"
                )
    
            # Check that the number of files+folders in each directory is less than
            # MAX_FILES_PER_FOLDER
            total_items = len(files) + len(dirs)
>           assert total_items <= MAX_FILES_PER_FOLDER, (
                f"Directory {root} has {total_items} items, exceeds {MAX_FILES_PER_FOLDER}"
            )
E           AssertionError: Directory /app/c4_test_31aacc97-83b5-414c-830d-fd9203766beb/ has 331 items, exceeds 30
E           assert 331 <= 30

/tests/test_outputs.py:135: AssertionError
---------------------------- Captured stdout setup -----------------------------
Generating test data.
Generated mini-shard 1/10000
Generated mini-shard 101/10000
Generated mini-shard 201/10000
Generated mini-shard 301/10000
Generated mini-shard 401/10000
Generated mini-shard 501/10000
Generated mini-shard 601/10000
Generated mini-shard 701/10000
Generated mini-shard 801/10000
Generated mini-shard 901/10000
Generated mini-shard 1001/10000
Generated mini-shard 1101/10000
Generated mini-shard 1201/10000
Generated mini-shard 1301/10000
Generated mini-shard 1401/10000
Generated mini-shard 1501/10000
Generated mini-shard 1601/10000
Generated mini-shard 1701/10000
Generated mini-shard 1801/10000
Generated mini-shard 1901/10000
Generated mini-shard 2001/10000
Generated mini-shard 2101/10000
Generated mini-shard 2201/10000
Generated mini-shard 2301/10000
Generated mini-shard 2401/10000
Generated mini-shard 2501/10000
Generated mini-shard 2601/10000
Generated mini-shard 2701/10000
Generated mini-shard 2801/10000
Generated mini-shard 2901/10000
Generated mini-shard 3001/10000
Generated mini-shard 3101/10000
Generated mini-shard 3201/10000
Generated mini-shard 3301/10000
Generated mini-shard 3401/10000
Generated mini-shard 3501/10000
Generated mini-shard 3601/10000
Generated mini-shard 3701/10000
Generated mini-shard 3801/10000
Generated mini-shard 3901/10000
Generated mini-shard 4001/10000
Generated mini-shard 4101/10000
Generated mini-shard 4201/10000
Generated mini-shard 4301/10000
Generated mini-shard 4401/10000
Generated mini-shard 4501/10000
Generated mini-shard 4601/10000
Generated mini-shard 4701/10000
Generated mini-shard 4801/10000
Generated mini-shard 4901/10000
Generated mini-shard 5001/10000
Generated mini-shard 5101/10000
Generated mini-shard 5201/10000
Generated mini-shard 5301/10000
Generated mini-shard 5401/10000
Generated mini-shard 5501/10000
Generated mini-shard 5601/10000
Generated mini-shard 5701/10000
Generated mini-shard 5801/10000
Generated mini-shard 5901/10000
Generated mini-shard 6001/10000
Generated mini-shard 6101/10000
Generated mini-shard 6201/10000
Generated mini-shard 6301/10000
Generated mini-shard 6401/10000
Generated mini-shard 6501/10000
Generated mini-shard 6601/10000
Generated mini-shard 6701/10000
Generated mini-shard 6801/10000
Generated mini-shard 6901/10000
Generated mini-shard 7001/10000
Generated mini-shard 7101/10000
Generated mini-shard 7201/10000
Generated mini-shard 7301/10000
Generated mini-shard 7401/10000
Generated mini-shard 7501/10000
Generated mini-shard 7601/10000
Generated mini-shard 7701/10000
Generated mini-shard 7801/10000
Generated mini-shard 7901/10000
Generated mini-shard 8001/10000
Generated mini-shard 8101/10000
Generated mini-shard 8201/10000
Generated mini-shard 8301/10000
Generated mini-shard 8401/10000
Generated mini-shard 8501/10000
Generated mini-shard 8601/10000
Generated mini-shard 8701/10000
Generated mini-shard 8801/10000
Generated mini-shard 8901/10000
Generated mini-shard 9001/10000
Generated mini-shard 9101/10000
Generated mini-shard 9201/10000
Generated mini-shard 9301/10000
Generated mini-shard 9401/10000
Generated mini-shard 9501/10000
Generated mini-shard 9601/10000
Generated mini-shard 9701/10000
Generated mini-shard 9801/10000
Generated 9898 test files
---------------------------- Captured stderr setup -----------------------------
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4628 examples [00:00, 17892.11 examples/s]
Generating train split: 9408 examples [00:00, 16180.56 examples/s]
Generating train split: 14037 examples [00:00, 16155.35 examples/s]
Generating train split: 18508 examples [00:01, 18625.53 examples/s]
Generating train split: 23057 examples [00:01, 22600.50 examples/s]
Generating train split: 27549 examples [00:01, 26025.34 examples/s]
Generating train split: 32060 examples [00:01, 26262.00 examples/s]
Generating train split: 41282 examples [00:01, 33327.17 examples/s]
Generating train split: 45801 examples [00:01, 34076.84 examples/s]
Generating train split: 50561 examples [00:01, 32023.61 examples/s]
Generating train split: 55346 examples [00:02, 35288.15 examples/s]
Generating train split: 59997 examples [00:02, 37482.85 examples/s]
Generating train split: 64252 examples [00:02, 36928.18 examples/s]
Generating train split: 68767 examples [00:02, 32879.86 examples/s]
Generating train split: 73564 examples [00:02, 35972.79 examples/s]
Generating train split: 78065 examples [00:02, 37588.92 examples/s]
Generating train split: 82514 examples [00:02, 37857.11 examples/s]
Generating train split: 87212 examples [00:02, 33736.10 examples/s]
Generating train split: 91794 examples [00:03, 36146.77 examples/s]
Generating train split: 96407 examples [00:03, 27385.73 examples/s]
Generating train split: 100921 examples [00:03, 28685.73 examples/s]
Generating train split: 105527 examples [00:03, 25705.12 examples/s]
Generating train split: 110011 examples [00:03, 24597.47 examples/s]
Generating train split: 114448 examples [00:04, 21780.04 examples/s]
Generating train split: 119080 examples [00:04, 24975.10 examples/s]
Generating train split: 123717 examples [00:04, 20983.12 examples/s]
Generating train split: 128199 examples [00:04, 21051.10 examples/s]
Generating train split: 132871 examples [00:04, 21841.26 examples/s]
Generating train split: 137465 examples [00:05, 22355.79 examples/s]
Generating train split: 142093 examples [00:05, 22101.34 examples/s]
Generating train split: 146537 examples [00:05, 22515.75 examples/s]
Generating train split: 151171 examples [00:05, 22626.42 examples/s]
Generating train split: 155695 examples [00:06, 20580.60 examples/s]
Generating train split: 160125 examples [00:06, 23614.47 examples/s]
Generating train split: 164831 examples [00:06, 26495.61 examples/s]
Generating train split: 169418 examples [00:06, 26539.02 examples/s]
Generating train split: 174224 examples [00:06, 26565.41 examples/s]
Generating train split: 178721 examples [00:06, 29104.18 examples/s]
Generating train split: 183305 examples [00:06, 31986.69 examples/s]
Generating train split: 187976 examples [00:07, 29159.37 examples/s]
Generating train split: 192291 examples [00:07, 31382.61 examples/s]
Generating train split: 196759 examples [00:07, 28231.85 examples/s]
Generating train split: 201256 examples [00:07, 31087.27 examples/s]
Generating train split: 205928 examples [00:07, 28927.70 examples/s]
Generating train split: 210384 examples [00:07, 31616.88 examples/s]
Generating train split: 215024 examples [00:07, 33971.00 examples/s]
Generating train split: 219465 examples [00:08, 31452.99 examples/s]
Generating train split: 224136 examples [00:08, 34428.02 examples/s]
Generating train split: 233020 examples [00:08, 32839.74 examples/s]
Generating train split: 237429 examples [00:08, 33917.21 examples/s]
Generating train split: 242080 examples [00:08, 36478.51 examples/s]
Generating train split: 246733 examples [00:08, 32704.84 examples/s]
Generating train split: 251191 examples [00:08, 34746.92 examples/s]
Generating train split: 255874 examples [00:09, 36220.46 examples/s]
Generating train split: 260334 examples [00:09, 32580.94 examples/s]
Generating train split: 264806 examples [00:09, 34736.79 examples/s]
Generating train split: 269418 examples [00:09, 36247.05 examples/s]
Generating train split: 274149 examples [00:09, 37422.06 examples/s]
Generating train split: 278833 examples [00:09, 32468.07 examples/s]
Generating train split: 283387 examples [00:09, 33872.21 examples/s]
Generating train split: 287951 examples [00:10, 32079.95 examples/s]
Generating train split: 297116 examples [00:10, 35018.09 examples/s]
Generating train split: 301743 examples [00:10, 32819.45 examples/s]
Generating train split: 306036 examples [00:10, 34652.37 examples/s]
Generating train split: 310765 examples [00:10, 36737.10 examples/s]
Generating train split: 315368 examples [00:10, 32484.83 examples/s]
Generating train split: 319765 examples [00:10, 33208.77 examples/s]
Generating train split: 324228 examples [00:11, 35704.85 examples/s]
Generating train split: 328806 examples [00:11, 31799.66 examples/s]
Generating train split: 338040 examples [00:11, 36524.30 examples/s]
Generating train split: 342797 examples [00:11, 36918.15 examples/s]
Generating train split: 347254 examples [00:11, 34114.34 examples/s]
Generating train split: 351738 examples [00:11, 35995.34 examples/s]
Generating train split: 356318 examples [00:12, 31999.46 examples/s]
Generating train split: 356318 examples [00:12, 29587.01 examples/s]
------------------------------ Captured log setup ------------------------------
WARNING  huggingface_hub.utils._http:_http.py:993 Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
----------------------------- Captured stdout call -----------------------------
Installing dependencies with uv sync...
Running compress script: /app/compress.py /app/c4_test_2a5f6fe6-01c1-45c3-80d0-8cd73fa8064a/ /app/c4_test_31aacc97-83b5-414c-830d-fd9203766beb/
=========================== short test summary info ============================
FAILED ../tests/test_outputs.py::test_compress_decompress_workflow - Assertio...
======================== 1 failed in 102.01s (0:01:42) =========================
