"./operations.test -test.v -test.timeout 2h0m0s -remote TestDrime: -list-retries 5 -verbose -test.run '^(TestCheck|TestCheckDownload|TestCheckSizeOnly|TestCheckSumDownload|TestCopyFileBackupDir|TestDelete|TestListDirSorted|TestListDirSortedFn|TestListJSON|TestLsWithFilesFrom|TestMaxDelete|TestMaxDeleteSize|TestMultithreadCopyAbort|TestParseSumFile|TestPurge|TestRmdirsWithFilter|TestStatJSON)$'" - Starting (try 3/5) 2026/09/09 02:35:22 DEBUG : Creating backend with remote "TestDrime:rclone-test-gomalep7vugo" 2026/09/09 02:35:22 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/09 02:35:23 DEBUG : Creating backend with remote "/tmp/rclone2590012887" === RUN TestMultithreadCopyAbort run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" 2026/09/09 02:35:26 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because source is local disk 2026/09/09 02:35:26 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2026/09/09 02:35:26 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2026/09/09 02:35:26 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 5Mi with 3 parallel streams 2026/09/09 02:35:26 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (10485760-10485761) size 1 starting 2026/09/09 02:35:26 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:26 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-5242880) size 5Mi starting 2026/09/09 02:35:26 DEBUG : Open with options = [RangeOption(0,5242879)] 2026/09/09 02:35:26 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (5242880-10485760) size 5Mi starting 2026/09/09 02:35:26 DEBUG : Open with options = [RangeOption(5242880,10485759)] 2026/09/09 02:35:27 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-5242880) size 5Mi finished 2026/09/09 02:35:27 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (5242880-10485760) size 5Mi finished 2026/09/09 02:35:27 DEBUG : Returning error reader 2026/09/09 02:35:27 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:27 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 0/10: BOOM: simulated read failure 2026/09/09 02:35:27 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:28 DEBUG : Returning error reader 2026/09/09 02:35:28 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:28 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 1/10: BOOM: simulated read failure 2026/09/09 02:35:28 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:29 DEBUG : Returning error reader 2026/09/09 02:35:29 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:29 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 2/10: BOOM: simulated read failure 2026/09/09 02:35:29 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:30 DEBUG : Returning error reader 2026/09/09 02:35:30 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:30 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 3/10: BOOM: simulated read failure 2026/09/09 02:35:30 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:31 DEBUG : Returning error reader 2026/09/09 02:35:31 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:31 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 4/10: BOOM: simulated read failure 2026/09/09 02:35:31 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:32 DEBUG : Returning error reader 2026/09/09 02:35:32 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:32 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 5/10: BOOM: simulated read failure 2026/09/09 02:35:32 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:33 DEBUG : Returning error reader 2026/09/09 02:35:33 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:33 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 6/10: BOOM: simulated read failure 2026/09/09 02:35:33 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:34 DEBUG : Returning error reader 2026/09/09 02:35:34 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:34 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 7/10: BOOM: simulated read failure 2026/09/09 02:35:34 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:35 DEBUG : Returning error reader 2026/09/09 02:35:35 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:35 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 8/10: BOOM: simulated read failure 2026/09/09 02:35:35 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:36 DEBUG : Returning error reader 2026/09/09 02:35:36 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:36 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 9/10: BOOM: simulated read failure 2026/09/09 02:35:36 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/09/09 02:35:37 DEBUG : Returning error reader 2026/09/09 02:35:37 DEBUG : BOOM: simulated read failure 2026/09/09 02:35:37 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 10/10: BOOM: simulated read failure 2026/09/09 02:35:37 DEBUG : test-multithread-abort: Reopen failed after offset 0 bytes read: failed to reopen: too many retries 2026/09/09 02:35:37 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 failed: multi-thread copy: failed to write chunk: failed to upload part 3: Put "https://drimestorage.18b804eaa6bc9759e2278f8d5367f42f.r2.cloudflarestorage.com/uploads/586f930d-b1c4-4f38-a491-da5da67d6663/586f930d-b1c4-4f38-a491-da5da67d6663?uploadId=AG1mqOgbOlzN2iJNsb2quR4--__VOsb58C3dMPVxXsDSoOMCS8czqlRDdI955UYsoVZr8w7GyWsdXOCgq__rva23JWLuvCf_d151p5sYIENuzQhiRlemFGpcandmTh6xUhaKOoUZHA1N9dbZpEwetkkOQ6ucts-9H8xpK8NNKz1TJJnhAjKuAoIrCr4KJN6GQ4bfeTkKuBNsRdRnKrrFJMvMm8OaJkgT3KRFiycCgyZ1i7mhsISL4b1n2s3ZgByqX5HZdDOngiskhvq2YY7F4ODVdYCX5sGnBbRuW_de-6iDHvOFU8IA7tt62x4Q3ucgTEX8d0CM1hsmraQdcB2vuEs&partNumber=3&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=9dd875828a029f3dc92060a58cb00078%2F20260909%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20260909T023527Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Signature=8a8db18b63fd00af54724263d1267422838059d65655aaece25a5b019404f3f5": BOOM: simulated read failure 2026/09/09 02:35:37 DEBUG : test-multithread-abort: multi-thread copy: cancelling transfer on exit run.go:173: removing file "test-multithread-abort" failed - try 1/5: sleeping 1s: object not found run.go:173: removing file "test-multithread-abort" failed - try 2/5: sleeping 1.5s: object not found run.go:173: removing file "test-multithread-abort" failed - try 3/5: sleeping 2.25s: object not found run.go:173: removing file "test-multithread-abort" failed - try 4/5: sleeping 3.375s: object not found run.go:173: removing file "test-multithread-abort" failed: object not found --- PASS: TestMultithreadCopyAbort (24.74s) === RUN TestCheck run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "rutabaga" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestCheck (0.97s) === RUN TestCheckDownload run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" === RUN TestCheckDownload/1 === RUN TestCheckDownload/2 === RUN TestCheckDownload/3 === RUN TestCheckDownload/4 === RUN TestCheckDownload/5 === RUN TestCheckDownload/6 === RUN TestCheckDownload/7 --- PASS: TestCheckDownload (6.15s) --- PASS: TestCheckDownload/1 (0.35s) --- PASS: TestCheckDownload/2 (0.31s) --- PASS: TestCheckDownload/3 (0.29s) --- PASS: TestCheckDownload/4 (0.43s) --- PASS: TestCheckDownload/5 (0.31s) --- PASS: TestCheckDownload/6 (0.28s) --- PASS: TestCheckDownload/7 (0.27s) === RUN TestCheckSizeOnly run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" === RUN TestCheckSizeOnly/1 === RUN TestCheckSizeOnly/2 === RUN TestCheckSizeOnly/3 === RUN TestCheckSizeOnly/4 === RUN TestCheckSizeOnly/5 === RUN TestCheckSizeOnly/6 === RUN TestCheckSizeOnly/7 --- PASS: TestCheckSizeOnly (5.78s) --- PASS: TestCheckSizeOnly/1 (0.09s) --- PASS: TestCheckSizeOnly/2 (0.09s) --- PASS: TestCheckSizeOnly/3 (0.12s) --- PASS: TestCheckSizeOnly/4 (0.10s) --- PASS: TestCheckSizeOnly/5 (0.11s) --- PASS: TestCheckSizeOnly/6 (0.10s) --- PASS: TestCheckSizeOnly/7 (0.22s) === RUN TestParseSumFile run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "test.sum" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestParseSumFile (0.60s) === RUN TestCheckSumDownload run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" 2026/09/09 02:36:01 DEBUG : Creating backend with remote "TestDrime:rclone-test-gomalep7vugo/data" run.go:303: Failed to put "data/banana" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:293: Sleeping for 6s just to make sure --- FAIL: TestCheckSumDownload (10.15s) === RUN TestCopyFileBackupDir run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "dst/file1" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:293: Sleeping for 6s just to make sure --- FAIL: TestCopyFileBackupDir (8.87s) === RUN TestListDirSorted run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "sub dir/sub sub dir/hello world3" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestListDirSorted (9.42s) === RUN TestListDirSortedFn run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "a.txt" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestListDirSortedFn (0.47s) === RUN TestListJSON run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "file1" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestListJSON (0.36s) === RUN TestStatJSON run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" === RUN TestStatJSON/Root === RUN TestStatJSON/RootFilesOnly === RUN TestStatJSON/RootDirsOnly === RUN TestStatJSON/Dir === RUN TestStatJSON/DirWithTrailingSlash === RUN TestStatJSON/File === RUN TestStatJSON/NotFound === RUN TestStatJSON/DirFilesOnly === RUN TestStatJSON/FileFilesOnly === RUN TestStatJSON/NotFoundFilesOnly === RUN TestStatJSON/DirDirsOnly === RUN TestStatJSON/FileDirsOnly === RUN TestStatJSON/NotFoundDirsOnly === RUN TestStatJSON/RootNotFound 2026/09/09 02:36:33 DEBUG : Creating backend with remote "TestDrime:rclone-test-gomalep7vugo/notfound" 2026/09/09 02:36:33 DEBUG : Config file has changed externally - reloading === NAME TestStatJSON run.go:186: removing dir "sub" failed - try 1/5: sleeping 1s: directory not empty --- PASS: TestStatJSON (6.24s) --- PASS: TestStatJSON/Root (0.09s) --- PASS: TestStatJSON/RootFilesOnly (0.00s) --- PASS: TestStatJSON/RootDirsOnly (0.09s) --- PASS: TestStatJSON/Dir (0.18s) --- PASS: TestStatJSON/DirWithTrailingSlash (0.09s) --- PASS: TestStatJSON/File (0.11s) --- PASS: TestStatJSON/NotFound (0.20s) --- PASS: TestStatJSON/DirFilesOnly (0.11s) --- PASS: TestStatJSON/FileFilesOnly (0.16s) --- PASS: TestStatJSON/NotFoundFilesOnly (0.10s) --- PASS: TestStatJSON/DirDirsOnly (0.12s) --- PASS: TestStatJSON/FileDirsOnly (0.09s) --- PASS: TestStatJSON/NotFoundDirsOnly (0.09s) --- PASS: TestStatJSON/RootNotFound (0.66s) === RUN TestLsWithFilesFrom run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "potato2" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestLsWithFilesFrom (0.50s) === RUN TestDelete run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "small" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestDelete (0.45s) === RUN TestMaxDelete run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "small" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestMaxDelete (0.70s) === RUN TestMaxDeleteSize run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" run.go:303: Failed to put "small" to "drime root 'rclone-test-gomalep7vugo'": failed to upload file: Error "The file failed to upload." --- FAIL: TestMaxDeleteSize (0.75s) === RUN TestPurge 2026/09/09 02:36:39 DEBUG : Creating backend with remote "TestDrime:rclone-test-lokumig1yuvi" 2026/09/09 02:36:39 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/09 02:36:39 DEBUG : Creating backend with remote "/tmp/rclone4217312190" run.go:198: Remote "drime root 'rclone-test-lokumig1yuvi'", Local "Local file system at /tmp/rclone4217312190", Modify Window "876000h0m0s" 2026/09/09 02:36:41 INFO : A2: Making directory 2026/09/09 02:36:41 INFO : A1/B2: Making directory 2026/09/09 02:36:41 INFO : A1/B2/C2: Making directory 2026/09/09 02:36:42 INFO : A1/B1/C3: Making directory 2026/09/09 02:36:42 INFO : A3: Making directory 2026/09/09 02:36:42 INFO : A3/B3: Making directory 2026/09/09 02:36:42 INFO : A3/B3/C4: Making directory 2026/09/09 02:36:45 ERROR : error listing: directory not found 2026/09/09 02:36:45 DEBUG : drime root 'rclone-test-lokumig1yuvi': Purge remote 2026/09/09 02:36:45 NOTICE: purge failed: directory not found --- PASS: TestPurge (6.20s) === RUN TestRmdirsWithFilter run.go:198: Remote "drime root 'rclone-test-gomalep7vugo'", Local "Local file system at /tmp/rclone2590012887", Modify Window "876000h0m0s" 2026/09/09 02:36:45 INFO : A1: Making directory 2026/09/09 02:36:46 INFO : A1/B1: Making directory 2026/09/09 02:36:46 INFO : A1/B1/C1: Making directory 2026/09/09 02:36:47 DEBUG : removing 1 level 3 directories 2026/09/09 02:36:47 INFO : A1/B1/C1: Removing directory 2026/09/09 02:36:48 DEBUG : removing 1 level 2 directories 2026/09/09 02:36:48 INFO : A1/B1: Removing directory 2026/09/09 02:36:48 ERROR : A1/B1: Failed to rmdir: directory not empty operations_test.go:805: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:805 Error: Received unexpected error: failed to remove directories: directory not empty Test: TestRmdirsWithFilter run.go:186: removing dir "A1" failed - try 1/5: sleeping 1s: directory not empty --- FAIL: TestRmdirsWithFilter (4.82s) FAIL 2026/09/09 02:36:50 DEBUG : drime root 'rclone-test-gomalep7vugo': Purge remote "./operations.test -test.v -test.timeout 2h0m0s -remote TestDrime: -list-retries 5 -verbose -test.run '^(TestCheck|TestCheckDownload|TestCheckSizeOnly|TestCheckSumDownload|TestCopyFileBackupDir|TestDelete|TestListDirSorted|TestListDirSortedFn|TestListJSON|TestLsWithFilesFrom|TestMaxDelete|TestMaxDeleteSize|TestMultithreadCopyAbort|TestParseSumFile|TestPurge|TestRmdirsWithFilter|TestStatJSON)$'" - Finished ERROR in 1m27.681314318s (try 3/5): exit status 1: Failed [TestCheck TestParseSumFile TestCheckSumDownload TestCopyFileBackupDir TestListDirSorted TestListDirSortedFn TestListJSON TestLsWithFilesFrom TestDelete TestMaxDelete TestMaxDeleteSize TestRmdirsWithFilter]