"./operations.test -test.v -test.timeout 1h0m0s -remote TestAzureBlob: -verbose -fast-list -test.run '^TestMultithreadCopyAbort$'" - Starting (try 4/5)
2025/01/24 04:05:15 DEBUG : Creating backend with remote "TestAzureBlob:rclone-test-pawifen7cela"
2025/01/24 04:05:15 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2025/01/24 04:05:15 DEBUG : Creating backend with remote "/tmp/rclone200389794"
=== RUN   TestMultithreadCopyAbort
    run.go:180: Remote "Azure container rclone-test-pawifen7cela", Local "Local file system at /tmp/rclone200389794", Modify Window "1ns"
2025/01/24 04:05:15 DEBUG : chunksize-probe: Multipart upload session started for 25 parts of size 4Mi
2025/01/24 04:05:15 DEBUG : chunksize-probe: open chunk writer: started multipart upload
2025/01/24 04:05:15 DEBUG : chunksize-probe: Clearing uncommitted blocks
2025/01/24 04:05:15 DEBUG : chunksize-probe: Committing 0 blocks to remove uncommitted blocks
2025/01/24 04:05:15 DEBUG : chunksize-probe: Removing empty object
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because source is local disk
2025/01/24 04:05:16 DEBUG : test-multithread-abort: Multipart upload session started for 3 parts of size 4Mi
2025/01/24 04:05:16 DEBUG : test-multithread-abort: open chunk writer: started multipart upload
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 16 instead of --multi-thread-streams 1
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: number of streams 16 was bigger than number of chunks 3
2025/01/24 04:05:16 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 4Mi with 3 parallel streams
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (8388608-8388609) size 1 starting
2025/01/24 04:05:16 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-4194304) size 4Mi starting
2025/01/24 04:05:16 DEBUG : Open with options = [RangeOption(0,4194303)]
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (4194304-8388608) size 4Mi starting
2025/01/24 04:05:16 DEBUG : Open with options = [RangeOption(4194304,8388607)]
2025/01/24 04:05:16 DEBUG : test-multithread-abort: Seek from 4194304 to 0
2025/01/24 04:05:16 DEBUG : Open with options = [RangeOption(4194304,8388607)]
2025/01/24 04:05:16 DEBUG : test-multithread-abort: Seek from 4194304 to 0
2025/01/24 04:05:16 DEBUG : Open with options = [RangeOption(0,4194303)]
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multipart upload wrote chunk 2 with 4194304 bytes
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (4194304-8388608) size 4Mi finished
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multipart upload wrote chunk 1 with 4194304 bytes
2025/01/24 04:05:16 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-4194304) size 4Mi finished
2025/01/24 04:05:17 DEBUG : Returning error reader
2025/01/24 04:05:17 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:17 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 1/10: BOOM: simulated read failure
2025/01/24 04:05:17 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:18 DEBUG : Returning error reader
2025/01/24 04:05:18 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:18 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 2/10: BOOM: simulated read failure
2025/01/24 04:05:18 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:19 DEBUG : Returning error reader
2025/01/24 04:05:19 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:19 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 3/10: BOOM: simulated read failure
2025/01/24 04:05:19 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:20 DEBUG : Returning error reader
2025/01/24 04:05:20 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:20 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 4/10: BOOM: simulated read failure
2025/01/24 04:05:20 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:21 DEBUG : Returning error reader
2025/01/24 04:05:21 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:21 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 5/10: BOOM: simulated read failure
2025/01/24 04:05:21 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:22 DEBUG : Returning error reader
2025/01/24 04:05:22 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:22 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 6/10: BOOM: simulated read failure
2025/01/24 04:05:22 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:23 DEBUG : Returning error reader
2025/01/24 04:05:23 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:23 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 7/10: BOOM: simulated read failure
2025/01/24 04:05:23 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:24 DEBUG : Returning error reader
2025/01/24 04:05:24 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:24 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 8/10: BOOM: simulated read failure
2025/01/24 04:05:24 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:25 DEBUG : Returning error reader
2025/01/24 04:05:25 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:25 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 9/10: BOOM: simulated read failure
2025/01/24 04:05:25 DEBUG : Open with options = [RangeOption(8388608,8388608)]
2025/01/24 04:05:26 DEBUG : Returning error reader
2025/01/24 04:05:26 DEBUG : BOOM: simulated read failure
2025/01/24 04:05:26 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 10/10: BOOM: simulated read failure
2025/01/24 04:05:26 DEBUG : test-multithread-abort: Reopen failed after offset 0 bytes read: failed to reopen: too many retries
2025/01/24 04:05:26 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 failed: multi-thread copy: failed to write chunk: BOOM: simulated read failure
2025/01/24 04:05:26 DEBUG : test-multithread-abort: multi-thread copy: cancelling transfer on exit
2025/01/24 04:05:26 DEBUG : test-multithread-abort: Clearing uncommitted blocks
2025/01/24 04:05:26 DEBUG : test-multithread-abort: 2 Uncommitted blocks found
2025/01/24 04:05:26 DEBUG : test-multithread-abort: Committing 0 blocks to remove uncommitted blocks
    fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3
    fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3
    fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3
    fstest.go:299: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:340
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:330
        	Error:      	Should be true
        	Test:       	TestMultithreadCopyAbort
        	Messages:   	listing wrong, want
        	            	  test-multithread-abort (100) got
        	            	  test-multithread-abort (0)
    fstest.go:144: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:144
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:189
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:340
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:330
        	Error:      	Not equal: 
        	            	expected: 100
        	            	actual  : 0
        	Test:       	TestMultithreadCopyAbort
        	Messages:   	test-multithread-abort: size incorrect file=100 vs obj=0
--- FAIL: TestMultithreadCopyAbort (18.39s)
FAIL
2025/01/24 04:05:33 DEBUG : Azure container rclone-test-pawifen7cela: Purge remote
"./operations.test -test.v -test.timeout 1h0m0s -remote TestAzureBlob: -verbose -fast-list -test.run '^TestMultithreadCopyAbort$'" - Finished ERROR in 18.491924812s (try 4/5): exit status 1: Failed [TestMultithreadCopyAbort]