"./operations.test -test.v -test.timeout 1h0m0s -remote TestB2: -list-retries 5 -verbose -fast-list" - Starting (try 1/5) 2025/01/06 01:11:48 DEBUG : Creating backend with remote "TestB2:rclone-test-womemiz7tuku" 2025/01/06 01:11:48 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/06 01:11:49 DEBUG : Creating backend with remote "/tmp/rclone730593119" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" 2025/01/06 01:11:50 DEBUG : chunksize-probe: Cancelling large file upload === RUN TestMultithreadCopy/upload=false,size=201326591,streams=2 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: write buffer set to 131072 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 3/3 (134217728-201326591) size 64.000Mi starting 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/06 01:12:12 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/06 01:12:13 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 2 2025/01/06 01:12:13 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 0 2025/01/06 01:12:13 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 1 2025/01/06 01:12:18 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 3/3 (134217728-201326591) size 64.000Mi finished 2025/01/06 01:12:19 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/01/06 01:12:21 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/01/06 01:12:21 DEBUG : test-multithread-copy-false-201326591-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=201326592,streams=2 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: write buffer set to 131072 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 3/3 (134217728-201326592) size 64Mi starting 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 2 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 0 2025/01/06 01:12:51 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 1 2025/01/06 01:12:56 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 3/3 (134217728-201326592) size 64Mi finished 2025/01/06 01:12:56 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/01/06 01:12:56 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/01/06 01:12:56 DEBUG : test-multithread-copy-false-201326592-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=201326593,streams=2 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: write buffer set to 131072 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: Starting multi-thread copy with 4 chunks of size 64Mi with 4 parallel streams 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 4/4 (201326592-201326593) size 1 starting 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 1/4 (0-67108864) size 64Mi starting 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 2/4 (67108864-134217728) size 64Mi starting 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 3/4 (134217728-201326592) size 64Mi starting 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 3 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 4/4 (201326592-201326593) size 1 finished 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 0 2025/01/06 01:13:26 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 1 2025/01/06 01:13:27 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 2 2025/01/06 01:13:31 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 1/4 (0-67108864) size 64Mi finished 2025/01/06 01:13:31 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 2/4 (67108864-134217728) size 64Mi finished 2025/01/06 01:13:32 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 3/4 (134217728-201326592) size 64Mi finished 2025/01/06 01:13:32 DEBUG : test-multithread-copy-false-201326593-2: Finished multi-thread copy with 4 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=201326591,streams=2 2025/01/06 01:13:54 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: disabling buffering because source is local disk 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: Starting multi-thread copy with 2 chunks of size 96Mi with 2 parallel streams 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 2/2 (100663296-201326591) size 96.000Mi starting 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi starting 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: Sending chunk 1 length 100663295 2025/01/06 01:13:55 DEBUG : test-multithread-copy-true-201326591-2: Sending chunk 0 length 100663296 2025/01/06 01:14:03 DEBUG : test-multithread-copy-true-201326591-2: Done sending chunk 1 2025/01/06 01:14:03 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 2/2 (100663296-201326591) size 96.000Mi finished 2025/01/06 01:14:09 DEBUG : test-multithread-copy-true-201326591-2: Done sending chunk 0 2025/01/06 01:14:09 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi finished 2025/01/06 01:14:09 DEBUG : test-multithread-copy-true-201326591-2: Finishing large file upload with 2 parts 2025/01/06 01:14:09 DEBUG : test-multithread-copy-true-201326591-2: Finished multi-thread copy with 2 parts of size 96Mi === RUN TestMultithreadCopy/upload=true,size=201326592,streams=2 2025/01/06 01:14:30 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: disabling buffering because source is local disk 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: Starting multi-thread copy with 2 chunks of size 96Mi with 2 parallel streams 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 2/2 (100663296-201326592) size 96Mi starting 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi starting 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: Sending chunk 1 length 100663296 2025/01/06 01:14:31 DEBUG : test-multithread-copy-true-201326592-2: Sending chunk 0 length 100663296 2025/01/06 01:14:39 DEBUG : test-multithread-copy-true-201326592-2: Done sending chunk 1 2025/01/06 01:14:39 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 2/2 (100663296-201326592) size 96Mi finished 2025/01/06 01:14:39 DEBUG : test-multithread-copy-true-201326592-2: Done sending chunk 0 2025/01/06 01:14:39 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi finished 2025/01/06 01:14:39 DEBUG : test-multithread-copy-true-201326592-2: Finishing large file upload with 2 parts 2025/01/06 01:14:40 DEBUG : test-multithread-copy-true-201326592-2: Finished multi-thread copy with 2 parts of size 96Mi === RUN TestMultithreadCopy/upload=true,size=201326593,streams=2 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: disabling buffering because source is local disk 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: Starting multi-thread copy with 3 chunks of size 96Mi with 3 parallel streams 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 starting 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 2 length 1 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi starting 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi starting 2025/01/06 01:15:02 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 0 length 100663296 2025/01/06 01:15:03 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 1 length 100663296 2025/01/06 01:15:03 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 2 2025/01/06 01:15:03 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 finished 2025/01/06 01:15:10 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 0 2025/01/06 01:15:10 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi finished 2025/01/06 01:15:11 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 1 2025/01/06 01:15:11 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi finished 2025/01/06 01:15:11 DEBUG : test-multithread-copy-true-201326593-2: Finishing large file upload with 3 parts 2025/01/06 01:15:12 DEBUG : test-multithread-copy-true-201326593-2: Finished multi-thread copy with 3 parts of size 96Mi --- PASS: TestMultithreadCopy (210.40s) --- PASS: TestMultithreadCopy/upload=false,size=201326591,streams=2 (38.43s) --- PASS: TestMultithreadCopy/upload=false,size=201326592,streams=2 (35.16s) --- PASS: TestMultithreadCopy/upload=false,size=201326593,streams=2 (35.42s) --- PASS: TestMultithreadCopy/upload=true,size=201326591,streams=2 (37.26s) --- PASS: TestMultithreadCopy/upload=true,size=201326592,streams=2 (31.33s) --- PASS: TestMultithreadCopy/upload=true,size=201326593,streams=2 (30.97s) === RUN TestMultithreadCopyAbort run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" 2025/01/06 01:15:20 DEBUG : chunksize-probe: Cancelling large file upload 2025/01/06 01:15:34 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because source is local disk 2025/01/06 01:15:35 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2025/01/06 01:15:35 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/06 01:15:35 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 96Mi with 3 parallel streams 2025/01/06 01:15:35 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 starting 2025/01/06 01:15:35 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:35 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi starting 2025/01/06 01:15:35 DEBUG : Open with options = [RangeOption(0,100663295)] 2025/01/06 01:15:35 DEBUG : test-multithread-abort: Sending chunk 0 length 100663296 2025/01/06 01:15:35 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi starting 2025/01/06 01:15:35 DEBUG : Open with options = [RangeOption(100663296,201326591)] 2025/01/06 01:15:35 DEBUG : test-multithread-abort: Sending chunk 1 length 100663296 2025/01/06 01:15:45 DEBUG : test-multithread-abort: Done sending chunk 0 2025/01/06 01:15:45 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi finished 2025/01/06 01:15:51 DEBUG : test-multithread-abort: Done sending chunk 1 2025/01/06 01:15:51 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi finished 2025/01/06 01:15:51 DEBUG : Returning error reader 2025/01/06 01:15:51 DEBUG : test-multithread-abort: Sending chunk 2 length 1 2025/01/06 01:15:51 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:51 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 0/10: BOOM: simulated read failure 2025/01/06 01:15:51 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:53 DEBUG : Returning error reader 2025/01/06 01:15:53 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:53 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 1/10: BOOM: simulated read failure 2025/01/06 01:15:53 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:54 DEBUG : Returning error reader 2025/01/06 01:15:54 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:54 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 2/10: BOOM: simulated read failure 2025/01/06 01:15:54 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:55 DEBUG : Returning error reader 2025/01/06 01:15:55 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:55 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 3/10: BOOM: simulated read failure 2025/01/06 01:15:55 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:56 DEBUG : Returning error reader 2025/01/06 01:15:56 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:56 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 4/10: BOOM: simulated read failure 2025/01/06 01:15:56 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:57 DEBUG : Returning error reader 2025/01/06 01:15:57 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:57 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 5/10: BOOM: simulated read failure 2025/01/06 01:15:57 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:58 DEBUG : Returning error reader 2025/01/06 01:15:58 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:58 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 6/10: BOOM: simulated read failure 2025/01/06 01:15:58 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:15:59 DEBUG : Returning error reader 2025/01/06 01:15:59 DEBUG : BOOM: simulated read failure 2025/01/06 01:15:59 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 7/10: BOOM: simulated read failure 2025/01/06 01:15:59 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:16:00 DEBUG : Returning error reader 2025/01/06 01:16:00 DEBUG : BOOM: simulated read failure 2025/01/06 01:16:00 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 8/10: BOOM: simulated read failure 2025/01/06 01:16:00 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:16:01 DEBUG : Returning error reader 2025/01/06 01:16:01 DEBUG : BOOM: simulated read failure 2025/01/06 01:16:01 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 9/10: BOOM: simulated read failure 2025/01/06 01:16:01 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/01/06 01:16:02 DEBUG : Returning error reader 2025/01/06 01:16:02 DEBUG : BOOM: simulated read failure 2025/01/06 01:16:02 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 10/10: BOOM: simulated read failure 2025/01/06 01:16:02 DEBUG : test-multithread-abort: Reopen failed after offset 0 bytes read: failed to reopen: too many retries 2025/01/06 01:16:02 DEBUG : test-multithread-abort: Error sending chunk 2 (retry=false): Post "https://pod-000-1181-17.backblaze.com/b2api/v1/b2_upload_part/4_z8f720dc320bd3b9b9f480e1b_f201468940cd35b10_d20250106_m011535_c001_v0001181_t0038_u01736126135572/0042": BOOM: simulated read failure: &url.Error{Op:"Post", URL:"https://pod-000-1181-17.backblaze.com/b2api/v1/b2_upload_part/4_z8f720dc320bd3b9b9f480e1b_f201468940cd35b10_d20250106_m011535_c001_v0001181_t0038_u01736126135572/0042", Err:(*errors.errorString)(0xc000485000)} 2025/01/06 01:16:02 DEBUG : test-multithread-abort: Error sending chunk 2: Post "https://pod-000-1181-17.backblaze.com/b2api/v1/b2_upload_part/4_z8f720dc320bd3b9b9f480e1b_f201468940cd35b10_d20250106_m011535_c001_v0001181_t0038_u01736126135572/0042": BOOM: simulated read failure 2025/01/06 01:16:02 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 failed: multi-thread copy: failed to write chunk: Post "https://pod-000-1181-17.backblaze.com/b2api/v1/b2_upload_part/4_z8f720dc320bd3b9b9f480e1b_f201468940cd35b10_d20250106_m011535_c001_v0001181_t0038_u01736126135572/0042": BOOM: simulated read failure 2025/01/06 01:16:02 DEBUG : test-multithread-abort: multi-thread copy: cancelling transfer on exit 2025/01/06 01:16:02 DEBUG : test-multithread-abort: Cancelling large file upload --- PASS: TestMultithreadCopyAbort (43.51s) === RUN TestSizeDiffers --- PASS: TestSizeDiffers (0.00s) === RUN TestReOpen === RUN TestReOpen/Normal === RUN TestReOpen/Normal/Basics 2025/01/06 01:16:03 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/ErrorAtStart === RUN TestReOpen/Normal/WithErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/Normal/TooManyErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/Normal/Seek 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Seek from 5 to 2 === RUN TestReOpen/Normal/AccountRead === RUN TestReOpen/Normal/AccountReadDelay 2025/01/06 01:16:03 DEBUG : potato: Seek from 10 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 10 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/AccountReadError === RUN TestReOpen/WithRangeOption === RUN TestReOpen/WithRangeOption/Basics 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/ErrorAtStart === RUN TestReOpen/WithRangeOption/WithErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithRangeOption/TooManyErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithRangeOption/Seek 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Seek from 5 to 2 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 4 === RUN TestReOpen/WithRangeOption/AccountRead === RUN TestReOpen/WithRangeOption/AccountReadDelay 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/AccountReadError === RUN TestReOpen/WithSeekOption === RUN TestReOpen/WithSeekOption/Basics 2025/01/06 01:16:03 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/ErrorAtStart === RUN TestReOpen/WithSeekOption/WithErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithSeekOption/TooManyErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithSeekOption/Seek 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Seek from 5 to 2 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 5 === RUN TestReOpen/WithSeekOption/AccountRead === RUN TestReOpen/WithSeekOption/AccountReadDelay 2025/01/06 01:16:03 DEBUG : potato: Seek from 8 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 8 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/AccountReadError === RUN TestReOpen/UnknownSize === RUN TestReOpen/UnknownSize/Basics 2025/01/06 01:16:03 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/ErrorAtStart === RUN TestReOpen/UnknownSize/WithErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/UnknownSize/TooManyErrors 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/06 01:16:03 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/UnknownSize/Seek 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/06 01:16:03 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/06 01:16:03 DEBUG : potato: Seek from 5 to 2 2025/01/06 01:16:03 DEBUG : potato: Seek from 7 to 6 === RUN TestReOpen/UnknownSize/AccountRead === RUN TestReOpen/UnknownSize/AccountReadDelay 2025/01/06 01:16:03 DEBUG : potato: Seek from 9 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 9 to 0 2025/01/06 01:16:03 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/AccountReadError --- PASS: TestReOpen (0.00s) --- PASS: TestReOpen/Normal (0.00s) --- PASS: TestReOpen/Normal/Basics (0.00s) --- PASS: TestReOpen/Normal/ErrorAtStart (0.00s) --- PASS: TestReOpen/Normal/WithErrors (0.00s) --- PASS: TestReOpen/Normal/TooManyErrors (0.00s) --- PASS: TestReOpen/Normal/Seek (0.00s) --- PASS: TestReOpen/Normal/AccountRead (0.00s) --- PASS: TestReOpen/Normal/AccountReadDelay (0.00s) --- PASS: TestReOpen/Normal/AccountReadError (0.00s) --- PASS: TestReOpen/WithRangeOption (0.00s) --- PASS: TestReOpen/WithRangeOption/Basics (0.00s) --- PASS: TestReOpen/WithRangeOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithRangeOption/WithErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/Seek (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountRead (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadError (0.00s) --- PASS: TestReOpen/WithSeekOption (0.00s) --- PASS: TestReOpen/WithSeekOption/Basics (0.00s) --- PASS: TestReOpen/WithSeekOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithSeekOption/WithErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/Seek (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountRead (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadError (0.00s) --- PASS: TestReOpen/UnknownSize (0.00s) --- PASS: TestReOpen/UnknownSize/Basics (0.00s) --- PASS: TestReOpen/UnknownSize/ErrorAtStart (0.00s) --- PASS: TestReOpen/UnknownSize/WithErrors (0.00s) --- PASS: TestReOpen/UnknownSize/TooManyErrors (0.00s) --- PASS: TestReOpen/UnknownSize/Seek (0.00s) --- PASS: TestReOpen/UnknownSize/AccountRead (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadDelay (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadError (0.00s) === RUN TestCheck run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" === RUN TestCheck/1 === RUN TestCheck/2 === RUN TestCheck/3 === RUN TestCheck/4 === RUN TestCheck/5 === RUN TestCheck/6 === RUN TestCheck/7 --- PASS: TestCheck (4.32s) --- PASS: TestCheck/1 (0.19s) --- PASS: TestCheck/2 (0.19s) --- PASS: TestCheck/3 (0.19s) --- PASS: TestCheck/4 (0.18s) --- PASS: TestCheck/5 (0.18s) --- PASS: TestCheck/6 (0.18s) --- PASS: TestCheck/7 (0.18s) === RUN TestCheckFsError 2025/01/06 01:16:07 DEBUG : Creating backend with remote "nonexistent" 2025/01/06 01:16:07 DEBUG : Config file has changed externally - reloading 2025/01/06 01:16:07 DEBUG : Creating backend with remote "nonexistent" 2025/01/06 01:16:07 DEBUG : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: Waiting for checks to finish 2025/01/06 01:16:07 ERROR : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: error reading source root directory: directory not found 2025/01/06 01:16:07 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 differences found 2025/01/06 01:16:07 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 errors while checking --- PASS: TestCheckFsError (0.00s) === RUN TestCheckDownload run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" === RUN TestCheckDownload/1 === RUN TestCheckDownload/2 === RUN TestCheckDownload/3 === RUN TestCheckDownload/4 === RUN TestCheckDownload/5 === RUN TestCheckDownload/6 === RUN TestCheckDownload/7 --- PASS: TestCheckDownload (5.80s) --- PASS: TestCheckDownload/1 (0.73s) --- PASS: TestCheckDownload/2 (0.36s) --- PASS: TestCheckDownload/3 (0.36s) --- PASS: TestCheckDownload/4 (0.55s) --- PASS: TestCheckDownload/5 (0.38s) --- PASS: TestCheckDownload/6 (0.37s) --- PASS: TestCheckDownload/7 (0.37s) === RUN TestCheckSizeOnly run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" === RUN TestCheckSizeOnly/1 === RUN TestCheckSizeOnly/2 === RUN TestCheckSizeOnly/3 === RUN TestCheckSizeOnly/4 === RUN TestCheckSizeOnly/5 === RUN TestCheckSizeOnly/6 === RUN TestCheckSizeOnly/7 --- PASS: TestCheckSizeOnly (4.52s) --- PASS: TestCheckSizeOnly/1 (0.19s) --- PASS: TestCheckSizeOnly/2 (0.18s) --- PASS: TestCheckSizeOnly/3 (0.19s) --- PASS: TestCheckSizeOnly/4 (0.18s) --- PASS: TestCheckSizeOnly/5 (0.18s) --- PASS: TestCheckSizeOnly/6 (0.18s) --- PASS: TestCheckSizeOnly/7 (0.18s) === RUN TestCheckEqualReaders --- PASS: TestCheckEqualReaders (0.00s) === RUN TestParseSumFile run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" 2025/01/06 01:16:19 NOTICE: test.sum: improperly formatted checksum line 4 2025/01/06 01:16:19 NOTICE: test.sum: improperly formatted checksum line 5 2025/01/06 01:16:19 NOTICE: test.sum: improperly formatted checksum line 6 2025/01/06 01:16:19 NOTICE: test.sum: 2 warning(s) suppressed... 2025/01/06 01:16:19 NOTICE: test.sum: improperly formatted checksum line 4 2025/01/06 01:16:19 NOTICE: test.sum: improperly formatted checksum line 5 2025/01/06 01:16:19 NOTICE: test.sum: improperly formatted checksum line 6 2025/01/06 01:16:19 NOTICE: test.sum: 2 warning(s) suppressed... --- PASS: TestParseSumFile (2.07s) === RUN TestCheckSum run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" 2025/01/06 01:16:20 DEBUG : Creating backend with remote "TestB2:rclone-test-womemiz7tuku/data" check_test.go:354: B2 bucket rclone-test-womemiz7tuku path data lacks md5, skipping --- SKIP: TestCheckSum (1.47s) === RUN TestCheckSumDownload run.go:180: Remote "B2 bucket rclone-test-womemiz7tuku", Local "Local file system at /tmp/rclone730593119", Modify Window "1ms" 2025/01/06 01:16:21 DEBUG : Creating backend with remote "TestB2:rclone-test-womemiz7tuku/data" === RUN TestCheckSumDownload/subtest1 === RUN TestCheckSumDownload/subtest2 === RUN TestCheckSumDownload/subtest3 "./operations.test -test.v -test.timeout 1h0m0s -remote TestB2: -list-retries 5 -verbose -fast-list" - Finished ERROR in 4m52.125790823s (try 1/5): signal: killed: Failed []