"./operations.test -test.v -test.timeout 1h0m0s -remote TestB2: -list-retries 5 -verbose" - Starting (try 3/5) 2024/11/14 21:57:02 DEBUG : Creating backend with remote "TestB2:rclone-test-lebifox6cidi" 2024/11/14 21:57:02 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2024/11/14 21:57:02 DEBUG : Creating backend with remote "/tmp/rclone411355367" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" 2024/11/14 21:57:03 DEBUG : chunksize-probe: Cancelling large file upload === RUN TestMultithreadCopy/upload=false,size=201326591,streams=2 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: write buffer set to 131072 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 3/3 (134217728-201326591) size 64.000Mi starting 2024/11/14 21:58:03 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2024/11/14 21:58:04 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 1 2024/11/14 21:58:04 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 0 2024/11/14 21:58:04 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 2 2024/11/14 21:58:09 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2024/11/14 21:58:09 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2024/11/14 21:58:09 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 3/3 (134217728-201326591) size 64.000Mi finished 2024/11/14 21:58:09 DEBUG : test-multithread-copy-false-201326591-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=201326592,streams=2 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: write buffer set to 131072 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 3/3 (134217728-201326592) size 64Mi starting 2024/11/14 21:58:40 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 0 2024/11/14 21:58:41 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 1 2024/11/14 21:58:41 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 2 2024/11/14 21:58:45 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2024/11/14 21:58:46 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2024/11/14 21:58:46 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 3/3 (134217728-201326592) size 64Mi finished 2024/11/14 21:58:46 DEBUG : test-multithread-copy-false-201326592-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=201326593,streams=2 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: write buffer set to 131072 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: Starting multi-thread copy with 4 chunks of size 64Mi with 4 parallel streams 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 3/4 (134217728-201326592) size 64Mi starting 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 1/4 (0-67108864) size 64Mi starting 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 2/4 (67108864-134217728) size 64Mi starting 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 4/4 (201326592-201326593) size 1 starting 2024/11/14 21:59:15 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 1 2024/11/14 21:59:16 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 3 2024/11/14 21:59:16 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 4/4 (201326592-201326593) size 1 finished 2024/11/14 21:59:16 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 0 2024/11/14 21:59:16 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 2 2024/11/14 21:59:21 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 2/4 (67108864-134217728) size 64Mi finished 2024/11/14 21:59:21 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 1/4 (0-67108864) size 64Mi finished 2024/11/14 21:59:21 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 3/4 (134217728-201326592) size 64Mi finished 2024/11/14 21:59:21 DEBUG : test-multithread-copy-false-201326593-2: Finished multi-thread copy with 4 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=201326591,streams=2 2024/11/14 21:59:42 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: disabling buffering because source is local disk 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: Starting multi-thread copy with 2 chunks of size 96Mi with 2 parallel streams 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 2/2 (100663296-201326591) size 96.000Mi starting 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi starting 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: Sending chunk 1 length 100663295 2024/11/14 21:59:43 DEBUG : test-multithread-copy-true-201326591-2: Sending chunk 0 length 100663296 2024/11/14 21:59:52 DEBUG : test-multithread-copy-true-201326591-2: Done sending chunk 0 2024/11/14 21:59:52 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi finished 2024/11/14 22:00:04 DEBUG : test-multithread-copy-true-201326591-2: Done sending chunk 1 2024/11/14 22:00:04 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 2/2 (100663296-201326591) size 96.000Mi finished 2024/11/14 22:00:04 DEBUG : test-multithread-copy-true-201326591-2: Finishing large file upload with 2 parts 2024/11/14 22:00:04 DEBUG : test-multithread-copy-true-201326591-2: Finished multi-thread copy with 2 parts of size 96Mi === RUN TestMultithreadCopy/upload=true,size=201326592,streams=2 2024/11/14 22:00:26 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: disabling buffering because source is local disk 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: Starting multi-thread copy with 2 chunks of size 96Mi with 2 parallel streams 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 2/2 (100663296-201326592) size 96Mi starting 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi starting 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: Sending chunk 1 length 100663296 2024/11/14 22:00:27 DEBUG : test-multithread-copy-true-201326592-2: Sending chunk 0 length 100663296 2024/11/14 22:00:36 DEBUG : test-multithread-copy-true-201326592-2: Done sending chunk 0 2024/11/14 22:00:36 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi finished 2024/11/14 22:00:36 DEBUG : test-multithread-copy-true-201326592-2: Done sending chunk 1 2024/11/14 22:00:36 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 2/2 (100663296-201326592) size 96Mi finished 2024/11/14 22:00:36 DEBUG : test-multithread-copy-true-201326592-2: Finishing large file upload with 2 parts 2024/11/14 22:00:37 DEBUG : test-multithread-copy-true-201326592-2: Finished multi-thread copy with 2 parts of size 96Mi === RUN TestMultithreadCopy/upload=true,size=201326593,streams=2 2024/11/14 22:00:58 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: disabling buffering because source is local disk 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: Starting multi-thread copy with 3 chunks of size 96Mi with 3 parallel streams 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 starting 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi starting 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi starting 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 2 length 1 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 1 length 100663296 2024/11/14 22:00:59 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 0 length 100663296 2024/11/14 22:01:00 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 2 2024/11/14 22:01:00 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 finished 2024/11/14 22:03:31 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 1 2024/11/14 22:03:31 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi finished 2024/11/14 22:03:33 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 0 2024/11/14 22:03:33 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi finished 2024/11/14 22:03:33 DEBUG : test-multithread-copy-true-201326593-2: Finishing large file upload with 3 parts 2024/11/14 22:03:34 DEBUG : test-multithread-copy-true-201326593-2: Finished multi-thread copy with 3 parts of size 96Mi --- PASS: TestMultithreadCopy (399.58s) --- PASS: TestMultithreadCopy/upload=false,size=201326591,streams=2 (73.54s) --- PASS: TestMultithreadCopy/upload=false,size=201326592,streams=2 (36.35s) --- PASS: TestMultithreadCopy/upload=false,size=201326593,streams=2 (35.27s) --- PASS: TestMultithreadCopy/upload=true,size=201326591,streams=2 (43.34s) --- PASS: TestMultithreadCopy/upload=true,size=201326592,streams=2 (32.26s) --- PASS: TestMultithreadCopy/upload=true,size=201326593,streams=2 (177.09s) === RUN TestMultithreadCopyAbort run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" 2024/11/14 22:03:42 DEBUG : chunksize-probe: Cancelling large file upload 2024/11/14 22:03:57 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because source is local disk 2024/11/14 22:03:58 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2024/11/14 22:03:58 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2024/11/14 22:03:58 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 96Mi with 3 parallel streams 2024/11/14 22:03:58 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 starting 2024/11/14 22:03:58 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi starting 2024/11/14 22:03:58 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi starting 2024/11/14 22:03:58 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:03:58 DEBUG : Open with options = [RangeOption(0,100663295)] 2024/11/14 22:03:58 DEBUG : Open with options = [RangeOption(100663296,201326591)] 2024/11/14 22:03:58 DEBUG : test-multithread-abort: Sending chunk 0 length 100663296 2024/11/14 22:03:58 DEBUG : test-multithread-abort: Sending chunk 1 length 100663296 2024/11/14 22:04:06 DEBUG : test-multithread-abort: Done sending chunk 0 2024/11/14 22:04:06 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi finished 2024/11/14 22:04:06 DEBUG : test-multithread-abort: Done sending chunk 1 2024/11/14 22:04:06 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi finished 2024/11/14 22:04:06 DEBUG : Returning error reader 2024/11/14 22:04:06 DEBUG : test-multithread-abort: Sending chunk 2 length 1 2024/11/14 22:04:06 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:06 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 0/10: BOOM: simulated read failure 2024/11/14 22:04:06 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:07 DEBUG : Returning error reader 2024/11/14 22:04:07 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:07 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 1/10: BOOM: simulated read failure 2024/11/14 22:04:07 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:08 DEBUG : Returning error reader 2024/11/14 22:04:08 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:08 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 2/10: BOOM: simulated read failure 2024/11/14 22:04:08 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:09 DEBUG : Returning error reader 2024/11/14 22:04:09 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:09 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 3/10: BOOM: simulated read failure 2024/11/14 22:04:09 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:10 DEBUG : Returning error reader 2024/11/14 22:04:10 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:10 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 4/10: BOOM: simulated read failure 2024/11/14 22:04:10 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:11 DEBUG : Returning error reader 2024/11/14 22:04:11 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:11 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 5/10: BOOM: simulated read failure 2024/11/14 22:04:11 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:12 DEBUG : Returning error reader 2024/11/14 22:04:12 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:12 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 6/10: BOOM: simulated read failure 2024/11/14 22:04:12 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:13 DEBUG : Returning error reader 2024/11/14 22:04:13 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:13 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 7/10: BOOM: simulated read failure 2024/11/14 22:04:13 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:14 DEBUG : Returning error reader 2024/11/14 22:04:14 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:14 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 8/10: BOOM: simulated read failure 2024/11/14 22:04:14 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:15 DEBUG : Returning error reader 2024/11/14 22:04:15 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:15 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 9/10: BOOM: simulated read failure 2024/11/14 22:04:15 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2024/11/14 22:04:16 DEBUG : Returning error reader 2024/11/14 22:04:16 DEBUG : BOOM: simulated read failure 2024/11/14 22:04:16 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 10/10: BOOM: simulated read failure 2024/11/14 22:04:16 DEBUG : test-multithread-abort: Reopen failed after offset 0 bytes read: failed to reopen: too many retries 2024/11/14 22:04:16 DEBUG : test-multithread-abort: Error sending chunk 2 (retry=false): Post "https://pod-000-1182-18.backblaze.com/b2api/v1/b2_upload_part/4_z8fa20d83a0ed2bcb9f380e1b_f201d4a02c3e539a9_d20241114_m220358_c001_v0001182_t0031_u01731621838356/0006": BOOM: simulated read failure: &url.Error{Op:"Post", URL:"https://pod-000-1182-18.backblaze.com/b2api/v1/b2_upload_part/4_z8fa20d83a0ed2bcb9f380e1b_f201d4a02c3e539a9_d20241114_m220358_c001_v0001182_t0031_u01731621838356/0006", Err:(*errors.errorString)(0xc0009531a0)} 2024/11/14 22:04:16 DEBUG : test-multithread-abort: Error sending chunk 2: Post "https://pod-000-1182-18.backblaze.com/b2api/v1/b2_upload_part/4_z8fa20d83a0ed2bcb9f380e1b_f201d4a02c3e539a9_d20241114_m220358_c001_v0001182_t0031_u01731621838356/0006": BOOM: simulated read failure 2024/11/14 22:04:16 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 failed: multi-thread copy: failed to write chunk: Post "https://pod-000-1182-18.backblaze.com/b2api/v1/b2_upload_part/4_z8fa20d83a0ed2bcb9f380e1b_f201d4a02c3e539a9_d20241114_m220358_c001_v0001182_t0031_u01731621838356/0006": BOOM: simulated read failure 2024/11/14 22:04:16 DEBUG : test-multithread-abort: multi-thread copy: cancelling transfer on exit 2024/11/14 22:04:16 DEBUG : test-multithread-abort: Cancelling large file upload --- PASS: TestMultithreadCopyAbort (35.63s) === RUN TestSizeDiffers --- PASS: TestSizeDiffers (0.00s) === RUN TestReOpen === RUN TestReOpen/Normal === RUN TestReOpen/Normal/Basics 2024/11/14 22:04:17 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/ErrorAtStart === RUN TestReOpen/Normal/WithErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/Normal/TooManyErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/Normal/Seek 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Seek from 5 to 2 === RUN TestReOpen/Normal/AccountRead === RUN TestReOpen/Normal/AccountReadDelay 2024/11/14 22:04:17 DEBUG : potato: Seek from 10 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 10 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/AccountReadError === RUN TestReOpen/WithRangeOption === RUN TestReOpen/WithRangeOption/Basics 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/ErrorAtStart === RUN TestReOpen/WithRangeOption/WithErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithRangeOption/TooManyErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithRangeOption/Seek 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Seek from 5 to 2 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 4 === RUN TestReOpen/WithRangeOption/AccountRead === RUN TestReOpen/WithRangeOption/AccountReadDelay 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/AccountReadError === RUN TestReOpen/WithSeekOption === RUN TestReOpen/WithSeekOption/Basics 2024/11/14 22:04:17 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/ErrorAtStart === RUN TestReOpen/WithSeekOption/WithErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithSeekOption/TooManyErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithSeekOption/Seek 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Seek from 5 to 2 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 5 === RUN TestReOpen/WithSeekOption/AccountRead === RUN TestReOpen/WithSeekOption/AccountReadDelay 2024/11/14 22:04:17 DEBUG : potato: Seek from 8 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 8 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/AccountReadError === RUN TestReOpen/UnknownSize === RUN TestReOpen/UnknownSize/Basics 2024/11/14 22:04:17 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/ErrorAtStart === RUN TestReOpen/UnknownSize/WithErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/UnknownSize/TooManyErrors 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/14 22:04:17 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/UnknownSize/Seek 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/14 22:04:17 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/14 22:04:17 DEBUG : potato: Seek from 5 to 2 2024/11/14 22:04:17 DEBUG : potato: Seek from 7 to 6 === RUN TestReOpen/UnknownSize/AccountRead === RUN TestReOpen/UnknownSize/AccountReadDelay 2024/11/14 22:04:17 DEBUG : potato: Seek from 9 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 9 to 0 2024/11/14 22:04:17 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/AccountReadError --- PASS: TestReOpen (0.00s) --- PASS: TestReOpen/Normal (0.00s) --- PASS: TestReOpen/Normal/Basics (0.00s) --- PASS: TestReOpen/Normal/ErrorAtStart (0.00s) --- PASS: TestReOpen/Normal/WithErrors (0.00s) --- PASS: TestReOpen/Normal/TooManyErrors (0.00s) --- PASS: TestReOpen/Normal/Seek (0.00s) --- PASS: TestReOpen/Normal/AccountRead (0.00s) --- PASS: TestReOpen/Normal/AccountReadDelay (0.00s) --- PASS: TestReOpen/Normal/AccountReadError (0.00s) --- PASS: TestReOpen/WithRangeOption (0.00s) --- PASS: TestReOpen/WithRangeOption/Basics (0.00s) --- PASS: TestReOpen/WithRangeOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithRangeOption/WithErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/Seek (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountRead (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadError (0.00s) --- PASS: TestReOpen/WithSeekOption (0.00s) --- PASS: TestReOpen/WithSeekOption/Basics (0.00s) --- PASS: TestReOpen/WithSeekOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithSeekOption/WithErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/Seek (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountRead (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadError (0.00s) --- PASS: TestReOpen/UnknownSize (0.00s) --- PASS: TestReOpen/UnknownSize/Basics (0.00s) --- PASS: TestReOpen/UnknownSize/ErrorAtStart (0.00s) --- PASS: TestReOpen/UnknownSize/WithErrors (0.00s) --- PASS: TestReOpen/UnknownSize/TooManyErrors (0.00s) --- PASS: TestReOpen/UnknownSize/Seek (0.00s) --- PASS: TestReOpen/UnknownSize/AccountRead (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadDelay (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadError (0.00s) === RUN TestCheck run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" === RUN TestCheck/1 === RUN TestCheck/2 === RUN TestCheck/3 === RUN TestCheck/4 === RUN TestCheck/5 === RUN TestCheck/6 === RUN TestCheck/7 --- PASS: TestCheck (6.65s) --- PASS: TestCheck/1 (0.19s) --- PASS: TestCheck/2 (0.18s) --- PASS: TestCheck/3 (0.19s) --- PASS: TestCheck/4 (0.18s) --- PASS: TestCheck/5 (0.18s) --- PASS: TestCheck/6 (0.18s) --- PASS: TestCheck/7 (0.19s) === RUN TestCheckFsError 2024/11/14 22:04:24 DEBUG : Creating backend with remote "nonexistent" 2024/11/14 22:04:24 DEBUG : Creating backend with remote "nonexistent" 2024/11/14 22:04:24 DEBUG : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: Waiting for checks to finish 2024/11/14 22:04:24 ERROR : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: error reading source root directory: directory not found 2024/11/14 22:04:24 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 differences found 2024/11/14 22:04:24 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 errors while checking --- PASS: TestCheckFsError (0.00s) === RUN TestCheckDownload run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" === RUN TestCheckDownload/1 === RUN TestCheckDownload/2 === RUN TestCheckDownload/3 === RUN TestCheckDownload/4 === RUN TestCheckDownload/5 === RUN TestCheckDownload/6 === RUN TestCheckDownload/7 --- PASS: TestCheckDownload (7.93s) --- PASS: TestCheckDownload/1 (0.74s) --- PASS: TestCheckDownload/2 (0.37s) --- PASS: TestCheckDownload/3 (0.36s) --- PASS: TestCheckDownload/4 (0.56s) --- PASS: TestCheckDownload/5 (0.39s) --- PASS: TestCheckDownload/6 (0.38s) --- PASS: TestCheckDownload/7 (0.38s) === RUN TestCheckSizeOnly run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" === RUN TestCheckSizeOnly/1 === RUN TestCheckSizeOnly/2 === RUN TestCheckSizeOnly/3 === RUN TestCheckSizeOnly/4 === RUN TestCheckSizeOnly/5 === RUN TestCheckSizeOnly/6 === RUN TestCheckSizeOnly/7 --- PASS: TestCheckSizeOnly (6.18s) --- PASS: TestCheckSizeOnly/1 (0.18s) --- PASS: TestCheckSizeOnly/2 (0.18s) --- PASS: TestCheckSizeOnly/3 (0.18s) --- PASS: TestCheckSizeOnly/4 (0.19s) --- PASS: TestCheckSizeOnly/5 (0.18s) --- PASS: TestCheckSizeOnly/6 (0.19s) --- PASS: TestCheckSizeOnly/7 (0.21s) === RUN TestCheckEqualReaders --- PASS: TestCheckEqualReaders (0.00s) === RUN TestParseSumFile run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" 2024/11/14 22:04:40 NOTICE: test.sum: improperly formatted checksum line 4 2024/11/14 22:04:40 NOTICE: test.sum: improperly formatted checksum line 5 2024/11/14 22:04:40 NOTICE: test.sum: improperly formatted checksum line 6 2024/11/14 22:04:40 NOTICE: test.sum: 2 warning(s) suppressed... 2024/11/14 22:04:41 NOTICE: test.sum: improperly formatted checksum line 4 2024/11/14 22:04:41 NOTICE: test.sum: improperly formatted checksum line 5 2024/11/14 22:04:41 NOTICE: test.sum: improperly formatted checksum line 6 2024/11/14 22:04:41 NOTICE: test.sum: 2 warning(s) suppressed... --- PASS: TestParseSumFile (3.44s) === RUN TestCheckSum run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" 2024/11/14 22:04:42 DEBUG : Creating backend with remote "TestB2:rclone-test-lebifox6cidi/data" check_test.go:354: B2 bucket rclone-test-lebifox6cidi path data lacks md5, skipping --- SKIP: TestCheckSum (1.52s) === RUN TestCheckSumDownload run.go:180: Remote "B2 bucket rclone-test-lebifox6cidi", Local "Local file system at /tmp/rclone411355367", Modify Window "1ms" 2024/11/14 22:04:43 DEBUG : Creating backend with remote "TestB2:rclone-test-lebifox6cidi/data" "./operations.test -test.v -test.timeout 1h0m0s -remote TestB2: -list-retries 5 -verbose" - Finished ERROR in 8m8.11831192s (try 3/5): signal: killed: Failed []