"./operations.test -test.v -test.timeout 1h0m0s -remote TestB2: -list-retries 5 -verbose -fast-list" - Starting (try 1/5) 2025/04/14 05:54:51 DEBUG : Creating backend with remote "TestB2:rclone-test-benipat1bobu" 2025/04/14 05:54:51 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/04/14 05:54:51 DEBUG : Creating backend with remote "/tmp/rclone3025274209" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "B2 bucket rclone-test-benipat1bobu", Local "Local file system at /tmp/rclone3025274209", Modify Window "1ms" 2025/04/14 05:54:53 DEBUG : chunksize-probe: Cancelling large file upload === RUN TestMultithreadCopy/upload=false,size=201326591,streams=2 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: write buffer set to 131072 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 3/3 (134217728-201326591) size 64.000Mi starting 2025/04/14 05:55:17 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 0 2025/04/14 05:55:18 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 2 2025/04/14 05:55:18 DEBUG : test-multithread-copy-false-201326591-2: writing chunk 1 2025/04/14 05:55:22 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/04/14 05:55:23 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 3/3 (134217728-201326591) size 64.000Mi finished 2025/04/14 05:55:23 DEBUG : test-multithread-copy-false-201326591-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/04/14 05:55:23 DEBUG : test-multithread-copy-false-201326591-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=201326592,streams=2 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: write buffer set to 131072 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 3/3 (134217728-201326592) size 64Mi starting 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 2 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 0 2025/04/14 05:55:58 DEBUG : test-multithread-copy-false-201326592-2: writing chunk 1 2025/04/14 05:56:03 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 3/3 (134217728-201326592) size 64Mi finished 2025/04/14 05:56:03 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/04/14 05:56:03 DEBUG : test-multithread-copy-false-201326592-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/04/14 05:56:03 DEBUG : test-multithread-copy-false-201326592-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=201326593,streams=2 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: write buffer set to 131072 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: Starting multi-thread copy with 4 chunks of size 64Mi with 4 parallel streams 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 4/4 (201326592-201326593) size 1 starting 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 1/4 (0-67108864) size 64Mi starting 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 2/4 (67108864-134217728) size 64Mi starting 2025/04/14 05:56:38 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 3/4 (134217728-201326592) size 64Mi starting 2025/04/14 05:56:39 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 3 2025/04/14 05:56:39 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 4/4 (201326592-201326593) size 1 finished 2025/04/14 05:56:39 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 1 2025/04/14 05:56:39 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 0 2025/04/14 05:56:39 DEBUG : test-multithread-copy-false-201326593-2: writing chunk 2 2025/04/14 05:56:44 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 2/4 (67108864-134217728) size 64Mi finished 2025/04/14 05:56:44 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 1/4 (0-67108864) size 64Mi finished 2025/04/14 05:56:44 DEBUG : test-multithread-copy-false-201326593-2: multi-thread copy: chunk 3/4 (134217728-201326592) size 64Mi finished 2025/04/14 05:56:44 DEBUG : test-multithread-copy-false-201326593-2: Finished multi-thread copy with 4 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=201326591,streams=2 2025/04/14 05:57:13 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: disabling buffering because source is local disk 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: Starting multi-thread copy with 2 chunks of size 96Mi with 2 parallel streams 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 2/2 (100663296-201326591) size 96.000Mi starting 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: Sending chunk 1 length 100663295 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi starting 2025/04/14 05:57:14 DEBUG : test-multithread-copy-true-201326591-2: Sending chunk 0 length 100663296 2025/04/14 05:57:22 DEBUG : test-multithread-copy-true-201326591-2: Done sending chunk 0 2025/04/14 05:57:22 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi finished 2025/04/14 05:57:23 DEBUG : test-multithread-copy-true-201326591-2: Done sending chunk 1 2025/04/14 05:57:23 DEBUG : test-multithread-copy-true-201326591-2: multi-thread copy: chunk 2/2 (100663296-201326591) size 96.000Mi finished 2025/04/14 05:57:23 DEBUG : test-multithread-copy-true-201326591-2: Finishing large file upload with 2 parts 2025/04/14 05:57:24 DEBUG : test-multithread-copy-true-201326591-2: Finished multi-thread copy with 2 parts of size 96Mi === RUN TestMultithreadCopy/upload=true,size=201326592,streams=2 2025/04/14 05:57:51 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: disabling buffering because source is local disk 2025/04/14 05:57:52 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/04/14 05:57:52 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/04/14 05:57:52 DEBUG : test-multithread-copy-true-201326592-2: Starting multi-thread copy with 2 chunks of size 96Mi with 2 parallel streams 2025/04/14 05:57:52 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 2/2 (100663296-201326592) size 96Mi starting 2025/04/14 05:57:52 DEBUG : test-multithread-copy-true-201326592-2: Sending chunk 1 length 100663296 2025/04/14 05:57:52 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi starting 2025/04/14 05:57:53 DEBUG : test-multithread-copy-true-201326592-2: Sending chunk 0 length 100663296 2025/04/14 05:58:06 DEBUG : test-multithread-copy-true-201326592-2: Done sending chunk 0 2025/04/14 05:58:06 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 1/2 (0-100663296) size 96Mi finished 2025/04/14 05:58:13 DEBUG : test-multithread-copy-true-201326592-2: Done sending chunk 1 2025/04/14 05:58:13 DEBUG : test-multithread-copy-true-201326592-2: multi-thread copy: chunk 2/2 (100663296-201326592) size 96Mi finished 2025/04/14 05:58:13 DEBUG : test-multithread-copy-true-201326592-2: Finishing large file upload with 2 parts 2025/04/14 05:58:13 DEBUG : test-multithread-copy-true-201326592-2: Finished multi-thread copy with 2 parts of size 96Mi === RUN TestMultithreadCopy/upload=true,size=201326593,streams=2 2025/04/14 05:58:42 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: disabling buffering because source is local disk 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: Starting multi-thread copy with 3 chunks of size 96Mi with 3 parallel streams 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 starting 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 2 length 1 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi starting 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi starting 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 1 length 100663296 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: Sending chunk 0 length 100663296 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 2 2025/04/14 05:58:43 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 finished 2025/04/14 05:58:51 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 1 2025/04/14 05:58:51 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi finished 2025/04/14 05:58:51 DEBUG : test-multithread-copy-true-201326593-2: Done sending chunk 0 2025/04/14 05:58:51 DEBUG : test-multithread-copy-true-201326593-2: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi finished 2025/04/14 05:58:51 DEBUG : test-multithread-copy-true-201326593-2: Finishing large file upload with 3 parts 2025/04/14 05:58:52 DEBUG : test-multithread-copy-true-201326593-2: Finished multi-thread copy with 3 parts of size 96Mi --- PASS: TestMultithreadCopy (250.49s) --- PASS: TestMultithreadCopy/upload=false,size=201326591,streams=2 (40.40s) --- PASS: TestMultithreadCopy/upload=false,size=201326592,streams=2 (40.73s) --- PASS: TestMultithreadCopy/upload=false,size=201326593,streams=2 (40.33s) --- PASS: TestMultithreadCopy/upload=true,size=201326591,streams=2 (39.45s) --- PASS: TestMultithreadCopy/upload=true,size=201326592,streams=2 (50.24s) --- PASS: TestMultithreadCopy/upload=true,size=201326593,streams=2 (37.66s) === RUN TestMultithreadCopyAbort run.go:180: Remote "B2 bucket rclone-test-benipat1bobu", Local "Local file system at /tmp/rclone3025274209", Modify Window "1ms" 2025/04/14 05:59:02 DEBUG : chunksize-probe: Cancelling large file upload 2025/04/14 05:59:21 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because source is local disk 2025/04/14 05:59:22 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2025/04/14 05:59:22 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/04/14 05:59:22 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 96Mi with 3 parallel streams 2025/04/14 05:59:22 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (201326592-201326593) size 1 starting 2025/04/14 05:59:22 DEBUG : Open with options = [RangeOption(201326592,201326592)] 2025/04/14 05:59:22 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi starting 2025/04/14 05:59:22 DEBUG : Open with options = [RangeOption(0,100663295)] 2025/04/14 05:59:22 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (100663296-201326592) size 96Mi starting 2025/04/14 05:59:22 DEBUG : Open with options = [RangeOption(100663296,201326591)] 2025/04/14 05:59:22 DEBUG : test-multithread-abort: Sending chunk 0 length 100663296 2025/04/14 05:59:22 DEBUG : test-multithread-abort: Sending chunk 1 length 100663296 2025/04/14 05:59:30 DEBUG : test-multithread-abort: Done sending chunk 0 2025/04/14 05:59:30 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-100663296) size 96Mi finished 2025/04/14 06:00:43 DEBUG : test-multithread-abort: Error sending chunk 1 (retry=true): The server did not receive a complete request message within the time that it was prepared to wait (408 request_timeout): &api.Error{Status:408, Code:"request_timeout", Message:"The server did not receive a complete request message within the time that it was prepared to wait"} 2025/04/14 06:00:43 DEBUG : test-multithread-abort: Clearing part upload URL because of error: The server did not receive a complete request message within the time that it was prepared to wait (408 request_timeout) 2025/04/14 06:00:43 DEBUG : pacer: low level retry 1/10 (error The server did not receive a complete request message within the time that it was prepared to wait (408 request_timeout)) 2025/04/14 06:00:43 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/04/14 06:00:43 DEBUG : test-multithread-abort: Sending chunk 1 length 100663296 2025/04/14 06:00:44 DEBUG : test-multithread-abort: Seek from 163840 to 0 2025/04/14 06:00:44 DEBUG : Open with options = [RangeOption(100663296,201326591)] panic: sync: WaitGroup is reused before previous Wait has returned goroutine 786 [running]: sync.(*WaitGroup).Wait(0x3b9aca00?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/sync/waitgroup.go:120 +0x74 github.com/rclone/rclone/fs/operations.errorObject.Open({{0x2a28e70?, 0xc00460c0c0?}, 0x475ad8?, 0xc001a97290?}, {0x2a13758, 0xc00049a140}, {0xc000068650, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:262 +0x179 github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc00026e1c0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:147 +0x109 github.com/rclone/rclone/fs/operations.NewReOpen({0x2a13758, 0xc00049a140}, {0x2a270b0, 0xc000216060}, 0xa, {0xc000068640, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:102 +0x50b github.com/rclone/rclone/fs/operations.Open({0x2a13758, 0xc00049a140}, {0x2a270b0, 0xc000216060}, {0xc000068640, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:124 +0x6b github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc00049a190, {0x2a13758, 0xc00049a140}, 0x2, {0x2a0db60, 0xc00268a0c0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:81 +0x2df github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:225 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:78 +0x50 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 728 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:75 +0x96 "./operations.test -test.v -test.timeout 1h0m0s -remote TestB2: -list-retries 5 -verbose -fast-list" - Finished ERROR in 5m53.042262166s (try 1/5): exit status 2: Failed []