"./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Starting (try 1/5) 2025/03/04 07:12:59 DEBUG : Creating backend with remote "TestPcloud:rclone-test-pixujuk4vowe" 2025/03/04 07:12:59 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/03/04 07:13:00 DEBUG : Creating backend with remote "/tmp/rclone2176294063" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "pcloud root 'rclone-test-pixujuk4vowe'", Local "Local file system at /tmp/rclone2176294063", Modify Window "1s" === RUN TestMultithreadCopy/upload=false,size=134217727,streams=2 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: write buffer set to 131072 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/03/04 07:13:08 DEBUG : Couldn't parse Date: from server edef9.pcloud.com: "Tue, 04 Mar 2025 07:13:08 +0000": parsing time "Tue, 04 Mar 2025 07:13:08 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:08 +0000" as " " 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 1 2025/03/04 07:13:08 DEBUG : Couldn't parse Date: from server edef9.pcloud.com: "Tue, 04 Mar 2025 07:13:08 +0000": parsing time "Tue, 04 Mar 2025 07:13:08 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:08 +0000" as " " 2025/03/04 07:13:08 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 0 2025/03/04 07:13:10 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2025/03/04 07:13:10 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/03/04 07:13:10 DEBUG : test-multithread-copy-false-134217727-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217728,streams=2 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: write buffer set to 131072 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/03/04 07:13:22 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Tue, 04 Mar 2025 07:13:22 +0000": parsing time "Tue, 04 Mar 2025 07:13:22 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:22 +0000" as " " 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 0 2025/03/04 07:13:22 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Tue, 04 Mar 2025 07:13:22 +0000": parsing time "Tue, 04 Mar 2025 07:13:22 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:22 +0000" as " " 2025/03/04 07:13:22 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 1 2025/03/04 07:13:24 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/03/04 07:13:24 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2025/03/04 07:13:24 DEBUG : test-multithread-copy-false-134217728-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217729,streams=2 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: write buffer set to 131072 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/03/04 07:13:38 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Tue, 04 Mar 2025 07:13:38 +0000": parsing time "Tue, 04 Mar 2025 07:13:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:38 +0000" as " " 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 2 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2025/03/04 07:13:38 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Tue, 04 Mar 2025 07:13:38 +0000": parsing time "Tue, 04 Mar 2025 07:13:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:38 +0000" as " " 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 1 2025/03/04 07:13:38 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Tue, 04 Mar 2025 07:13:38 +0000": parsing time "Tue, 04 Mar 2025 07:13:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 04 Mar 2025 07:13:38 +0000" as " " 2025/03/04 07:13:38 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 0 2025/03/04 07:13:40 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/03/04 07:13:40 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/03/04 07:13:40 DEBUG : test-multithread-copy-false-134217729-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=134217727,streams=2 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: write buffer set to 131072 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 1 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/03/04 07:13:54 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 0 2025/03/04 07:23:30 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/03/04 07:23:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/03/04 07:23:30 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/03/04 07:23:36 DEBUG : pacer: Reducing sleep to 15ms 2025/03/04 07:23:36 DEBUG : pacer: Reducing sleep to 11.25ms 2025/03/04 07:23:36 DEBUG : pacer: Reducing sleep to 10ms 2025/03/04 07:23:36 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 failed: multi-thread copy: failed to write chunk: context canceled 2025/03/04 07:23:36 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: cancelling transfer on exit 2025/03/04 07:23:36 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 0/5 2025/03/04 07:23:37 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 1/5 2025/03/04 07:23:38 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 2/5 2025/03/04 07:23:39 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 3/5 2025/03/04 07:23:41 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 4/5 2025/03/04 07:23:42 ERROR : test-multithread-copy-true-134217727-2: multi-thread copy: failed to close file before aborting: incorrect size after upload: got 119144448, want 134217727 multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217728,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: write buffer set to 131072 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 1 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/03/04 07:23:58 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 0 2025/03/04 07:46:43 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2025/03/04 07:46:49 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/03/04 07:46:49 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 0/5 2025/03/04 07:46:49 ERROR : test-multithread-copy-true-134217728-2: multi-thread copy: can't set metadata as SetMetadata isn't implemented in: pcloud root 'rclone-test-pixujuk4vowe' 2025/03/04 07:46:49 DEBUG : test-multithread-copy-true-134217728-2: Finished multi-thread copy with 2 parts of size 64Mi fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:224 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:224 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" fstest.go:121: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:121 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:126 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:145 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:189 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:225 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: test-multithread-copy-true-134217728-2: Modification time difference too big |-211083h41m48.500000001s| > 1s (want 2001-02-03 04:05:06.499999999 +0000 UTC vs got 2025-03-04 07:46:55 +0000 UTC) (precision 1s) === RUN TestMultithreadCopy/upload=true,size=134217729,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217729-2 (134217729) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217729-2 (134217729) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: write buffer set to 131072 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 0 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 2 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 1 2025/03/04 07:47:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2025/03/04 08:08:10 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/03/04 08:08:10 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/03/04 08:08:10 DEBUG : test-multithread-copy-true-134217729-2: checking file size: try 0/5 2025/03/04 08:08:10 DEBUG : test-multithread-copy-true-134217729-2: Finished multi-thread copy with 3 parts of size 64Mi fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:224 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217729-2 (134217729) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217729-2 (134217729) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:224 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" fstest.go:121: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:121 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:126 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:145 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:189 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:225 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: test-multithread-copy-true-134217729-2: Modification time difference too big |-211084h3m11.500000001s| > 1s (want 2001-02-03 04:05:06.499999999 +0000 UTC vs got 2025-03-04 08:08:18 +0000 UTC) (precision 1s) --- FAIL: TestMultithreadCopy (3322.35s) --- PASS: TestMultithreadCopy/upload=false,size=134217727,streams=2 (14.70s) --- PASS: TestMultithreadCopy/upload=false,size=134217728,streams=2 (14.33s) --- PASS: TestMultithreadCopy/upload=false,size=134217729,streams=2 (15.83s) --- FAIL: TestMultithreadCopy/upload=true,size=134217727,streams=2 (596.91s) --- FAIL: TestMultithreadCopy/upload=true,size=134217728,streams=2 (1399.47s) --- FAIL: TestMultithreadCopy/upload=true,size=134217729,streams=2 (1280.94s) === RUN TestMultithreadCopyAbort run.go:180: Remote "pcloud root 'rclone-test-pixujuk4vowe'", Local "Local file system at /tmp/rclone2176294063", Modify Window "1s" 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: write buffer set to 131072 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/03/04 08:08:32 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/03/04 08:08:32 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/03/04 08:08:32 DEBUG : Open with options = [RangeOption(0,67108863)] 2025/03/04 08:08:32 DEBUG : test-multithread-abort: writing chunk 0 2025/03/04 08:08:32 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/03/04 08:08:32 DEBUG : Open with options = [RangeOption(67108864,134217727)] 2025/03/04 08:08:32 DEBUG : test-multithread-abort: writing chunk 1 panic: test timed out after 1h0m0s running tests: TestMultithreadCopyAbort (4m38s) goroutine 49497 [running]: testing.(*M).startAlarm.func1() /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:2373 +0x385 created by time.goFunc /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 4 minutes]: testing.(*T).Run(0xc00016a000, {0x260dc1e?, 0x0?}, 0x2725c48) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:1751 +0x3ab testing.runTests.func1(0xc00016a000) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:2168 +0x37 testing.tRunner(0xc00016a000, 0xc000b4bc18) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:1690 +0xf4 testing.runTests(0xc000b1c630, {0x3d066a0, 0x6c, 0x6c}, {0x3d1ae20?, 0x1?, 0x3d1ae20?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:2166 +0x43d testing.(*M).Run(0xc00033cfa0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:2034 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc00033cfa0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:263 +0xa9 goroutine 27 [syscall, 60 minutes]: os/signal.signal_recv() /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/os/signal/signal.go:151 +0x1f goroutine 28 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 29 [select, 60 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc000410880) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 37931 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc000b30fc0) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:339 +0x13d created by github.com/rclone/rclone/fs/accounting.(*StatsInfo)._startAverageLoop in goroutine 37938 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:404 +0xb6 goroutine 37938 [semacquire, 4 minutes]: sync.runtime_Semacquire(0xc00057c708?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/sema.go:71 +0x25 sync.(*WaitGroup).Wait(0x23df2e0?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/sync/waitgroup.go:118 +0x48 golang.org/x/sync/errgroup.(*Group).Wait(0xc000410240) /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:56 +0x25 github.com/rclone/rclone/fs/operations.multiThreadCopy({0x2a02460, 0x3d3e300}, {0x2a18260, 0xc00033d040}, {0x2606b55, 0x16}, {0x2a15e10, 0xc00019cce0}, 0x1, 0xc00019f440, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:229 +0xe3f github.com/rclone/rclone/fs/operations.TestMultithreadCopyAbort(0xc0003e4000) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:322 +0x669 testing.tRunner(0xc0003e4000, 0x2725c48) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:1690 +0xf4 created by testing.(*T).Run in goroutine 1 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/testing/testing.go:1743 +0x390 goroutine 49471 [select]: net/http.(*persistConn).writeLoop(0xc0009785a0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2519 +0xe7 created by net/http.(*Transport).dialConn in goroutine 49456 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:1875 +0x15a5 goroutine 136 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 67 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 49495 [IO wait]: internal/poll.runtime_pollWait(0x7f9759aa6468, 0x72) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc000d85300?, 0xc000b73980?, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000d85300, {0xc000b73980, 0x1980, 0x1980}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc000d85300, {0xc000b73980?, 0xc000b73980?, 0x2?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000614670, {0xc000b73980?, 0x3d1ae20?, 0xc000a099d0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/net.go:189 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000a0ef48, {0xc000b73980?, 0x18?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc01fa77ab8, {0xc000b73980?, 0x0?, 0xc01fa77ab8?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:809 +0x3b bytes.(*Buffer).ReadFrom(0xc00084ad38, {0x29dc780, 0xc01fa77ab8}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc00084aa88, {0x7f9743ff2e18, 0xc000a0ef48}, 0x10000001980?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:831 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc00084aa88, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:629 +0x3cf crypto/tls.(*Conn).readRecord(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:591 crypto/tls.(*Conn).Read(0xc00084aa88, {0xc000daa000, 0x1000, 0x30?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:1385 +0x150 net/http.(*persistConn).Read(0xc000d38000, {0xc000daa000?, 0x40a8a5?, 0x68?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2052 +0x4a bufio.(*Reader).fill(0xc0009d8120) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc0009d8120, 0x1) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc000d38000) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2205 +0x185 created by net/http.(*Transport).dialConn in goroutine 49473 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:1874 +0x154f goroutine 49470 [IO wait]: internal/poll.runtime_pollWait(0x7f9759aa6678, 0x72) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc00034d280?, 0xc000b72000?, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00034d280, {0xc000b72000, 0x1980, 0x1980}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc00034d280, {0xc000b72000?, 0xc000b72000?, 0x2?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000614618, {0xc000b72000?, 0x3d1ae20?, 0xc000c0d9d0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/net.go:189 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000a0ebe8, {0xc000b72000?, 0x18?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc01fa77a88, {0xc000b72000?, 0x0?, 0xc01fa77a88?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:809 +0x3b bytes.(*Buffer).ReadFrom(0xc00084a9b8, {0x29dc780, 0xc01fa77a88}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc00084a708, {0x7f9743ff2e18, 0xc000a0ebe8}, 0x10000001980?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:831 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc00084a708, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:629 +0x3cf crypto/tls.(*Conn).readRecord(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:591 crypto/tls.(*Conn).Read(0xc00084a708, {0xc000d36000, 0x1000, 0x30?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/crypto/tls/conn.go:1385 +0x150 net/http.(*persistConn).Read(0xc0009785a0, {0xc000d36000?, 0x40a8a5?, 0x68?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2052 +0x4a bufio.(*Reader).fill(0xc01a6eed80) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc01a6eed80, 0x1) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc0009785a0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2205 +0x185 created by net/http.(*Transport).dialConn in goroutine 49456 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:1874 +0x154f goroutine 37985 [semacquire, 4 minutes]: sync.runtime_Semacquire(0xc000376d10?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/sema.go:71 +0x25 sync.(*WaitGroup).Wait(0x3b9aca00?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/sync/waitgroup.go:118 +0x48 github.com/rclone/rclone/fs/operations.errorObject.Open({{0x2a17bd0?, 0xc00154cd20?}, 0x475ad8?, 0xc001657a10?}, {0x2a02578, 0xc0000be1e0}, {0xc000376d10, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:262 +0x179 github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc0001fa1c0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:147 +0x109 github.com/rclone/rclone/fs/operations.NewReOpen({0x2a02578, 0xc0000be1e0}, {0x2a15e10, 0xc00019cce0}, 0xa, {0xc000376d00, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:102 +0x50b github.com/rclone/rclone/fs/operations.Open({0x2a02578, 0xc0000be1e0}, {0x2a15e10, 0xc00019cce0}, {0xc000376d00, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:124 +0x6b github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0000be230, {0x2a02578, 0xc0000be1e0}, 0x2, {0x29fb1d8, 0xc000cfe300}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:81 +0x2df github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:225 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:78 +0x50 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 37938 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:75 +0x96 goroutine 37986 [chan receive, 4 minutes]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000410440) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 37985 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 37983 [select]: net/http.(*persistConn).roundTrip(0xc000d38000, 0xc0000bfc20) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2755 +0x7b0 net/http.(*Transport).roundTrip(0xc000d0ad80, 0xc000b56c80) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:644 +0xb91 net/http.(*Transport).RoundTrip(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/roundtrip.go:30 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0003f5960, 0xc000b56c80) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:349 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc0005e96c0, 0xc000b56b40) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.27.0/transport.go:55 +0x3de net/http.send(0xc000b56b40, {0x29de6c0, 0xc0005e96c0}, {0xc001d96e01?, 0x41790b?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc0006367e0, 0xc000b56b40, {0xc000060887?, 0xa?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc0006367e0, 0xc000b56b40) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:725 +0x8bc net/http.(*Client).Do(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc0000bfb80, {0x2a02460, 0x3d3e300}, 0xc000b56a00) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:353 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc0000bfb80, {0x2a02460, 0x3d3e300}, 0x41790b?, {0x0?, 0x0?}, {0x21280a0, 0xc0006368a0}, 0x41790b?, 0x2727188, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:550 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:493 github.com/rclone/rclone/backend/pcloud.fileOpen.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:153 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc0006368d0?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000385e00, 0xc0006368d0, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileOpen({0x2a02460, 0x3d3e300}, 0xc0000bfb80, 0xc00033d040, 0xe54653dbf) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:152 +0x3a7 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc000410200, {0xc001e94000, 0x20000, 0x20000}, 0x3740000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:73 +0x187 io.(*OffsetWriter).Write(0xc00019d1c0, {0xc001e94000?, 0xa967c0?, 0xc001d8c010?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc0000b2280) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc0000b2280, {0x29dac60, 0xc001d8c000}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x29dac20, 0xc0000b2280}, {0x29dac60, 0xc001d8c000}, {0x0, 0x0, 0x0}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/io/io.go:415 +0x151 io.Copy(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc000cfe300, {0xc0000be1e0?, 0x2a15e10?}, 0x0, {0x29ede40, 0xc001d8c000}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:305 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0000be230, {0x2a02578, 0xc0000be1e0}, 0x0, {0x29fb1d8, 0xc000cfe300}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:107 +0x703 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:225 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:78 +0x50 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 37938 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:75 +0x96 goroutine 37984 [select]: net/http.(*persistConn).roundTrip(0xc0009785a0, 0xc0001a1d60) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2755 +0x7b0 net/http.(*Transport).roundTrip(0xc000a9f200, 0xc000cbe3c0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:644 +0xb91 net/http.(*Transport).RoundTrip(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/roundtrip.go:30 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc000344af0, 0xc000cbe3c0) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:349 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc00048b780, 0xc000cbe280) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.27.0/transport.go:55 +0x3de net/http.send(0xc000cbe280, {0x29de6c0, 0xc00048b780}, {0xc001e02e01?, 0x4194e8?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc000a31560, 0xc000cbe280, {0x0?, 0xa?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc000a31560, 0xc000cbe280) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:725 +0x8bc net/http.(*Client).Do(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc0001a1cc0, {0x2a02460, 0x3d3e300}, 0xc000cbe140) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:353 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc0001a1cc0, {0x2a02460, 0x3d3e300}, 0x41790b?, {0x0?, 0x0?}, {0x21280a0, 0xc000a31620}, 0x41790b?, 0x2727188, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:550 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:493 github.com/rclone/rclone/backend/pcloud.fileOpen.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:153 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc000a31650?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000385e00, 0xc000a31650, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileOpen({0x2a02460, 0x3d3e300}, 0xc0001a1cc0, 0xc00033d040, 0xe54653dbf) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:152 +0x3a7 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc000410200, {0xc001ec6000, 0x20000, 0x20000}, 0x7700000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:73 +0x187 io.(*OffsetWriter).Write(0xc000072160, {0xc001ec6000?, 0xa967c0?, 0xc0001fa2b0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000410540) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000410540, {0x29dac60, 0xc0001fa2a0}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x29dac20, 0xc000410540}, {0x29dac60, 0xc0001fa2a0}, {0x0, 0x0, 0x0}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/io/io.go:415 +0x151 io.Copy(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc000cfe300, {0xc0000be1e0?, 0x2a15e10?}, 0x1, {0x29ede40, 0xc0001fa2a0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:305 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0000be230, {0x2a02578, 0xc0000be1e0}, 0x1, {0x29fb1d8, 0xc000cfe300}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:107 +0x703 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:225 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:78 +0x50 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 37938 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.11.0/errgroup/errgroup.go:75 +0x96 goroutine 49496 [select]: net/http.(*persistConn).writeLoop(0xc000d38000) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:2519 +0xe7 created by net/http.(*Transport).dialConn in goroutine 49473 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/transport.go:1875 +0x15a5 goroutine 37937 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc0000b2240) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 37983 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 37982 [select]: github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc00075c000) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:221 +0xed created by github.com/rclone/rclone/fs/accounting.newAccountSizeName in goroutine 37938 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:119 +0x438 goroutine 37987 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000410480) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 37984 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f "./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Finished ERROR in 1h0m0.581485033s (try 1/5): exit status 2: Failed [TestMultithreadCopy/upload=true,size=134217727,streams=2 TestMultithreadCopy/upload=true,size=134217728,streams=2 TestMultithreadCopy/upload=true,size=134217729,streams=2]