"./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Starting (try 1/5) 2025/01/12 02:20:57 DEBUG : Creating backend with remote "TestPcloud:rclone-test-pakukoz7huso" 2025/01/12 02:20:57 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/12 02:20:58 DEBUG : Creating backend with remote "/tmp/rclone1834897841" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "pcloud root 'rclone-test-pakukoz7huso'", Local "Local file system at /tmp/rclone1834897841", Modify Window "1s" === RUN TestMultithreadCopy/upload=false,size=134217727,streams=2 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: write buffer set to 131072 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/12 02:21:07 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Sun, 12 Jan 2025 02:21:07 +0000": parsing time "Sun, 12 Jan 2025 02:21:07 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:07 +0000" as " " 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 1 2025/01/12 02:21:07 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Sun, 12 Jan 2025 02:21:07 +0000": parsing time "Sun, 12 Jan 2025 02:21:07 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:07 +0000" as " " 2025/01/12 02:21:07 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 0 2025/01/12 02:21:09 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2025/01/12 02:21:09 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/01/12 02:21:09 DEBUG : test-multithread-copy-false-134217727-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217728,streams=2 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: write buffer set to 131072 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/12 02:21:22 DEBUG : Couldn't parse Date: from server edef3.pcloud.com: "Sun, 12 Jan 2025 02:21:22 +0000": parsing time "Sun, 12 Jan 2025 02:21:22 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:22 +0000" as " " 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 1 2025/01/12 02:21:22 DEBUG : Couldn't parse Date: from server edef3.pcloud.com: "Sun, 12 Jan 2025 02:21:22 +0000": parsing time "Sun, 12 Jan 2025 02:21:22 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:22 +0000" as " " 2025/01/12 02:21:22 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 0 2025/01/12 02:21:24 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/01/12 02:21:24 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2025/01/12 02:21:24 DEBUG : test-multithread-copy-false-134217728-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217729,streams=2 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: write buffer set to 131072 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/12 02:21:38 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Sun, 12 Jan 2025 02:21:38 +0000": parsing time "Sun, 12 Jan 2025 02:21:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:38 +0000" as " " 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 2 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2025/01/12 02:21:38 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Sun, 12 Jan 2025 02:21:38 +0000": parsing time "Sun, 12 Jan 2025 02:21:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:38 +0000" as " " 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 0 2025/01/12 02:21:38 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Sun, 12 Jan 2025 02:21:38 +0000": parsing time "Sun, 12 Jan 2025 02:21:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 12 Jan 2025 02:21:38 +0000" as " " 2025/01/12 02:21:38 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 1 2025/01/12 02:21:40 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/01/12 02:21:40 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/01/12 02:21:40 DEBUG : test-multithread-copy-false-134217729-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=134217727,streams=2 2025/01/12 02:21:53 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: write buffer set to 131072 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 1 2025/01/12 02:21:54 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 0 2025/01/12 02:50:27 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/12 02:50:27 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/12 02:50:27 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/12 02:50:31 DEBUG : pacer: Reducing sleep to 15ms 2025/01/12 02:50:31 DEBUG : pacer: Reducing sleep to 11.25ms 2025/01/12 02:50:31 DEBUG : pacer: Reducing sleep to 10ms 2025/01/12 02:50:31 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 failed: multi-thread copy: failed to write chunk: context canceled 2025/01/12 02:50:31 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: cancelling transfer on exit 2025/01/12 02:50:31 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 0/5 2025/01/12 02:50:32 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 1/5 2025/01/12 02:50:33 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 2/5 2025/01/12 02:50:34 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 3/5 2025/01/12 02:50:35 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 4/5 2025/01/12 02:50:37 ERROR : test-multithread-copy-true-134217727-2: multi-thread copy: failed to close file before aborting: incorrect size after upload: got 130285568, want 134217727 multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217728,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: write buffer set to 131072 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 1 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/12 02:50:54 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 0 2025/01/12 03:17:13 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/12 03:17:13 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/12 03:17:13 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/12 03:17:18 DEBUG : pacer: Reducing sleep to 15ms 2025/01/12 03:17:18 DEBUG : pacer: Reducing sleep to 11.25ms 2025/01/12 03:17:18 DEBUG : pacer: Reducing sleep to 10ms 2025/01/12 03:17:18 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 failed: multi-thread copy: failed to write chunk: context canceled 2025/01/12 03:17:18 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: cancelling transfer on exit 2025/01/12 03:17:18 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 0/5 2025/01/12 03:17:19 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 1/5 2025/01/12 03:17:20 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 2/5 2025/01/12 03:17:21 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 3/5 2025/01/12 03:17:22 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 4/5 2025/01/12 03:17:23 ERROR : test-multithread-copy-true-134217728-2: multi-thread copy: failed to close file before aborting: incorrect size after upload: got 122683392, want 134217728 multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217729,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217729-2 (134217729) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728), test-multithread-copy-true-134217729-2 (134217729) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217728-2" 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: write buffer set to 131072 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 0 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 2 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 1 2025/01/12 03:17:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2025/01/12 03:19:48 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/12 03:19:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/12 03:19:48 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/12 03:19:54 DEBUG : pacer: Reducing sleep to 15ms 2025/01/12 03:19:54 DEBUG : pacer: Reducing sleep to 11.25ms 2025/01/12 03:19:54 DEBUG : pacer: Reducing sleep to 10ms 2025/01/12 03:19:54 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 failed: multi-thread copy: failed to write chunk: context canceled 2025/01/12 03:19:54 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: cancelling transfer on exit 2025/01/12 03:19:54 DEBUG : test-multithread-copy-true-134217729-2: checking file size: try 0/5 multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 --- FAIL: TestMultithreadCopy (3536.53s) --- PASS: TestMultithreadCopy/upload=false,size=134217727,streams=2 (15.98s) --- PASS: TestMultithreadCopy/upload=false,size=134217728,streams=2 (14.76s) --- PASS: TestMultithreadCopy/upload=false,size=134217729,streams=2 (15.89s) --- FAIL: TestMultithreadCopy/upload=true,size=134217727,streams=2 (1732.27s) --- FAIL: TestMultithreadCopy/upload=true,size=134217728,streams=2 (1606.69s) --- FAIL: TestMultithreadCopy/upload=true,size=134217729,streams=2 (150.77s) === RUN TestMultithreadCopyAbort run.go:180: Remote "pcloud root 'rclone-test-pakukoz7huso'", Local "Local file system at /tmp/rclone1834897841", Modify Window "1s" 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: write buffer set to 131072 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/12 03:20:04 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/12 03:20:04 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/12 03:20:04 DEBUG : Open with options = [RangeOption(0,67108863)] 2025/01/12 03:20:04 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/12 03:20:04 DEBUG : Open with options = [RangeOption(67108864,134217727)] 2025/01/12 03:20:04 DEBUG : test-multithread-abort: writing chunk 0 2025/01/12 03:20:04 DEBUG : test-multithread-abort: writing chunk 1 panic: test timed out after 1h0m0s running tests: TestMultithreadCopyAbort (1m3s) goroutine 26176 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2366 +0x385 created by time.goFunc /usr/local/go/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive, 1 minutes]: testing.(*T).Run(0xc0004201a0, {0x24549a3?, 0x0?}, 0x25614f0) /usr/local/go/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc0004201a0) /usr/local/go/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc0004201a0, 0xc000631c00) /usr/local/go/src/testing/testing.go:1689 +0xfb testing.runTests(0xc0002924f8, {0x3a13f40, 0x6c, 0x6c}, {0x1?, 0xedf151e0a?, 0x3a27be0?}) /usr/local/go/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc0004db860) /usr/local/go/src/testing/testing.go:2027 +0x68b github.com/rclone/rclone/fstest.TestMain(0xc0004db860) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0x196 goroutine 23006 [select]: net/http.(*persistConn).roundTrip(0xc0008aa7e0, 0xc000ad99c0) /usr/local/go/src/net/http/transport.go:2674 +0x979 net/http.(*Transport).roundTrip(0xc0008a6640, 0xc000a7c7e0) /usr/local/go/src/net/http/transport.go:608 +0x79a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0005c8690, 0xc000a7c7e0) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc01faef620, 0xc000a7c6c0) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc000a7c6c0, {0x2806280, 0xc01faef620}, {0xc001be4d01?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc02050e360, 0xc000a7c6c0, {0x413fdb?, 0x1?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc02050e360, 0xc000a7c6c0) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc01a3986e0, {0x2828380, 0x3a8ad60}, 0xc01a487cc0) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:356 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc01a3986e0, {0x2828380, 0x3a8ad60}, 0xc001be56b0?, {0x0?, 0x0?}, {0x1f97b40, 0xc0005172a0}, 0x30?, 0x25629a0, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:553 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:496 github.com/rclone/rclone/backend/pcloud.filePWrite.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:224 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc002063590?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000290d80, 0xc002063590, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.filePWrite({0x2828380, 0x3a8ad60}, 0xc01a3986e0, 0xc0002960a0, 0x1, 0x10a0000, {0xc001d00000, 0x20000, 0x20000}) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:223 +0x4af github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc01a40a8c0, {0xc001d00000, 0x20000, 0x20000}, 0x10a0000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:95 +0x354 io.(*OffsetWriter).Write(0xc000428080, {0xc001d00000?, 0xa47780?, 0xc0005a4010?}) /usr/local/go/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000ad8100) /usr/local/go/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000ad8100, {0x2802b60, 0xc0005a4000}) /usr/local/go/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x2802b20, 0xc000ad8100}, {0x2802b60, 0xc0005a4000}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:415 +0x151 io.Copy(...) /usr/local/go/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc000d0eae0, {0xc01a2bc5a0?, 0x283a988?}, 0x0, {0x2814f68, 0xc0005a4000}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc01a2bc5f0, {0x2828498, 0xc01a2bc5a0}, 0x0, {0x2821660, 0xc000d0eae0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 22922 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 goroutine 35 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 36 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 37 [select, 60 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc00051fdc0) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 22922 [semacquire, 1 minutes]: sync.runtime_Semacquire(0xc0005a6978?) /usr/local/go/src/runtime/sema.go:62 +0x25 sync.(*WaitGroup).Wait(0x2238300?) /usr/local/go/src/sync/waitgroup.go:116 +0x48 golang.org/x/sync/errgroup.(*Group).Wait(0xc01a40a900) /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:56 +0x25 github.com/rclone/rclone/fs/operations.multiThreadCopy({0x2828380, 0x3a8ad60}, {0x283cdd8, 0xc000524320}, {0x244e103, 0x16}, {0x283a988, 0xc0003b8000}, 0x1, 0xc000000000, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:232 +0xe25 github.com/rclone/rclone/fs/operations.TestMultithreadCopyAbort(0xc01a46a1a0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:322 +0x68a testing.tRunner(0xc01a46a1a0, 0x25614f0) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 23008 [semacquire, 1 minutes]: sync.runtime_Semacquire(0xc000468ea0?) /usr/local/go/src/runtime/sema.go:62 +0x25 sync.(*WaitGroup).Wait(0x3b9aca00?) /usr/local/go/src/sync/waitgroup.go:116 +0x48 github.com/rclone/rclone/fs/operations.errorObject.Open({{0x283c748?, 0xc000774000?}, 0x41b178?, 0xc00088f620?}, {0x2828498, 0xc01a2bc5a0}, {0xc000468ea0, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:262 +0x179 github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc00017c9a0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:147 +0x109 github.com/rclone/rclone/fs/operations.NewReOpen({0x2828498, 0xc01a2bc5a0}, {0x283a988, 0xc0003b8000}, 0xa, {0xc000468e70, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:102 +0x50b github.com/rclone/rclone/fs/operations.Open({0x2828498, 0xc01a2bc5a0}, {0x283a988, 0xc0003b8000}, {0xc000468e70, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/reopen.go:124 +0x6b github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc01a2bc5f0, {0x2828498, 0xc01a2bc5a0}, 0x2, {0x2821660, 0xc000d0eae0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:84 +0x2ff github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 22922 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 goroutine 26171 [IO wait]: internal/poll.runtime_pollWait(0x7f5844206ef8, 0x72) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc01f874800?, 0xc020203300?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc01f874800, {0xc020203300, 0x1980, 0x1980}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc01f874800, {0xc020203300?, 0xc000a40850?, 0x41cc25?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc01e5d8230, {0xc020203300?, 0x7f583c360df0?, 0xc01fc64de0?}) /usr/local/go/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc01fc64a08, {0xc020203300?, 0x0?, 0xa40930?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc01fc64de0, {0xc020203300?, 0x0?, 0xc01fc64de0?}) /usr/local/go/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc01a17c2b0, {0x2804440, 0xc01fc64de0}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc01a17c008, {0x7f583c638b18, 0xc01fc64a08}, 0xc000a40978?) /usr/local/go/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc01a17c008, 0x0) /usr/local/go/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).Read(0xc01a17c008, {0xc00067b000, 0x1000, 0x8?}) /usr/local/go/src/crypto/tls/conn.go:1370 +0x156 net/http.(*persistConn).Read(0xc0008aa7e0, {0xc00067b000?, 0xc0009a6120?, 0xc000a40d38?}) /usr/local/go/src/net/http/transport.go:1976 +0x4a bufio.(*Reader).fill(0xc000241500) /usr/local/go/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc000241500, 0x1) /usr/local/go/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc0008aa7e0) /usr/local/go/src/net/http/transport.go:2140 +0x1b9 created by net/http.(*Transport).dialConn in goroutine 26217 /usr/local/go/src/net/http/transport.go:1798 +0x152f goroutine 134 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 60 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 26140 [select]: net/http.(*persistConn).readLoop(0xc0008aac60) /usr/local/go/src/net/http/transport.go:2260 +0xd3a created by net/http.(*Transport).dialConn in goroutine 26223 /usr/local/go/src/net/http/transport.go:1798 +0x152f goroutine 23007 [runnable]: net/http.(*persistConn).roundTrip(0xc0008aac60, 0xc000ad98c0) /usr/local/go/src/net/http/transport.go:2674 +0x979 net/http.(*Transport).roundTrip(0xc0001f17c0, 0xc000a7c480) /usr/local/go/src/net/http/transport.go:608 +0x79a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc00040b880, 0xc000a7c480) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc000462c20, 0xc000a7c360) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc000a7c360, {0x2806280, 0xc000462c20}, {0xc001be8d01?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc01a599f20, 0xc000a7c360, {0x413fdb?, 0x1?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc01a599f20, 0xc000a7c360) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc01a2bcc80, {0x2828380, 0x3a8ad60}, 0xc01a487a40) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:356 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc01a2bcc80, {0x2828380, 0x3a8ad60}, 0x443bae?, {0x0?, 0x0?}, {0x1f97ae0, 0xc01fff1360}, 0xc01bd02780?, 0x25629a0, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:553 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:496 github.com/rclone/rclone/backend/pcloud.fileChecksum.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:186 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc0020632c0?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000290d80, 0xc0020632c0, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileChecksum({0x2828380, 0x3a8ad60}, 0xc01a2bcc80, 0xc0002960a0, 0x1, 0x50c0000, 0x20000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:185 +0x4d7 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc01a40a8c0, {0xc001d38000, 0x20000, 0x20000}, 0x50c0000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:79 +0x225 io.(*OffsetWriter).Write(0xc0003b8da0, {0xc001d38000?, 0xa47780?, 0xc00017cb70?}) /usr/local/go/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc01a40aa80) /usr/local/go/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc01a40aa80, {0x2802b60, 0xc00017cb60}) /usr/local/go/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x2802b20, 0xc01a40aa80}, {0x2802b60, 0xc00017cb60}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:415 +0x151 io.Copy(...) /usr/local/go/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc000d0eae0, {0xc01a2bc5a0?, 0x283a988?}, 0x1, {0x2814f68, 0xc00017cb60}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc01a2bc5f0, {0x2828498, 0xc01a2bc5a0}, 0x1, {0x2821660, 0xc000d0eae0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 22922 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 goroutine 22951 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000ad80c0) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 23006 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 23009 [chan receive, 1 minutes]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc01a40aa00) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 23008 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 26175 [sleep]: time.Sleep(0x989680) /usr/local/go/src/runtime/time.go:195 +0x115 github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall.func1(0xc000a70a50?) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:167 +0x1c created by github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall in goroutine 23006 /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:166 +0xef goroutine 26172 [IO wait]: internal/poll.runtime_pollWait(0x7f5844206ef8, 0x77) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc01f874800?, 0xc01a202068?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitWrite(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:93 internal/poll.(*FD).Write(0xc01f874800, {0xc01a200000, 0x3c50, 0x5500}) /usr/local/go/src/internal/poll/fd_unix.go:388 +0x2d9 net.(*netFD).Write(0xc01f874800, {0xc01a200000?, 0x643385?, 0xc01e5d8230?}) /usr/local/go/src/net/fd_posix.go:96 +0x25 net.(*conn).Write(0xc01e5d8230, {0xc01a200000?, 0x0?, 0x0?}) /usr/local/go/src/net/net.go:191 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Write(0xc01fc64a08, {0xc01a200000, 0x3c50, 0x5500}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:122 +0x58 crypto/tls.(*Conn).write(0xc01a17c008, {0xc01a200000?, 0x5?, 0x5500?}) /usr/local/go/src/crypto/tls/conn.go:945 +0x109 crypto/tls.(*Conn).writeRecordLocked(0xc01a17c008, 0x17, {0xc000a80000, 0x8000, 0x8000}) /usr/local/go/src/crypto/tls/conn.go:1026 +0x685 crypto/tls.(*Conn).Write(0xc01fc64e40?, {0xc000a80000, 0x8000, 0x8000}) /usr/local/go/src/crypto/tls/conn.go:1233 +0x399 io.copyBuffer({0x7f583c5ef340, 0xc01a17c008}, {0x2802980, 0xc01fc64e40}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:431 +0x1de io.Copy(...) /usr/local/go/src/io/io.go:388 net/http.persistConnWriter.ReadFrom({0xc000ad9740?}, {0x2802980?, 0xc01fc64e40?}) /usr/local/go/src/net/http/transport.go:1823 +0x5b bufio.(*Writer).ReadFrom(0xc000ad9740, {0x2802980, 0xc01fc64e40}) /usr/local/go/src/bufio/bufio.go:794 +0x17c io.copyBuffer({0x2802b20, 0xc000ad9740}, {0x2802980, 0xc01fc64e40}, {0xc0007a0000, 0x8000, 0x8000}) /usr/local/go/src/io/io.go:415 +0x151 io.CopyBuffer({0x2802b20?, 0xc000ad9740?}, {0x2802980?, 0xc01fc64e40?}, {0xc0007a0000?, 0x18?, 0xc00008d008?}) /usr/local/go/src/io/io.go:402 +0x36 net/http.(*transferWriter).doBodyCopy(0xc0007870e0, {0x2802b20, 0xc000ad9740}, {0x2802980, 0xc01fc64e40}) /usr/local/go/src/net/http/transfer.go:416 +0xe8 net/http.(*transferWriter).writeBody(0xc0007870e0, {0x2802b20, 0xc000ad9740}) /usr/local/go/src/net/http/transfer.go:371 +0x370 net/http.(*Request).write(0xc000a7c900, {0x2802b20, 0xc000ad9740}, 0x0, 0xc0020636b0, 0x0) /usr/local/go/src/net/http/request.go:741 +0xb0d net/http.(*persistConn).writeLoop(0xc0008aa7e0) /usr/local/go/src/net/http/transport.go:2446 +0x18e created by net/http.(*Transport).dialConn in goroutine 26217 /usr/local/go/src/net/http/transport.go:1799 +0x1585 goroutine 23005 [select]: github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc000157700) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:221 +0xf6 created by github.com/rclone/rclone/fs/accounting.newAccountSizeName in goroutine 22922 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:119 +0x438 goroutine 26141 [select]: net/http.(*persistConn).writeLoop(0xc0008aac60) /usr/local/go/src/net/http/transport.go:2443 +0xf0 created by net/http.(*Transport).dialConn in goroutine 26223 /usr/local/go/src/net/http/transport.go:1799 +0x1585 goroutine 23010 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc01a40aa40) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 23007 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f "./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Finished ERROR in 1h0m0.538998193s (try 1/5): exit status 2: Failed [TestMultithreadCopy/upload=true,size=134217727,streams=2 TestMultithreadCopy/upload=true,size=134217728,streams=2 TestMultithreadCopy/upload=true,size=134217729,streams=2]