"./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose -test.run '^TestMultithreadCopy$/^(upload=true,size=134217727,streams=2|upload=true,size=134217728,streams=2|upload=true,size=134217729,streams=2)$'" - Starting (try 2/5) 2025/01/11 05:37:12 DEBUG : Creating backend with remote "TestPcloud:rclone-test-holinef0vixo" 2025/01/11 05:37:12 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/11 05:37:13 DEBUG : Creating backend with remote "/tmp/rclone1869746242" === RUN TestMultithreadCopy run.go:180: Remote "pcloud root 'rclone-test-holinef0vixo'", Local "Local file system at /tmp/rclone1869746242", Modify Window "1s" === RUN TestMultithreadCopy/upload=true,size=134217727,streams=2 2025/01/11 05:37:22 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: write buffer set to 131072 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 1 2025/01/11 05:37:23 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 0 2025/01/11 06:06:25 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/01/11 06:07:16 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2025/01/11 06:07:16 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 0/5 2025/01/11 06:07:17 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 1/5 2025/01/11 06:07:19 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 2/5 2025/01/11 06:07:20 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 3/5 2025/01/11 06:07:21 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 4/5 2025/01/11 06:07:22 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: cancelling transfer on exit multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to close object after copy: incorrect size after upload: got 134086656, want 134217727 Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217728,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: write buffer set to 131072 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 1 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/11 06:07:38 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 0 2025/01/11 06:30:36 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2025/01/11 06:31:12 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/01/11 06:31:12 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 0/5 2025/01/11 06:31:13 ERROR : test-multithread-copy-true-134217728-2: multi-thread copy: can't set metadata as SetMetadata isn't implemented in: pcloud root 'rclone-test-holinef0vixo' 2025/01/11 06:31:13 DEBUG : test-multithread-copy-true-134217728-2: Finished multi-thread copy with 2 parts of size 64Mi fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:224 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:224 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" fstest.go:121: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:121 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:126 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:145 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:189 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:225 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: test-multithread-copy-true-134217728-2: Modification time difference too big |-209834h26m14.500000001s| > 1s (want 2001-02-03 04:05:06.499999999 +0000 UTC vs got 2025-01-11 06:31:21 +0000 UTC) (precision 1s) === RUN TestMultithreadCopy/upload=true,size=134217729,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217729-2 (134217729) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217729-2 (134217729) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/01/11 06:31:40 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: write buffer set to 131072 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 2 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 0 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 1 2025/01/11 06:31:41 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished panic: test timed out after 1h0m0s running tests: TestMultithreadCopy (1h0m0s) TestMultithreadCopy/upload=true,size=134217729,streams=2 (5m49s) goroutine 25997 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2366 +0x385 created by time.goFunc /usr/local/go/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc0000fc680, {0x2441a23?, 0x0?}, 0x255e208) /usr/local/go/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc0000fc680) /usr/local/go/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc0000fc680, 0xc000987c00) /usr/local/go/src/testing/testing.go:1689 +0xfb testing.runTests(0xc00091a678, {0x3a0ff20, 0x6c, 0x6c}, {0x1?, 0xedf13fa89?, 0x3a23bc0?}) /usr/local/go/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc00050b0e0) /usr/local/go/src/testing/testing.go:2027 +0x68b github.com/rclone/rclone/fstest.TestMain(0xc00050b0e0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0x196 goroutine 50 [chan receive, 5 minutes]: testing.(*T).Run(0xc0000fc820, {0xc00051ca50?, 0xc000a01f10?}, 0xc000694c40) /usr/local/go/src/testing/testing.go:1750 +0x3ab github.com/rclone/rclone/fs/operations.TestMultithreadCopy(0xc0000fc820) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:167 +0x1f7 testing.tRunner(0xc0000fc820, 0x255e208) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 35 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 36 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 37 [select, 60 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc000694200) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 53 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 51 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 24906 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc000780708?) /usr/local/go/src/runtime/sema.go:62 +0x25 sync.(*WaitGroup).Wait(0x22352c0?) /usr/local/go/src/sync/waitgroup.go:116 +0x48 golang.org/x/sync/errgroup.(*Group).Wait(0xc0002b7b00) /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:56 +0x25 github.com/rclone/rclone/fs/operations.multiThreadCopy({0x2824f20, 0xc000917830}, {0x2839898, 0xc0005d00a0}, {0xc00051c9f0, 0x26}, {0x2839208, 0xc00010e360}, 0x2, 0xc019cce000, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:232 +0xe25 github.com/rclone/rclone/fs/operations.TestMultithreadCopy.func1(0xc0000fc9c0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:219 +0x905 testing.tRunner(0xc0000fc9c0, 0xc000694c40) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 50 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 24945 [select]: net/http.(*Transport).getConn(0xc000442c80, 0xc00098c400, {{}, 0x0, {0xc0004180a0, 0x5}, {0xc000446300, 0x13}, 0x0}) /usr/local/go/src/net/http/transport.go:1406 +0x5a5 net/http.(*Transport).roundTrip(0xc000442c80, 0xc000984fc0) /usr/local/go/src/net/http/transport.go:595 +0x73a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0003344d0, 0xc000984fc0) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc00011a2a0, 0xc000984ea0) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc000984ea0, {0x2802d60, 0xc00011a2a0}, {0xc01971ae01?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc000607e00, 0xc000984ea0, {0x413fdb?, 0x7f59140caa68?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc000607e00, 0xc000984ea0) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc0000bac30, {0x2824f20, 0xc000917830}, 0xc000442dc0) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:325 +0xcc5 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc0000bac30, {0x2824f20, 0xc000917830}, 0xc01971b710?, {0x0?, 0x0?}, {0x1f94a40, 0xc00021e060}, 0x30?, 0x255f6b8, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:522 +0x3e5 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:465 github.com/rclone/rclone/backend/pcloud.fileOpen.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:153 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc00021e1b0?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000400360, 0xc00021e1b0, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileOpen({0x2824f20, 0xc000917830}, 0xc0000bac30, 0xc0005d00a0, 0xd8642a65a) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:152 +0x387 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc0002b7ac0, {0xc019dc0000, 0x20000, 0x20000}, 0x4560000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:73 +0x187 io.(*OffsetWriter).Write(0xc0003a2fc0, {0xc019dc0000?, 0xa477c0?, 0xc000a11970?}) /usr/local/go/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000694500) /usr/local/go/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000694500, {0x27ff620, 0xc000a11960}) /usr/local/go/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x27ff5e0, 0xc000694500}, {0x27ff620, 0xc000a11960}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:415 +0x151 io.Copy(...) /usr/local/go/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc019f4ec00, {0xc0000bab40?, 0x2839208?}, 0x1, {0x2811a28, 0xc000a11960}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0000bab90, {0x2824f58, 0xc0000bab40}, 0x1, {0x281e120, 0xc019f4ec00}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 24906 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 goroutine 24895 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc0006944c0) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 24945 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 26023 [IO wait]: internal/poll.runtime_pollWait(0x7f58cc642e00, 0x72) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc000435580?, 0xc00038cd80?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000435580, {0xc00038cd80, 0x240, 0x240}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc000435580, {0xc00038cd80?, 0xc000863688?, 0x41cc25?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000089f40, {0xc00038cd80?, 0x7f58cc43d558?, 0xc000706690?}) /usr/local/go/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000706678, {0xc00038cd80?, 0x20cd8a0?, 0x205?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc000706690, {0xc00038cd80?, 0xc0002296c0?, 0xc000706690?}) /usr/local/go/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0003ce9b0, {0x2800f00, 0xc000706690}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0003ce708, {0x7f58cc4207f0, 0xc000706678}, 0x4bb420?) /usr/local/go/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0003ce708, 0x0) /usr/local/go/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).readHandshakeBytes(0xc0003ce708, 0x4) /usr/local/go/src/crypto/tls/conn.go:1075 +0x51 crypto/tls.(*Conn).readHandshake(0xc0003ce708, {0x0, 0x0}) /usr/local/go/src/crypto/tls/conn.go:1086 +0x34 crypto/tls.(*Conn).clientHandshake(0xc0003ce708, {0x2824f58, 0xc0000baf00}) /usr/local/go/src/crypto/tls/handshake_client.go:225 +0x27b crypto/tls.(*Conn).handshakeContext(0xc0003ce708, {0x2824f20, 0xc000917830}) /usr/local/go/src/crypto/tls/conn.go:1553 +0x3cb crypto/tls.(*Conn).HandshakeContext(...) /usr/local/go/src/crypto/tls/conn.go:1493 net/http.(*persistConn).addTLS.func2() /usr/local/go/src/net/http/transport.go:1572 +0x6e created by net/http.(*persistConn).addTLS in goroutine 25884 /usr/local/go/src/net/http/transport.go:1568 +0x309 goroutine 26022 [select]: net.(*netFD).connect.func2() /usr/local/go/src/net/fd_unix.go:118 +0x7a created by net.(*netFD).connect in goroutine 26018 /usr/local/go/src/net/fd_unix.go:117 +0x37c goroutine 26018 [IO wait]: internal/poll.runtime_pollWait(0x7f58cc642ff0, 0x77) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc01a2c0000?, 0x0?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitWrite(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:93 internal/poll.(*FD).WaitWrite(...) /usr/local/go/src/internal/poll/fd_unix.go:683 net.(*netFD).connect(0xc01a2c0000, {0x2824fc8, 0xc000334770}, {0xc00003d198?, 0x414145?}, {0x27ffa80?, 0xc000988a40?}) /usr/local/go/src/net/fd_unix.go:141 +0x70b net.(*netFD).dial(0xc01a2c0000, {0x2824fc8, 0xc000334770}, {0x282aed8?, 0x0?}, {0x282aed8, 0xc00039be30}, 0x90?) /usr/local/go/src/net/sock_posix.go:124 +0x3bc net.socket({0x2824fc8, 0xc000334770}, {0x241cafd, 0x3}, 0x2, 0x1, 0x41cefe?, 0x0, {0x282aed8, 0x0}, ...) /usr/local/go/src/net/sock_posix.go:70 +0x29b net.internetSocket({0x2824fc8, 0xc000334770}, {0x241cafd, 0x3}, {0x282aed8, 0x0}, {0x282aed8, 0xc00039be30}, 0x1, 0x0, ...) /usr/local/go/src/net/ipsock_posix.go:154 +0xf8 net.(*sysDialer).doDialTCPProto(0xc000440960, {0x2824fc8, 0xc000334770}, 0x0, 0xc00039be30, 0x0) /usr/local/go/src/net/tcpsock_posix.go:85 +0xec net.(*sysDialer).doDialTCP(...) /usr/local/go/src/net/tcpsock_posix.go:75 net.(*sysDialer).dialTCP(0x10?, {0x2824fc8?, 0xc000334770?}, 0xc00003d4a0?, 0x5392bb?) /usr/local/go/src/net/tcpsock_posix.go:71 +0x65 net.(*sysDialer).dialSingle(0xc000440960, {0x2824fc8, 0xc000334770}, {0x2812790, 0xc00039be30}) /usr/local/go/src/net/dial.go:651 +0x27d net.(*sysDialer).dialSerial(0xc000440960, {0x2824fc8, 0xc0003345b0}, {0xc0009bc300?, 0xa, 0xc0009bc300?}) /usr/local/go/src/net/dial.go:616 +0x24e net.(*sysDialer).dialParallel(0xc0009bc200?, {0x2824fc8?, 0xc0003345b0?}, {0xc0009bc300?, 0xc0003345b0?, 0x241d61d?}, {0x0?, 0x241cafd?, 0xae511487880b1321?}) /usr/local/go/src/net/dial.go:517 +0x3b4 net.(*Dialer).DialContext(0xc000435680, {0x2824f20, 0xc000917830}, {0x241cafd, 0x3}, {0xc000446300, 0x13}) /usr/local/go/src/net/dial.go:508 +0x69a github.com/rclone/rclone/fs/fshttp.(*Dialer).DialContext(0xc000435680, {0x2824f20, 0xc000917830}, {0x241cafd?, 0x60?}, {0xc000446300, 0x13}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:62 +0x125 github.com/rclone/rclone/fs/fshttp.NewTransportCustom.func1({0x2824f20, 0xc000917830}, {0x241cafd, 0x3}, {0xc000446300, 0x13}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:109 +0x65 net/http.(*Transport).dial(0x0?, {0x2824f20?, 0xc000917830?}, {0x241cafd?, 0x71cc00?}, {0xc000446300?, 0xc000663c28?}) /usr/local/go/src/net/http/transport.go:1187 +0xd2 net/http.(*Transport).dialConn(0xc000442c80, {0x2824f20, 0xc000917830}, {{}, 0x0, {0xc0004180a0, 0x5}, {0xc000446300, 0x13}, 0x0}) /usr/local/go/src/net/http/transport.go:1647 +0x7e8 net/http.(*Transport).dialConnFor(0xc000442c80, 0xc0009c6000) /usr/local/go/src/net/http/transport.go:1484 +0xca created by net/http.(*Transport).queueForDial in goroutine 24945 /usr/local/go/src/net/http/transport.go:1461 +0x1cd goroutine 25884 [chan receive]: net/http.(*persistConn).addTLS(0xc000984c60, {0x2824f20, 0xc000917830}, {0xc000446228, 0xf}, 0x0) /usr/local/go/src/net/http/transport.go:1578 +0x327 net/http.(*Transport).dialConn(0xc000442780, {0x2824f20, 0xc000917830}, {{}, 0x0, {0xc000418050, 0x5}, {0xc000446228, 0x13}, 0x0}) /usr/local/go/src/net/http/transport.go:1657 +0x99c net/http.(*Transport).dialConnFor(0xc000442780, 0xc0002ace70) /usr/local/go/src/net/http/transport.go:1484 +0xca created by net/http.(*Transport).queueForDial in goroutine 24944 /usr/local/go/src/net/http/transport.go:1461 +0x1cd goroutine 24944 [select]: net/http.(*Transport).getConn(0xc000442780, 0xc00098c180, {{}, 0x0, {0xc000418050, 0x5}, {0xc000446228, 0x13}, 0x0}) /usr/local/go/src/net/http/transport.go:1406 +0x5a5 net/http.(*Transport).roundTrip(0xc000442780, 0xc000984a20) /usr/local/go/src/net/http/transport.go:595 +0x73a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc000334070, 0xc000984a20) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc00011a100, 0xc000984900) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc000984900, {0x2802d60, 0xc00011a100}, {0xc019716e01?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc000607140, 0xc000984900, {0x413fdb?, 0x1?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc000607140, 0xc000984900) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc0000ba910, {0x2824f20, 0xc000917830}, 0xc0004428c0) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:325 +0xcc5 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc0000ba910, {0x2824f20, 0xc000917830}, 0xc019717710?, {0x0?, 0x0?}, {0x1f94a40, 0xc000607200}, 0x30?, 0x255f6b8, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:522 +0x3e5 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:465 github.com/rclone/rclone/backend/pcloud.fileOpen.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:153 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc000607230?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000400360, 0xc000607230, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileOpen({0x2824f20, 0xc000917830}, 0xc0000ba910, 0xc0005d00a0, 0xd8642a65a) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:152 +0x387 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc0002b7ac0, {0xc019da0000, 0x20000, 0x20000}, 0x560000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:73 +0x187 io.(*OffsetWriter).Write(0xc000350640, {0xc019da0000?, 0xa477c0?, 0xc019d32010?}) /usr/local/go/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc00051eb00) /usr/local/go/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc00051eb00, {0x27ff620, 0xc019d32000}) /usr/local/go/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x27ff5e0, 0xc00051eb00}, {0x27ff620, 0xc019d32000}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:415 +0x151 io.Copy(...) /usr/local/go/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc019f4ec00, {0xc0000bab40?, 0x2839208?}, 0x0, {0x2811a28, 0xc019d32000}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0000bab90, {0x2824f58, 0xc0000bab40}, 0x0, {0x281e120, 0xc019f4ec00}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 24906 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 goroutine 24833 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc00051eac0) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 24944 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 24943 [select]: github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc00062e800) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:221 +0xf6 created by github.com/rclone/rclone/fs/accounting.newAccountSizeName in goroutine 24906 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:119 +0x438 "./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose -test.run '^TestMultithreadCopy$/^(upload=true,size=134217727,streams=2|upload=true,size=134217728,streams=2|upload=true,size=134217729,streams=2)$'" - Finished ERROR in 1h0m0.52687773s (try 2/5): exit status 2: Failed [TestMultithreadCopy/upload=true,size=134217727,streams=2 TestMultithreadCopy/upload=true,size=134217728,streams=2 TestMultithreadCopy/upload=true,size=134217729,streams=2]