"./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose -fast-list -test.run '^TestMultithreadCopy$/^(upload=true,size=134217727,streams=2|upload=true,size=134217728,streams=2|upload=true,size=134217729,streams=2)$'" - Starting (try 5/5) 2025/02/21 04:18:36 DEBUG : Creating backend with remote "TestPcloud:rclone-test-dafibaz0sozi" 2025/02/21 04:18:36 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/02/21 04:18:37 DEBUG : Creating backend with remote "/tmp/rclone992077836" === RUN TestMultithreadCopy run.go:180: Remote "pcloud root 'rclone-test-dafibaz0sozi'", Local "Local file system at /tmp/rclone992077836", Modify Window "1s" === RUN TestMultithreadCopy/upload=true,size=134217727,streams=2 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: write buffer set to 131072 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 1 2025/02/21 04:18:46 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 0 2025/02/21 04:44:35 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2025/02/21 04:44:35 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/02/21 04:44:35 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 0/5 2025/02/21 04:44:36 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 1/5 2025/02/21 04:44:37 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 2/5 2025/02/21 04:44:38 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 3/5 2025/02/21 04:44:39 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 4/5 2025/02/21 04:44:40 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: cancelling transfer on exit multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to close object after copy: incorrect size after upload: got 127008768, want 134217727 Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217728,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/02/21 04:44:56 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: write buffer set to 131072 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 1 2025/02/21 04:44:57 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 0 2025/02/21 05:12:06 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/02/21 05:12:55 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2025/02/21 05:12:55 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 0/5 2025/02/21 05:12:56 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 1/5 2025/02/21 05:12:57 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 2/5 2025/02/21 05:12:58 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 3/5 2025/02/21 05:12:59 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 4/5 2025/02/21 05:13:00 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: cancelling transfer on exit multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to close object after copy: incorrect size after upload: got 134086656, want 134217728 Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217729,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217729-2 (134217729) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728), test-multithread-copy-true-134217729-2 (134217729) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217728-2" 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: write buffer set to 131072 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 2 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 0 2025/02/21 05:13:17 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 1 2025/02/21 05:13:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished panic: test timed out after 1h0m0s running tests: TestMultithreadCopy (1h0m0s) TestMultithreadCopy/upload=true,size=134217729,streams=2 (5m36s) goroutine 25863 [running]: testing.(*M).startAlarm.func1() /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:2366 +0x385 created by time.goFunc /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc0004ce4e0, {0x25831da?, 0x0?}, 0x26a9bf0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc0004ce4e0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc0004ce4e0, 0xc000a0fc00) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1689 +0xfb testing.runTests(0xc000b1a780, {0x3c3d5c0, 0x6c, 0x6c}, {0x1?, 0xedf49f59d?, 0x3c51b20?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc00041b680) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:2027 +0x68b github.com/rclone/rclone/fstest.TestMain(0xc00041b680) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0x196 goroutine 59 [chan receive, 5 minutes]: testing.(*T).Run(0xc0004ce680, {0xc00080aea0?, 0xc000a57f18?}, 0xc00031ef00) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1750 +0x3ab github.com/rclone/rclone/fs/operations.TestMultithreadCopy(0xc0004ce680) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:167 +0x1ef testing.tRunner(0xc0004ce680, 0x26a9bf0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1742 +0x390 goroutine 20 [syscall, 60 minutes]: os/signal.signal_recv() /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/os/signal/signal.go:151 +0x1f goroutine 21 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 22 [select, 60 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0008970c0) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 131 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 60 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 24817 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc000b38540) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:338 +0x150 created by github.com/rclone/rclone/fs/accounting.(*StatsInfo)._startAverageLoop in goroutine 24871 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:403 +0xb6 goroutine 24871 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc00096a480?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/sema.go:62 +0x25 sync.(*WaitGroup).Wait(0x236b600?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/waitgroup.go:116 +0x48 golang.org/x/sync/errgroup.(*Group).Wait(0xc0002bde40) /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:56 +0x25 github.com/rclone/rclone/fs/operations.multiThreadCopy({0x29908e0, 0xc000b17890}, {0x29a5718, 0xc000571720}, {0xc00080ae70, 0x26}, {0x29a5088, 0xc000a24180}, 0x2, 0xc0005a69c0, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:232 +0xe25 github.com/rclone/rclone/fs/operations.TestMultithreadCopy.func1(0xc000d3e4e0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:219 +0x8fc testing.tRunner(0xc000d3e4e0, 0xc00031ef00) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 59 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1742 +0x390 goroutine 24877 [select]: github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc0001d8a00) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:221 +0xf6 created by github.com/rclone/rclone/fs/accounting.newAccountSizeName in goroutine 24871 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:119 +0x438 goroutine 24947 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000897100) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 24879 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 24878 [select]: net/http.(*persistConn).roundTrip(0xc000646c60, 0xc000c8e900) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:2674 +0x979 net/http.(*Transport).roundTrip(0xc000c8af00, 0xc00039a7e0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:608 +0x79a net/http.(*Transport).RoundTrip(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc00036d180, 0xc00039a7e0) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc000554400, 0xc00039a6c0) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.25.0/transport.go:55 +0x3de net/http.send(0xc00039a6c0, {0x296d820, 0xc000554400}, {0xc019edce01?, 0x41cc25?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc01a05fda0, 0xc00039a6c0, {0x413fdb?, 0x1?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc01a05fda0, 0xc00039a6c0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc00089a690, {0x29908e0, 0xc000b17890}, 0xc000c8b040) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:356 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc00089a690, {0x29908e0, 0xc000b17890}, 0xc019edd708?, {0x0?, 0x0?}, {0x20bc220, 0xc01a05fe90}, 0x30?, 0x26ab0b0, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:553 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:496 github.com/rclone/rclone/backend/pcloud.fileOpen.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:153 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc01a05fec0?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0000ae720, 0xc01a05fec0, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileOpen({0x29908e0, 0xc000b17890}, 0xc00089a690, 0xc000571720, 0xe27314d56) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:152 +0x3a7 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc0002bde00, {0xc019f20000, 0x20000, 0x20000}, 0x4c0000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:73 +0x187 io.(*OffsetWriter).Write(0xc0003a6620, {0xc019f20000?, 0xa5e4e0?, 0xc019bb2010?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000692740) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000692740, {0x296a0a0, 0xc019bb2000}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x296a060, 0xc000692740}, {0x296a0a0, 0xc019bb2000}, {0x0, 0x0, 0x0}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/io/io.go:415 +0x151 io.Copy(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc01ac46540, {0xc000112910?, 0x29a5088?}, 0x0, {0x297cca8, 0xc019bb2000}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc000112960, {0x2990918, 0xc000112910}, 0x0, {0x2989508, 0xc01ac46540}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 24871 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 goroutine 25880 [select]: net/http.(*persistConn).writeLoop(0xc000646c60) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:2443 +0xf0 created by net/http.(*Transport).dialConn in goroutine 25841 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:1799 +0x1585 goroutine 25859 [IO wait]: internal/poll.runtime_pollWait(0x7f296a3c3ef8, 0x72) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc01a0dc780?, 0xc0007a1980?, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc01a0dc780, {0xc0007a1980, 0x1980, 0x1980}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc01a0dc780, {0xc0007a1980?, 0xc000a51850?, 0x41cc25?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc00073e0e0, {0xc0007a1980?, 0x7f29689e87b0?, 0xc0008861c8?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000b1a558, {0xc0007a1980?, 0x20?, 0x2555be0?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc0008861c8, {0xc0007a1980?, 0x0?, 0xc0008861c8?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0005d57b0, {0x296b980, 0xc0008861c8}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0005d5508, {0x7f2968058f18, 0xc000b1a558}, 0x1980?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0005d5508, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).Read(0xc0005d5508, {0xc000bc2000, 0x1000, 0x100b1e4b108?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:1370 +0x156 net/http.(*persistConn).Read(0xc000bd9560, {0xc000bc2000?, 0x40b7c5?, 0x60?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:1976 +0x4a bufio.(*Reader).fill(0xc01a0b60c0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc01a0b60c0, 0x1) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc000bd9560) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:2140 +0x1b9 created by net/http.(*Transport).dialConn in goroutine 25821 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:1798 +0x152f goroutine 24935 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000692700) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 24878 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 25879 [IO wait]: internal/poll.runtime_pollWait(0x7f296a3c3ff0, 0x72) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc01a16f000?, 0xc000a03980?, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc01a16f000, {0xc000a03980, 0x1980, 0x1980}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc01a16f000, {0xc000a03980?, 0xc000a56850?, 0x41cc25?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000884248, {0xc000a03980?, 0x7f29689e7810?, 0xc000b1abb8?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000886918, {0xc000a03980?, 0x20?, 0x2555be0?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc000b1abb8, {0xc000a03980?, 0x0?, 0xc000b1abb8?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0001dd430, {0x296b980, 0xc000b1abb8}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0001dd188, {0x7f2968058f18, 0xc000886918}, 0x1980?) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0001dd188, 0x0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).Read(0xc0001dd188, {0xc000790000, 0x1000, 0x100b1e4b5b8?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/crypto/tls/conn.go:1370 +0x156 net/http.(*persistConn).Read(0xc000646c60, {0xc000790000?, 0x40b7c5?, 0x60?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:1976 +0x4a bufio.(*Reader).fill(0xc000e794a0) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc000e794a0, 0x1) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc000646c60) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:2140 +0x1b9 created by net/http.(*Transport).dialConn in goroutine 25841 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:1798 +0x152f goroutine 25860 [select]: net/http.(*persistConn).writeLoop(0xc000bd9560) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:2443 +0xf0 created by net/http.(*Transport).dialConn in goroutine 25821 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:1799 +0x1585 goroutine 24879 [select]: net/http.(*persistConn).roundTrip(0xc000bd9560, 0xc000ad8800) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:2674 +0x979 net/http.(*Transport).roundTrip(0xc000ac8dc0, 0xc000bd9320) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/transport.go:608 +0x79a net/http.(*Transport).RoundTrip(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc00058c930, 0xc000bd9320) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc000414340, 0xc000bd9200) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.25.0/transport.go:55 +0x3de net/http.send(0xc000bd9200, {0x296d820, 0xc000414340}, {0xc000b02e01?, 0x41cc25?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc01a35ce10, 0xc000bd9200, {0x413fdb?, 0x1?, 0x0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc01a35ce10, 0xc000bd9200) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc000d00460, {0x29908e0, 0xc000b17890}, 0xc000ac8f00) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:356 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc000d00460, {0x29908e0, 0xc000b17890}, 0xc000b03708?, {0x0?, 0x0?}, {0x20bc220, 0xc01a35ced0}, 0x30?, 0x26ab0b0, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:553 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:496 github.com/rclone/rclone/backend/pcloud.fileOpen.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:153 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc01a35cf00?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0000ae720, 0xc01a35cf00, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileOpen({0x29908e0, 0xc000b17890}, 0xc000d00460, 0xc000571720, 0xe27314d56) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:152 +0x3a7 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc0002bde00, {0xc019f40000, 0x20000, 0x20000}, 0x44a0000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:73 +0x187 io.(*OffsetWriter).Write(0xc000b1e340, {0xc019f40000?, 0xa5e4e0?, 0xc0001501d0?}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000897140) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000897140, {0x296a0a0, 0xc0001501c0}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x296a060, 0xc000897140}, {0x296a0a0, 0xc0001501c0}, {0x0, 0x0, 0x0}) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/io/io.go:415 +0x151 io.Copy(...) /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc01ac46540, {0xc000112910?, 0x29a5088?}, 0x1, {0x297cca8, 0xc0001501c0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc000112960, {0x2990918, 0xc000112910}, 0x1, {0x2989508, 0xc01ac46540}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 24871 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96 "./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose -fast-list -test.run '^TestMultithreadCopy$/^(upload=true,size=134217727,streams=2|upload=true,size=134217728,streams=2|upload=true,size=134217729,streams=2)$'" - Finished ERROR in 1h0m0.502202439s (try 5/5): exit status 2: Failed [TestMultithreadCopy/upload=true,size=134217727,streams=2 TestMultithreadCopy/upload=true,size=134217728,streams=2 TestMultithreadCopy/upload=true,size=134217729,streams=2]