"./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Starting (try 1/5) 2024/12/07 04:10:15 DEBUG : Creating backend with remote "TestPcloud:rclone-test-patayal8yovo" 2024/12/07 04:10:15 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2024/12/07 04:10:15 DEBUG : Creating backend with remote "/tmp/rclone1323810672" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "pcloud root 'rclone-test-patayal8yovo'", Local "Local file system at /tmp/rclone1323810672", Modify Window "1s" === RUN TestMultithreadCopy/upload=false,size=134217727,streams=2 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: write buffer set to 131072 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2024/12/07 04:10:23 DEBUG : Couldn't parse Date: from server edef4.pcloud.com: "Sat, 07 Dec 2024 04:10:23 +0000": parsing time "Sat, 07 Dec 2024 04:10:23 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:23 +0000" as " " 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 1 2024/12/07 04:10:23 DEBUG : Couldn't parse Date: from server edef4.pcloud.com: "Sat, 07 Dec 2024 04:10:23 +0000": parsing time "Sat, 07 Dec 2024 04:10:23 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:23 +0000" as " " 2024/12/07 04:10:23 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 0 2024/12/07 04:10:25 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2024/12/07 04:10:25 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2024/12/07 04:10:25 DEBUG : test-multithread-copy-false-134217727-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217728,streams=2 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: write buffer set to 131072 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2024/12/07 04:10:38 DEBUG : Couldn't parse Date: from server edef4.pcloud.com: "Sat, 07 Dec 2024 04:10:38 +0000": parsing time "Sat, 07 Dec 2024 04:10:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:38 +0000" as " " 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 0 2024/12/07 04:10:38 DEBUG : Couldn't parse Date: from server edef4.pcloud.com: "Sat, 07 Dec 2024 04:10:38 +0000": parsing time "Sat, 07 Dec 2024 04:10:38 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:38 +0000" as " " 2024/12/07 04:10:38 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 1 2024/12/07 04:10:40 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2024/12/07 04:10:40 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2024/12/07 04:10:40 DEBUG : test-multithread-copy-false-134217728-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217729,streams=2 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: write buffer set to 131072 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2024/12/07 04:10:52 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Sat, 07 Dec 2024 04:10:52 +0000": parsing time "Sat, 07 Dec 2024 04:10:52 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:52 +0000" as " " 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 2 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2024/12/07 04:10:52 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Sat, 07 Dec 2024 04:10:52 +0000": parsing time "Sat, 07 Dec 2024 04:10:52 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:52 +0000" as " " 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 0 2024/12/07 04:10:52 DEBUG : Couldn't parse Date: from server edef6.pcloud.com: "Sat, 07 Dec 2024 04:10:52 +0000": parsing time "Sat, 07 Dec 2024 04:10:52 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 07 Dec 2024 04:10:52 +0000" as " " 2024/12/07 04:10:52 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 1 2024/12/07 04:10:54 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2024/12/07 04:10:54 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2024/12/07 04:10:54 DEBUG : test-multithread-copy-false-134217729-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=134217727,streams=2 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: write buffer set to 131072 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 1 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2024/12/07 04:11:08 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 0 2024/12/07 04:44:36 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2024/12/07 04:44:44 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2024/12/07 04:44:44 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 0/5 2024/12/07 04:44:44 DEBUG : test-multithread-copy-true-134217727-2: Finished multi-thread copy with 2 parts of size 64Mi fstest.go:137: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:137 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:189 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:225 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 Messages: pcloud root 'rclone-test-patayal8yovo'/test-multithread-copy-true-134217727-2: sha1 hash incorrect - expecting "8e3b2ac68113715abfd1823f74557bfef5f0737c" got "411a274368a47b9e29e017f4ed5245760c943f0f" fstest.go:137: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:137 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:189 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:225 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 Messages: pcloud root 'rclone-test-patayal8yovo'/test-multithread-copy-true-134217727-2: sha256 hash incorrect - expecting "53016fa9e9d55b737544dbb4b6d64aa0bfc3a6316f13373074a825c1849ec88d" got "a14cb19456c0b038e025a9baeff4a71d1998f8e8ea6b2e13aa89722cbc14c6f1" === RUN TestMultithreadCopy/upload=true,size=134217728,streams=2 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: write buffer set to 131072 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 0 2024/12/07 04:44:58 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 1 panic: test timed out after 1h0m0s running tests: TestMultithreadCopy (1h0m0s) TestMultithreadCopy/upload=true,size=134217728,streams=2 (25m26s) goroutine 22223 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2366 +0x385 created by time.goFunc /usr/local/go/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc00015a340, {0x23f23df?, 0x0?}, 0x250e808) /usr/local/go/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc00015a340) /usr/local/go/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc00015a340, 0xc00076bc00) /usr/local/go/src/testing/testing.go:1689 +0xfb testing.runTests(0xc0005ce1b0, {0x399e3e0, 0x6c, 0x6c}, {0x1?, 0xedee5c1a7?, 0x39b21e0?}) /usr/local/go/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc0002c7cc0) /usr/local/go/src/testing/testing.go:2027 +0x68b github.com/rclone/rclone/fstest.TestMain(0xc0002c7cc0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0x196 goroutine 12939 [select]: net/http.(*persistConn).roundTrip(0xc000312c60, 0xc0009d3bc0) /usr/local/go/src/net/http/transport.go:2674 +0x979 net/http.(*Transport).roundTrip(0xc0004d5b80, 0xc0007e2120) /usr/local/go/src/net/http/transport.go:608 +0x79a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc000052460, 0xc0007e2120) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc0004d7a00, 0xc0007e2000) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc0007e2000, {0x27a8be0, 0xc0004d7a00}, {0xc000c7ed01?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc002d412c0, 0xc0007e2000, {0x414126?, 0xc000c7ee38?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc002d412c0, 0xc0007e2000) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc0000bb4f0, {0x27cacc0, 0xc0004e7290}, 0xc0007e0000) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:325 +0xcc5 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc0000bb4f0, {0x27cacc0, 0xc0004e7290}, 0xc000c7f6a0?, {0x0?, 0x0?}, {0x1f52820, 0xc0000bb7c0}, 0x30?, 0x250fcb8, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:522 +0x3e5 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:465 github.com/rclone/rclone/backend/pcloud.fileChecksum.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:186 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc002d41f20?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0005879e0, 0xc002d41f20, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.fileChecksum({0x27cacc0, 0xc0004e7290}, 0xc0000bb4f0, 0xc0005a9100, 0x1, 0x7040000, 0x20000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:185 +0x4b7 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc000591280, {0xc001da0000, 0x20000, 0x20000}, 0x7040000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:79 +0x225 io.(*OffsetWriter).Write(0xc0004d6140, {0xc001da0000?, 0xa48760?, 0xc000200d30?}) /usr/local/go/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000ade080) /usr/local/go/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000ade080, {0x27a5600, 0xc000200d20}) /usr/local/go/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x27a55c0, 0xc000ade080}, {0x27a5600, 0xc000200d20}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:415 +0x151 io.Copy(...) /usr/local/go/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc000b2f620, {0xc0004af590?, 0x27def48?}, 0x1, {0x27b79a8, 0xc000200d20}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0004af630, {0x27cacf8, 0xc0004af590}, 0x1, {0x27c3ed8, 0xc000b2f620}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.9.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 12900 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.9.0/errgroup/errgroup.go:75 +0x96 goroutine 8 [select]: go.opencensus.io/stats/view.(*worker).start(0xc000460d00) /home/rclone/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f created by go.opencensus.io/stats/view.init.0 in goroutine 1 /home/rclone/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d goroutine 35 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 36 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 37 [select, 60 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc000136c40) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 63 [chan receive, 25 minutes]: testing.(*T).Run(0xc00015b040, {0xc0008908a0?, 0xc000093f10?}, 0xc000590d80) /usr/local/go/src/testing/testing.go:1750 +0x3ab github.com/rclone/rclone/fs/operations.TestMultithreadCopy(0xc00015b040) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:167 +0x1f7 testing.tRunner(0xc00015b040, 0x250e808) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 12900 [semacquire, 25 minutes]: sync.runtime_Semacquire(0xc000a4c678?) /usr/local/go/src/runtime/sema.go:62 +0x25 sync.(*WaitGroup).Wait(0x21ec5e0?) /usr/local/go/src/sync/waitgroup.go:116 +0x48 golang.org/x/sync/errgroup.(*Group).Wait(0xc000591300) /home/rclone/go/pkg/mod/golang.org/x/sync@v0.9.0/errgroup/errgroup.go:56 +0x25 github.com/rclone/rclone/fs/operations.multiThreadCopy({0x27cacc0, 0xc0004e7290}, {0x27df5d8, 0xc0002c7d60}, {0xc000890870, 0x26}, {0x27def48, 0xc0008c4540}, 0x2, 0xc0005d4180, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:232 +0xe25 github.com/rclone/rclone/fs/operations.TestMultithreadCopy.func1(0xc00015a1a0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:219 +0x905 testing.tRunner(0xc00015a1a0, 0xc000590d80) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 63 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 22221 [IO wait]: internal/poll.runtime_pollWait(0x7f40aa6d3ff0, 0x72) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc0004bd900?, 0xc000949300?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0004bd900, {0xc000949300, 0x1980, 0x1980}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc0004bd900, {0xc000949300?, 0xc000a47850?, 0x41cc25?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000a80200, {0xc000949300?, 0x7f40a8ca69a8?, 0xc0005cfae8?}) /usr/local/go/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc0004fcbb8, {0xc000949300?, 0x0?, 0xa47930?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc0005cfae8, {0xc000949300?, 0x0?, 0xc0005cfae8?}) /usr/local/go/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0005a70b0, {0x27a6ee0, 0xc0005cfae8}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0005a6e08, {0x7f40a8362e48, 0xc0004fcbb8}, 0xc000a47978?) /usr/local/go/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0005a6e08, 0x0) /usr/local/go/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).Read(0xc0005a6e08, {0xc00083c000, 0x1000, 0x8?}) /usr/local/go/src/crypto/tls/conn.go:1370 +0x156 net/http.(*persistConn).Read(0xc000536900, {0xc00083c000?, 0xc000b10e40?, 0xc000a47d38?}) /usr/local/go/src/net/http/transport.go:1976 +0x4a bufio.(*Reader).fill(0xc0029d47e0) /usr/local/go/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc0029d47e0, 0x1) /usr/local/go/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc000536900) /usr/local/go/src/net/http/transport.go:2140 +0x1b9 created by net/http.(*Transport).dialConn in goroutine 22182 /usr/local/go/src/net/http/transport.go:1798 +0x152f goroutine 12937 [select]: github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc001cbc000) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:221 +0xf6 created by github.com/rclone/rclone/fs/accounting.newAccountSizeName in goroutine 12900 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:119 +0x438 goroutine 21 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 64 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 12938 [select]: net/http.(*persistConn).roundTrip(0xc000536900, 0xc0009d3900) /usr/local/go/src/net/http/transport.go:2674 +0x979 net/http.(*Transport).roundTrip(0xc0002663c0, 0xc0002786c0) /usr/local/go/src/net/http/transport.go:608 +0x79a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc00013e9a0, 0xc0002786c0) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc00047ea20, 0xc0002785a0) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc0002785a0, {0x27a8be0, 0xc00047ea20}, {0xc000c7ad01?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc00013c5a0, 0xc0002785a0, {0x413f26?, 0xc000c7ae50?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc00013c5a0, 0xc0002785a0) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc0005055e0, {0x27cacc0, 0xc0004e7290}, 0xc0004d5900) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:325 +0xcc5 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc0005055e0, {0x27cacc0, 0xc0004e7290}, 0xc000c7b6b8?, {0x0?, 0x0?}, {0x1f52880, 0xc0004d79a0}, 0x30?, 0x250fcb8, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:522 +0x3e5 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:465 github.com/rclone/rclone/backend/pcloud.filePWrite.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:224 +0x6e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc002d41080?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0005879e0, 0xc002d41080, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:225 github.com/rclone/rclone/backend/pcloud.filePWrite({0x27cacc0, 0xc0004e7290}, 0xc0005055e0, 0xc0005a9100, 0x1, 0x3080000, {0xc001d80000, 0x20000, 0x20000}) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:223 +0x485 github.com/rclone/rclone/backend/pcloud.(*writerAt).WriteAt(0xc000591280, {0xc001d80000, 0x20000, 0x20000}, 0x3080000) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/writer_at.go:95 +0x354 io.(*OffsetWriter).Write(0xc00047e3a0, {0xc001d80000?, 0xa48760?, 0xc001d08010?}) /usr/local/go/src/io/io.go:583 +0x2c bufio.(*Writer).Flush(0xc000136340) /usr/local/go/src/bufio/bufio.go:639 +0x55 bufio.(*Writer).ReadFrom(0xc000136340, {0x27a5600, 0xc001d08000}) /usr/local/go/src/bufio/bufio.go:789 +0xbf io.copyBuffer({0x27a55c0, 0xc000136340}, {0x27a5600, 0xc001d08000}, {0x0, 0x0, 0x0}) /usr/local/go/src/io/io.go:415 +0x151 io.Copy(...) /usr/local/go/src/io/io.go:388 github.com/rclone/rclone/fs/operations.(*writerAtChunkWriter).WriteChunk(0xc000b2f620, {0xc0004af590?, 0x27def48?}, 0x0, {0x27b79a8, 0xc001d08000}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:308 +0x287 github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc0004af630, {0x27cacf8, 0xc0004af590}, 0x0, {0x27c3ed8, 0xc000b2f620}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:110 +0x743 github.com/rclone/rclone/fs/operations.multiThreadCopy.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread.go:228 +0x2b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/rclone/go/pkg/mod/golang.org/x/sync@v0.9.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 12900 /home/rclone/go/pkg/mod/golang.org/x/sync@v0.9.0/errgroup/errgroup.go:75 +0x96 goroutine 22191 [IO wait]: internal/poll.runtime_pollWait(0x7f40aa6d3e00, 0x72) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc0025c1000?, 0xc0001ba600?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0025c1000, {0xc0001ba600, 0x1980, 0x1980}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc0025c1000, {0xc0001ba600?, 0xc00099d850?, 0x41cc25?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000a80288, {0xc0001ba600?, 0x7f40a8ca69a8?, 0xc0005cfe78?}) /usr/local/go/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc0004fce70, {0xc0001ba600?, 0x0?, 0x99d930?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc0005cfe78, {0xc0001ba600?, 0x0?, 0xc0005cfe78?}) /usr/local/go/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0005a7430, {0x27a6ee0, 0xc0005cfe78}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0005a7188, {0x7f40a8362e48, 0xc0004fce70}, 0xc00099d978?) /usr/local/go/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0005a7188, 0x0) /usr/local/go/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).Read(0xc0005a7188, {0xc0008e4000, 0x1000, 0x8?}) /usr/local/go/src/crypto/tls/conn.go:1370 +0x156 net/http.(*persistConn).Read(0xc000312c60, {0xc0008e4000?, 0xc000b113e0?, 0xc00099dd38?}) /usr/local/go/src/net/http/transport.go:1976 +0x4a bufio.(*Reader).fill(0xc000a78d20) /usr/local/go/src/bufio/bufio.go:110 +0x103 bufio.(*Reader).Peek(0xc000a78d20, 0x1) /usr/local/go/src/bufio/bufio.go:148 +0x53 net/http.(*persistConn).readLoop(0xc000312c60) /usr/local/go/src/net/http/transport.go:2140 +0x1b9 created by net/http.(*Transport).dialConn in goroutine 22230 /usr/local/go/src/net/http/transport.go:1798 +0x152f goroutine 22192 [select]: net/http.(*persistConn).writeLoop(0xc000312c60) /usr/local/go/src/net/http/transport.go:2443 +0xf0 created by net/http.(*Transport).dialConn in goroutine 22230 /usr/local/go/src/net/http/transport.go:1799 +0x1585 goroutine 22235 [sleep]: time.Sleep(0x989680) /usr/local/go/src/runtime/time.go:195 +0x115 github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall.func1(0xc000a8abb0?) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:167 +0x1c created by github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall in goroutine 12939 /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:166 +0xef goroutine 12940 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc0005916c0) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 12939 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 22222 [select]: net/http.(*persistConn).writeLoop(0xc000536900) /usr/local/go/src/net/http/transport.go:2443 +0xf0 created by net/http.(*Transport).dialConn in goroutine 22182 /usr/local/go/src/net/http/transport.go:1799 +0x1585 goroutine 12905 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000136300) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 12938 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f "./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Finished ERROR in 1h0m0.499104403s (try 1/5): exit status 2: Failed []