"./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Starting (try 1/5) 2025/01/14 03:47:27 DEBUG : Creating backend with remote "TestPcloud:rclone-test-cazulan1heku" 2025/01/14 03:47:27 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/14 03:47:33 DEBUG : Creating backend with remote "/tmp/rclone1153736003" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" === RUN TestMultithreadCopy/upload=false,size=134217727,streams=2 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: write buffer set to 131072 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/01/14 03:47:43 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:47:43 +0000": parsing time "Tue, 14 Jan 2025 03:47:43 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:47:43 +0000" as " " 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 1 2025/01/14 03:47:43 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:47:43 +0000": parsing time "Tue, 14 Jan 2025 03:47:43 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:47:43 +0000" as " " 2025/01/14 03:47:43 DEBUG : test-multithread-copy-false-134217727-2: writing chunk 0 2025/01/14 03:47:45 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi finished 2025/01/14 03:47:45 DEBUG : test-multithread-copy-false-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/01/14 03:47:45 DEBUG : test-multithread-copy-false-134217727-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217728,streams=2 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: write buffer set to 131072 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/14 03:48:01 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:48:01 +0000": parsing time "Tue, 14 Jan 2025 03:48:01 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:48:01 +0000" as " " 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 1 2025/01/14 03:48:01 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:48:01 +0000": parsing time "Tue, 14 Jan 2025 03:48:01 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:48:01 +0000" as " " 2025/01/14 03:48:01 DEBUG : test-multithread-copy-false-134217728-2: writing chunk 0 2025/01/14 03:48:03 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi finished 2025/01/14 03:48:03 DEBUG : test-multithread-copy-false-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi finished 2025/01/14 03:48:03 DEBUG : test-multithread-copy-false-134217728-2: Finished multi-thread copy with 2 parts of size 64Mi === RUN TestMultithreadCopy/upload=false,size=134217729,streams=2 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: write buffer set to 131072 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/14 03:48:17 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:48:17 +0000": parsing time "Tue, 14 Jan 2025 03:48:17 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:48:17 +0000" as " " 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 2 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2025/01/14 03:48:17 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:48:17 +0000": parsing time "Tue, 14 Jan 2025 03:48:17 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:48:17 +0000" as " " 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 0 2025/01/14 03:48:17 DEBUG : Couldn't parse Date: from server edef8.pcloud.com: "Tue, 14 Jan 2025 03:48:17 +0000": parsing time "Tue, 14 Jan 2025 03:48:17 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 03:48:17 +0000" as " " 2025/01/14 03:48:17 DEBUG : test-multithread-copy-false-134217729-2: writing chunk 1 2025/01/14 03:48:19 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/01/14 03:48:19 DEBUG : test-multithread-copy-false-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/01/14 03:48:19 DEBUG : test-multithread-copy-false-134217729-2: Finished multi-thread copy with 3 parts of size 64Mi === RUN TestMultithreadCopy/upload=true,size=134217727,streams=2 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: write buffer set to 131072 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 (67108864-134217727) size 64.000Mi starting 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 1 2025/01/14 03:48:33 DEBUG : test-multithread-copy-true-134217727-2: writing chunk 0 2025/01/14 03:54:26 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/14 03:54:26 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/14 03:54:26 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 2/2 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/14 03:54:37 DEBUG : pacer: Reducing sleep to 15ms 2025/01/14 03:54:37 DEBUG : pacer: Reducing sleep to 11.25ms 2025/01/14 03:54:37 DEBUG : pacer: Reducing sleep to 10ms 2025/01/14 03:54:37 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: chunk 1/2 failed: multi-thread copy: failed to write chunk: context canceled 2025/01/14 03:54:37 DEBUG : test-multithread-copy-true-134217727-2: multi-thread copy: cancelling transfer on exit 2025/01/14 03:54:37 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 0/5 2025/01/14 03:54:38 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 1/5 2025/01/14 03:54:39 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 2/5 2025/01/14 03:54:40 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 3/5 2025/01/14 03:54:41 DEBUG : test-multithread-copy-true-134217727-2: checking file size: try 4/5 2025/01/14 03:54:42 ERROR : test-multithread-copy-true-134217727-2: multi-thread copy: failed to close file before aborting: incorrect size after upload: got 123207680, want 134217727 multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217727,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217728,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217728-2 (134217728) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: write buffer set to 131072 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: Starting multi-thread copy with 2 chunks of size 64Mi with 2 parallel streams 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 (67108864-134217728) size 64Mi starting 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 (0-67108864) size 64Mi starting 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 1 2025/01/14 03:54:58 DEBUG : test-multithread-copy-true-134217728-2: writing chunk 0 2025/01/14 04:08:39 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/14 04:08:39 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/14 04:08:39 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 1/2 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/14 04:08:39 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/14 04:08:39 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/01/14 04:08:39 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: chunk 2/2 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/14 04:08:39 DEBUG : test-multithread-copy-true-134217728-2: multi-thread copy: cancelling transfer on exit 2025/01/14 04:08:39 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 0/5 2025/01/14 04:08:40 DEBUG : pacer: Reducing sleep to 30ms 2025/01/14 04:08:41 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 1/5 2025/01/14 04:08:41 DEBUG : pacer: Reducing sleep to 22.5ms 2025/01/14 04:08:42 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 2/5 2025/01/14 04:08:42 DEBUG : pacer: Reducing sleep to 16.875ms 2025/01/14 04:08:43 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 3/5 2025/01/14 04:08:43 DEBUG : pacer: Reducing sleep to 12.65625ms 2025/01/14 04:08:44 DEBUG : test-multithread-copy-true-134217728-2: checking file size: try 4/5 2025/01/14 04:08:44 DEBUG : pacer: Reducing sleep to 10ms 2025/01/14 04:08:45 ERROR : test-multithread-copy-true-134217728-2: multi-thread copy: failed to close file before aborting: incorrect size after upload: got 80740352, want 134217728 multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217728,streams=2 === RUN TestMultithreadCopy/upload=true,size=134217729,streams=2 fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: listing wrong, want test-multithread-copy-true-134217729-2 (134217729) got test-multithread-copy-true-134217727-2 (134217727), test-multithread-copy-true-134217728-2 (134217728), test-multithread-copy-true-134217729-2 (134217729) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217727-2" fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:189 Error: Should be true Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 Messages: Unexpected file "test-multithread-copy-true-134217728-2" 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: write buffer set to 131072 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 2 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 0 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/14 04:09:01 DEBUG : test-multithread-copy-true-134217729-2: writing chunk 1 2025/01/14 04:09:02 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 finished 2025/01/14 04:16:18 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/14 04:16:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/14 04:16:18 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 1/3 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/14 04:16:30 DEBUG : pacer: low level retry 1/1 (error pcloud error: Internal upload error. (5001)) 2025/01/14 04:16:30 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/01/14 04:16:30 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: chunk 2/3 failed: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) 2025/01/14 04:16:30 DEBUG : test-multithread-copy-true-134217729-2: multi-thread copy: cancelling transfer on exit 2025/01/14 04:16:30 DEBUG : test-multithread-copy-true-134217729-2: checking file size: try 0/5 2025/01/14 04:16:30 DEBUG : pacer: Reducing sleep to 30ms 2025/01/14 04:16:30 DEBUG : pacer: Reducing sleep to 22.5ms 2025/01/14 04:16:30 DEBUG : pacer: Reducing sleep to 16.875ms multithread_test.go:220: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:220 Error: Received unexpected error: multi-thread copy: failed to write chunk: open file: open new file descriptor: pcloud error: Internal upload error. (5001) Test: TestMultithreadCopy/upload=true,size=134217729,streams=2 2025/01/14 04:16:30 DEBUG : pacer: Reducing sleep to 12.65625ms 2025/01/14 04:16:30 DEBUG : pacer: Reducing sleep to 10ms --- FAIL: TestMultithreadCopy (1737.63s) --- PASS: TestMultithreadCopy/upload=false,size=134217727,streams=2 (16.97s) --- PASS: TestMultithreadCopy/upload=false,size=134217728,streams=2 (17.82s) --- PASS: TestMultithreadCopy/upload=false,size=134217729,streams=2 (16.03s) --- FAIL: TestMultithreadCopy/upload=true,size=134217727,streams=2 (378.78s) --- FAIL: TestMultithreadCopy/upload=true,size=134217728,streams=2 (842.71s) --- FAIL: TestMultithreadCopy/upload=true,size=134217729,streams=2 (465.15s) === RUN TestMultithreadCopyAbort run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: write buffer set to 131072 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/01/14 04:16:40 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 64Mi with 3 parallel streams 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (134217728-134217729) size 1 starting 2025/01/14 04:16:40 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi starting 2025/01/14 04:16:40 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi starting 2025/01/14 04:16:40 DEBUG : Open with options = [RangeOption(67108864,134217727)] 2025/01/14 04:16:40 DEBUG : Open with options = [RangeOption(0,67108863)] 2025/01/14 04:16:40 DEBUG : test-multithread-abort: writing chunk 1 2025/01/14 04:16:40 DEBUG : test-multithread-abort: writing chunk 0 2025/01/14 04:46:51 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-67108864) size 64Mi finished 2025/01/14 04:46:54 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (67108864-134217728) size 64Mi finished 2025/01/14 04:46:54 DEBUG : Returning error reader 2025/01/14 04:46:54 DEBUG : test-multithread-abort: writing chunk 2 2025/01/14 04:46:54 DEBUG : BOOM: simulated read failure 2025/01/14 04:46:54 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 1/10: BOOM: simulated read failure 2025/01/14 04:46:54 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:46:55 DEBUG : Returning error reader 2025/01/14 04:46:55 DEBUG : BOOM: simulated read failure 2025/01/14 04:46:55 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 2/10: BOOM: simulated read failure 2025/01/14 04:46:55 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:46:56 DEBUG : Returning error reader 2025/01/14 04:46:56 DEBUG : BOOM: simulated read failure 2025/01/14 04:46:56 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 3/10: BOOM: simulated read failure 2025/01/14 04:46:56 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:46:57 DEBUG : Returning error reader 2025/01/14 04:46:57 DEBUG : BOOM: simulated read failure 2025/01/14 04:46:57 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 4/10: BOOM: simulated read failure 2025/01/14 04:46:57 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:46:58 DEBUG : Returning error reader 2025/01/14 04:46:58 DEBUG : BOOM: simulated read failure 2025/01/14 04:46:58 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 5/10: BOOM: simulated read failure 2025/01/14 04:46:58 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:46:59 DEBUG : Returning error reader 2025/01/14 04:46:59 DEBUG : BOOM: simulated read failure 2025/01/14 04:46:59 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 6/10: BOOM: simulated read failure 2025/01/14 04:46:59 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:47:00 DEBUG : Returning error reader 2025/01/14 04:47:00 DEBUG : BOOM: simulated read failure 2025/01/14 04:47:00 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 7/10: BOOM: simulated read failure 2025/01/14 04:47:00 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:47:01 DEBUG : Returning error reader 2025/01/14 04:47:01 DEBUG : BOOM: simulated read failure 2025/01/14 04:47:01 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 8/10: BOOM: simulated read failure 2025/01/14 04:47:01 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:47:02 DEBUG : Returning error reader 2025/01/14 04:47:02 DEBUG : BOOM: simulated read failure 2025/01/14 04:47:02 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 9/10: BOOM: simulated read failure 2025/01/14 04:47:02 DEBUG : Open with options = [RangeOption(134217728,134217728)] 2025/01/14 04:47:03 DEBUG : Returning error reader 2025/01/14 04:47:03 DEBUG : BOOM: simulated read failure 2025/01/14 04:47:03 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 10/10: BOOM: simulated read failure 2025/01/14 04:47:03 DEBUG : test-multithread-abort: Reopen failed after offset 0 bytes read: failed to reopen: too many retries 2025/01/14 04:47:03 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 failed: multi-thread copy: failed to write chunk: BOOM: simulated read failure 2025/01/14 04:47:03 DEBUG : test-multithread-abort: multi-thread copy: cancelling transfer on exit 2025/01/14 04:47:03 DEBUG : test-multithread-abort: checking file size: try 0/5 2025/01/14 04:47:05 DEBUG : test-multithread-abort: checking file size: try 1/5 2025/01/14 04:47:06 DEBUG : test-multithread-abort: checking file size: try 2/5 2025/01/14 04:47:07 DEBUG : test-multithread-abort: checking file size: try 3/5 2025/01/14 04:47:08 DEBUG : test-multithread-abort: checking file size: try 4/5 2025/01/14 04:47:09 ERROR : test-multithread-abort: multi-thread copy: failed to close file before aborting: incorrect size after upload: got 134217728, want 134217729 --- PASS: TestMultithreadCopyAbort (1838.86s) === RUN TestSizeDiffers --- PASS: TestSizeDiffers (0.00s) === RUN TestReOpen === RUN TestReOpen/Normal === RUN TestReOpen/Normal/Basics 2025/01/14 04:47:09 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/ErrorAtStart === RUN TestReOpen/Normal/WithErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/Normal/TooManyErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/Normal/Seek 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Seek from 5 to 2 === RUN TestReOpen/Normal/AccountRead === RUN TestReOpen/Normal/AccountReadDelay 2025/01/14 04:47:09 DEBUG : potato: Seek from 10 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 10 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/AccountReadError === RUN TestReOpen/WithRangeOption === RUN TestReOpen/WithRangeOption/Basics 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/ErrorAtStart === RUN TestReOpen/WithRangeOption/WithErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithRangeOption/TooManyErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithRangeOption/Seek 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Seek from 5 to 2 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 4 === RUN TestReOpen/WithRangeOption/AccountRead === RUN TestReOpen/WithRangeOption/AccountReadDelay 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/AccountReadError === RUN TestReOpen/WithSeekOption === RUN TestReOpen/WithSeekOption/Basics 2025/01/14 04:47:09 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/ErrorAtStart === RUN TestReOpen/WithSeekOption/WithErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithSeekOption/TooManyErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithSeekOption/Seek 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Seek from 5 to 2 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 5 === RUN TestReOpen/WithSeekOption/AccountRead === RUN TestReOpen/WithSeekOption/AccountReadDelay 2025/01/14 04:47:09 DEBUG : potato: Seek from 8 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 8 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/AccountReadError === RUN TestReOpen/UnknownSize === RUN TestReOpen/UnknownSize/Basics 2025/01/14 04:47:09 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/ErrorAtStart === RUN TestReOpen/UnknownSize/WithErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/UnknownSize/TooManyErrors 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/01/14 04:47:09 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/UnknownSize/Seek 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/01/14 04:47:09 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/01/14 04:47:09 DEBUG : potato: Seek from 5 to 2 2025/01/14 04:47:09 DEBUG : potato: Seek from 7 to 6 === RUN TestReOpen/UnknownSize/AccountRead === RUN TestReOpen/UnknownSize/AccountReadDelay 2025/01/14 04:47:09 DEBUG : potato: Seek from 9 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 9 to 0 2025/01/14 04:47:09 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/AccountReadError --- PASS: TestReOpen (0.00s) --- PASS: TestReOpen/Normal (0.00s) --- PASS: TestReOpen/Normal/Basics (0.00s) --- PASS: TestReOpen/Normal/ErrorAtStart (0.00s) --- PASS: TestReOpen/Normal/WithErrors (0.00s) --- PASS: TestReOpen/Normal/TooManyErrors (0.00s) --- PASS: TestReOpen/Normal/Seek (0.00s) --- PASS: TestReOpen/Normal/AccountRead (0.00s) --- PASS: TestReOpen/Normal/AccountReadDelay (0.00s) --- PASS: TestReOpen/Normal/AccountReadError (0.00s) --- PASS: TestReOpen/WithRangeOption (0.00s) --- PASS: TestReOpen/WithRangeOption/Basics (0.00s) --- PASS: TestReOpen/WithRangeOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithRangeOption/WithErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/Seek (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountRead (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadError (0.00s) --- PASS: TestReOpen/WithSeekOption (0.00s) --- PASS: TestReOpen/WithSeekOption/Basics (0.00s) --- PASS: TestReOpen/WithSeekOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithSeekOption/WithErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/Seek (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountRead (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadError (0.00s) --- PASS: TestReOpen/UnknownSize (0.00s) --- PASS: TestReOpen/UnknownSize/Basics (0.00s) --- PASS: TestReOpen/UnknownSize/ErrorAtStart (0.00s) --- PASS: TestReOpen/UnknownSize/WithErrors (0.00s) --- PASS: TestReOpen/UnknownSize/TooManyErrors (0.00s) --- PASS: TestReOpen/UnknownSize/Seek (0.00s) --- PASS: TestReOpen/UnknownSize/AccountRead (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadDelay (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadError (0.00s) === RUN TestCheck run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" === RUN TestCheck/1 === RUN TestCheck/2 === RUN TestCheck/3 === RUN TestCheck/4 === RUN TestCheck/5 === RUN TestCheck/6 === RUN TestCheck/7 --- PASS: TestCheck (3.22s) --- PASS: TestCheck/1 (0.16s) --- PASS: TestCheck/2 (0.16s) --- PASS: TestCheck/3 (0.17s) --- PASS: TestCheck/4 (0.18s) --- PASS: TestCheck/5 (0.17s) --- PASS: TestCheck/6 (0.18s) --- PASS: TestCheck/7 (0.18s) === RUN TestCheckFsError 2025/01/14 04:47:12 DEBUG : Creating backend with remote "nonexistent" 2025/01/14 04:47:12 DEBUG : Config file has changed externally - reloading 2025/01/14 04:47:12 DEBUG : Creating backend with remote "nonexistent" 2025/01/14 04:47:12 DEBUG : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: Waiting for checks to finish 2025/01/14 04:47:12 ERROR : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: error reading source root directory: directory not found 2025/01/14 04:47:12 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 differences found 2025/01/14 04:47:12 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 errors while checking --- PASS: TestCheckFsError (0.00s) === RUN TestCheckDownload run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" === RUN TestCheckDownload/1 === RUN TestCheckDownload/2 === RUN TestCheckDownload/3 === RUN TestCheckDownload/4 === RUN TestCheckDownload/5 === RUN TestCheckDownload/6 === RUN TestCheckDownload/7 --- PASS: TestCheckDownload (3.90s) --- PASS: TestCheckDownload/1 (0.25s) --- PASS: TestCheckDownload/2 (0.26s) --- PASS: TestCheckDownload/3 (0.25s) --- PASS: TestCheckDownload/4 (0.27s) --- PASS: TestCheckDownload/5 (0.25s) --- PASS: TestCheckDownload/6 (0.27s) --- PASS: TestCheckDownload/7 (0.26s) === RUN TestCheckSizeOnly run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" === RUN TestCheckSizeOnly/1 === RUN TestCheckSizeOnly/2 === RUN TestCheckSizeOnly/3 === RUN TestCheckSizeOnly/4 === RUN TestCheckSizeOnly/5 === RUN TestCheckSizeOnly/6 === RUN TestCheckSizeOnly/7 --- PASS: TestCheckSizeOnly (2.63s) --- PASS: TestCheckSizeOnly/1 (0.08s) --- PASS: TestCheckSizeOnly/2 (0.08s) --- PASS: TestCheckSizeOnly/3 (0.08s) --- PASS: TestCheckSizeOnly/4 (0.08s) --- PASS: TestCheckSizeOnly/5 (0.09s) --- PASS: TestCheckSizeOnly/6 (0.08s) --- PASS: TestCheckSizeOnly/7 (0.08s) === RUN TestCheckEqualReaders --- PASS: TestCheckEqualReaders (0.00s) === RUN TestParseSumFile run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" 2025/01/14 04:47:19 DEBUG : Couldn't parse Date: from server edef12.pcloud.com: "Tue, 14 Jan 2025 04:47:19 +0000": parsing time "Tue, 14 Jan 2025 04:47:19 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 04:47:19 +0000" as " " 2025/01/14 04:47:19 NOTICE: test.sum: improperly formatted checksum line 4 2025/01/14 04:47:19 NOTICE: test.sum: improperly formatted checksum line 5 2025/01/14 04:47:19 NOTICE: test.sum: improperly formatted checksum line 6 2025/01/14 04:47:19 NOTICE: test.sum: 2 warning(s) suppressed... 2025/01/14 04:47:20 DEBUG : Couldn't parse Date: from server edef4.pcloud.com: "Tue, 14 Jan 2025 04:47:20 +0000": parsing time "Tue, 14 Jan 2025 04:47:20 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 04:47:20 +0000" as " " 2025/01/14 04:47:20 NOTICE: test.sum: improperly formatted checksum line 4 2025/01/14 04:47:20 NOTICE: test.sum: improperly formatted checksum line 5 2025/01/14 04:47:20 NOTICE: test.sum: improperly formatted checksum line 6 2025/01/14 04:47:20 NOTICE: test.sum: 2 warning(s) suppressed... --- PASS: TestParseSumFile (0.93s) === RUN TestCheckSum run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" 2025/01/14 04:47:20 DEBUG : Creating backend with remote "TestPcloud:rclone-test-cazulan1heku/data" check_test.go:353: pcloud root 'rclone-test-cazulan1heku/data' lacks md5, skipping --- SKIP: TestCheckSum (0.78s) === RUN TestCheckSumDownload run.go:180: Remote "pcloud root 'rclone-test-cazulan1heku'", Local "Local file system at /tmp/rclone1153736003", Modify Window "1s" 2025/01/14 04:47:21 DEBUG : Creating backend with remote "TestPcloud:rclone-test-cazulan1heku/data" === RUN TestCheckSumDownload/subtest1 === RUN TestCheckSumDownload/subtest2 === RUN TestCheckSumDownload/subtest3 === RUN TestCheckSumDownload/subtest4 === RUN TestCheckSumDownload/subtest5 === RUN TestCheckSumDownload/subtest6 === RUN TestCheckSumDownload/subtest7 --- PASS: TestCheckSumDownload (8.21s) --- PASS: TestCheckSumDownload/subtest1 (0.58s) --- PASS: TestCheckSumDownload/subtest2 (0.49s) --- PASS: TestCheckSumDownload/subtest3 (0.52s) --- PASS: TestCheckSumDownload/subtest4 (0.51s) --- PASS: TestCheckSumDownload/subtest5 (0.51s) --- PASS: TestCheckSumDownload/subtest6 (0.54s) --- PASS: TestCheckSumDownload/subtest7 (0.51s) === RUN TestApplyTransforms 2025/01/14 04:47:29 DEBUG : Creating backend with remote "TestPcloud:rclone-test-rimugum7zike" 2025/01/14 04:47:29 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/14 04:47:29 DEBUG : Creating backend with remote "/tmp/rclone239505511" run.go:180: Remote "pcloud root 'rclone-test-rimugum7zike'", Local "Local file system at /tmp/rclone239505511", Modify Window "1s" upper checkfile vs. lower remote (without normalization) 2025/01/14 04:47:30 ERROR : hello, world!: sum not found 2025/01/14 04:47:30 ERROR : HELLO, WORLD!: file not in pcloud root 'rclone-test-rimugum7zike' 2025/01/14 04:47:30 NOTICE: pcloud root 'rclone-test-rimugum7zike': 1 files missing 2025/01/14 04:47:30 NOTICE: 1 hashes missing 2025/01/14 04:47:30 NOTICE: pcloud root 'rclone-test-rimugum7zike': 2 differences found 2025/01/14 04:47:30 NOTICE: pcloud root 'rclone-test-rimugum7zike': 2 errors while checking upper checkfile vs. lower remote (with normalization) 2025/01/14 04:47:30 DEBUG : Couldn't parse Date: from server edef11.pcloud.com: "Tue, 14 Jan 2025 04:47:30 +0000": parsing time "Tue, 14 Jan 2025 04:47:30 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 04:47:30 +0000" as " " 2025/01/14 04:47:30 DEBUG : hello, world!: md5 = 65a8e27d8879283831b664bd8b7f0ad4 OK 2025/01/14 04:47:30 NOTICE: pcloud root 'rclone-test-rimugum7zike': 0 differences found 2025/01/14 04:47:30 NOTICE: pcloud root 'rclone-test-rimugum7zike': 1 matching files 2025/01/14 04:47:30 DEBUG : Creating backend with remote "TestPcloud:rclone-test-pavicok4lidu" 2025/01/14 04:47:30 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/14 04:47:31 DEBUG : Creating backend with remote "/tmp/rclone14038520" run.go:180: Remote "pcloud root 'rclone-test-pavicok4lidu'", Local "Local file system at /tmp/rclone14038520", Modify Window "1s" lower checkfile vs. upper remote (without normalization) 2025/01/14 04:47:31 ERROR : HELLO, WORLD!: sum not found 2025/01/14 04:47:31 ERROR : hello, world!: file not in pcloud root 'rclone-test-pavicok4lidu' 2025/01/14 04:47:31 NOTICE: pcloud root 'rclone-test-pavicok4lidu': 1 files missing 2025/01/14 04:47:31 NOTICE: 1 hashes missing 2025/01/14 04:47:31 NOTICE: pcloud root 'rclone-test-pavicok4lidu': 2 differences found 2025/01/14 04:47:31 NOTICE: pcloud root 'rclone-test-pavicok4lidu': 2 errors while checking lower checkfile vs. upper remote (with normalization) 2025/01/14 04:47:31 DEBUG : Couldn't parse Date: from server edef11.pcloud.com: "Tue, 14 Jan 2025 04:47:31 +0000": parsing time "Tue, 14 Jan 2025 04:47:31 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 04:47:31 +0000" as " " 2025/01/14 04:47:31 DEBUG : HELLO, WORLD!: md5 = 65a8e27d8879283831b664bd8b7f0ad4 OK 2025/01/14 04:47:31 NOTICE: pcloud root 'rclone-test-pavicok4lidu': 0 differences found 2025/01/14 04:47:31 NOTICE: pcloud root 'rclone-test-pavicok4lidu': 1 matching files 2025/01/14 04:47:31 DEBUG : Creating backend with remote "TestPcloud:rclone-test-qefufoq7nora" 2025/01/14 04:47:31 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/14 04:47:32 DEBUG : Creating backend with remote "/tmp/rclone2227167929" run.go:180: Remote "pcloud root 'rclone-test-qefufoq7nora'", Local "Local file system at /tmp/rclone2227167929", Modify Window "1s" lower checkfile vs. upperlowermixed remote (without normalization) 2025/01/14 04:47:32 ERROR : HeLlO, wOrLd!: sum not found 2025/01/14 04:47:32 ERROR : hello, world!: file not in pcloud root 'rclone-test-qefufoq7nora' 2025/01/14 04:47:32 NOTICE: pcloud root 'rclone-test-qefufoq7nora': 1 files missing 2025/01/14 04:47:32 NOTICE: 1 hashes missing 2025/01/14 04:47:32 NOTICE: pcloud root 'rclone-test-qefufoq7nora': 2 differences found 2025/01/14 04:47:32 NOTICE: pcloud root 'rclone-test-qefufoq7nora': 2 errors while checking lower checkfile vs. upperlowermixed remote (with normalization) 2025/01/14 04:47:33 DEBUG : Couldn't parse Date: from server edef11.pcloud.com: "Tue, 14 Jan 2025 04:47:33 +0000": parsing time "Tue, 14 Jan 2025 04:47:33 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 14 Jan 2025 04:47:33 +0000" as " " 2025/01/14 04:47:33 DEBUG : HeLlO, wOrLd!: md5 = 65a8e27d8879283831b664bd8b7f0ad4 OK 2025/01/14 04:47:33 NOTICE: pcloud root 'rclone-test-qefufoq7nora': 0 differences found 2025/01/14 04:47:33 NOTICE: pcloud root 'rclone-test-qefufoq7nora': 1 matching files 2025/01/14 04:47:33 DEBUG : Creating backend with remote "TestPcloud:rclone-test-royanar1bube" 2025/01/14 04:47:33 DEBUG : Using config file from "/home/rclone/.rclone.conf" panic: test timed out after 1h0m0s running tests: TestApplyTransforms (4s) goroutine 30538 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2366 +0x385 created by time.goFunc /usr/local/go/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive]: testing.(*T).Run(0xc000162340, {0x2444ad4?, 0x0?}, 0x2561508) /usr/local/go/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc000162340) /usr/local/go/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc000162340, 0xc000787c00) /usr/local/go/src/testing/testing.go:1689 +0xfb testing.runTests(0xc000802528, {0x3a13f40, 0x6c, 0x6c}, {0x1?, 0xedf17d555?, 0x3a27be0?}) /usr/local/go/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc0004fa6e0) /usr/local/go/src/testing/testing.go:2027 +0x68b github.com/rclone/rclone/fstest.TestMain(0xc0004fa6e0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0x196 goroutine 29173 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc00039f240) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 29172 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 7 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 8 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 9 [select, 60 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0002c3140) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 27419 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27410 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 30142 [select]: net/http.(*Transport).getConn(0xc0003a0b40, 0xc0002c23c0, {{}, 0x0, {0xc0008215c0, 0x5}, {0xc000586738, 0x13}, 0x0}) /usr/local/go/src/net/http/transport.go:1406 +0x5a5 net/http.(*Transport).roundTrip(0xc0003a0b40, 0xc0009ef200) /usr/local/go/src/net/http/transport.go:595 +0x73a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0003c45b0, 0xc0009ef200) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 golang.org/x/oauth2.(*Transport).RoundTrip(0xc0004ad3e0, 0xc0009ef0e0) /home/rclone/go/pkg/mod/golang.org/x/oauth2@v0.24.0/transport.go:55 +0x3de net/http.send(0xc0009ef0e0, {0x2806280, 0xc0004ad3e0}, {0xc00006a901?, 0x41cc25?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc01a6c9dd0, 0xc0009ef0e0, {0x413fdb?, 0x1?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc01a6c9dd0, 0xc0009ef0e0) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc000665b80, {0x2828380, 0x3a8ad60}, 0xc0003a0c80) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:356 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc000665b80, {0x2828380, 0x3a8ad60}, 0xc00006b1d8?, {0x0?, 0x0?}, {0x1f979c0, 0xc0009eefc0}, 0xc00006b250?, 0x25629a0, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:553 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:496 github.com/rclone/rclone/backend/pcloud.(*Fs).listAll.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/pcloud.go:542 +0x8f github.com/rclone/rclone/fs.pacerInvoker(0x1, 0xa, 0x40?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:88 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0006e0660, 0xc0002c2380, 0xa) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc0006e0660, 0xc0002c2380) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:216 +0x92 github.com/rclone/rclone/backend/pcloud.(*Fs).listAll(0xc000562d20, {0x2828380, 0x3a8ad60}, {0xc000014c60, 0x2}, 0x1, 0x0, 0x0, 0xc00006b5a8) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/pcloud.go:541 +0x3b9 github.com/rclone/rclone/backend/pcloud.(*Fs).FindLeaf(0xc00082045b?, {0x2828380?, 0x3a8ad60?}, {0xc000014c60?, 0x1?}, {0xc00082045b?, 0xc01a6c9ef0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/pcloud.go:462 +0x7a github.com/rclone/rclone/lib/dircache.(*DirCache)._findDir(0xc00053ae10, {0x2828380, 0x3a8ad60}, {0xc00082045b, 0x18}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:234 +0x112 github.com/rclone/rclone/lib/dircache.(*DirCache)._findRoot(0xc00053ae10, {0x2828380?, 0x3a8ad60?}, 0x90?) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:305 +0x45 github.com/rclone/rclone/lib/dircache.(*DirCache).FindRoot(0xc00082045b?, {0x2828380?, 0x3a8ad60?}, 0x2?) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:291 +0xc5 github.com/rclone/rclone/backend/pcloud.NewFs({0x2828380, 0x3a8ad60}, {0xc000820450, 0xa}, {0xc00082045b, 0x18}, {0x2814fb8, 0xc01a6c9b90}) /home/rclone/go/src/github.com/rclone/rclone/backend/pcloud/pcloud.go:349 +0x7bc github.com/rclone/rclone/fs.NewFs({0x2828380, 0x3a8ad60}, {0xc000820450, 0x23}) /home/rclone/go/src/github.com/rclone/rclone/fs/newfs.go:67 +0x33b github.com/rclone/rclone/fstest.RandomRemote() /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:459 +0x51 github.com/rclone/rclone/fstest.newRun() /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:96 +0x73 github.com/rclone/rclone/fstest.newRunIndividual(0xc0003a76c0, 0xc0?) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:143 +0x25 github.com/rclone/rclone/fstest.NewRunIndividual(...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:196 github.com/rclone/rclone/fs/operations_test.TestApplyTransforms.func1({0x2433df5, 0xd}, {0x2433ddb, 0xd}, {0x24935af, 0x2a}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:564 +0xa5 github.com/rclone/rclone/fs/operations_test.TestApplyTransforms(0xc0003a76c0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:601 +0x24a testing.tRunner(0xc0003a76c0, 0x2561508) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 30311 [chan receive]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 77 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 67 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 28899 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28826 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 30144 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc000ac7840) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 27756 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27747 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 30483 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc0001925a0) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:335 +0x156 created by github.com/rclone/rclone/fs/accounting.(*StatsInfo).startAverageLoop.func1 in goroutine 30510 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:367 +0x69 goroutine 30529 [chan receive]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 30312 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0003eda80) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 28423 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28382 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 29004 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28995 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 30537 [IO wait]: internal/poll.runtime_pollWait(0x7f7fb0509660, 0x72) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc0000d3a00?, 0xc0006f1680?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0000d3a00, {0xc0006f1680, 0x240, 0x240}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc0000d3a00, {0xc0006f1680?, 0xc000885688?, 0x41cc25?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000142a60, {0xc0006f1680?, 0x7f7fb023be88?, 0xc0006364c8?}) /usr/local/go/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000636498, {0xc0006f1680?, 0x20d0860?, 0x205?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc0006364c8, {0xc0006f1680?, 0x0?, 0xc0006364c8?}) /usr/local/go/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0002c9eb0, {0x2804440, 0xc0006364c8}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0002c9c08, {0x7f7fb0219298, 0xc000636498}, 0x4bb420?) /usr/local/go/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0002c9c08, 0x0) /usr/local/go/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).readHandshakeBytes(0xc0002c9c08, 0x4) /usr/local/go/src/crypto/tls/conn.go:1075 +0x51 crypto/tls.(*Conn).readHandshake(0xc0002c9c08, {0x0, 0x0}) /usr/local/go/src/crypto/tls/conn.go:1086 +0x34 crypto/tls.(*Conn).clientHandshake(0xc0002c9c08, {0x2828498, 0xc000665db0}) /usr/local/go/src/crypto/tls/handshake_client.go:225 +0x27b crypto/tls.(*Conn).handshakeContext(0xc0002c9c08, {0x2828380, 0x3a8ad60}) /usr/local/go/src/crypto/tls/conn.go:1553 +0x3cb crypto/tls.(*Conn).HandshakeContext(...) /usr/local/go/src/crypto/tls/conn.go:1493 net/http.(*persistConn).addTLS.func2() /usr/local/go/src/net/http/transport.go:1572 +0x6e created by net/http.(*persistConn).addTLS in goroutine 30532 /usr/local/go/src/net/http/transport.go:1568 +0x309 goroutine 30532 [chan receive]: net/http.(*persistConn).addTLS(0xc0009ef440, {0x2828380, 0x3a8ad60}, {0xc000586738, 0xf}, 0x0) /usr/local/go/src/net/http/transport.go:1578 +0x327 net/http.(*Transport).dialConn(0xc0003a0b40, {0x2828380, 0x3a8ad60}, {{}, 0x0, {0xc0008215c0, 0x5}, {0xc000586738, 0x13}, 0x0}) /usr/local/go/src/net/http/transport.go:1657 +0x99c net/http.(*Transport).dialConnFor(0xc0003a0b40, 0xc0003adc30) /usr/local/go/src/net/http/transport.go:1484 +0xca created by net/http.(*Transport).queueForDial in goroutine 30142 /usr/local/go/src/net/http/transport.go:1449 +0x3c9 goroutine 30143 [chan receive]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 27349 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27340 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 27792 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27783 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28302 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28293 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 27914 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27825 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 27488 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27479 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 30347 [chan receive]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 28835 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28810 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 27960 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27935 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28026 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27985 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 27648 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27639 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 29065 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc000ac6240) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 29064 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 30530 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0002c2340) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 28756 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28635 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28669 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28660 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28983 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28926 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28704 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28695 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28116 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28011 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 28215 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28158 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 30348 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0004aef40) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x89 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 30142 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 28493 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 28484 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 27566 [chan receive (nil chan)]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 27557 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 "./operations.test -test.v -test.timeout 1h0m0s -remote TestPcloud: -verbose" - Finished ERROR in 1h0m6.162456557s (try 1/5): exit status 2: Failed [TestMultithreadCopy/upload=true,size=134217727,streams=2 TestMultithreadCopy/upload=true,size=134217728,streams=2 TestMultithreadCopy/upload=true,size=134217729,streams=2]