"./operations.test -test.v -test.timeout 1h0m0s -remote TestKoofr: -verbose" - Starting (try 1/5) 2024/11/09 04:26:45 DEBUG : Creating backend with remote "TestKoofr:rclone-test-qesuxex7jute" 2024/11/09 04:26:45 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2024/11/09 04:26:46 DEBUG : Creating backend with remote "/tmp/rclone4253133895" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" multithread_test.go:121: multithread writing not supported 2024/11/09 04:26:46 ERROR : : error listing: directory not found --- SKIP: TestMultithreadCopy (0.04s) === RUN TestMultithreadCopyAbort run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" multithread_test.go:121: multithread writing not supported 2024/11/09 04:26:46 ERROR : : error listing: directory not found --- SKIP: TestMultithreadCopyAbort (0.04s) === RUN TestSizeDiffers --- PASS: TestSizeDiffers (0.00s) === RUN TestReOpen === RUN TestReOpen/Normal === RUN TestReOpen/Normal/Basics 2024/11/09 04:26:46 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/ErrorAtStart === RUN TestReOpen/Normal/WithErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/Normal/TooManyErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/Normal/Seek 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Seek from 5 to 2 === RUN TestReOpen/Normal/AccountRead === RUN TestReOpen/Normal/AccountReadDelay 2024/11/09 04:26:46 DEBUG : potato: Seek from 10 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 10 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/AccountReadError === RUN TestReOpen/WithRangeOption === RUN TestReOpen/WithRangeOption/Basics 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/ErrorAtStart === RUN TestReOpen/WithRangeOption/WithErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithRangeOption/TooManyErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithRangeOption/Seek 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Seek from 5 to 2 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 4 === RUN TestReOpen/WithRangeOption/AccountRead === RUN TestReOpen/WithRangeOption/AccountReadDelay 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/AccountReadError === RUN TestReOpen/WithSeekOption === RUN TestReOpen/WithSeekOption/Basics 2024/11/09 04:26:46 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/ErrorAtStart === RUN TestReOpen/WithSeekOption/WithErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithSeekOption/TooManyErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithSeekOption/Seek 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Seek from 5 to 2 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 5 === RUN TestReOpen/WithSeekOption/AccountRead === RUN TestReOpen/WithSeekOption/AccountReadDelay 2024/11/09 04:26:46 DEBUG : potato: Seek from 8 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 8 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/AccountReadError === RUN TestReOpen/UnknownSize === RUN TestReOpen/UnknownSize/Basics 2024/11/09 04:26:46 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/ErrorAtStart === RUN TestReOpen/UnknownSize/WithErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/UnknownSize/TooManyErrors 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2024/11/09 04:26:46 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/UnknownSize/Seek 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2024/11/09 04:26:46 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2024/11/09 04:26:46 DEBUG : potato: Seek from 5 to 2 2024/11/09 04:26:46 DEBUG : potato: Seek from 7 to 6 === RUN TestReOpen/UnknownSize/AccountRead === RUN TestReOpen/UnknownSize/AccountReadDelay 2024/11/09 04:26:46 DEBUG : potato: Seek from 9 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 9 to 0 2024/11/09 04:26:46 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/AccountReadError --- PASS: TestReOpen (0.00s) --- PASS: TestReOpen/Normal (0.00s) --- PASS: TestReOpen/Normal/Basics (0.00s) --- PASS: TestReOpen/Normal/ErrorAtStart (0.00s) --- PASS: TestReOpen/Normal/WithErrors (0.00s) --- PASS: TestReOpen/Normal/TooManyErrors (0.00s) --- PASS: TestReOpen/Normal/Seek (0.00s) --- PASS: TestReOpen/Normal/AccountRead (0.00s) --- PASS: TestReOpen/Normal/AccountReadDelay (0.00s) --- PASS: TestReOpen/Normal/AccountReadError (0.00s) --- PASS: TestReOpen/WithRangeOption (0.00s) --- PASS: TestReOpen/WithRangeOption/Basics (0.00s) --- PASS: TestReOpen/WithRangeOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithRangeOption/WithErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/Seek (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountRead (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadError (0.00s) --- PASS: TestReOpen/WithSeekOption (0.00s) --- PASS: TestReOpen/WithSeekOption/Basics (0.00s) --- PASS: TestReOpen/WithSeekOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithSeekOption/WithErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/Seek (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountRead (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadError (0.00s) --- PASS: TestReOpen/UnknownSize (0.00s) --- PASS: TestReOpen/UnknownSize/Basics (0.00s) --- PASS: TestReOpen/UnknownSize/ErrorAtStart (0.00s) --- PASS: TestReOpen/UnknownSize/WithErrors (0.00s) --- PASS: TestReOpen/UnknownSize/TooManyErrors (0.00s) --- PASS: TestReOpen/UnknownSize/Seek (0.00s) --- PASS: TestReOpen/UnknownSize/AccountRead (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadDelay (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadError (0.00s) === RUN TestCheck run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" === RUN TestCheck/1 === RUN TestCheck/2 === RUN TestCheck/3 === RUN TestCheck/4 === RUN TestCheck/5 === RUN TestCheck/6 === RUN TestCheck/7 --- PASS: TestCheck (4.70s) --- PASS: TestCheck/1 (0.20s) --- PASS: TestCheck/2 (0.16s) --- PASS: TestCheck/3 (0.09s) --- PASS: TestCheck/4 (0.06s) --- PASS: TestCheck/5 (0.05s) --- PASS: TestCheck/6 (0.06s) --- PASS: TestCheck/7 (0.05s) === RUN TestCheckFsError 2024/11/09 04:26:50 DEBUG : Creating backend with remote "nonexistent" 2024/11/09 04:26:50 DEBUG : Creating backend with remote "nonexistent" 2024/11/09 04:26:50 DEBUG : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: Waiting for checks to finish 2024/11/09 04:26:50 ERROR : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: error reading source root directory: directory not found 2024/11/09 04:26:50 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 differences found 2024/11/09 04:26:50 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 2 errors while checking --- PASS: TestCheckFsError (0.00s) === RUN TestCheckDownload run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" === RUN TestCheckDownload/1 === RUN TestCheckDownload/2 === RUN TestCheckDownload/3 === RUN TestCheckDownload/4 === RUN TestCheckDownload/5 === RUN TestCheckDownload/6 === RUN TestCheckDownload/7 --- PASS: TestCheckDownload (5.52s) --- PASS: TestCheckDownload/1 (0.17s) --- PASS: TestCheckDownload/2 (0.32s) --- PASS: TestCheckDownload/3 (0.20s) --- PASS: TestCheckDownload/4 (0.33s) --- PASS: TestCheckDownload/5 (0.26s) --- PASS: TestCheckDownload/6 (0.22s) --- PASS: TestCheckDownload/7 (0.17s) === RUN TestCheckSizeOnly run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" === RUN TestCheckSizeOnly/1 === RUN TestCheckSizeOnly/2 === RUN TestCheckSizeOnly/3 === RUN TestCheckSizeOnly/4 === RUN TestCheckSizeOnly/5 === RUN TestCheckSizeOnly/6 === RUN TestCheckSizeOnly/7 --- PASS: TestCheckSizeOnly (5.35s) --- PASS: TestCheckSizeOnly/1 (0.05s) --- PASS: TestCheckSizeOnly/2 (0.05s) --- PASS: TestCheckSizeOnly/3 (0.10s) --- PASS: TestCheckSizeOnly/4 (0.06s) --- PASS: TestCheckSizeOnly/5 (0.13s) --- PASS: TestCheckSizeOnly/6 (0.06s) --- PASS: TestCheckSizeOnly/7 (0.08s) === RUN TestCheckEqualReaders --- PASS: TestCheckEqualReaders (0.00s) === RUN TestParseSumFile run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" 2024/11/09 04:27:03 NOTICE: test.sum: improperly formatted checksum line 4 2024/11/09 04:27:03 NOTICE: test.sum: improperly formatted checksum line 5 2024/11/09 04:27:03 NOTICE: test.sum: improperly formatted checksum line 6 2024/11/09 04:27:03 NOTICE: test.sum: 2 warning(s) suppressed... 2024/11/09 04:27:04 NOTICE: test.sum: improperly formatted checksum line 4 2024/11/09 04:27:04 NOTICE: test.sum: improperly formatted checksum line 5 2024/11/09 04:27:04 NOTICE: test.sum: improperly formatted checksum line 6 2024/11/09 04:27:04 NOTICE: test.sum: 2 warning(s) suppressed... --- PASS: TestParseSumFile (3.79s) === RUN TestCheckSum run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" 2024/11/09 04:27:05 DEBUG : Creating backend with remote "TestKoofr:rclone-test-qesuxex7jute/data" === RUN TestCheckSum/subtest1 === RUN TestCheckSum/subtest2 === RUN TestCheckSum/subtest3 === RUN TestCheckSum/subtest4 === RUN TestCheckSum/subtest5 === RUN TestCheckSum/subtest6 === RUN TestCheckSum/subtest7 --- PASS: TestCheckSum (15.97s) --- PASS: TestCheckSum/subtest1 (0.45s) --- PASS: TestCheckSum/subtest2 (0.51s) --- PASS: TestCheckSum/subtest3 (0.79s) --- PASS: TestCheckSum/subtest4 (0.36s) --- PASS: TestCheckSum/subtest5 (0.45s) --- PASS: TestCheckSum/subtest6 (0.55s) --- PASS: TestCheckSum/subtest7 (0.31s) === RUN TestCheckSumDownload run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" 2024/11/09 04:27:21 DEBUG : Creating backend with remote "TestKoofr:rclone-test-qesuxex7jute/data" === RUN TestCheckSumDownload/subtest1 === RUN TestCheckSumDownload/subtest2 === RUN TestCheckSumDownload/subtest3 === RUN TestCheckSumDownload/subtest4 === RUN TestCheckSumDownload/subtest5 === RUN TestCheckSumDownload/subtest6 === RUN TestCheckSumDownload/subtest7 --- PASS: TestCheckSumDownload (16.03s) --- PASS: TestCheckSumDownload/subtest1 (0.34s) --- PASS: TestCheckSumDownload/subtest2 (0.71s) --- PASS: TestCheckSumDownload/subtest3 (0.96s) --- PASS: TestCheckSumDownload/subtest4 (0.47s) --- PASS: TestCheckSumDownload/subtest5 (0.67s) --- PASS: TestCheckSumDownload/subtest6 (0.68s) --- PASS: TestCheckSumDownload/subtest7 (0.84s) === RUN TestApplyTransforms 2024/11/09 04:27:37 DEBUG : Creating backend with remote "TestKoofr:rclone-test-ficehal9fojo" 2024/11/09 04:27:37 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2024/11/09 04:27:37 DEBUG : Creating backend with remote "/tmp/rclone2153148937" run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-ficehal9fojo", Local "Local file system at /tmp/rclone2153148937", Modify Window "1ms" run.go:285: Failed to put "hello, world!" to "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-ficehal9fojo": Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=hello%2C+world%21&info=true&modified=1324817999123&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-ficehal9fojo": http2: timeout awaiting response headers 2024/11/09 04:32:38 DEBUG : koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-ficehal9fojo: Purge dir "" --- FAIL: TestApplyTransforms (300.91s) === RUN TestTruncateString --- PASS: TestTruncateString (0.00s) === RUN TestCopyFile run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" 2024/11/09 04:32:38 DEBUG : file1: Need to transfer - File not found at Destination 2024/11/09 04:37:39 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 0/10 2024/11/09 04:42:39 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 1/10 2024/11/09 04:47:39 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 2/10 2024/11/09 04:52:40 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 3/10 2024/11/09 04:57:40 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 4/10 2024/11/09 05:02:40 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 5/10 2024/11/09 05:07:40 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 6/10 2024/11/09 05:12:41 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 7/10 2024/11/09 05:17:41 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 8/10 2024/11/09 05:22:41 DEBUG : file1: Received error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers - low level retry 9/10 2024/11/09 05:22:41 ERROR : file1: Failed to copy: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers copy_test.go:113: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:113 Error: Received unexpected error: Post "https:/content/api/v2/mounts/4266cdc3-3d29-4a88-ac94-738914c162b4/files/put?autorename=false&filename=file2&info=true&modified=981173106499&overwrite=true&overwriteIgnoreNonexisting=&path=%2Frclone-test-qesuxex7jute%2Fsub": http2: timeout awaiting response headers Test: TestCopyFile --- FAIL: TestCopyFile (3003.58s) === RUN TestCopyLongFile run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" copy_test.go:153: Test only runs on local --- SKIP: TestCopyLongFile (0.09s) === RUN TestCopyFileBackupDir run.go:180: Remote "koofr:4266cdc3-3d29-4a88-ac94-738914c162b4:rclone-test-qesuxex7jute", Local "Local file system at /tmp/rclone4253133895", Modify Window "1ms" panic: test timed out after 1h0m0s running tests: TestCopyFileBackupDir (4m4s) goroutine 1649 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2366 +0x385 created by time.goFunc /usr/local/go/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive, 4 minutes]: testing.(*T).Run(0xc000524b60, {0x2358428?, 0x0?}, 0x246bd68) /usr/local/go/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc000524b60) /usr/local/go/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc000524b60, 0xc0006b5c00) /usr/local/go/src/testing/testing.go:1689 +0xfb testing.runTests(0xc000906960, {0x38adac0, 0x6c, 0x6c}, {0x1?, 0xedec0db86?, 0x38c18e0?}) /usr/local/go/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc000334f00) /usr/local/go/src/testing/testing.go:2027 +0x68b github.com/rclone/rclone/fstest.TestMain(0xc000334f00) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0x196 goroutine 8 [select]: go.opencensus.io/stats/view.(*worker).start(0xc000311580) /home/rclone/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f created by go.opencensus.io/stats/view.init.0 in goroutine 1 /home/rclone/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d goroutine 11 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 12 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 159 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 150 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 1686 [select, 4 minutes]: net/http.(*http2ClientConn).RoundTrip(0xc000814000, 0xc00083cc60) /usr/local/go/src/net/http/h2_bundle.go:8323 +0x434 net/http.(*http2Transport).RoundTripOpt(0xc0006582d0, 0xc00083cc60, {0x38?}) /usr/local/go/src/net/http/h2_bundle.go:7591 +0x1b1 net/http.(*http2Transport).RoundTrip(...) /usr/local/go/src/net/http/h2_bundle.go:7540 net/http.http2noDialH2RoundTripper.RoundTrip({0xc000470140?}, 0xc00083cc60?) /usr/local/go/src/net/http/h2_bundle.go:10167 +0x1a net/http.(*Transport).roundTrip(0xc000470140, 0xc00083cb40) /usr/local/go/src/net/http/transport.go:553 +0x39c net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:17 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0004801c0, 0xc00083cb40) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:352 +0x345 net/http.send(0xc00083cb40, {0x26f61a0, 0xc0004801c0}, {0x1?, 0x8?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e4 net/http.(*Client).send(0xc000689800, 0xc00083cb40, {0x413fdb?, 0x160?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x98 net/http.(*Client).do(0xc000689800, 0xc00083cb40) /usr/local/go/src/net/http/client.go:724 +0x8dc net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:590 github.com/koofr/go-httpclient.(*HTTPClient).Request(0xc0005a88c0, 0xc000607878) /home/rclone/go/pkg/mod/github.com/koofr/go-httpclient@v0.0.0-20240520111329-e20f8f203988/httpclient.go:346 +0x505 github.com/koofr/go-koofrclient.(*KoofrClient).FilesPutWithOptions(0xc000689830, {0xc0008dd650, 0x24}, {0xc000860660, 0x1d}, {0xc00086067e, 0x5}, {0x26f5280, 0xc000168fc0}, 0xc000607bc0) /home/rclone/go/pkg/mod/github.com/koofr/go-koofrclient@v0.0.0-20221207135200-cbd7fc9ad6a6/client_files.go:285 +0xaad github.com/rclone/rclone/backend/koofr.(*Fs).Put(0xc0008ed680, {0x271a1c0?, 0xc00034d350?}, {0x26f5280, 0xc000168fc0}, {0x27233f0, 0xc000486150}, {0x0, 0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/backend/koofr/koofr.go:424 +0x238 github.com/rclone/rclone/fstest.(*Run).WriteObjectTo(0xc0005781c0, {0x271a1c0, 0xc00034d350}, {0x272e0d0, 0xc0008ed680}, {0x23375b3, 0x9}, {0x234f7c1, 0x12}, {0x1dcd64ff, ...}, ...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:275 +0x418 github.com/rclone/rclone/fstest.(*Run).WriteObject(...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 github.com/rclone/rclone/fs/operations_test.TestCopyFileBackupDir(0xc0000fc4e0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:181 +0x2da testing.tRunner(0xc0000fc4e0, 0x246bd68) /usr/local/go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1742 +0x390 goroutine 869 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 828 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 661 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 556 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 265 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 144 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 325 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 316 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 544 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 535 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 514 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 505 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 201 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 32 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 466 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 457 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 239 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 230 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 286 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 277 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 527 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 518 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 781 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 772 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 426 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 369 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 626 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 569 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 352 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 343 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 952 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 927 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 696 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 591 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 1525 [chan receive, 59 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1524 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab goroutine 1590 [IO wait, 4 minutes]: internal/poll.runtime_pollWait(0x7f638f712ff0, 0x72) /usr/local/go/src/runtime/netpoll.go:345 +0x85 internal/poll.(*pollDesc).wait(0xc0009f8980?, 0xc000b38600?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0009f8980, {0xc000b38600, 0x1300, 0x1300}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a net.(*netFD).Read(0xc0009f8980, {0xc000b38600?, 0xc000095838?, 0x41c385?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000506050, {0xc000b38600?, 0x7f638dce8058?, 0xc000988870?}) /usr/local/go/src/net/net.go:179 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000988180, {0xc000b38600?, 0xc0006800cc?, 0x4?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x2c crypto/tls.(*atLeastReader).Read(0xc000988870, {0xc000b38600?, 0x0?, 0xc000988870?}) /usr/local/go/src/crypto/tls/conn.go:806 +0x3b bytes.(*Buffer).ReadFrom(0xc0004c6630, {0x26f6800, 0xc000988870}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc0004c6388, {0x7f638cbb4a58, 0xc000988180}, 0xc000095960?) /usr/local/go/src/crypto/tls/conn.go:828 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc0004c6388, 0x0) /usr/local/go/src/crypto/tls/conn.go:626 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:588 crypto/tls.(*Conn).Read(0xc0004c6388, {0xc0003e9000, 0x1000, 0x10?}) /usr/local/go/src/crypto/tls/conn.go:1370 +0x156 bufio.(*Reader).Read(0xc000587ec0, {0xc000200d60, 0x9, 0xc000718c20?}) /usr/local/go/src/bufio/bufio.go:241 +0x197 io.ReadAtLeast({0x26f51a0, 0xc000587ec0}, {0xc000200d60, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:335 +0x90 io.ReadFull(...) /usr/local/go/src/io/io.go:354 net/http.http2readFrameHeader({0xc000200d60, 0x9, 0x4857f3?}, {0x26f51a0?, 0xc000587ec0?}) /usr/local/go/src/net/http/h2_bundle.go:1638 +0x65 net/http.(*http2Framer).ReadFrame(0xc000200d20) /usr/local/go/src/net/http/h2_bundle.go:1902 +0x85 net/http.(*http2clientConnReadLoop).run(0xc000095fa8) /usr/local/go/src/net/http/h2_bundle.go:9303 +0x12c net/http.(*http2ClientConn).readLoop(0xc000814000) /usr/local/go/src/net/http/h2_bundle.go:9198 +0x65 created by net/http.(*http2Transport).newClientConn in goroutine 1589 /usr/local/go/src/net/http/h2_bundle.go:7848 +0xca6 goroutine 919 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 670 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 807 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 798 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 845 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 836 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 941 [chan receive (nil chan), 59 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:211 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 932 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:209 +0x378 goroutine 1732 [select, 4 minutes]: net/http.(*http2clientStream).writeRequest(0xc00053f200, 0xc00083cc60) /usr/local/go/src/net/http/h2_bundle.go:8491 +0xacc net/http.(*http2clientStream).doRequest(0xc00053f200, 0xc00079e758?) /usr/local/go/src/net/http/h2_bundle.go:8353 +0x18 created by net/http.(*http2ClientConn).RoundTrip in goroutine 1686 /usr/local/go/src/net/http/h2_bundle.go:8259 +0x2ed "./operations.test -test.v -test.timeout 1h0m0s -remote TestKoofr: -verbose" - Finished ERROR in 1h0m0.373373967s (try 1/5): exit status 2: Failed [TestApplyTransforms TestCopyFile]