"./operations.test -test.v -test.timeout 1h0m0s -remote TestS3Rclone: -verbose -fast-list" - Starting (try 1/5) 2025/10/10 05:42:07 DEBUG : Creating backend with remote "TestS3Rclone:rclone-test-ledowuf8siqa" 2025/10/10 05:42:07 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/10/10 05:42:07 DEBUG : Setting type="s3" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_TYPE 2025/10/10 05:42:07 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 05:42:07 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 05:42:07 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : TestS3Rclone: detected overridden config - adding "{pEvFN}" suffix to name 2025/10/10 05:42:07 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 05:42:07 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 05:42:07 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 05:42:07 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 05:42:07 DEBUG : Creating backend with remote "/tmp/rclone3483282939" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 05:42:07 INFO : S3 bucket rclone-test-ledowuf8siqa: Bucket "rclone-test-ledowuf8siqa" created with ACL "" 2025/10/10 05:42:07 DEBUG : chunksize-probe: open chunk writer: started multipart upload: 28 2025/10/10 05:42:07 DEBUG : chunksize-probe: multipart upload "28" aborted === RUN TestMultithreadCopy/upload=false,size=10485759,streams=2 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: write buffer set to 131072 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: chunk size 64Mi was bigger than source file size 10.000Mi 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: number of streams 4 was bigger than number of chunks 1 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: Starting multi-thread copy with 1 chunks of size 10.000Mi with 1 parallel streams 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: chunk 1/1 (0-10485759) size 10.000Mi starting 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: writing chunk 0 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: multi-thread copy: chunk 1/1 (0-10485759) size 10.000Mi finished 2025/10/10 05:42:08 DEBUG : test-multithread-copy-false-10485759-2: Finished multi-thread copy with 1 parts of size 10.000Mi === RUN TestMultithreadCopy/upload=false,size=10485760,streams=2 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: write buffer set to 131072 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: chunk size 64Mi was bigger than source file size 10Mi 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: number of streams 4 was bigger than number of chunks 1 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: Starting multi-thread copy with 1 chunks of size 10Mi with 1 parallel streams 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: chunk 1/1 (0-10485760) size 10Mi starting 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: writing chunk 0 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: multi-thread copy: chunk 1/1 (0-10485760) size 10Mi finished 2025/10/10 05:42:09 DEBUG : test-multithread-copy-false-10485760-2: Finished multi-thread copy with 1 parts of size 10Mi === RUN TestMultithreadCopy/upload=false,size=10485761,streams=2 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: disabling buffering because destination uses OpenWriterAt 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: write buffer set to 131072 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: chunk size 64Mi was bigger than source file size 10.000Mi 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: number of streams 4 was bigger than number of chunks 1 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: Starting multi-thread copy with 1 chunks of size 10.000Mi with 1 parallel streams 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: chunk 1/1 (0-10485761) size 10.000Mi starting 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: writing chunk 0 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: multi-thread copy: chunk 1/1 (0-10485761) size 10.000Mi finished 2025/10/10 05:42:10 DEBUG : test-multithread-copy-false-10485761-2: Finished multi-thread copy with 1 parts of size 10.000Mi === RUN TestMultithreadCopy/upload=true,size=10485759,streams=2 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: disabling buffering because source is local disk 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: open chunk writer: started multipart upload: 32 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: Starting multi-thread copy with 2 chunks of size 5Mi with 2 parallel streams 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: chunk 2/2 (5242880-10485759) size 5.000Mi starting 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: chunk 1/2 (0-5242880) size 5Mi starting 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: Seek from 5242879 to 0 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: Seek from 5242880 to 0 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: Seek from 5242879 to 0 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: Seek from 5242880 to 0 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multipart upload wrote chunk 2 with 5242879 bytes and etag "f1218df50401de5420640a72f335cb2b" 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: chunk 2/2 (5242880-10485759) size 5.000Mi finished 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multipart upload wrote chunk 1 with 5242880 bytes and etag "e9cc1344e79c111651754a7ce77b6861" 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multi-thread copy: chunk 1/2 (0-5242880) size 5Mi finished 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: multipart upload "32" finished 2025/10/10 05:42:12 DEBUG : test-multithread-copy-true-10485759-2: Finished multi-thread copy with 2 parts of size 5Mi === RUN TestMultithreadCopy/upload=true,size=10485760,streams=2 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: disabling buffering because source is local disk 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: open chunk writer: started multipart upload: 35 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: number of streams 4 was bigger than number of chunks 2 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: Starting multi-thread copy with 2 chunks of size 5Mi with 2 parallel streams 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: chunk 2/2 (5242880-10485760) size 5Mi starting 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: chunk 1/2 (0-5242880) size 5Mi starting 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: Seek from 5242880 to 0 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: Seek from 5242880 to 0 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: Seek from 5242880 to 0 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: Seek from 5242880 to 0 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multipart upload wrote chunk 1 with 5242880 bytes and etag "55f173914a9a414f104a7e33d6bdb3cf" 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: chunk 1/2 (0-5242880) size 5Mi finished 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multipart upload wrote chunk 2 with 5242880 bytes and etag "3ee5d0df00a9ff225430e3836c6f34d3" 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multi-thread copy: chunk 2/2 (5242880-10485760) size 5Mi finished 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: multipart upload "35" finished 2025/10/10 05:42:14 DEBUG : test-multithread-copy-true-10485760-2: Finished multi-thread copy with 2 parts of size 5Mi === RUN TestMultithreadCopy/upload=true,size=10485761,streams=2 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: disabling buffering because source is local disk 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: open chunk writer: started multipart upload: 37 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 2 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Starting multi-thread copy with 3 chunks of size 5Mi with 3 parallel streams 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: chunk 3/3 (10485760-10485761) size 1 starting 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: chunk 1/3 (0-5242880) size 5Mi starting 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: chunk 2/3 (5242880-10485760) size 5Mi starting 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Seek from 1 to 0 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Seek from 1 to 0 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multipart upload wrote chunk 3 with 1 bytes and etag "415290769594460e2e485922904f345d" 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: chunk 3/3 (10485760-10485761) size 1 finished 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Seek from 5242880 to 0 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Seek from 5242880 to 0 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Seek from 5242880 to 0 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Seek from 5242880 to 0 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multipart upload wrote chunk 2 with 5242880 bytes and etag "08aa954db8a436c294320d8070757a58" 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: chunk 2/3 (5242880-10485760) size 5Mi finished 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multipart upload wrote chunk 1 with 5242880 bytes and etag "ab8bf7f0d16b6025e0a2d95f520ff943" 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multi-thread copy: chunk 1/3 (0-5242880) size 5Mi finished 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: multipart upload "37" finished 2025/10/10 05:42:16 DEBUG : test-multithread-copy-true-10485761-2: Finished multi-thread copy with 3 parts of size 5Mi 2025/10/10 05:44:11 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:44:11 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/10 05:46:06 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:46:06 DEBUG : pacer: Rate limited, increasing sleep to 20ms fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 05:48:03 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:48:03 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/10 05:50:03 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:50:03 DEBUG : pacer: Rate limited, increasing sleep to 80ms === NAME TestMultithreadCopy run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Received unexpected error: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused Test: TestMultithreadCopy --- FAIL: TestMultithreadCopy (476.24s) --- PASS: TestMultithreadCopy/upload=false,size=10485759,streams=2 (1.14s) --- PASS: TestMultithreadCopy/upload=false,size=10485760,streams=2 (1.13s) --- PASS: TestMultithreadCopy/upload=false,size=10485761,streams=2 (1.25s) --- PASS: TestMultithreadCopy/upload=true,size=10485759,streams=2 (1.92s) --- PASS: TestMultithreadCopy/upload=true,size=10485760,streams=2 (1.94s) --- FAIL: TestMultithreadCopy/upload=true,size=10485761,streams=2 (231.12s) === RUN TestMultithreadCopyAbort run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 05:51:55 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateMultipartUpload, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Post "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/chunksize-probe?uploads=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:51:55 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/10 05:53:52 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateMultipartUpload, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Post "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/chunksize-probe?uploads=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:53:52 DEBUG : pacer: Rate limited, increasing sleep to 320ms multithread_test.go:139: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:139 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:295 Error: Received unexpected error: create multipart upload failed: operation error S3: CreateMultipartUpload, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Post "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/chunksize-probe?uploads=": dial tcp 127.0.0.1:28624: connect: connection refused Test: TestMultithreadCopyAbort 2025/10/10 05:55:35 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:55:35 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/10 05:57:24 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:24 DEBUG : pacer: Rate limited, increasing sleep to 1.28s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:139 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:295 Error: Received unexpected error: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused Test: TestMultithreadCopyAbort --- FAIL: TestMultithreadCopyAbort (440.56s) === RUN TestSizeDiffers --- PASS: TestSizeDiffers (0.00s) === RUN TestReOpen === RUN TestReOpen/Normal === RUN TestReOpen/Normal/Basics 2025/10/10 05:57:24 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/ErrorAtStart === RUN TestReOpen/Normal/WithErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/Normal/TooManyErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/Normal/Seek 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Seek from 5 to 2 === RUN TestReOpen/Normal/AccountRead === RUN TestReOpen/Normal/AccountReadDelay 2025/10/10 05:57:24 DEBUG : potato: Seek from 10 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 10 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/AccountReadError === RUN TestReOpen/WithRangeOption === RUN TestReOpen/WithRangeOption/Basics 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/ErrorAtStart === RUN TestReOpen/WithRangeOption/WithErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithRangeOption/TooManyErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithRangeOption/Seek 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Seek from 5 to 2 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 4 === RUN TestReOpen/WithRangeOption/AccountRead === RUN TestReOpen/WithRangeOption/AccountReadDelay 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/AccountReadError === RUN TestReOpen/WithSeekOption === RUN TestReOpen/WithSeekOption/Basics 2025/10/10 05:57:24 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/ErrorAtStart === RUN TestReOpen/WithSeekOption/WithErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithSeekOption/TooManyErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithSeekOption/Seek 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Seek from 5 to 2 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 5 === RUN TestReOpen/WithSeekOption/AccountRead === RUN TestReOpen/WithSeekOption/AccountReadDelay 2025/10/10 05:57:24 DEBUG : potato: Seek from 8 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 8 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/AccountReadError === RUN TestReOpen/UnknownSize === RUN TestReOpen/UnknownSize/Basics 2025/10/10 05:57:24 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/ErrorAtStart === RUN TestReOpen/UnknownSize/WithErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/UnknownSize/TooManyErrors 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/10/10 05:57:24 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/UnknownSize/Seek 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/10/10 05:57:24 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/10/10 05:57:24 DEBUG : potato: Seek from 5 to 2 2025/10/10 05:57:24 DEBUG : potato: Seek from 7 to 6 === RUN TestReOpen/UnknownSize/AccountRead === RUN TestReOpen/UnknownSize/AccountReadDelay 2025/10/10 05:57:24 DEBUG : potato: Seek from 9 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 9 to 0 2025/10/10 05:57:24 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/AccountReadError --- PASS: TestReOpen (0.00s) --- PASS: TestReOpen/Normal (0.00s) --- PASS: TestReOpen/Normal/Basics (0.00s) --- PASS: TestReOpen/Normal/ErrorAtStart (0.00s) --- PASS: TestReOpen/Normal/WithErrors (0.00s) --- PASS: TestReOpen/Normal/TooManyErrors (0.00s) --- PASS: TestReOpen/Normal/Seek (0.00s) --- PASS: TestReOpen/Normal/AccountRead (0.00s) --- PASS: TestReOpen/Normal/AccountReadDelay (0.00s) --- PASS: TestReOpen/Normal/AccountReadError (0.00s) --- PASS: TestReOpen/WithRangeOption (0.00s) --- PASS: TestReOpen/WithRangeOption/Basics (0.00s) --- PASS: TestReOpen/WithRangeOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithRangeOption/WithErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/Seek (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountRead (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadError (0.00s) --- PASS: TestReOpen/WithSeekOption (0.00s) --- PASS: TestReOpen/WithSeekOption/Basics (0.00s) --- PASS: TestReOpen/WithSeekOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithSeekOption/WithErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/Seek (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountRead (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadError (0.00s) --- PASS: TestReOpen/UnknownSize (0.00s) --- PASS: TestReOpen/UnknownSize/Basics (0.00s) --- PASS: TestReOpen/UnknownSize/ErrorAtStart (0.00s) --- PASS: TestReOpen/UnknownSize/WithErrors (0.00s) --- PASS: TestReOpen/UnknownSize/TooManyErrors (0.00s) --- PASS: TestReOpen/UnknownSize/Seek (0.00s) --- PASS: TestReOpen/UnknownSize/AccountRead (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadDelay (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadError (0.00s) === RUN TestCheck run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 05:57:24 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:24 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:26 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:28 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:30 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:32 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:34 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:36 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:38 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:40 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 05:57:42 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "rutabaga" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 05:59:36 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:31 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:303 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:98 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:181 Error: Received unexpected error: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused Test: TestCheck --- FAIL: TestCheck (247.02s) === RUN TestCheckFsError 2025/10/10 06:01:31 DEBUG : Creating backend with remote "nonexistent" 2025/10/10 06:01:31 DEBUG : Config file has changed externally - reloading 2025/10/10 06:01:31 DEBUG : Creating backend with remote "nonexistent" 2025/10/10 06:01:31 DEBUG : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: Waiting for checks to finish 2025/10/10 06:01:31 ERROR : Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: error reading source root directory: directory not found 2025/10/10 06:01:31 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 0 differences found 2025/10/10 06:01:31 NOTICE: Local file system at /home/rclone/go/src/github.com/rclone/rclone/fs/operations/nonexistent: 1 errors while checking --- PASS: TestCheckFsError (0.00s) === RUN TestCheckDownload run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:01:31 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:33 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:35 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:37 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:39 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:41 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:43 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:45 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:47 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:01:49 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "rutabaga" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:03:52 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:03:54 DEBUG : pacer: Reducing sleep to 1.5s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:303 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:98 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:204 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCheckDownload --- FAIL: TestCheckDownload (143.16s) === RUN TestCheckSizeOnly run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:03:54 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:03:54 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:03:56 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:03:58 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:00 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:02 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:04 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:06 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:08 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:10 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "rutabaga" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:12 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "rutabaga" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/rutabaga?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:04:14 DEBUG : pacer: Reducing sleep to 1.5s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:303 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:98 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:181 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCheckSizeOnly --- FAIL: TestCheckSizeOnly (20.24s) === RUN TestCheckEqualReaders --- PASS: TestCheckEqualReaders (0.00s) === RUN TestParseSumFile run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:04:16 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:16 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:18 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:20 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:22 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:24 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:26 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:28 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:30 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:32 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "test.sum" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:04:34 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "test.sum" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/test.sum?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:04:36 DEBUG : pacer: Reducing sleep to 1.5s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:308 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestParseSumFile --- FAIL: TestParseSumFile (22.02s) === RUN TestCheckSum run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:04:36 DEBUG : Creating backend with remote "TestS3Rclone:rclone-test-ledowuf8siqa/data" 2025/10/10 06:04:36 DEBUG : Setting type="s3" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_TYPE 2025/10/10 06:04:36 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:04:36 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:04:36 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:04:36 DEBUG : TestS3Rclone: detected overridden config - adding "{pEvFN}" suffix to name 2025/10/10 06:04:36 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:04:36 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:04:36 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:04:36 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:06:32 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:06:32 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/10 06:08:26 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:26 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/10 06:08:26 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:26 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:28 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:30 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:32 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:34 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:36 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:38 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:40 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:42 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:08:44 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "data/banana" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:08:46 DEBUG : pacer: Reducing sleep to 1.5s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:355 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:537 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCheckSum --- FAIL: TestCheckSum (250.14s) === RUN TestCheckSumDownload run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:08:46 DEBUG : Creating backend with remote "TestS3Rclone:rclone-test-ledowuf8siqa/data" 2025/10/10 06:08:46 DEBUG : Setting type="s3" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_TYPE 2025/10/10 06:08:46 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:08:46 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:08:46 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:08:46 DEBUG : TestS3Rclone: detected overridden config - adding "{pEvFN}" suffix to name 2025/10/10 06:08:46 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:08:46 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:08:46 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:08:46 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:10:40 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:10:40 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/10 06:12:30 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/10 06:12:30 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:30 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:32 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:34 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:36 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:38 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:40 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:42 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:44 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:46 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "data/banana" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:12:48 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "data/banana" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/data/banana?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:12:50 DEBUG : pacer: Reducing sleep to 1.5s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:355 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:541 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCheckSumDownload --- FAIL: TestCheckSumDownload (243.22s) === RUN TestApplyTransforms 2025/10/10 06:12:50 DEBUG : Creating backend with remote "TestS3Rclone:rclone-test-riwinan4ruwi" 2025/10/10 06:12:50 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/10/10 06:12:50 DEBUG : Setting type="s3" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_TYPE 2025/10/10 06:12:50 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:12:50 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:12:50 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : TestS3Rclone: detected overridden config - adding "{pEvFN}" suffix to name 2025/10/10 06:12:50 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:12:50 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:12:50 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:12:50 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:12:50 DEBUG : Creating backend with remote "/tmp/rclone3866386681" run.go:180: Remote "S3 bucket rclone-test-riwinan4ruwi", Local "Local file system at /tmp/rclone3866386681", Modify Window "1ns" 2025/10/10 06:14:43 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-riwinan4ruwi": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:14:43 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/10 06:16:49 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-riwinan4ruwi": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:16:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms run.go:237: Failed to mkdir "S3 bucket rclone-test-riwinan4ruwi": operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-riwinan4ruwi": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:16:49 DEBUG : S3 bucket rclone-test-riwinan4ruwi: Purge remote 2025/10/10 06:18:45 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-riwinan4ruwi?versioning=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:18:45 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/10 06:20:48 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-riwinan4ruwi?versioning=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:20:48 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/10 06:20:48 ERROR : S3 bucket rclone-test-riwinan4ruwi: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-riwinan4ruwi?versioning=": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:20:48 DEBUG : S3 bucket rclone-test-riwinan4ruwi: bucket is versioned: false 2025/10/10 06:20:48 DEBUG : Waiting for deletions to finish 2025/10/10 06:22:48 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-riwinan4ruwi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:22:48 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/10 06:24:34 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-riwinan4ruwi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:24:34 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/10 06:26:25 DEBUG : pacer: low level retry 1/2 (error operation error S3: DeleteBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Delete "http://127.0.0.1:28624/rclone-test-riwinan4ruwi": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:26:25 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/10 06:28:14 DEBUG : pacer: low level retry 2/2 (error operation error S3: DeleteBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Delete "http://127.0.0.1:28624/rclone-test-riwinan4ruwi": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:14 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/10/10 06:28:14 NOTICE: purge failed: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-riwinan4ruwi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused --- FAIL: TestApplyTransforms (924.83s) === RUN TestTruncateString --- PASS: TestTruncateString (0.00s) === RUN TestCopyFile run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:108 Error: Should be true Test: TestCopyFile Messages: listing wrong, want file1 (14) got file1 (14), rutabaga (8), test-multithread-copy-true-10485761-2 (10485761) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:108 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFile Messages: Unexpected file "rutabaga" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:108 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFile Messages: Unexpected file "test-multithread-copy-true-10485761-2" 2025/10/10 06:28:22 DEBUG : pacer: Reducing sleep to 1.125s copy_test.go:114: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:114 Error: Received unexpected error: operation error S3: HeadObject, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/sub/file2": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCopyFile 2025/10/10 06:28:23 DEBUG : pacer: Reducing sleep to 843.75ms run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:114 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCopyFile --- FAIL: TestCopyFile (8.51s) === RUN TestCopyLongFile run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" copy_test.go:154: Test only runs on local 2025/10/10 06:28:24 DEBUG : pacer: Reducing sleep to 632.8125ms run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1156 /usr/local/go/src/testing/testing.go:1132 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:154 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCopyLongFile --- FAIL: TestCopyLongFile (1.13s) === RUN TestCopyFileBackupDir run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:180 Error: Should be true Test: TestCopyFileBackupDir Messages: listing wrong, want dst/file1 (14) got dst/file1 (14), file1 (14), rutabaga (8), test-multithread-copy-true-10485761-2 (10485761) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:180 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileBackupDir Messages: Unexpected file "file1" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:180 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileBackupDir Messages: Unexpected file "rutabaga" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:180 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileBackupDir Messages: Unexpected file "test-multithread-copy-true-10485761-2" 2025/10/10 06:28:31 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:31 DEBUG : pacer: Rate limited, increasing sleep to 1.265625s run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:33 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:33 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:35 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:37 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:39 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:41 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:43 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:45 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:47 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:281: Retry Put of "dst/file1" to S3 bucket rclone-test-ledowuf8siqa: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:28:49 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused) run.go:285: Failed to put "dst/file1" to "S3 bucket rclone-test-ledowuf8siqa": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/file1?x-id=PutObject": dial tcp 127.0.0.1:28624: connect: connection refused 2025/10/10 06:28:51 DEBUG : pacer: Reducing sleep to 1.5s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:182 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCopyFileBackupDir --- FAIL: TestCopyFileBackupDir (27.03s) === RUN TestCopyFileCompareDest run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:28:51 DEBUG : Creating backend with remote "TestS3Rclone:rclone-test-ledowuf8siqa/dst" 2025/10/10 06:28:51 DEBUG : Config file has changed externally - reloading 2025/10/10 06:28:51 DEBUG : Setting type="s3" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_TYPE 2025/10/10 06:28:51 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:28:51 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:28:51 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:28:51 DEBUG : TestS3Rclone: detected overridden config - adding "{pEvFN}" suffix to name 2025/10/10 06:28:51 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:28:51 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:28:51 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:28:51 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:30:41 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:30:41 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/10 06:32:44 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:32:44 DEBUG : pacer: Rate limited, increasing sleep to 20ms fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:204 Error: Should be true Test: TestCopyFileCompareDest Messages: listing wrong, want one (3) got dst/file1 (14), file1 (14), one (3), rutabaga (8), test-multithread-copy-true-10485761-2 (10485761) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:204 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCompareDest Messages: Unexpected file "file1" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:204 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCompareDest Messages: Unexpected file "rutabaga" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:204 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCompareDest Messages: Unexpected file "test-multithread-copy-true-10485761-2" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:204 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCompareDest Messages: Unexpected file "dst/file1" 2025/10/10 06:34:54 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/one": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:34:54 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/10 06:36:42 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/one": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:36:42 DEBUG : pacer: Rate limited, increasing sleep to 80ms copy_test.go:207: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:207 Error: Received unexpected error: operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst/one": dial tcp 127.0.0.1:28624: connect: connection refused Test: TestCopyFileCompareDest 2025/10/10 06:36:42 DEBUG : pacer: Reducing sleep to 1.125s run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:207 Error: Received unexpected error: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28624/rclone-test-ledowuf8siqa?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp 127.0.0.1:28624: connect: connection refused failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Test: TestCopyFileCompareDest --- FAIL: TestCopyFileCompareDest (470.42s) === RUN TestCopyFileCopyDest run.go:180: Remote "S3 bucket rclone-test-ledowuf8siqa", Local "Local file system at /tmp/rclone3483282939", Modify Window "1ns" 2025/10/10 06:36:42 DEBUG : Creating backend with remote "TestS3Rclone:rclone-test-ledowuf8siqa/dst" 2025/10/10 06:36:42 DEBUG : Setting type="s3" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_TYPE 2025/10/10 06:36:42 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:36:42 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:36:42 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:36:42 DEBUG : TestS3Rclone: detected overridden config - adding "{pEvFN}" suffix to name 2025/10/10 06:36:42 DEBUG : Setting provider="Rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_PROVIDER 2025/10/10 06:36:42 DEBUG : Setting access_key_id="rclone" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ACCESS_KEY_ID 2025/10/10 06:36:42 DEBUG : Setting secret_access_key="JoltRogueVerde5" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_SECRET_ACCESS_KEY 2025/10/10 06:36:42 DEBUG : Setting endpoint="http://127.0.0.1:28624/" for "TestS3Rclone" from environment variable RCLONE_CONFIG_TESTS3RCLONE_ENDPOINT 2025/10/10 06:38:33 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:38:33 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/10 06:40:21 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28624/rclone-test-ledowuf8siqa/dst": dial tcp 127.0.0.1:28624: connect: connection refused) 2025/10/10 06:40:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:287 Error: Should be true Test: TestCopyFileCopyDest Messages: listing wrong, want one (3) got dst/file1 (14), file1 (14), one (3), rutabaga (8), test-multithread-copy-true-10485761-2 (10485761) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:287 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCopyDest Messages: Unexpected file "file1" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:287 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCopyDest Messages: Unexpected file "rutabaga" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:287 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCopyDest Messages: Unexpected file "test-multithread-copy-true-10485761-2" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:334 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:287 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestCopyFileCopyDest Messages: Unexpected file "dst/file1" panic: test timed out after 1h0m0s running tests: TestCopyFileCopyDest (5m25s) goroutine 1477 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 6 minutes]: testing.(*T).Run(0xc000582c40, {0x27a4214?, 0xc000671ad8?}, 0x28be720) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc000582c40) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc000582c40, 0xc000671c18) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc00069fbd8, {0x4001c40, 0x6e, 0x6e}, {0xee07a93af?, 0x4017580?, 0x4017580?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc0004217c0) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc0004217c0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:267 +0xa9 goroutine 15 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 16 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 1371 [select]: github.com/aws/aws-sdk-go-v2/internal/sdk.sleepWithContext({0x2c0dda8, 0xc0005ac0c0}, 0x2bf39c0?) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/sdk/time.go:36 +0xc5 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).handleAttempt(0xc000573b30, {0x2c0dda8, 0xc0005ac0c0}, {{0x272b2c0?, 0xc00030fe30?}}, 0xc0001213e0, {0x2bef420, 0xc0003fc140}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:276 +0x9c8 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).HandleFinalize(0xc000573b30, {0x2c0dda8, 0xc00068f830}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef420, 0xc0003fc140}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:121 +0x525 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptBeforeRetryLoop).HandleFinalize(0xc00064d938?, {0x2c0dda8, 0xc00068f830}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef420, 0xc0003fc160}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:131 +0x10a github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanRetryLoop).HandleFinalize(0xc00064da40?, {0x2c0dda8, 0xc00068f830}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef420, 0xc0003fc180}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:693 +0x129 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize(0x41821f?, {0x2c0dda8, 0xc00068f830}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef420, 0xc0003fc1a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1/accept_encoding_gzip.go:67 +0x15e github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware(0x2c0dda8?, {0x2c0dda8, 0xc00068f830}, {0x272b2c0, 0xc00068f3b0}, {0x2bee540, 0xc00035ba00}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestEnd).HandleBuild(0x4982a0?, {0x2c0dda8?, 0xc00068f830?}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef480, 0xc0001163e0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1444 +0x8b github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RecursionDetection).HandleBuild(0x14ffdd8?, {0x2c0dda8, 0xc00068f830}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef4a0, 0xc00035be20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/recursion_detection.go:46 +0x258 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/internal/middleware.AddTimeOffsetMiddleware.HandleBuild({0x27863dc?}, {0x2c0dda8?, 0xc00068f6b0?}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef4a0?, 0xc00035be40?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/middleware/middleware.go:30 +0xb2 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RequestUserAgent).HandleBuild(0x0?, {0x2c0dda8, 0xc00068f6b0}, {{0x272b2c0, 0xc00068f3b0}}, {0x2bef4a0, 0xc00035be60}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/user_agent.go:343 +0x8d github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/service/s3.setCredentialSourceMiddleware.HandleBuild({0xc0003dc4c8, {{0x403ab60, 0x0, 0x0}, {0x0, 0x0}, 0xc000959b50, 0x0, 0x0, {0x2be7da0, ...}, ...}}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:916 +0x148 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild(0x194154393335e59c?, {0x2c0dda8, 0xc00068f6b0}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef4a0, 0xc00035bea0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/middleware_content_length.go:49 +0x12a github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild({}, {0x2c0dda8, 0xc00068f6b0}, {{0x272b2c0?, 0xc00068f3b0?}}, {0x2bef4a0, 0xc00035bec0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/middleware.go:42 +0x1ed github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware(0x6?, {0x2c0dda8, 0xc00068f6b0}, {0x272b2c0, 0xc00068f3b0}, {0x2bee540, 0xc00035ba60}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:192 github.com/aws/aws-sdk-go-v2/service/s3.(*isExpressUserAgent).HandleSerialize(0xc00003c0c8, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef380, 0xc0001163c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/express_user_agent.go:33 +0xf5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*removeBucketFromPathMiddleware).HandleSerialize(0x22df2c0?, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bc60}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/remove_bucket_middleware.go:26 +0x1fd github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*updateEndpoint).HandleSerialize(0xc0003dc4e0, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bc80}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/update_endpoint.go:145 +0x52b github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptAfterSerialization).HandleSerialize(0xc0003dc558, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bca0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:106 +0x1d0 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_serializeOpHeadObject).HandleSerialize(0x2399ee0?, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bce0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serializers.go:5025 +0x86a github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptBeforeSerialization).HandleSerialize(0x22df2c0?, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bd00}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:79 +0x146 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*serializeImmutableHostnameBucketMiddleware).HandleSerialize(0xc00049449d, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bd20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serialize_immutable_hostname_bucket.go:53 +0x3a6 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*s3ObjectLambdaEndpoint).HandleSerialize(0xc0005769c0, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bd40}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/s3_object_lambda.go:34 +0x697 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*processARNResource).HandleSerialize(0xc000576900, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bd60}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/process_arn_resource.go:54 +0x1219 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*ResolveEndpoint).HandleSerialize(0xc00098e230, {0x2c0dda8, 0xc00068f6b0}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bd80}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/endpoints.go:89 +0x755 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*setOperationInputMiddleware).HandleSerialize(0x403ab60?, {0x2c0dda8?, 0xc00068f620?}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bda0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:355 +0xb5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*putBucketContextMiddleware).HandleSerialize(0x41b074?, {0x2c0dda8, 0xc00068f380}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bdc0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/bucket_context.go:26 +0xdc github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestStart).HandleSerialize(0x90?, {0x2c0dda8?, 0xc00068f380?}, {{0x2399ee0?, 0xc00043e540?}, {0x272b2c0?, 0xc00068f3b0?}}, {0x2bef3a0, 0xc00035bde0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1426 +0xaf github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware(0xc000116310, {0x2c0dda8, 0xc00068f380}, {0x2399ee0, 0xc00043e540}, {0x2bee540, 0xc00035ba80}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:122 +0x26f github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeEnd).HandleInitialize(0xc00003c090?, {0x2c0dda8?, 0xc00068f380?}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3c0, 0xc000116380}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1409 +0x8b github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*validateOpHeadObject).HandleInitialize(0xc00003c001?, {0x2c0dda8, 0xc00068f380}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3e0, 0xc00035bb40}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/validators.go:1210 +0xc6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize(0x0?, {0x2c0dda8?, 0xc00068f350?}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3e0, 0xc00035bb60}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/logging.go:45 +0x8c github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*ARNLookup).HandleInitialize(0xc00068f350?, {0x2c0dda8, 0xc00068f350}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3e0, 0xc00035bb80}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.8/arn_lookup.go:39 +0x262 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*legacyEndpointContextSetter).HandleInitialize(0x14ffdd8?, {0x2c0dda8?, 0xc00068f350?}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3e0?, 0xc00035bba0?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:418 +0x9e github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize({{0x2771f32, 0x2}, {0x0, 0x0}, {0x27842ad, 0x9}, {0x2787782, 0xa}}, {0x2c0dda8, 0xc00068f200}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/metadata.go:41 +0x1c6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptExecution).HandleInitialize(0xc000573ec0, {0x2c0dda8, 0xc00068f0e0}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3e0, 0xc00035bc00}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:46 +0x1f0 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeStart).HandleInitialize(0x41821f?, {0x2c0dda8?, 0xc00068f0e0?}, {{0x2399ee0?, 0xc00043e540?}}, {0x2bef3e0, 0xc00035bc20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1391 +0x85 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware(0xc0001549c0?, {0x2c0dda8, 0xc00068f0e0}, {0x2399ee0, 0xc00043e540}, {0x2bee540, 0xc00035bae0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware(0x418201?, {0x2c0dda8, 0xc00068f0e0}, {0x2399ee0, 0xc00043e540}, {0x2bee4e0?, 0xc00035b9c0?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/stack.go:109 +0x183 github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).invokeOperation(0xc0006d6a88, {0x2c0dda8?, 0xc0006a13b0?}, {0x2787782, 0xa}, {0x2399ee0, 0xc00043e540}, {0x0, 0x0, 0x0}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:305 +0xb75 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).HeadObject(0xc000651760?, {0x2c0dda8?, 0xc0006a13b0?}, 0x41b074?, {0x0?, 0xc0006517d8?, 0x41373d?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_op_HeadObject.go:147 +0x10c github.com/rclone/rclone/backend/s3.(*Fs).headObject.func1() /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6074 +0x4e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x2, 0x30?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:86 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0032071a0, 0xc0005735f0, 0x2) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:232 +0xc5 github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc0032071a0, 0xc0005735f0) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:251 +0x93 github.com/rclone/rclone/backend/s3.(*Fs).headObject(0xc0004bc308, {0x2c0dda8, 0xc0006a13b0}, 0xc00043e540) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6072 +0x1ae github.com/rclone/rclone/backend/s3.(*Object).headObject(0xc000698000, {0x2c0dda8, 0xc0006a13b0}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6056 +0x129 github.com/rclone/rclone/backend/s3.(*Object).readMetaData(0xc000698000, {0x2c0dda8?, 0xc0006a13b0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6096 +0x29 github.com/rclone/rclone/backend/s3.(*Fs).newObjectWithInfo(0xc0004bc308, {0x2c0dda8, 0xc0006a13b0}, {0x277221d, 0x3}, 0x0, 0x0) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:4190 +0x505 github.com/rclone/rclone/backend/s3.(*Fs).NewObject(0xc000651c00?, {0x2c0dda8?, 0xc0006a13b0?}, {0x277221d?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:4201 +0x28 github.com/rclone/rclone/fs/operations.moveOrCopyFile({0x2c0dda8, 0xc0006a13b0}, {0x2c258f0, 0xc0004bc308}, {0x2c24a80, 0xc00043e240}, {0x277221d, 0x3}, {0x277221d, 0x3}, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations.go:2029 +0x49b github.com/rclone/rclone/fs/operations.CopyFile(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy.go:421 github.com/rclone/rclone/fs/operations_test.TestCopyFileCopyDest(0xc0006081c0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy_test.go:289 +0x35c testing.tRunner(0xc0006081c0, 0x28be720) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 53 [chan receive, 59 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 19 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 646 [chan receive, 29 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 645 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 669 [chan receive (nil chan), 42 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:257 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 660 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:255 +0x378 "./operations.test -test.v -test.timeout 1h0m0s -remote TestS3Rclone: -verbose -fast-list" - Finished ERROR in 1h0m0.061144446s (try 1/5): exit status 2: Failed [TestMultithreadCopy/upload=true,size=10485761,streams=2 TestMultithreadCopyAbort TestCheck TestCheckDownload TestCheckSizeOnly TestParseSumFile TestCheckSum TestCheckSumDownload TestApplyTransforms TestCopyFile TestCopyLongFile TestCopyFileBackupDir TestCopyFileCompareDest]