"./bisync.test -test.v -test.timeout 1h0m0s -remote TestS3MinioEdge: -verbose" - Starting (try 2/5) 2025/10/26 02:01:07 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-diyajuz8baro" 2025/10/26 02:01:07 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/10/26 02:01:07 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:01:07 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:01:07 DEBUG : Creating backend with remote "/tmp/rclone1345631169" === RUN TestBisyncRemoteLocal 2025/10/26 02:01:07 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi" 2025/10/26 02:01:07 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:01:07 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:01:07 NOTICE: remote: TestS3MinioEdge:rclone-test-tecolid1fuzi === RUN TestBisyncRemoteLocal/all_changed 2025/10/26 02:01:07 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li" 2025/10/26 02:01:07 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:01:07 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:03:03 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:03:03 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 02:04:57 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:04:57 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 02:04:57 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tecolid1fuzi/020107li" 2025/10/26 02:04:57 INFO : path1: Making directory 2025/10/26 02:06:47 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:06:47 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 02:08:49 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:08:49 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/all_changed Messages: Mkdir path1 2025/10/26 02:08:49 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li/all_changed/path1" 2025/10/26 02:08:49 DEBUG : Config file has changed externally - reloading 2025/10/26 02:08:49 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:08:49 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:08:49 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:08:49 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:08:49 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:08:49 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:08:49 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:08:49 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:08:49 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:08:49 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:10:35 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li/all_changed/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:10:35 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 02:12:33 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li/all_changed/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:12:33 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 02:12:33 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li/all_changed/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tecolid1fuzi/020107li/all_changed/path1" 2025/10/26 02:12:33 DEBUG : Creating backend with remote "/tmp/020107li" 2025/10/26 02:12:33 DEBUG : Config file has changed externally - reloading 2025/10/26 02:12:33 DEBUG : Creating backend with remote "/tmp/020107li/all_changed/path2" 2025/10/26 02:12:33 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_all_changed/initial" 2025/10/26 02:12:33 DEBUG : Creating backend with remote "/tmp/020107li/initdir/test_all_changed-pubeneh4" 2025/10/26 02:12:33 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : file1.copy1.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : file1.copy2.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : file1.copy3.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : file1.copy4.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : file1.copy5.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : subdir: Making directory with metadata 2025/10/26 02:12:33 INFO : subdir: Made directory with metadata (mtime=2021-11-02T05:00:07Z) 2025/10/26 02:12:33 DEBUG : Added delayed dir = "subdir", newDst=subdir 2025/10/26 02:12:33 DEBUG : subdir/file20.txt: Need to transfer - File not found at Destination 2025/10/26 02:12:33 DEBUG : Local file system at /tmp/020107li/initdir/test_all_changed-pubeneh4: Waiting for checks to finish 2025/10/26 02:12:33 DEBUG : Local file system at /tmp/020107li/initdir/test_all_changed-pubeneh4: Waiting for transfers to finish 2025/10/26 02:12:33 DEBUG : file1.copy2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : file1.copy2.txt.455881d5.partial: renamed to: file1.copy2.txt 2025/10/26 02:12:33 INFO : file1.copy2.txt: Copied (new) 2025/10/26 02:12:33 DEBUG : file1.copy1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : file1.copy3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 02:12:33 DEBUG : file1.copy3.txt.d0285540.partial: renamed to: file1.copy3.txt 2025/10/26 02:12:33 INFO : file1.copy3.txt: Copied (new) 2025/10/26 02:12:33 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 02:12:33 INFO : RCLONE_TEST: Copied (new) 2025/10/26 02:12:33 DEBUG : file1.copy4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : file1.copy4.txt.8c987629.partial: renamed to: file1.copy4.txt 2025/10/26 02:12:33 INFO : file1.copy4.txt: Copied (new) 2025/10/26 02:12:33 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : file1.copy5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : subdir/file20.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:12:33 DEBUG : file1.copy5.txt.19e8a2bc.partial: renamed to: file1.copy5.txt 2025/10/26 02:12:33 INFO : file1.copy5.txt: Copied (new) 2025/10/26 02:12:33 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 02:12:33 DEBUG : subdir/file20.txt.1ac3db2f.partial: renamed to: subdir/file20.txt 2025/10/26 02:12:33 INFO : subdir/file20.txt: Copied (new) 2025/10/26 02:12:33 INFO : file1.txt: Copied (new) 2025/10/26 02:12:33 DEBUG : file1.copy1.txt.21b8fa2b.partial: renamed to: file1.copy1.txt 2025/10/26 02:12:33 INFO : file1.copy1.txt: Copied (new) 2025/10/26 02:12:33 INFO : subdir: Set directory modification time (using SetModTime) 2025/10/26 02:14:25 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:14:25 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 02:16:12 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:16:12 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/26 02:16:12 ERROR : S3 bucket rclone-test-tecolid1fuzi path 020107li/all_changed/path1: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused 2025/10/26 02:16:12 DEBUG : S3 bucket rclone-test-tecolid1fuzi path 020107li/all_changed/path1: bucket is versioned: false 2025/10/26 02:16:12 DEBUG : Waiting for deletions to finish 2025/10/26 02:18:00 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:18:00 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/26 02:19:53 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:19:53 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/26 02:19:53 DEBUG : Waiting for deletions to finish 2025/10/26 02:19:53 ERROR : error listing: directory not found 2025/10/26 02:19:53 ERROR : Local file system at /tmp/020107li/all_changed/path2: Failed to list "": directory not found 2025/10/26 02:19:53 DEBUG : removing 1 level 0 directories 2025/10/26 02:19:53 INFO : Local file system at /tmp/020107li/all_changed/path2: Removing directory 2025/10/26 02:19:53 ERROR : Failed to rmdir: stat /tmp/020107li/all_changed/path2: no such file or directory 2025/10/26 02:21:48 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:21:48 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/26 02:23:51 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:23:51 DEBUG : pacer: Rate limited, increasing sleep to 1.28s fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused === RUN TestBisyncRemoteLocal/backupdir 2025/10/26 02:23:51 DEBUG : fs cache: switching user supplied name "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li" for canonical name "TestS3MinioEdge{FbV4e}:rclone-test-tecolid1fuzi/020107li" 2025/10/26 02:23:51 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li" 2025/10/26 02:23:51 DEBUG : Config file has changed externally - reloading 2025/10/26 02:23:51 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:23:51 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:23:51 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:23:51 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:23:51 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:23:51 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:23:51 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:23:51 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:23:51 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:23:51 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:25:54 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:25:54 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 02:27:51 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:27:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 02:27:51 INFO : path1: Making directory 2025/10/26 02:29:44 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:29:44 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 02:31:45 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:31:45 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/backupdir Messages: Mkdir path1 2025/10/26 02:31:45 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li/backupdir/path1" 2025/10/26 02:31:45 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:31:45 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:31:45 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:31:45 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:31:45 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:31:45 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:31:45 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:31:45 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:31:45 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:31:45 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:33:44 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li/backupdir/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:33:44 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 02:35:42 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li/backupdir/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:35:42 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 02:35:42 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li/backupdir/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tecolid1fuzi/020107li/backupdir/path1" 2025/10/26 02:35:42 DEBUG : Creating backend with remote "/tmp/020107li" 2025/10/26 02:35:42 DEBUG : Creating backend with remote "/tmp/020107li/backupdir/path2" 2025/10/26 02:35:42 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_backupdir/initial" 2025/10/26 02:35:42 DEBUG : Creating backend with remote "/tmp/020107li/initdir/test_backupdir-golaxar3" 2025/10/26 02:35:42 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file2.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file3.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file4.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file5.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file6.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file7.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file8.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : Local file system at /tmp/020107li/initdir/test_backupdir-golaxar3: Waiting for checks to finish 2025/10/26 02:35:42 DEBUG : file3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : Local file system at /tmp/020107li/initdir/test_backupdir-golaxar3: Waiting for transfers to finish 2025/10/26 02:35:42 DEBUG : file3.txt.a74079f2.partial: renamed to: file3.txt 2025/10/26 02:35:42 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : file2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 INFO : file3.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 02:35:42 INFO : file1.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file2.txt.3230ad67.partial: renamed to: file2.txt 2025/10/26 02:35:42 INFO : file2.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : file4.txt.fbf05a9b.partial: renamed to: file4.txt 2025/10/26 02:35:42 INFO : file4.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : file6.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : file5.txt.6e808e0e.partial: renamed to: file5.txt 2025/10/26 02:35:42 INFO : file5.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file7.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : file6.txt.0a60f5f0.partial: renamed to: file6.txt 2025/10/26 02:35:42 INFO : file6.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file7.txt.9f102165.partial: renamed to: file7.txt 2025/10/26 02:35:42 INFO : file7.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file8.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:35:42 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 02:35:42 DEBUG : file8.txt.b300b322.partial: renamed to: file8.txt 2025/10/26 02:35:42 INFO : file8.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 02:35:42 INFO : RCLONE_TEST: Copied (new) 2025/10/26 02:35:42 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_backupdir/modfiles" 2025/10/26 02:35:42 DEBUG : Creating backend with remote "/tmp/020107li/datadir/test_backupdir-lunabax6" 2025/10/26 02:35:42 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file10.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file11.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file2.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file5L.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file5R.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file6.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : file7.txt: Need to transfer - File not found at Destination 2025/10/26 02:35:42 DEBUG : Local file system at /tmp/020107li/datadir/test_backupdir-lunabax6: Waiting for checks to finish 2025/10/26 02:35:42 DEBUG : Local file system at /tmp/020107li/datadir/test_backupdir-lunabax6: Waiting for transfers to finish 2025/10/26 02:35:42 DEBUG : file1.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 02:35:42 DEBUG : file1.txt.75b77f44.partial: renamed to: file1.txt 2025/10/26 02:35:42 INFO : file1.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file2.txt: md5 = fb3ecfb2800400fb01b0bfd39903e9fb OK 2025/10/26 02:35:42 DEBUG : file11.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 02:35:42 DEBUG : file2.txt.4526d70e.partial: renamed to: file2.txt 2025/10/26 02:35:42 INFO : file2.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file11.txt.40d797d4.partial: renamed to: file11.txt 2025/10/26 02:35:42 INFO : file11.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file10.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 02:35:42 DEBUG : file10.txt.c02780cb.partial: renamed to: file10.txt 2025/10/26 02:35:42 INFO : file10.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file5L.txt: md5 = 0860a03592626642f8fd6c8bfb447d2a OK 2025/10/26 02:35:42 DEBUG : file5L.txt.8dd6541a.partial: renamed to: file5L.txt 2025/10/26 02:35:42 INFO : file5L.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file5R.txt: md5 = 979a803b15d27df0c31ad7d29006d10b OK 2025/10/26 02:35:42 DEBUG : file5R.txt.ee3b9525.partial: renamed to: file5R.txt 2025/10/26 02:35:42 INFO : file5R.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file6.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 02:35:42 DEBUG : file6.txt.4185169b.partial: renamed to: file6.txt 2025/10/26 02:35:42 INFO : file6.txt: Copied (new) 2025/10/26 02:35:42 DEBUG : file7.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 02:35:42 DEBUG : file7.txt.c1750184.partial: renamed to: file7.txt 2025/10/26 02:35:42 INFO : file7.txt: Copied (new) 2025/10/26 02:37:38 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:37:38 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 02:39:41 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:39:41 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/26 02:39:41 ERROR : S3 bucket rclone-test-tecolid1fuzi path 020107li/backupdir/path1: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused 2025/10/26 02:39:41 DEBUG : S3 bucket rclone-test-tecolid1fuzi path 020107li/backupdir/path1: bucket is versioned: false 2025/10/26 02:39:41 DEBUG : Waiting for deletions to finish 2025/10/26 02:41:35 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:41:35 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/26 02:43:37 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:43:37 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/26 02:43:37 DEBUG : Waiting for deletions to finish 2025/10/26 02:43:37 ERROR : error listing: directory not found 2025/10/26 02:43:37 ERROR : Local file system at /tmp/020107li/backupdir/path2: Failed to list "": directory not found 2025/10/26 02:43:37 DEBUG : removing 1 level 0 directories 2025/10/26 02:43:37 INFO : Local file system at /tmp/020107li/backupdir/path2: Removing directory 2025/10/26 02:43:37 ERROR : Failed to rmdir: stat /tmp/020107li/backupdir/path2: no such file or directory 2025/10/26 02:45:32 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:45:32 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/26 02:47:28 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:47:28 DEBUG : pacer: Rate limited, increasing sleep to 1.28s fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=020107li%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused === RUN TestBisyncRemoteLocal/basic 2025/10/26 02:47:28 DEBUG : fs cache: switching user supplied name "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li" for canonical name "TestS3MinioEdge{FbV4e}:rclone-test-tecolid1fuzi/020107li" 2025/10/26 02:47:28 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li" 2025/10/26 02:47:28 DEBUG : Config file has changed externally - reloading 2025/10/26 02:47:28 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:47:28 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:47:28 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:47:28 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:47:28 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:47:28 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:47:28 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:47:28 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:47:28 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:47:28 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:49:23 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:49:23 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 02:51:19 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:51:19 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 02:51:19 INFO : path1: Making directory 2025/10/26 02:53:09 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:53:09 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 02:55:12 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:55:12 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tecolid1fuzi": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/basic Messages: Mkdir path1 2025/10/26 02:55:12 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li/basic/path1" 2025/10/26 02:55:12 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 02:55:12 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:55:12 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:55:12 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:55:12 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:55:12 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 02:55:12 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 02:55:12 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 02:55:12 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 02:55:12 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 02:56:59 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li/basic/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:56:59 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 02:58:52 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tecolid1fuzi/020107li/basic/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 02:58:52 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 02:58:52 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tecolid1fuzi/020107li/basic/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tecolid1fuzi/020107li/basic/path1" 2025/10/26 02:58:52 DEBUG : Creating backend with remote "/tmp/020107li" 2025/10/26 02:58:52 DEBUG : Config file has changed externally - reloading 2025/10/26 02:58:52 DEBUG : Creating backend with remote "/tmp/020107li/basic/path2" 2025/10/26 02:58:52 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_basic/initial" 2025/10/26 02:58:52 DEBUG : Creating backend with remote "/tmp/020107li/initdir/test_basic-piyaqax0" 2025/10/26 02:58:52 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.copy1.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.copy2.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.copy3.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.copy4.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.copy5.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : subdir: Making directory with metadata 2025/10/26 02:58:52 INFO : subdir: Made directory with metadata (mtime=2021-11-02T05:00:07Z) 2025/10/26 02:58:52 DEBUG : Added delayed dir = "subdir", newDst=subdir 2025/10/26 02:58:52 DEBUG : file1.copy2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 DEBUG : file1.copy1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 DEBUG : subdir/file20.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : file1.copy2.txt.455881d5.partial: renamed to: file1.copy2.txt 2025/10/26 02:58:52 DEBUG : file1.copy1.txt.21b8fa2b.partial: renamed to: file1.copy1.txt 2025/10/26 02:58:52 INFO : file1.copy2.txt: Copied (new) 2025/10/26 02:58:52 INFO : file1.copy1.txt: Copied (new) 2025/10/26 02:58:52 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 02:58:52 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 02:58:52 INFO : RCLONE_TEST: Copied (new) 2025/10/26 02:58:52 DEBUG : Local file system at /tmp/020107li/initdir/test_basic-piyaqax0: Waiting for checks to finish 2025/10/26 02:58:52 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 02:58:52 DEBUG : Local file system at /tmp/020107li/initdir/test_basic-piyaqax0: Waiting for transfers to finish 2025/10/26 02:58:52 INFO : file1.txt: Copied (new) 2025/10/26 02:58:52 DEBUG : file1.copy5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 DEBUG : file1.copy3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 DEBUG : file1.copy4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 DEBUG : file1.copy5.txt.19e8a2bc.partial: renamed to: file1.copy5.txt 2025/10/26 02:58:52 INFO : file1.copy5.txt: Copied (new) 2025/10/26 02:58:52 DEBUG : file1.copy3.txt.d0285540.partial: renamed to: file1.copy3.txt 2025/10/26 02:58:52 INFO : file1.copy3.txt: Copied (new) 2025/10/26 02:58:52 DEBUG : file1.copy4.txt.8c987629.partial: renamed to: file1.copy4.txt 2025/10/26 02:58:52 DEBUG : subdir/file20.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 02:58:52 INFO : file1.copy4.txt: Copied (new) 2025/10/26 02:58:52 DEBUG : subdir/file20.txt.1ac3db2f.partial: renamed to: subdir/file20.txt 2025/10/26 02:58:52 INFO : subdir/file20.txt: Copied (new) 2025/10/26 02:58:52 INFO : subdir: Set directory modification time (using SetModTime) 2025/10/26 02:58:52 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_basic/modfiles" 2025/10/26 02:58:52 DEBUG : Creating backend with remote "/tmp/020107li/datadir/test_basic-juqidar9" 2025/10/26 02:58:52 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 02:58:52 DEBUG : Local file system at /tmp/020107li/datadir/test_basic-juqidar9: Waiting for checks to finish 2025/10/26 02:58:52 DEBUG : Local file system at /tmp/020107li/datadir/test_basic-juqidar9: Waiting for transfers to finish 2025/10/26 02:58:52 DEBUG : file1.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 02:58:52 DEBUG : file1.txt.75b77f44.partial: renamed to: file1.txt 2025/10/26 02:58:52 INFO : file1.txt: Copied (new) 2025/10/26 03:00:46 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tecolid1fuzi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:00:46 DEBUG : pacer: Rate limited, increasing sleep to 40ms panic: test timed out after 1h0m0s running tests: TestBisyncRemoteLocal (1h0m0s) TestBisyncRemoteLocal/basic (13m38s) goroutine 1051 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc000504e00, {0x27ff354?, 0xc00006d6e0?}, 0x2930810) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc000504e00) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc000504e00, 0xc00006d820) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc00055d9f8, {0x40a3200, 0x4, 0x4}, {0xee08f77e3?, 0x40e0a00?, 0x40e0a00?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc0004ab900) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc0004ab900) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/cmd/bisync_test.TestMain(0xc0004ab900) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:243 +0xf2 main.main() _testmain.go:55 +0xa8 goroutine 10 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 11 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 12 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 13 [chan receive, 15 minutes]: testing.(*T).Run(0xc000504fc0, {0xc0005f0a65?, 0x2c4fa68?}, 0xc0001711a0) /usr/local/go/src/testing/testing.go:1859 +0x431 github.com/rclone/rclone/cmd/bisync_test.testBisync({0x2c8a6e8, 0xc00054dad0}, 0xc000504fc0, {0xc000684000, 0x28}, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:398 +0x94f github.com/rclone/rclone/cmd/bisync_test.TestBisyncRemoteLocal(0xc000504fc0) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:256 +0x175 testing.tRunner(0xc000504fc0, 0x2930810) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 14 [chan receive, 60 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 13 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 818 [select]: github.com/aws/aws-sdk-go-v2/internal/sdk.sleepWithContext({0x2c8a6e8, 0xc0008e0ff0}, 0x2c700e0?) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/sdk/time.go:36 +0xc5 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).handleAttempt(0xc0004adc80, {0x2c8a6e8, 0xc0008e0ff0}, {{0x2781760?, 0xc0008e0b40?}}, 0xc000497ed0, {0x2c6b980, 0xc0005e8760}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:276 +0x9c8 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).HandleFinalize(0xc0004adc80, {0x2c8a6e8, 0xc0004c68d0}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6b980, 0xc0005e8760}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:121 +0x525 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptBeforeRetryLoop).HandleFinalize(0x156d445?, {0x2c8a6e8, 0xc0004c68d0}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6b980, 0xc0005e8780}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:131 +0x10a github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanRetryLoop).HandleFinalize(0xc00075b8d0?, {0x2c8a6e8, 0xc0004c68d0}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6b980, 0xc0005e87a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:693 +0x129 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize(0x41821f?, {0x2c8a6e8, 0xc0004c68d0}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6b980, 0xc0005e87c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1/accept_encoding_gzip.go:67 +0x15e github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware(0x2c8a6e8?, {0x2c8a6e8, 0xc0004c68d0}, {0x2781760, 0xc0004c6540}, {0x2c6aaa0, 0xc0005e8100}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestEnd).HandleBuild(0x4982a0?, {0x2c8a6e8?, 0xc0004c68d0?}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6b9e0, 0xc0004a8010}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1444 +0x8b github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RecursionDetection).HandleBuild(0x1568ed8?, {0x2c8a6e8, 0xc0004c68d0}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6ba00, 0xc0005e8520}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/recursion_detection.go:46 +0x258 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/internal/middleware.AddTimeOffsetMiddleware.HandleBuild({0x27debf9?}, {0x2c8a6e8?, 0xc0004c6720?}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6ba00?, 0xc0005e8540?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/middleware/middleware.go:30 +0xb2 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RequestUserAgent).HandleBuild(0x0?, {0x2c8a6e8, 0xc0004c6720}, {{0x2781760, 0xc0004c6540}}, {0x2c6ba00, 0xc0005e8560}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/user_agent.go:343 +0x8d github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/service/s3.setCredentialSourceMiddleware.HandleBuild({0xc0008a2600, {{0x4104100, 0x0, 0x0}, {0x0, 0x0}, 0xc000714290, 0x0, 0x0, {0x2c63f00, ...}, ...}}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:916 +0x148 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild(0xcf4bc226021245e9?, {0x2c8a6e8, 0xc0004c6720}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6ba00, 0xc0005e85a0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/middleware_content_length.go:49 +0x12a github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild({}, {0x2c8a6e8, 0xc0004c6720}, {{0x2781760?, 0xc0004c6540?}}, {0x2c6ba00, 0xc0005e85c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/middleware.go:42 +0x1ed github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware(0x2000?, {0x2c8a6e8, 0xc0004c6720}, {0x2781760, 0xc0004c6540}, {0x2c6aaa0, 0xc0005e8120}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:192 github.com/aws/aws-sdk-go-v2/service/s3.(*isExpressUserAgent).HandleSerialize(0xc00084c4d8, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b8e0, 0xc0008d3f40}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/express_user_agent.go:33 +0xf5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*removeBucketFromPathMiddleware).HandleSerialize(0x232b420?, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e82a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/remove_bucket_middleware.go:26 +0x1fd github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*updateEndpoint).HandleSerialize(0xc0008a2618, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e82c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/update_endpoint.go:145 +0x52b github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptAfterSerialization).HandleSerialize(0xc0008a2660, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e82e0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:106 +0x1d0 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_serializeOpGetBucketVersioning).HandleSerialize(0x2781760?, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e8300}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serializers.go:3984 +0x86a github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptBeforeSerialization).HandleSerialize(0x232b420?, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e8320}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:79 +0x146 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*serializeImmutableHostnameBucketMiddleware).HandleSerialize(0xc0006815bd, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e8340}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serialize_immutable_hostname_bucket.go:53 +0x3a6 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*s3ObjectLambdaEndpoint).HandleSerialize(0xc00090c000, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e8360}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/s3_object_lambda.go:34 +0x697 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*processARNResource).HandleSerialize(0xc000925ec0, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e8380}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/process_arn_resource.go:54 +0x1219 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*ResolveEndpoint).HandleSerialize(0xc000114500, {0x2c8a6e8, 0xc0004c6720}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e83a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/endpoints.go:89 +0x755 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*setOperationInputMiddleware).HandleSerialize(0x4104100?, {0x2c8a6e8?, 0xc0004c6630?}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e83c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:355 +0xb5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*putBucketContextMiddleware).HandleSerialize(0x41b09a?, {0x2c8a6e8, 0xc0004c6480}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e83e0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/bucket_context.go:26 +0xdc github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestStart).HandleSerialize(0x90?, {0x2c8a6e8?, 0xc0004c6480?}, {{0x23e68a0?, 0xc000402330?}, {0x2781760?, 0xc0004c6540?}}, {0x2c6b900, 0xc0005e84c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1426 +0xaf github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware(0xc0008d3e90, {0x2c8a6e8, 0xc0004c6480}, {0x23e68a0, 0xc000402330}, {0x2c6aaa0, 0xc0005e8140}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:122 +0x26f github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeEnd).HandleInitialize(0xc00084c4a8?, {0x2c8a6e8?, 0xc0004c6480?}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b920, 0xc0008d3f00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1409 +0x8b github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*validateOpGetBucketVersioning).HandleInitialize(0x19?, {0x2c8a6e8, 0xc0004c6480}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b940, 0xc0005e81a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/validators.go:970 +0xc6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize(0x0?, {0x2c8a6e8?, 0xc0004c6450?}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b940, 0xc0005e81c0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/logging.go:45 +0x8c github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*ARNLookup).HandleInitialize(0xc0004c6450?, {0x2c8a6e8, 0xc0004c6450}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b940, 0xc0005e81e0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.8/arn_lookup.go:39 +0x262 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*legacyEndpointContextSetter).HandleInitialize(0x1568ed8?, {0x2c8a6e8?, 0xc0004c6450?}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b940?, 0xc0005e8200?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:418 +0x9e github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize({{0x27ca68e, 0x2}, {0x0, 0x0}, {0x27dc772, 0x9}, {0x27faacb, 0x13}}, {0x2c8a6e8, 0xc0004c6270}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/metadata.go:41 +0x1c6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptExecution).HandleInitialize(0xc0004adf20, {0x2c8a6e8, 0xc0004c60c0}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b940, 0xc0005e8240}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:46 +0x1f0 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeStart).HandleInitialize(0x41821f?, {0x2c8a6e8?, 0xc0004c60c0?}, {{0x23e68a0?, 0xc000402330?}}, {0x2c6b940, 0xc0005e8260}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1391 +0x85 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware(0xc00074bd40?, {0x2c8a6e8, 0xc0004c60c0}, {0x23e68a0, 0xc000402330}, {0x2c6aaa0, 0xc0005e8160}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware(0x6?, {0x2c8a6e8, 0xc0004c60c0}, {0x23e68a0, 0xc000402330}, {0x2c6aa40?, 0xc0005e80c0?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/stack.go:109 +0x183 github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).invokeOperation(0xc000617508, {0x2c8a6e8?, 0xc00054dad0?}, {0x27faacb, 0x13}, {0x23e68a0, 0xc000402330}, {0x0, 0x0, 0x0}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:305 +0xb75 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).GetBucketVersioning(0x21f69a0?, {0x2c8a6e8?, 0xc00054dad0?}, 0xc00075f640?, {0x0?, 0xc0005056c0?, 0xc00075f670?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_op_GetBucketVersioning.go:43 +0x10c github.com/rclone/rclone/backend/s3.(*Fs).setGetVersioning.func2() /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3421 +0x57 github.com/rclone/rclone/fs.pacerInvoker(0x2, 0x2, 0x2c61740?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:86 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000794960, 0xc0003d8900, 0x2) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:232 +0xc5 github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc000794960, 0xc0003d8900) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:251 +0x93 github.com/rclone/rclone/backend/s3.(*Fs).setGetVersioning(0xc000112908, {0x2c8a6e8, 0xc00054dad0}, {0x0, 0x0, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3420 +0x34c github.com/rclone/rclone/backend/s3.(*Fs).isVersioned(0xc000112908, {0x2c8a6e8?, 0xc00054dad0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3378 +0xcb github.com/rclone/rclone/backend/s3.(*Fs).purge(0xc000112908, {0x2c8a6e8, 0xc00054dad0}, {0x0, 0x0}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3455 +0x10b github.com/rclone/rclone/backend/s3.(*Fs).Purge(0x9f575c?, {0x2c8a6e8?, 0xc00054dad0?}, {0x0?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3535 +0x25 github.com/rclone/rclone/fs/operations.Purge({0x2c8a6e8, 0xc00054dad0}, {0x2ca2a00, 0xc000112908}, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations.go:1149 +0xfe github.com/rclone/rclone/cmd/bisync_test.(*bisyncTest).cleanupCase(0xc00074ce00, {0x2c8a6e8, 0xc00054dad0}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:653 +0x3e github.com/rclone/rclone/cmd/bisync_test.(*bisyncTest).runTestCase(0xc00074ce00, {0x2c8a6e8, 0xc00054dad0}, 0xc00074cc40, {0xc0005f0a65, 0x5}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:478 +0x72b github.com/rclone/rclone/cmd/bisync_test.testBisync.func4(0xc00074cc40) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 +0x234 testing.tRunner(0xc00074cc40, 0xc0001711a0) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 13 /usr/local/go/src/testing/testing.go:1851 +0x413 "./bisync.test -test.v -test.timeout 1h0m0s -remote TestS3MinioEdge: -verbose" - Finished ERROR in 1h0m0.086198023s (try 2/5): exit status 2: Failed []