"./bisync.test -test.v -test.timeout 1h0m0s -remote TestS3MinioEdge: -verbose" - Starting (try 3/5) 2025/10/26 03:01:07 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kibadaf4soxu" 2025/10/26 03:01:07 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/10/26 03:01:07 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:01:07 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:01:07 DEBUG : Creating backend with remote "/tmp/rclone377628179" === RUN TestBisyncRemoteLocal 2025/10/26 03:01:07 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye" 2025/10/26 03:01:07 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:01:07 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:01:07 NOTICE: remote: TestS3MinioEdge:rclone-test-tamazij3muye === RUN TestBisyncRemoteLocal/all_changed 2025/10/26 03:01:07 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji" 2025/10/26 03:01:07 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:01:07 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:01:07 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:01:07 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:01:07 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:01:07 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:03:02 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:03:02 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 03:04:55 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:04:55 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 03:04:55 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tamazij3muye/030107ji" 2025/10/26 03:04:55 INFO : path1: Making directory 2025/10/26 03:06:40 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:06:40 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 03:08:34 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:08:34 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/all_changed Messages: Mkdir path1 2025/10/26 03:08:34 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji/all_changed/path1" 2025/10/26 03:08:34 DEBUG : Config file has changed externally - reloading 2025/10/26 03:08:34 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:08:34 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:08:34 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:08:34 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:08:34 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:08:34 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:08:34 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:08:34 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:08:34 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:08:34 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:10:34 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji/all_changed/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:10:34 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 03:12:32 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji/all_changed/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:12:32 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 03:12:32 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji/all_changed/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tamazij3muye/030107ji/all_changed/path1" 2025/10/26 03:12:32 DEBUG : Creating backend with remote "/tmp/030107ji" 2025/10/26 03:12:32 DEBUG : Creating backend with remote "/tmp/030107ji/all_changed/path2" 2025/10/26 03:12:32 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_all_changed/initial" 2025/10/26 03:12:32 DEBUG : Creating backend with remote "/tmp/030107ji/initdir/test_all_changed-zobizin0" 2025/10/26 03:12:32 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.copy1.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.copy2.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.copy3.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.copy4.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.copy5.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : subdir: Making directory with metadata 2025/10/26 03:12:32 INFO : subdir: Made directory with metadata (mtime=2021-11-02T05:00:07Z) 2025/10/26 03:12:32 DEBUG : Added delayed dir = "subdir", newDst=subdir 2025/10/26 03:12:32 DEBUG : subdir/file20.txt: Need to transfer - File not found at Destination 2025/10/26 03:12:32 DEBUG : file1.copy1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : file1.copy1.txt.21b8fa2b.partial: renamed to: file1.copy1.txt 2025/10/26 03:12:32 INFO : file1.copy1.txt: Copied (new) 2025/10/26 03:12:32 DEBUG : file1.copy2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : file1.copy2.txt.455881d5.partial: renamed to: file1.copy2.txt 2025/10/26 03:12:32 INFO : file1.copy2.txt: Copied (new) 2025/10/26 03:12:32 DEBUG : file1.copy3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : file1.copy3.txt.d0285540.partial: renamed to: file1.copy3.txt 2025/10/26 03:12:32 INFO : file1.copy3.txt: Copied (new) 2025/10/26 03:12:32 DEBUG : file1.copy4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : file1.copy5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : file1.copy5.txt.19e8a2bc.partial: renamed to: file1.copy5.txt 2025/10/26 03:12:32 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 03:12:32 INFO : file1.copy5.txt: Copied (new) 2025/10/26 03:12:32 INFO : file1.txt: Copied (new) 2025/10/26 03:12:32 DEBUG : Local file system at /tmp/030107ji/initdir/test_all_changed-zobizin0: Waiting for checks to finish 2025/10/26 03:12:32 DEBUG : Local file system at /tmp/030107ji/initdir/test_all_changed-zobizin0: Waiting for transfers to finish 2025/10/26 03:12:32 DEBUG : subdir/file20.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:12:32 DEBUG : subdir/file20.txt.1ac3db2f.partial: renamed to: subdir/file20.txt 2025/10/26 03:12:32 INFO : subdir/file20.txt: Copied (new) 2025/10/26 03:12:32 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 03:12:32 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 03:12:32 DEBUG : file1.copy4.txt.8c987629.partial: renamed to: file1.copy4.txt 2025/10/26 03:12:32 INFO : file1.copy4.txt: Copied (new) 2025/10/26 03:12:32 INFO : RCLONE_TEST: Copied (new) 2025/10/26 03:12:32 INFO : subdir: Set directory modification time (using SetModTime) 2025/10/26 03:14:34 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:14:34 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 03:16:30 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:16:30 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/26 03:16:30 ERROR : S3 bucket rclone-test-tamazij3muye path 030107ji/all_changed/path1: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused 2025/10/26 03:16:30 DEBUG : S3 bucket rclone-test-tamazij3muye path 030107ji/all_changed/path1: bucket is versioned: false 2025/10/26 03:16:30 DEBUG : Waiting for deletions to finish 2025/10/26 03:18:24 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:18:24 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/26 03:20:28 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:20:28 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/26 03:20:28 DEBUG : Waiting for deletions to finish 2025/10/26 03:20:28 ERROR : error listing: directory not found 2025/10/26 03:20:28 ERROR : Local file system at /tmp/030107ji/all_changed/path2: Failed to list "": directory not found 2025/10/26 03:20:28 DEBUG : removing 1 level 0 directories 2025/10/26 03:20:28 INFO : Local file system at /tmp/030107ji/all_changed/path2: Removing directory 2025/10/26 03:20:28 ERROR : Failed to rmdir: stat /tmp/030107ji/all_changed/path2: no such file or directory 2025/10/26 03:22:28 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:22:28 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/26 03:24:16 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:24:16 DEBUG : pacer: Rate limited, increasing sleep to 1.28s fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused === RUN TestBisyncRemoteLocal/backupdir 2025/10/26 03:24:16 DEBUG : fs cache: switching user supplied name "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji" for canonical name "TestS3MinioEdge{FbV4e}:rclone-test-tamazij3muye/030107ji" 2025/10/26 03:24:16 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji" 2025/10/26 03:24:16 DEBUG : Config file has changed externally - reloading 2025/10/26 03:24:16 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:24:16 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:24:16 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:24:16 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:24:16 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:24:16 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:24:16 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:24:16 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:24:16 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:24:16 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:26:07 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:26:07 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 03:28:09 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:28:09 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 03:28:09 INFO : path1: Making directory 2025/10/26 03:30:06 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:30:06 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 03:32:05 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:32:05 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/backupdir Messages: Mkdir path1 2025/10/26 03:32:05 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji/backupdir/path1" 2025/10/26 03:32:05 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:32:05 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:32:05 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:32:05 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:32:05 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:32:05 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:32:05 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:32:05 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:32:05 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:32:05 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:33:54 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji/backupdir/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:33:54 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 03:35:47 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji/backupdir/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:35:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 03:35:47 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji/backupdir/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tamazij3muye/030107ji/backupdir/path1" 2025/10/26 03:35:47 DEBUG : Creating backend with remote "/tmp/030107ji" 2025/10/26 03:35:47 DEBUG : Config file has changed externally - reloading 2025/10/26 03:35:47 DEBUG : Creating backend with remote "/tmp/030107ji/backupdir/path2" 2025/10/26 03:35:47 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_backupdir/initial" 2025/10/26 03:35:47 DEBUG : Creating backend with remote "/tmp/030107ji/initdir/test_backupdir-kixureb1" 2025/10/26 03:35:47 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file2.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file3.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file4.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file5.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file6.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file7.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file8.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : Local file system at /tmp/030107ji/initdir/test_backupdir-kixureb1: Waiting for checks to finish 2025/10/26 03:35:47 DEBUG : Local file system at /tmp/030107ji/initdir/test_backupdir-kixureb1: Waiting for transfers to finish 2025/10/26 03:35:47 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : file2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 03:35:47 DEBUG : file3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 INFO : file1.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file2.txt.3230ad67.partial: renamed to: file2.txt 2025/10/26 03:35:47 INFO : file2.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 03:35:47 DEBUG : file4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 03:35:47 DEBUG : file4.txt.fbf05a9b.partial: renamed to: file4.txt 2025/10/26 03:35:47 INFO : RCLONE_TEST: Copied (new) 2025/10/26 03:35:47 INFO : file4.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : file7.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : file5.txt.6e808e0e.partial: renamed to: file5.txt 2025/10/26 03:35:47 INFO : file5.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file6.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : file7.txt.9f102165.partial: renamed to: file7.txt 2025/10/26 03:35:47 INFO : file7.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file6.txt.0a60f5f0.partial: renamed to: file6.txt 2025/10/26 03:35:47 INFO : file6.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file3.txt.a74079f2.partial: renamed to: file3.txt 2025/10/26 03:35:47 INFO : file3.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file8.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:35:47 DEBUG : file8.txt.b300b322.partial: renamed to: file8.txt 2025/10/26 03:35:47 INFO : file8.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_backupdir/modfiles" 2025/10/26 03:35:47 DEBUG : Creating backend with remote "/tmp/030107ji/datadir/test_backupdir-xagukar9" 2025/10/26 03:35:47 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file10.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file11.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file2.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file5L.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file5R.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file6.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : file7.txt: Need to transfer - File not found at Destination 2025/10/26 03:35:47 DEBUG : Local file system at /tmp/030107ji/datadir/test_backupdir-xagukar9: Waiting for checks to finish 2025/10/26 03:35:47 DEBUG : Local file system at /tmp/030107ji/datadir/test_backupdir-xagukar9: Waiting for transfers to finish 2025/10/26 03:35:47 DEBUG : file1.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 03:35:47 DEBUG : file2.txt: md5 = fb3ecfb2800400fb01b0bfd39903e9fb OK 2025/10/26 03:35:47 DEBUG : file1.txt.75b77f44.partial: renamed to: file1.txt 2025/10/26 03:35:47 INFO : file1.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file2.txt.4526d70e.partial: renamed to: file2.txt 2025/10/26 03:35:47 INFO : file2.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file11.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 03:35:47 DEBUG : file11.txt.40d797d4.partial: renamed to: file11.txt 2025/10/26 03:35:47 INFO : file11.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file10.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 03:35:47 DEBUG : file10.txt.c02780cb.partial: renamed to: file10.txt 2025/10/26 03:35:47 INFO : file10.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file6.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 03:35:47 DEBUG : file6.txt.4185169b.partial: renamed to: file6.txt 2025/10/26 03:35:47 DEBUG : file5R.txt: md5 = 979a803b15d27df0c31ad7d29006d10b OK 2025/10/26 03:35:47 DEBUG : file5L.txt: md5 = 0860a03592626642f8fd6c8bfb447d2a OK 2025/10/26 03:35:47 INFO : file6.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file5R.txt.ee3b9525.partial: renamed to: file5R.txt 2025/10/26 03:35:47 INFO : file5R.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file5L.txt.8dd6541a.partial: renamed to: file5L.txt 2025/10/26 03:35:47 INFO : file5L.txt: Copied (new) 2025/10/26 03:35:47 DEBUG : file7.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 03:35:47 DEBUG : file7.txt.c1750184.partial: renamed to: file7.txt 2025/10/26 03:35:47 INFO : file7.txt: Copied (new) 2025/10/26 03:37:51 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:37:51 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 03:39:39 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:39:39 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/26 03:39:39 ERROR : S3 bucket rclone-test-tamazij3muye path 030107ji/backupdir/path1: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused 2025/10/26 03:39:39 DEBUG : S3 bucket rclone-test-tamazij3muye path 030107ji/backupdir/path1: bucket is versioned: false 2025/10/26 03:39:39 DEBUG : Waiting for deletions to finish 2025/10/26 03:41:25 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:41:25 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/26 03:43:24 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:43:24 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/26 03:43:24 DEBUG : Waiting for deletions to finish 2025/10/26 03:43:24 ERROR : error listing: directory not found 2025/10/26 03:43:24 ERROR : Local file system at /tmp/030107ji/backupdir/path2: Failed to list "": directory not found 2025/10/26 03:43:24 DEBUG : removing 1 level 0 directories 2025/10/26 03:43:24 INFO : Local file system at /tmp/030107ji/backupdir/path2: Removing directory 2025/10/26 03:43:24 ERROR : Failed to rmdir: stat /tmp/030107ji/backupdir/path2: no such file or directory 2025/10/26 03:45:16 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:45:16 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/26 03:47:12 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:47:12 DEBUG : pacer: Rate limited, increasing sleep to 1.28s fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=030107ji%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused === RUN TestBisyncRemoteLocal/basic 2025/10/26 03:47:12 DEBUG : fs cache: switching user supplied name "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji" for canonical name "TestS3MinioEdge{FbV4e}:rclone-test-tamazij3muye/030107ji" 2025/10/26 03:47:12 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji" 2025/10/26 03:47:12 DEBUG : Config file has changed externally - reloading 2025/10/26 03:47:12 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:47:12 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:47:12 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:47:12 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:47:12 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:47:12 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:47:12 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:47:12 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:47:12 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:47:12 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:49:09 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:49:09 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 03:51:07 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:51:07 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 03:51:07 INFO : path1: Making directory 2025/10/26 03:52:58 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:52:58 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 03:54:58 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:54:58 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-tamazij3muye": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/basic Messages: Mkdir path1 2025/10/26 03:54:58 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji/basic/path1" 2025/10/26 03:54:58 DEBUG : Config file has changed externally - reloading 2025/10/26 03:54:58 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 03:54:58 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:54:58 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:54:58 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:54:58 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:54:58 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 03:54:58 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 03:54:58 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 03:54:58 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 03:54:58 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 03:56:58 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji/basic/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:56:58 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 03:58:56 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-tamazij3muye/030107ji/basic/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 03:58:56 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 03:58:56 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-tamazij3muye/030107ji/basic/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-tamazij3muye/030107ji/basic/path1" 2025/10/26 03:58:56 DEBUG : Creating backend with remote "/tmp/030107ji" 2025/10/26 03:58:56 DEBUG : Creating backend with remote "/tmp/030107ji/basic/path2" 2025/10/26 03:58:56 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_basic/initial" 2025/10/26 03:58:56 DEBUG : Creating backend with remote "/tmp/030107ji/initdir/test_basic-noficeq1" 2025/10/26 03:58:56 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.copy1.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.copy2.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.copy3.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.copy4.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.copy5.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : subdir: Making directory with metadata 2025/10/26 03:58:56 INFO : subdir: Made directory with metadata (mtime=2021-11-02T05:00:07Z) 2025/10/26 03:58:56 DEBUG : Added delayed dir = "subdir", newDst=subdir 2025/10/26 03:58:56 DEBUG : subdir/file20.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : file1.copy2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : Local file system at /tmp/030107ji/initdir/test_basic-noficeq1: Waiting for checks to finish 2025/10/26 03:58:56 DEBUG : Local file system at /tmp/030107ji/initdir/test_basic-noficeq1: Waiting for transfers to finish 2025/10/26 03:58:56 DEBUG : file1.copy1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : file1.copy3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 03:58:56 DEBUG : file1.copy2.txt.455881d5.partial: renamed to: file1.copy2.txt 2025/10/26 03:58:56 INFO : file1.copy2.txt: Copied (new) 2025/10/26 03:58:56 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 03:58:56 INFO : RCLONE_TEST: Copied (new) 2025/10/26 03:58:56 DEBUG : file1.copy1.txt.21b8fa2b.partial: renamed to: file1.copy1.txt 2025/10/26 03:58:56 INFO : file1.copy1.txt: Copied (new) 2025/10/26 03:58:56 DEBUG : file1.copy3.txt.d0285540.partial: renamed to: file1.copy3.txt 2025/10/26 03:58:56 INFO : file1.copy3.txt: Copied (new) 2025/10/26 03:58:56 DEBUG : file1.copy4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : file1.copy5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : file1.copy4.txt.8c987629.partial: renamed to: file1.copy4.txt 2025/10/26 03:58:56 INFO : file1.copy4.txt: Copied (new) 2025/10/26 03:58:56 DEBUG : file1.copy5.txt.19e8a2bc.partial: renamed to: file1.copy5.txt 2025/10/26 03:58:56 INFO : file1.copy5.txt: Copied (new) 2025/10/26 03:58:56 DEBUG : subdir/file20.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 03:58:56 DEBUG : subdir/file20.txt.1ac3db2f.partial: renamed to: subdir/file20.txt 2025/10/26 03:58:56 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 03:58:56 INFO : subdir/file20.txt: Copied (new) 2025/10/26 03:58:56 INFO : file1.txt: Copied (new) 2025/10/26 03:58:56 INFO : subdir: Set directory modification time (using SetModTime) 2025/10/26 03:58:56 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_basic/modfiles" 2025/10/26 03:58:56 DEBUG : Creating backend with remote "/tmp/030107ji/datadir/test_basic-kakayoz9" 2025/10/26 03:58:56 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 03:58:56 DEBUG : Local file system at /tmp/030107ji/datadir/test_basic-kakayoz9: Waiting for checks to finish 2025/10/26 03:58:56 DEBUG : Local file system at /tmp/030107ji/datadir/test_basic-kakayoz9: Waiting for transfers to finish 2025/10/26 03:58:56 DEBUG : file1.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 03:58:56 DEBUG : file1.txt.75b77f44.partial: renamed to: file1.txt 2025/10/26 03:58:56 INFO : file1.txt: Copied (new) 2025/10/26 04:00:49 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-tamazij3muye?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:00:49 DEBUG : pacer: Rate limited, increasing sleep to 40ms panic: test timed out after 1h0m0s running tests: TestBisyncRemoteLocal (1h0m0s) TestBisyncRemoteLocal/basic (13m54s) goroutine 922 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc000102e00, {0x27ff354?, 0xc00006d6e0?}, 0x2930810) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc000102e00) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc000102e00, 0xc00006d820) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc00042d680, {0x40a3200, 0x4, 0x4}, {0xee08f85f3?, 0x40e0a00?, 0x40e0a00?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc0006fae60) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc0006fae60) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/cmd/bisync_test.TestMain(0xc0006fae60) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:243 +0xf2 main.main() _testmain.go:55 +0xa8 goroutine 16 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 50 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 51 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 52 [chan receive, 14 minutes]: testing.(*T).Run(0xc000102fc0, {0xc0007060f5?, 0x2c4fa68?}, 0xc00011ad80) /usr/local/go/src/testing/testing.go:1859 +0x431 github.com/rclone/rclone/cmd/bisync_test.testBisync({0x2c8a6e8, 0xc000831890}, 0xc000102fc0, {0xc0004a44e0, 0x28}, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:398 +0x94f github.com/rclone/rclone/cmd/bisync_test.TestBisyncRemoteLocal(0xc000102fc0) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:256 +0x175 testing.tRunner(0xc000102fc0, 0x2930810) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 22 [chan receive, 60 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 52 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 830 [select]: github.com/aws/aws-sdk-go-v2/internal/sdk.sleepWithContext({0x2c8a6e8, 0xc000324c90}, 0x2c700e0?) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/sdk/time.go:36 +0xc5 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).handleAttempt(0xc00030f440, {0x2c8a6e8, 0xc000324c90}, {{0x2781760?, 0xc000324990?}}, 0xc00013f4a0, {0x2c6b980, 0xc00071d1c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:276 +0x9c8 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).HandleFinalize(0xc00030f440, {0x2c8a6e8, 0xc00030fec0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6b980, 0xc00071d1c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:121 +0x525 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptBeforeRetryLoop).HandleFinalize(0x156d445?, {0x2c8a6e8, 0xc00030fec0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6b980, 0xc00071d1e0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:131 +0x10a github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanRetryLoop).HandleFinalize(0xc00075f8d0?, {0x2c8a6e8, 0xc00030fec0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6b980, 0xc00071d200}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:693 +0x129 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize(0x41821f?, {0x2c8a6e8, 0xc00030fec0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6b980, 0xc00071d220}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1/accept_encoding_gzip.go:67 +0x15e github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware(0x2c8a6e8?, {0x2c8a6e8, 0xc00030fec0}, {0x2781760, 0xc00030fb30}, {0x2c6aaa0, 0xc00071cb20}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestEnd).HandleBuild(0x4982a0?, {0x2c8a6e8?, 0xc00030fec0?}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6b9e0, 0xc0006aea70}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1444 +0x8b github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RecursionDetection).HandleBuild(0x1568ed8?, {0x2c8a6e8, 0xc00030fec0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6ba00, 0xc00071cec0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/recursion_detection.go:46 +0x258 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/internal/middleware.AddTimeOffsetMiddleware.HandleBuild({0x27debf9?}, {0x2c8a6e8?, 0xc00030fce0?}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6ba00?, 0xc00071cee0?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/middleware/middleware.go:30 +0xb2 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RequestUserAgent).HandleBuild(0x0?, {0x2c8a6e8, 0xc00030fce0}, {{0x2781760, 0xc00030fb30}}, {0x2c6ba00, 0xc00071cf00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/user_agent.go:343 +0x8d github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/service/s3.setCredentialSourceMiddleware.HandleBuild({0xc00038a510, {{0x4104100, 0x0, 0x0}, {0x0, 0x0}, 0xc000487250, 0x0, 0x0, {0x2c63f00, ...}, ...}}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:916 +0x148 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild(0x4545f0bed147dc50?, {0x2c8a6e8, 0xc00030fce0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6ba00, 0xc00071cf40}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/middleware_content_length.go:49 +0x12a github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild({}, {0x2c8a6e8, 0xc00030fce0}, {{0x2781760?, 0xc00030fb30?}}, {0x2c6ba00, 0xc00071cf60}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/middleware.go:42 +0x1ed github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware(0x2000?, {0x2c8a6e8, 0xc00030fce0}, {0x2781760, 0xc00030fb30}, {0x2c6aaa0, 0xc00071cb40}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:192 github.com/aws/aws-sdk-go-v2/service/s3.(*isExpressUserAgent).HandleSerialize(0xc00008c5c0, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b8e0, 0xc0006aea30}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/express_user_agent.go:33 +0xf5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*removeBucketFromPathMiddleware).HandleSerialize(0x232b420?, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071ccc0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/remove_bucket_middleware.go:26 +0x1fd github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*updateEndpoint).HandleSerialize(0xc00038a528, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cce0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/update_endpoint.go:145 +0x52b github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptAfterSerialization).HandleSerialize(0xc00038a570, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cd00}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:106 +0x1d0 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_serializeOpGetBucketVersioning).HandleSerialize(0x2781760?, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cd20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serializers.go:3984 +0x86a github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptBeforeSerialization).HandleSerialize(0x232b420?, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cd40}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:79 +0x146 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*serializeImmutableHostnameBucketMiddleware).HandleSerialize(0xc00040f88d, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cd60}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serialize_immutable_hostname_bucket.go:53 +0x3a6 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*s3ObjectLambdaEndpoint).HandleSerialize(0xc00011b020, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cd80}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/s3_object_lambda.go:34 +0x697 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*processARNResource).HandleSerialize(0xc00011af00, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cda0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/process_arn_resource.go:54 +0x1219 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*ResolveEndpoint).HandleSerialize(0xc0003f2b90, {0x2c8a6e8, 0xc00030fce0}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cdc0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/endpoints.go:89 +0x755 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*setOperationInputMiddleware).HandleSerialize(0x4104100?, {0x2c8a6e8?, 0xc00030fc50?}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071cde0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:355 +0xb5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*putBucketContextMiddleware).HandleSerialize(0x41b074?, {0x2c8a6e8, 0xc00030fb00}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071ce00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/bucket_context.go:26 +0xdc github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestStart).HandleSerialize(0x90?, {0x2c8a6e8?, 0xc00030fb00?}, {{0x23e68a0?, 0xc0005e2438?}, {0x2781760?, 0xc00030fb30?}}, {0x2c6b900, 0xc00071ce20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1426 +0xaf github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware(0xc0006ae980, {0x2c8a6e8, 0xc00030fb00}, {0x23e68a0, 0xc0005e2438}, {0x2c6aaa0, 0xc00071cb60}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:122 +0x26f github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeEnd).HandleInitialize(0xc00008c590?, {0x2c8a6e8?, 0xc00030fb00?}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b920, 0xc0006ae9f0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1409 +0x8b github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*validateOpGetBucketVersioning).HandleInitialize(0xc00008c501?, {0x2c8a6e8, 0xc00030fb00}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b940, 0xc00071cbc0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/validators.go:970 +0xc6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize(0x0?, {0x2c8a6e8?, 0xc00030fad0?}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b940, 0xc00071cbe0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/logging.go:45 +0x8c github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*ARNLookup).HandleInitialize(0xc00030fad0?, {0x2c8a6e8, 0xc00030fad0}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b940, 0xc00071cc00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.8/arn_lookup.go:39 +0x262 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*legacyEndpointContextSetter).HandleInitialize(0x1568ed8?, {0x2c8a6e8?, 0xc00030fad0?}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b940?, 0xc00071cc20?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:418 +0x9e github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize({{0x27ca68e, 0x2}, {0x0, 0x0}, {0x27dc772, 0x9}, {0x27faacb, 0x13}}, {0x2c8a6e8, 0xc00030f830}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/metadata.go:41 +0x1c6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptExecution).HandleInitialize(0xc00030f620, {0x2c8a6e8, 0xc00030f710}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b940, 0xc00071cc60}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:46 +0x1f0 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeStart).HandleInitialize(0x41821f?, {0x2c8a6e8?, 0xc00030f710?}, {{0x23e68a0?, 0xc0005e2438?}}, {0x2c6b940, 0xc00071cc80}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1391 +0x85 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware(0xc0003ea270?, {0x2c8a6e8, 0xc00030f710}, {0x23e68a0, 0xc0005e2438}, {0x2c6aaa0, 0xc00071cb80}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware(0x418201?, {0x2c8a6e8, 0xc00030f710}, {0x23e68a0, 0xc0005e2438}, {0x2c6aa40?, 0xc00071cae0?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/stack.go:109 +0x183 github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).invokeOperation(0xc00066c708, {0x2c8a6e8?, 0xc000831890?}, {0x27faacb, 0x13}, {0x23e68a0, 0xc0005e2438}, {0x0, 0x0, 0x0}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:305 +0xb75 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).GetBucketVersioning(0x21f69a0?, {0x2c8a6e8?, 0xc000831890?}, 0xc000763640?, {0x0?, 0xc00050ac40?, 0xc000763670?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_op_GetBucketVersioning.go:43 +0x10c github.com/rclone/rclone/backend/s3.(*Fs).setGetVersioning.func2() /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3421 +0x57 github.com/rclone/rclone/fs.pacerInvoker(0x2, 0x2, 0x2c61740?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:86 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc00038f6e0, 0xc000828540, 0x2) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:232 +0xc5 github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc00038f6e0, 0xc000828540) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:251 +0x93 github.com/rclone/rclone/backend/s3.(*Fs).setGetVersioning(0xc0003da008, {0x2c8a6e8, 0xc000831890}, {0x0, 0x0, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3420 +0x34c github.com/rclone/rclone/backend/s3.(*Fs).isVersioned(0xc0003da008, {0x2c8a6e8?, 0xc000831890?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3378 +0xcb github.com/rclone/rclone/backend/s3.(*Fs).purge(0xc0003da008, {0x2c8a6e8, 0xc000831890}, {0x0, 0x0}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3455 +0x10b github.com/rclone/rclone/backend/s3.(*Fs).Purge(0x9f575c?, {0x2c8a6e8?, 0xc000831890?}, {0x0?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3535 +0x25 github.com/rclone/rclone/fs/operations.Purge({0x2c8a6e8, 0xc000831890}, {0x2ca2a00, 0xc0003da008}, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations.go:1149 +0xfe github.com/rclone/rclone/cmd/bisync_test.(*bisyncTest).cleanupCase(0xc0005488c0, {0x2c8a6e8, 0xc000831890}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:653 +0x3e github.com/rclone/rclone/cmd/bisync_test.(*bisyncTest).runTestCase(0xc0005488c0, {0x2c8a6e8, 0xc000831890}, 0xc000548700, {0xc0007060f5, 0x5}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:478 +0x72b github.com/rclone/rclone/cmd/bisync_test.testBisync.func4(0xc000548700) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 +0x234 testing.tRunner(0xc000548700, 0xc00011ad80) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 52 /usr/local/go/src/testing/testing.go:1851 +0x413 "./bisync.test -test.v -test.timeout 1h0m0s -remote TestS3MinioEdge: -verbose" - Finished ERROR in 1h0m0.083769341s (try 3/5): exit status 2: Failed []