"./bisync.test -test.v -test.timeout 1h0m0s -remote TestS3MinioEdge: -verbose" - Starting (try 4/5) 2025/10/26 04:01:37 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-hibetaq8kafo" 2025/10/26 04:01:37 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/10/26 04:01:37 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:01:37 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:01:37 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:01:37 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:01:37 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:01:37 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:01:37 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:01:37 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:01:37 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:01:37 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:01:37 DEBUG : Creating backend with remote "/tmp/rclone1453644308" === RUN TestBisyncRemoteLocal 2025/10/26 04:01:37 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi" 2025/10/26 04:01:37 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:01:37 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:01:37 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:01:37 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:01:37 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:01:37 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:01:37 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:01:37 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:01:37 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:01:37 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:01:37 NOTICE: remote: TestS3MinioEdge:rclone-test-kiyajel1gizi === RUN TestBisyncRemoteLocal/all_changed 2025/10/26 04:01:37 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no" 2025/10/26 04:01:37 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:01:37 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:01:37 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:01:37 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:01:37 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:01:37 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:01:37 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:01:37 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:01:37 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:01:37 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:03:39 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:03:39 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 04:05:40 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:05:40 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 04:05:40 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-kiyajel1gizi/040137no" 2025/10/26 04:05:40 INFO : path1: Making directory 2025/10/26 04:07:35 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:07:35 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 04:09:23 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:09:23 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/all_changed Messages: Mkdir path1 2025/10/26 04:09:23 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no/all_changed/path1" 2025/10/26 04:09:23 DEBUG : Config file has changed externally - reloading 2025/10/26 04:09:23 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:09:23 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:09:23 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:09:23 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:09:23 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:09:23 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:09:23 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:09:23 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:09:23 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:09:23 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:11:21 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no/all_changed/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:11:21 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 04:13:11 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no/all_changed/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:13:11 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 04:13:11 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no/all_changed/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-kiyajel1gizi/040137no/all_changed/path1" 2025/10/26 04:13:11 DEBUG : Creating backend with remote "/tmp/040137no" 2025/10/26 04:13:11 DEBUG : Creating backend with remote "/tmp/040137no/all_changed/path2" 2025/10/26 04:13:11 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_all_changed/initial" 2025/10/26 04:13:11 DEBUG : Creating backend with remote "/tmp/040137no/initdir/test_all_changed-soyitoj3" 2025/10/26 04:13:11 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : file1.copy1.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : file1.copy2.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : file1.copy3.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : file1.copy4.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : file1.copy5.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : subdir: Making directory with metadata 2025/10/26 04:13:11 INFO : subdir: Made directory with metadata (mtime=2021-11-02T05:00:07Z) 2025/10/26 04:13:11 DEBUG : Added delayed dir = "subdir", newDst=subdir 2025/10/26 04:13:11 DEBUG : file1.copy2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 04:13:11 DEBUG : subdir/file20.txt: Need to transfer - File not found at Destination 2025/10/26 04:13:11 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 04:13:11 INFO : RCLONE_TEST: Copied (new) 2025/10/26 04:13:11 DEBUG : Local file system at /tmp/040137no/initdir/test_all_changed-soyitoj3: Waiting for checks to finish 2025/10/26 04:13:11 DEBUG : file1.copy4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : file1.copy4.txt.8c987629.partial: renamed to: file1.copy4.txt 2025/10/26 04:13:11 INFO : file1.copy4.txt: Copied (new) 2025/10/26 04:13:11 DEBUG : file1.copy3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : Local file system at /tmp/040137no/initdir/test_all_changed-soyitoj3: Waiting for transfers to finish 2025/10/26 04:13:11 DEBUG : file1.copy5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : file1.copy3.txt.d0285540.partial: renamed to: file1.copy3.txt 2025/10/26 04:13:11 DEBUG : file1.copy5.txt.19e8a2bc.partial: renamed to: file1.copy5.txt 2025/10/26 04:13:11 INFO : file1.copy5.txt: Copied (new) 2025/10/26 04:13:11 INFO : file1.copy3.txt: Copied (new) 2025/10/26 04:13:11 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 04:13:11 INFO : file1.txt: Copied (new) 2025/10/26 04:13:11 DEBUG : subdir/file20.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : file1.copy1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:13:11 DEBUG : file1.copy1.txt.21b8fa2b.partial: renamed to: file1.copy1.txt 2025/10/26 04:13:11 INFO : file1.copy1.txt: Copied (new) 2025/10/26 04:13:11 DEBUG : file1.copy2.txt.455881d5.partial: renamed to: file1.copy2.txt 2025/10/26 04:13:11 INFO : file1.copy2.txt: Copied (new) 2025/10/26 04:13:11 DEBUG : subdir/file20.txt.1ac3db2f.partial: renamed to: subdir/file20.txt 2025/10/26 04:13:11 INFO : subdir/file20.txt: Copied (new) 2025/10/26 04:13:11 INFO : subdir: Set directory modification time (using SetModTime) 2025/10/26 04:15:00 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:15:00 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 04:16:48 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:16:48 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/26 04:16:48 ERROR : S3 bucket rclone-test-kiyajel1gizi path 040137no/all_changed/path1: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused 2025/10/26 04:16:48 DEBUG : S3 bucket rclone-test-kiyajel1gizi path 040137no/all_changed/path1: bucket is versioned: false 2025/10/26 04:16:48 DEBUG : Waiting for deletions to finish 2025/10/26 04:18:45 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:18:45 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/26 04:20:38 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:20:38 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/26 04:20:38 DEBUG : Waiting for deletions to finish 2025/10/26 04:20:38 ERROR : error listing: directory not found 2025/10/26 04:20:38 ERROR : Local file system at /tmp/040137no/all_changed/path2: Failed to list "": directory not found 2025/10/26 04:20:38 DEBUG : removing 1 level 0 directories 2025/10/26 04:20:38 INFO : Local file system at /tmp/040137no/all_changed/path2: Removing directory 2025/10/26 04:20:38 ERROR : Failed to rmdir: stat /tmp/040137no/all_changed/path2: no such file or directory 2025/10/26 04:22:36 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:22:36 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/26 04:24:31 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:24:31 DEBUG : pacer: Rate limited, increasing sleep to 1.28s fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fall_changed%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused === RUN TestBisyncRemoteLocal/backupdir 2025/10/26 04:24:31 DEBUG : fs cache: switching user supplied name "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no" for canonical name "TestS3MinioEdge{FbV4e}:rclone-test-kiyajel1gizi/040137no" 2025/10/26 04:24:31 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no" 2025/10/26 04:24:31 DEBUG : Config file has changed externally - reloading 2025/10/26 04:24:31 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:24:31 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:24:31 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:24:31 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:24:31 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:24:31 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:24:31 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:24:31 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:24:31 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:24:31 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:26:23 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:26:23 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 04:28:12 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:28:12 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 04:28:12 INFO : path1: Making directory 2025/10/26 04:29:59 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:29:59 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 04:31:41 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:31:41 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/backupdir Messages: Mkdir path1 2025/10/26 04:31:41 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no/backupdir/path1" 2025/10/26 04:31:41 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:31:41 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:31:41 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:31:41 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:31:41 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:31:41 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:31:41 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:31:41 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:31:41 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:31:41 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:33:32 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no/backupdir/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:33:32 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 04:35:18 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no/backupdir/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:35:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 04:35:18 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no/backupdir/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-kiyajel1gizi/040137no/backupdir/path1" 2025/10/26 04:35:18 DEBUG : Creating backend with remote "/tmp/040137no" 2025/10/26 04:35:18 DEBUG : Creating backend with remote "/tmp/040137no/backupdir/path2" 2025/10/26 04:35:18 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_backupdir/initial" 2025/10/26 04:35:18 DEBUG : Creating backend with remote "/tmp/040137no/initdir/test_backupdir-qotegab2" 2025/10/26 04:35:18 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file2.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file3.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file4.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file5.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file6.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file7.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file8.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : Local file system at /tmp/040137no/initdir/test_backupdir-qotegab2: Waiting for checks to finish 2025/10/26 04:35:18 DEBUG : Local file system at /tmp/040137no/initdir/test_backupdir-qotegab2: Waiting for transfers to finish 2025/10/26 04:35:18 DEBUG : file2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : file3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : file2.txt.3230ad67.partial: renamed to: file2.txt 2025/10/26 04:35:18 INFO : file2.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file3.txt.a74079f2.partial: renamed to: file3.txt 2025/10/26 04:35:18 INFO : file3.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 04:35:18 DEBUG : file4.txt.fbf05a9b.partial: renamed to: file4.txt 2025/10/26 04:35:18 INFO : file4.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 04:35:18 INFO : RCLONE_TEST: Copied (new) 2025/10/26 04:35:18 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 04:35:18 DEBUG : file6.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 INFO : file1.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file6.txt.0a60f5f0.partial: renamed to: file6.txt 2025/10/26 04:35:18 INFO : file6.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file7.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : file7.txt.9f102165.partial: renamed to: file7.txt 2025/10/26 04:35:18 INFO : file7.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file8.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : file5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:35:18 DEBUG : file8.txt.b300b322.partial: renamed to: file8.txt 2025/10/26 04:35:18 INFO : file8.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file5.txt.6e808e0e.partial: renamed to: file5.txt 2025/10/26 04:35:18 INFO : file5.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_backupdir/modfiles" 2025/10/26 04:35:18 DEBUG : Creating backend with remote "/tmp/040137no/datadir/test_backupdir-fexomep9" 2025/10/26 04:35:18 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file10.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file11.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file2.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file5L.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file5R.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file6.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file7.txt: Need to transfer - File not found at Destination 2025/10/26 04:35:18 DEBUG : file1.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 04:35:18 DEBUG : file1.txt.75b77f44.partial: renamed to: file1.txt 2025/10/26 04:35:18 INFO : file1.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : Local file system at /tmp/040137no/datadir/test_backupdir-fexomep9: Waiting for checks to finish 2025/10/26 04:35:18 DEBUG : Local file system at /tmp/040137no/datadir/test_backupdir-fexomep9: Waiting for transfers to finish 2025/10/26 04:35:18 DEBUG : file11.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 04:35:18 DEBUG : file10.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 04:35:18 DEBUG : file11.txt.40d797d4.partial: renamed to: file11.txt 2025/10/26 04:35:18 INFO : file11.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file10.txt.c02780cb.partial: renamed to: file10.txt 2025/10/26 04:35:18 DEBUG : file2.txt: md5 = fb3ecfb2800400fb01b0bfd39903e9fb OK 2025/10/26 04:35:18 INFO : file10.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file2.txt.4526d70e.partial: renamed to: file2.txt 2025/10/26 04:35:18 INFO : file2.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file5L.txt: md5 = 0860a03592626642f8fd6c8bfb447d2a OK 2025/10/26 04:35:18 DEBUG : file5L.txt.8dd6541a.partial: renamed to: file5L.txt 2025/10/26 04:35:18 INFO : file5L.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file5R.txt: md5 = 979a803b15d27df0c31ad7d29006d10b OK 2025/10/26 04:35:18 DEBUG : file5R.txt.ee3b9525.partial: renamed to: file5R.txt 2025/10/26 04:35:18 INFO : file5R.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file6.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 04:35:18 DEBUG : file7.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 04:35:18 DEBUG : file7.txt.c1750184.partial: renamed to: file7.txt 2025/10/26 04:35:18 INFO : file7.txt: Copied (new) 2025/10/26 04:35:18 DEBUG : file6.txt.4185169b.partial: renamed to: file6.txt 2025/10/26 04:35:18 INFO : file6.txt: Copied (new) 2025/10/26 04:37:07 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:37:07 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 04:38:57 DEBUG : pacer: low level retry 2/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:38:57 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/10/26 04:38:57 ERROR : S3 bucket rclone-test-kiyajel1gizi path 040137no/backupdir/path1: Failed to read versioning status, assuming unversioned: operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused 2025/10/26 04:38:57 DEBUG : S3 bucket rclone-test-kiyajel1gizi path 040137no/backupdir/path1: bucket is versioned: false 2025/10/26 04:38:57 DEBUG : Waiting for deletions to finish 2025/10/26 04:40:58 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:40:58 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/10/26 04:42:53 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:42:53 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/10/26 04:42:53 DEBUG : Waiting for deletions to finish 2025/10/26 04:42:53 ERROR : error listing: directory not found 2025/10/26 04:42:53 ERROR : Local file system at /tmp/040137no/backupdir/path2: Failed to list "": directory not found 2025/10/26 04:42:53 DEBUG : removing 1 level 0 directories 2025/10/26 04:42:53 INFO : Local file system at /tmp/040137no/backupdir/path2: Removing directory 2025/10/26 04:42:53 ERROR : Failed to rmdir: stat /tmp/040137no/backupdir/path2: no such file or directory 2025/10/26 04:44:51 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:44:51 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/10/26 04:46:50 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:46:50 DEBUG : pacer: Rate limited, increasing sleep to 1.28s fstest.go:282: Error listing: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=040137no%2Fbackupdir%2Fpath1%2F": dial tcp 127.0.0.1:28626: connect: connection refused === RUN TestBisyncRemoteLocal/basic 2025/10/26 04:46:50 DEBUG : fs cache: switching user supplied name "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no" for canonical name "TestS3MinioEdge{FbV4e}:rclone-test-kiyajel1gizi/040137no" 2025/10/26 04:46:50 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no" 2025/10/26 04:46:50 DEBUG : Config file has changed externally - reloading 2025/10/26 04:46:50 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:46:50 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:46:50 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:46:50 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:46:50 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:46:50 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:46:50 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:46:50 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:46:50 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:46:50 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:48:44 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:48:44 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 04:50:42 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:50:42 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 04:50:42 INFO : path1: Making directory 2025/10/26 04:52:27 DEBUG : pacer: low level retry 1/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:52:27 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/10/26 04:54:17 DEBUG : pacer: low level retry 2/2 (error operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:54:17 DEBUG : pacer: Rate limited, increasing sleep to 80ms bisync_test.go:2013: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:2013 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:639 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:432 /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 Error: Received unexpected error: operation error S3: CreateBucket, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "http://127.0.0.1:28626/rclone-test-kiyajel1gizi": dial tcp 127.0.0.1:28626: connect: connection refused Test: TestBisyncRemoteLocal/basic Messages: Mkdir path1 2025/10/26 04:54:17 DEBUG : Creating backend with remote "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no/basic/path1" 2025/10/26 04:54:17 DEBUG : Config file has changed externally - reloading 2025/10/26 04:54:17 DEBUG : Setting type="s3" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_TYPE 2025/10/26 04:54:17 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:54:17 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:54:17 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:54:17 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:54:17 DEBUG : TestS3MinioEdge: detected overridden config - adding "{FbV4e}" suffix to name 2025/10/26 04:54:17 DEBUG : Setting provider="Minio" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_PROVIDER 2025/10/26 04:54:17 DEBUG : Setting access_key_id="rclone" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ACCESS_KEY_ID 2025/10/26 04:54:17 DEBUG : Setting secret_access_key="DeniseOxygenEiffel4" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_SECRET_ACCESS_KEY 2025/10/26 04:54:17 DEBUG : Setting endpoint="http://127.0.0.1:28626/" for "TestS3MinioEdge" from environment variable RCLONE_CONFIG_TESTS3MINIOEDGE_ENDPOINT 2025/10/26 04:56:20 DEBUG : pacer: low level retry 1/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no/basic/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:56:20 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/10/26 04:58:12 DEBUG : pacer: low level retry 2/2 (error operation error S3: HeadObject, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://127.0.0.1:28626/rclone-test-kiyajel1gizi/040137no/basic/path1": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 04:58:12 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/10/26 04:58:12 DEBUG : fs cache: renaming cache item "TestS3MinioEdge:rclone-test-kiyajel1gizi/040137no/basic/path1" to be canonical "TestS3MinioEdge{FbV4e}:rclone-test-kiyajel1gizi/040137no/basic/path1" 2025/10/26 04:58:12 DEBUG : Creating backend with remote "/tmp/040137no" 2025/10/26 04:58:12 DEBUG : Config file has changed externally - reloading 2025/10/26 04:58:12 DEBUG : Creating backend with remote "/tmp/040137no/basic/path2" 2025/10/26 04:58:12 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_basic/initial" 2025/10/26 04:58:12 DEBUG : Creating backend with remote "/tmp/040137no/initdir/test_basic-mogaxas5" 2025/10/26 04:58:12 DEBUG : RCLONE_TEST: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : file1.copy1.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : file1.copy2.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : file1.copy3.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : file1.copy4.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : file1.copy5.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : subdir: Making directory with metadata 2025/10/26 04:58:12 INFO : subdir: Made directory with metadata (mtime=2021-11-02T05:00:07Z) 2025/10/26 04:58:12 DEBUG : Added delayed dir = "subdir", newDst=subdir 2025/10/26 04:58:12 DEBUG : subdir/file20.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : Local file system at /tmp/040137no/initdir/test_basic-mogaxas5: Waiting for checks to finish 2025/10/26 04:58:12 DEBUG : Local file system at /tmp/040137no/initdir/test_basic-mogaxas5: Waiting for transfers to finish 2025/10/26 04:58:12 DEBUG : file1.copy1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : file1.copy1.txt.21b8fa2b.partial: renamed to: file1.copy1.txt 2025/10/26 04:58:12 INFO : file1.copy1.txt: Copied (new) 2025/10/26 04:58:12 DEBUG : file1.copy2.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : file1.copy2.txt.455881d5.partial: renamed to: file1.copy2.txt 2025/10/26 04:58:12 INFO : file1.copy2.txt: Copied (new) 2025/10/26 04:58:12 DEBUG : file1.copy4.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : file1.copy3.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : file1.copy4.txt.8c987629.partial: renamed to: file1.copy4.txt 2025/10/26 04:58:12 INFO : file1.copy4.txt: Copied (new) 2025/10/26 04:58:12 DEBUG : file1.copy3.txt.d0285540.partial: renamed to: file1.copy3.txt 2025/10/26 04:58:12 INFO : file1.copy3.txt: Copied (new) 2025/10/26 04:58:12 DEBUG : RCLONE_TEST: md5 = 294d25b294ff26a5243dba914ac3fbf7 OK 2025/10/26 04:58:12 DEBUG : file1.copy5.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : subdir/file20.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : subdir/file20.txt.1ac3db2f.partial: renamed to: subdir/file20.txt 2025/10/26 04:58:12 INFO : subdir/file20.txt: Copied (new) 2025/10/26 04:58:12 DEBUG : RCLONE_TEST.abb09051.partial: renamed to: RCLONE_TEST 2025/10/26 04:58:12 INFO : RCLONE_TEST: Copied (new) 2025/10/26 04:58:12 DEBUG : file1.copy5.txt.19e8a2bc.partial: renamed to: file1.copy5.txt 2025/10/26 04:58:12 INFO : file1.copy5.txt: Copied (new) 2025/10/26 04:58:12 DEBUG : file1.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK 2025/10/26 04:58:12 DEBUG : file1.txt.56d0d699.partial: renamed to: file1.txt 2025/10/26 04:58:12 INFO : file1.txt: Copied (new) 2025/10/26 04:58:12 INFO : subdir: Set directory modification time (using SetModTime) 2025/10/26 04:58:12 DEBUG : Creating backend with remote "/home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/testdata/test_basic/modfiles" 2025/10/26 04:58:12 DEBUG : Creating backend with remote "/tmp/040137no/datadir/test_basic-rugihed3" 2025/10/26 04:58:12 DEBUG : file1.txt: Need to transfer - File not found at Destination 2025/10/26 04:58:12 DEBUG : Local file system at /tmp/040137no/datadir/test_basic-rugihed3: Waiting for checks to finish 2025/10/26 04:58:12 DEBUG : Local file system at /tmp/040137no/datadir/test_basic-rugihed3: Waiting for transfers to finish 2025/10/26 04:58:12 DEBUG : file1.txt: md5 = 7fe98ed88552b828777d8630900346b8 OK 2025/10/26 04:58:12 DEBUG : file1.txt.75b77f44.partial: renamed to: file1.txt 2025/10/26 04:58:12 INFO : file1.txt: Copied (new) 2025/10/26 05:00:11 DEBUG : pacer: low level retry 1/2 (error operation error S3: GetBucketVersioning, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://127.0.0.1:28626/rclone-test-kiyajel1gizi?versioning=": dial tcp 127.0.0.1:28626: connect: connection refused) 2025/10/26 05:00:11 DEBUG : pacer: Rate limited, increasing sleep to 40ms panic: test timed out after 1h0m0s running tests: TestBisyncRemoteLocal (1h0m0s) TestBisyncRemoteLocal/basic (14m46s) goroutine 1072 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc0008bc1c0, {0x27ff354?, 0xc00006d6e0?}, 0x2930810) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc0008bc1c0) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc0008bc1c0, 0xc00006d820) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc0001f54a0, {0x40a3200, 0x4, 0x4}, {0xee08f9421?, 0x40e0a00?, 0x40e0a00?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc00075ea00) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc00075ea00) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/cmd/bisync_test.TestMain(0xc00075ea00) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:243 +0xf2 main.main() _testmain.go:55 +0xa8 goroutine 7 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 6 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 19 [chan receive, 15 minutes]: testing.(*T).Run(0xc0008bc380, {0xc000880d65?, 0x2c4fa68?}, 0xc00013d440) /usr/local/go/src/testing/testing.go:1859 +0x431 github.com/rclone/rclone/cmd/bisync_test.testBisync({0x2c8a6e8, 0xc0000f1800}, 0xc0008bc380, {0xc0008b1380, 0x28}, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:398 +0x94f github.com/rclone/rclone/cmd/bisync_test.TestBisyncRemoteLocal(0xc0008bc380) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:256 +0x175 testing.tRunner(0xc0008bc380, 0x2930810) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 8 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 42 [chan receive, 60 minutes]: github.com/rclone/rclone/lib/atexit.Register.func1.1() /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x29 created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 19 /home/rclone/go/src/github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x68 goroutine 824 [select]: github.com/aws/aws-sdk-go-v2/internal/sdk.sleepWithContext({0x2c8a6e8, 0xc0000b24b0}, 0x2c700e0?) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/sdk/time.go:36 +0xc5 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).handleAttempt(0xc00068dcb0, {0x2c8a6e8, 0xc0000b24b0}, {{0x2781760?, 0xc0000b2060?}}, 0xc000512e90, {0x2c6b980, 0xc00050f620}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:276 +0x9c8 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).HandleFinalize(0xc00068dcb0, {0x2c8a6e8, 0xc000544660}, {{0x2781760?, 0xc000544270?}}, {0x2c6b980, 0xc00050f620}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/retry/middleware.go:121 +0x525 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptBeforeRetryLoop).HandleFinalize(0x156d445?, {0x2c8a6e8, 0xc000544660}, {{0x2781760?, 0xc000544270?}}, {0x2c6b980, 0xc00050f640}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:131 +0x10a github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanRetryLoop).HandleFinalize(0xc00068c810?, {0x2c8a6e8, 0xc000544660}, {{0x2781760?, 0xc000544270?}}, {0x2c6b980, 0xc00050f660}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:693 +0x129 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize(0x41821f?, {0x2c8a6e8, 0xc000544660}, {{0x2781760?, 0xc000544270?}}, {0x2c6b980, 0xc00050f680}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.13.1/accept_encoding_gzip.go:67 +0x15e github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:200 github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware(0x2c8a6e8?, {0x2c8a6e8, 0xc000544660}, {0x2781760, 0xc000544270}, {0x2c6aaa0, 0xc00050f060}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_finalize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestEnd).HandleBuild(0x4982a0?, {0x2c8a6e8?, 0xc000544660?}, {{0x2781760?, 0xc000544270?}}, {0x2c6b9e0, 0xc000769450}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1444 +0x8b github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RecursionDetection).HandleBuild(0x1568ed8?, {0x2c8a6e8, 0xc000544660}, {{0x2781760?, 0xc000544270?}}, {0x2c6ba00, 0xc00050f3c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/recursion_detection.go:46 +0x258 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/internal/middleware.AddTimeOffsetMiddleware.HandleBuild({0x27debf9?}, {0x2c8a6e8?, 0xc000544480?}, {{0x2781760?, 0xc000544270?}}, {0x2c6ba00?, 0xc00050f3e0?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/internal/middleware/middleware.go:30 +0xb2 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.(*RequestUserAgent).HandleBuild(0x0?, {0x2c8a6e8, 0xc000544480}, {{0x2781760, 0xc000544270}}, {0x2c6ba00, 0xc00050f400}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/user_agent.go:343 +0x8d github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/service/s3.setCredentialSourceMiddleware.HandleBuild({0xc0007b25a0, {{0x4104100, 0x0, 0x0}, {0x0, 0x0}, 0xc0006c5250, 0x0, 0x0, {0x2c63f00, ...}, ...}}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:916 +0x148 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild(0xb048e59df22f8bdc?, {0x2c8a6e8, 0xc000544480}, {{0x2781760?, 0xc000544270?}}, {0x2c6ba00, 0xc00050f440}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/middleware_content_length.go:49 +0x12a github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild({}, {0x2c8a6e8, 0xc000544480}, {{0x2781760?, 0xc000544270?}}, {0x2c6ba00, 0xc00050f460}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/middleware.go:42 +0x1ed github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:200 github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware(0x1826876?, {0x2c8a6e8, 0xc000544480}, {0x2781760, 0xc000544270}, {0x2c6aaa0, 0xc00050f080}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_build.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:192 github.com/aws/aws-sdk-go-v2/service/s3.(*isExpressUserAgent).HandleSerialize(0xc000870238, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b8e0, 0xc000769410}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/express_user_agent.go:33 +0xf5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*removeBucketFromPathMiddleware).HandleSerialize(0x232b420?, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f200}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/remove_bucket_middleware.go:26 +0x1fd github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*updateEndpoint).HandleSerialize(0xc0007b25b8, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f220}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/update_endpoint.go:145 +0x52b github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptAfterSerialization).HandleSerialize(0xc0007b2600, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f240}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:106 +0x1d0 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_serializeOpGetBucketVersioning).HandleSerialize(0x2781760?, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f260}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serializers.go:3984 +0x86a github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/transport/http.(*InterceptBeforeSerialization).HandleSerialize(0x232b420?, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f280}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:79 +0x146 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*serializeImmutableHostnameBucketMiddleware).HandleSerialize(0xc0008803bd, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f2a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/serialize_immutable_hostname_bucket.go:53 +0x3a6 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*s3ObjectLambdaEndpoint).HandleSerialize(0xc0008083c0, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f2c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/s3_object_lambda.go:34 +0x697 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*processARNResource).HandleSerialize(0xc000808300, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f2e0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/internal/customizations/process_arn_resource.go:54 +0x1219 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*ResolveEndpoint).HandleSerialize(0xc00064f540, {0x2c8a6e8, 0xc000544480}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f300}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/endpoints.go:89 +0x755 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*setOperationInputMiddleware).HandleSerialize(0x4104100?, {0x2c8a6e8?, 0xc000544390?}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f320}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:355 +0xb5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*putBucketContextMiddleware).HandleSerialize(0x41b074?, {0x2c8a6e8, 0xc0005441b0}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f340}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/bucket_context.go:26 +0xdc github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestStart).HandleSerialize(0x90?, {0x2c8a6e8?, 0xc0005441b0?}, {{0x23e68a0?, 0xc00054a7c8?}, {0x2781760?, 0xc000544270?}}, {0x2c6b900, 0xc00050f360}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1426 +0xaf github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:208 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware(0xc000769360, {0x2c8a6e8, 0xc0005441b0}, {0x23e68a0, 0xc00054a7c8}, {0x2c6aaa0, 0xc00050f0a0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_serialize.go:122 +0x26f github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeEnd).HandleInitialize(0xc000870208?, {0x2c8a6e8?, 0xc0005441b0?}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b920, 0xc0007693d0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1409 +0x8b github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*validateOpGetBucketVersioning).HandleInitialize(0xc000870201?, {0x2c8a6e8, 0xc0005441b0}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b940, 0xc00050f100}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/validators.go:970 +0xc6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize(0x0?, {0x2c8a6e8?, 0xc0005440f0?}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b940, 0xc00050f120}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/logging.go:45 +0x8c github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*ARNLookup).HandleInitialize(0xc0005440f0?, {0x2c8a6e8, 0xc0005440f0}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b940, 0xc00050f140}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.19.8/arn_lookup.go:39 +0x262 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*legacyEndpointContextSetter).HandleInitialize(0x1568ed8?, {0x2c8a6e8?, 0xc0005440f0?}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b940?, 0xc00050f160?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:418 +0x9e github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize({{0x27ca68e, 0x2}, {0x0, 0x0}, {0x27dc772, 0x9}, {0x27faacb, 0x13}}, {0x2c8a6e8, 0xc000118420}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.39.1/aws/middleware/metadata.go:41 +0x1c6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/transport/http.(*InterceptExecution).HandleInitialize(0xc00068de60, {0x2c8a6e8, 0xc00068df50}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b940, 0xc00050f1a0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/transport/http/interceptor_middleware.go:46 +0x1f0 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeStart).HandleInitialize(0x41821f?, {0x2c8a6e8?, 0xc00068df50?}, {{0x23e68a0?, 0xc00054a7c8?}}, {0x2c6b940, 0xc00050f1c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:1391 +0x85 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware(0xc000574340?, {0x2c8a6e8, 0xc00068df50}, {0x23e68a0, 0xc00054a7c8}, {0x2c6aaa0, 0xc00050f0c0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/step_initialize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware(0x418201?, {0x2c8a6e8, 0xc00068df50}, {0x23e68a0, 0xc00054a7c8}, {0x2c6aa40?, 0xc00050f020?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/stack.go:109 +0x183 github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.23.0/middleware/middleware.go:57 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).invokeOperation(0xc00008ca88, {0x2c8a6e8?, 0xc0000f1800?}, {0x27faacb, 0x13}, {0x23e68a0, 0xc00054a7c8}, {0x0, 0x0, 0x0}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_client.go:305 +0xb75 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).GetBucketVersioning(0x21f69a0?, {0x2c8a6e8?, 0xc0000f1800?}, 0xc0006e3640?, {0x0?, 0xc000187dc0?, 0xc0006e3670?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.88.2/api_op_GetBucketVersioning.go:43 +0x10c github.com/rclone/rclone/backend/s3.(*Fs).setGetVersioning.func2() /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3421 +0x57 github.com/rclone/rclone/fs.pacerInvoker(0x2, 0x2, 0x2c61740?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:86 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000893920, 0xc00089db00, 0x2) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:232 +0xc5 github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc000893920, 0xc00089db00) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:251 +0x93 github.com/rclone/rclone/backend/s3.(*Fs).setGetVersioning(0xc00046ef08, {0x2c8a6e8, 0xc0000f1800}, {0x0, 0x0, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3420 +0x34c github.com/rclone/rclone/backend/s3.(*Fs).isVersioned(0xc00046ef08, {0x2c8a6e8?, 0xc0000f1800?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3378 +0xcb github.com/rclone/rclone/backend/s3.(*Fs).purge(0xc00046ef08, {0x2c8a6e8, 0xc0000f1800}, {0x0, 0x0}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3455 +0x10b github.com/rclone/rclone/backend/s3.(*Fs).Purge(0x9f575c?, {0x2c8a6e8?, 0xc0000f1800?}, {0x0?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:3535 +0x25 github.com/rclone/rclone/fs/operations.Purge({0x2c8a6e8, 0xc0000f1800}, {0x2ca2a00, 0xc00046ef08}, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations.go:1149 +0xfe github.com/rclone/rclone/cmd/bisync_test.(*bisyncTest).cleanupCase(0xc0008c8e00, {0x2c8a6e8, 0xc0000f1800}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:653 +0x3e github.com/rclone/rclone/cmd/bisync_test.(*bisyncTest).runTestCase(0xc0008c8e00, {0x2c8a6e8, 0xc0000f1800}, 0xc0008c8c40, {0xc000880d65, 0x5}) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:478 +0x72b github.com/rclone/rclone/cmd/bisync_test.testBisync.func4(0xc0008c8c40) /home/rclone/go/src/github.com/rclone/rclone/cmd/bisync/bisync_test.go:405 +0x234 testing.tRunner(0xc0008c8c40, 0xc00013d440) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 19 /usr/local/go/src/testing/testing.go:1851 +0x413 "./bisync.test -test.v -test.timeout 1h0m0s -remote TestS3MinioEdge: -verbose" - Finished ERROR in 1h0m0.095594567s (try 4/5): exit status 2: Failed []