"go test -v -timeout 1h0m0s -remote TestS3Wasabi: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsEncoding$/^(URL_encoding|invalid_UTF-8|leading_dot|trailing_CR|trailing_HT|trailing_LF|trailing_VT|trailing_dot|trailing_space)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^(FsListDirFile2|FsListFile1|FsListFile1and2|FsListLevel2|FsListR|FsListRDirFile2|FsListRLevel2|FsPurge|FsPurgeRoot|ObjectRemove)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^(List|ListEntries|ListR)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^Put$/^Remove$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FsCopy$/^Metadata$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^Cleanup$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^AfterDelete$/^List$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^AfterOne$/^List$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^AfterTwo$/^List$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^Before$/^List$'" - Starting (try 2/5) === RUN TestIntegration fstests.go:438: Using remote "TestS3Wasabi:" 2025/07/30 02:09:58 NOTICE: TestS3Wasabi: Starting server 2025/07/30 02:09:58 DEBUG : Creating backend with remote "TestS3Wasabi:rclone-test-gizazos9zefi" 2025/07/30 02:09:58 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/07/30 02:09:59 INFO : S3 bucket rclone-test-gizazos9zefi: Bucket "rclone-test-gizazos9zefi" created with ACL "" === RUN TestIntegration/FsMkdir === RUN TestIntegration/FsMkdir/FsEncoding === RUN TestIntegration/FsMkdir/FsEncoding/leading_dot fstests.go:707: testing ".leading dot" === RUN TestIntegration/FsMkdir/FsEncoding/trailing_space fstests.go:707: testing "trailing space " === RUN TestIntegration/FsMkdir/FsEncoding/trailing_CR fstests.go:707: testing "trailing CR␍" === RUN TestIntegration/FsMkdir/FsEncoding/trailing_LF fstests.go:707: testing "trailing LF␊" === RUN TestIntegration/FsMkdir/FsEncoding/trailing_HT fstests.go:707: testing "trailing HT␉" === RUN TestIntegration/FsMkdir/FsEncoding/trailing_VT fstests.go:707: testing "trailing VT␋" === RUN TestIntegration/FsMkdir/FsEncoding/trailing_dot fstests.go:707: testing "trailing dot." === RUN TestIntegration/FsMkdir/FsEncoding/invalid_UTF-8 fstests.go:707: testing "invalid utf-8\xfe" === RUN TestIntegration/FsMkdir/FsEncoding/URL_encoding fstests.go:707: testing "test%46.txt" === RUN TestIntegration/FsMkdir/FsPutFiles === RUN TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListRDirFile2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListR === RUN TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListRLevel2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListFile1 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsPurge 2025/07/30 02:10:16 DEBUG : S3 bucket rclone-test-gizazos9zefi: bucket is versioned: false 2025/07/30 02:10:16 DEBUG : Waiting for deletions to finish 2025/07/30 02:10:16 DEBUG : "dirToPurge/fileToPurge.txt" version false 2025/07/30 02:10:16 DEBUG : dirToPurge/fileToPurge.txt: Deleting (id "") 2025/07/30 02:10:16 INFO : dirToPurge/fileToPurge.txt: Deleted === RUN TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot 2025/07/30 02:10:18 DEBUG : Creating backend with remote "TestS3Wasabi:rclone-test-gizazos9zefi/dirToPurgeFromRoot" 2025/07/30 02:10:19 DEBUG : S3 bucket rclone-test-gizazos9zefi path dirToPurgeFromRoot: bucket is versioned: false 2025/07/30 02:10:19 DEBUG : Waiting for deletions to finish 2025/07/30 02:10:19 DEBUG : "fileToPurgeFromRoot.txt" version false 2025/07/30 02:10:19 DEBUG : fileToPurgeFromRoot.txt: Deleting (id "") 2025/07/30 02:10:19 INFO : fileToPurgeFromRoot.txt: Deleted === RUN TestIntegration/FsMkdir/FsPutFiles/FsCopy === RUN TestIntegration/FsMkdir/FsPutFiles/FsCopy/Metadata === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot fstests.go:1911: Opening root remote "TestS3Wasabi:" path "rclone-test-gizazos9zefi" from "TestS3Wasabi:rclone-test-gizazos9zefi" 2025/07/30 02:10:24 DEBUG : Creating backend with remote "TestS3Wasabi:" === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/List === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR 2025/07/30 02:17:58 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 3122B79D9C9E0713:A, HostID: dyWHotoWiToSKMdyV/ZUJKFa+eOGFNVaafJBSQDECSkdCOGlkuoTyem1GKVu17dN6pIFR9dYY6IK, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:17:58 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2025/07/30 02:25:36 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 6E4CA2BB686788F0:B, HostID: ZwuaA0Au9GgPgDxJdKe1KF2CR57JlAeakYGaXgtGo76EIL6oj+ANDLYdRtunR3efVkHwTdNcG9e3, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:25:36 DEBUG : pacer: Rate limited, increasing sleep to 20ms fstests.go:1970: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1970 Error: Received unexpected error: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 6E4CA2BB686788F0:B, HostID: ZwuaA0Au9GgPgDxJdKe1KF2CR57JlAeakYGaXgtGo76EIL6oj+ANDLYdRtunR3efVkHwTdNcG9e3, api error InternalError: We encountered an internal error. Please retry the operation again later. Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR fstests.go:1973: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1973 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR Messages: file1Root "rclone-test-gizazos9zefi/file name.txt" not found fstests.go:1974: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1974 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR Messages: file2Root "rclone-test-gizazos9zefi/hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt" not found === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put 2025/07/30 02:25:37 DEBUG : pacer: Reducing sleep to 15ms 2025/07/30 02:25:37 DEBUG : pacer: Reducing sleep to 11.25ms 2025/07/30 02:25:37 DEBUG : pacer: Reducing sleep to 0s === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectRemove === RUN TestIntegration/FsMkdir/FsPutFiles/Internal === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/Before === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/Before/List === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterOne === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterOne/List === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterDelete === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterDelete/List === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterTwo === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterTwo/List === RUN TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/Cleanup 2025/07/30 02:25:49 DEBUG : Waiting for deletions to finish 2025/07/30 02:25:49 DEBUG : "hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt" version false 2025/07/30 02:25:49 DEBUG : hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt: Not deleting current version "null" 2025/07/30 02:25:49 DEBUG : "versions/test-versions.txt" version false 2025/07/30 02:25:49 DEBUG : versions/test-versions.txt: Not deleting current version "001753842346707163166-47Sgpw4MU6" 2025/07/30 02:25:49 DEBUG : "versions/test-versions-v2025-07-30-022545-000.txt" version true 2025/07/30 02:25:49 DEBUG : versions/test-versions-v2025-07-30-022545-000.txt: Deleting delete marker (id "001753842344532516331-W64gGZn79k") 2025/07/30 02:25:49 DEBUG : "versions/test-versions-v2025-07-30-022542-000.txt" version true 2025/07/30 02:25:49 DEBUG : versions/test-versions-v2025-07-30-022542-000.txt: Deleting (id "001753842342122842966-T8Ei4DEWAz") 2025/07/30 02:25:49 INFO : versions/test-versions-v2025-07-30-022545-000.txt: Deleted 2025/07/30 02:25:49 INFO : versions/test-versions-v2025-07-30-022542-000.txt: Deleted 2025/07/30 02:25:51 DEBUG : Waiting for deletions to finish 2025/07/30 02:25:51 DEBUG : "hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt" version false 2025/07/30 02:25:51 DEBUG : hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt: Deleting (id "null") 2025/07/30 02:25:51 DEBUG : "versions/test-versions-v2025-07-30-022551-000.txt" version true 2025/07/30 02:25:51 DEBUG : versions/test-versions-v2025-07-30-022551-000.txt: Deleting delete marker (id "001753842350863031540-MzFWc2tBqC") 2025/07/30 02:25:51 DEBUG : "versions/test-versions-v2025-07-30-022547-000.txt" version true 2025/07/30 02:25:51 DEBUG : versions/test-versions-v2025-07-30-022547-000.txt: Deleting (id "001753842346707163166-47Sgpw4MU6") 2025/07/30 02:25:51 INFO : versions/test-versions-v2025-07-30-022551-000.txt: Deleted 2025/07/30 02:25:51 INFO : hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt: Deleted 2025/07/30 02:25:51 INFO : versions/test-versions-v2025-07-30-022547-000.txt: Deleted 2025/07/30 02:25:54 INFO : S3 bucket rclone-test-gizazos9zefi: Bucket "rclone-test-gizazos9zefi" deleted 2025/07/30 02:25:54 DEBUG : Waiting for deletions to finish --- FAIL: TestIntegration (956.51s) --- FAIL: TestIntegration/FsMkdir (955.63s) --- PASS: TestIntegration/FsMkdir/FsEncoding (11.03s) --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_dot (1.00s) --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_space (1.34s) --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_CR (1.04s) --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_LF (1.06s) --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_HT (1.61s) --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_VT (1.14s) --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_dot (0.86s) --- PASS: TestIntegration/FsMkdir/FsEncoding/invalid_UTF-8 (1.58s) --- PASS: TestIntegration/FsMkdir/FsEncoding/URL_encoding (1.27s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles (940.81s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 (0.58s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRDirFile2 (0.81s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListR (0.11s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 (0.23s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRLevel2 (0.12s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListFile1 (0.33s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 (0.38s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPurge (2.46s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot (3.43s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsCopy (3.68s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsCopy/Metadata (2.15s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot (914.56s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List (1.12s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries (1.06s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR (910.57s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put (1.81s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove (0.49s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectRemove (1.07s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal (11.48s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions (11.48s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt (2.11s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/Before (0.37s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/Before/List (0.37s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterOne (0.89s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterOne/List (0.88s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterDelete (0.34s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterDelete/List (0.34s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterTwo (0.51s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterTwo/List (0.51s) --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/Cleanup (1.69s) FAIL exit status 1 FAIL github.com/rclone/rclone/backend/s3 956.558s "go test -v -timeout 1h0m0s -remote TestS3Wasabi: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsEncoding$/^(URL_encoding|invalid_UTF-8|leading_dot|trailing_CR|trailing_HT|trailing_LF|trailing_VT|trailing_dot|trailing_space)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^(FsListDirFile2|FsListFile1|FsListFile1and2|FsListLevel2|FsListR|FsListRDirFile2|FsListRLevel2|FsPurge|FsPurgeRoot|ObjectRemove)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^(List|ListEntries|ListR)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^Put$/^Remove$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FsCopy$/^Metadata$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^Cleanup$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^AfterDelete$/^List$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^AfterOne$/^List$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^AfterTwo$/^List$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^Internal$/^Versions$/^VersionAt$/^Before$/^List$'" - Finished ERROR in 15m57.836261736s (try 2/5): exit status 1: Failed [TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR]