"./vfs.test -test.v -test.timeout 1h0m0s -remote TestS3Wasabi: -verbose" - Starting (try 1/5) 2025/07/30 01:06:43 DEBUG : Creating backend with remote "TestS3Wasabi:rclone-test-dabilar2bopi" 2025/07/30 01:06:43 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/07/30 01:06:43 DEBUG : Creating backend with remote "/tmp/rclone1500284885" === RUN TestDirHandleMethods run.go:180: Remote "S3 bucket rclone-test-dabilar2bopi", Local "Local file system at /tmp/rclone1500284885", Modify Window "1ns" 2025/07/30 01:06:43 INFO : S3 bucket rclone-test-dabilar2bopi: poll-interval is not supported by this remote 2025/07/30 01:06:44 INFO : S3 bucket rclone-test-dabilar2bopi: Bucket "rclone-test-dabilar2bopi" created with ACL "" 2025/07/30 01:06:45 DEBUG : WaitForWriters: timeout=30s 2025/07/30 01:06:45 DEBUG : dir: Looking for writers 2025/07/30 01:06:45 DEBUG : Looking for writers 2025/07/30 01:06:45 DEBUG : dir: reading active writers 2025/07/30 01:06:45 DEBUG : >WaitForWriters: --- PASS: TestDirHandleMethods (2.30s) === RUN TestDirHandleReaddir run.go:180: Remote "S3 bucket rclone-test-dabilar2bopi", Local "Local file system at /tmp/rclone1500284885", Modify Window "1ns" 2025/07/30 01:06:45 INFO : S3 bucket rclone-test-dabilar2bopi: poll-interval is not supported by this remote 2025/07/30 01:08:46 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 61FBCECB17316D1B:A, HostID: fWgpIpy0gR2rBDQexBusDW2l7LDMhXUz4G67g2KpqhDtxxJ/QCFe9xoOIyqbQLcQVhUOEYjtn5Wx, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:08:46 DEBUG : pacer: Rate limited, increasing sleep to 10ms run.go:281: Retry Put of "dir/file2" to S3 bucket rclone-test-dabilar2bopi: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 61FBCECB17316D1B:A, HostID: fWgpIpy0gR2rBDQexBusDW2l7LDMhXUz4G67g2KpqhDtxxJ/QCFe9xoOIyqbQLcQVhUOEYjtn5Wx, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:09:20 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 6D298A56AFAABF29:B, HostID: hDnsm+HU3KWzIzSTzOdKwGqJPF0i96YDQMEqooP7FlHSysi6x86EwNt0YOAOxv3CMLQ2RKfAXcj4, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:09:20 DEBUG : pacer: Rate limited, increasing sleep to 20ms run.go:281: Retry Put of "dir/file2" to S3 bucket rclone-test-dabilar2bopi: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 6D298A56AFAABF29:B, HostID: hDnsm+HU3KWzIzSTzOdKwGqJPF0i96YDQMEqooP7FlHSysi6x86EwNt0YOAOxv3CMLQ2RKfAXcj4, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:09:52 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 4C07FAAAC7C9B18A:B, HostID: c4ZOjmWRJISZF42AsZ3UjYwE530/u8Ce61ul9jOkZpuT/oPPUZPzuKZonu9AemxmRf62xXWxXfpM, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:09:52 DEBUG : pacer: Rate limited, increasing sleep to 40ms run.go:281: Retry Put of "dir/file2" to S3 bucket rclone-test-dabilar2bopi: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 4C07FAAAC7C9B18A:B, HostID: c4ZOjmWRJISZF42AsZ3UjYwE530/u8Ce61ul9jOkZpuT/oPPUZPzuKZonu9AemxmRf62xXWxXfpM, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:11:56 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 6C73FDF5C500E575:B, HostID: RWtZ7irJu6HXnS5oxj9vtYsYI5xsuFLAHVo9nw52dKbi78AL/OWShkg0K3VDtezxrTb8EpSoSDya, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:11:56 DEBUG : pacer: Rate limited, increasing sleep to 80ms run.go:281: Retry Put of "dir/file2" to S3 bucket rclone-test-dabilar2bopi: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 6C73FDF5C500E575:B, HostID: RWtZ7irJu6HXnS5oxj9vtYsYI5xsuFLAHVo9nw52dKbi78AL/OWShkg0K3VDtezxrTb8EpSoSDya, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:12:34 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 3B589C30486F1097:B, HostID: 7GjG7Xslmxf6kZyfwF7Qmgt/hA0qY176aDa6L/l4e7N2icRaW7Rx/1QKVTbESepUBDsX0OfT0up6, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:12:34 DEBUG : pacer: Rate limited, increasing sleep to 160ms run.go:281: Retry Put of "dir/file2" to S3 bucket rclone-test-dabilar2bopi: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 3B589C30486F1097:B, HostID: 7GjG7Xslmxf6kZyfwF7Qmgt/hA0qY176aDa6L/l4e7N2icRaW7Rx/1QKVTbESepUBDsX0OfT0up6, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:13:07 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9DF14F2BF899B577:B, HostID: 5Z3nqdZ0luEpbVmBdRX0dDMCzx95AkvURL7rHh8Jo2RmFOGoe/TbeujRW8gaD2b4Qkr8uMAet/SZ, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:13:07 DEBUG : pacer: Rate limited, increasing sleep to 320ms run.go:281: Retry Put of "dir/file2" to S3 bucket rclone-test-dabilar2bopi: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9DF14F2BF899B577:B, HostID: 5Z3nqdZ0luEpbVmBdRX0dDMCzx95AkvURL7rHh8Jo2RmFOGoe/TbeujRW8gaD2b4Qkr8uMAet/SZ, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:13:14 DEBUG : pacer: Reducing sleep to 240ms 2025/07/30 01:13:15 DEBUG : pacer: Reducing sleep to 180ms 2025/07/30 01:13:21 DEBUG : pacer: Reducing sleep to 135ms 2025/07/30 01:13:25 DEBUG : pacer: Reducing sleep to 101.25ms 2025/07/30 01:13:30 DEBUG : pacer: Reducing sleep to 75.9375ms 2025/07/30 01:13:33 DEBUG : pacer: Reducing sleep to 56.953125ms 2025/07/30 01:13:33 DEBUG : pacer: Reducing sleep to 42.714843ms 2025/07/30 01:13:35 DEBUG : pacer: Reducing sleep to 32.036132ms 2025/07/30 01:13:35 DEBUG : pacer: Reducing sleep to 24.027099ms 2025/07/30 01:13:36 DEBUG : pacer: Reducing sleep to 18.020324ms 2025/07/30 01:13:36 DEBUG : WaitForWriters: timeout=30s 2025/07/30 01:13:36 DEBUG : dir/subdir: Looking for writers 2025/07/30 01:13:36 DEBUG : dir: Looking for writers 2025/07/30 01:13:36 DEBUG : file1: reading active writers 2025/07/30 01:13:36 DEBUG : file2: reading active writers 2025/07/30 01:13:36 DEBUG : subdir: reading active writers 2025/07/30 01:13:36 DEBUG : Looking for writers 2025/07/30 01:13:36 DEBUG : dir: reading active writers 2025/07/30 01:13:36 DEBUG : >WaitForWriters: 2025/07/30 01:13:41 DEBUG : pacer: Reducing sleep to 13.515243ms 2025/07/30 01:13:42 DEBUG : pacer: Reducing sleep to 10.136432ms 2025/07/30 01:13:42 DEBUG : pacer: Reducing sleep to 0s --- PASS: TestDirHandleReaddir (417.05s) === RUN TestDirHandleReaddirnames run.go:180: Remote "S3 bucket rclone-test-dabilar2bopi", Local "Local file system at /tmp/rclone1500284885", Modify Window "1ns" 2025/07/30 01:13:42 INFO : S3 bucket rclone-test-dabilar2bopi: poll-interval is not supported by this remote 2025/07/30 01:15:43 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 22F079FBF8F7A64B:B, HostID: 9f+vVt/KxVFlKe5xzc4aOEXsVZZkPIV8EFP0b3ibqMQaLsYL9VOq7IJDZZId7hST3SvW0bzDIKC2, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:15:43 DEBUG : pacer: Rate limited, increasing sleep to 10ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 22F079FBF8F7A64B:B, HostID: 9f+vVt/KxVFlKe5xzc4aOEXsVZZkPIV8EFP0b3ibqMQaLsYL9VOq7IJDZZId7hST3SvW0bzDIKC2, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:16:16 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 126955E27540C946:B, HostID: g1NPANSNvEE9bloYe8csRBG9yQAdVUI3YL//vdQF90ve42HHjrfQOu+RF6sj90zESDsnPZpetxsL, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:16:16 DEBUG : pacer: Rate limited, increasing sleep to 20ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 126955E27540C946:B, HostID: g1NPANSNvEE9bloYe8csRBG9yQAdVUI3YL//vdQF90ve42HHjrfQOu+RF6sj90zESDsnPZpetxsL, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:16:23 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: F45863A5CFCF611E:A, HostID: xEacau02BGpt9Oa8Of5egj3hB+ZZ2NK50covNNxuehLFCLT4mnlC5iC3eCbFyYIfa5wJPKRylwy9, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:16:23 DEBUG : pacer: Rate limited, increasing sleep to 40ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: F45863A5CFCF611E:A, HostID: xEacau02BGpt9Oa8Of5egj3hB+ZZ2NK50covNNxuehLFCLT4mnlC5iC3eCbFyYIfa5wJPKRylwy9, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:16:45 DEBUG : dir: forgetting directory cache 2025/07/30 01:16:45 DEBUG : forgetting directory cache 2025/07/30 01:16:45 DEBUG : dir: forgetting directory cache 2025/07/30 01:16:57 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 671D6DC58C8D5769:A, HostID: i8VEl9Y5ZxOhdFs04GgDIQKRUg3SYU5FMTGb41byKpcMfF5mx6lzGUZwQCweglz5ZFXXEMrteMuF, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:16:57 DEBUG : pacer: Rate limited, increasing sleep to 80ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 671D6DC58C8D5769:A, HostID: i8VEl9Y5ZxOhdFs04GgDIQKRUg3SYU5FMTGb41byKpcMfF5mx6lzGUZwQCweglz5ZFXXEMrteMuF, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:17:32 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 3F57386536929372:A, HostID: VyBmCwQHbP/Cxk4Jc+1eAC7+NNRiK3qDgUHt7NFvergVWZ4PoPnk9X0lzhL/aZSw0QGQIdq80VDl, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:17:32 DEBUG : pacer: Rate limited, increasing sleep to 160ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 3F57386536929372:A, HostID: VyBmCwQHbP/Cxk4Jc+1eAC7+NNRiK3qDgUHt7NFvergVWZ4PoPnk9X0lzhL/aZSw0QGQIdq80VDl, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:19:01 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 12FDEE8259126AB3:A, HostID: 6Qx4h2BrkkoMke0QjWyH0j/vqMBHbaW+N1qgEX3/+STsfYKXkjQV7WvyvFiPosQY7W6T6CdTZrh7, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:19:01 DEBUG : pacer: Rate limited, increasing sleep to 320ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 12FDEE8259126AB3:A, HostID: 6Qx4h2BrkkoMke0QjWyH0j/vqMBHbaW+N1qgEX3/+STsfYKXkjQV7WvyvFiPosQY7W6T6CdTZrh7, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:19:44 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: AEEA0F511D9DC193:A, HostID: MsV5+Kbj58ck6AKMpv9ENviZ8wj62luPshnF3kTj4fhKZXsWVbEotQvFtvWOcBE91V6ib7CuB5c3, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:19:44 DEBUG : pacer: Rate limited, increasing sleep to 640ms run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: AEEA0F511D9DC193:A, HostID: MsV5+Kbj58ck6AKMpv9ENviZ8wj62luPshnF3kTj4fhKZXsWVbEotQvFtvWOcBE91V6ib7CuB5c3, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:20:22 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 85B2B1D92579C84F:B, HostID: FjP8RRymphFEffZi49nSDyk8wGVM9VPZAaUuLQxoickVgOjm4FwLT2d2M5WnUUVe63GMjOCA9xNY, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:20:22 DEBUG : pacer: Rate limited, increasing sleep to 1.28s run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 85B2B1D92579C84F:B, HostID: FjP8RRymphFEffZi49nSDyk8wGVM9VPZAaUuLQxoickVgOjm4FwLT2d2M5WnUUVe63GMjOCA9xNY, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:21:06 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9A3CD3C119B9D350:A, HostID: KhUqMujeXJBufwVuZKvBGOXCZv0nfrtiEWDo9HhZgcWnAKnI2druHFFfhsQ/+WAWQlpdYoyj/UyG, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:21:06 DEBUG : pacer: Rate limited, increasing sleep to 2s run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9A3CD3C119B9D350:A, HostID: KhUqMujeXJBufwVuZKvBGOXCZv0nfrtiEWDo9HhZgcWnAKnI2druHFFfhsQ/+WAWQlpdYoyj/UyG, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:21:53 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: B2FA91F7FFAB522C:A, HostID: 71IQ4oY9twc8jYJXYVBXxKN5ryFYSoWVvkeahnT6TY/UtAvp/ZUsI24LOHqt82x492JlHKTy3f9F, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:285: Failed to put "dir/file1" to "S3 bucket rclone-test-dabilar2bopi": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: B2FA91F7FFAB522C:A, HostID: 71IQ4oY9twc8jYJXYVBXxKN5ryFYSoWVvkeahnT6TY/UtAvp/ZUsI24LOHqt82x492JlHKTy3f9F, api error InternalError: We encountered an internal error. Please retry the operation again later. 2025/07/30 01:21:53 DEBUG : WaitForWriters: timeout=30s 2025/07/30 01:21:53 DEBUG : Looking for writers 2025/07/30 01:21:53 DEBUG : >WaitForWriters: 2025/07/30 01:23:35 DEBUG : forgetting directory cache 2025/07/30 01:23:35 DEBUG : dir: forgetting directory cache 2025/07/30 01:23:35 DEBUG : dir/subdir: forgetting directory cache 2025/07/30 01:23:42 DEBUG : forgetting directory cache 2025/07/30 01:30:30 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 3BA405A491CF2261:B, HostID: CCKsMQ4NXY5p0Usronbz26+UtjpQwxzhvk3iONIAGizRF8vPz+FaJUYPbkc9OG8SvpjL280jnnxc, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:38:39 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 85B737CC3C23B20D:A, HostID: p/gQvWJazolg0eoHpWKfmOEZk4YA20WUrWFaXfm/ha8ieUNbBxBujQ1whsRbdCaEx6fKxx9QKYyh, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/vfs/dir_test.go:25 /home/rclone/go/src/github.com/rclone/rclone/vfs/dir_handle_test.go:94 Error: Received unexpected error: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 85B737CC3C23B20D:A, HostID: p/gQvWJazolg0eoHpWKfmOEZk4YA20WUrWFaXfm/ha8ieUNbBxBujQ1whsRbdCaEx6fKxx9QKYyh, api error InternalError: We encountered an internal error. Please retry the operation again later. Test: TestDirHandleReaddirnames --- FAIL: TestDirHandleReaddirnames (1496.88s) === RUN TestDirMethods run.go:180: Remote "S3 bucket rclone-test-dabilar2bopi", Local "Local file system at /tmp/rclone1500284885", Modify Window "1ns" 2025/07/30 01:38:39 INFO : S3 bucket rclone-test-dabilar2bopi: poll-interval is not supported by this remote 2025/07/30 01:39:12 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9560BCFBBF07EA6E:B, HostID: 2Q8QS4MkGo5BNtwWzOLrdbeTwBOKl7cKOvlaO2KT7gZbsTPoeWGK+lwnWE3lRPM8mDTuS42BAzSG, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9560BCFBBF07EA6E:B, HostID: 2Q8QS4MkGo5BNtwWzOLrdbeTwBOKl7cKOvlaO2KT7gZbsTPoeWGK+lwnWE3lRPM8mDTuS42BAzSG, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:39:52 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 80E71383C4BF9AB7:A, HostID: QdWY+sHTCk2itHtt9ahc7vp88JRPvsGtR02ZwiW8H4dgjEBkczZXl8cBM15LRIuEZqv8kb1JH5Zh, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 80E71383C4BF9AB7:A, HostID: QdWY+sHTCk2itHtt9ahc7vp88JRPvsGtR02ZwiW8H4dgjEBkczZXl8cBM15LRIuEZqv8kb1JH5Zh, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:40:34 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 8CD9171FB43CACF0:B, HostID: uQQYWueit+7RJk0cRhFEXkBTQxljNRK4z2UYrGN9kKbzpBZLWYs9gpVZj/W2eDAwoMcEqybmXYvV, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 8CD9171FB43CACF0:B, HostID: uQQYWueit+7RJk0cRhFEXkBTQxljNRK4z2UYrGN9kKbzpBZLWYs9gpVZj/W2eDAwoMcEqybmXYvV, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:41:15 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 987B98D1AF545AD2:B, HostID: zJyaCUl0TB5PnewW3MYT5yE0Agw0o9+jntyKWB5vHbzINzm/wKcjQzJsE8s7TobLURfVKVmzhzYV, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 987B98D1AF545AD2:B, HostID: zJyaCUl0TB5PnewW3MYT5yE0Agw0o9+jntyKWB5vHbzINzm/wKcjQzJsE8s7TobLURfVKVmzhzYV, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:41:57 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: E5BCF786A0F56520:B, HostID: JDgzJ0JsWRda43VezEub30hlPCOR+vMr+8pznGZ1dTLxcaFVA0a/H38xFlApdHw71S9S7sNwsc/j, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: E5BCF786A0F56520:B, HostID: JDgzJ0JsWRda43VezEub30hlPCOR+vMr+8pznGZ1dTLxcaFVA0a/H38xFlApdHw71S9S7sNwsc/j, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:42:37 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: BDC21CAB19172739:B, HostID: MX91qcjFcLshAZnBMZaS2kUM/ltUfWShcflbk7luleXlqebi9IVWuMOTOaghKXoZdrMyNZXJp1ZY, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: BDC21CAB19172739:B, HostID: MX91qcjFcLshAZnBMZaS2kUM/ltUfWShcflbk7luleXlqebi9IVWuMOTOaghKXoZdrMyNZXJp1ZY, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:43:15 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9FCEB68CEAB77499:B, HostID: sqJTnL58jv5t95aHp+O7sls/5JjyzU2ui2vM6cyN7IEbqyFt6bRwMtB4whnGsWDJNXeN1k186+fe, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 9FCEB68CEAB77499:B, HostID: sqJTnL58jv5t95aHp+O7sls/5JjyzU2ui2vM6cyN7IEbqyFt6bRwMtB4whnGsWDJNXeN1k186+fe, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:43:57 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 5707972B76D93F90:A, HostID: qfz0LO3zgTBM2frXSAEVcmWE3NptG7DX0QVv5Ft2JJExOM0m9J7iyfYSaqXEza9r33efcLjFEa5m, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 8/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 5707972B76D93F90:A, HostID: qfz0LO3zgTBM2frXSAEVcmWE3NptG7DX0QVv5Ft2JJExOM0m9J7iyfYSaqXEza9r33efcLjFEa5m, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:44:38 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 62BD05FC93B009BF:B, HostID: tAH+6HNJoiXiFsks6qIREd2jpKS4S6gTvvItAuntguysbT+ATfXEpvkjnbNmEbOYrEeFxhYB81I6, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 9/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 62BD05FC93B009BF:B, HostID: tAH+6HNJoiXiFsks6qIREd2jpKS4S6gTvvItAuntguysbT+ATfXEpvkjnbNmEbOYrEeFxhYB81I6, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 01:45:16 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 614B83B19FEBCCEF:B, HostID: CbbXAcrOfP5zYoW28vTRT53L7B4XHEfEuSEcaPppU7GVGOD3SKgxj74b0n4T2ymDTjMrYHbI9tRU, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:285: Failed to put "dir/file1" to "S3 bucket rclone-test-dabilar2bopi": operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 614B83B19FEBCCEF:B, HostID: CbbXAcrOfP5zYoW28vTRT53L7B4XHEfEuSEcaPppU7GVGOD3SKgxj74b0n4T2ymDTjMrYHbI9tRU, api error InternalError: We encountered an internal error. Please retry the operation again later. 2025/07/30 01:45:16 DEBUG : WaitForWriters: timeout=30s 2025/07/30 01:45:16 DEBUG : Looking for writers 2025/07/30 01:45:16 DEBUG : >WaitForWriters: 2025/07/30 01:48:39 DEBUG : forgetting directory cache 2025/07/30 01:53:51 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 10BBEDD17168EFAF:A, HostID: AKJZTy3gwV/Je+gs3ORo6iCUqc2bP7Ie1IBnyaSJCt29jfPZTg4Y8TNfbUvn0Tfjj44+0APL5GuN, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:01:49 DEBUG : pacer: low level retry 2/2 (error operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 2B41B399102A8E58:A, HostID: JBoJTvg67FwC+H2KsrB7+E6jULiysLRH8TXC59zGYmXgu2Wf/Ucvvzly98HtUzIMxc3ujt2yqaf8, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:164: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:164 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 /usr/local/go/src/runtime/panic.go:636 /usr/local/go/src/testing/testing.go:1041 /usr/local/go/src/testing/testing.go:1125 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:285 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 /home/rclone/go/src/github.com/rclone/rclone/vfs/dir_test.go:25 /home/rclone/go/src/github.com/rclone/rclone/vfs/dir_test.go:36 Error: Received unexpected error: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 500, RequestID: 2B41B399102A8E58:A, HostID: JBoJTvg67FwC+H2KsrB7+E6jULiysLRH8TXC59zGYmXgu2Wf/Ucvvzly98HtUzIMxc3ujt2yqaf8, api error InternalError: We encountered an internal error. Please retry the operation again later. Test: TestDirMethods --- FAIL: TestDirMethods (1389.36s) === RUN TestDirForgetAll run.go:180: Remote "S3 bucket rclone-test-dabilar2bopi", Local "Local file system at /tmp/rclone1500284885", Modify Window "1ns" 2025/07/30 02:01:49 INFO : S3 bucket rclone-test-dabilar2bopi: poll-interval is not supported by this remote 2025/07/30 02:01:58 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 3084CE74EEF9A46A:A, HostID: o7WRmLlZl7RrxBG2dZRkVkSTtDnky/b+SvQfFClVnuKC+rSEiwR+yIpghwJ6ohwPr48x4hgPyCWt, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 1/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 3084CE74EEF9A46A:A, HostID: o7WRmLlZl7RrxBG2dZRkVkSTtDnky/b+SvQfFClVnuKC+rSEiwR+yIpghwJ6ohwPr48x4hgPyCWt, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:02:33 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 1FD349DACBC07849:B, HostID: k9huI7vx8jKerkoel/n+PWaDkn4vnpl70e16EfOD9AAUBhU5rAGjRS/BX9Gl+fIPwQ2xzXhrUyOy, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 2/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 1FD349DACBC07849:B, HostID: k9huI7vx8jKerkoel/n+PWaDkn4vnpl70e16EfOD9AAUBhU5rAGjRS/BX9Gl+fIPwQ2xzXhrUyOy, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:03:12 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 8D3EBA4DD7D86F83:B, HostID: rANlwVcwum9aaBJcnhkp61E6UlI56Usv5pES47Pdnvtua3gdtqQ+crWx2zPBCFXp+FhMu8Ws4J5R, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 3/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 8D3EBA4DD7D86F83:B, HostID: rANlwVcwum9aaBJcnhkp61E6UlI56Usv5pES47Pdnvtua3gdtqQ+crWx2zPBCFXp+FhMu8Ws4J5R, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:04:05 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 7E7D44D5A8EE734A:B, HostID: Sw2UtzVSVrzhk2D2H+fRmUZ5FtP35lFTC9SlyxFKi4I35ZhrKvA9pVxPOKpjci/iVOi/ToafzLJ/, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 4/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 7E7D44D5A8EE734A:B, HostID: Sw2UtzVSVrzhk2D2H+fRmUZ5FtP35lFTC9SlyxFKi4I35ZhrKvA9pVxPOKpjci/iVOi/ToafzLJ/, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:04:46 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 196AD492081F9A79:A, HostID: urFL86wfu6wd97SMlYWSFolgcjOUtrTI4jKqxWMGujNumCniYolJoIW1Un7E1/LXTmX39ArHAbva, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 5/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 196AD492081F9A79:A, HostID: urFL86wfu6wd97SMlYWSFolgcjOUtrTI4jKqxWMGujNumCniYolJoIW1Un7E1/LXTmX39ArHAbva, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:05:28 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 100BC5239F8A57B7:A, HostID: Un5ZROoHFqBGpZUQrglWlVoVMrK8KFZ+4yLk+F8Wb+bzCEMXM8R/QRZyyDHRk9yjmUnFl8tKYwoi, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 6/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: 100BC5239F8A57B7:A, HostID: Un5ZROoHFqBGpZUQrglWlVoVMrK8KFZ+4yLk+F8Wb+bzCEMXM8R/QRZyyDHRk9yjmUnFl8tKYwoi, api error InternalError: We encountered an internal error. Please retry the operation again later.) 2025/07/30 02:06:08 DEBUG : pacer: low level retry 1/1 (error operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: F98E028D102C1354:B, HostID: 0rXoH5ToiU8DV9Wa6vUhrdYe3ikBiIBhOwWjc9uXBSCMcy77lKMabiDgLFM2et0jzvi5+EB4ut+e, api error InternalError: We encountered an internal error. Please retry the operation again later.) run.go:281: Retry Put of "dir/file1" to S3 bucket rclone-test-dabilar2bopi: 7/10 (operation error S3: PutObject, exceeded maximum number of attempts, 1, https response error StatusCode: 500, RequestID: F98E028D102C1354:B, HostID: 0rXoH5ToiU8DV9Wa6vUhrdYe3ikBiIBhOwWjc9uXBSCMcy77lKMabiDgLFM2et0jzvi5+EB4ut+e, api error InternalError: We encountered an internal error. Please retry the operation again later.) panic: test timed out after 1h0m0s running tests: TestDirForgetAll (4m54s) goroutine 765 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 5 minutes]: testing.(*T).Run(0xc0005dbc00, {0x274ac8e?, 0xc000551ad8?}, 0x28872d8) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc0005dbc00) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc0005dbc00, 0xc000551c18) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc000586000, {0x3fc67a0, 0x55, 0x55}, {0xee01b6723?, 0x3ff96c0?, 0x3ff96c0?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc000304c80) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc000304c80) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/vfs.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/vfs/vfs_test.go:37 main.main() _testmain.go:217 +0xa9 goroutine 60 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc0002d0f00, {0x2bcab20, 0xc0004cc230}) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:339 +0x145 created by github.com/rclone/rclone/fs/accounting.(*StatsInfo)._startAverageLoop in goroutine 50 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:376 +0x117 goroutine 15 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 16 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 707 [select]: net/http.(*persistConn).roundTrip(0xc0003cb200, 0xc0001c4910) /usr/local/go/src/net/http/transport.go:2826 +0x785 net/http.(*Transport).roundTrip(0xc0004d7ba0, 0xc0007b6a00) /usr/local/go/src/net/http/transport.go:696 +0xb9a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:30 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc000332850, 0xc0007b6a00) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:362 +0x345 net/http.send(0xc0007b6a00, {0x2ba3a60, 0xc000332850}, {0x201?, 0x2381c00?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e2 net/http.(*Client).send(0xc0003a5dd0, 0xc0007b6a00, {0xc0007a90e0?, 0x2641900?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x91 net/http.(*Client).do(0xc0003a5dd0, 0xc0007b6a00) /usr/local/go/src/net/http/client.go:728 +0x989 net/http.(*Client).Do(0xc0007b4740?, 0x26f6620?) /usr/local/go/src/net/http/client.go:587 +0x13 github.com/aws/smithy-go/transport/http.(*timedClientDo).Do(0xc000a32978, 0xc0007b6a00) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/transport/http/metrics.go:47 +0xb0 github.com/aws/smithy-go/transport/http.ClientHandler.Handle({{0x76eb38302b40?, 0xc0003a5dd0?}, {0x2be6428?, 0x401cde0?}}, {0x2bcaae8?, 0xc000486960?}, {0x26deb20, 0xc000492bd0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/transport/http/client.go:92 +0x358 github.com/aws/smithy-go/middleware.deserializeWrapHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:190 github.com/aws/smithy-go/transport/http.(*RequestResponseLogger).HandleDeserialize(0xc0007aab08, {0x2bcaae8, 0xc000486960}, {{0x26deb20, 0xc000492bd0}}, {0x2bac5e0, 0xc000a8ec60}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/transport/http/middleware_http_logging.go:58 +0x28f github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/aws-sdk-go-v2/aws/middleware.RecordResponseTiming.HandleDeserialize({}, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8ee0?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.36.3/aws/middleware/middleware.go:57 +0x6e github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/aws-sdk-go-v2/internal/middleware.(*AddTimeOffsetMiddleware).HandleDeserialize(0xc0005b2200, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600, 0xc0007a8f00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.36.3/internal/middleware/middleware.go:41 +0xa2 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_deserializeOpPutObject).HandleDeserialize(0x8?, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8f20?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/deserializers.go:12020 +0x8e github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*metadataRetriever).HandleDeserialize(0xc0007b68c0?, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8f40?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.15/metadata_retriever.go:32 +0x5c github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*errorWrapper).HandleDeserialize(0xc000753ee8?, {0x2bcaae8?, 0xc000486960?}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8f60?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.15/response_error_middleware.go:30 +0x53 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/smithy-go/transport/http.(*closeResponseBody).HandleDeserialize(0x2bab5e0?, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8f80?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/transport/http/middleware_close_response_body.go:60 +0x53 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/smithy-go/transport/http.(*errorCloseResponseBodyMiddleware).HandleDeserialize(0x2737ea6?, {0x2bcaae8?, 0xc000486960?}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8fa0?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/transport/http/middleware_close_response_body.go:29 +0x53 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/aws-sdk-go-v2/aws/middleware.AddRawResponse.HandleDeserialize({}, {0x2bcaae8?, 0xc000486960?}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac600?, 0xc0007a8fc0?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.36.3/aws/middleware/middleware.go:154 +0x62 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:206 github.com/aws/smithy-go/middleware.(*DeserializeStep).HandleMiddleware(0x2bb0e00?, {0x2bcaae8, 0xc000486960}, {0x26deb20, 0xc000492bd0}, {0x2bab660, 0xc0007a8720}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_deserialize.go:120 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.finalizeWrapHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*signRequestMiddleware).HandleFinalize(0x2bcaae8?, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5a0, 0xc000a8e970}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/auth.go:346 +0x4f8 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*setLegacyContextSigningOptionsMiddleware).HandleFinalize(0x26deb0a?, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8ba0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/auth.go:54 +0x5e2 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*ContentSHA256Header).HandleFinalize(0x14c6218?, {0x2bcaae8, 0xc000486960}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8bc0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.36.3/aws/signer/v4/middleware.go:242 +0x1ba github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*UnsignedPayload).HandleFinalize(0x2bcaae8?, {0x2bcaae8, 0xc000486870}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8be0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.36.3/aws/signer/v4/middleware.go:123 +0xdb github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/checksum.(*AddInputChecksumTrailer).HandleFinalize(0xc0007aab04, {0x2bcaae8, 0xc000486870}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8c00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.7.2/middleware_compute_input_checksum.go:261 +0x2e6 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/checksum.(*ComputeInputPayloadChecksum).HandleFinalize(0xc0007aab00, {0x2bcaae8, 0xc000486870}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8c20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.7.2/middleware_compute_input_checksum.go:112 +0x283 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*disableHTTPSMiddleware).HandleFinalize(0x14c6218?, {0x2bcaae8?, 0xc000486870?}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0?, 0xc0007a8c40?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:1276 +0x143 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*resolveEndpointV2Middleware).HandleFinalize(0xc00067f400, {0x2bcaae8, 0xc000486870}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8c60}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/endpoints.go:7643 +0xe82 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*getIdentityMiddleware).HandleFinalize(0xc00067f200, {0x2bcaae8, 0xc000486120}, {{0x26deb20?, 0xc000492bd0?}}, {0x2bac5c0, 0xc0007a8c80}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/auth.go:288 +0x3a2 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_finalize.go:200 ...44 frames elided... github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*s3ObjectLambdaEndpoint).HandleSerialize(0xc000a91920, {0x2bcaae8, 0xc000492480}, {{0x23498a0?, 0xc0001964e0?}, {0x26deb20?, 0xc0004921e0?}}, {0x2bac540, 0xc0007a8980}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/internal/customizations/s3_object_lambda.go:34 +0x697 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*processARNResource).HandleSerialize(0xc000a91860, {0x2bcaae8, 0xc000492480}, {{0x23498a0?, 0xc0001964e0?}, {0x26deb20?, 0xc0004921e0?}}, {0x2bac540, 0xc0007a89a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/internal/customizations/process_arn_resource.go:54 +0x1219 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*ResolveEndpoint).HandleSerialize(0xc0004cc550, {0x2bcaae8, 0xc000492480}, {{0x23498a0?, 0xc0001964e0?}, {0x26deb20?, 0xc0004921e0?}}, {0x2bac540, 0xc0007a89c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/endpoints.go:89 +0x755 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*setOperationInputMiddleware).HandleSerialize(0x401cde0?, {0x2bcaae8?, 0xc000492360?}, {{0x23498a0?, 0xc0001964e0?}, {0x26deb20?, 0xc0004921e0?}}, {0x2bac540, 0xc0007a89e0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:351 +0xb5 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*putBucketContextMiddleware).HandleSerialize(0x3f9e0d0?, {0x2bcaae8, 0xc0004921b0}, {{0x23498a0?, 0xc0001964e0?}, {0x26deb20?, 0xc0004921e0?}}, {0x2bac540, 0xc0007a8a00}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/bucket_context.go:26 +0xdc github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/aws-sdk-go-v2/service/s3.(*spanBuildRequestStart).HandleSerialize(0x90?, {0x2bcaae8?, 0xc0004921b0?}, {{0x23498a0?, 0xc0001964e0?}, {0x26deb20?, 0xc0004921e0?}}, {0x2bac540, 0xc0007a8a20}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:1334 +0xaf github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:208 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware(0xc000a8e740, {0x2bcaae8, 0xc0004921b0}, {0x23498a0, 0xc0001964e0}, {0x2bab6c0, 0xc0007a87a0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_serialize.go:122 +0x26f github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:184 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeEnd).HandleInitialize(0x13?, {0x2bcaae8?, 0xc0004921b0?}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac560, 0xc000a8e7e0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:1317 +0x8b github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*validateOpPutObject).HandleInitialize(0x1?, {0x2bcaae8, 0xc0004921b0}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac580, 0xc0007a8800}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/validators.go:1730 +0xc6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize(0x0?, {0x2bcaae8?, 0xc000492000?}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac580, 0xc0007a8820}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/logging.go:45 +0x8c github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*ARNLookup).HandleInitialize(0xc000492000?, {0x2bcaae8, 0xc000492000}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac580, 0xc0007a8840}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.15/arn_lookup.go:39 +0x262 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*legacyEndpointContextSetter).HandleInitialize(0x14c6218?, {0x2bcaae8?, 0xc000492000?}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac580?, 0xc0007a8860?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:414 +0x9e github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize({{0x2725c9a, 0x2}, {0x0, 0x0}, {0x2737ea6, 0x9}, {0x27389ef, 0x9}}, {0x2bcaae8, 0xc00089be60}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.36.3/aws/middleware/metadata.go:41 +0x1c6 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/internal/checksum.(*SetupInputContext).HandleInitialize(0xc000a326a8?, {0x2bcaae8, 0xc00089be60}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac580, 0xc0007a88a0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.7.2/middleware_setup_context.go:62 +0xf2 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/aws-sdk-go-v2/service/s3.(*spanInitializeStart).HandleInitialize(0x41821f?, {0x2bcaae8?, 0xc00089be60?}, {{0x23498a0?, 0xc0001964e0?}}, {0x2bac580, 0xc0007a88c0}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:1299 +0x85 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:200 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware(0xc00073eea0?, {0x2bcaae8, 0xc00089be60}, {0x23498a0, 0xc0001964e0}, {0x2bab6c0, 0xc0007a87c0}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/step_initialize.go:114 +0x22c github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/middleware.go:57 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware(0x418201?, {0x2bcaae8, 0xc00089be60}, {0x23498a0, 0xc0001964e0}, {0x2bab660?, 0xc0007a8720?}) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/stack.go:109 +0x183 github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...) /home/rclone/go/pkg/mod/github.com/aws/smithy-go@v1.22.3/middleware/middleware.go:57 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).invokeOperation(0xc0001a5c00, {0x2bcaa08?, 0x401cde0?}, {0x27389ef, 0x9}, {0x23498a0, 0xc0001964e0}, {0xc000a8e6e0, 0x2, 0x2}, ...) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_client.go:301 +0xc15 github.com/aws/aws-sdk-go-v2/service/s3.(*Client).PutObject(0xc0007597f8?, {0x2bcaa08?, 0x401cde0?}, 0xc000759698?, {0xc000a8e6e0?, 0xc000102700?, 0xc0007596c8?}) /home/rclone/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/s3@v1.80.0/api_op_PutObject.go:138 +0x10c github.com/rclone/rclone/backend/s3.(*Object).uploadSinglepartPutObject.func2() /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6633 +0x5e github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0x2?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:86 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc00058cc60, 0xc0004cc370, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:199 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:227 github.com/rclone/rclone/backend/s3.(*Object).uploadSinglepartPutObject(0xc0005a6b40, {0x2bcaa08, 0x401cde0}, 0xc0001964e0, 0xc0003885b0?, {0x2ba2800, 0xc00089bb00}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6632 +0x3ba github.com/rclone/rclone/backend/s3.(*Object).Update(0xc0005a6b40, {0x2bcaa08, 0x401cde0}, {0x2ba2800, 0xc00089bb00}, {0x2bd6860, 0xc0003885b0}, {0x0, 0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:6903 +0x414 github.com/rclone/rclone/backend/s3.(*Fs).Put(0xc00041f208, {0x2bcaa08, 0x401cde0}, {0x2ba2800, 0xc00089bb00}, {0x2bd6860, 0xc0003885b0}, {0x0, 0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/backend/s3/s3.go:4840 +0x108 github.com/rclone/rclone/fstest.(*Run).WriteObjectTo(0xc000388d20, {0x2bcaa08, 0x401cde0}, {0x2be2e40, 0xc00041f208}, {0x2737069, 0x9}, {0x274511a, 0xe}, {0x1dcd64ff, ...}, ...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:275 +0x3f2 github.com/rclone/rclone/fstest.(*Run).WriteObject(...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 github.com/rclone/rclone/vfs.dirCreate(0xc0007e96c0) /home/rclone/go/src/github.com/rclone/rclone/vfs/dir_test.go:25 +0xbb github.com/rclone/rclone/vfs.TestDirForgetAll(0xc0007e96c0) /home/rclone/go/src/github.com/rclone/rclone/vfs/dir_test.go:87 +0x4b testing.tRunner(0xc0007e96c0, 0x28872d8) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 753 [select]: net/http.(*persistConn).writeLoop(0xc0003cb200) /usr/local/go/src/net/http/transport.go:2590 +0xe7 created by net/http.(*Transport).dialConn in goroutine 747 /usr/local/go/src/net/http/transport.go:1945 +0x17a5 goroutine 752 [IO wait]: internal/poll.runtime_pollWait(0x76eb822447c8, 0x72) /usr/local/go/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc0005ea780?, 0xc000796600?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0005ea780, {0xc000796600, 0x1300, 0x1300}) /usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc0005ea780, {0xc000796600?, 0x12fb?, 0xc0007a9420?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc00007e708, {0xc000796600?, 0xc000796600?, 0x5?}) /usr/local/go/src/net/net.go:194 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000a84120, {0xc000796600?, 0x76eb3839c448?, 0x76eb822cc108?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x29 crypto/tls.(*atLeastReader).Read(0xc000a84240, {0xc000796600?, 0x12fb?, 0x0?}) /usr/local/go/src/crypto/tls/conn.go:809 +0x3b bytes.(*Buffer).ReadFrom(0xc000201438, {0x2ba5140, 0xc000a84240}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc000201188, {0x76eb38305d18, 0xc000a84120}, 0x1300?) /usr/local/go/src/crypto/tls/conn.go:831 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc000201188, 0x0) /usr/local/go/src/crypto/tls/conn.go:629 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:591 crypto/tls.(*Conn).Read(0xc000201188, {0xc000553000, 0x1000, 0x76eb822cca78?}) /usr/local/go/src/crypto/tls/conn.go:1385 +0x145 net/http.(*persistConn).Read(0xc0003cb200, {0xc000553000?, 0x412565?, 0x68?}) /usr/local/go/src/net/http/transport.go:2122 +0x47 bufio.(*Reader).fill(0xc000a607e0) /usr/local/go/src/bufio/bufio.go:113 +0x103 bufio.(*Reader).Peek(0xc000a607e0, 0x1) /usr/local/go/src/bufio/bufio.go:152 +0x53 net/http.(*persistConn).readLoop(0xc0003cb200) /usr/local/go/src/net/http/transport.go:2275 +0x172 created by net/http.(*Transport).dialConn in goroutine 747 /usr/local/go/src/net/http/transport.go:1944 +0x174f "./vfs.test -test.v -test.timeout 1h0m0s -remote TestS3Wasabi: -verbose" - Finished ERROR in 1h0m0.059020992s (try 1/5): exit status 2: Failed [TestDirHandleReaddirnames TestDirMethods]