"./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -test.run '^(TestServerSideCopyOverSelf|TestSyncBackupDirSuffixOnly|TestSyncBackupDirWithSuffix|TestSyncBackupDirWithSuffixKeepExtension)$'" - Starting (try 2/5) 2024/11/05 03:04:10 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sozagov0moqo" 2024/11/05 03:04:10 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2024/11/05 03:04:11 DEBUG : Creating backend with remote "/tmp/rclone3545047134" === RUN TestServerSideCopyOverSelf run.go:180: Remote "gofile root 'rclone-test-sozagov0moqo'", Local "Local file system at /tmp/rclone3545047134", Modify Window "1s" 2024/11/05 03:04:12 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:04:17 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/11/05 03:04:17 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/11/05 03:04:17 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:04:22 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2024/11/05 03:04:22 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2024/11/05 03:04:22 DEBUG : pacer: Reducing sleep to 20ms 2024/11/05 03:04:22 DEBUG : pacer: Reducing sleep to 10ms 2024/11/05 03:04:23 DEBUG : Upload server store4 (eu) responded in 125.905331ms 2024/11/05 03:04:23 DEBUG : Upload server store10 (eu) responded in 126.845577ms 2024/11/05 03:04:23 DEBUG : sub dir/hello world: Using upload server store4 (eu) 2024/11/05 03:04:23 DEBUG : Creating backend with remote "TestGoFile:rclone-test-qaduhar9zeki" sync_test.go:626: Server side copy (if possible) gofile root 'rclone-test-sozagov0moqo' -> gofile root 'rclone-test-qaduhar9zeki' 2024/11/05 03:04:24 DEBUG : Added delayed dir = "sub dir", newDst= 2024/11/05 03:04:24 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2024/11/05 03:04:24 DEBUG : gofile root 'rclone-test-qaduhar9zeki': Waiting for checks to finish 2024/11/05 03:04:24 DEBUG : gofile root 'rclone-test-qaduhar9zeki': Waiting for transfers to finish 2024/11/05 03:04:24 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:04:29 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/11/05 03:04:29 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/11/05 03:04:30 DEBUG : pacer: Reducing sleep to 10ms 2024/11/05 03:04:31 DEBUG : sub dir/hello world: md5 = 5eb63bbbe01eeed093cb22bb8f5acdc3 OK 2024/11/05 03:04:31 INFO : sub dir/hello world: Copied (server-side copy) 2024/11/05 03:04:31 INFO : sub dir: Set directory modification time (using DirSetModTime) run.go:180: Remote "gofile root 'rclone-test-sozagov0moqo'", Local "Local file system at /tmp/rclone3545047134", Modify Window "1s" 2024/11/05 03:04:31 DEBUG : sub dir/hello world: Using upload server store10 (eu) 2024/11/05 03:04:31 DEBUG : sub dir/hello world: Removing old object on successful upload 2024/11/05 03:04:32 DEBUG : Added delayed dir = "sub dir", newDst=sub dir 2024/11/05 03:04:32 DEBUG : sub dir/hello world: Sizes differ (src 17 vs dst 11) 2024/11/05 03:04:32 DEBUG : gofile root 'rclone-test-qaduhar9zeki': Waiting for checks to finish 2024/11/05 03:04:32 DEBUG : gofile root 'rclone-test-qaduhar9zeki': Waiting for transfers to finish 2024/11/05 03:04:33 DEBUG : sub dir/hello world: md5 = 00321a507bdb420dbbd4f53c733b1505 OK 2024/11/05 03:04:33 INFO : sub dir/hello world: Copied (server-side copy) 2024/11/05 03:04:33 INFO : sub dir: Set directory modification time (using SetModTime) run.go:180: Remote "gofile root 'rclone-test-sozagov0moqo'", Local "Local file system at /tmp/rclone3545047134", Modify Window "1s" fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:295: Flushing the directory cache fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:641 Error: Should be true Test: TestServerSideCopyOverSelf Messages: listing wrong, want sub dir/hello world (17) got sub dir/hello world (11), sub dir/hello world (17) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:641 Error: Should be true Test: TestServerSideCopyOverSelf Messages: Unexpected file "sub dir/hello world" 2024/11/05 03:04:41 DEBUG : gofile root 'rclone-test-qaduhar9zeki': Purge remote --- FAIL: TestServerSideCopyOverSelf (33.22s) === RUN TestSyncBackupDirWithSuffix run.go:180: Remote "gofile root 'rclone-test-sozagov0moqo'", Local "Local file system at /tmp/rclone3545047134", Modify Window "1s" 2024/11/05 03:04:45 DEBUG : dst/one: Using upload server store10 (eu) 2024/11/05 03:04:46 DEBUG : dst/two: Using upload server store10 (eu) 2024/11/05 03:04:47 DEBUG : dst/three.txt: Using upload server store10 (eu) 2024/11/05 03:04:48 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sozagov0moqo/dst" 2024/11/05 03:04:48 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sozagov0moqo/backup" 2024/11/05 03:04:49 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/11/05 03:04:49 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for checks to finish 2024/11/05 03:04:49 DEBUG : two: Unchanged skipping 2024/11/05 03:04:49 DEBUG : one: Sizes differ (src 4 vs dst 3) 2024/11/05 03:04:49 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:04:54 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/11/05 03:04:54 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/11/05 03:04:55 DEBUG : pacer: Reducing sleep to 10ms 2024/11/05 03:04:55 INFO : one: Moved (server-side) to: one.bak 2024/11/05 03:04:55 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for transfers to finish 2024/11/05 03:04:56 DEBUG : Upload server store4 (eu) responded in 127.539414ms 2024/11/05 03:04:56 DEBUG : Upload server store5 (eu) responded in 127.823867ms 2024/11/05 03:04:56 DEBUG : one: Using upload server store5 (eu) 2024/11/05 03:04:56 DEBUG : one: md5 = c7957179c41f69d44f217a108c7915d8 OK 2024/11/05 03:04:56 INFO : one: Copied (new) 2024/11/05 03:04:56 DEBUG : Waiting for deletions to finish 2024/11/05 03:04:57 INFO : three.txt: Moved (server-side) to: three.txt.bak 2024/11/05 03:04:57 INFO : three.txt: Moved into backup dir 2024/11/05 03:04:58 DEBUG : dst/three.txt: Using upload server store10 (eu) 2024/11/05 03:04:59 DEBUG : one: Sizes differ (src 5 vs dst 4) 2024/11/05 03:04:59 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/11/05 03:04:59 DEBUG : two: Unchanged skipping 2024/11/05 03:04:59 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for checks to finish 2024/11/05 03:04:59 INFO : one.bak: Deleted 2024/11/05 03:05:00 INFO : one: Moved (server-side) to: one.bak 2024/11/05 03:05:00 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for transfers to finish 2024/11/05 03:05:00 DEBUG : one: Using upload server store5 (eu) 2024/11/05 03:05:00 DEBUG : one: md5 = 0f93e81041f0cab37c37a05ae998b219 OK 2024/11/05 03:05:00 INFO : one: Copied (new) 2024/11/05 03:05:00 DEBUG : Waiting for deletions to finish 2024/11/05 03:05:01 INFO : three.txt.bak: Deleted 2024/11/05 03:05:01 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:06 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/11/05 03:05:06 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/11/05 03:05:07 DEBUG : pacer: Reducing sleep to 10ms 2024/11/05 03:05:07 INFO : three.txt: Moved (server-side) to: three.txt.bak 2024/11/05 03:05:07 INFO : three.txt: Moved into backup dir --- PASS: TestSyncBackupDirWithSuffix (26.31s) === RUN TestSyncBackupDirWithSuffixKeepExtension run.go:180: Remote "gofile root 'rclone-test-sozagov0moqo'", Local "Local file system at /tmp/rclone3545047134", Modify Window "1s" 2024/11/05 03:05:11 DEBUG : dst/one: Using upload server store10 (eu) 2024/11/05 03:05:11 DEBUG : dst/two: Using upload server store10 (eu) 2024/11/05 03:05:12 DEBUG : dst/three.txt: Using upload server store10 (eu) 2024/11/05 03:05:12 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sozagov0moqo/dst" 2024/11/05 03:05:13 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sozagov0moqo/backup" 2024/11/05 03:05:13 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/11/05 03:05:13 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for checks to finish 2024/11/05 03:05:13 DEBUG : two: Unchanged skipping 2024/11/05 03:05:13 DEBUG : one: Sizes differ (src 4 vs dst 3) 2024/11/05 03:05:14 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:19 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/11/05 03:05:19 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/11/05 03:05:19 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:24 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2024/11/05 03:05:24 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2024/11/05 03:05:24 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:29 DEBUG : pacer: low level retry 3/10 (error Error "error-rateLimit") 2024/11/05 03:05:29 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2024/11/05 03:05:30 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:35 DEBUG : pacer: low level retry 4/10 (error Error "error-rateLimit") 2024/11/05 03:05:35 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2024/11/05 03:05:35 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:40 DEBUG : pacer: low level retry 5/10 (error Error "error-rateLimit") 2024/11/05 03:05:40 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2024/11/05 03:05:40 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:45 DEBUG : pacer: low level retry 6/10 (error Error "error-rateLimit") 2024/11/05 03:05:45 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2024/11/05 03:05:45 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:50 DEBUG : pacer: low level retry 7/10 (error Error "error-rateLimit") 2024/11/05 03:05:50 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2024/11/05 03:05:50 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:05:55 DEBUG : pacer: low level retry 8/10 (error Error "error-rateLimit") 2024/11/05 03:05:55 DEBUG : pacer: Rate limited, increasing sleep to 2.56s 2024/11/05 03:05:55 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:06:00 DEBUG : pacer: low level retry 9/10 (error Error "error-rateLimit") 2024/11/05 03:06:00 DEBUG : pacer: Rate limited, increasing sleep to 5.12s 2024/11/05 03:06:00 DEBUG : Rate limited, sleep for 5s 2024/11/05 03:06:05 DEBUG : pacer: low level retry 10/10 (error Error "error-rateLimit") 2024/11/05 03:06:05 DEBUG : pacer: Rate limited, increasing sleep to 10.24s 2024/11/05 03:06:05 ERROR : one: Couldn't move: failed to move item: Error "error-rateLimit" 2024/11/05 03:06:05 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for transfers to finish 2024/11/05 03:06:05 ERROR : gofile root 'rclone-test-sozagov0moqo/dst': not deleting files as there were IO errors 2024/11/05 03:06:05 ERROR : gofile root 'rclone-test-sozagov0moqo/dst': not deleting directories as there were IO errors sync_test.go:2385: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2385 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2435 Error: Received unexpected error: failed to move item: Error "error-rateLimit" Test: TestSyncBackupDirWithSuffixKeepExtension --- FAIL: TestSyncBackupDirWithSuffixKeepExtension (57.68s) === RUN TestSyncBackupDirSuffixOnly run.go:180: Remote "gofile root 'rclone-test-sozagov0moqo'", Local "Local file system at /tmp/rclone3545047134", Modify Window "1s" 2024/11/05 03:06:09 DEBUG : Upload server store10 (eu) responded in 38.195044ms 2024/11/05 03:06:09 DEBUG : Upload server store5 (eu) responded in 124.385253ms 2024/11/05 03:06:09 DEBUG : dst/one: Using upload server store10 (eu) 2024/11/05 03:06:10 DEBUG : dst/two: Using upload server store5 (eu) 2024/11/05 03:06:10 DEBUG : dst/three.txt: Using upload server store5 (eu) 2024/11/05 03:06:11 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sozagov0moqo/dst" 2024/11/05 03:06:11 DEBUG : one: Sizes differ (src 4 vs dst 3) 2024/11/05 03:06:11 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for checks to finish 2024/11/05 03:06:11 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/11/05 03:06:11 DEBUG : two: Unchanged skipping 2024/11/05 03:06:12 INFO : one: Moved (server-side) to: one.bak 2024/11/05 03:06:12 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for transfers to finish 2024/11/05 03:06:12 DEBUG : Upload server store5 (eu) responded in 123.273566ms 2024/11/05 03:06:12 DEBUG : Upload server store10 (eu) responded in 125.664143ms 2024/11/05 03:06:12 DEBUG : one: Using upload server store5 (eu) 2024/11/05 03:06:12 DEBUG : one: md5 = c7957179c41f69d44f217a108c7915d8 OK 2024/11/05 03:06:12 INFO : one: Copied (new) 2024/11/05 03:06:12 DEBUG : Waiting for deletions to finish 2024/11/05 03:06:13 INFO : three.txt: Moved (server-side) to: three.txt.bak 2024/11/05 03:06:13 INFO : three.txt: Moved into backup dir 2024/11/05 03:06:13 DEBUG : dst/three.txt: Using upload server store5 (eu) 2024/11/05 03:06:14 DEBUG : three.txt.bak: Excluded (Path Filter) 2024/11/05 03:06:14 DEBUG : three.txt.bak: Excluded 2024/11/05 03:06:14 DEBUG : one.bak: Excluded (Path Filter) 2024/11/05 03:06:14 DEBUG : one.bak: Excluded 2024/11/05 03:06:14 DEBUG : one: Sizes differ (src 5 vs dst 4) 2024/11/05 03:06:14 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/11/05 03:06:14 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for checks to finish 2024/11/05 03:06:14 DEBUG : two: Unchanged skipping 2024/11/05 03:06:14 INFO : one.bak: Deleted 2024/11/05 03:06:15 INFO : one: Moved (server-side) to: one.bak 2024/11/05 03:06:15 DEBUG : gofile root 'rclone-test-sozagov0moqo/dst': Waiting for transfers to finish 2024/11/05 03:06:15 DEBUG : one: Using upload server store5 (eu) 2024/11/05 03:06:15 DEBUG : one: md5 = 0f93e81041f0cab37c37a05ae998b219 OK 2024/11/05 03:06:15 INFO : one: Copied (new) 2024/11/05 03:06:15 DEBUG : Waiting for deletions to finish 2024/11/05 03:06:15 INFO : three.txt.bak: Deleted 2024/11/05 03:06:16 INFO : three.txt: Moved (server-side) to: three.txt.bak 2024/11/05 03:06:16 INFO : three.txt: Moved into backup dir --- PASS: TestSyncBackupDirSuffixOnly (10.79s) FAIL 2024/11/05 03:06:19 DEBUG : gofile root 'rclone-test-sozagov0moqo': Purge remote "./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -test.run '^(TestServerSideCopyOverSelf|TestSyncBackupDirSuffixOnly|TestSyncBackupDirWithSuffix|TestSyncBackupDirWithSuffixKeepExtension)$'" - Finished ERROR in 2m8.713505128s (try 2/5): exit status 1: Failed [TestServerSideCopyOverSelf TestSyncBackupDirWithSuffixKeepExtension]