"./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -fast-list -test.run '^(TestServerSideCopyOverSelf|TestSyncBackupDirWithSuffix)$'" - Starting (try 4/5)
2024/11/03 04:51:18 DEBUG : Creating backend with remote "TestGoFile:rclone-test-cayenih0toxi"
2024/11/03 04:51:18 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2024/11/03 04:51:18 DEBUG : Creating backend with remote "/tmp/rclone859752518"
=== RUN   TestServerSideCopyOverSelf
    run.go:180: Remote "gofile root 'rclone-test-cayenih0toxi'", Local "Local file system at /tmp/rclone859752518", Modify Window "1s"
2024/11/03 04:51:20 DEBUG : Upload server store10 (eu) responded in 125.952925ms
2024/11/03 04:51:20 DEBUG : Upload server store6 (eu) responded in 127.003499ms
2024/11/03 04:51:20 DEBUG : sub dir/hello world: Using upload server store6 (eu)
2024/11/03 04:51:20 DEBUG : Creating backend with remote "TestGoFile:rclone-test-tudehux9zodi"
    sync_test.go:626: Server side copy (if possible) gofile root 'rclone-test-cayenih0toxi' -> gofile root 'rclone-test-tudehux9zodi'
2024/11/03 04:51:21 DEBUG : Added delayed dir = "sub dir", newDst=<nil>
2024/11/03 04:51:21 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination
2024/11/03 04:51:21 DEBUG : gofile root 'rclone-test-tudehux9zodi': Waiting for checks to finish
2024/11/03 04:51:21 DEBUG : gofile root 'rclone-test-tudehux9zodi': Waiting for transfers to finish
2024/11/03 04:51:22 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:51:27 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit")
2024/11/03 04:51:27 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/11/03 04:51:27 DEBUG : pacer: Reducing sleep to 10ms
2024/11/03 04:51:27 DEBUG : sub dir/hello world: md5 = 5eb63bbbe01eeed093cb22bb8f5acdc3 OK
2024/11/03 04:51:27 INFO  : sub dir/hello world: Copied (server-side copy)
2024/11/03 04:51:27 INFO  : sub dir: Set directory modification time (using DirSetModTime)
    run.go:180: Remote "gofile root 'rclone-test-cayenih0toxi'", Local "Local file system at /tmp/rclone859752518", Modify Window "1s"
2024/11/03 04:51:27 DEBUG : sub dir/hello world: Using upload server store10 (eu)
2024/11/03 04:51:27 DEBUG : sub dir/hello world: Removing old object on successful upload
2024/11/03 04:51:28 DEBUG : Added delayed dir = "sub dir", newDst=sub dir
2024/11/03 04:51:28 DEBUG : sub dir/hello world: Sizes differ (src 17 vs dst 11)
2024/11/03 04:51:28 DEBUG : gofile root 'rclone-test-tudehux9zodi': Waiting for checks to finish
2024/11/03 04:51:28 DEBUG : gofile root 'rclone-test-tudehux9zodi': Waiting for transfers to finish
2024/11/03 04:51:28 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:51:33 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit")
2024/11/03 04:51:33 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/11/03 04:51:33 DEBUG : pacer: Reducing sleep to 10ms
2024/11/03 04:51:33 DEBUG : sub dir/hello world: md5 = 00321a507bdb420dbbd4f53c733b1505 OK
2024/11/03 04:51:33 INFO  : sub dir/hello world: Copied (server-side copy)
2024/11/03 04:51:33 INFO  : sub dir: Set directory modification time (using SetModTime)
    run.go:180: Remote "gofile root 'rclone-test-cayenih0toxi'", Local "Local file system at /tmp/rclone859752518", Modify Window "1s"
    fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3
    fstest.go:295: Flushing the directory cache
    fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3
    fstest.go:295: Flushing the directory cache
    fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3
    fstest.go:295: Flushing the directory cache
    fstest.go:299: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:641
        	Error:      	Should be true
        	Test:       	TestServerSideCopyOverSelf
        	Messages:   	listing wrong, want
        	            	  sub dir/hello world (17) got
        	            	  sub dir/hello world (11), sub dir/hello world (17)
    fstest.go:185: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:641
        	Error:      	Should be true
        	Test:       	TestServerSideCopyOverSelf
        	Messages:   	Unexpected file "sub dir/hello world"
2024/11/03 04:51:41 DEBUG : gofile root 'rclone-test-tudehux9zodi': Purge remote
--- FAIL: TestServerSideCopyOverSelf (24.37s)
=== RUN   TestSyncBackupDirWithSuffix
    run.go:180: Remote "gofile root 'rclone-test-cayenih0toxi'", Local "Local file system at /tmp/rclone859752518", Modify Window "1s"
2024/11/03 04:51:43 DEBUG : dst/one: Using upload server store6 (eu)
2024/11/03 04:51:43 DEBUG : dst/two: Using upload server store6 (eu)
2024/11/03 04:51:44 DEBUG : dst/three.txt: Using upload server store6 (eu)
2024/11/03 04:51:44 DEBUG : Creating backend with remote "TestGoFile:rclone-test-cayenih0toxi/dst"
2024/11/03 04:51:44 DEBUG : Creating backend with remote "TestGoFile:rclone-test-cayenih0toxi/backup"
2024/11/03 04:51:45 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s)
2024/11/03 04:51:45 DEBUG : two: Unchanged skipping
2024/11/03 04:51:45 DEBUG : one: Sizes differ (src 4 vs dst 3)
2024/11/03 04:51:45 DEBUG : gofile root 'rclone-test-cayenih0toxi/dst': Waiting for checks to finish
2024/11/03 04:51:45 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:51:50 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit")
2024/11/03 04:51:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/11/03 04:51:51 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:51:56 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit")
2024/11/03 04:51:56 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/11/03 04:51:56 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:01 DEBUG : pacer: low level retry 3/10 (error Error "error-rateLimit")
2024/11/03 04:52:01 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2024/11/03 04:52:01 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:06 DEBUG : pacer: low level retry 4/10 (error Error "error-rateLimit")
2024/11/03 04:52:06 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2024/11/03 04:52:06 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:11 DEBUG : pacer: low level retry 5/10 (error Error "error-rateLimit")
2024/11/03 04:52:11 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2024/11/03 04:52:11 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:16 DEBUG : pacer: low level retry 6/10 (error Error "error-rateLimit")
2024/11/03 04:52:16 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2024/11/03 04:52:16 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:21 DEBUG : pacer: low level retry 7/10 (error Error "error-rateLimit")
2024/11/03 04:52:21 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2024/11/03 04:52:21 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:26 DEBUG : pacer: low level retry 8/10 (error Error "error-rateLimit")
2024/11/03 04:52:26 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2024/11/03 04:52:26 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:31 DEBUG : pacer: low level retry 9/10 (error Error "error-rateLimit")
2024/11/03 04:52:31 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2024/11/03 04:52:31 DEBUG : pacer: Reducing sleep to 2.56s
2024/11/03 04:52:31 INFO  : one: Moved (server-side) to: one.bak
2024/11/03 04:52:31 DEBUG : gofile root 'rclone-test-cayenih0toxi/dst': Waiting for transfers to finish
2024/11/03 04:52:31 DEBUG : Upload server store5 (eu) responded in 123.228158ms
2024/11/03 04:52:31 DEBUG : Upload server store4 (eu) responded in 124.759893ms
2024/11/03 04:52:31 DEBUG : one: Using upload server store5 (eu)
2024/11/03 04:52:31 DEBUG : one: md5 = c7957179c41f69d44f217a108c7915d8 OK
2024/11/03 04:52:31 INFO  : one: Copied (new)
2024/11/03 04:52:31 DEBUG : Waiting for deletions to finish
2024/11/03 04:52:36 DEBUG : pacer: Reducing sleep to 1.28s
2024/11/03 04:52:39 DEBUG : pacer: Reducing sleep to 640ms
2024/11/03 04:52:40 DEBUG : pacer: Reducing sleep to 320ms
2024/11/03 04:52:40 INFO  : three.txt: Moved (server-side) to: three.txt.bak
2024/11/03 04:52:40 INFO  : three.txt: Moved into backup dir
2024/11/03 04:52:41 DEBUG : Upload server store6 (eu) responded in 37.675762ms
2024/11/03 04:52:41 DEBUG : Upload server store5 (eu) responded in 124.136145ms
2024/11/03 04:52:41 DEBUG : dst/three.txt: Using upload server store6 (eu)
2024/11/03 04:52:42 DEBUG : one: Sizes differ (src 5 vs dst 4)
2024/11/03 04:52:42 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s)
2024/11/03 04:52:42 DEBUG : two: Unchanged skipping
2024/11/03 04:52:42 DEBUG : gofile root 'rclone-test-cayenih0toxi/dst': Waiting for checks to finish
2024/11/03 04:52:42 DEBUG : pacer: Reducing sleep to 160ms
2024/11/03 04:52:43 DEBUG : pacer: Reducing sleep to 80ms
2024/11/03 04:52:43 INFO  : one.bak: Deleted
2024/11/03 04:52:43 DEBUG : pacer: Reducing sleep to 40ms
2024/11/03 04:52:43 DEBUG : pacer: Reducing sleep to 20ms
2024/11/03 04:52:43 INFO  : one: Moved (server-side) to: one.bak
2024/11/03 04:52:43 DEBUG : gofile root 'rclone-test-cayenih0toxi/dst': Waiting for transfers to finish
2024/11/03 04:52:43 DEBUG : one: Using upload server store4 (eu)
2024/11/03 04:52:43 DEBUG : one: md5 = 0f93e81041f0cab37c37a05ae998b219 OK
2024/11/03 04:52:43 INFO  : one: Copied (new)
2024/11/03 04:52:43 DEBUG : Waiting for deletions to finish
2024/11/03 04:52:43 DEBUG : pacer: Reducing sleep to 10ms
2024/11/03 04:52:43 INFO  : three.txt.bak: Deleted
2024/11/03 04:52:43 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:48 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit")
2024/11/03 04:52:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/11/03 04:52:49 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:54 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit")
2024/11/03 04:52:54 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/11/03 04:52:54 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:52:59 DEBUG : pacer: low level retry 3/10 (error Error "error-rateLimit")
2024/11/03 04:52:59 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2024/11/03 04:52:59 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:04 DEBUG : pacer: low level retry 4/10 (error Error "error-rateLimit")
2024/11/03 04:53:04 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2024/11/03 04:53:04 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:09 DEBUG : pacer: low level retry 5/10 (error Error "error-rateLimit")
2024/11/03 04:53:09 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2024/11/03 04:53:09 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:14 DEBUG : pacer: low level retry 6/10 (error Error "error-rateLimit")
2024/11/03 04:53:14 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2024/11/03 04:53:14 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:19 DEBUG : pacer: low level retry 7/10 (error Error "error-rateLimit")
2024/11/03 04:53:19 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2024/11/03 04:53:19 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:24 DEBUG : pacer: low level retry 8/10 (error Error "error-rateLimit")
2024/11/03 04:53:24 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2024/11/03 04:53:24 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:29 DEBUG : pacer: low level retry 9/10 (error Error "error-rateLimit")
2024/11/03 04:53:29 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2024/11/03 04:53:29 DEBUG : Rate limited, sleep for 5s
2024/11/03 04:53:34 DEBUG : pacer: low level retry 10/10 (error Error "error-rateLimit")
2024/11/03 04:53:34 DEBUG : pacer: Rate limited, increasing sleep to 10.24s
2024/11/03 04:53:34 ERROR : three.txt: Couldn't move: failed to move item: Error "error-rateLimit"
2024/11/03 04:53:34 ERROR : three.txt: Couldn't move into backup dir: failed to move item: Error "error-rateLimit"
2024/11/03 04:53:34 ERROR : gofile root 'rclone-test-cayenih0toxi/dst': not deleting directories as there were IO errors
    sync_test.go:2410: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2410
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2431
        	Error:      	Received unexpected error:
        	            	failed to delete 1 files
        	Test:       	TestSyncBackupDirWithSuffix
--- FAIL: TestSyncBackupDirWithSuffix (113.20s)
FAIL
2024/11/03 04:53:36 DEBUG : gofile root 'rclone-test-cayenih0toxi': Purge remote
"./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -fast-list -test.run '^(TestServerSideCopyOverSelf|TestSyncBackupDirWithSuffix)$'" - Finished ERROR in 2m18.280059092s (try 4/5): exit status 1: Failed [TestServerSideCopyOverSelf TestSyncBackupDirWithSuffix]