"./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -test.run '^(TestServerSideCopyOverSelf|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveWithFilter|TestSyncBackupDir)$'" - Starting (try 2/5) 2024/10/29 03:56:00 DEBUG : Creating backend with remote "TestGoFile:rclone-test-cipojip5xupi" 2024/10/29 03:56:00 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2024/10/29 03:56:00 DEBUG : Creating backend with remote "/tmp/rclone1067233939" === RUN TestServerSideCopyOverSelf run.go:180: Remote "gofile root 'rclone-test-cipojip5xupi'", Local "Local file system at /tmp/rclone1067233939", Modify Window "1s" 2024/10/29 03:56:02 DEBUG : Upload server store4 (eu) responded in 123.329707ms 2024/10/29 03:56:02 DEBUG : Upload server store2 (eu) responded in 127.494091ms 2024/10/29 03:56:02 DEBUG : sub dir/hello world: Using upload server store4 (eu) 2024/10/29 03:56:02 DEBUG : Creating backend with remote "TestGoFile:rclone-test-pusawix0kuqa" sync_test.go:626: Server side copy (if possible) gofile root 'rclone-test-cipojip5xupi' -> gofile root 'rclone-test-pusawix0kuqa' 2024/10/29 03:56:03 DEBUG : Added delayed dir = "sub dir", newDst= 2024/10/29 03:56:03 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2024/10/29 03:56:03 DEBUG : gofile root 'rclone-test-pusawix0kuqa': Waiting for checks to finish 2024/10/29 03:56:03 DEBUG : gofile root 'rclone-test-pusawix0kuqa': Waiting for transfers to finish 2024/10/29 03:56:04 DEBUG : sub dir/hello world: md5 = 5eb63bbbe01eeed093cb22bb8f5acdc3 OK 2024/10/29 03:56:04 INFO : sub dir/hello world: Copied (server-side copy) 2024/10/29 03:56:04 INFO : sub dir: Set directory modification time (using DirSetModTime) run.go:180: Remote "gofile root 'rclone-test-cipojip5xupi'", Local "Local file system at /tmp/rclone1067233939", Modify Window "1s" 2024/10/29 03:56:05 DEBUG : sub dir/hello world: Using upload server store4 (eu) 2024/10/29 03:56:05 DEBUG : sub dir/hello world: Removing old object on successful upload 2024/10/29 03:56:05 DEBUG : Added delayed dir = "sub dir", newDst=sub dir 2024/10/29 03:56:05 DEBUG : sub dir/hello world: Sizes differ (src 17 vs dst 11) 2024/10/29 03:56:05 DEBUG : gofile root 'rclone-test-pusawix0kuqa': Waiting for checks to finish 2024/10/29 03:56:05 DEBUG : gofile root 'rclone-test-pusawix0kuqa': Waiting for transfers to finish 2024/10/29 03:56:06 DEBUG : sub dir/hello world: md5 = 00321a507bdb420dbbd4f53c733b1505 OK 2024/10/29 03:56:06 INFO : sub dir/hello world: Copied (server-side copy) 2024/10/29 03:56:06 INFO : sub dir: Set directory modification time (using SetModTime) run.go:180: Remote "gofile root 'rclone-test-cipojip5xupi'", Local "Local file system at /tmp/rclone1067233939", Modify Window "1s" fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:295: Flushing the directory cache fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:641 Error: Should be true Test: TestServerSideCopyOverSelf Messages: listing wrong, want sub dir/hello world (17) got sub dir/hello world (11), sub dir/hello world (17) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:641 Error: Should be true Test: TestServerSideCopyOverSelf Messages: Unexpected file "sub dir/hello world" 2024/10/29 03:56:14 DEBUG : gofile root 'rclone-test-pusawix0kuqa': Purge remote --- FAIL: TestServerSideCopyOverSelf (15.76s) === RUN TestServerSideMoveWithFilter run.go:180: Remote "gofile root 'rclone-test-cipojip5xupi'", Local "Local file system at /tmp/rclone1067233939", Modify Window "1s" 2024/10/29 03:56:16 DEBUG : Creating backend with remote "TestGoFile:rclone-test-bikokov2koja" 2024/10/29 03:56:17 DEBUG : potato2: Using upload server store4 (eu) 2024/10/29 03:56:17 DEBUG : empty space: Using upload server store4 (eu) 2024/10/29 03:56:17 DEBUG : potato3: Using upload server store2 (eu) sync_test.go:1705: Server side move (if possible) gofile root 'rclone-test-cipojip5xupi' -> gofile root 'rclone-test-bikokov2koja' 2024/10/29 03:56:18 DEBUG : Upload server store5 (eu) responded in 135.802318ms 2024/10/29 03:56:18 DEBUG : Upload server store7 (eu) responded in 136.085527ms 2024/10/29 03:56:18 DEBUG : Upload server store2 (eu) responded in 136.304637ms 2024/10/29 03:56:18 DEBUG : empty space: Using upload server store7 (eu) 2024/10/29 03:56:18 DEBUG : potato3: Using upload server store7 (eu) 2024/10/29 03:56:19 DEBUG : empty space: Excluded (Size Filter) 2024/10/29 03:56:19 DEBUG : empty space: Excluded 2024/10/29 03:56:19 DEBUG : empty space: Excluded (Size Filter) 2024/10/29 03:56:19 DEBUG : empty space: Excluded 2024/10/29 03:56:19 DEBUG : potato2: Need to transfer - File not found at Destination 2024/10/29 03:56:19 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2024/10/29 03:56:19 DEBUG : gofile root 'rclone-test-bikokov2koja': Waiting for checks to finish 2024/10/29 03:56:19 DEBUG : gofile root 'rclone-test-bikokov2koja': Waiting for transfers to finish 2024/10/29 03:56:19 INFO : potato3: Deleted 2024/10/29 03:56:19 INFO : potato2: Moved (server-side) 2024/10/29 03:56:19 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:24 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/10/29 03:56:24 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/10/29 03:56:24 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:29 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2024/10/29 03:56:29 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2024/10/29 03:56:29 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:34 DEBUG : pacer: low level retry 3/10 (error Error "error-rateLimit") 2024/10/29 03:56:34 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2024/10/29 03:56:34 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:39 DEBUG : pacer: low level retry 4/10 (error Error "error-rateLimit") 2024/10/29 03:56:39 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2024/10/29 03:56:39 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:44 DEBUG : pacer: low level retry 5/10 (error Error "error-rateLimit") 2024/10/29 03:56:44 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2024/10/29 03:56:44 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:49 DEBUG : pacer: low level retry 6/10 (error Error "error-rateLimit") 2024/10/29 03:56:49 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2024/10/29 03:56:49 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:54 DEBUG : pacer: low level retry 7/10 (error Error "error-rateLimit") 2024/10/29 03:56:54 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2024/10/29 03:56:54 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:56:59 DEBUG : pacer: low level retry 8/10 (error Error "error-rateLimit") 2024/10/29 03:56:59 DEBUG : pacer: Rate limited, increasing sleep to 2.56s 2024/10/29 03:56:59 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:57:04 DEBUG : pacer: low level retry 9/10 (error Error "error-rateLimit") 2024/10/29 03:57:04 DEBUG : pacer: Rate limited, increasing sleep to 5.12s 2024/10/29 03:57:04 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:57:09 DEBUG : pacer: low level retry 10/10 (error Error "error-rateLimit") 2024/10/29 03:57:09 DEBUG : pacer: Rate limited, increasing sleep to 10.24s 2024/10/29 03:57:09 ERROR : potato3: Couldn't move: failed to move item: Error "error-rateLimit" sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1894 Error: Received unexpected error: failed to move item: Error "error-rateLimit" Test: TestServerSideMoveWithFilter 2024/10/29 03:57:09 DEBUG : gofile root 'rclone-test-bikokov2koja': Purge remote 2024/10/29 03:57:10 DEBUG : pacer: Reducing sleep to 5.12s --- FAIL: TestServerSideMoveWithFilter (54.10s) === RUN TestServerSideMoveDeleteEmptySourceDirs run.go:180: Remote "gofile root 'rclone-test-cipojip5xupi'", Local "Local file system at /tmp/rclone1067233939", Modify Window "1s" 2024/10/29 03:57:10 DEBUG : Creating backend with remote "TestGoFile:rclone-test-xiduzev0zigo" 2024/10/29 03:57:11 DEBUG : Upload server store2 (eu) responded in 37.809179ms 2024/10/29 03:57:11 DEBUG : Upload server store4 (eu) responded in 37.657125ms 2024/10/29 03:57:11 DEBUG : potato2: Using upload server store4 (eu) 2024/10/29 03:57:11 DEBUG : empty space: Using upload server store2 (eu) 2024/10/29 03:57:11 DEBUG : potato3: Using upload server store2 (eu) 2024/10/29 03:57:11 DEBUG : tomatoDir: Making directory sync_test.go:1705: Server side move (if possible) gofile root 'rclone-test-cipojip5xupi' -> gofile root 'rclone-test-xiduzev0zigo' 2024/10/29 03:57:13 DEBUG : Upload server store4 (eu) responded in 127.407465ms 2024/10/29 03:57:13 DEBUG : Upload server store10 (eu) responded in 132.758786ms 2024/10/29 03:57:13 DEBUG : empty space: Using upload server store4 (eu) 2024/10/29 03:57:13 DEBUG : potato3: Using upload server store10 (eu) 2024/10/29 03:57:13 DEBUG : gofile root 'rclone-test-xiduzev0zigo': Using server-side directory move 2024/10/29 03:57:13 INFO : gofile root 'rclone-test-xiduzev0zigo': Server side directory move failed - fallback to file moves: can't copy directory - destination already exists 2024/10/29 03:57:13 DEBUG : potato2: Need to transfer - File not found at Destination 2024/10/29 03:57:13 DEBUG : Added delayed dir = "tomatoDir", newDst= 2024/10/29 03:57:13 DEBUG : empty space: Size and modification time the same (differ by 0s, within tolerance 1s) 2024/10/29 03:57:13 DEBUG : empty space: Unchanged skipping 2024/10/29 03:57:13 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2024/10/29 03:57:13 INFO : empty space: Deleted 2024/10/29 03:57:13 INFO : potato3: Deleted 2024/10/29 03:57:13 INFO : potato2: Moved (server-side) 2024/10/29 03:57:14 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:57:14 DEBUG : gofile root 'rclone-test-xiduzev0zigo': Waiting for checks to finish 2024/10/29 03:57:14 DEBUG : gofile root 'rclone-test-xiduzev0zigo': Waiting for transfers to finish 2024/10/29 03:57:19 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/10/29 03:57:19 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/10/29 03:57:19 DEBUG : pacer: Reducing sleep to 10ms 2024/10/29 03:57:19 INFO : potato3: Moved (server-side) 2024/10/29 03:57:19 INFO : tomatoDir: Removing directory 2024/10/29 03:57:19 DEBUG : gofile root 'rclone-test-cipojip5xupi': deleted 1 directories 2024/10/29 03:57:19 DEBUG : Creating backend with remote "TestGoFile:rclone-test-tayarob0dera" 2024/10/29 03:57:20 DEBUG : tomatoDir: Making directory 2024/10/29 03:57:20 DEBUG : gofile root 'rclone-test-tayarob0dera': Using server-side directory move 2024/10/29 03:57:20 INFO : gofile root 'rclone-test-tayarob0dera': Server side directory move succeeded 2024/10/29 03:57:21 DEBUG : gofile root 'rclone-test-tayarob0dera': Purge remote 2024/10/29 03:57:22 DEBUG : gofile root 'rclone-test-xiduzev0zigo': Purge remote 2024/10/29 03:57:22 NOTICE: purge failed: directory not found --- PASS: TestServerSideMoveDeleteEmptySourceDirs (11.78s) === RUN TestSyncBackupDir run.go:180: Remote "gofile root 'rclone-test-cipojip5xupi'", Local "Local file system at /tmp/rclone1067233939", Modify Window "1s" 2024/10/29 03:57:22 DEBUG : dst/one: Using upload server store2 (eu) 2024/10/29 03:57:23 DEBUG : dst/two: Using upload server store4 (eu) 2024/10/29 03:57:23 DEBUG : dst/three.txt: Using upload server store2 (eu) 2024/10/29 03:57:23 DEBUG : Creating backend with remote "TestGoFile:rclone-test-cipojip5xupi/dst" 2024/10/29 03:57:24 DEBUG : Creating backend with remote "TestGoFile:rclone-test-cipojip5xupi/backup" 2024/10/29 03:57:25 DEBUG : one: Sizes differ (src 4 vs dst 3) 2024/10/29 03:57:25 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/10/29 03:57:25 DEBUG : two: Unchanged skipping 2024/10/29 03:57:25 DEBUG : gofile root 'rclone-test-cipojip5xupi/dst': Waiting for checks to finish 2024/10/29 03:57:25 INFO : one: Moved (server-side) 2024/10/29 03:57:25 DEBUG : gofile root 'rclone-test-cipojip5xupi/dst': Waiting for transfers to finish 2024/10/29 03:57:25 DEBUG : Upload server store5 (eu) responded in 125.915836ms 2024/10/29 03:57:25 DEBUG : Upload server store6 (eu) responded in 125.855461ms 2024/10/29 03:57:25 DEBUG : one: Using upload server store5 (eu) 2024/10/29 03:57:26 DEBUG : one: md5 = c7957179c41f69d44f217a108c7915d8 OK 2024/10/29 03:57:26 INFO : one: Copied (new) 2024/10/29 03:57:26 DEBUG : Waiting for deletions to finish 2024/10/29 03:57:26 INFO : three.txt: Moved (server-side) 2024/10/29 03:57:26 INFO : three.txt: Moved into backup dir 2024/10/29 03:57:26 DEBUG : dst/three.txt: Using upload server store4 (eu) 2024/10/29 03:57:27 DEBUG : one: Sizes differ (src 5 vs dst 4) 2024/10/29 03:57:27 DEBUG : two: Size and modification time the same (differ by -499.999999ms, within tolerance 1s) 2024/10/29 03:57:27 DEBUG : two: Unchanged skipping 2024/10/29 03:57:27 DEBUG : gofile root 'rclone-test-cipojip5xupi/dst': Waiting for checks to finish 2024/10/29 03:57:27 INFO : one: Deleted 2024/10/29 03:57:27 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:57:32 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2024/10/29 03:57:32 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2024/10/29 03:57:32 DEBUG : Rate limited, sleep for 5s 2024/10/29 03:57:37 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2024/10/29 03:57:37 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2024/10/29 03:57:38 DEBUG : pacer: Reducing sleep to 20ms 2024/10/29 03:57:38 INFO : one: Moved (server-side) 2024/10/29 03:57:38 DEBUG : gofile root 'rclone-test-cipojip5xupi/dst': Waiting for transfers to finish 2024/10/29 03:57:38 DEBUG : one: Using upload server store6 (eu) 2024/10/29 03:57:38 DEBUG : one: md5 = 0f93e81041f0cab37c37a05ae998b219 OK 2024/10/29 03:57:38 INFO : one: Copied (new) 2024/10/29 03:57:38 DEBUG : Waiting for deletions to finish 2024/10/29 03:57:38 DEBUG : pacer: Reducing sleep to 10ms 2024/10/29 03:57:38 INFO : three.txt: Deleted 2024/10/29 03:57:38 INFO : three.txt: Moved (server-side) 2024/10/29 03:57:38 INFO : three.txt: Moved into backup dir --- PASS: TestSyncBackupDir (17.59s) FAIL 2024/10/29 03:57:40 DEBUG : gofile root 'rclone-test-cipojip5xupi': Purge remote "./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -test.run '^(TestServerSideCopyOverSelf|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveWithFilter|TestSyncBackupDir)$'" - Finished ERROR in 1m39.919123964s (try 2/5): exit status 1: Failed [TestServerSideCopyOverSelf TestServerSideMoveWithFilter]