"./operations.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -test.run '^(TestDirMove|TestMaxDeleteSize|TestRmdirsNoLeaveRoot)$'" - Starting (try 2/5) 2025/01/24 01:56:42 DEBUG : Creating backend with remote "TestGoFile:rclone-test-sazulaj2zehe" 2025/01/24 01:56:42 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/01/24 01:56:43 DEBUG : Creating backend with remote "/tmp/rclone2196673668" === RUN TestMaxDeleteSize run.go:180: Remote "gofile root 'rclone-test-sazulaj2zehe'", Local "Local file system at /tmp/rclone2196673668", Modify Window "1s" 2025/01/24 01:56:44 DEBUG : Upload server store-eu-par-1 (eu) responded in 131.497445ms 2025/01/24 01:56:44 DEBUG : Upload server store7 (eu) responded in 134.898297ms 2025/01/24 01:56:44 DEBUG : small: Using upload server store7 (eu) 2025/01/24 01:56:45 DEBUG : medium: Using upload server store-eu-par-1 (eu) 2025/01/24 01:56:46 DEBUG : large: Using upload server store-eu-par-1 (eu) 2025/01/24 01:56:46 DEBUG : Waiting for deletions to finish 2025/01/24 01:56:47 ERROR : small: Got fatal error on delete: --max-delete-size threshold reached 2025/01/24 01:56:47 INFO : medium: Deleted 2025/01/24 01:56:47 INFO : large: Deleted --- PASS: TestMaxDeleteSize (6.18s) === RUN TestRmdirsNoLeaveRoot run.go:180: Remote "gofile root 'rclone-test-sazulaj2zehe'", Local "Local file system at /tmp/rclone2196673668", Modify Window "1s" 2025/01/24 01:56:50 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:56:55 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:56:55 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:56:55 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:56:56 DEBUG : A1/B1/C1/one: Using upload server store7 (eu) 2025/01/24 01:56:56 INFO : A2: Making directory 2025/01/24 01:56:57 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:57:02 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:57:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:57:02 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:57:07 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2025/01/24 01:57:07 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/01/24 01:57:07 DEBUG : pacer: Reducing sleep to 20ms 2025/01/24 01:57:07 INFO : A1/B2: Making directory 2025/01/24 01:57:07 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:57:08 INFO : A1/B2/C2: Making directory 2025/01/24 01:57:08 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:57:13 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:57:13 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:57:13 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:57:18 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2025/01/24 01:57:18 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/01/24 01:57:18 DEBUG : pacer: Reducing sleep to 20ms 2025/01/24 01:57:18 INFO : A1/B1/C3: Making directory 2025/01/24 01:57:19 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:57:19 INFO : A3: Making directory 2025/01/24 01:57:19 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:57:24 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:57:24 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:57:25 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:57:25 INFO : A3/B3: Making directory 2025/01/24 01:57:25 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:57:30 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:57:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:57:30 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:57:30 INFO : A3/B3/C4: Making directory 2025/01/24 01:57:31 DEBUG : A1/two: Using upload server store-eu-par-1 (eu) 2025/01/24 01:57:33 DEBUG : removing 1 level 3 directories 2025/01/24 01:57:33 INFO : A3/B3/C4: Removing directory 2025/01/24 01:57:36 DEBUG : removing 2 level 3 directories 2025/01/24 01:57:36 INFO : A1/B2/C2: Removing directory 2025/01/24 01:57:36 INFO : A1/B1/C3: Removing directory 2025/01/24 01:57:37 DEBUG : removing 2 level 2 directories 2025/01/24 01:57:37 INFO : A3/B3: Removing directory 2025/01/24 01:57:37 INFO : A1/B2: Removing directory 2025/01/24 01:57:37 DEBUG : removing 2 level 1 directories 2025/01/24 01:57:37 INFO : A3: Removing directory 2025/01/24 01:57:37 INFO : A2: Removing directory 2025/01/24 01:57:41 DEBUG : removing 1 level 3 directories 2025/01/24 01:57:41 INFO : A1/B1/C1: Removing directory 2025/01/24 01:57:41 DEBUG : removing 1 level 2 directories 2025/01/24 01:57:41 INFO : A1/B1: Removing directory 2025/01/24 01:57:42 DEBUG : removing 1 level 1 directories 2025/01/24 01:57:42 INFO : A1: Removing directory 2025/01/24 01:57:43 DEBUG : removing 1 level 0 directories 2025/01/24 01:57:43 INFO : gofile root 'rclone-test-sazulaj2zehe': Removing directory --- PASS: TestRmdirsNoLeaveRoot (54.75s) === RUN TestDirMove run.go:180: Remote "gofile root 'rclone-test-sazulaj2zehe'", Local "Local file system at /tmp/rclone2196673668", Modify Window "1s" 2025/01/24 01:57:45 DEBUG : Upload server store-eu-par-1 (eu) responded in 41.871454ms 2025/01/24 01:57:46 DEBUG : Upload server store-eu-par-5 (eu) responded in 120.771247ms 2025/01/24 01:57:46 DEBUG : A1/one: Using upload server store-eu-par-5 (eu) 2025/01/24 01:57:46 DEBUG : A1/two: Using upload server store-eu-par-1 (eu) 2025/01/24 01:57:47 DEBUG : A1/B1/three: Using upload server store-eu-par-1 (eu) 2025/01/24 01:57:48 DEBUG : A1/B1/C1/four: Using upload server store-eu-par-5 (eu) 2025/01/24 01:57:48 DEBUG : A1/B1/C2/five: Using upload server store-eu-par-5 (eu) 2025/01/24 01:57:48 INFO : A1/B2: Making directory 2025/01/24 01:57:49 INFO : A1/B1/C3: Making directory 2025/01/24 01:57:56 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:01 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:01 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:58:01 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:58:02 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:07 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:07 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:58:07 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:58:07 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:07 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:07 INFO : A2/B1/C2/five: Moved (server-side) to: A3/B1/C2/five 2025/01/24 01:58:07 INFO : A2/B1/C1/four: Moved (server-side) to: A3/B1/C1/four 2025/01/24 01:58:07 INFO : A2/two: Moved (server-side) to: A3/two 2025/01/24 01:58:12 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:12 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:58:12 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:12 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:12 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/01/24 01:58:12 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:17 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2025/01/24 01:58:17 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/01/24 01:58:17 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2025/01/24 01:58:17 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/01/24 01:58:17 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:18 DEBUG : pacer: Reducing sleep to 80ms 2025/01/24 01:58:18 INFO : A2/one: Moved (server-side) to: A3/one 2025/01/24 01:58:22 DEBUG : pacer: low level retry 3/10 (error Error "error-rateLimit") 2025/01/24 01:58:22 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/01/24 01:58:23 DEBUG : pacer: Reducing sleep to 80ms 2025/01/24 01:58:23 INFO : A2/B1/three: Moved (server-side) to: A3/B1/three 2025/01/24 01:58:23 DEBUG : pacer: Reducing sleep to 40ms 2025/01/24 01:58:23 DEBUG : pacer: Reducing sleep to 20ms 2025/01/24 01:58:24 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:58:27 INFO : gofile root 'rclone-test-sazulaj2zehe': Can't DirMove - falling back to file moves: can't move directory - incompatible remotes 2025/01/24 01:58:32 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:37 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:37 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:58:37 DEBUG : pacer: Reducing sleep to 10ms 2025/01/24 01:58:37 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:37 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:38 INFO : A3/B1/C1/four: Moved (server-side) to: A4/B1/C1/four 2025/01/24 01:58:38 INFO : A3/B1/three: Moved (server-side) to: A4/B1/three 2025/01/24 01:58:38 INFO : A3/B1/C2/five: Moved (server-side) to: A4/B1/C2/five 2025/01/24 01:58:42 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:42 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/01/24 01:58:42 DEBUG : pacer: low level retry 1/10 (error Error "error-rateLimit") 2025/01/24 01:58:42 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/01/24 01:58:42 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:43 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:47 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2025/01/24 01:58:47 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/01/24 01:58:48 DEBUG : pacer: low level retry 2/10 (error Error "error-rateLimit") 2025/01/24 01:58:48 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/01/24 01:58:48 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:48 DEBUG : pacer: Reducing sleep to 80ms 2025/01/24 01:58:48 INFO : A3/two: Moved (server-side) to: A4/two 2025/01/24 01:58:53 DEBUG : pacer: low level retry 3/10 (error Error "error-rateLimit") 2025/01/24 01:58:53 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/01/24 01:58:53 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:58:58 DEBUG : pacer: low level retry 4/10 (error Error "error-rateLimit") 2025/01/24 01:58:58 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/01/24 01:58:58 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:59:03 DEBUG : pacer: low level retry 5/10 (error Error "error-rateLimit") 2025/01/24 01:59:03 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/01/24 01:59:03 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:59:08 DEBUG : pacer: low level retry 6/10 (error Error "error-rateLimit") 2025/01/24 01:59:08 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/01/24 01:59:08 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:59:13 DEBUG : pacer: low level retry 7/10 (error Error "error-rateLimit") 2025/01/24 01:59:13 DEBUG : pacer: Rate limited, increasing sleep to 2.56s 2025/01/24 01:59:13 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:59:18 DEBUG : pacer: low level retry 8/10 (error Error "error-rateLimit") 2025/01/24 01:59:18 DEBUG : pacer: Rate limited, increasing sleep to 5.12s 2025/01/24 01:59:18 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:59:23 DEBUG : pacer: low level retry 9/10 (error Error "error-rateLimit") 2025/01/24 01:59:23 DEBUG : pacer: Rate limited, increasing sleep to 10.24s 2025/01/24 01:59:23 DEBUG : Rate limited, sleep for 5s 2025/01/24 01:59:28 DEBUG : pacer: low level retry 10/10 (error Error "error-rateLimit") 2025/01/24 01:59:28 DEBUG : pacer: Rate limited, increasing sleep to 20s 2025/01/24 01:59:28 ERROR : A3/one: Couldn't move: failed to move item: Error "error-rateLimit" operations_test.go:1433: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1433 Error: Received unexpected error: RenameDir renames: failed to move item: Error "error-rateLimit" Test: TestDirMove 2025/01/24 01:59:35 DEBUG : pacer: Reducing sleep to 10s fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache 2025/01/24 01:59:53 DEBUG : pacer: Reducing sleep to 5s 2025/01/24 02:00:05 DEBUG : pacer: Reducing sleep to 2.5s fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:295: Flushing the directory cache 2025/01/24 02:00:08 DEBUG : pacer: Reducing sleep to 1.25s 2025/01/24 02:00:13 DEBUG : pacer: Reducing sleep to 625ms fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:295: Flushing the directory cache fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1439 Error: Should be true Test: TestDirMove Messages: listing wrong, want A4/B1/C1/four (4), A4/B1/C2/five (4), A4/B1/three (5), A4/one (3), A4/two (3) got A3/one (3), A4/B1/C1/four (4), A4/B1/C2/five (4), A4/B1/three (5), A4/two (3) fstest.go:185: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:185 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:302 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1439 Error: Should be true Test: TestDirMove Messages: Unexpected file "A3/one" fstest.go:197: Not found "A4/one" fstest.go:200: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:200 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:304 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1439 Error: Not equal: expected: 0 actual : 1 Test: TestDirMove Messages: 1 objects not found fstest.go:321: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:321 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1439 Error: Not equal: expected: []string{"A4", "A4/B1", "A4/B1/C1", "A4/B1/C2", "A4/B1/C3", "A4/B2"} actual : []string{"A3", "A3/B1", "A3/B1/C1", "A3/B1/C2", "A3/B1/C3", "A3/B2", "A4", "A4/B1", "A4/B1/C1", "A4/B1/C2", "A4/B1/C3", "A4/B2"} Diff: --- Expected +++ Actual @@ -1,2 +1,8 @@ -([]string) (len=6) { +([]string) (len=12) { + (string) (len=2) "A3", + (string) (len=5) "A3/B1", + (string) (len=8) "A3/B1/C1", + (string) (len=8) "A3/B1/C2", + (string) (len=8) "A3/B1/C3", + (string) (len=5) "A3/B2", (string) (len=2) "A4", Test: TestDirMove Messages: directories 2025/01/24 02:00:17 DEBUG : pacer: Reducing sleep to 312.5ms 2025/01/24 02:00:19 DEBUG : pacer: Reducing sleep to 156.25ms 2025/01/24 02:00:20 DEBUG : pacer: Reducing sleep to 78.125ms 2025/01/24 02:00:20 DEBUG : pacer: Reducing sleep to 39.0625ms 2025/01/24 02:00:20 DEBUG : pacer: Reducing sleep to 19.53125ms 2025/01/24 02:00:20 DEBUG : pacer: Reducing sleep to 10ms --- FAIL: TestDirMove (163.83s) FAIL 2025/01/24 02:00:28 DEBUG : gofile root 'rclone-test-sazulaj2zehe': Purge remote "./operations.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose -test.run '^(TestDirMove|TestMaxDeleteSize|TestRmdirsNoLeaveRoot)$'" - Finished ERROR in 3m45.828895412s (try 2/5): exit status 1: Failed [TestDirMove]