"./operations.test -test.v -test.timeout 1h0m0s -remote TestHubic: -verbose -test.run '^(TestCheckDownload|TestCheckSizeOnly|TestCopyFileCompareDest|TestCopyFileCopyDest|TestCopyURL|TestDirMove|TestLsd|TestMoveFile|TestMoveFileBackupDir)$/^(1|2|3)$'" - Starting (try 2/5) 2021/06/01 05:11:05 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0" 2021/06/01 05:11:05 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2021/06/01 05:11:05 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h50m43.092438763s) 2021/06/01 05:11:05 DEBUG : Creating backend with remote "/tmp/rclone794030473" === RUN TestCheckDownload run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:11:05 INFO : Swift container rclone-test-lacaroy1cicapus6sebaday0: Container "rclone-test-lacaroy1cicapus6sebaday0" created === RUN TestCheckDownload/1 === RUN TestCheckDownload/2 === RUN TestCheckDownload/3 --- PASS: TestCheckDownload (10.62s) --- PASS: TestCheckDownload/1 (0.30s) --- PASS: TestCheckDownload/2 (0.30s) --- PASS: TestCheckDownload/3 (0.28s) === RUN TestCheckSizeOnly run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" === RUN TestCheckSizeOnly/1 === RUN TestCheckSizeOnly/2 === RUN TestCheckSizeOnly/3 --- PASS: TestCheckSizeOnly (8.95s) --- PASS: TestCheckSizeOnly/1 (0.03s) --- PASS: TestCheckSizeOnly/2 (0.03s) --- PASS: TestCheckSizeOnly/3 (0.03s) === RUN TestLsd run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" fstest.go:299: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:293: Sleeping for 6s just to make sure --- PASS: TestLsd (9.06s) === RUN TestCopyURL run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:11:36 DEBUG : filename.txt: File name found in url --- PASS: TestCopyURL (7.17s) === RUN TestMoveFile run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:11:42 DEBUG : file1: Need to transfer - File not found at Destination 2021/06/01 05:11:44 DEBUG : file1: md5 = 0ef726ce9b1a7692357ff70dd321d595 OK 2021/06/01 05:11:44 INFO : file1: Copied (new) to: sub/file2 2021/06/01 05:11:44 INFO : file1: Deleted 2021/06/01 05:11:44 DEBUG : file1: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:11:44 DEBUG : file1: Unchanged skipping 2021/06/01 05:11:44 INFO : file1: Deleted 2021/06/01 05:11:44 DEBUG : Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0: don't need to copy/move sub/file2, it is already at target location --- PASS: TestMoveFile (3.86s) === RUN TestMoveFileBackupDir run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:11:46 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0/backup" 2021/06/01 05:11:47 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h50m1.981363935s) 2021/06/01 05:11:47 DEBUG : dst/file1: Sizes differ (src 14 vs dst 18) 2021/06/01 05:11:47 INFO : Swift container rclone-test-lacaroy1cicapus6sebaday0 path backup: Container "rclone-test-lacaroy1cicapus6sebaday0" created 2021/06/01 05:11:48 DEBUG : dst/file1: md5 = 05164b153084ba910184c26e561a7c18 OK 2021/06/01 05:11:48 INFO : dst/file1: Copied (server-side copy) 2021/06/01 05:11:49 INFO : dst/file1: Deleted 2021/06/01 05:11:50 DEBUG : dst/file1: md5 = 0ef726ce9b1a7692357ff70dd321d595 OK 2021/06/01 05:11:50 INFO : dst/file1: Copied (new) 2021/06/01 05:11:50 INFO : dst/file1: Deleted --- PASS: TestMoveFileBackupDir (6.70s) === RUN TestCopyFileCompareDest run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:11:52 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0/dst" 2021/06/01 05:11:52 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h49m56.625170674s) 2021/06/01 05:11:52 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0/CompareDest" 2021/06/01 05:11:53 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h49m55.997062919s) 2021/06/01 05:11:53 DEBUG : one: Need to transfer - File not found at Destination 2021/06/01 05:11:53 INFO : Swift container rclone-test-lacaroy1cicapus6sebaday0 path dst: Container "rclone-test-lacaroy1cicapus6sebaday0" created 2021/06/01 05:11:54 DEBUG : one: md5 = f97c5d29941bfb1b2fdab0874906ab82 OK 2021/06/01 05:11:54 INFO : one: Copied (new) 2021/06/01 05:11:55 DEBUG : one: Sizes differ (src 5 vs dst 3) 2021/06/01 05:11:56 DEBUG : one: md5 = 07912d142f5d63ee918b34796b5a2432 OK 2021/06/01 05:11:56 INFO : one: Copied (replaced existing) 2021/06/01 05:11:59 DEBUG : one: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:11:59 DEBUG : one: Destination found in --compare-dest, skipping 2021/06/01 05:12:01 DEBUG : two: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:12:01 DEBUG : two: Destination found in --compare-dest, skipping 2021/06/01 05:12:02 DEBUG : two: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:12:02 DEBUG : two: Destination found in --compare-dest, skipping 2021/06/01 05:12:02 DEBUG : two: Sizes differ (src 5 vs dst 3) 2021/06/01 05:12:02 DEBUG : two: Need to transfer - File not found at Destination 2021/06/01 05:12:03 DEBUG : two: md5 = 2379e4ce8c3380e996ab0509f17069ad OK 2021/06/01 05:12:03 INFO : two: Copied (new) --- PASS: TestCopyFileCompareDest (14.26s) === RUN TestCopyFileCopyDest run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:12:06 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0/dst" 2021/06/01 05:12:06 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h49m42.350327721s) 2021/06/01 05:12:06 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0/CopyDest" 2021/06/01 05:12:07 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h49m41.859768568s) 2021/06/01 05:12:07 DEBUG : one: Need to transfer - File not found at Destination 2021/06/01 05:12:07 INFO : Swift container rclone-test-lacaroy1cicapus6sebaday0 path dst: Container "rclone-test-lacaroy1cicapus6sebaday0" created 2021/06/01 05:12:08 DEBUG : one: md5 = f97c5d29941bfb1b2fdab0874906ab82 OK 2021/06/01 05:12:08 INFO : one: Copied (new) 2021/06/01 05:12:09 DEBUG : one: Sizes differ (src 5 vs dst 3) 2021/06/01 05:12:10 DEBUG : one: md5 = 07912d142f5d63ee918b34796b5a2432 OK 2021/06/01 05:12:10 INFO : one: Copied (replaced existing) 2021/06/01 05:12:12 DEBUG : Creating backend with remote "TestHubic:rclone-test-lacaroy1cicapus6sebaday0/BackupDir" 2021/06/01 05:12:13 DEBUG : Hubic: Got swift credentials (expiry 2021-06-02 07:01:49 +0200 +0200 in 23h49m35.914382489s) 2021/06/01 05:12:13 DEBUG : one: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:12:13 DEBUG : one: Sizes differ (src 5 vs dst 3) 2021/06/01 05:12:13 INFO : Swift container rclone-test-lacaroy1cicapus6sebaday0 path BackupDir: Container "rclone-test-lacaroy1cicapus6sebaday0" created 2021/06/01 05:12:15 DEBUG : one: md5 = f97c5d29941bfb1b2fdab0874906ab82 OK 2021/06/01 05:12:15 INFO : one: Copied (server-side copy) 2021/06/01 05:12:15 INFO : one: Deleted 2021/06/01 05:12:16 DEBUG : one: md5 = 07912d142f5d63ee918b34796b5a2432 OK 2021/06/01 05:12:16 INFO : one: Copied (server-side copy) 2021/06/01 05:12:16 DEBUG : one: Destination found in --copy-dest, using server-side copy 2021/06/01 05:12:18 DEBUG : two: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:12:19 DEBUG : two: md5 = b8a9f715dbb64fd5c56e7783c6820a61 OK 2021/06/01 05:12:19 INFO : two: Copied (server-side copy) 2021/06/01 05:12:19 DEBUG : two: Destination found in --copy-dest, using server-side copy 2021/06/01 05:12:20 DEBUG : two: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:12:20 DEBUG : two: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/06/01 05:12:20 DEBUG : two: Unchanged skipping 2021/06/01 05:12:21 DEBUG : three: Sizes differ (src 7 vs dst 5) 2021/06/01 05:12:21 DEBUG : three: Destination not found in --copy-dest 2021/06/01 05:12:21 DEBUG : three: Need to transfer - File not found at Destination 2021/06/01 05:12:23 DEBUG : three: md5 = 1bccb9dccb3e9f6a3f9d2a8bdb54b7f5 OK 2021/06/01 05:12:23 INFO : three: Copied (new) fstest.go:299: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:293: Sleeping for 6s just to make sure --- PASS: TestCopyFileCopyDest (29.29s) === RUN TestDirMove run.go:176: Remote "Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0", Local "Local file system at /tmp/rclone794030473", Modify Window "1ns" 2021/06/01 05:12:41 DEBUG : A1/B2: Making directory 2021/06/01 05:12:41 DEBUG : A1/B1/C3: Making directory fstest.go:251: Filtering empty directory "A1/B2" fstest.go:251: Filtering empty directory "A1/B1/C3" 2021/06/01 05:12:43 DEBUG : A1/one: md5 = f97c5d29941bfb1b2fdab0874906ab82 OK 2021/06/01 05:12:43 INFO : A1/one: Copied (server-side copy) to: A2/one 2021/06/01 05:12:43 DEBUG : A1/B1/C2/five: md5 = 30056e1cab7a61d256fc8edd970d14f5 OK 2021/06/01 05:12:43 INFO : A1/B1/C2/five: Copied (server-side copy) to: A2/B1/C2/five 2021/06/01 05:12:43 DEBUG : A1/B1/C1/four: md5 = 8cbad96aced40b3838dd9f07f6ef5772 OK 2021/06/01 05:12:43 INFO : A1/B1/C1/four: Copied (server-side copy) to: A2/B1/C1/four 2021/06/01 05:12:44 DEBUG : A1/two: md5 = b8a9f715dbb64fd5c56e7783c6820a61 OK 2021/06/01 05:12:44 INFO : A1/two: Copied (server-side copy) to: A2/two 2021/06/01 05:12:44 INFO : A1/one: Deleted 2021/06/01 05:12:44 INFO : A1/B1/C2/five: Deleted 2021/06/01 05:12:44 INFO : A1/B1/C1/four: Deleted 2021/06/01 05:12:44 INFO : A1/two: Deleted 2021/06/01 05:12:46 DEBUG : A1/B1/three: md5 = 35d6d33467aae9a2e3dccb4b6b027878 OK 2021/06/01 05:12:46 INFO : A1/B1/three: Copied (server-side copy) to: A2/B1/three 2021/06/01 05:12:46 INFO : A1/B1/three: Deleted fstest.go:251: Filtering empty directory "A2/B2" fstest.go:251: Filtering empty directory "A2/B1/C3" fstest.go:299: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:299: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:299: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:306: Error Trace: fstest.go:306 fstest.go:339 operations_test.go:1354 Error: Should be true Test: TestDirMove Messages: listing wrong, want A2/B1/C1/four (4), A2/B1/C2/five (4), A2/B1/three (5), A2/one (3), A2/two (3) got A2/B1/C1/four (4), A2/B1/three (5), A2/one (3), A2/two (3) fstest.go:204: Not found "A2/B1/C2/five" fstest.go:207: Error Trace: fstest.go:207 fstest.go:311 fstest.go:339 operations_test.go:1354 Error: Not equal: expected: 0 actual : 1 Test: TestDirMove Messages: 1 objects not found fstest.go:328: Error Trace: fstest.go:328 fstest.go:339 operations_test.go:1354 Error: Not equal: expected: []string{"A2", "A2/B1", "A2/B1/C1", "A2/B1/C2"} actual : []string{"A2", "A2/B1", "A2/B1/C1"} Diff: --- Expected +++ Actual @@ -1,6 +1,5 @@ -([]string) (len=4) { +([]string) (len=3) { (string) (len=2) "A2", (string) (len=5) "A2/B1", - (string) (len=8) "A2/B1/C1", - (string) (len=8) "A2/B1/C2" + (string) (len=8) "A2/B1/C1" } Test: TestDirMove Messages: directories 2021/06/01 05:12:55 DEBUG : A2/one: md5 = f97c5d29941bfb1b2fdab0874906ab82 OK 2021/06/01 05:12:55 INFO : A2/one: Copied (server-side copy) to: A3/one 2021/06/01 05:12:55 DEBUG : A2/B1/three: md5 = 35d6d33467aae9a2e3dccb4b6b027878 OK 2021/06/01 05:12:55 INFO : A2/B1/three: Copied (server-side copy) to: A3/B1/three 2021/06/01 05:12:55 DEBUG : A2/B1/C1/four: md5 = 8cbad96aced40b3838dd9f07f6ef5772 OK 2021/06/01 05:12:55 INFO : A2/B1/C1/four: Copied (server-side copy) to: A3/B1/C1/four 2021/06/01 05:12:56 DEBUG : A2/two: md5 = b8a9f715dbb64fd5c56e7783c6820a61 OK 2021/06/01 05:12:56 INFO : A2/two: Copied (server-side copy) to: A3/two 2021/06/01 05:12:56 INFO : A2/one: Deleted 2021/06/01 05:12:56 INFO : A2/B1/three: Deleted 2021/06/01 05:12:56 INFO : A2/B1/C1/four: Deleted 2021/06/01 05:12:56 INFO : A2/two: Deleted fstest.go:251: Filtering empty directory "A3/B2" fstest.go:251: Filtering empty directory "A3/B1/C3" fstest.go:299: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:299: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:299: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:306: Error Trace: fstest.go:306 fstest.go:339 operations_test.go:1379 Error: Should be true Test: TestDirMove Messages: listing wrong, want A3/B1/C1/four (4), A3/B1/C2/five (4), A3/B1/three (5), A3/one (3), A3/two (3) got A3/B1/C1/four (4), A3/B1/three (5), A3/one (3), A3/two (3) fstest.go:204: Not found "A3/B1/C2/five" fstest.go:207: Error Trace: fstest.go:207 fstest.go:311 fstest.go:339 operations_test.go:1379 Error: Not equal: expected: 0 actual : 1 Test: TestDirMove Messages: 1 objects not found fstest.go:328: Error Trace: fstest.go:328 fstest.go:339 operations_test.go:1379 Error: Not equal: expected: []string{"A3", "A3/B1", "A3/B1/C1", "A3/B1/C2"} actual : []string{"A3", "A3/B1", "A3/B1/C1"} Diff: --- Expected +++ Actual @@ -1,6 +1,5 @@ -([]string) (len=4) { +([]string) (len=3) { (string) (len=2) "A3", (string) (len=5) "A3/B1", - (string) (len=8) "A3/B1/C1", - (string) (len=8) "A3/B1/C2" + (string) (len=8) "A3/B1/C1" } Test: TestDirMove Messages: directories --- FAIL: TestDirMove (31.23s) FAIL 2021/06/01 05:13:07 DEBUG : Hubic Swift container rclone-test-lacaroy1cicapus6sebaday0: Purge remote 2021/06/01 05:13:07 DEBUG : Waiting for deletions to finish 2021/06/01 05:13:07 DEBUG : pacer: low level retry 1/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 10ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 2/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 3/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 4/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 5/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 6/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 7/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2021/06/01 05:13:07 DEBUG : pacer: low level retry 8/10 (error Container Not Empty) 2021/06/01 05:13:07 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2021/06/01 05:13:08 DEBUG : pacer: low level retry 9/10 (error Container Not Empty) 2021/06/01 05:13:08 DEBUG : pacer: Rate limited, increasing sleep to 2s 2021/06/01 05:13:09 DEBUG : pacer: low level retry 10/10 (error Container Not Empty) 2021/06/01 05:13:09 purge failed: Container Not Empty "./operations.test -test.v -test.timeout 1h0m0s -remote TestHubic: -verbose -test.run '^(TestCheckDownload|TestCheckSizeOnly|TestCopyFileCompareDest|TestCopyFileCopyDest|TestCopyURL|TestDirMove|TestLsd|TestMoveFile|TestMoveFileBackupDir)$/^(1|2|3)$'" - Finished ERROR in 2m4.133374053s (try 2/5): exit status 1: Failed [TestDirMove]