"./sync.test -test.v -test.timeout 1h0m0s -remote TestChunkerChunk50bSHA1QuickS3: -verbose -fast-list -size-limit 1024 -test.run '^(TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs)$'" - Starting (try 3/5) 2021/03/04 05:29:17 DEBUG : Creating backend with remote "TestChunkerChunk50bSHA1QuickS3:rclone-test-fucuxum6meyovuv6rarasuv3" 2021/03/04 05:29:17 DEBUG : Creating backend with remote "TestS3:rclone-test-fucuxum6meyovuv6rarasuv3" 2021/03/04 05:29:17 DEBUG : Reset feature "ListR" 2021/03/04 05:29:17 DEBUG : Creating backend with remote "/tmp/rclone078575353" === RUN TestServerSideMove run.go:176: Remote "Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-fucuxum6meyovuv6rarasuv3'", Local "Local file system at /tmp/rclone078575353", Modify Window "1ns" 2021/03/04 05:29:17 DEBUG : Creating backend with remote "TestChunkerChunk50bSHA1QuickS3:rclone-test-hureyeg6xehehin7nozeweh2" 2021/03/04 05:29:17 DEBUG : Creating backend with remote "TestS3:rclone-test-hureyeg6xehehin7nozeweh2" 2021/03/04 05:29:17 DEBUG : Reset feature "ListR" 2021/03/04 05:29:19 INFO : S3 bucket rclone-test-fucuxum6meyovuv6rarasuv3: Bucket "rclone-test-fucuxum6meyovuv6rarasuv3" created with ACL "private" 2021/03/04 05:29:19 DEBUG : potato2.rclone_chunk.001_fofqhz: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:19 INFO : potato2.rclone_chunk.001_fofqhz: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/03/04 05:29:19 INFO : potato2.rclone_chunk.001_fofqhz: Deleted 2021/03/04 05:29:19 DEBUG : potato2.rclone_chunk.002_fofqhz: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/03/04 05:29:19 INFO : potato2.rclone_chunk.002_fofqhz: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/03/04 05:29:19 INFO : potato2.rclone_chunk.002_fofqhz: Deleted 2021/03/04 05:29:19 DEBUG : empty space.rclone_chunk.001_fofqmq: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/03/04 05:29:19 INFO : empty space.rclone_chunk.001_fofqmq: Copied (server-side copy) to: empty space 2021/03/04 05:29:19 INFO : empty space.rclone_chunk.001_fofqmq: Deleted 2021/03/04 05:29:19 DEBUG : potato3.rclone_chunk.001_fofqg1: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:19 INFO : potato3.rclone_chunk.001_fofqg1: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:19 INFO : potato3.rclone_chunk.001_fofqg1: Deleted 2021/03/04 05:29:20 DEBUG : potato3.rclone_chunk.002_fofqg1: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/03/04 05:29:20 INFO : potato3.rclone_chunk.002_fofqg1: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:20 INFO : potato3.rclone_chunk.002_fofqg1: Deleted sync_test.go:1294: Server side move (if possible) Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-fucuxum6meyovuv6rarasuv3' -> Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hureyeg6xehehin7nozeweh2' 2021/03/04 05:29:20 INFO : S3 bucket rclone-test-hureyeg6xehehin7nozeweh2: Bucket "rclone-test-hureyeg6xehehin7nozeweh2" created with ACL "private" 2021/03/04 05:29:20 DEBUG : empty space.rclone_chunk.001_fofrih: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/03/04 05:29:20 INFO : empty space.rclone_chunk.001_fofrih: Copied (server-side copy) to: empty space 2021/03/04 05:29:20 INFO : empty space.rclone_chunk.001_fofrih: Deleted 2021/03/04 05:29:21 DEBUG : potato3.rclone_chunk.001_fofr95: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.001_fofr95: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.001_fofr95: Deleted 2021/03/04 05:29:21 DEBUG : potato3.rclone_chunk.002_fofr95: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.002_fofr95: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.002_fofr95: Deleted 2021/03/04 05:29:21 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2021/03/04 05:29:21 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hureyeg6xehehin7nozeweh2': Waiting for checks to finish 2021/03/04 05:29:21 DEBUG : potato2: Can't copy - required hash not found 2021/03/04 05:29:21 DEBUG : potato3: Can't copy - required hash not found 2021/03/04 05:29:21 DEBUG : empty space: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/03/04 05:29:21 DEBUG : empty space: Unchanged skipping 2021/03/04 05:29:21 INFO : empty space: Deleted 2021/03/04 05:29:21 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hureyeg6xehehin7nozeweh2': Waiting for transfers to finish 2021/03/04 05:29:21 DEBUG : potato2.rclone_chunk.001_fofsst: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:21 INFO : potato2.rclone_chunk.001_fofsst: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/03/04 05:29:21 INFO : potato2.rclone_chunk.001_fofsst: Deleted 2021/03/04 05:29:21 DEBUG : potato3.rclone_chunk.001_fofs4b: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.001_fofs4b: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:21 DEBUG : potato2.rclone_chunk.002_fofsst: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/03/04 05:29:21 INFO : potato2.rclone_chunk.002_fofsst: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.001_fofs4b: Deleted 2021/03/04 05:29:21 INFO : potato2.rclone_chunk.002_fofsst: Deleted 2021/03/04 05:29:21 DEBUG : potato3.rclone_chunk.002_fofs4b: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.002_fofs4b: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:21 INFO : potato2: Copied (new) 2021/03/04 05:29:21 INFO : potato3.rclone_chunk.002_fofs4b: Deleted 2021/03/04 05:29:21 INFO : potato3: Copied (replaced existing) 2021/03/04 05:29:21 INFO : potato2: Deleted 2021/03/04 05:29:21 INFO : potato3: Deleted 2021/03/04 05:29:22 DEBUG : Creating backend with remote "TestChunkerChunk50bSHA1QuickS3:rclone-test-pucopun5zosilov2pacaful6" 2021/03/04 05:29:22 DEBUG : Creating backend with remote "TestS3:rclone-test-pucopun5zosilov2pacaful6" 2021/03/04 05:29:22 DEBUG : Reset feature "ListR" 2021/03/04 05:29:22 DEBUG : empty space: copy non-chunked object... 2021/03/04 05:29:22 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-pucopun5zosilov2pacaful6': Waiting for checks to finish 2021/03/04 05:29:22 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-pucopun5zosilov2pacaful6': Waiting for transfers to finish 2021/03/04 05:29:22 DEBUG : potato2: Can't copy - required hash not found 2021/03/04 05:29:22 DEBUG : potato3: Can't copy - required hash not found 2021/03/04 05:29:22 INFO : S3 bucket rclone-test-pucopun5zosilov2pacaful6: Bucket "rclone-test-pucopun5zosilov2pacaful6" created with ACL "private" 2021/03/04 05:29:22 INFO : empty space: Copied (server-side copy) 2021/03/04 05:29:22 INFO : empty space: Deleted 2021/03/04 05:29:22 ERROR : potato2: Failed to copy: object not found 2021/03/04 05:29:22 ERROR : potato2: Not deleting source as copy failed: object not found 2021/03/04 05:29:22 DEBUG : potato3.rclone_chunk.001_foftjw: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:22 INFO : potato3.rclone_chunk.001_foftjw: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:23 INFO : potato3.rclone_chunk.001_foftjw: Deleted 2021/03/04 05:29:23 DEBUG : potato3.rclone_chunk.002_foftjw: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/03/04 05:29:23 INFO : potato3.rclone_chunk.002_foftjw: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:23 INFO : potato3.rclone_chunk.002_foftjw: Deleted 2021/03/04 05:29:23 INFO : potato3: Copied (new) 2021/03/04 05:29:23 INFO : potato3: Deleted sync_test.go:1331: Error Trace: sync_test.go:1331 sync_test.go:1398 Error: Received unexpected error: object not found Test: TestServerSideMove 2021/03/04 05:29:23 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-pucopun5zosilov2pacaful6': Purge object "empty space" 2021/03/04 05:29:23 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-pucopun5zosilov2pacaful6': Purge object "potato3" 2021/03/04 05:29:23 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-pucopun5zosilov2pacaful6': Purge dir "" 2021/03/04 05:29:23 purge failed to rmdir "": BucketNotEmpty: The bucket you tried to delete is not empty status code: 409, request id: 90B507E459254E75, host id: S+53uxnO+I4CB6xMN23oaAGc4Yu++MEpUckLNwRcNZ2M9nP7VV7zbu38qlDC+lI6Fr2ypDBoNXU= 2021/03/04 05:29:23 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hureyeg6xehehin7nozeweh2': Purge object "potato2" 2021/03/04 05:29:23 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hureyeg6xehehin7nozeweh2': Purge dir "" 2021/03/04 05:29:23 INFO : S3 bucket rclone-test-hureyeg6xehehin7nozeweh2: Bucket "rclone-test-hureyeg6xehehin7nozeweh2" deleted --- FAIL: TestServerSideMove (6.03s) === RUN TestServerSideMoveDeleteEmptySourceDirs run.go:176: Remote "Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-fucuxum6meyovuv6rarasuv3'", Local "Local file system at /tmp/rclone078575353", Modify Window "1ns" 2021/03/04 05:29:23 DEBUG : Creating backend with remote "TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6" 2021/03/04 05:29:23 DEBUG : Creating backend with remote "TestS3:rclone-test-hipocoz4matazeg5nuziwab6" 2021/03/04 05:29:23 DEBUG : Reset feature "ListR" 2021/03/04 05:29:24 DEBUG : potato2.rclone_chunk.001_fofucd: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:24 INFO : potato2.rclone_chunk.001_fofucd: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/03/04 05:29:24 INFO : potato2.rclone_chunk.001_fofucd: Deleted 2021/03/04 05:29:24 DEBUG : potato2.rclone_chunk.002_fofucd: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/03/04 05:29:24 INFO : potato2.rclone_chunk.002_fofucd: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/03/04 05:29:24 INFO : potato2.rclone_chunk.002_fofucd: Deleted 2021/03/04 05:29:24 DEBUG : empty space.rclone_chunk.001_fofv54: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/03/04 05:29:24 INFO : empty space.rclone_chunk.001_fofv54: Copied (server-side copy) to: empty space 2021/03/04 05:29:24 INFO : empty space.rclone_chunk.001_fofv54: Deleted 2021/03/04 05:29:24 DEBUG : potato3.rclone_chunk.001_fofvnd: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:24 INFO : potato3.rclone_chunk.001_fofvnd: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:24 INFO : potato3.rclone_chunk.001_fofvnd: Deleted 2021/03/04 05:29:24 DEBUG : potato3.rclone_chunk.002_fofvnd: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/03/04 05:29:24 INFO : potato3.rclone_chunk.002_fofvnd: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:24 INFO : potato3.rclone_chunk.002_fofvnd: Deleted 2021/03/04 05:29:24 DEBUG : tomatoDir: Making directory sync_test.go:1294: Server side move (if possible) Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-fucuxum6meyovuv6rarasuv3' -> Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6' 2021/03/04 05:29:25 INFO : S3 bucket rclone-test-hipocoz4matazeg5nuziwab6: Bucket "rclone-test-hipocoz4matazeg5nuziwab6" created with ACL "private" 2021/03/04 05:29:25 DEBUG : empty space.rclone_chunk.001_fofwms: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/03/04 05:29:25 INFO : empty space.rclone_chunk.001_fofwms: Copied (server-side copy) to: empty space 2021/03/04 05:29:25 INFO : empty space.rclone_chunk.001_fofwms: Deleted 2021/03/04 05:29:25 DEBUG : potato3.rclone_chunk.001_fofwes: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:25 INFO : potato3.rclone_chunk.001_fofwes: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:25 INFO : potato3.rclone_chunk.001_fofwes: Deleted 2021/03/04 05:29:25 DEBUG : potato3.rclone_chunk.002_fofwes: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/03/04 05:29:25 INFO : potato3.rclone_chunk.002_fofwes: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:25 INFO : potato3.rclone_chunk.002_fofwes: Deleted 2021/03/04 05:29:26 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2021/03/04 05:29:26 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6': Waiting for checks to finish 2021/03/04 05:29:26 DEBUG : potato2: Can't copy - required hash not found 2021/03/04 05:29:26 DEBUG : potato3: Can't copy - required hash not found 2021/03/04 05:29:26 DEBUG : empty space: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/03/04 05:29:26 DEBUG : empty space: Unchanged skipping 2021/03/04 05:29:26 INFO : empty space: Deleted 2021/03/04 05:29:26 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6': Waiting for transfers to finish 2021/03/04 05:29:26 DEBUG : potato2.rclone_chunk.001_fofxz8: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:26 INFO : potato2.rclone_chunk.001_fofxz8: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/03/04 05:29:26 INFO : potato2.rclone_chunk.001_fofxz8: Deleted 2021/03/04 05:29:26 DEBUG : potato2.rclone_chunk.002_fofxz8: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/03/04 05:29:26 INFO : potato2.rclone_chunk.002_fofxz8: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/03/04 05:29:26 INFO : potato2.rclone_chunk.002_fofxz8: Deleted 2021/03/04 05:29:26 INFO : potato2: Copied (new) 2021/03/04 05:29:26 DEBUG : potato3.rclone_chunk.001_fofxgs: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/03/04 05:29:26 INFO : potato3.rclone_chunk.001_fofxgs: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/03/04 05:29:26 INFO : potato3.rclone_chunk.001_fofxgs: Deleted 2021/03/04 05:29:26 INFO : potato2: Deleted 2021/03/04 05:29:26 DEBUG : potato3.rclone_chunk.002_fofxgs: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/03/04 05:29:26 INFO : potato3.rclone_chunk.002_fofxgs: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/03/04 05:29:26 INFO : potato3.rclone_chunk.002_fofxgs: Deleted 2021/03/04 05:29:26 INFO : potato3: Copied (replaced existing) 2021/03/04 05:29:26 INFO : potato3: Deleted 2021/03/04 05:29:27 DEBUG : Creating backend with remote "TestChunkerChunk50bSHA1QuickS3:rclone-test-cagokut7rakineg5bigipor9" 2021/03/04 05:29:27 DEBUG : Creating backend with remote "TestS3:rclone-test-cagokut7rakineg5bigipor9" 2021/03/04 05:29:27 DEBUG : Reset feature "ListR" 2021/03/04 05:29:27 DEBUG : tomatoDir: Making directory 2021/03/04 05:29:27 DEBUG : empty space: copy non-chunked object... 2021/03/04 05:29:27 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-cagokut7rakineg5bigipor9': Waiting for checks to finish 2021/03/04 05:29:27 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-cagokut7rakineg5bigipor9': Waiting for transfers to finish 2021/03/04 05:29:27 DEBUG : potato2: Can't copy - required hash not found 2021/03/04 05:29:27 DEBUG : potato3: Can't copy - required hash not found 2021/03/04 05:29:27 INFO : S3 bucket rclone-test-cagokut7rakineg5bigipor9: Bucket "rclone-test-cagokut7rakineg5bigipor9" created with ACL "private" 2021/03/04 05:29:27 INFO : empty space: Copied (server-side copy) 2021/03/04 05:29:27 INFO : empty space: Deleted 2021/03/04 05:29:27 ERROR : potato3: Failed to copy: object not found 2021/03/04 05:29:27 ERROR : potato3: Not deleting source as copy failed: object not found 2021/03/04 05:29:28 ERROR : potato2: Failed to copy: object not found 2021/03/04 05:29:28 ERROR : potato2: Not deleting source as copy failed: object not found sync_test.go:1331: Error Trace: sync_test.go:1331 sync_test.go:1420 Error: Received unexpected error: object not found Test: TestServerSideMoveDeleteEmptySourceDirs 2021/03/04 05:29:28 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-cagokut7rakineg5bigipor9': Purge object "empty space" 2021/03/04 05:29:28 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-cagokut7rakineg5bigipor9': Purge dir "" 2021/03/04 05:29:28 purge failed to rmdir "": BucketNotEmpty: The bucket you tried to delete is not empty status code: 409, request id: 772010E9152E34B2, host id: 1xRROafHj7B57kzGwZYXcZzMgf1qu3jzeGE6ZwQ1sMFq9/suOrCVD4tt/n9Eu3QNn1h0yeYAW8E= 2021/03/04 05:29:28 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6': Purge object "potato2" 2021/03/04 05:29:28 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6': Purge object "potato3" 2021/03/04 05:29:28 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-hipocoz4matazeg5nuziwab6': Purge dir "" 2021/03/04 05:29:28 INFO : S3 bucket rclone-test-hipocoz4matazeg5nuziwab6: Bucket "rclone-test-hipocoz4matazeg5nuziwab6" deleted --- FAIL: TestServerSideMoveDeleteEmptySourceDirs (4.76s) FAIL 2021/03/04 05:29:28 DEBUG : Chunked 'TestChunkerChunk50bSHA1QuickS3:rclone-test-fucuxum6meyovuv6rarasuv3': Purge dir "" 2021/03/04 05:29:28 INFO : S3 bucket rclone-test-fucuxum6meyovuv6rarasuv3: Bucket "rclone-test-fucuxum6meyovuv6rarasuv3" deleted "./sync.test -test.v -test.timeout 1h0m0s -remote TestChunkerChunk50bSHA1QuickS3: -verbose -fast-list -size-limit 1024 -test.run '^(TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs)$'" - Finished ERROR in 11.263775051s (try 3/5): exit status 1: Failed [TestServerSideMove TestServerSideMoveDeleteEmptySourceDirs]