"./sync.test -test.v -test.timeout 1h0m0s -remote TestChunkerChunk50bMD5QuickS3: -verbose -fast-list -size-limit 1024 -test.run '^(TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs)$'" - Starting (try 2/5) 2021/01/04 05:04:09 DEBUG : Creating backend with remote "TestChunkerChunk50bMD5QuickS3:rclone-test-waxadug4gasufot5deyiwep0" 2021/01/04 05:04:09 DEBUG : Creating backend with remote "TestS3:rclone-test-waxadug4gasufot5deyiwep0" 2021/01/04 05:04:09 DEBUG : Reset feature "ListR" 2021/01/04 05:04:09 DEBUG : Creating backend with remote "/tmp/rclone044544909" === RUN TestServerSideMove run.go:176: Remote "Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-waxadug4gasufot5deyiwep0'", Local "Local file system at /tmp/rclone044544909", Modify Window "1ns" 2021/01/04 05:04:09 DEBUG : Creating backend with remote "TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0" 2021/01/04 05:04:09 DEBUG : Creating backend with remote "TestS3:rclone-test-lahopex6porumic7fiwugim0" 2021/01/04 05:04:09 DEBUG : Reset feature "ListR" 2021/01/04 05:04:10 INFO : S3 bucket rclone-test-waxadug4gasufot5deyiwep0: Bucket "rclone-test-waxadug4gasufot5deyiwep0" created with ACL "private" 2021/01/04 05:04:10 DEBUG : potato2.rclone_chunk.001_edx8jt: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:10 INFO : potato2.rclone_chunk.001_edx8jt: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/01/04 05:04:10 INFO : potato2.rclone_chunk.001_edx8jt: Deleted 2021/01/04 05:04:10 DEBUG : potato2.rclone_chunk.002_edx8jt: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/01/04 05:04:10 INFO : potato2.rclone_chunk.002_edx8jt: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/01/04 05:04:10 INFO : potato2.rclone_chunk.002_edx8jt: Deleted 2021/01/04 05:04:10 DEBUG : empty space.rclone_chunk.001_edx8a8: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/01/04 05:04:10 INFO : empty space.rclone_chunk.001_edx8a8: Copied (server-side copy) to: empty space 2021/01/04 05:04:10 INFO : empty space.rclone_chunk.001_edx8a8: Deleted 2021/01/04 05:04:11 DEBUG : potato3.rclone_chunk.001_edx8z3: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:11 INFO : potato3.rclone_chunk.001_edx8z3: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:11 INFO : potato3.rclone_chunk.001_edx8z3: Deleted 2021/01/04 05:04:11 DEBUG : potato3.rclone_chunk.002_edx8z3: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/01/04 05:04:11 INFO : potato3.rclone_chunk.002_edx8z3: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:11 INFO : potato3.rclone_chunk.002_edx8z3: Deleted sync_test.go:1294: Server side move (if possible) Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-waxadug4gasufot5deyiwep0' -> Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0' 2021/01/04 05:04:12 INFO : S3 bucket rclone-test-lahopex6porumic7fiwugim0: Bucket "rclone-test-lahopex6porumic7fiwugim0" created with ACL "private" 2021/01/04 05:04:12 DEBUG : empty space.rclone_chunk.001_edxa86: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/01/04 05:04:12 INFO : empty space.rclone_chunk.001_edxa86: Copied (server-side copy) to: empty space 2021/01/04 05:04:12 INFO : empty space.rclone_chunk.001_edxa86: Deleted 2021/01/04 05:04:12 DEBUG : potato3.rclone_chunk.001_edxazt: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:12 INFO : potato3.rclone_chunk.001_edxazt: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:12 INFO : potato3.rclone_chunk.001_edxazt: Deleted 2021/01/04 05:04:12 DEBUG : potato3.rclone_chunk.002_edxazt: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/01/04 05:04:12 INFO : potato3.rclone_chunk.002_edxazt: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:12 INFO : potato3.rclone_chunk.002_edxazt: Deleted 2021/01/04 05:04:12 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2021/01/04 05:04:12 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0': Waiting for checks to finish 2021/01/04 05:04:12 DEBUG : potato2: Can't copy - required hash not found 2021/01/04 05:04:12 DEBUG : empty space: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/01/04 05:04:12 DEBUG : empty space: Unchanged skipping 2021/01/04 05:04:12 DEBUG : potato3: Can't copy - required hash not found 2021/01/04 05:04:12 INFO : empty space: Deleted 2021/01/04 05:04:12 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0': Waiting for transfers to finish 2021/01/04 05:04:13 DEBUG : potato2.rclone_chunk.001_edxaqe: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:13 INFO : potato2.rclone_chunk.001_edxaqe: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/01/04 05:04:13 INFO : potato2.rclone_chunk.001_edxaqe: Deleted 2021/01/04 05:04:13 DEBUG : potato2.rclone_chunk.002_edxaqe: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/01/04 05:04:13 INFO : potato2.rclone_chunk.002_edxaqe: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/01/04 05:04:13 DEBUG : potato3.rclone_chunk.001_edxa1p: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:13 INFO : potato3.rclone_chunk.001_edxa1p: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:13 INFO : potato2.rclone_chunk.002_edxaqe: Deleted 2021/01/04 05:04:13 INFO : potato3.rclone_chunk.001_edxa1p: Deleted 2021/01/04 05:04:13 INFO : potato2: Copied (new) 2021/01/04 05:04:13 DEBUG : potato3.rclone_chunk.002_edxa1p: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/01/04 05:04:13 INFO : potato3.rclone_chunk.002_edxa1p: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:13 INFO : potato3.rclone_chunk.002_edxa1p: Deleted 2021/01/04 05:04:13 INFO : potato2: Deleted 2021/01/04 05:04:13 INFO : potato3: Copied (replaced existing) 2021/01/04 05:04:13 INFO : potato3: Deleted 2021/01/04 05:04:13 DEBUG : Creating backend with remote "TestChunkerChunk50bMD5QuickS3:rclone-test-bivayoh6duxayan2lepaxal2" 2021/01/04 05:04:13 DEBUG : Creating backend with remote "TestS3:rclone-test-bivayoh6duxayan2lepaxal2" 2021/01/04 05:04:13 DEBUG : Reset feature "ListR" 2021/01/04 05:04:13 DEBUG : empty space: copy non-chunked object... 2021/01/04 05:04:13 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-bivayoh6duxayan2lepaxal2': Waiting for checks to finish 2021/01/04 05:04:13 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-bivayoh6duxayan2lepaxal2': Waiting for transfers to finish 2021/01/04 05:04:13 DEBUG : potato3: Can't copy - required hash not found 2021/01/04 05:04:13 DEBUG : potato2: Can't copy - required hash not found 2021/01/04 05:04:14 INFO : S3 bucket rclone-test-bivayoh6duxayan2lepaxal2: Bucket "rclone-test-bivayoh6duxayan2lepaxal2" created with ACL "private" 2021/01/04 05:04:14 INFO : empty space: Copied (server-side copy) 2021/01/04 05:04:14 ERROR : potato2: Failed to copy: object not found 2021/01/04 05:04:14 ERROR : potato2: Not deleting source as copy failed: object not found 2021/01/04 05:04:14 INFO : empty space: Deleted 2021/01/04 05:04:14 ERROR : potato3: Failed to copy: object not found 2021/01/04 05:04:14 ERROR : potato3: Not deleting source as copy failed: object not found sync_test.go:1331: Error Trace: sync_test.go:1331 sync_test.go:1398 Error: Received unexpected error: object not found Test: TestServerSideMove 2021/01/04 05:04:14 ERROR : : error listing: directory not found 2021/01/04 05:04:14 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-bivayoh6duxayan2lepaxal2': Purge dir "" 2021/01/04 05:04:14 purge failed to rmdir "": BucketNotEmpty: The bucket you tried to delete is not empty status code: 409, request id: 3AF55EF0DF97BEE7, host id: 5m73vPkWrPY5bfb7k8EN631ZVs0xATkdMNUJVMmD6u2CkQCCz4sgLzO3Gr2FA5WM5DPVTjEZXL4= 2021/01/04 05:04:14 purge failed: directory not found 2021/01/04 05:04:14 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0': Purge object "potato2" 2021/01/04 05:04:14 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0': Purge object "potato3" 2021/01/04 05:04:14 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-lahopex6porumic7fiwugim0': Purge dir "" 2021/01/04 05:04:14 INFO : S3 bucket rclone-test-lahopex6porumic7fiwugim0: Bucket "rclone-test-lahopex6porumic7fiwugim0" deleted --- FAIL: TestServerSideMove (5.60s) === RUN TestServerSideMoveDeleteEmptySourceDirs run.go:176: Remote "Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-waxadug4gasufot5deyiwep0'", Local "Local file system at /tmp/rclone044544909", Modify Window "1ns" 2021/01/04 05:04:14 DEBUG : Creating backend with remote "TestChunkerChunk50bMD5QuickS3:rclone-test-dutisoh7tafimud0texadoc9" 2021/01/04 05:04:14 DEBUG : Creating backend with remote "TestS3:rclone-test-dutisoh7tafimud0texadoc9" 2021/01/04 05:04:14 DEBUG : Reset feature "ListR" 2021/01/04 05:04:14 DEBUG : potato2.rclone_chunk.001_edxcza: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:14 INFO : potato2.rclone_chunk.001_edxcza: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/01/04 05:04:15 INFO : potato2.rclone_chunk.001_edxcza: Deleted 2021/01/04 05:04:15 DEBUG : potato2.rclone_chunk.002_edxcza: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/01/04 05:04:15 INFO : potato2.rclone_chunk.002_edxcza: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/01/04 05:04:15 INFO : potato2.rclone_chunk.002_edxcza: Deleted 2021/01/04 05:04:15 DEBUG : empty space.rclone_chunk.001_edxdbb: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/01/04 05:04:15 INFO : empty space.rclone_chunk.001_edxdbb: Copied (server-side copy) to: empty space 2021/01/04 05:04:15 INFO : empty space.rclone_chunk.001_edxdbb: Deleted 2021/01/04 05:04:15 DEBUG : potato3.rclone_chunk.001_edxdfx: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:15 INFO : potato3.rclone_chunk.001_edxdfx: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:15 INFO : potato3.rclone_chunk.001_edxdfx: Deleted 2021/01/04 05:04:15 DEBUG : potato3.rclone_chunk.002_edxdfx: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/01/04 05:04:15 INFO : potato3.rclone_chunk.002_edxdfx: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:15 INFO : potato3.rclone_chunk.002_edxdfx: Deleted 2021/01/04 05:04:15 DEBUG : tomatoDir: Making directory sync_test.go:1294: Server side move (if possible) Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-waxadug4gasufot5deyiwep0' -> Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-dutisoh7tafimud0texadoc9' 2021/01/04 05:04:16 INFO : S3 bucket rclone-test-dutisoh7tafimud0texadoc9: Bucket "rclone-test-dutisoh7tafimud0texadoc9" created with ACL "private" 2021/01/04 05:04:16 DEBUG : empty space.rclone_chunk.001_edxe7i: MD5 = 336d5ebc5436534e61d16e63ddfca327 OK 2021/01/04 05:04:16 INFO : empty space.rclone_chunk.001_edxe7i: Copied (server-side copy) to: empty space 2021/01/04 05:04:16 INFO : empty space.rclone_chunk.001_edxe7i: Deleted 2021/01/04 05:04:16 DEBUG : potato3.rclone_chunk.001_edxeo1: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:16 INFO : potato3.rclone_chunk.001_edxeo1: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:16 INFO : potato3.rclone_chunk.001_edxeo1: Deleted 2021/01/04 05:04:17 DEBUG : potato3.rclone_chunk.002_edxeo1: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/01/04 05:04:17 INFO : potato3.rclone_chunk.002_edxeo1: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:17 INFO : potato3.rclone_chunk.002_edxeo1: Deleted 2021/01/04 05:04:17 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2021/01/04 05:04:17 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-dutisoh7tafimud0texadoc9': Waiting for checks to finish 2021/01/04 05:04:17 DEBUG : potato3: Can't copy - required hash not found 2021/01/04 05:04:17 DEBUG : potato2: Can't copy - required hash not found 2021/01/04 05:04:17 DEBUG : empty space: Size and modification time the same (differ by 0s, within tolerance 1ns) 2021/01/04 05:04:17 DEBUG : empty space: Unchanged skipping 2021/01/04 05:04:17 INFO : empty space: Deleted 2021/01/04 05:04:17 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-dutisoh7tafimud0texadoc9': Waiting for transfers to finish 2021/01/04 05:04:17 DEBUG : potato2.rclone_chunk.001_edxf5n: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:17 INFO : potato2.rclone_chunk.001_edxf5n: Copied (server-side copy) to: potato2.rclone_chunk.001 2021/01/04 05:04:17 INFO : potato2.rclone_chunk.001_edxf5n: Deleted 2021/01/04 05:04:17 DEBUG : potato2.rclone_chunk.002_edxf5n: MD5 = 01a6651a94ab602dbfc47587d234b52c OK 2021/01/04 05:04:17 INFO : potato2.rclone_chunk.002_edxf5n: Copied (server-side copy) to: potato2.rclone_chunk.002 2021/01/04 05:04:17 DEBUG : potato3.rclone_chunk.001_edxf7c: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:17 INFO : potato3.rclone_chunk.001_edxf7c: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:17 INFO : potato2.rclone_chunk.002_edxf5n: Deleted 2021/01/04 05:04:17 INFO : potato3.rclone_chunk.001_edxf7c: Deleted 2021/01/04 05:04:17 INFO : potato2: Copied (new) 2021/01/04 05:04:17 DEBUG : potato3.rclone_chunk.002_edxf7c: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/01/04 05:04:17 INFO : potato3.rclone_chunk.002_edxf7c: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:17 INFO : potato3.rclone_chunk.002_edxf7c: Deleted 2021/01/04 05:04:17 INFO : potato2: Deleted 2021/01/04 05:04:17 INFO : potato3: Copied (replaced existing) 2021/01/04 05:04:17 INFO : potato3: Deleted 2021/01/04 05:04:17 DEBUG : Creating backend with remote "TestChunkerChunk50bMD5QuickS3:rclone-test-solerav6dimixot0bebamif6" 2021/01/04 05:04:17 DEBUG : Creating backend with remote "TestS3:rclone-test-solerav6dimixot0bebamif6" 2021/01/04 05:04:17 DEBUG : Reset feature "ListR" 2021/01/04 05:04:17 DEBUG : tomatoDir: Making directory 2021/01/04 05:04:17 DEBUG : empty space: copy non-chunked object... 2021/01/04 05:04:17 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-solerav6dimixot0bebamif6': Waiting for checks to finish 2021/01/04 05:04:17 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-solerav6dimixot0bebamif6': Waiting for transfers to finish 2021/01/04 05:04:17 DEBUG : potato3: Can't copy - required hash not found 2021/01/04 05:04:17 DEBUG : potato2: Can't copy - required hash not found 2021/01/04 05:04:18 INFO : S3 bucket rclone-test-solerav6dimixot0bebamif6: Bucket "rclone-test-solerav6dimixot0bebamif6" created with ACL "private" 2021/01/04 05:04:18 INFO : empty space: Copied (server-side copy) 2021/01/04 05:04:18 INFO : empty space: Deleted 2021/01/04 05:04:18 DEBUG : potato3.rclone_chunk.001_edxgpi: MD5 = dbbc040ac83a6f6c6816767a36d34c7b OK 2021/01/04 05:04:18 INFO : potato3.rclone_chunk.001_edxgpi: Copied (server-side copy) to: potato3.rclone_chunk.001 2021/01/04 05:04:18 ERROR : potato2: Failed to copy: object not found 2021/01/04 05:04:18 ERROR : potato2: Not deleting source as copy failed: object not found 2021/01/04 05:04:18 INFO : potato3.rclone_chunk.001_edxgpi: Deleted 2021/01/04 05:04:18 DEBUG : potato3.rclone_chunk.002_edxgpi: MD5 = 111c387f2584838efb7e0ec6f2fa2e10 OK 2021/01/04 05:04:18 INFO : potato3.rclone_chunk.002_edxgpi: Copied (server-side copy) to: potato3.rclone_chunk.002 2021/01/04 05:04:18 INFO : potato3.rclone_chunk.002_edxgpi: Deleted 2021/01/04 05:04:18 INFO : potato3: Copied (new) 2021/01/04 05:04:19 INFO : potato3: Deleted sync_test.go:1331: Error Trace: sync_test.go:1331 sync_test.go:1420 Error: Received unexpected error: object not found Test: TestServerSideMoveDeleteEmptySourceDirs 2021/01/04 05:04:19 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-solerav6dimixot0bebamif6': Purge object "empty space" 2021/01/04 05:04:19 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-solerav6dimixot0bebamif6': Purge object "potato3" 2021/01/04 05:04:19 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-solerav6dimixot0bebamif6': Purge dir "" 2021/01/04 05:04:19 purge failed to rmdir "": BucketNotEmpty: The bucket you tried to delete is not empty status code: 409, request id: AE630FF24DFF4E37, host id: lblwRC0zjBt5UDIwOws+QbP7EZQZHRc7XX2a2EMHSYmFLLy8aqQB/3Avj/KYZqm8/f/v4/wdF+s= 2021/01/04 05:04:19 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-dutisoh7tafimud0texadoc9': Purge object "potato2" 2021/01/04 05:04:19 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-dutisoh7tafimud0texadoc9': Purge dir "" 2021/01/04 05:04:19 INFO : S3 bucket rclone-test-dutisoh7tafimud0texadoc9: Bucket "rclone-test-dutisoh7tafimud0texadoc9" deleted --- FAIL: TestServerSideMoveDeleteEmptySourceDirs (4.75s) FAIL 2021/01/04 05:04:19 DEBUG : Chunked 'TestChunkerChunk50bMD5QuickS3:rclone-test-waxadug4gasufot5deyiwep0': Purge dir "" 2021/01/04 05:04:19 INFO : S3 bucket rclone-test-waxadug4gasufot5deyiwep0: Bucket "rclone-test-waxadug4gasufot5deyiwep0" deleted "./sync.test -test.v -test.timeout 1h0m0s -remote TestChunkerChunk50bMD5QuickS3: -verbose -fast-list -size-limit 1024 -test.run '^(TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs)$'" - Finished ERROR in 10.684876026s (try 2/5): exit status 1: Failed [TestServerSideMove TestServerSideMoveDeleteEmptySourceDirs]