"./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter|TestSyncCompareDest|TestSyncMultipleCompareDest)$'" - Starting (try 2/5) 2025/02/15 03:16:32 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-bakijop2payu" 2025/02/15 03:16:32 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/02/15 03:16:33 DEBUG : Creating backend with remote "/tmp/rclone2264702700" === RUN TestServerSideCopy run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:16:40 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-qurecas0piti" sync_test.go:590: Server side copy (if possible) Linkbox root 'rclone-test-bakijop2payu' -> Linkbox root 'rclone-test-qurecas0piti' 2025/02/15 03:16:42 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/15 03:16:42 DEBUG : Linkbox root 'rclone-test-qurecas0piti': Waiting for checks to finish 2025/02/15 03:16:42 DEBUG : Linkbox root 'rclone-test-qurecas0piti': Waiting for transfers to finish 2025/02/15 03:16:42 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:594: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:594 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideCopy 2025/02/15 03:16:42 DEBUG : Linkbox root 'rclone-test-qurecas0piti': Purge remote 2025/02/15 03:16:42 NOTICE: purge failed: directory not found --- FAIL: TestServerSideCopy (13.23s) === RUN TestServerSideCopyOverSelf run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:16:52 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-yubibol2faka" sync_test.go:626: Server side copy (if possible) Linkbox root 'rclone-test-bakijop2payu' -> Linkbox root 'rclone-test-yubibol2faka' 2025/02/15 03:16:53 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/15 03:16:53 DEBUG : Linkbox root 'rclone-test-yubibol2faka': Waiting for checks to finish 2025/02/15 03:16:53 DEBUG : Linkbox root 'rclone-test-yubibol2faka': Waiting for transfers to finish 2025/02/15 03:16:53 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:630: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:630 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideCopyOverSelf 2025/02/15 03:16:53 DEBUG : Linkbox root 'rclone-test-yubibol2faka': Purge remote 2025/02/15 03:16:54 NOTICE: purge failed: directory not found --- FAIL: TestServerSideCopyOverSelf (11.38s) === RUN TestServerSideMoveOverSelf run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:17:04 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-pisoyir9ziqi" sync_test.go:671: Server side copy (if possible) Linkbox root 'rclone-test-bakijop2payu' -> Linkbox root 'rclone-test-pisoyir9ziqi' 2025/02/15 03:17:05 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/15 03:17:05 DEBUG : Linkbox root 'rclone-test-pisoyir9ziqi': Waiting for checks to finish 2025/02/15 03:17:05 DEBUG : Linkbox root 'rclone-test-pisoyir9ziqi': Waiting for transfers to finish 2025/02/15 03:17:05 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:675: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:675 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMoveOverSelf 2025/02/15 03:17:05 DEBUG : Linkbox root 'rclone-test-pisoyir9ziqi': Purge remote 2025/02/15 03:17:06 NOTICE: purge failed: directory not found --- FAIL: TestServerSideMoveOverSelf (12.17s) === RUN TestCopyRedownload run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:17:16 DEBUG : Added delayed dir = "sub dir", newDst= 2025/02/15 03:17:16 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/15 03:17:16 DEBUG : Local file system at /tmp/rclone2264702700: Waiting for checks to finish 2025/02/15 03:17:16 DEBUG : Local file system at /tmp/rclone2264702700: Waiting for transfers to finish 2025/02/15 03:17:16 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:17:16 ERROR : sub dir: Failed to update directory timestamp or metadata: chtimes /tmp/rclone2264702700/sub dir: no such file or directory sync_test.go:732: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:732 Error: Received unexpected error: failed to set directory modtime: chtimes /tmp/rclone2264702700/sub dir: no such file or directory Test: TestCopyRedownload --- FAIL: TestCopyRedownload (11.09s) === RUN TestServerSideMove run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:17:21 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-tuxifuv0yoso" sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-bakijop2payu' -> Linkbox root 'rclone-test-tuxifuv0yoso' 2025/02/15 03:17:39 DEBUG : potato2: Need to transfer - File not found at Destination 2025/02/15 03:17:39 DEBUG : empty space: Sizes identical 2025/02/15 03:17:39 DEBUG : empty space: Unchanged skipping 2025/02/15 03:17:39 DEBUG : Linkbox root 'rclone-test-tuxifuv0yoso': Waiting for checks to finish 2025/02/15 03:17:39 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/02/15 03:17:39 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:17:39 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:17:39 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:17:39 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:17:40 INFO : empty space: Deleted 2025/02/15 03:17:40 DEBUG : Linkbox root 'rclone-test-tuxifuv0yoso': Waiting for transfers to finish sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1881 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMove 2025/02/15 03:17:40 DEBUG : Linkbox root 'rclone-test-tuxifuv0yoso': Purge remote --- FAIL: TestServerSideMove (24.31s) === RUN TestServerSideMoveWithFilter run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:17:45 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-mamegec3gica" sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-bakijop2payu' -> Linkbox root 'rclone-test-mamegec3gica' 2025/02/15 03:18:02 DEBUG : empty space: Excluded (Size Filter) 2025/02/15 03:18:02 DEBUG : empty space: Excluded 2025/02/15 03:18:02 DEBUG : empty space: Excluded (Size Filter) 2025/02/15 03:18:02 DEBUG : empty space: Excluded 2025/02/15 03:18:02 DEBUG : potato2: Need to transfer - File not found at Destination 2025/02/15 03:18:02 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/02/15 03:18:02 DEBUG : Linkbox root 'rclone-test-mamegec3gica': Waiting for checks to finish 2025/02/15 03:18:02 DEBUG : Linkbox root 'rclone-test-mamegec3gica': Waiting for transfers to finish 2025/02/15 03:18:02 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:02 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:03 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:03 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1894 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMoveWithFilter 2025/02/15 03:18:03 DEBUG : Linkbox root 'rclone-test-mamegec3gica': Purge remote --- FAIL: TestServerSideMoveWithFilter (24.01s) === RUN TestServerSideMoveDeleteEmptySourceDirs run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:18:09 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-tozavew1digo" 2025/02/15 03:18:18 INFO : tomatoDir: Making directory sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-bakijop2payu' -> Linkbox root 'rclone-test-tozavew1digo' 2025/02/15 03:18:29 DEBUG : potato2: Need to transfer - File not found at Destination 2025/02/15 03:18:29 DEBUG : empty space: Sizes identical 2025/02/15 03:18:29 DEBUG : empty space: Unchanged skipping 2025/02/15 03:18:29 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/02/15 03:18:29 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:29 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:29 DEBUG : Linkbox root 'rclone-test-tozavew1digo': Waiting for checks to finish 2025/02/15 03:18:30 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:30 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/15 03:18:31 INFO : empty space: Deleted 2025/02/15 03:18:31 DEBUG : Linkbox root 'rclone-test-tozavew1digo': Waiting for transfers to finish 2025/02/15 03:18:31 ERROR : Linkbox root 'rclone-test-bakijop2payu': not deleting directories as there were IO errors sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1901 Error: Received unexpected error: not deleting directories as there were IO errors Test: TestServerSideMoveDeleteEmptySourceDirs 2025/02/15 03:18:31 DEBUG : Linkbox root 'rclone-test-tozavew1digo': Purge remote --- FAIL: TestServerSideMoveDeleteEmptySourceDirs (29.12s) === RUN TestSyncCompareDest run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:18:39 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-bakijop2payu/dst" 2025/02/15 03:18:40 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-bakijop2payu/CompareDest" 2025/02/15 03:18:42 DEBUG : one: Need to transfer - File not found at Destination 2025/02/15 03:18:42 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for checks to finish 2025/02/15 03:18:42 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for transfers to finish 2025/02/15 03:18:47 INFO : one: Copied (new) 2025/02/15 03:18:47 DEBUG : Waiting for deletions to finish 2025/02/15 03:18:49 DEBUG : one: Sizes differ (src 5 vs dst 3) 2025/02/15 03:18:49 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for checks to finish 2025/02/15 03:18:49 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for transfers to finish 2025/02/15 03:18:49 DEBUG : one: Update: removing old file 2025/02/15 03:18:53 INFO : one: Copied (replaced existing) 2025/02/15 03:18:53 DEBUG : Waiting for deletions to finish 2025/02/15 03:18:54 DEBUG : dst/one: Update: removing old file 2025/02/15 03:19:04 DEBUG : one: Sizes differ (src 5 vs dst 3) 2025/02/15 03:19:04 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for checks to finish 2025/02/15 03:19:05 DEBUG : one: Sizes identical 2025/02/15 03:19:05 DEBUG : one: Destination found in --compare-dest, skipping 2025/02/15 03:19:05 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for transfers to finish 2025/02/15 03:19:05 DEBUG : Waiting for deletions to finish 2025/02/15 03:19:05 INFO : There was nothing to transfer 2025/02/15 03:19:10 DEBUG : two: Sizes identical 2025/02/15 03:19:10 DEBUG : two: Destination found in --compare-dest, skipping 2025/02/15 03:19:10 DEBUG : one: Sizes differ (src 5 vs dst 3) 2025/02/15 03:19:10 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for checks to finish 2025/02/15 03:19:10 DEBUG : one: Sizes identical 2025/02/15 03:19:10 DEBUG : one: Destination found in --compare-dest, skipping 2025/02/15 03:19:10 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for transfers to finish 2025/02/15 03:19:10 DEBUG : Waiting for deletions to finish 2025/02/15 03:19:10 INFO : There was nothing to transfer 2025/02/15 03:19:12 DEBUG : two: Sizes identical 2025/02/15 03:19:12 DEBUG : two: Destination found in --compare-dest, skipping 2025/02/15 03:19:12 DEBUG : one: Sizes differ (src 5 vs dst 3) 2025/02/15 03:19:12 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for checks to finish 2025/02/15 03:19:12 DEBUG : one: Sizes identical 2025/02/15 03:19:12 DEBUG : one: Destination found in --compare-dest, skipping 2025/02/15 03:19:12 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for transfers to finish 2025/02/15 03:19:12 DEBUG : Waiting for deletions to finish 2025/02/15 03:19:12 INFO : There was nothing to transfer sync_test.go:2175: No hash on uploaded file so skipping compare timestamp test 2025/02/15 03:19:15 DEBUG : two: Sizes differ (src 5 vs dst 3) 2025/02/15 03:19:15 DEBUG : two: Need to transfer - File not found at Destination 2025/02/15 03:19:15 DEBUG : one: Sizes differ (src 5 vs dst 3) 2025/02/15 03:19:15 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for checks to finish 2025/02/15 03:19:15 DEBUG : one: Sizes identical 2025/02/15 03:19:15 DEBUG : one: Destination found in --compare-dest, skipping 2025/02/15 03:19:15 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dst': Waiting for transfers to finish 2025/02/15 03:19:17 INFO : two: Copied (new) 2025/02/15 03:19:17 DEBUG : Waiting for deletions to finish --- PASS: TestSyncCompareDest (50.87s) === RUN TestSyncMultipleCompareDest run.go:180: Remote "Linkbox root 'rclone-test-bakijop2payu'", Local "Local file system at /tmp/rclone2264702700", Modify Window "876000h0m0s" 2025/02/15 03:19:41 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-bakijop2payu/dest" 2025/02/15 03:19:42 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-bakijop2payu/pre-dest1" 2025/02/15 03:19:43 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-bakijop2payu/pre-dest2" 2025/02/15 03:19:45 DEBUG : 1: Sizes identical 2025/02/15 03:19:45 DEBUG : 1: Destination found in --compare-dest, skipping 2025/02/15 03:19:45 DEBUG : 2: Sizes identical 2025/02/15 03:19:45 DEBUG : 2: Destination found in --compare-dest, skipping 2025/02/15 03:19:46 DEBUG : 3: Need to transfer - File not found at Destination 2025/02/15 03:19:46 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dest': Waiting for checks to finish 2025/02/15 03:19:46 DEBUG : Linkbox root 'rclone-test-bakijop2payu/dest': Waiting for transfers to finish 2025/02/15 03:19:50 INFO : 3: Copied (new) 2025/02/15 03:19:50 DEBUG : Waiting for deletions to finish --- PASS: TestSyncMultipleCompareDest (34.15s) FAIL 2025/02/15 03:20:04 DEBUG : Linkbox root 'rclone-test-bakijop2payu': Purge remote "./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter|TestSyncCompareDest|TestSyncMultipleCompareDest)$'" - Finished ERROR in 3m32.895308276s (try 2/5): exit status 1: Failed [TestServerSideCopy TestServerSideCopyOverSelf TestServerSideMoveOverSelf TestCopyRedownload TestServerSideMove TestServerSideMoveWithFilter TestServerSideMoveDeleteEmptySourceDirs]