"./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestMoveWithIgnoreExisting|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter)$'" - Starting (try 2/5) 2025/04/26 04:54:23 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-kubopit7ruto" 2025/04/26 04:54:23 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/04/26 04:54:24 DEBUG : Creating backend with remote "/tmp/rclone536725891" === RUN TestServerSideCopy run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:54:31 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-wanuxip1xaru" 2025/04/26 04:54:31 DEBUG : Config file has changed externally - reloading sync_test.go:590: Server side copy (if possible) Linkbox root 'rclone-test-kubopit7ruto' -> Linkbox root 'rclone-test-wanuxip1xaru' 2025/04/26 04:54:33 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/04/26 04:54:33 DEBUG : Linkbox root 'rclone-test-wanuxip1xaru': Waiting for checks to finish 2025/04/26 04:54:33 DEBUG : Linkbox root 'rclone-test-wanuxip1xaru': Waiting for transfers to finish 2025/04/26 04:54:33 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:594: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:594 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideCopy 2025/04/26 04:54:33 DEBUG : Linkbox root 'rclone-test-wanuxip1xaru': Purge remote 2025/04/26 04:54:33 NOTICE: purge failed: directory not found --- FAIL: TestServerSideCopy (14.43s) === RUN TestServerSideCopyOverSelf run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:54:44 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-wutowux0caxo" sync_test.go:626: Server side copy (if possible) Linkbox root 'rclone-test-kubopit7ruto' -> Linkbox root 'rclone-test-wutowux0caxo' 2025/04/26 04:54:45 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/04/26 04:54:45 DEBUG : Linkbox root 'rclone-test-wutowux0caxo': Waiting for checks to finish 2025/04/26 04:54:45 DEBUG : Linkbox root 'rclone-test-wutowux0caxo': Waiting for transfers to finish 2025/04/26 04:54:45 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:630: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:630 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideCopyOverSelf 2025/04/26 04:54:45 DEBUG : Linkbox root 'rclone-test-wutowux0caxo': Purge remote 2025/04/26 04:54:45 NOTICE: purge failed: directory not found --- FAIL: TestServerSideCopyOverSelf (11.99s) === RUN TestServerSideMoveOverSelf run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:54:56 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-mewasih4dazu" 2025/04/26 04:54:56 DEBUG : Config file has changed externally - reloading sync_test.go:671: Server side copy (if possible) Linkbox root 'rclone-test-kubopit7ruto' -> Linkbox root 'rclone-test-mewasih4dazu' 2025/04/26 04:54:57 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/04/26 04:54:57 DEBUG : Linkbox root 'rclone-test-mewasih4dazu': Waiting for checks to finish 2025/04/26 04:54:57 DEBUG : Linkbox root 'rclone-test-mewasih4dazu': Waiting for transfers to finish 2025/04/26 04:54:57 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:675: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:675 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMoveOverSelf 2025/04/26 04:54:57 DEBUG : Linkbox root 'rclone-test-mewasih4dazu': Purge remote 2025/04/26 04:54:58 NOTICE: purge failed: directory not found run.go:125: removing dir "sub dir" failed - try 1/3: directory not empty run.go:125: removing dir "sub dir" failed - try 2/3: directory not empty fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:295: Flushing the directory cache fstest.go:286: Sleeping for 7s just to make sure --- FAIL: TestServerSideMoveOverSelf (27.28s) === RUN TestCopyRedownload run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:55:23 DEBUG : Added delayed dir = "sub dir", newDst= 2025/04/26 04:55:23 DEBUG : Local file system at /tmp/rclone536725891: Waiting for checks to finish 2025/04/26 04:55:23 DEBUG : Local file system at /tmp/rclone536725891: Waiting for transfers to finish 2025/04/26 04:55:23 INFO : There was nothing to transfer run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:55:23 ERROR : : error listing: directory not found fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 2025/04/26 04:55:24 ERROR : : error listing: directory not found fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 2025/04/26 04:55:26 ERROR : : error listing: directory not found fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:299: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:299 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:350 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:736 Error: Should be true Test: TestCopyRedownload Messages: listing wrong, want sub dir/hello world (11) got fstest.go:197: Not found "sub dir/hello world" fstest.go:200: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:200 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:304 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:350 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:736 Error: Not equal: expected: 0 actual : 1 Test: TestCopyRedownload Messages: 1 objects not found run.go:125: removing dir "sub dir" failed - try 1/3: directory not empty run.go:125: removing dir "sub dir" failed - try 2/3: directory not empty fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:286: Sleeping for 6s just to make sure --- FAIL: TestCopyRedownload (29.80s) === RUN TestMoveWithIgnoreExisting run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:55:47 DEBUG : existing: Need to transfer - File not found at Destination 2025/04/26 04:55:47 DEBUG : existing-b: Need to transfer - File not found at Destination 2025/04/26 04:55:47 DEBUG : Linkbox root 'rclone-test-kubopit7ruto': Waiting for checks to finish 2025/04/26 04:55:47 DEBUG : Linkbox root 'rclone-test-kubopit7ruto': Waiting for transfers to finish 2025/04/26 04:55:51 DEBUG : existing: Trying to read object after upload: try again in 100ms (1/10) 2025/04/26 04:55:51 DEBUG : existing-b: Trying to read object after upload: try again in 100ms (1/10) 2025/04/26 04:55:51 DEBUG : existing: Trying to read object after upload: try again in 200ms (2/10) 2025/04/26 04:55:52 DEBUG : existing-b: Trying to read object after upload: try again in 200ms (2/10) 2025/04/26 04:55:52 DEBUG : existing: Trying to read object after upload: try again in 400ms (3/10) 2025/04/26 04:55:52 DEBUG : existing-b: Trying to read object after upload: try again in 400ms (3/10) 2025/04/26 04:55:53 DEBUG : existing: Trying to read object after upload: try again in 800ms (4/10) 2025/04/26 04:55:53 DEBUG : existing-b: Trying to read object after upload: try again in 800ms (4/10) 2025/04/26 04:55:54 DEBUG : existing: Trying to read object after upload: try again in 1.6s (5/10) 2025/04/26 04:55:54 INFO : existing-b: Copied (new) 2025/04/26 04:55:54 INFO : existing-b: Deleted 2025/04/26 04:55:56 INFO : existing: Copied (new) 2025/04/26 04:55:56 INFO : existing: Deleted 2025/04/26 04:55:57 DEBUG : existing: Destination exists, skipping 2025/04/26 04:55:57 DEBUG : existing: Not removing source file as destination file exists and --ignore-existing is set 2025/04/26 04:55:57 DEBUG : Linkbox root 'rclone-test-kubopit7ruto': Waiting for checks to finish 2025/04/26 04:55:57 DEBUG : Linkbox root 'rclone-test-kubopit7ruto': Waiting for transfers to finish 2025/04/26 04:55:57 INFO : There was nothing to transfer run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" --- PASS: TestMoveWithIgnoreExisting (14.97s) === RUN TestServerSideMove run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:56:02 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-gekugoj5fovo" 2025/04/26 04:56:02 DEBUG : Config file has changed externally - reloading sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-kubopit7ruto' -> Linkbox root 'rclone-test-gekugoj5fovo' 2025/04/26 04:56:20 DEBUG : potato2: Need to transfer - File not found at Destination 2025/04/26 04:56:20 DEBUG : Linkbox root 'rclone-test-gekugoj5fovo': Waiting for checks to finish 2025/04/26 04:56:20 DEBUG : empty space: Sizes identical 2025/04/26 04:56:20 DEBUG : empty space: Unchanged skipping 2025/04/26 04:56:20 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/04/26 04:56:20 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:20 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:20 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:20 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:22 INFO : empty space: Deleted 2025/04/26 04:56:22 DEBUG : Linkbox root 'rclone-test-gekugoj5fovo': Waiting for transfers to finish sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1881 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMove 2025/04/26 04:56:22 DEBUG : Linkbox root 'rclone-test-gekugoj5fovo': Purge remote --- FAIL: TestServerSideMove (25.01s) === RUN TestServerSideMoveWithFilter run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:56:27 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-makanoj4rese" sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-kubopit7ruto' -> Linkbox root 'rclone-test-makanoj4rese' 2025/04/26 04:56:45 DEBUG : empty space: Excluded (Size Filter) 2025/04/26 04:56:45 DEBUG : empty space: Excluded 2025/04/26 04:56:45 DEBUG : empty space: Excluded (Size Filter) 2025/04/26 04:56:45 DEBUG : empty space: Excluded 2025/04/26 04:56:45 DEBUG : potato2: Need to transfer - File not found at Destination 2025/04/26 04:56:45 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/04/26 04:56:45 DEBUG : Linkbox root 'rclone-test-makanoj4rese': Waiting for checks to finish 2025/04/26 04:56:45 DEBUG : Linkbox root 'rclone-test-makanoj4rese': Waiting for transfers to finish 2025/04/26 04:56:45 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:45 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:45 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:56:45 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1894 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMoveWithFilter 2025/04/26 04:56:45 DEBUG : Linkbox root 'rclone-test-makanoj4rese': Purge remote --- FAIL: TestServerSideMoveWithFilter (25.51s) === RUN TestServerSideMoveDeleteEmptySourceDirs run.go:180: Remote "Linkbox root 'rclone-test-kubopit7ruto'", Local "Local file system at /tmp/rclone536725891", Modify Window "876000h0m0s" 2025/04/26 04:56:53 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-niyogen0mepe" 2025/04/26 04:56:53 DEBUG : Config file has changed externally - reloading 2025/04/26 04:57:02 INFO : tomatoDir: Making directory sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-kubopit7ruto' -> Linkbox root 'rclone-test-niyogen0mepe' 2025/04/26 04:57:14 DEBUG : potato2: Need to transfer - File not found at Destination 2025/04/26 04:57:14 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/04/26 04:57:14 DEBUG : empty space: Sizes identical 2025/04/26 04:57:14 DEBUG : empty space: Unchanged skipping 2025/04/26 04:57:15 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:57:15 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:57:15 DEBUG : Linkbox root 'rclone-test-niyogen0mepe': Waiting for checks to finish 2025/04/26 04:57:15 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:57:15 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/04/26 04:57:16 INFO : empty space: Deleted 2025/04/26 04:57:16 DEBUG : Linkbox root 'rclone-test-niyogen0mepe': Waiting for transfers to finish 2025/04/26 04:57:16 ERROR : Linkbox root 'rclone-test-kubopit7ruto': not deleting directories as there were IO errors sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1901 Error: Received unexpected error: not deleting directories as there were IO errors Test: TestServerSideMoveDeleteEmptySourceDirs 2025/04/26 04:57:16 DEBUG : Linkbox root 'rclone-test-niyogen0mepe': Purge remote --- FAIL: TestServerSideMoveDeleteEmptySourceDirs (31.87s) FAIL 2025/04/26 04:57:24 DEBUG : Linkbox root 'rclone-test-kubopit7ruto': Purge remote "./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestMoveWithIgnoreExisting|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter)$'" - Finished ERROR in 3m3.389610036s (try 2/5): exit status 1: Failed [TestServerSideCopy TestServerSideCopyOverSelf TestServerSideMoveOverSelf TestCopyRedownload TestServerSideMove TestServerSideMoveWithFilter TestServerSideMoveDeleteEmptySourceDirs]