"./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter)$'" - Starting (try 4/5) 2025/02/18 02:44:52 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-zisatib2qimu" 2025/02/18 02:44:52 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/02/18 02:44:53 DEBUG : Creating backend with remote "/tmp/rclone1303082226" === RUN TestServerSideCopy run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:45:00 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-pacifet2vara" sync_test.go:590: Server side copy (if possible) Linkbox root 'rclone-test-zisatib2qimu' -> Linkbox root 'rclone-test-pacifet2vara' 2025/02/18 02:45:02 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/18 02:45:02 DEBUG : Linkbox root 'rclone-test-pacifet2vara': Waiting for checks to finish 2025/02/18 02:45:02 DEBUG : Linkbox root 'rclone-test-pacifet2vara': Waiting for transfers to finish 2025/02/18 02:45:02 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:594: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:594 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideCopy 2025/02/18 02:45:02 DEBUG : Linkbox root 'rclone-test-pacifet2vara': Purge remote 2025/02/18 02:45:02 NOTICE: purge failed: directory not found --- FAIL: TestServerSideCopy (13.77s) === RUN TestServerSideCopyOverSelf run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:45:12 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-jixayep5loxe" sync_test.go:626: Server side copy (if possible) Linkbox root 'rclone-test-zisatib2qimu' -> Linkbox root 'rclone-test-jixayep5loxe' 2025/02/18 02:45:13 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/18 02:45:13 DEBUG : Linkbox root 'rclone-test-jixayep5loxe': Waiting for checks to finish 2025/02/18 02:45:13 DEBUG : Linkbox root 'rclone-test-jixayep5loxe': Waiting for transfers to finish 2025/02/18 02:45:13 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:630: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:630 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideCopyOverSelf 2025/02/18 02:45:13 DEBUG : Linkbox root 'rclone-test-jixayep5loxe': Purge remote 2025/02/18 02:45:14 NOTICE: purge failed: directory not found --- FAIL: TestServerSideCopyOverSelf (11.63s) === RUN TestServerSideMoveOverSelf run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:45:23 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:45:24 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:45:24 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:45:25 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 800ms (4/10) 2025/02/18 02:45:26 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 1.6s (5/10) 2025/02/18 02:45:28 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 3.2s (6/10) 2025/02/18 02:45:32 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-fekuguv2sape" sync_test.go:671: Server side copy (if possible) Linkbox root 'rclone-test-zisatib2qimu' -> Linkbox root 'rclone-test-fekuguv2sape' 2025/02/18 02:45:34 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/18 02:45:34 DEBUG : Linkbox root 'rclone-test-fekuguv2sape': Waiting for checks to finish 2025/02/18 02:45:34 DEBUG : Linkbox root 'rclone-test-fekuguv2sape': Waiting for transfers to finish 2025/02/18 02:45:34 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:675: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:675 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMoveOverSelf 2025/02/18 02:45:34 DEBUG : Linkbox root 'rclone-test-fekuguv2sape': Purge remote 2025/02/18 02:45:34 NOTICE: purge failed: directory not found run.go:125: removing dir "sub dir" failed - try 1/3: directory not empty fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:286: Sleeping for 6s just to make sure --- FAIL: TestServerSideMoveOverSelf (30.46s) === RUN TestCopyRedownload run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:45:53 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:45:54 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:45:54 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:45:55 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 800ms (4/10) 2025/02/18 02:45:56 DEBUG : sub dir/hello world: Trying to read object after upload: try again in 1.6s (5/10) 2025/02/18 02:45:59 DEBUG : Added delayed dir = "sub dir", newDst= 2025/02/18 02:45:59 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/02/18 02:45:59 DEBUG : Local file system at /tmp/rclone1303082226: Waiting for checks to finish 2025/02/18 02:45:59 DEBUG : Local file system at /tmp/rclone1303082226: Waiting for transfers to finish 2025/02/18 02:45:59 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:45:59 ERROR : sub dir: Failed to update directory timestamp or metadata: chtimes /tmp/rclone1303082226/sub dir: no such file or directory sync_test.go:732: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:732 Error: Received unexpected error: failed to set directory modtime: chtimes /tmp/rclone1303082226/sub dir: no such file or directory Test: TestCopyRedownload run.go:125: removing dir "sub dir" failed - try 1/3: directory not empty run.go:125: removing dir "sub dir" failed - try 2/3: directory not empty run.go:125: removing dir "sub dir" failed - try 3/3: directory not empty run.go:128: removing dir "sub dir" failed: directory not empty fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:295: Flushing the directory cache fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:295: Flushing the directory cache fstest.go:321: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:321 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:332 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:173 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:388 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1175 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1353 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1683 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/panic.go:626 /home/rclone/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/testing/testing.go:1005 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:732 Error: Not equal: expected: []string{} actual : []string{"sub dir"} Diff: --- Expected +++ Actual @@ -1,2 +1,3 @@ -([]string) { +([]string) (len=1) { + (string) (len=7) "sub dir" } Test: TestCopyRedownload Messages: directories --- FAIL: TestCopyRedownload (27.11s) === RUN TestServerSideMove run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:46:16 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-kapibip7cija" 2025/02/18 02:46:20 DEBUG : potato2: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:46:20 DEBUG : potato2: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:46:21 DEBUG : potato2: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:46:21 DEBUG : potato2: Trying to read object after upload: try again in 800ms (4/10) 2025/02/18 02:46:22 DEBUG : potato2: Trying to read object after upload: try again in 1.6s (5/10) 2025/02/18 02:46:24 DEBUG : potato2: Trying to read object after upload: try again in 3.2s (6/10) 2025/02/18 02:46:31 DEBUG : empty space: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:46:31 DEBUG : empty space: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:46:32 DEBUG : empty space: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:46:33 DEBUG : empty space: Trying to read object after upload: try again in 800ms (4/10) 2025/02/18 02:46:34 DEBUG : empty space: Trying to read object after upload: try again in 1.6s (5/10) 2025/02/18 02:46:38 DEBUG : potato3: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:46:39 DEBUG : potato3: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:46:39 DEBUG : potato3: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:46:40 DEBUG : potato3: Trying to read object after upload: try again in 800ms (4/10) fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:295: Flushing the directory cache fstest.go:286: Sleeping for 6s just to make sure sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-zisatib2qimu' -> Linkbox root 'rclone-test-kapibip7cija' 2025/02/18 02:46:54 DEBUG : empty space: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:46:55 DEBUG : empty space: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:46:56 DEBUG : empty space: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:46:56 DEBUG : empty space: Trying to read object after upload: try again in 800ms (4/10) 2025/02/18 02:46:57 DEBUG : empty space: Trying to read object after upload: try again in 1.6s (5/10) 2025/02/18 02:47:02 DEBUG : potato3: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:47:03 DEBUG : potato3: Trying to read object after upload: try again in 200ms (2/10) 2025/02/18 02:47:03 DEBUG : potato3: Trying to read object after upload: try again in 400ms (3/10) 2025/02/18 02:47:04 DEBUG : potato3: Trying to read object after upload: try again in 800ms (4/10) 2025/02/18 02:47:06 DEBUG : potato2: Need to transfer - File not found at Destination 2025/02/18 02:47:06 DEBUG : empty space: Sizes identical 2025/02/18 02:47:06 DEBUG : empty space: Unchanged skipping 2025/02/18 02:47:06 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/02/18 02:47:06 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:06 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:06 DEBUG : Linkbox root 'rclone-test-kapibip7cija': Waiting for checks to finish 2025/02/18 02:47:06 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:06 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:08 INFO : empty space: Deleted 2025/02/18 02:47:08 DEBUG : Linkbox root 'rclone-test-kapibip7cija': Waiting for transfers to finish sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1881 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMove 2025/02/18 02:47:08 DEBUG : Linkbox root 'rclone-test-kapibip7cija': Purge remote --- FAIL: TestServerSideMove (59.30s) === RUN TestServerSideMoveWithFilter run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:47:15 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-fehehuw1taka" 2025/02/18 02:47:15 DEBUG : Config file has changed externally - reloading sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-zisatib2qimu' -> Linkbox root 'rclone-test-fehehuw1taka' 2025/02/18 02:47:32 DEBUG : empty space: Excluded (Size Filter) 2025/02/18 02:47:32 DEBUG : empty space: Excluded 2025/02/18 02:47:32 DEBUG : empty space: Excluded (Size Filter) 2025/02/18 02:47:32 DEBUG : empty space: Excluded 2025/02/18 02:47:32 DEBUG : potato2: Need to transfer - File not found at Destination 2025/02/18 02:47:32 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/02/18 02:47:32 DEBUG : Linkbox root 'rclone-test-fehehuw1taka': Waiting for checks to finish 2025/02/18 02:47:32 DEBUG : Linkbox root 'rclone-test-fehehuw1taka': Waiting for transfers to finish 2025/02/18 02:47:32 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:32 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:32 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:32 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1894 Error: Received unexpected error: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" Test: TestServerSideMoveWithFilter 2025/02/18 02:47:32 DEBUG : Linkbox root 'rclone-test-fehehuw1taka': Purge remote --- FAIL: TestServerSideMoveWithFilter (24.06s) === RUN TestServerSideMoveDeleteEmptySourceDirs run.go:180: Remote "Linkbox root 'rclone-test-zisatib2qimu'", Local "Local file system at /tmp/rclone1303082226", Modify Window "876000h0m0s" 2025/02/18 02:47:39 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-hekeced5tisu" 2025/02/18 02:47:48 INFO : tomatoDir: Making directory sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-zisatib2qimu' -> Linkbox root 'rclone-test-hekeced5tisu' 2025/02/18 02:47:58 DEBUG : potato3: Trying to read object after upload: try again in 100ms (1/10) 2025/02/18 02:47:59 DEBUG : potato2: Need to transfer - File not found at Destination 2025/02/18 02:47:59 DEBUG : empty space: Sizes identical 2025/02/18 02:47:59 DEBUG : empty space: Unchanged skipping 2025/02/18 02:47:59 DEBUG : potato3: Sizes differ (src 68 vs dst 60) 2025/02/18 02:47:59 DEBUG : Linkbox root 'rclone-test-hekeced5tisu': Waiting for checks to finish 2025/02/18 02:47:59 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:59 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:59 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:47:59 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect" 2025/02/18 02:48:00 INFO : empty space: Deleted 2025/02/18 02:48:00 DEBUG : Linkbox root 'rclone-test-hekeced5tisu': Waiting for transfers to finish 2025/02/18 02:48:00 ERROR : Linkbox root 'rclone-test-zisatib2qimu': not deleting directories as there were IO errors sync_test.go:1716: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1901 Error: Received unexpected error: not deleting directories as there were IO errors Test: TestServerSideMoveDeleteEmptySourceDirs 2025/02/18 02:48:00 DEBUG : Linkbox root 'rclone-test-hekeced5tisu': Purge remote --- FAIL: TestServerSideMoveDeleteEmptySourceDirs (29.03s) FAIL 2025/02/18 02:48:08 DEBUG : Linkbox root 'rclone-test-zisatib2qimu': Purge remote "./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter)$'" - Finished ERROR in 3m17.921027837s (try 4/5): exit status 1: Failed [TestServerSideCopy TestServerSideCopyOverSelf TestServerSideMoveOverSelf TestCopyRedownload TestServerSideMove TestServerSideMoveWithFilter TestServerSideMoveDeleteEmptySourceDirs]