"./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestMoveWithIgnoreExisting|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter)$'" - Starting (try 2/5)
2025/04/05 06:38:54 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-royiduq8jita"
2025/04/05 06:38:54 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2025/04/05 06:38:55 DEBUG : Creating backend with remote "/tmp/rclone330271572"
=== RUN   TestServerSideCopy
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:39:03 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-lomagos6goca"
    sync_test.go:590: Server side copy (if possible) Linkbox root 'rclone-test-royiduq8jita' -> Linkbox root 'rclone-test-lomagos6goca'
2025/04/05 06:39:04 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination
2025/04/05 06:39:04 DEBUG : Linkbox root 'rclone-test-lomagos6goca': Waiting for checks to finish
2025/04/05 06:39:04 DEBUG : Linkbox root 'rclone-test-lomagos6goca': Waiting for transfers to finish
2025/04/05 06:39:05 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
    sync_test.go:594: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:594
        	Error:      	Received unexpected error:
        	            	failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
        	Test:       	TestServerSideCopy
2025/04/05 06:39:05 DEBUG : Linkbox root 'rclone-test-lomagos6goca': Purge remote
2025/04/05 06:39:05 NOTICE: purge failed: directory not found
--- FAIL: TestServerSideCopy (14.02s)
=== RUN   TestServerSideCopyOverSelf
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:39:15 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-huheguj5siba"
    sync_test.go:626: Server side copy (if possible) Linkbox root 'rclone-test-royiduq8jita' -> Linkbox root 'rclone-test-huheguj5siba'
2025/04/05 06:39:17 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination
2025/04/05 06:39:17 DEBUG : Linkbox root 'rclone-test-huheguj5siba': Waiting for checks to finish
2025/04/05 06:39:17 DEBUG : Linkbox root 'rclone-test-huheguj5siba': Waiting for transfers to finish
2025/04/05 06:39:17 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
    sync_test.go:630: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:630
        	Error:      	Received unexpected error:
        	            	failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
        	Test:       	TestServerSideCopyOverSelf
2025/04/05 06:39:17 DEBUG : Linkbox root 'rclone-test-huheguj5siba': Purge remote
2025/04/05 06:39:17 NOTICE: purge failed: directory not found
--- FAIL: TestServerSideCopyOverSelf (12.36s)
=== RUN   TestServerSideMoveOverSelf
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:39:28 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-kawegud9cuhu"
    sync_test.go:671: Server side copy (if possible) Linkbox root 'rclone-test-royiduq8jita' -> Linkbox root 'rclone-test-kawegud9cuhu'
2025/04/05 06:39:30 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination
2025/04/05 06:39:30 DEBUG : Linkbox root 'rclone-test-kawegud9cuhu': Waiting for checks to finish
2025/04/05 06:39:30 DEBUG : Linkbox root 'rclone-test-kawegud9cuhu': Waiting for transfers to finish
2025/04/05 06:39:30 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
    sync_test.go:675: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:675
        	Error:      	Received unexpected error:
        	            	failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
        	Test:       	TestServerSideMoveOverSelf
2025/04/05 06:39:30 DEBUG : Linkbox root 'rclone-test-kawegud9cuhu': Purge remote
2025/04/05 06:39:31 NOTICE: purge failed: directory not found
--- FAIL: TestServerSideMoveOverSelf (13.75s)
=== RUN   TestCopyRedownload
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:39:42 DEBUG : Added delayed dir = "sub dir", newDst=<nil>
2025/04/05 06:39:42 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination
2025/04/05 06:39:42 DEBUG : Local file system at /tmp/rclone330271572: Waiting for checks to finish
2025/04/05 06:39:42 DEBUG : Local file system at /tmp/rclone330271572: Waiting for transfers to finish
2025/04/05 06:39:42 ERROR : sub dir/hello world: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:39:42 ERROR : sub dir: Failed to update directory timestamp or metadata: chtimes /tmp/rclone330271572/sub dir: no such file or directory
    sync_test.go:732: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:732
        	Error:      	Received unexpected error:
        	            	failed to set directory modtime: chtimes /tmp/rclone330271572/sub dir: no such file or directory
        	Test:       	TestCopyRedownload
--- FAIL: TestCopyRedownload (12.15s)
=== RUN   TestMoveWithIgnoreExisting
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:39:48 DEBUG : existing: Need to transfer - File not found at Destination
2025/04/05 06:39:48 DEBUG : existing-b: Need to transfer - File not found at Destination
2025/04/05 06:39:48 DEBUG : Linkbox root 'rclone-test-royiduq8jita': Waiting for checks to finish
2025/04/05 06:39:48 DEBUG : Linkbox root 'rclone-test-royiduq8jita': Waiting for transfers to finish
2025/04/05 06:39:51 INFO  : existing-b: Copied (new)
2025/04/05 06:39:51 INFO  : existing-b: Deleted
2025/04/05 06:39:52 DEBUG : existing: Trying to read object after upload: try again in 100ms (1/10)
2025/04/05 06:39:52 DEBUG : existing: Trying to read object after upload: try again in 200ms (2/10)
2025/04/05 06:39:53 DEBUG : existing: Trying to read object after upload: try again in 400ms (3/10)
2025/04/05 06:39:54 DEBUG : existing: Trying to read object after upload: try again in 800ms (4/10)
2025/04/05 06:39:55 DEBUG : existing: Trying to read object after upload: try again in 1.6s (5/10)
2025/04/05 06:39:57 DEBUG : existing: Trying to read object after upload: try again in 3.2s (6/10)
2025/04/05 06:40:01 DEBUG : existing: Trying to read object after upload: try again in 6.4s (7/10)
2025/04/05 06:40:08 DEBUG : existing: Trying to read object after upload: try again in 12.8s (8/10)
2025/04/05 06:40:21 DEBUG : existing: Trying to read object after upload: try again in 25.6s (9/10)
2025/04/05 06:40:47 DEBUG : existing: Trying to read object after upload: try again in 51.2s (10/10)
2025/04/05 06:41:38 ERROR : existing: Failed to copy: object not found
2025/04/05 06:41:38 ERROR : existing: Not deleting source as copy failed: object not found
    sync_test.go:1835: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1835
        	Error:      	Received unexpected error:
        	            	object not found
        	Test:       	TestMoveWithIgnoreExisting
--- FAIL: TestMoveWithIgnoreExisting (114.89s)
=== RUN   TestServerSideMove
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:41:43 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-fajofob8code"
    sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-royiduq8jita' -> Linkbox root 'rclone-test-fajofob8code'
2025/04/05 06:42:02 DEBUG : potato2: Need to transfer - File not found at Destination
2025/04/05 06:42:02 DEBUG : empty space: Sizes identical
2025/04/05 06:42:02 DEBUG : empty space: Unchanged skipping
2025/04/05 06:42:02 DEBUG : Linkbox root 'rclone-test-fajofob8code': Waiting for checks to finish
2025/04/05 06:42:02 DEBUG : potato3: Sizes differ (src 68 vs dst 60)
2025/04/05 06:42:02 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:02 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:02 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:02 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:03 INFO  : empty space: Deleted
2025/04/05 06:42:03 DEBUG : Linkbox root 'rclone-test-fajofob8code': Waiting for transfers to finish
    sync_test.go:1716: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1881
        	Error:      	Received unexpected error:
        	            	failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
        	Test:       	TestServerSideMove
2025/04/05 06:42:03 DEBUG : Linkbox root 'rclone-test-fajofob8code': Purge remote
--- FAIL: TestServerSideMove (26.02s)
=== RUN   TestServerSideMoveWithFilter
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:42:09 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-niroyej3tagu"
    sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-royiduq8jita' -> Linkbox root 'rclone-test-niroyej3tagu'
2025/04/05 06:42:26 DEBUG : empty space: Excluded (Size Filter)
2025/04/05 06:42:26 DEBUG : empty space: Excluded
2025/04/05 06:42:26 DEBUG : empty space: Excluded (Size Filter)
2025/04/05 06:42:26 DEBUG : empty space: Excluded
2025/04/05 06:42:26 DEBUG : potato2: Need to transfer - File not found at Destination
2025/04/05 06:42:26 DEBUG : potato3: Sizes differ (src 68 vs dst 60)
2025/04/05 06:42:26 DEBUG : Linkbox root 'rclone-test-niroyej3tagu': Waiting for checks to finish
2025/04/05 06:42:26 DEBUG : Linkbox root 'rclone-test-niroyej3tagu': Waiting for transfers to finish
2025/04/05 06:42:26 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:26 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:27 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:27 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
    sync_test.go:1716: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1894
        	Error:      	Received unexpected error:
        	            	failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
        	Test:       	TestServerSideMoveWithFilter
2025/04/05 06:42:27 DEBUG : Linkbox root 'rclone-test-niroyej3tagu': Purge remote
--- FAIL: TestServerSideMoveWithFilter (24.84s)
=== RUN   TestServerSideMoveDeleteEmptySourceDirs
    run.go:180: Remote "Linkbox root 'rclone-test-royiduq8jita'", Local "Local file system at /tmp/rclone330271572", Modify Window "876000h0m0s"
2025/04/05 06:42:33 DEBUG : Creating backend with remote "TestLinkbox:rclone-test-foheneq5wili"
2025/04/05 06:42:42 INFO  : tomatoDir: Making directory
    sync_test.go:1705: Server side move (if possible) Linkbox root 'rclone-test-royiduq8jita' -> Linkbox root 'rclone-test-foheneq5wili'
2025/04/05 06:42:53 DEBUG : potato2: Need to transfer - File not found at Destination
2025/04/05 06:42:53 DEBUG : empty space: Sizes identical
2025/04/05 06:42:53 DEBUG : empty space: Unchanged skipping
2025/04/05 06:42:53 DEBUG : potato3: Sizes differ (src 68 vs dst 60)
2025/04/05 06:42:54 DEBUG : Linkbox root 'rclone-test-foheneq5wili': Waiting for checks to finish
2025/04/05 06:42:54 ERROR : potato2: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:54 ERROR : potato2: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:54 ERROR : potato3: Failed to copy: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:54 ERROR : potato3: Not deleting source as copy failed: failed to open source object: Open failed: HTTP error 403 (403 Forbidden) returned body: "ip incorrect"
2025/04/05 06:42:55 INFO  : empty space: Deleted
2025/04/05 06:42:55 DEBUG : Linkbox root 'rclone-test-foheneq5wili': Waiting for transfers to finish
2025/04/05 06:42:55 ERROR : Linkbox root 'rclone-test-royiduq8jita': not deleting directories as there were IO errors
    sync_test.go:1716: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1716
        	            				/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1901
        	Error:      	Received unexpected error:
        	            	not deleting directories as there were IO errors
        	Test:       	TestServerSideMoveDeleteEmptySourceDirs
2025/04/05 06:42:55 DEBUG : Linkbox root 'rclone-test-foheneq5wili': Purge remote
--- FAIL: TestServerSideMoveDeleteEmptySourceDirs (28.98s)
FAIL
2025/04/05 06:43:02 DEBUG : Linkbox root 'rclone-test-royiduq8jita': Purge remote
"./sync.test -test.v -test.timeout 1h0m0s -remote TestLinkbox: -verbose -test.run '^(TestCopyRedownload|TestMoveWithIgnoreExisting|TestServerSideCopy|TestServerSideCopyOverSelf|TestServerSideMove|TestServerSideMoveDeleteEmptySourceDirs|TestServerSideMoveOverSelf|TestServerSideMoveWithFilter)$'" - Finished ERROR in 4m9.744048077s (try 2/5): exit status 1: Failed [TestServerSideCopy TestServerSideCopyOverSelf TestServerSideMoveOverSelf TestCopyRedownload TestMoveWithIgnoreExisting TestServerSideMove TestServerSideMoveWithFilter TestServerSideMoveDeleteEmptySourceDirs]