"./sync.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^(TestServerSideMoveDeleteEmptySourceDirs|TestSyncWithExcludeAndDeleteExcluded)$'" - Starting (try 3/5)
2026/09/23 03:40:37 DEBUG : Creating backend with remote "TestShade:rclone-test-gisesuf8quwa"
2026/09/23 03:40:37 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2026/09/23 03:40:37 DEBUG : Creating new ShadeFS backend with drive: cf193d00-c783-4bab-aa7a-fd84b3700d27
2026/09/23 03:40:37 DEBUG : Creating backend with remote "/tmp/rclone2104318659"
=== RUN   TestSyncWithExcludeAndDeleteExcluded
    run.go:198: Remote "Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa", Local "Local file system at /tmp/rclone2104318659", Modify Window "876000h0m0s"
2026/09/23 03:40:38 DEBUG : potato2: multipart upload: starting chunk 0 size 60 offset 0/60
2026/09/23 03:40:40 DEBUG : empty space: multipart upload: starting chunk 0 size 1 offset 0/1
2026/09/23 03:40:43 DEBUG : enormous: multipart upload: starting chunk 0 size 100 offset 0/100
2026/09/23 03:40:45 DEBUG : enormous: Excluded (Size Filter)
2026/09/23 03:40:45 DEBUG : enormous: Excluded
2026/09/23 03:40:45 DEBUG : potato2: Excluded (Size Filter)
2026/09/23 03:40:45 DEBUG : potato2: Excluded
2026/09/23 03:40:45 DEBUG : empty space: size = 1 OK
2026/09/23 03:40:45 DEBUG : empty space: Sizes identical
2026/09/23 03:40:45 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa: Waiting for checks to finish
2026/09/23 03:40:45 DEBUG : empty space: Unchanged skipping
2026/09/23 03:40:45 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa: Waiting for transfers to finish
2026/09/23 03:40:45 DEBUG : Waiting for deletions to finish
2026/09/23 03:40:45 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:45 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/09/23 03:40:45 DEBUG : pacer: Reducing sleep to 10ms
2026/09/23 03:40:45 INFO  : potato2: Deleted
2026/09/23 03:40:45 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:45 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/09/23 03:40:45 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:45 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2026/09/23 03:40:45 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:45 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2026/09/23 03:40:46 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:46 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2026/09/23 03:40:46 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:46 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/09/23 03:40:46 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:46 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/09/23 03:40:46 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:46 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/09/23 03:40:47 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:47 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/09/23 03:40:48 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:40:48 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/09/23 03:40:48 ERROR : enormous: Couldn't delete: HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND"
2026/09/23 03:40:48 ERROR : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa: not deleting directories as there were IO errors
    sync_test.go:1434: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:1434
        	Error:      	Received unexpected error:
        	            	failed to delete 1 files
        	Test:       	TestSyncWithExcludeAndDeleteExcluded
2026/09/23 03:40:51 DEBUG : pacer: Reducing sleep to 2.56s
2026/09/23 03:40:56 DEBUG : pacer: Reducing sleep to 1.28s
2026/09/23 03:40:58 DEBUG : pacer: Reducing sleep to 640ms
2026/09/23 03:41:00 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:00 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/09/23 03:41:00 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:00 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/09/23 03:41:02 DEBUG : pacer: Reducing sleep to 1.28s
--- FAIL: TestSyncWithExcludeAndDeleteExcluded (24.18s)
=== RUN   TestServerSideMoveDeleteEmptySourceDirs
    run.go:198: Remote "Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa", Local "Local file system at /tmp/rclone2104318659", Modify Window "876000h0m0s"
2026/09/23 03:41:02 DEBUG : Creating backend with remote "TestShade:rclone-test-nicohip6jaka"
2026/09/23 03:41:02 DEBUG : Config file has changed externally - reloading
2026/09/23 03:41:02 DEBUG : Creating new ShadeFS backend with drive: cf193d00-c783-4bab-aa7a-fd84b3700d27
2026/09/23 03:41:04 DEBUG : pacer: Reducing sleep to 640ms
2026/09/23 03:41:04 DEBUG : potato2: multipart upload: starting chunk 0 size 60 offset 0/60
2026/09/23 03:41:05 DEBUG : pacer: Reducing sleep to 320ms
2026/09/23 03:41:06 DEBUG : pacer: Reducing sleep to 160ms
2026/09/23 03:41:08 DEBUG : pacer: Reducing sleep to 80ms
2026/09/23 03:41:08 DEBUG : pacer: Reducing sleep to 40ms
2026/09/23 03:41:08 DEBUG : empty space: multipart upload: starting chunk 0 size 1 offset 0/1
2026/09/23 03:41:08 DEBUG : pacer: Reducing sleep to 20ms
2026/09/23 03:41:09 DEBUG : pacer: Reducing sleep to 10ms
2026/09/23 03:41:10 DEBUG : potato3: multipart upload: starting chunk 0 size 68 offset 0/68
2026/09/23 03:41:12 INFO  : tomatoDir: Making directory
    sync_test.go:1736: Server side move (if possible) Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa -> Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka
2026/09/23 03:41:13 DEBUG : empty space: multipart upload: starting chunk 0 size 1 offset 0/1
2026/09/23 03:41:15 DEBUG : potato3: multipart upload: starting chunk 0 size 60 offset 0/60
2026/09/23 03:41:17 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Using server-side directory move
2026/09/23 03:41:17 INFO  : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Server side directory move failed - fallback to file moves: can't copy directory - destination already exists
2026/09/23 03:41:18 DEBUG : potato2: Need to transfer - File not found at Destination
2026/09/23 03:41:18 DEBUG : empty space: size = 1 OK
2026/09/23 03:41:18 DEBUG : empty space: Sizes identical
2026/09/23 03:41:18 DEBUG : empty space: Unchanged skipping
2026/09/23 03:41:18 DEBUG : potato3: size = 68 (Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa)
2026/09/23 03:41:18 DEBUG : potato3: size = 60 (Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka)
2026/09/23 03:41:18 DEBUG : potato3: Sizes differ
2026/09/23 03:41:18 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Waiting for checks to finish
2026/09/23 03:41:18 INFO  : potato2: Moved (server-side)
2026/09/23 03:41:18 INFO  : empty space: Deleted
2026/09/23 03:41:18 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Waiting for transfers to finish
2026/09/23 03:41:18 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/09/23 03:41:18 DEBUG : pacer: Reducing sleep to 10ms
2026/09/23 03:41:18 INFO  : potato3: Deleted
2026/09/23 03:41:18 INFO  : potato3: Moved (server-side)
2026/09/23 03:41:18 INFO  : tomatoDir: Removing directory
2026/09/23 03:41:18 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa: deleted 1 directories
2026/09/23 03:41:18 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/09/23 03:41:19 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:19 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2026/09/23 03:41:19 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:19 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2026/09/23 03:41:19 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:19 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2026/09/23 03:41:19 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:19 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/09/23 03:41:19 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:19 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/09/23 03:41:20 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:20 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/09/23 03:41:20 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:20 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/09/23 03:41:21 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:21 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/09/23 03:41:24 DEBUG : pacer: Reducing sleep to 2.56s
2026/09/23 03:41:29 DEBUG : pacer: Reducing sleep to 1.28s
2026/09/23 03:41:29 DEBUG : Creating backend with remote "TestShade:rclone-test-geyodoq7cici"
2026/09/23 03:41:29 DEBUG : Config file has changed externally - reloading
2026/09/23 03:41:29 DEBUG : Creating new ShadeFS backend with drive: cf193d00-c783-4bab-aa7a-fd84b3700d27
2026/09/23 03:41:29 INFO  : tomatoDir: Making directory
2026/09/23 03:41:30 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Using server-side directory move
2026/09/23 03:41:30 INFO  : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Server side directory move succeeded
2026/09/23 03:41:30 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/09/23 03:41:30 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:30 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2026/09/23 03:41:31 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:31 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2026/09/23 03:41:31 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:31 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2026/09/23 03:41:31 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:31 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/09/23 03:41:31 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:31 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/09/23 03:41:31 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:31 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/09/23 03:41:32 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:32 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/09/23 03:41:33 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:33 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/09/23 03:41:36 DEBUG : pacer: Reducing sleep to 2.56s
2026/09/23 03:41:36 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Error from List call: HTTP error 404 (404 Not Found) returned body: "ENTRY_NOT_FOUND"
2026/09/23 03:41:36 ERROR : error listing: directory not found
2026/09/23 03:41:36 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Error from List call: HTTP error 404 (404 Not Found) returned body: "ENTRY_NOT_FOUND"
2026/09/23 03:41:36 ERROR : error listing: directory not found
2026/09/23 03:41:41 DEBUG : pacer: Reducing sleep to 1.28s
2026/09/23 03:41:41 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Purge object "empty space"
2026/09/23 03:41:43 DEBUG : pacer: Reducing sleep to 640ms
2026/09/23 03:41:43 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Purge object "potato2"
2026/09/23 03:41:45 DEBUG : pacer: Reducing sleep to 320ms
2026/09/23 03:41:45 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Purge object "potato3"
2026/09/23 03:41:45 DEBUG : pacer: Reducing sleep to 160ms
2026/09/23 03:41:46 DEBUG : pacer: Reducing sleep to 80ms
2026/09/23 03:41:46 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Purge dir "tomatoDir"
2026/09/23 03:41:46 DEBUG : pacer: Reducing sleep to 40ms
2026/09/23 03:41:46 DEBUG : pacer: Reducing sleep to 20ms
2026/09/23 03:41:46 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-geyodoq7cici: Purge dir ""
2026/09/23 03:41:46 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:46 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2026/09/23 03:41:46 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:46 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2026/09/23 03:41:46 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:46 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2026/09/23 03:41:47 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:47 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/09/23 03:41:47 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:47 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/09/23 03:41:47 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:47 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/09/23 03:41:48 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND")
2026/09/23 03:41:48 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/09/23 03:41:49 DEBUG : pacer: Reducing sleep to 1.28s
2026/09/23 03:41:52 DEBUG : pacer: Reducing sleep to 640ms
2026/09/23 03:41:52 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Error from List call: HTTP error 404 (404 Not Found) returned body: "ENTRY_NOT_FOUND"
2026/09/23 03:41:52 ERROR : error listing: directory not found
2026/09/23 03:41:52 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-nicohip6jaka: Purge dir ""
2026/09/23 03:41:52 NOTICE: purge failed to rmdir "": HTTP error 404 (404 Not Found) returned body: "ENTRY_NOT_FOUND"
2026/09/23 03:41:52 NOTICE: purge failed: directory not found
2026/09/23 03:41:52 DEBUG : pacer: Reducing sleep to 640ms
2026/09/23 03:41:53 DEBUG : pacer: Reducing sleep to 320ms
--- PASS: TestServerSideMoveDeleteEmptySourceDirs (51.68s)
FAIL
2026/09/23 03:41:54 DEBUG : pacer: Reducing sleep to 160ms
2026/09/23 03:41:54 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-gisesuf8quwa: Purge dir ""
2026/09/23 03:41:54 DEBUG : pacer: Reducing sleep to 80ms
2026/09/23 03:41:54 DEBUG : pacer: Reducing sleep to 40ms
"./sync.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^(TestServerSideMoveDeleteEmptySourceDirs|TestSyncWithExcludeAndDeleteExcluded)$'" - Finished ERROR in 1m17.312495378s (try 3/5): exit status 1: Failed [TestSyncWithExcludeAndDeleteExcluded]