"./sync.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^TestCopy$'" - Starting (try 5/5)
2026/07/26 08:41:54 DEBUG : Creating backend with remote "TestShade:rclone-test-pekigev9rizu"
2026/07/26 08:41:54 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2026/07/26 08:41:54 DEBUG : Creating new ShadeFS backend with drive: cf193d00-c783-4bab-aa7a-fd84b3700d27
2026/07/26 08:41:54 DEBUG : Creating backend with remote "/tmp/rclone3665955131"
=== RUN   TestCopy
    run.go:198: Remote "Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-pekigev9rizu", Local "Local file system at /tmp/rclone3665955131", Modify Window "876000h0m0s"
2026/07/26 08:41:54 INFO  : sub dir: Set directory modification time (using DirSetModTime)
2026/07/26 08:41:54 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination
2026/07/26 08:41:54 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-pekigev9rizu: Waiting for checks to finish
2026/07/26 08:41:54 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-pekigev9rizu: Waiting for transfers to finish
2026/07/26 08:41:55 DEBUG : sub dir/hello world: multipart upload: starting chunk 0 size 11 offset 0/11
2026/07/26 08:41:56 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/26 08:41:56 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/07/26 08:41:57 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:57 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2026/07/26 08:41:57 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:57 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2026/07/26 08:41:57 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:57 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2026/07/26 08:41:57 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:57 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/07/26 08:41:58 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:58 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/07/26 08:41:58 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:58 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/07/26 08:41:59 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:41:59 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/07/26 08:42:00 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:00 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/07/26 08:42:02 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:02 DEBUG : pacer: Rate limited, increasing sleep to 10.24s
2026/07/26 08:42:02 DEBUG : sub dir/hello world: Cancelling multipart upload
2026/07/26 08:42:08 DEBUG : pacer: Reducing sleep to 5.12s
2026/07/26 08:42:08 DEBUG : sub dir/hello world: Received error: multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error" - low level retry 0/10
2026/07/26 08:42:18 DEBUG : pacer: Reducing sleep to 2.56s
2026/07/26 08:42:18 DEBUG : sub dir/hello world: multipart upload: starting chunk 0 size 11 offset 0/11
2026/07/26 08:42:23 DEBUG : pacer: Reducing sleep to 1.28s
2026/07/26 08:42:25 DEBUG : pacer: Reducing sleep to 640ms
2026/07/26 08:42:27 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/26 08:42:27 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/07/26 08:42:28 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:28 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/07/26 08:42:29 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:29 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/07/26 08:42:32 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:32 DEBUG : pacer: Rate limited, increasing sleep to 10.24s
2026/07/26 08:42:37 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:37 DEBUG : pacer: Rate limited, increasing sleep to 20.48s
2026/07/26 08:42:47 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:42:47 DEBUG : pacer: Rate limited, increasing sleep to 40.96s
2026/07/26 08:43:07 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:43:07 DEBUG : pacer: Rate limited, increasing sleep to 1m21.92s
2026/07/26 08:43:48 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:43:48 DEBUG : pacer: Rate limited, increasing sleep to 2m43.84s
2026/07/26 08:45:10 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:45:10 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2026/07/26 08:47:54 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 08:47:54 DEBUG : sub dir/hello world: Cancelling multipart upload
2026/07/26 08:52:54 DEBUG : pacer: Reducing sleep to 2m30s
2026/07/26 08:52:54 DEBUG : sub dir/hello world: Received error: multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error" - low level retry 1/10
2026/07/26 08:57:54 DEBUG : pacer: Reducing sleep to 1m15s
2026/07/26 08:57:54 DEBUG : sub dir/hello world: multipart upload: starting chunk 0 size 11 offset 0/11
2026/07/26 09:00:24 DEBUG : pacer: Reducing sleep to 37.5s
2026/07/26 09:01:39 DEBUG : pacer: Reducing sleep to 18.75s
2026/07/26 09:02:17 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/26 09:02:17 DEBUG : pacer: Rate limited, increasing sleep to 37.5s
2026/07/26 09:02:35 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:02:35 DEBUG : pacer: Rate limited, increasing sleep to 1m15s
2026/07/26 09:03:13 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:03:13 DEBUG : pacer: Rate limited, increasing sleep to 2m30s
2026/07/26 09:04:28 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:04:28 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2026/07/26 09:06:58 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:11:58 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:16:58 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:21:58 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/26 09:26:58 DEBUG : pacer: Reducing sleep to 2m30s
2026/07/26 09:26:58 DEBUG : sub dir/hello world: Cancelling multipart upload
2026/07/26 09:31:58 DEBUG : pacer: Reducing sleep to 1m15s
2026/07/26 09:31:58 DEBUG : Saving config "token" in section "TestShade" of the config file
2026/07/26 09:31:58 DEBUG : Config file has changed externally - reloading
2026/07/26 09:31:58 DEBUG : Saving config "token_expiry" in section "TestShade" of the config file
2026/07/26 09:34:28 DEBUG : pacer: Reducing sleep to 37.5s
2026/07/26 09:34:28 ERROR : sub dir/hello world: Failed to copy: multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 400 (400 Bad Request) returned body: "Invalid token"
    sync_test.go:79: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:79
        	Error:      	Received unexpected error:
        	            	multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 400 (400 Bad Request) returned body: "Invalid token"
        	Test:       	TestCopy
2026/07/26 09:35:43 DEBUG : pacer: Reducing sleep to 18.75s
2026/07/26 09:36:20 DEBUG : pacer: Reducing sleep to 9.375s
2026/07/26 09:36:39 DEBUG : pacer: Reducing sleep to 4.6875s
2026/07/26 09:36:48 DEBUG : pacer: Reducing sleep to 2.34375s
2026/07/26 09:36:53 DEBUG : pacer: Reducing sleep to 1.171875s
2026/07/26 09:36:55 DEBUG : pacer: Reducing sleep to 585.9375ms
--- FAIL: TestCopy (3301.26s)
FAIL
2026/07/26 09:36:57 DEBUG : pacer: Reducing sleep to 292.96875ms
2026/07/26 09:36:57 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-pekigev9rizu: Purge dir ""
2026/07/26 09:36:57 DEBUG : pacer: Reducing sleep to 146.484375ms
2026/07/26 09:36:57 DEBUG : pacer: Reducing sleep to 73.242187ms
"./sync.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^TestCopy$'" - Finished ERROR in 55m3.550350298s (try 5/5): exit status 1: Failed [TestCopy]