"./operations.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^TestMultithreadCopy$/^upload=false,size=134217727,streams=2$'" - Starting (try 2/5)
2026/07/27 03:02:39 DEBUG : Creating backend with remote "TestShade:rclone-test-xuxokaz1taqi"
2026/07/27 03:02:39 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2026/07/27 03:02:39 DEBUG : Creating new ShadeFS backend with drive: cf193d00-c783-4bab-aa7a-fd84b3700d27
2026/07/27 03:02:40 DEBUG : Creating backend with remote "/tmp/rclone435787152"
=== RUN   TestMultithreadCopy
    run.go:198: Remote "Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-xuxokaz1taqi", Local "Local file system at /tmp/rclone435787152", Modify Window "876000h0m0s"
=== RUN   TestMultithreadCopy/upload=false,size=134217727,streams=2
2026/07/27 03:02:42 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 0 size 64Mi offset 0/128.000Mi
2026/07/27 03:02:42 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 1 size 64.000Mi offset 64Mi/128.000Mi
2026/07/27 03:02:46 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/27 03:02:46 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2026/07/27 03:02:47 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:47 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2026/07/27 03:02:47 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:47 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2026/07/27 03:02:47 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:47 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2026/07/27 03:02:47 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:47 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/07/27 03:02:48 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:48 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/07/27 03:02:48 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:48 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/07/27 03:02:49 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:49 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/07/27 03:02:50 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:50 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/07/27 03:02:53 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:02:53 DEBUG : pacer: Rate limited, increasing sleep to 10.24s
2026/07/27 03:02:53 DEBUG : test-multithread-copy-false-134217727-2: Cancelling multipart upload
2026/07/27 03:02:58 DEBUG : pacer: Reducing sleep to 5.12s
=== NAME  TestMultithreadCopy
    run.go:299: Retry Put of "test-multithread-copy-false-134217727-2" to Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-xuxokaz1taqi: 1/10 (multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:08 DEBUG : pacer: Reducing sleep to 2.56s
2026/07/27 03:03:08 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 0 size 64Mi offset 0/128.000Mi
2026/07/27 03:03:08 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 1 size 64.000Mi offset 64Mi/128.000Mi
2026/07/27 03:03:13 DEBUG : pacer: Reducing sleep to 1.28s
2026/07/27 03:03:15 DEBUG : pacer: Reducing sleep to 640ms
2026/07/27 03:03:19 DEBUG : pacer: Reducing sleep to 320ms
2026/07/27 03:03:19 DEBUG : pacer: Reducing sleep to 160ms
2026/07/27 03:03:20 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/27 03:03:20 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2026/07/27 03:03:21 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:21 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2026/07/27 03:03:21 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:21 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2026/07/27 03:03:22 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:22 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2026/07/27 03:03:23 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:23 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/07/27 03:03:25 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:25 DEBUG : pacer: Rate limited, increasing sleep to 10.24s
2026/07/27 03:03:30 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:30 DEBUG : pacer: Rate limited, increasing sleep to 20.48s
2026/07/27 03:03:41 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:03:41 DEBUG : pacer: Rate limited, increasing sleep to 40.96s
2026/07/27 03:04:01 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:04:01 DEBUG : pacer: Rate limited, increasing sleep to 1m21.92s
2026/07/27 03:04:42 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:04:42 DEBUG : pacer: Rate limited, increasing sleep to 2m43.84s
2026/07/27 03:04:42 DEBUG : test-multithread-copy-false-134217727-2: Cancelling multipart upload
2026/07/27 03:06:04 DEBUG : pacer: Reducing sleep to 1m21.92s
    run.go:299: Retry Put of "test-multithread-copy-false-134217727-2" to Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-xuxokaz1taqi: 2/10 (multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:08:48 DEBUG : pacer: Reducing sleep to 40.96s
2026/07/27 03:08:48 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 0 size 64Mi offset 0/128.000Mi
2026/07/27 03:08:48 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 1 size 64.000Mi offset 64Mi/128.000Mi
2026/07/27 03:10:10 DEBUG : pacer: Reducing sleep to 20.48s
2026/07/27 03:10:51 DEBUG : pacer: Reducing sleep to 10.24s
2026/07/27 03:11:13 DEBUG : pacer: Reducing sleep to 5.12s
2026/07/27 03:11:23 DEBUG : pacer: Reducing sleep to 2.56s
2026/07/27 03:11:29 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/27 03:11:29 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2026/07/27 03:11:29 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:11:29 DEBUG : pacer: Rate limited, increasing sleep to 10.24s
2026/07/27 03:11:35 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:11:35 DEBUG : pacer: Rate limited, increasing sleep to 20.48s
2026/07/27 03:11:45 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:11:45 DEBUG : pacer: Rate limited, increasing sleep to 40.96s
2026/07/27 03:12:05 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:12:05 DEBUG : pacer: Rate limited, increasing sleep to 1m21.92s
2026/07/27 03:12:46 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:12:46 DEBUG : pacer: Rate limited, increasing sleep to 2m43.84s
2026/07/27 03:14:08 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:14:08 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2026/07/27 03:16:52 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:21:52 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:26:52 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:26:52 DEBUG : test-multithread-copy-false-134217727-2: Cancelling multipart upload
2026/07/27 03:31:52 DEBUG : pacer: Reducing sleep to 2m30s
    run.go:299: Retry Put of "test-multithread-copy-false-134217727-2" to Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-xuxokaz1taqi: 3/10 (multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:36:52 DEBUG : pacer: Reducing sleep to 1m15s
2026/07/27 03:36:52 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 0 size 64Mi offset 0/128.000Mi
2026/07/27 03:36:53 DEBUG : test-multithread-copy-false-134217727-2: multipart upload: starting chunk 1 size 64.000Mi offset 64Mi/128.000Mi
2026/07/27 03:39:22 DEBUG : pacer: Reducing sleep to 37.5s
2026/07/27 03:40:37 DEBUG : pacer: Reducing sleep to 18.75s
2026/07/27 03:41:17 DEBUG : pacer: Reducing sleep to 9.375s
2026/07/27 03:41:36 DEBUG : pacer: Reducing sleep to 4.6875s
2026/07/27 03:41:44 DEBUG : pacer: low level retry 1/10 (error EOF)
2026/07/27 03:41:44 DEBUG : pacer: Rate limited, increasing sleep to 9.375s
2026/07/27 03:41:48 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:41:48 DEBUG : pacer: Rate limited, increasing sleep to 18.75s
2026/07/27 03:41:57 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:41:57 DEBUG : pacer: Rate limited, increasing sleep to 37.5s
2026/07/27 03:42:16 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:42:16 DEBUG : pacer: Rate limited, increasing sleep to 1m15s
2026/07/27 03:42:53 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:42:53 DEBUG : pacer: Rate limited, increasing sleep to 2m30s
2026/07/27 03:44:08 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:44:08 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2026/07/27 03:46:38 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "Internal Server Error")
2026/07/27 03:51:38 DEBUG : pacer: Reducing sleep to 2m30s
2026/07/27 03:51:38 DEBUG : test-multithread-copy-false-134217727-2: Cancelling multipart upload
2026/07/27 03:56:38 DEBUG : pacer: Reducing sleep to 1m15s
2026/07/27 03:56:38 DEBUG : Saving config "token" in section "TestShade" of the config file
2026/07/27 03:56:38 DEBUG : Config file has changed externally - reloading
2026/07/27 03:56:38 DEBUG : Saving config "token_expiry" in section "TestShade" of the config file
2026/07/27 03:59:08 DEBUG : pacer: Reducing sleep to 37.5s
    run.go:303: Failed to put "test-multithread-copy-false-134217727-2" to "Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-xuxokaz1taqi": multipart upload: failed to finalise: failed to complete multipart upload: HTTP error 400 (400 Bad Request) returned body: "Invalid token"
=== NAME  TestMultithreadCopy/upload=false,size=134217727,streams=2
    testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
2026/07/27 04:00:23 DEBUG : pacer: Reducing sleep to 18.75s
2026/07/27 04:01:01 DEBUG : pacer: Reducing sleep to 9.375s
2026/07/27 04:01:20 DEBUG : pacer: Reducing sleep to 4.6875s
--- FAIL: TestMultithreadCopy (3519.93s)
    --- FAIL: TestMultithreadCopy/upload=false,size=134217727,streams=2 (3387.95s)
FAIL
2026/07/27 04:01:29 DEBUG : pacer: Reducing sleep to 2.34375s
2026/07/27 04:01:29 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-xuxokaz1taqi: Purge dir ""
2026/07/27 04:01:34 DEBUG : pacer: Reducing sleep to 1.171875s
2026/07/27 04:01:36 DEBUG : pacer: Reducing sleep to 585.9375ms
"./operations.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^TestMultithreadCopy$/^upload=false,size=134217727,streams=2$'" - Finished ERROR in 58m56.659835189s (try 2/5): exit status 1: Failed [TestMultithreadCopy/upload=false,size=134217727,streams=2]