"go test -v -timeout 1h0m0s -remote TestWebdavInfiniteScale: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1.049Mi#01$/^(1100010|1100011|2200023)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1.049Mi$/^(1099999|1100000|2200001)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1.333Mi$/^(1398100|1398101|2796203)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1Mi$/^(1048575|1048576|2097153)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^2Mi$/^(2097151|2097152|4194305)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^(FsCopy|FsDirMove|FsListDirFile2|FsListFile1|FsListFile1and2|FsListLevel2|FsListR|FsMove|FsPurge|FsPurgeRoot|ObjectOpen|ObjectOpenRange|ObjectOpenSeek|ObjectPartialRead|ObjectRemove|ObjectSetModTime|ObjectUpdate)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^(List|ListEntries)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^Put$/^Remove$|^TestIntegration$/^FsMkdir$/^FsPutZeroLength$'" - Starting (try 3/5) === RUN TestIntegration fstests.go:438: Using remote "TestWebdavInfiniteScale:" 2025/11/10 03:45:01 NOTICE: TestWebdavInfiniteScale: Starting server 2025/11/10 03:45:03 NOTICE: TestWebdavInfiniteScale: Attempting to connect to "127.0.0.1:28639" try 1/100 2025/11/10 03:45:04 DEBUG : TestWebdavInfiniteScale: Read 0, error: read tcp 127.0.0.1:58480->127.0.0.1:28639: i/o timeout 2025/11/10 03:45:04 NOTICE: TestWebdavInfiniteScale: Connect delay 5s 2025/11/10 03:45:09 NOTICE: TestWebdavInfiniteScale: Started server and connected to "127.0.0.1:28639" 2025/11/10 03:45:09 DEBUG : Creating backend with remote "TestWebdavInfiniteScale:rclone-test-kowosoy2vewu" 2025/11/10 03:45:09 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/11/10 03:45:09 DEBUG : Setting type="webdav" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_TYPE 2025/11/10 03:45:09 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:45:09 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:45:09 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:45:09 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:45:09 DEBUG : TestWebdavInfiniteScale: detected overridden config - adding "{BOGkL}" suffix to name 2025/11/10 03:45:09 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:45:09 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:45:09 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:45:09 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:45:09 DEBUG : found headers: 2025/11/10 03:45:09 DEBUG : Setting type="webdav" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_TYPE === RUN TestIntegration/FsMkdir === RUN TestIntegration/FsMkdir/FsPutZeroLength 2025/11/10 03:45:09 DEBUG : zero-length-file: Update will use the tus protocol to upload 2025/11/10 03:45:09 INFO : zero-length-file: Failed to read metadata: object not found 2025/11/10 03:45:09 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:45:09 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:253 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestIntegration/FsMkdir/FsPutZeroLength Messages: zero-length-file: Modification time difference too big |-217103h40m2.500000001s| > 1s (want 2001-02-03 04:05:06.499999999 +0000 UTC vs got 2025-11-10 03:45:09 +0000 GMT) (precision 1s) fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:257 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestIntegration/FsMkdir/FsPutZeroLength Messages: zero-length-file: Modification time difference too big |-217103h40m2.500000001s| > 1s (want 2001-02-03 04:05:06.499999999 +0000 UTC vs got 2025-11-10 03:45:09 +0000 GMT) (precision 1s) 2025/11/10 03:45:09 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:45:09 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/11/10 03:45:09 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:45:09 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/11/10 03:45:10 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:45:10 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/11/10 03:45:10 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:45:10 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/11/10 03:45:10 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:45:10 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/11/10 03:45:10 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:45:10 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/11/10 03:45:10 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:45:10 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/11/10 03:45:11 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:45:11 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:45:12 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:45:14 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutZeroLength === RUN TestIntegration/FsMkdir/FsPutFiles 2025/11/10 03:45:16 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:45:16 DEBUG : file name.txt: Update will use the tus protocol to upload 2025/11/10 03:45:18 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:45:18 INFO : file name.txt: Failed to read metadata: object not found 2025/11/10 03:45:20 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:45:20 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:45:21 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:45:21 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 100 2025/11/10 03:45:21 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:45:22 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:45:22 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:45:23 DEBUG : pacer: Reducing sleep to 266.967773ms 2025/11/10 03:45:23 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:45:23 DEBUG : pacer: Reducing sleep to 150.169371ms 2025/11/10 03:45:23 DEBUG : pacer: Reducing sleep to 112.627028ms 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 84.470271ms 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 63.352703ms 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 47.514527ms 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 35.635895ms 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 26.726921ms 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 20.04519ms 2025/11/10 03:45:24 DEBUG : hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt: Update will use the tus protocol to upload 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 15.033892ms 2025/11/10 03:45:24 INFO : hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt: Failed to read metadata: object not found 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 11.275419ms 2025/11/10 03:45:24 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:45:24 DEBUG : pacer: Reducing sleep to 10ms 2025/11/10 03:45:24 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 100 2025/11/10 03:45:24 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:45:24 DEBUG : file name.txt: Update will use the tus protocol to upload 2025/11/10 03:45:25 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:45:25 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 100 2025/11/10 03:45:25 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished === RUN TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 fstests.go:979: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:979 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:998 Error: Not equal: expected: []string{"file name.txt"} actual : []string{"file name.txt", "zero-length-file"} Diff: --- Expected +++ Actual @@ -1,3 +1,4 @@ -([]string) (len=1) { - (string) (len=13) "file name.txt" +([]string) (len=2) { + (string) (len=13) "file name.txt", + (string) (len=16) "zero-length-file" } Test: TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListR fstests.go:1021: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1021 Error: Not equal: expected: []string{"file name.txt", "hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt"} actual : []string{"file name.txt", "hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt", "zero-length-file"} Diff: --- Expected +++ Actual @@ -1,4 +1,5 @@ -([]string) (len=2) { +([]string) (len=3) { (string) (len=13) "file name.txt", - (string) (len=60) "hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt" + (string) (len=60) "hello? sausage/êé/Hello, 世界/ \" ' @ < > & ? + ≠/z.txt", + (string) (len=16) "zero-length-file" } Test: TestIntegration/FsMkdir/FsPutFiles/FsListR === RUN TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 fstests.go:1093: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1093 Error: Not equal: expected: []string{"file name.txt"} actual : []string{"file name.txt", "zero-length-file"} Diff: --- Expected +++ Actual @@ -1,3 +1,4 @@ -([]string) (len=1) { - (string) (len=13) "file name.txt" +([]string) (len=2) { + (string) (len=13) "file name.txt", + (string) (len=16) "zero-length-file" } Test: TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 === RUN TestIntegration/FsMkdir/FsPutFiles/FsListFile1 fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1107 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsListFile1 Messages: listing wrong, want file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1107 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsListFile1 Messages: Unexpected file "zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1134 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 Messages: listing wrong, want file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1134 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 Messages: Unexpected file "zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FsPurge 2025/11/10 03:45:42 DEBUG : dirToPurge/fileToPurge.txt: Update will use the tus protocol to upload 2025/11/10 03:45:43 INFO : dirToPurge/fileToPurge.txt: Failed to read metadata: object not found 2025/11/10 03:45:43 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:45:43 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 100 2025/11/10 03:45:43 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1163 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurge Messages: listing wrong, want dirToPurge/fileToPurge.txt (100), file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got dirToPurge/fileToPurge.txt (100), file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1163 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurge Messages: Unexpected file "zero-length-file" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1175 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurge Messages: listing wrong, want file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1175 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurge Messages: Unexpected file "zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot 2025/11/10 03:45:59 DEBUG : dirToPurgeFromRoot/fileToPurgeFromRoot.txt: Update will use the tus protocol to upload 2025/11/10 03:45:59 INFO : dirToPurgeFromRoot/fileToPurgeFromRoot.txt: Failed to read metadata: object not found 2025/11/10 03:45:59 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:45:59 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 100 2025/11/10 03:45:59 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1200 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot Messages: listing wrong, want dirToPurgeFromRoot/fileToPurgeFromRoot.txt (100), file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got dirToPurgeFromRoot/fileToPurgeFromRoot.txt (100), file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1200 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot Messages: Unexpected file "zero-length-file" 2025/11/10 03:46:08 DEBUG : Creating backend with remote "TestWebdavInfiniteScale:rclone-test-kowosoy2vewu/dirToPurgeFromRoot" 2025/11/10 03:46:08 DEBUG : Setting type="webdav" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_TYPE 2025/11/10 03:46:08 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:46:08 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:46:08 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:46:08 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:46:08 DEBUG : TestWebdavInfiniteScale: detected overridden config - adding "{BOGkL}" suffix to name 2025/11/10 03:46:08 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:46:08 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:46:08 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:46:08 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:46:08 DEBUG : found headers: fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1217 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot Messages: listing wrong, want file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1217 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot Messages: Unexpected file "zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FsCopy 2025/11/10 03:46:16 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:46:16 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/11/10 03:46:17 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:46:17 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/11/10 03:46:17 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:46:17 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/11/10 03:46:17 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:46:17 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/11/10 03:46:17 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:46:17 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/11/10 03:46:17 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:46:17 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/11/10 03:46:17 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:46:17 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/11/10 03:46:18 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:46:18 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:46:19 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:46:21 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:1267: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1267 Error: Received unexpected error: copy call failed: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutFiles/FsCopy Messages: Error: &fmt.wrapError{msg:"copy call failed: 500 Internal Server Error", err:fserrors.wrappedRetryError{error:(*api.Error)(0xc000612400)}} === RUN TestIntegration/FsMkdir/FsPutFiles/FsMove 2025/11/10 03:46:23 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:46:25 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:46:27 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:46:28 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:46:29 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:46:29 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:46:30 DEBUG : pacer: Reducing sleep to 266.967773ms 2025/11/10 03:46:30 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:46:30 DEBUG : pacer: Reducing sleep to 150.169371ms fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 2025/11/10 03:46:32 DEBUG : pacer: Reducing sleep to 112.627028ms 2025/11/10 03:46:32 DEBUG : pacer: Reducing sleep to 84.470271ms 2025/11/10 03:46:32 DEBUG : pacer: Reducing sleep to 63.352703ms 2025/11/10 03:46:32 DEBUG : pacer: Reducing sleep to 47.514527ms 2025/11/10 03:46:32 DEBUG : pacer: Reducing sleep to 35.635895ms fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 2025/11/10 03:46:34 DEBUG : pacer: Reducing sleep to 26.726921ms 2025/11/10 03:46:34 DEBUG : pacer: Reducing sleep to 20.04519ms 2025/11/10 03:46:34 DEBUG : pacer: Reducing sleep to 15.033892ms 2025/11/10 03:46:34 DEBUG : pacer: Reducing sleep to 11.275419ms 2025/11/10 03:46:34 DEBUG : pacer: Reducing sleep to 10ms fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1362 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: listing wrong, want file name.txt (100), other.txt (100) got file name.txt (100), other.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1362 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: Unexpected file "zero-length-file" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1373 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: listing wrong, want moveTest/other.txt (100), other.txt (100) got moveTest/other.txt (100), other.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1373 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: Unexpected file "zero-length-file" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1381 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: listing wrong, want file name.txt (100), other.txt (100) got file name.txt (100), other.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1381 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: Unexpected file "zero-length-file" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1388 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: listing wrong, want file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1388 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsMove Messages: Unexpected file "zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FsMove/Metadata fstests.go:1398: Skipping test as can't write metadata === RUN TestIntegration/FsMkdir/FsPutFiles/FsDirMove 2025/11/10 03:47:05 DEBUG : Creating backend with remote "TestWebdavInfiniteScale:rclone-test-xeyobex3dote" 2025/11/10 03:47:05 DEBUG : Config file has changed externally - reloading 2025/11/10 03:47:05 DEBUG : Setting type="webdav" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_TYPE 2025/11/10 03:47:05 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:47:05 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:47:05 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:47:05 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:47:05 DEBUG : TestWebdavInfiniteScale: detected overridden config - adding "{BOGkL}" suffix to name 2025/11/10 03:47:05 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:47:05 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:47:05 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:47:05 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:47:05 DEBUG : found headers: fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1485 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsDirMove Messages: listing wrong, want new_name/sub_new_name/file name.txt (100), new_name/sub_new_name/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got new_name/sub_new_name/file name.txt (100), new_name/sub_new_name/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), new_name/sub_new_name/zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1485 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsDirMove Messages: Unexpected file "new_name/sub_new_name/zero-length-file" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1499 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsDirMove Messages: listing wrong, want file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt (100), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1499 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FsDirMove Messages: Unexpected file "zero-length-file" 2025/11/10 03:47:23 DEBUG : webdav root 'rclone-test-xeyobex3dote': Purge remote === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectSetModTime fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1580 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1758 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectSetModTime Messages: file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectOpen 2025/11/10 03:47:23 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:47:23 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/11/10 03:47:24 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:47:24 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/11/10 03:47:24 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:47:24 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/11/10 03:47:24 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:47:24 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/11/10 03:47:24 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:47:24 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/11/10 03:47:24 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:47:24 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/11/10 03:47:24 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:47:24 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/11/10 03:47:25 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:47:25 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:47:26 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:47:28 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:287: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:287 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1772 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutFiles/ObjectOpen Messages: readObject("file name.txt") limit=-1, options=[] === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek 2025/11/10 03:47:30 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:47:32 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:47:32 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:47:34 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:47:36 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:47:38 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:47:40 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:47:42 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:47:44 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:47:46 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:47:48 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:47:50 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:287: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:287 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1779 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek Messages: readObject("file name.txt") limit=-1, options=[SeekOption(50)] === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectOpenRange 2025/11/10 03:47:52 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:47:54 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:47:54 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:47:55 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:47:57 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:47:59 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:48:01 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:48:03 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:48:05 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:48:07 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:48:09 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:48:11 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:287: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:287 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1798 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutFiles/ObjectOpenRange Messages: readObject("file name.txt") limit=-1, options=[RangeOption(5,15)] === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectPartialRead 2025/11/10 03:48:13 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:48:15 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:48:15 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:48:17 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:48:19 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:48:21 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:48:23 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:48:25 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:48:27 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:48:29 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:48:31 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:48:33 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:287: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:287 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1809 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutFiles/ObjectPartialRead Messages: readObject("file name.txt") limit=50, options=[] === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate 2025/11/10 03:48:35 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:48:37 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:48:37 DEBUG : file name.txt-should-be-ignored.bin: Update will use the tus protocol to upload 2025/11/10 03:48:38 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:48:38 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:48:39 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:48:39 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 200 2025/11/10 03:48:39 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:48:40 DEBUG : pacer: Reducing sleep to 474.609375ms fstest.go:143: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:149 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1833 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: webdav root 'rclone-test-kowosoy2vewu'/file name.txt: sha1 hash incorrect - expecting "89c8bcb196fbc4dc9305df7eb10c30241b4c6537" got "7623f40b821628e10e191cf080357709228e09d2" fstest.go:150: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:150 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1833 Error: Not equal: expected: 200 actual : 100 Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: file name.txt: size incorrect file=200 vs obj=100 fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1833 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) 2025/11/10 03:48:41 DEBUG : pacer: Reducing sleep to 355.957031ms fstest.go:143: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:149 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1837 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: webdav root 'rclone-test-kowosoy2vewu'/file name.txt: sha1 hash incorrect - expecting "89c8bcb196fbc4dc9305df7eb10c30241b4c6537" got "7623f40b821628e10e191cf080357709228e09d2" fstest.go:150: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:150 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1837 Error: Not equal: expected: 200 actual : 100 Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: file name.txt: size incorrect file=200 vs obj=100 fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1837 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) 2025/11/10 03:48:41 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:48:41 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:48:42 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:48:42 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:48:42 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:48:42 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:48:44 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:48:46 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:48:48 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:48:50 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:48:52 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:48:54 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:48:56 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:287: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:287 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1840 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate Messages: readObject("file name.txt") limit=-1, options=[] === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot fstests.go:1911: Opening root remote "TestWebdavInfiniteScale:" path "rclone-test-kowosoy2vewu" from "TestWebdavInfiniteScale:rclone-test-kowosoy2vewu" 2025/11/10 03:48:56 DEBUG : Creating backend with remote "TestWebdavInfiniteScale:" 2025/11/10 03:48:56 DEBUG : Setting type="webdav" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_TYPE 2025/11/10 03:48:56 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:48:56 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:48:56 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:48:56 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:48:56 DEBUG : TestWebdavInfiniteScale: detected overridden config - adding "{BOGkL}" suffix to name 2025/11/10 03:48:56 DEBUG : Setting url="https://127.0.0.1:28639/dav/spaces/some-admin-user-id-0000-100000000000" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_URL 2025/11/10 03:48:56 DEBUG : Setting vendor="infinitescale" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_VENDOR 2025/11/10 03:48:56 DEBUG : Setting user="admin" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_USER 2025/11/10 03:48:56 DEBUG : Setting pass="s1XACOzYfBJSIJpCO1gQdySCkV1E" for "TestWebdavInfiniteScale" from environment variable RCLONE_CONFIG_TESTWEBDAVINFINITESCALE_PASS 2025/11/10 03:48:56 DEBUG : found headers: === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/List fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1931 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List Messages: listing wrong, want rclone-test-kowosoy2vewu/file name.txt (200), rclone-test-kowosoy2vewu/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got rclone-test-kowosoy2vewu/file name.txt (100), rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin (200), rclone-test-kowosoy2vewu/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), rclone-test-kowosoy2vewu/zero-length-file (0) fstest.go:143: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:149 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1931 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List Messages: webdav root ''/rclone-test-kowosoy2vewu/file name.txt: sha1 hash incorrect - expecting "89c8bcb196fbc4dc9305df7eb10c30241b4c6537" got "7623f40b821628e10e191cf080357709228e09d2" fstest.go:150: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:150 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1931 Error: Not equal: expected: 200 actual : 100 Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List Messages: rclone-test-kowosoy2vewu/file name.txt: size incorrect file=200 vs obj=100 fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1931 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List Messages: rclone-test-kowosoy2vewu/file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1931 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List Messages: Unexpected file "rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1931 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List Messages: Unexpected file "rclone-test-kowosoy2vewu/zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries fstest.go:380: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:380 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1938 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries Messages: ListEntries not equal, want rclone-test-kowosoy2vewu/file name.txt (200) got rclone-test-kowosoy2vewu/file name.txt (100), rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin (200), rclone-test-kowosoy2vewu/zero-length-file (0) fstest.go:143: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:149 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:383 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1938 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries Messages: webdav root ''/rclone-test-kowosoy2vewu/file name.txt: sha1 hash incorrect - expecting "89c8bcb196fbc4dc9305df7eb10c30241b4c6537" got "7623f40b821628e10e191cf080357709228e09d2" fstest.go:150: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:150 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:383 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1938 Error: Not equal: expected: 200 actual : 100 Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries Messages: rclone-test-kowosoy2vewu/file name.txt: size incorrect file=200 vs obj=100 fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:383 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1938 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries Messages: rclone-test-kowosoy2vewu/file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:383 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1938 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries Messages: Unexpected file "rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:383 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1938 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries Messages: Unexpected file "rclone-test-kowosoy2vewu/zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put 2025/11/10 03:49:05 DEBUG : rclone-test-kowosoy2vewu/created from root.txt: Update will use the tus protocol to upload 2025/11/10 03:49:05 INFO : rclone-test-kowosoy2vewu/created from root.txt: Failed to read metadata: object not found 2025/11/10 03:49:05 DEBUG : webdav root '': Uploaded starts 2025/11/10 03:49:05 DEBUG : webdav root '': Uploaded chunk no 1 ok, range 0 -> 100 2025/11/10 03:49:05 DEBUG : webdav root '': -- Uploaded finished fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1987 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put Messages: listing wrong, want rclone-test-kowosoy2vewu/created from root.txt (100), rclone-test-kowosoy2vewu/file name.txt (200), rclone-test-kowosoy2vewu/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got rclone-test-kowosoy2vewu/created from root.txt (100), rclone-test-kowosoy2vewu/file name.txt (100), rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin (200), rclone-test-kowosoy2vewu/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), rclone-test-kowosoy2vewu/zero-length-file (0) fstest.go:143: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:149 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1987 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put Messages: webdav root ''/rclone-test-kowosoy2vewu/file name.txt: sha1 hash incorrect - expecting "89c8bcb196fbc4dc9305df7eb10c30241b4c6537" got "7623f40b821628e10e191cf080357709228e09d2" fstest.go:150: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:150 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1987 Error: Not equal: expected: 200 actual : 100 Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put Messages: rclone-test-kowosoy2vewu/file name.txt: size incorrect file=200 vs obj=100 fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1987 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put Messages: rclone-test-kowosoy2vewu/file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1987 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put Messages: Unexpected file "rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1987 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put Messages: Unexpected file "rclone-test-kowosoy2vewu/zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1992 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove Messages: listing wrong, want rclone-test-kowosoy2vewu/file name.txt (200), rclone-test-kowosoy2vewu/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got rclone-test-kowosoy2vewu/file name.txt (100), rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin (200), rclone-test-kowosoy2vewu/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), rclone-test-kowosoy2vewu/zero-length-file (0) fstest.go:143: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:143 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:149 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1992 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove Messages: webdav root ''/rclone-test-kowosoy2vewu/file name.txt: sha1 hash incorrect - expecting "89c8bcb196fbc4dc9305df7eb10c30241b4c6537" got "7623f40b821628e10e191cf080357709228e09d2" fstest.go:150: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:150 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1992 Error: Not equal: expected: 200 actual : 100 Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove Messages: rclone-test-kowosoy2vewu/file name.txt: size incorrect file=200 vs obj=100 fstest.go:127: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:127 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:132 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:151 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:195 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1992 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove Messages: rclone-test-kowosoy2vewu/file name.txt: Modification time difference too big |95170h10m10.999999999s| > 1s (want 2011-12-13 14:15:16.999999999 +0000 UTC vs got 2001-02-03 04:05:06 +0000 GMT) (precision 1s) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1992 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove Messages: Unexpected file "rclone-test-kowosoy2vewu/file name.txt-should-be-ignored.bin" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:1992 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove Messages: Unexpected file "rclone-test-kowosoy2vewu/zero-length-file" === RUN TestIntegration/FsMkdir/FsPutFiles/ObjectRemove 2025/11/10 03:49:23 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:49:25 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:49:26 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:49:27 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:49:28 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:49:29 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:49:29 DEBUG : pacer: Reducing sleep to 266.967773ms fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 2025/11/10 03:49:30 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:49:31 DEBUG : pacer: Reducing sleep to 150.169371ms 2025/11/10 03:49:31 DEBUG : pacer: Reducing sleep to 112.627028ms 2025/11/10 03:49:31 DEBUG : pacer: Reducing sleep to 84.470271ms 2025/11/10 03:49:31 DEBUG : pacer: Reducing sleep to 63.352703ms fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 2025/11/10 03:49:33 DEBUG : pacer: Reducing sleep to 47.514527ms 2025/11/10 03:49:33 DEBUG : pacer: Reducing sleep to 35.635895ms 2025/11/10 03:49:33 DEBUG : pacer: Reducing sleep to 26.726921ms 2025/11/10 03:49:33 DEBUG : pacer: Reducing sleep to 20.04519ms 2025/11/10 03:49:33 DEBUG : pacer: Reducing sleep to 15.033892ms fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:2269 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectRemove Messages: listing wrong, want hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100) got file name.txt-should-be-ignored.bin (200), hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt (100), zero-length-file (0) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:2269 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectRemove Messages: Unexpected file "file name.txt-should-be-ignored.bin" fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:2269 Error: Should be true Test: TestIntegration/FsMkdir/FsPutFiles/ObjectRemove Messages: Unexpected file "zero-length-file" === RUN TestIntegration/FsMkdir/FsPutChunked === RUN TestIntegration/FsMkdir/FsPutChunked/1Mi === RUN TestIntegration/FsMkdir/FsPutChunked/1Mi/1048575 2025/11/10 03:49:38 DEBUG : pacer: Reducing sleep to 11.275419ms 2025/11/10 03:49:38 DEBUG : chunked-1Mi-1023.999Ki.bin: Update will use the tus protocol to upload 2025/11/10 03:49:38 DEBUG : pacer: Reducing sleep to 10ms 2025/11/10 03:49:38 INFO : chunked-1Mi-1023.999Ki.bin: Failed to read metadata: object not found 2025/11/10 03:49:38 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:49:38 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1048575 2025/11/10 03:49:38 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:49:38 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:49:38 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/11/10 03:49:38 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:49:38 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/11/10 03:49:38 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:49:38 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/11/10 03:49:38 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:49:38 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/11/10 03:49:38 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:49:38 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/11/10 03:49:39 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:49:39 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/11/10 03:49:39 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:49:39 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/11/10 03:49:40 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:49:40 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:49:41 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:49:43 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1Mi/1048575 === RUN TestIntegration/FsMkdir/FsPutChunked/1Mi/1048576 2025/11/10 03:49:45 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:49:45 DEBUG : chunked-1Mi-1Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:49:47 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:49:47 INFO : chunked-1Mi-1Mi.bin: Failed to read metadata: object not found 2025/11/10 03:49:48 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:49:48 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:49:49 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:49:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1048576 2025/11/10 03:49:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:49:50 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:49:51 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:49:51 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:49:51 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:49:52 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:49:52 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:49:52 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:49:52 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:49:54 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:49:56 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:49:58 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:50:00 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:50:02 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:50:04 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:50:06 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1Mi/1048576 === RUN TestIntegration/FsMkdir/FsPutChunked/1Mi/2097153 2025/11/10 03:50:08 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:50:08 DEBUG : chunked-1Mi-2.000Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:50:10 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:50:10 INFO : chunked-1Mi-2.000Mi.bin: Failed to read metadata: object not found 2025/11/10 03:50:12 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:50:12 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:50:12 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:50:12 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1048576 2025/11/10 03:50:13 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:50:13 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 2 ok, range 1048576 -> 2097152 2025/11/10 03:50:14 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:50:14 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 3 ok, range 2097152 -> 2097153 2025/11/10 03:50:14 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:50:15 DEBUG : pacer: Reducing sleep to 266.967773ms 2025/11/10 03:50:15 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:50:15 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:50:15 DEBUG : pacer: Rate limited, increasing sleep to 400.451658ms 2025/11/10 03:50:16 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:50:16 DEBUG : pacer: Rate limited, increasing sleep to 800.903316ms 2025/11/10 03:50:16 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:50:16 DEBUG : pacer: Rate limited, increasing sleep to 1.601806632s 2025/11/10 03:50:17 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:50:17 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:50:18 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:50:20 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:50:23 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:50:24 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:50:27 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:50:28 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1Mi/2097153 === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1099999 2025/11/10 03:50:30 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:50:30 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:50:33 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:50:33 INFO : chunked-1.049Mi-1.049Mi.bin: Failed to read metadata: object not found 2025/11/10 03:50:34 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:50:34 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:50:35 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:50:35 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1099999 2025/11/10 03:50:35 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:50:36 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:50:37 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:50:37 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:50:37 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:50:37 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:50:37 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:50:38 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:50:38 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:50:39 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:50:41 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:50:43 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:50:46 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:50:48 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:50:50 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:50:52 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1099999 === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1100000 2025/11/10 03:50:54 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:50:54 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:50:56 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:50:57 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:50:57 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:50:58 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:50:58 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1100000 2025/11/10 03:50:58 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:50:59 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:51:00 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:51:00 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:51:00 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:51:01 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:51:01 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:51:01 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:51:01 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:51:03 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:51:05 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:51:07 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:51:09 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:51:11 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:51:13 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:51:15 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1100000 === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi/2200001 2025/11/10 03:51:17 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:51:17 DEBUG : chunked-1.049Mi-2.098Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:51:19 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:51:19 INFO : chunked-1.049Mi-2.098Mi.bin: Failed to read metadata: object not found 2025/11/10 03:51:20 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:51:20 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:51:21 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:51:21 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1100000 2025/11/10 03:51:22 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:51:22 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 2 ok, range 1100000 -> 2200000 2025/11/10 03:51:23 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:51:23 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 3 ok, range 2200000 -> 2200001 2025/11/10 03:51:23 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:51:23 DEBUG : pacer: Reducing sleep to 266.967773ms 2025/11/10 03:51:24 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:51:24 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:51:24 DEBUG : pacer: Rate limited, increasing sleep to 400.451658ms 2025/11/10 03:51:24 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:51:24 DEBUG : pacer: Rate limited, increasing sleep to 800.903316ms 2025/11/10 03:51:24 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:51:24 DEBUG : pacer: Rate limited, increasing sleep to 1.601806632s 2025/11/10 03:51:25 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:51:25 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:51:27 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:51:29 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:51:31 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:51:33 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:51:35 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:51:37 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.049Mi/2200001 === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01 === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100010 2025/11/10 03:51:39 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:51:39 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:51:41 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:51:42 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:51:42 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:51:43 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:51:43 DEBUG : pacer: Rate limited, increasing sleep to 1.6875s 2025/11/10 03:51:43 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:51:43 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 1/10 2025/11/10 03:51:45 DEBUG : pacer: Reducing sleep to 1.265625s 2025/11/10 03:51:45 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:51:47 DEBUG : pacer: Reducing sleep to 949.21875ms 2025/11/10 03:51:48 DEBUG : pacer: Reducing sleep to 711.914062ms 2025/11/10 03:51:48 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:51:49 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:51:49 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:51:49 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:51:49 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 2/10 2025/11/10 03:51:51 DEBUG : pacer: Reducing sleep to 1.067871093s 2025/11/10 03:51:51 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:51:53 DEBUG : pacer: Reducing sleep to 800.903319ms 2025/11/10 03:51:54 DEBUG : pacer: Reducing sleep to 600.677489ms 2025/11/10 03:51:54 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:51:55 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:51:55 DEBUG : pacer: Rate limited, increasing sleep to 1.201354978s 2025/11/10 03:51:55 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:51:55 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 3/10 2025/11/10 03:51:57 DEBUG : pacer: Reducing sleep to 901.016233ms 2025/11/10 03:51:57 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:51:58 DEBUG : pacer: Reducing sleep to 675.762174ms 2025/11/10 03:51:59 DEBUG : pacer: Reducing sleep to 506.82163ms 2025/11/10 03:51:59 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:51:59 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:51:59 DEBUG : pacer: Rate limited, increasing sleep to 1.01364326s 2025/11/10 03:51:59 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:51:59 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 4/10 2025/11/10 03:52:01 DEBUG : pacer: Reducing sleep to 760.232445ms 2025/11/10 03:52:01 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:03 DEBUG : pacer: Reducing sleep to 570.174333ms 2025/11/10 03:52:03 DEBUG : pacer: Reducing sleep to 427.630749ms 2025/11/10 03:52:03 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:04 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:04 DEBUG : pacer: Rate limited, increasing sleep to 855.261498ms 2025/11/10 03:52:04 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:04 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 5/10 2025/11/10 03:52:06 DEBUG : pacer: Reducing sleep to 641.446123ms 2025/11/10 03:52:06 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:07 DEBUG : pacer: Reducing sleep to 481.084592ms 2025/11/10 03:52:07 DEBUG : pacer: Reducing sleep to 360.813444ms 2025/11/10 03:52:07 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:08 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:08 DEBUG : pacer: Rate limited, increasing sleep to 721.626888ms 2025/11/10 03:52:08 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:08 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 6/10 2025/11/10 03:52:10 DEBUG : pacer: Reducing sleep to 541.220166ms 2025/11/10 03:52:10 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:11 DEBUG : pacer: Reducing sleep to 405.915124ms 2025/11/10 03:52:11 DEBUG : pacer: Reducing sleep to 304.436343ms 2025/11/10 03:52:11 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:11 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:11 DEBUG : pacer: Rate limited, increasing sleep to 608.872686ms 2025/11/10 03:52:11 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:11 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 7/10 2025/11/10 03:52:14 DEBUG : pacer: Reducing sleep to 456.654514ms 2025/11/10 03:52:14 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:14 DEBUG : pacer: Reducing sleep to 342.490885ms 2025/11/10 03:52:15 DEBUG : pacer: Reducing sleep to 256.868163ms 2025/11/10 03:52:15 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:15 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:15 DEBUG : pacer: Rate limited, increasing sleep to 513.736326ms 2025/11/10 03:52:15 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:15 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 8/10 2025/11/10 03:52:17 DEBUG : pacer: Reducing sleep to 385.302244ms 2025/11/10 03:52:17 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:17 DEBUG : pacer: Reducing sleep to 288.976683ms 2025/11/10 03:52:18 DEBUG : pacer: Reducing sleep to 216.732512ms 2025/11/10 03:52:18 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:18 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:18 DEBUG : pacer: Rate limited, increasing sleep to 433.465024ms 2025/11/10 03:52:18 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:18 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 9/10 2025/11/10 03:52:20 DEBUG : pacer: Reducing sleep to 325.098768ms 2025/11/10 03:52:20 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:21 DEBUG : pacer: Reducing sleep to 243.824076ms 2025/11/10 03:52:21 DEBUG : pacer: Reducing sleep to 182.868057ms 2025/11/10 03:52:21 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:21 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:21 DEBUG : pacer: Rate limited, increasing sleep to 365.736114ms 2025/11/10 03:52:21 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:21 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 10/10 fstests.go:146: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:146 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:236 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: tus update failed: uploadChunk failed: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100010 Messages: PutLarge === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100011 2025/11/10 03:52:23 DEBUG : pacer: Reducing sleep to 274.302085ms 2025/11/10 03:52:23 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:24 DEBUG : pacer: Reducing sleep to 205.726563ms 2025/11/10 03:52:24 DEBUG : pacer: Reducing sleep to 154.294922ms 2025/11/10 03:52:24 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:24 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:24 DEBUG : pacer: Rate limited, increasing sleep to 308.589844ms 2025/11/10 03:52:24 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:24 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 1/10 2025/11/10 03:52:26 DEBUG : pacer: Reducing sleep to 231.442383ms 2025/11/10 03:52:26 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:27 DEBUG : pacer: Reducing sleep to 173.581787ms 2025/11/10 03:52:27 DEBUG : pacer: Reducing sleep to 130.18634ms 2025/11/10 03:52:27 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:27 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:27 DEBUG : pacer: Rate limited, increasing sleep to 260.37268ms 2025/11/10 03:52:27 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:27 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 2/10 2025/11/10 03:52:29 DEBUG : pacer: Reducing sleep to 195.27951ms 2025/11/10 03:52:29 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:29 DEBUG : pacer: Reducing sleep to 146.459632ms 2025/11/10 03:52:29 DEBUG : pacer: Reducing sleep to 109.844724ms 2025/11/10 03:52:29 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:29 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:29 DEBUG : pacer: Rate limited, increasing sleep to 219.689448ms 2025/11/10 03:52:29 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:29 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 3/10 2025/11/10 03:52:31 DEBUG : pacer: Reducing sleep to 164.767086ms 2025/11/10 03:52:31 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:32 DEBUG : pacer: Reducing sleep to 123.575314ms 2025/11/10 03:52:32 DEBUG : pacer: Reducing sleep to 92.681485ms 2025/11/10 03:52:32 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:32 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:32 DEBUG : pacer: Rate limited, increasing sleep to 185.36297ms 2025/11/10 03:52:32 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:32 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 4/10 2025/11/10 03:52:34 DEBUG : pacer: Reducing sleep to 139.022227ms 2025/11/10 03:52:34 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:34 DEBUG : pacer: Reducing sleep to 104.26667ms 2025/11/10 03:52:34 DEBUG : pacer: Reducing sleep to 78.200002ms 2025/11/10 03:52:34 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:34 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:34 DEBUG : pacer: Rate limited, increasing sleep to 156.400004ms 2025/11/10 03:52:34 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:34 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 5/10 2025/11/10 03:52:36 DEBUG : pacer: Reducing sleep to 117.300003ms 2025/11/10 03:52:36 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:37 DEBUG : pacer: Reducing sleep to 87.975002ms 2025/11/10 03:52:37 DEBUG : pacer: Reducing sleep to 65.981251ms 2025/11/10 03:52:37 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:37 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:37 DEBUG : pacer: Rate limited, increasing sleep to 131.962502ms 2025/11/10 03:52:37 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:37 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 6/10 2025/11/10 03:52:39 DEBUG : pacer: Reducing sleep to 98.971876ms 2025/11/10 03:52:39 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:39 DEBUG : pacer: Reducing sleep to 74.228907ms 2025/11/10 03:52:39 DEBUG : pacer: Reducing sleep to 55.67168ms 2025/11/10 03:52:39 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:39 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:39 DEBUG : pacer: Rate limited, increasing sleep to 111.34336ms 2025/11/10 03:52:39 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:39 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 7/10 2025/11/10 03:52:41 DEBUG : pacer: Reducing sleep to 83.50752ms 2025/11/10 03:52:41 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:41 DEBUG : pacer: Reducing sleep to 62.63064ms 2025/11/10 03:52:42 DEBUG : pacer: Reducing sleep to 46.97298ms 2025/11/10 03:52:42 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:42 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:42 DEBUG : pacer: Rate limited, increasing sleep to 93.94596ms 2025/11/10 03:52:42 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:42 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 8/10 2025/11/10 03:52:44 DEBUG : pacer: Reducing sleep to 70.45947ms 2025/11/10 03:52:44 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:44 DEBUG : pacer: Reducing sleep to 52.844602ms 2025/11/10 03:52:44 DEBUG : pacer: Reducing sleep to 39.633451ms 2025/11/10 03:52:44 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:44 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:44 DEBUG : pacer: Rate limited, increasing sleep to 79.266902ms 2025/11/10 03:52:44 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:44 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 9/10 2025/11/10 03:52:46 DEBUG : pacer: Reducing sleep to 59.450176ms 2025/11/10 03:52:46 DEBUG : chunked-1.049Mi-1.049Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:46 DEBUG : pacer: Reducing sleep to 44.587632ms 2025/11/10 03:52:46 DEBUG : pacer: Reducing sleep to 33.440724ms 2025/11/10 03:52:46 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:46 DEBUG : pacer: low level retry 1/1 (error 500 Internal Server Error) 2025/11/10 03:52:46 DEBUG : pacer: Rate limited, increasing sleep to 66.881448ms 2025/11/10 03:52:46 ERROR : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 failed: uploadChunk failed: 500 Internal Server Error 2025/11/10 03:52:46 DEBUG : chunked-1.049Mi-1.049Mi.bin: tus update failed. fstests.go:143: PutLarge error: tus update failed: uploadChunk failed: 500 Internal Server Error - low level retry 10/10 fstests.go:146: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:146 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:236 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: tus update failed: uploadChunk failed: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100011 Messages: PutLarge === RUN TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/2200023 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 50.161086ms 2025/11/10 03:52:49 DEBUG : chunked-1.049Mi-2.098Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 37.620814ms 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 28.21561ms 2025/11/10 03:52:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 21.161707ms 2025/11/10 03:52:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1100011 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 15.87128ms 2025/11/10 03:52:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 2 ok, range 1100011 -> 2200022 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 11.90346ms 2025/11/10 03:52:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 3 ok, range 2200022 -> 2200023 2025/11/10 03:52:49 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:52:49 DEBUG : pacer: Reducing sleep to 10ms 2025/11/10 03:52:49 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:52:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/11/10 03:52:49 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:52:49 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2025/11/10 03:52:49 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:52:49 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2025/11/10 03:52:50 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:52:50 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2025/11/10 03:52:50 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:52:50 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2025/11/10 03:52:50 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:52:50 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2025/11/10 03:52:50 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:52:50 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2025/11/10 03:52:51 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:52:51 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:52:52 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:52:54 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/2200023 === RUN TestIntegration/FsMkdir/FsPutChunked/1.333Mi === RUN TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398100 2025/11/10 03:52:56 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:52:56 DEBUG : chunked-1.333Mi-1.333Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:52:58 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:52:58 INFO : chunked-1.333Mi-1.333Mi.bin: Failed to read metadata: object not found 2025/11/10 03:53:00 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:53:00 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:53:01 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:53:01 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1398100 2025/11/10 03:53:01 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:53:02 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:53:02 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:53:03 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:53:03 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:53:03 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:53:03 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:53:04 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:53:04 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:53:05 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:53:07 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:53:09 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:53:11 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:53:13 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:53:15 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:53:17 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398100 === RUN TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398101 2025/11/10 03:53:19 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:53:19 DEBUG : chunked-1.333Mi-1.333Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:53:21 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:53:23 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:53:23 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:53:24 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:53:24 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1398101 2025/11/10 03:53:24 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:53:25 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:53:25 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:53:26 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:53:26 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:53:26 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:53:26 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:53:27 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:53:27 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:53:28 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:53:30 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:53:32 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:53:34 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:53:36 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:53:38 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:53:40 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398101 === RUN TestIntegration/FsMkdir/FsPutChunked/1.333Mi/2796203 2025/11/10 03:53:42 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:53:42 DEBUG : chunked-1.333Mi-2.667Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:53:44 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:53:44 INFO : chunked-1.333Mi-2.667Mi.bin: Failed to read metadata: object not found 2025/11/10 03:53:46 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:53:46 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:53:47 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:53:47 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 1398101 2025/11/10 03:53:48 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:53:48 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 2 ok, range 1398101 -> 2796202 2025/11/10 03:53:48 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:53:48 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 3 ok, range 2796202 -> 2796203 2025/11/10 03:53:48 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:53:49 DEBUG : pacer: Reducing sleep to 266.967773ms 2025/11/10 03:53:49 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:53:50 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:53:50 DEBUG : pacer: Rate limited, increasing sleep to 400.451658ms 2025/11/10 03:53:50 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:53:50 DEBUG : pacer: Rate limited, increasing sleep to 800.903316ms 2025/11/10 03:53:50 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:53:50 DEBUG : pacer: Rate limited, increasing sleep to 1.601806632s 2025/11/10 03:53:51 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:53:51 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:53:53 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:53:55 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:53:57 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:53:59 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:54:01 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:54:03 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/1.333Mi/2796203 === RUN TestIntegration/FsMkdir/FsPutChunked/2Mi === RUN TestIntegration/FsMkdir/FsPutChunked/2Mi/2097151 2025/11/10 03:54:05 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:54:05 DEBUG : chunked-2Mi-2.000Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:54:07 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:54:07 INFO : chunked-2Mi-2.000Mi.bin: Failed to read metadata: object not found 2025/11/10 03:54:08 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:54:08 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:54:09 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:54:09 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 2097151 2025/11/10 03:54:09 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:54:10 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:54:11 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:54:11 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:54:11 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:54:12 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:54:12 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:54:12 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:54:12 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:54:14 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:54:16 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:54:18 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:54:20 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:54:22 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:54:24 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:54:26 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/2Mi/2097151 === RUN TestIntegration/FsMkdir/FsPutChunked/2Mi/2097152 2025/11/10 03:54:28 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:54:28 DEBUG : chunked-2Mi-2Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:54:30 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:54:30 INFO : chunked-2Mi-2Mi.bin: Failed to read metadata: object not found 2025/11/10 03:54:31 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:54:31 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:54:32 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:54:32 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 2097152 2025/11/10 03:54:32 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:54:33 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:54:34 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:54:34 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:54:34 DEBUG : pacer: Rate limited, increasing sleep to 711.914062ms 2025/11/10 03:54:35 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:54:35 DEBUG : pacer: Rate limited, increasing sleep to 1.423828124s 2025/11/10 03:54:35 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:54:35 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:54:37 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:54:39 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:54:41 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:54:43 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:54:45 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:54:47 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:54:49 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/2Mi/2097152 === RUN TestIntegration/FsMkdir/FsPutChunked/2Mi/4194305 2025/11/10 03:54:51 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:54:51 DEBUG : chunked-2Mi-4.000Mi.bin: Update will use the tus protocol to upload 2025/11/10 03:54:53 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:54:53 INFO : chunked-2Mi-4.000Mi.bin: Failed to read metadata: object not found 2025/11/10 03:54:54 DEBUG : pacer: Reducing sleep to 843.75ms 2025/11/10 03:54:54 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded starts 2025/11/10 03:54:55 DEBUG : pacer: Reducing sleep to 632.8125ms 2025/11/10 03:54:55 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 1 ok, range 0 -> 2097152 2025/11/10 03:54:56 DEBUG : pacer: Reducing sleep to 474.609375ms 2025/11/10 03:54:56 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 2 ok, range 2097152 -> 4194304 2025/11/10 03:54:57 DEBUG : pacer: Reducing sleep to 355.957031ms 2025/11/10 03:54:57 DEBUG : webdav root 'rclone-test-kowosoy2vewu': Uploaded chunk no 3 ok, range 4194304 -> 4194305 2025/11/10 03:54:57 DEBUG : webdav root 'rclone-test-kowosoy2vewu': -- Uploaded finished 2025/11/10 03:54:58 DEBUG : pacer: Reducing sleep to 266.967773ms 2025/11/10 03:54:58 DEBUG : pacer: Reducing sleep to 200.225829ms 2025/11/10 03:54:58 DEBUG : pacer: low level retry 1/10 (error 500 Internal Server Error) 2025/11/10 03:54:58 DEBUG : pacer: Rate limited, increasing sleep to 400.451658ms 2025/11/10 03:54:59 DEBUG : pacer: low level retry 2/10 (error 500 Internal Server Error) 2025/11/10 03:54:59 DEBUG : pacer: Rate limited, increasing sleep to 800.903316ms 2025/11/10 03:54:59 DEBUG : pacer: low level retry 3/10 (error 500 Internal Server Error) 2025/11/10 03:54:59 DEBUG : pacer: Rate limited, increasing sleep to 1.601806632s 2025/11/10 03:55:00 DEBUG : pacer: low level retry 4/10 (error 500 Internal Server Error) 2025/11/10 03:55:00 DEBUG : pacer: Rate limited, increasing sleep to 2s 2025/11/10 03:55:01 DEBUG : pacer: low level retry 5/10 (error 500 Internal Server Error) 2025/11/10 03:55:03 DEBUG : pacer: low level retry 6/10 (error 500 Internal Server Error) 2025/11/10 03:55:05 DEBUG : pacer: low level retry 7/10 (error 500 Internal Server Error) 2025/11/10 03:55:07 DEBUG : pacer: low level retry 8/10 (error 500 Internal Server Error) 2025/11/10 03:55:09 DEBUG : pacer: low level retry 9/10 (error 500 Internal Server Error) 2025/11/10 03:55:11 DEBUG : pacer: low level retry 10/10 (error 500 Internal Server Error) fstests.go:262: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:262 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:275 Error: Received unexpected error: 500 Internal Server Error Test: TestIntegration/FsMkdir/FsPutChunked/2Mi/4194305 2025/11/10 03:55:13 DEBUG : pacer: Reducing sleep to 1.5s 2025/11/10 03:55:15 DEBUG : pacer: Reducing sleep to 1.125s 2025/11/10 03:55:15 ERROR : error listing: directory not found 2025/11/10 03:55:17 DEBUG : pacer: Reducing sleep to 843.75ms === NAME TestIntegration/FsMkdir fstests.go:2817: Warning: this should produce fs.ErrorDirNotFound 2025/11/10 03:55:17 NOTICE: TestWebdavInfiniteScale: Stopping server --- FAIL: TestIntegration (615.35s) --- FAIL: TestIntegration/FsMkdir (607.97s) --- FAIL: TestIntegration/FsMkdir/FsPutZeroLength (5.14s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles (263.31s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 (0.35s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsListR (0.36s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 (0.14s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsListFile1 (8.30s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 (8.37s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsPurge (16.86s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot (17.06s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsCopy (5.06s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsMove (43.59s) --- SKIP: TestIntegration/FsMkdir/FsPutFiles/FsMove/Metadata (0.00s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FsDirMove (18.15s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectSetModTime (0.28s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectOpen (4.86s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek (21.52s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectOpenRange (21.48s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectPartialRead (21.50s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate (23.07s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot (26.67s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List (8.86s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries (0.14s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put (17.66s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove (8.69s) --- FAIL: TestIntegration/FsMkdir/FsPutFiles/ObjectRemove (14.94s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked (333.93s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1Mi (50.97s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1Mi/1048575 (5.40s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1Mi/1048576 (23.28s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1Mi/2097153 (22.29s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi (68.36s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1099999 (23.21s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1100000 (23.18s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi/2200001 (21.97s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01 (77.38s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100010 (46.34s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100011 (25.34s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/2200023 (5.70s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.333Mi (68.76s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398100 (23.11s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398101 (23.05s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/1.333Mi/2796203 (22.60s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/2Mi (68.45s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/2Mi/2097151 (22.88s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/2Mi/2097152 (23.13s) --- FAIL: TestIntegration/FsMkdir/FsPutChunked/2Mi/4194305 (22.44s) FAIL exit status 1 FAIL github.com/rclone/rclone/backend/webdav 615.424s "go test -v -timeout 1h0m0s -remote TestWebdavInfiniteScale: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1.049Mi#01$/^(1100010|1100011|2200023)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1.049Mi$/^(1099999|1100000|2200001)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1.333Mi$/^(1398100|1398101|2796203)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^1Mi$/^(1048575|1048576|2097153)$|^TestIntegration$/^FsMkdir$/^FsPutChunked$/^2Mi$/^(2097151|2097152|4194305)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^(FsCopy|FsDirMove|FsListDirFile2|FsListFile1|FsListFile1and2|FsListLevel2|FsListR|FsMove|FsPurge|FsPurgeRoot|ObjectOpen|ObjectOpenRange|ObjectOpenSeek|ObjectPartialRead|ObjectRemove|ObjectSetModTime|ObjectUpdate)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^(List|ListEntries)$|^TestIntegration$/^FsMkdir$/^FsPutFiles$/^FromRoot$/^Put$/^Remove$|^TestIntegration$/^FsMkdir$/^FsPutZeroLength$'" - Finished ERROR in 10m16.477838681s (try 3/5): exit status 1: Failed [TestIntegration/FsMkdir/FsPutZeroLength TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 TestIntegration/FsMkdir/FsPutFiles/FsListR TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 TestIntegration/FsMkdir/FsPutFiles/FsListFile1 TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 TestIntegration/FsMkdir/FsPutFiles/FsPurge TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot TestIntegration/FsMkdir/FsPutFiles/FsCopy TestIntegration/FsMkdir/FsPutFiles/FsMove TestIntegration/FsMkdir/FsPutFiles/FsDirMove TestIntegration/FsMkdir/FsPutFiles/ObjectSetModTime TestIntegration/FsMkdir/FsPutFiles/ObjectOpen TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek TestIntegration/FsMkdir/FsPutFiles/ObjectOpenRange TestIntegration/FsMkdir/FsPutFiles/ObjectPartialRead TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate TestIntegration/FsMkdir/FsPutFiles/FromRoot/List TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove TestIntegration/FsMkdir/FsPutFiles/ObjectRemove TestIntegration/FsMkdir/FsPutChunked/1Mi/1048575 TestIntegration/FsMkdir/FsPutChunked/1Mi/1048576 TestIntegration/FsMkdir/FsPutChunked/1Mi/2097153 TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1099999 TestIntegration/FsMkdir/FsPutChunked/1.049Mi/1100000 TestIntegration/FsMkdir/FsPutChunked/1.049Mi/2200001 TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100010 TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/1100011 TestIntegration/FsMkdir/FsPutChunked/1.049Mi#01/2200023 TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398100 TestIntegration/FsMkdir/FsPutChunked/1.333Mi/1398101 TestIntegration/FsMkdir/FsPutChunked/1.333Mi/2796203 TestIntegration/FsMkdir/FsPutChunked/2Mi/2097151 TestIntegration/FsMkdir/FsPutChunked/2Mi/2097152 TestIntegration/FsMkdir/FsPutChunked/2Mi/4194305]