"./sync.test -test.v -test.timeout 1h0m0s -remote TestChunkerChunk50bYandex: -verbose -size-limit 1024 -test.run '^(TestSyncConcurrentDelete|TestSyncConcurrentTruncate)$'" - Starting (try 5/5)
2025/02/01 06:05:46 DEBUG : Creating backend with remote "TestChunkerChunk50bYandex:rclone-test-mojiyat9lopi"
2025/02/01 06:05:46 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2025/02/01 06:05:46 DEBUG : Creating backend with remote "TestYandex:rclone-test-mojiyat9lopi"
2025/02/01 06:05:46 DEBUG : Reset feature "ListR"
2025/02/01 06:05:46 DEBUG : Creating backend with remote "/tmp/rclone3309297668"
=== RUN   TestSyncConcurrentDelete
    run.go:180: Remote "Chunked 'TestChunkerChunk50bYandex:rclone-test-mojiyat9lopi'", Local "Local file system at /tmp/rclone3309297668", Modify Window "1ns"
2025/02/01 06:05:50 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:05:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:05:51 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:05:52 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:05:52 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:05:52 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2025/02/01 06:05:52 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:05:53 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:05:53 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:05:54 INFO  : both0.rclone_chunk.001_zvupsp: Moved (server-side) to: both0
2025/02/01 06:05:54 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:05:54 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:05:55 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:05:55 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:05:56 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:05:59 INFO  : only0.rclone_chunk.001_zvuwtw: Moved (server-side) to: only0
2025/02/01 06:05:59 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:05:59 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:05:59 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:05:59 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:05:59 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2025/02/01 06:06:00 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:06:00 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:00 DEBUG : pacer: Rate limited, increasing sleep to 45ms
2025/02/01 06:06:00 DEBUG : pacer: Reducing sleep to 33.75ms
2025/02/01 06:06:01 DEBUG : pacer: Reducing sleep to 25.3125ms
2025/02/01 06:06:01 DEBUG : pacer: Reducing sleep to 18.984375ms
2025/02/01 06:06:02 DEBUG : pacer: Reducing sleep to 14.238281ms
2025/02/01 06:06:03 DEBUG : pacer: Reducing sleep to 10.67871ms
2025/02/01 06:06:03 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:06:04 ERROR : both1.rclone_chunk.001_zvv1l4: Couldn't move: couldn't move file: [500 - InternalServerError] Internal Server Error (Ошибка сервера.)
2025/02/01 06:06:07 ERROR : both1.rclone_chunk.001_zvv1l4: Failed to remove temporary chunk: [500 - InternalServerError] Internal Server Error (Ошибка сервера.)
    run.go:285: Failed to put "both1" to "Chunked 'TestChunkerChunk50bYandex:rclone-test-mojiyat9lopi'": couldn't move file: [500 - InternalServerError] Internal Server Error (Ошибка сервера.)
2025/02/01 06:06:14 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:14 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:06:14 DEBUG : pacer: low level retry 2/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:14 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2025/02/01 06:06:14 DEBUG : pacer: Reducing sleep to 30ms
--- FAIL: TestSyncConcurrentDelete (28.01s)
=== RUN   TestSyncConcurrentTruncate
    run.go:180: Remote "Chunked 'TestChunkerChunk50bYandex:rclone-test-mojiyat9lopi'", Local "Local file system at /tmp/rclone3309297668", Modify Window "1ns"
2025/02/01 06:06:15 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:06:15 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:06:15 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:06:16 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:06:17 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:17 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:06:18 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:06:18 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:06:19 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:06:19 INFO  : both0.rclone_chunk.001_zvvh2d: Moved (server-side) to: both0
2025/02/01 06:06:19 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:19 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:06:20 DEBUG : pacer: low level retry 2/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:20 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2025/02/01 06:06:20 DEBUG : pacer: Reducing sleep to 30ms
2025/02/01 06:06:20 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:20 DEBUG : pacer: Rate limited, increasing sleep to 60ms
2025/02/01 06:06:20 DEBUG : pacer: Reducing sleep to 45ms
2025/02/01 06:06:21 DEBUG : pacer: Reducing sleep to 33.75ms
2025/02/01 06:06:21 DEBUG : pacer: Reducing sleep to 25.3125ms
2025/02/01 06:06:22 DEBUG : pacer: Reducing sleep to 18.984375ms
2025/02/01 06:06:22 DEBUG : pacer: Reducing sleep to 14.238281ms
2025/02/01 06:06:23 DEBUG : pacer: Reducing sleep to 10.67871ms
2025/02/01 06:06:23 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:06:27 INFO  : only0.rclone_chunk.001_zvvmy1: Moved (server-side) to: only0
2025/02/01 06:06:29 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:29 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:06:29 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:06:30 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:06:30 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:30 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2025/02/01 06:06:31 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:06:32 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:06:32 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:32 DEBUG : pacer: Rate limited, increasing sleep to 25.3125ms
2025/02/01 06:06:32 DEBUG : pacer: Reducing sleep to 18.984375ms
2025/02/01 06:06:33 DEBUG : pacer: Reducing sleep to 14.238281ms
2025/02/01 06:06:33 DEBUG : pacer: Reducing sleep to 10.67871ms
2025/02/01 06:06:38 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:06:38 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:38 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:06:39 DEBUG : pacer: low level retry 2/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:39 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2025/02/01 06:06:39 DEBUG : pacer: Reducing sleep to 30ms
2025/02/01 06:06:39 INFO  : both1.rclone_chunk.001_zvvt5s: Moved (server-side) to: both1
2025/02/01 06:06:39 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:06:40 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:06:40 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:06:41 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:06:44 INFO  : only1.rclone_chunk.001_zvw5zx: Moved (server-side) to: only1
2025/02/01 06:06:49 INFO  : both2.rclone_chunk.001_zvwawg: Moved (server-side) to: both2
2025/02/01 06:06:55 INFO  : only2.rclone_chunk.001_zvwg0m: Moved (server-side) to: only2
2025/02/01 06:06:58 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:06:58 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:06:58 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:06:58 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:06:59 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:00 INFO  : both3.rclone_chunk.001_zvwlju: Moved (server-side) to: both3
2025/02/01 06:07:01 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:01 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:01 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:07:01 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:01 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2025/02/01 06:07:02 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:07:04 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:07:05 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:07:05 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:08 INFO  : only3.rclone_chunk.001_zvwqbp: Moved (server-side) to: only3
2025/02/01 06:07:08 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:08 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:08 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:07:09 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:09 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2025/02/01 06:07:09 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:07:09 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:07:10 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:10 DEBUG : pacer: Rate limited, increasing sleep to 33.75ms
2025/02/01 06:07:10 DEBUG : pacer: Reducing sleep to 25.3125ms
2025/02/01 06:07:11 DEBUG : pacer: Reducing sleep to 18.984375ms
2025/02/01 06:07:11 DEBUG : pacer: Reducing sleep to 14.238281ms
2025/02/01 06:07:11 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:11 DEBUG : pacer: Rate limited, increasing sleep to 28.476562ms
2025/02/01 06:07:11 DEBUG : pacer: Reducing sleep to 21.357421ms
2025/02/01 06:07:12 DEBUG : pacer: Reducing sleep to 16.018065ms
2025/02/01 06:07:12 DEBUG : pacer: Reducing sleep to 12.013548ms
2025/02/01 06:07:12 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:12 DEBUG : pacer: Rate limited, increasing sleep to 24.027096ms
2025/02/01 06:07:13 DEBUG : pacer: Reducing sleep to 18.020322ms
2025/02/01 06:07:17 DEBUG : pacer: Reducing sleep to 13.515241ms
2025/02/01 06:07:18 DEBUG : pacer: Reducing sleep to 10.13643ms
2025/02/01 06:07:18 INFO  : both4.rclone_chunk.001_zvwynr: Moved (server-side) to: both4
2025/02/01 06:07:18 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:21 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:21 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:07:21 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:07:23 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:23 INFO  : only4.rclone_chunk.001_zvx8zp: Moved (server-side) to: only4
2025/02/01 06:07:25 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:25 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:26 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:07:26 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:26 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2025/02/01 06:07:26 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:07:27 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:07:27 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:07:27 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:28 INFO  : both5.rclone_chunk.001_zvxd00: Moved (server-side) to: both5
2025/02/01 06:07:32 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:32 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:32 DEBUG : pacer: low level retry 2/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:32 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2025/02/01 06:07:32 DEBUG : pacer: Reducing sleep to 30ms
2025/02/01 06:07:33 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:07:34 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:07:34 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:07:34 INFO  : only5.rclone_chunk.001_zvxjo1: Moved (server-side) to: only5
2025/02/01 06:07:34 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:39 INFO  : both6.rclone_chunk.001_zvxofx: Moved (server-side) to: both6
2025/02/01 06:07:44 INFO  : only6.rclone_chunk.001_zvxt2o: Moved (server-side) to: only6
2025/02/01 06:07:47 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:47 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:07:48 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:07:48 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:07:49 INFO  : both7.rclone_chunk.001_zvxya6: Moved (server-side) to: both7
2025/02/01 06:07:54 INFO  : only7.rclone_chunk.001_zvy3bf: Moved (server-side) to: only7
2025/02/01 06:07:58 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:58 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:07:59 DEBUG : pacer: low level retry 2/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:07:59 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2025/02/01 06:07:59 DEBUG : pacer: Reducing sleep to 30ms
2025/02/01 06:07:59 DEBUG : pacer: Reducing sleep to 22.5ms
2025/02/01 06:08:00 DEBUG : pacer: Reducing sleep to 16.875ms
2025/02/01 06:08:00 DEBUG : pacer: Reducing sleep to 12.65625ms
2025/02/01 06:08:01 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:08:01 INFO  : both8.rclone_chunk.001_zvy8lq: Moved (server-side) to: both8
2025/02/01 06:08:07 INFO  : only8.rclone_chunk.001_zvygmz: Moved (server-side) to: only8
2025/02/01 06:08:09 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:08:09 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:08:10 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:08:13 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:08:13 DEBUG : pacer: Reducing sleep to 10ms
2025/02/01 06:08:14 ERROR : both9.rclone_chunk.001_zvylml: Couldn't move: couldn't move file: [500 - InternalServerError] Internal Server Error (Ошибка сервера.)
2025/02/01 06:08:17 ERROR : both9.rclone_chunk.001_zvylml: Failed to remove temporary chunk: [404 - DiskNotFoundError] Resource not found. (Не удалось найти запрошенный ресурс.)
    run.go:285: Failed to put "both9" to "Chunked 'TestChunkerChunk50bYandex:rclone-test-mojiyat9lopi'": couldn't move file: [500 - InternalServerError] Internal Server Error (Ошибка сервера.)
2025/02/01 06:08:18 DEBUG : pacer: low level retry 1/10 (error [500 - InternalServerError] Internal Server Error (Ошибка сервера.))
2025/02/01 06:08:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2025/02/01 06:08:18 DEBUG : pacer: Reducing sleep to 15ms
2025/02/01 06:08:21 DEBUG : pacer: Reducing sleep to 11.25ms
2025/02/01 06:08:25 DEBUG : pacer: Reducing sleep to 10ms
--- FAIL: TestSyncConcurrentTruncate (180.97s)
FAIL
2025/02/01 06:09:15 DEBUG : Chunked 'TestChunkerChunk50bYandex:rclone-test-mojiyat9lopi': Purge remote
"./sync.test -test.v -test.timeout 1h0m0s -remote TestChunkerChunk50bYandex: -verbose -size-limit 1024 -test.run '^(TestSyncConcurrentDelete|TestSyncConcurrentTruncate)$'" - Finished ERROR in 3m32.205048117s (try 5/5): exit status 1: Failed [TestSyncConcurrentDelete TestSyncConcurrentTruncate]