"./sync.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^TestSyncConcurrentDelete$'" - Starting (try 4/5) 2026/09/05 02:53:11 DEBUG : Creating backend with remote "TestShade:rclone-test-toqahuv7refe" 2026/09/05 02:53:11 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/05 02:53:11 DEBUG : Creating new ShadeFS backend with drive: cf193d00-c783-4bab-aa7a-fd84b3700d27 2026/09/05 02:53:11 DEBUG : Creating backend with remote "/tmp/rclone516911185" === RUN TestSyncConcurrentDelete run.go:198: Remote "Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-toqahuv7refe", Local "Local file system at /tmp/rclone516911185", Modify Window "876000h0m0s" 2026/09/05 02:53:12 DEBUG : both0: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:14 DEBUG : only0: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:16 DEBUG : both1: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:18 DEBUG : only1: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:20 DEBUG : both2: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:22 DEBUG : only2: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:24 DEBUG : both3: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:26 DEBUG : only3: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:33 DEBUG : both4: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:36 DEBUG : only4: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:38 DEBUG : both5: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:40 DEBUG : only5: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:42 DEBUG : both6: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:44 DEBUG : only6: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:45 DEBUG : both7: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:47 DEBUG : only7: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:49 DEBUG : both8: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:51 DEBUG : only8: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:53 DEBUG : both9: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:55 DEBUG : only9: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:57 DEBUG : both10: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:53:59 DEBUG : only10: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:01 DEBUG : both11: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:03 DEBUG : only11: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:05 DEBUG : both12: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:07 DEBUG : only12: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:09 DEBUG : both13: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:11 DEBUG : only13: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:13 DEBUG : both14: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:15 DEBUG : only14: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:17 DEBUG : both15: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:19 DEBUG : only15: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:20 DEBUG : both16: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:23 DEBUG : only16: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:25 DEBUG : both17: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:27 DEBUG : only17: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:28 DEBUG : both18: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:30 DEBUG : only18: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:32 DEBUG : both19: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:35 DEBUG : only19: multipart upload: starting chunk 0 size 6 offset 0/6 2026/09/05 02:54:37 DEBUG : both11: size = 6 OK 2026/09/05 02:54:37 DEBUG : both12: size = 6 OK 2026/09/05 02:54:37 DEBUG : both12: Sizes identical 2026/09/05 02:54:37 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-toqahuv7refe: Waiting for checks to finish 2026/09/05 02:54:37 DEBUG : both11: Sizes identical 2026/09/05 02:54:37 DEBUG : both0: size = 6 OK 2026/09/05 02:54:37 DEBUG : both0: Sizes identical 2026/09/05 02:54:37 DEBUG : both10: size = 6 OK 2026/09/05 02:54:37 DEBUG : both10: Sizes identical 2026/09/05 02:54:37 DEBUG : both12: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both13: size = 6 OK 2026/09/05 02:54:37 DEBUG : both13: Sizes identical 2026/09/05 02:54:37 DEBUG : both10: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both14: size = 6 OK 2026/09/05 02:54:37 DEBUG : both14: Sizes identical 2026/09/05 02:54:37 DEBUG : both11: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both15: size = 6 OK 2026/09/05 02:54:37 DEBUG : both15: Sizes identical 2026/09/05 02:54:37 DEBUG : both15: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both16: size = 6 OK 2026/09/05 02:54:37 DEBUG : both16: Sizes identical 2026/09/05 02:54:37 DEBUG : both16: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both17: size = 6 OK 2026/09/05 02:54:37 DEBUG : both17: Sizes identical 2026/09/05 02:54:37 DEBUG : both17: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both18: size = 6 OK 2026/09/05 02:54:37 DEBUG : both18: Sizes identical 2026/09/05 02:54:37 DEBUG : both18: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both19: size = 6 OK 2026/09/05 02:54:37 DEBUG : both19: Sizes identical 2026/09/05 02:54:37 DEBUG : both0: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both1: size = 6 OK 2026/09/05 02:54:37 DEBUG : both1: Sizes identical 2026/09/05 02:54:37 DEBUG : both13: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both2: size = 6 OK 2026/09/05 02:54:37 DEBUG : both2: Sizes identical 2026/09/05 02:54:37 DEBUG : both2: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both3: size = 6 OK 2026/09/05 02:54:37 DEBUG : both3: Sizes identical 2026/09/05 02:54:37 DEBUG : both14: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both4: size = 6 OK 2026/09/05 02:54:37 DEBUG : both4: Sizes identical 2026/09/05 02:54:37 DEBUG : both4: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both5: size = 6 OK 2026/09/05 02:54:37 DEBUG : both5: Sizes identical 2026/09/05 02:54:37 DEBUG : both5: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both6: size = 6 OK 2026/09/05 02:54:37 DEBUG : both6: Sizes identical 2026/09/05 02:54:37 DEBUG : both6: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both7: size = 6 OK 2026/09/05 02:54:37 DEBUG : both7: Sizes identical 2026/09/05 02:54:37 DEBUG : both7: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both8: size = 6 OK 2026/09/05 02:54:37 DEBUG : both8: Sizes identical 2026/09/05 02:54:37 DEBUG : both8: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both9: size = 6 OK 2026/09/05 02:54:37 DEBUG : both9: Sizes identical 2026/09/05 02:54:37 DEBUG : both9: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both19: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both1: Unchanged skipping 2026/09/05 02:54:37 DEBUG : both3: Unchanged skipping 2026/09/05 02:54:37 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-toqahuv7refe: Waiting for transfers to finish 2026/09/05 02:54:37 DEBUG : Waiting for deletions to finish 2026/09/05 02:54:37 INFO : only16: Deleted 2026/09/05 02:54:37 INFO : only17: Deleted 2026/09/05 02:54:37 INFO : only18: Deleted 2026/09/05 02:54:37 INFO : only3: Deleted 2026/09/05 02:54:37 INFO : only11: Deleted 2026/09/05 02:54:37 INFO : only14: Deleted 2026/09/05 02:54:37 INFO : only15: Deleted 2026/09/05 02:54:37 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:37 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/05 02:54:37 DEBUG : pacer: Reducing sleep to 10ms 2026/09/05 02:54:37 INFO : only1: Deleted 2026/09/05 02:54:37 INFO : only4: Deleted 2026/09/05 02:54:37 INFO : only7: Deleted 2026/09/05 02:54:37 DEBUG : pacer: low level retry 2/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:37 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/05 02:54:37 DEBUG : pacer: Reducing sleep to 10ms 2026/09/05 02:54:37 INFO : only0: Deleted 2026/09/05 02:54:37 INFO : only13: Deleted 2026/09/05 02:54:37 INFO : only8: Deleted 2026/09/05 02:54:37 DEBUG : pacer: low level retry 3/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:37 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/05 02:54:38 DEBUG : pacer: Reducing sleep to 10ms 2026/09/05 02:54:38 INFO : only10: Deleted 2026/09/05 02:54:38 INFO : only12: Deleted 2026/09/05 02:54:38 INFO : only2: Deleted 2026/09/05 02:54:38 DEBUG : pacer: low level retry 4/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/05 02:54:38 DEBUG : pacer: Reducing sleep to 10ms 2026/09/05 02:54:38 INFO : only5: Deleted 2026/09/05 02:54:38 INFO : only6: Deleted 2026/09/05 02:54:38 INFO : only9: Deleted 2026/09/05 02:54:38 DEBUG : pacer: low level retry 5/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/05 02:54:38 DEBUG : pacer: low level retry 6/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2026/09/05 02:54:38 DEBUG : pacer: low level retry 7/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2026/09/05 02:54:38 DEBUG : pacer: low level retry 8/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2026/09/05 02:54:38 DEBUG : pacer: low level retry 9/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2026/09/05 02:54:38 DEBUG : pacer: low level retry 10/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:38 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2026/09/05 02:54:38 ERROR : only19: Couldn't delete: HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND" 2026/09/05 02:54:38 ERROR : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-toqahuv7refe: not deleting directories as there were IO errors sync_test.go:2783: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2783 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2789 Error: Received unexpected error: failed to delete 1 files Test: TestSyncConcurrentDelete Messages: Sync must not return a error sync_test.go:2784: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2784 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2789 Error: Should be false Test: TestSyncConcurrentDelete Messages: Low level errors must not have happened 2026/09/05 02:54:39 DEBUG : pacer: Reducing sleep to 320ms fstest.go:299: Sleeping for 1s for list eventual consistency: 1/3 2026/09/05 02:54:40 DEBUG : pacer: Reducing sleep to 160ms fstest.go:299: Sleeping for 2s for list eventual consistency: 2/3 2026/09/05 02:54:42 DEBUG : pacer: Reducing sleep to 80ms fstest.go:299: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:306: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:306 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:358 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2785 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2789 /usr/local/go/src/runtime/asm_amd64.s:1264 Error: Should be true Test: TestSyncConcurrentDelete Messages: listing wrong, want both0 (6), both1 (6), both10 (6), both11 (6), both12 (6), both13 (6), both14 (6), both15 (6), both16 (6), both17 (6), both18 (6), both19 (6), both2 (6), both3 (6), both4 (6), both5 (6), both6 (6), both7 (6), both8 (6), both9 (6) got both0 (6), both1 (6), both10 (6), both11 (6), both12 (6), both13 (6), both14 (6), both15 (6), both16 (6), both17 (6), both18 (6), both19 (6), both2 (6), both3 (6), both4 (6), both5 (6), both6 (6), both7 (6), both8 (6), both9 (6), only19 (6) fstest.go:192: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:192 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:309 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:358 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2785 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:2789 Error: Should be true Test: TestSyncConcurrentDelete Messages: Unexpected file "only19" 2026/09/05 02:54:46 DEBUG : pacer: Reducing sleep to 40ms 2026/09/05 02:54:46 DEBUG : pacer: Reducing sleep to 20ms 2026/09/05 02:54:46 DEBUG : pacer: Reducing sleep to 10ms 2026/09/05 02:54:49 DEBUG : pacer: low level retry 1/10 (error HTTP error 500 (500 Internal Server Error) returned body: "ENTRY_NOT_FOUND") 2026/09/05 02:54:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/05 02:54:50 DEBUG : pacer: Reducing sleep to 10ms --- FAIL: TestSyncConcurrentDelete (98.21s) FAIL 2026/09/05 02:54:50 DEBUG : Shade drive cf193d00-c783-4bab-aa7a-fd84b3700d27 path rclone-test-toqahuv7refe: Purge dir "" "./sync.test -test.v -test.timeout 1h0m0s -remote TestShade: -verbose -test.run '^TestSyncConcurrentDelete$'" - Finished ERROR in 1m38.962808805s (try 4/5): exit status 1: Failed [TestSyncConcurrentDelete]