"./gitannex.test -test.v -test.timeout 2h0m0s -remote TestDrime: -verbose -test.run '^TestGitAnnexFstestBackendCases$/^(TransferAndCheckpresentWithRealisticKey|TransferStoreRelative)$'" - Starting (try 4/5) 2026/02/07 04:24:02 DEBUG : Creating backend with remote "TestDrime:rclone-test-losifur9zawa" 2026/02/07 04:24:02 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/02/07 04:24:04 DEBUG : Creating backend with remote "/tmp/rclone2735802647" === RUN TestGitAnnexFstestBackendCases === RUN TestGitAnnexFstestBackendCases/TransferStoreRelative run.go:185: Remote "drime root 'rclone-test-losifur9zawa'", Local "Local file system at /tmp/rclone2735802647", Modify Window "876000h0m0s" 2026/02/07 04:24:04 DEBUG : Creating backend with remote "TestDrime:rclone-test-losifur9zawa" 2026/02/07 04:24:04 DEBUG : Creating backend with remote "../../rclone2735802647" 2026/02/07 04:24:04 DEBUG : fs cache: renaming cache item "../../rclone2735802647" to be canonical "/tmp/rclone2735802647" 2026/02/07 04:24:04 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:24:14 DEBUG : KeyRelative: size = 5 OK 2026/02/07 04:24:14 INFO : file.txt: Copied (new) to: KeyRelative fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:301: Flushing the directory cache fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferStoreRelative Messages: listing wrong, want got KeyRelative (5) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferStoreRelative Messages: Unexpected file "KeyRelative" === RUN TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey run.go:185: Remote "drime root 'rclone-test-losifur9zawa'", Local "Local file system at /tmp/rclone2735802647", Modify Window "876000h0m0s" 2026/02/07 04:24:25 DEBUG : Creating backend with remote "TestDrime:rclone-test-losifur9zawa" 2026/02/07 04:24:26 DEBUG : Creating backend with remote "/tmp/rclone2735802647" 2026/02/07 04:24:26 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:24:30 DEBUG : SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key: size = 5 OK 2026/02/07 04:24:30 INFO : file.txt: Copied (new) to: SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:301: Flushing the directory cache fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:345 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1188 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1446 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey Messages: listing wrong, want SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key (5) got fstest.go:203: Not found "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" fstest.go:206: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:206 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:310 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:345 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1188 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1446 Error: Not equal: expected: 0 actual : 1 Test: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey Messages: 1 objects not found fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:301: Flushing the directory cache fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey Messages: listing wrong, want got SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key (5) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey Messages: Unexpected file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" 2026/02/07 04:24:49 DEBUG : pacer: low level retry 1/10 (error Error "Server Error") 2026/02/07 04:24:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/02/07 04:24:49 DEBUG : pacer: low level retry 2/10 (error Error "Server Error") 2026/02/07 04:24:49 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2026/02/07 04:24:49 DEBUG : pacer: low level retry 3/10 (error Error "Server Error") 2026/02/07 04:24:49 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2026/02/07 04:24:50 DEBUG : pacer: low level retry 4/10 (error Error "Server Error") 2026/02/07 04:24:50 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2026/02/07 04:24:50 DEBUG : pacer: low level retry 5/10 (error Error "Server Error") 2026/02/07 04:24:50 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2026/02/07 04:24:50 DEBUG : pacer: low level retry 6/10 (error Error "Server Error") 2026/02/07 04:24:50 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2026/02/07 04:24:51 DEBUG : pacer: low level retry 7/10 (error Error "Server Error") 2026/02/07 04:24:51 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2026/02/07 04:24:51 DEBUG : pacer: low level retry 8/10 (error Error "Server Error") 2026/02/07 04:24:51 DEBUG : pacer: Rate limited, increasing sleep to 2.56s 2026/02/07 04:24:52 DEBUG : pacer: low level retry 9/10 (error Error "Server Error") 2026/02/07 04:24:52 DEBUG : pacer: Rate limited, increasing sleep to 5.12s 2026/02/07 04:24:55 DEBUG : pacer: low level retry 10/10 (error Error "Server Error") 2026/02/07 04:24:55 DEBUG : pacer: Rate limited, increasing sleep to 10.24s run.go:130: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed - try 1/3: failed to delete item: Error "Server Error" 2026/02/07 04:25:00 DEBUG : pacer: low level retry 1/10 (error Error "Server Error") 2026/02/07 04:25:00 DEBUG : pacer: Rate limited, increasing sleep to 20s 2026/02/07 04:25:11 DEBUG : pacer: low level retry 2/10 (error Error "Server Error") 2026/02/07 04:25:31 DEBUG : pacer: low level retry 3/10 (error Error "Server Error") 2026/02/07 04:25:51 DEBUG : pacer: low level retry 4/10 (error Error "Server Error") 2026/02/07 04:26:11 DEBUG : pacer: low level retry 5/10 (error Error "Server Error") 2026/02/07 04:26:31 DEBUG : pacer: low level retry 6/10 (error Error "Server Error") 2026/02/07 04:26:51 DEBUG : pacer: low level retry 7/10 (error Error "Server Error") 2026/02/07 04:27:11 DEBUG : pacer: low level retry 8/10 (error Error "Server Error") 2026/02/07 04:27:31 DEBUG : pacer: low level retry 9/10 (error Error "Server Error") 2026/02/07 04:27:51 DEBUG : pacer: low level retry 10/10 (error Error "Server Error") run.go:130: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed - try 2/3: failed to delete item: Error "Server Error" 2026/02/07 04:28:11 DEBUG : pacer: low level retry 1/10 (error Error "Server Error") 2026/02/07 04:28:31 DEBUG : pacer: low level retry 2/10 (error Error "Server Error") 2026/02/07 04:28:51 DEBUG : pacer: low level retry 3/10 (error Error "Server Error") 2026/02/07 04:29:11 DEBUG : pacer: low level retry 4/10 (error Error "Server Error") 2026/02/07 04:29:31 DEBUG : pacer: low level retry 5/10 (error Error "Server Error") 2026/02/07 04:29:51 DEBUG : pacer: low level retry 6/10 (error Error "Server Error") 2026/02/07 04:30:11 DEBUG : pacer: low level retry 7/10 (error Error "Server Error") 2026/02/07 04:30:31 DEBUG : pacer: low level retry 8/10 (error Error "Server Error") 2026/02/07 04:30:51 DEBUG : pacer: low level retry 9/10 (error Error "Server Error") 2026/02/07 04:31:11 DEBUG : pacer: low level retry 10/10 (error Error "Server Error") run.go:130: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed - try 3/3: failed to delete item: Error "Server Error" run.go:133: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed: failed to delete item: Error "Server Error" 2026/02/07 04:31:31 DEBUG : pacer: Reducing sleep to 10s --- FAIL: TestGitAnnexFstestBackendCases (447.51s) --- FAIL: TestGitAnnexFstestBackendCases/TransferStoreRelative (21.74s) --- FAIL: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey (425.76s) FAIL 2026/02/07 04:31:31 DEBUG : drime root 'rclone-test-losifur9zawa': Purge remote 2026/02/07 04:31:52 DEBUG : pacer: Reducing sleep to 5s "./gitannex.test -test.v -test.timeout 2h0m0s -remote TestDrime: -verbose -test.run '^TestGitAnnexFstestBackendCases$/^(TransferAndCheckpresentWithRealisticKey|TransferStoreRelative)$'" - Finished ERROR in 7m49.423549647s (try 4/5): exit status 1: Failed [TestGitAnnexFstestBackendCases/TransferStoreRelative TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey]