"./gitannex.test -test.v -test.timeout 2h0m0s -remote TestDrime: -verbose -test.run '^TestGitAnnexFstestBackendCases$/^(CheckPresentAndTransfer|CheckpresentTransferCheckpresent|Remove|RemovePreexistingFile|TransferAndCheckpresentWithRealisticKey|TransferStoreRelative)$'" - Starting (try 2/5) 2026/02/07 04:12:52 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:12:52 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/02/07 04:12:53 DEBUG : Creating backend with remote "/tmp/rclone10713644" === RUN TestGitAnnexFstestBackendCases === RUN TestGitAnnexFstestBackendCases/TransferStoreRelative run.go:185: Remote "drime root 'rclone-test-yesipuq7jayo'", Local "Local file system at /tmp/rclone10713644", Modify Window "876000h0m0s" 2026/02/07 04:12:53 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:12:54 DEBUG : Creating backend with remote "../../rclone10713644" 2026/02/07 04:12:54 DEBUG : fs cache: renaming cache item "../../rclone10713644" to be canonical "/tmp/rclone10713644" 2026/02/07 04:12:54 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:12:58 DEBUG : KeyRelative: size = 5 OK 2026/02/07 04:12:58 INFO : file.txt: Copied (new) to: KeyRelative 2026/02/07 04:12:58 ERROR : error listing: directory not found fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache 2026/02/07 04:13:00 ERROR : error listing: directory not found fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache 2026/02/07 04:13:02 ERROR : error listing: directory not found fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:301: Flushing the directory cache fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:345 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1047 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1446 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferStoreRelative Messages: listing wrong, want KeyRelative (5) got fstest.go:203: Not found "KeyRelative" fstest.go:206: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:206 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:310 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:350 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:345 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1047 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1446 Error: Not equal: expected: 0 actual : 1 Test: TestGitAnnexFstestBackendCases/TransferStoreRelative Messages: 1 objects not found fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:301: Flushing the directory cache fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferStoreRelative Messages: listing wrong, want got KeyRelative (5) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferStoreRelative Messages: Unexpected file "KeyRelative" === RUN TestGitAnnexFstestBackendCases/CheckPresentAndTransfer run.go:185: Remote "drime root 'rclone-test-yesipuq7jayo'", Local "Local file system at /tmp/rclone10713644", Modify Window "876000h0m0s" 2026/02/07 04:13:17 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:13:17 DEBUG : Creating backend with remote "/tmp/rclone10713644" 2026/02/07 04:13:17 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:13:23 DEBUG : KeyAbsolute: size = 5 OK 2026/02/07 04:13:23 INFO : file.txt: Copied (new) to: KeyAbsolute fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 7s just to make sure fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure === RUN TestGitAnnexFstestBackendCases/CheckpresentTransferCheckpresent run.go:185: Remote "drime root 'rclone-test-yesipuq7jayo'", Local "Local file system at /tmp/rclone10713644", Modify Window "876000h0m0s" 2026/02/07 04:13:44 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:13:44 DEBUG : Creating backend with remote "/tmp/rclone10713644" 2026/02/07 04:13:45 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:13:47 DEBUG : foo: size = 5 OK 2026/02/07 04:13:47 INFO : file.txt: Copied (new) to: foo fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 7s just to make sure === RUN TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey run.go:185: Remote "drime root 'rclone-test-yesipuq7jayo'", Local "Local file system at /tmp/rclone10713644", Modify Window "876000h0m0s" 2026/02/07 04:14:08 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:14:09 DEBUG : Creating backend with remote "/tmp/rclone10713644" 2026/02/07 04:14:09 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:14:13 DEBUG : SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key: size = 5 OK 2026/02/07 04:14:13 INFO : file.txt: Copied (new) to: SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3 fstest.go:301: Flushing the directory cache fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3 fstest.go:301: Flushing the directory cache fstest.go:305: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:305 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey Messages: listing wrong, want got SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key (5) fstest.go:191: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:191 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:308 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:178 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 /home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1417 /usr/local/go/src/testing/testing.go:1211 /usr/local/go/src/testing/testing.go:1445 /usr/local/go/src/testing/testing.go:1786 Error: Should be true Test: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey Messages: Unexpected file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" 2026/02/07 04:14:24 DEBUG : pacer: low level retry 1/10 (error Error "Server Error") 2026/02/07 04:14:24 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/02/07 04:14:24 DEBUG : pacer: low level retry 2/10 (error Error "Server Error") 2026/02/07 04:14:24 DEBUG : pacer: Rate limited, increasing sleep to 40ms 2026/02/07 04:14:24 DEBUG : pacer: low level retry 3/10 (error Error "Server Error") 2026/02/07 04:14:24 DEBUG : pacer: Rate limited, increasing sleep to 80ms 2026/02/07 04:14:25 DEBUG : pacer: low level retry 4/10 (error Error "Server Error") 2026/02/07 04:14:25 DEBUG : pacer: Rate limited, increasing sleep to 160ms 2026/02/07 04:14:25 DEBUG : pacer: low level retry 5/10 (error Error "Server Error") 2026/02/07 04:14:25 DEBUG : pacer: Rate limited, increasing sleep to 320ms 2026/02/07 04:14:25 DEBUG : pacer: low level retry 6/10 (error Error "Server Error") 2026/02/07 04:14:25 DEBUG : pacer: Rate limited, increasing sleep to 640ms 2026/02/07 04:14:26 DEBUG : pacer: low level retry 7/10 (error Error "Server Error") 2026/02/07 04:14:26 DEBUG : pacer: Rate limited, increasing sleep to 1.28s 2026/02/07 04:14:26 DEBUG : pacer: low level retry 8/10 (error Error "Server Error") 2026/02/07 04:14:26 DEBUG : pacer: Rate limited, increasing sleep to 2.56s 2026/02/07 04:14:27 DEBUG : pacer: low level retry 9/10 (error Error "Server Error") 2026/02/07 04:14:27 DEBUG : pacer: Rate limited, increasing sleep to 5.12s 2026/02/07 04:14:30 DEBUG : pacer: low level retry 10/10 (error Error "Server Error") 2026/02/07 04:14:30 DEBUG : pacer: Rate limited, increasing sleep to 10.24s run.go:130: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed - try 1/3: failed to delete item: Error "Server Error" 2026/02/07 04:14:35 DEBUG : pacer: low level retry 1/10 (error Error "Server Error") 2026/02/07 04:14:35 DEBUG : pacer: Rate limited, increasing sleep to 20s 2026/02/07 04:14:45 DEBUG : pacer: low level retry 2/10 (error Error "Server Error") 2026/02/07 04:15:05 DEBUG : pacer: low level retry 3/10 (error Error "Server Error") 2026/02/07 04:15:25 DEBUG : pacer: low level retry 4/10 (error Error "Server Error") 2026/02/07 04:15:45 DEBUG : pacer: low level retry 5/10 (error Error "Server Error") 2026/02/07 04:16:05 DEBUG : pacer: low level retry 6/10 (error Error "Server Error") 2026/02/07 04:16:26 DEBUG : pacer: low level retry 7/10 (error Error "Server Error") 2026/02/07 04:16:45 DEBUG : pacer: low level retry 8/10 (error Error "Server Error") 2026/02/07 04:17:05 DEBUG : pacer: low level retry 9/10 (error Error "Server Error") 2026/02/07 04:17:25 DEBUG : pacer: low level retry 10/10 (error Error "Server Error") run.go:130: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed - try 2/3: failed to delete item: Error "Server Error" 2026/02/07 04:17:46 DEBUG : pacer: low level retry 1/10 (error Error "Server Error") 2026/02/07 04:18:07 DEBUG : pacer: low level retry 2/10 (error Error "Server Error") 2026/02/07 04:18:28 DEBUG : pacer: low level retry 3/10 (error Error "Server Error") 2026/02/07 04:18:48 DEBUG : pacer: low level retry 4/10 (error Error "Server Error") 2026/02/07 04:19:08 DEBUG : pacer: low level retry 5/10 (error Error "Server Error") 2026/02/07 04:19:27 DEBUG : pacer: low level retry 6/10 (error Error "Server Error") 2026/02/07 04:19:47 DEBUG : pacer: low level retry 7/10 (error Error "Server Error") 2026/02/07 04:20:08 DEBUG : pacer: low level retry 8/10 (error Error "Server Error") 2026/02/07 04:20:27 DEBUG : pacer: low level retry 9/10 (error Error "Server Error") 2026/02/07 04:20:47 DEBUG : pacer: low level retry 10/10 (error Error "Server Error") run.go:130: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed - try 3/3: failed to delete item: Error "Server Error" run.go:133: removing file "SHA256E-s1048576--7ba87e06b9b7903cfbaf4a38736766c161e3e7b42f06fe57f040aa410a8f0701.this-is-a-test-key" failed: failed to delete item: Error "Server Error" 2026/02/07 04:21:08 DEBUG : pacer: Reducing sleep to 10s === RUN TestGitAnnexFstestBackendCases/RemovePreexistingFile run.go:185: Remote "drime root 'rclone-test-yesipuq7jayo'", Local "Local file system at /tmp/rclone10713644", Modify Window "876000h0m0s" 2026/02/07 04:21:30 DEBUG : pacer: Reducing sleep to 5s 2026/02/07 04:21:46 DEBUG : pacer: Reducing sleep to 2.5s 2026/02/07 04:21:47 DEBUG : pacer: Reducing sleep to 1.25s 2026/02/07 04:21:47 DEBUG : Config file has changed externally - reloading 2026/02/07 04:21:47 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:21:55 DEBUG : pacer: Reducing sleep to 625ms 2026/02/07 04:22:04 INFO : SomeKey: Deleted 2026/02/07 04:22:08 DEBUG : pacer: Reducing sleep to 312.5ms fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache 2026/02/07 04:22:13 DEBUG : pacer: Reducing sleep to 156.25ms 2026/02/07 04:22:15 DEBUG : pacer: Reducing sleep to 78.125ms fstest.go:292: Sleeping for 6s just to make sure 2026/02/07 04:22:25 DEBUG : pacer: Reducing sleep to 39.0625ms 2026/02/07 04:22:27 DEBUG : pacer: Reducing sleep to 19.53125ms 2026/02/07 04:22:29 DEBUG : pacer: Reducing sleep to 10ms === RUN TestGitAnnexFstestBackendCases/Remove run.go:185: Remote "drime root 'rclone-test-yesipuq7jayo'", Local "Local file system at /tmp/rclone10713644", Modify Window "876000h0m0s" 2026/02/07 04:22:32 DEBUG : Config file has changed externally - reloading 2026/02/07 04:22:32 DEBUG : Creating backend with remote "TestDrime:rclone-test-yesipuq7jayo" 2026/02/07 04:22:34 DEBUG : Creating backend with remote "/tmp/rclone10713644" 2026/02/07 04:22:36 DEBUG : file.txt: Need to transfer - File not found at Destination 2026/02/07 04:22:49 DEBUG : SomeKey: size = 5 OK 2026/02/07 04:22:49 INFO : file.txt: Copied (new) to: SomeKey 2026/02/07 04:22:54 INFO : SomeKey: Deleted fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure --- FAIL: TestGitAnnexFstestBackendCases (618.77s) --- FAIL: TestGitAnnexFstestBackendCases/TransferStoreRelative (23.48s) --- PASS: TestGitAnnexFstestBackendCases/CheckPresentAndTransfer (26.99s) --- PASS: TestGitAnnexFstestBackendCases/CheckpresentTransferCheckpresent (24.60s) --- FAIL: TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey (419.52s) --- PASS: TestGitAnnexFstestBackendCases/RemovePreexistingFile (83.96s) --- PASS: TestGitAnnexFstestBackendCases/Remove (40.21s) FAIL 2026/02/07 04:23:12 DEBUG : drime root 'rclone-test-yesipuq7jayo': Purge remote "./gitannex.test -test.v -test.timeout 2h0m0s -remote TestDrime: -verbose -test.run '^TestGitAnnexFstestBackendCases$/^(CheckPresentAndTransfer|CheckpresentTransferCheckpresent|Remove|RemovePreexistingFile|TransferAndCheckpresentWithRealisticKey|TransferStoreRelative)$'" - Finished ERROR in 10m21.021887403s (try 2/5): exit status 1: Failed [TestGitAnnexFstestBackendCases/TransferStoreRelative TestGitAnnexFstestBackendCases/TransferAndCheckpresentWithRealisticKey]