"./operations.test -test.v -test.timeout 2h0m0s -remote TestInternxt: -list-retries 5 -verbose -test.run '^(TestCopyFileBackupDir|TestPurge|TestRemoveExisting|TestRmdirsLeaveRoot|TestRmdirsNoLeaveRoot|TestRmdirsWithFilter)$|^TestMultithreadCopy$/^(upload=true,size=62914560,streams=2|upload=true,size=62914561,streams=2)$'" - Starting (try 2/5) 2026/08/31 05:53:59 DEBUG : Creating backend with remote "TestInternxt:rclone-test-qotimav9pisa" 2026/08/31 05:53:59 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/08/31 05:53:59 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/08/31 05:53:59 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/08/31 05:53:59 DEBUG : TestInternxt: Saved new token in config file 2026/08/31 05:53:59 DEBUG : Internxt root 'rclone-test-qotimav9pisa': Persisted rotated token from user info, expiry: 2026-09-07 05:53:59 +0000 UTC 2026/08/31 05:53:59 DEBUG : Creating backend with remote "/tmp/rclone543845900" === RUN TestMultithreadCopy run.go:198: Remote "Internxt root 'rclone-test-qotimav9pisa'", Local "Local file system at /tmp/rclone543845900", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } 2026/08/31 05:54:07 NOTICE: Internxt root 'rclone-test-qotimav9pisa': Multipart upload aborted for chunksize-probe === RUN TestMultithreadCopy/upload=true,size=62914560,streams=2 2026/08/31 05:54:12 DEBUG : test-multithread-copy-true-62914560-2: multi-thread copy: disabling buffering because source is local disk multithread_test.go:225: file too small for multipart upload: multi-thread copy: failed to open chunk writer: file too small for multipart upload (minimum 104857600 bytes) === RUN TestMultithreadCopy/upload=true,size=62914561,streams=2 fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:306: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:306 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:352 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:193 Error: Should be true Test: TestMultithreadCopy/upload=true,size=62914561,streams=2 Messages: listing wrong, want test-multithread-copy-true-62914561-2 (62914561) got test-multithread-copy-true-62914560-2 (62914560), test-multithread-copy-true-62914561-2 (62914561) fstest.go:192: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:192 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:309 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:352 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:193 /usr/local/go/src/runtime/asm_amd64.s:1771 Error: Should be true Test: TestMultithreadCopy/upload=true,size=62914561,streams=2 Messages: Unexpected file "test-multithread-copy-true-62914560-2" 2026/08/31 05:54:49 DEBUG : test-multithread-copy-true-62914561-2: multi-thread copy: disabling buffering because source is local disk multithread_test.go:225: file too small for multipart upload: multi-thread copy: failed to open chunk writer: file too small for multipart upload (minimum 104857600 bytes) --- FAIL: TestMultithreadCopy (50.99s) --- SKIP: TestMultithreadCopy/upload=true,size=62914560,streams=2 (5.38s) --- FAIL: TestMultithreadCopy/upload=true,size=62914561,streams=2 (36.31s) === RUN TestCopyFileBackupDir run.go:198: Remote "Internxt root 'rclone-test-qotimav9pisa'", Local "Local file system at /tmp/rclone543845900", Modify Window "876000h0m0s" 2026/08/31 05:54:52 DEBUG : Creating backend with remote "TestInternxt:rclone-test-qotimav9pisa/backup" 2026/08/31 05:54:53 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/08/31 05:54:53 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/08/31 05:54:53 DEBUG : TestInternxt: Saved new token in config file 2026/08/31 05:54:53 DEBUG : Internxt root 'rclone-test-qotimav9pisa/backup': Persisted rotated token from user info, expiry: 2026-09-07 05:54:53 +0000 UTC 2026/08/31 05:54:53 DEBUG : dst/file1: size = 14 (Local file system at /tmp/rclone543845900) 2026/08/31 05:54:53 DEBUG : dst/file1: size = 18 (Internxt root 'rclone-test-qotimav9pisa') 2026/08/31 05:54:53 DEBUG : dst/file1: Sizes differ 2026/08/31 05:54:56 INFO : dst/file1: Moved (server-side) 2026/08/31 05:54:57 DEBUG : dst/file1: size = 14 OK 2026/08/31 05:54:57 INFO : dst/file1: Copied (new) run.go:186: removing dir "dst" failed - try 1/5: sleeping 1s: directory not empty --- PASS: TestCopyFileBackupDir (11.50s) === RUN TestPurge 2026/08/31 05:55:02 DEBUG : Creating backend with remote "TestInternxt:rclone-test-suyusog0konu" 2026/08/31 05:55:02 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/08/31 05:55:02 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/08/31 05:55:02 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/08/31 05:55:02 DEBUG : TestInternxt: Saved new token in config file 2026/08/31 05:55:02 DEBUG : Internxt root 'rclone-test-suyusog0konu': Persisted rotated token from user info, expiry: 2026-09-07 05:55:02 +0000 UTC 2026/08/31 05:55:03 DEBUG : Creating backend with remote "/tmp/rclone1896907674" run.go:198: Remote "Internxt root 'rclone-test-suyusog0konu'", Local "Local file system at /tmp/rclone1896907674", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } 2026/08/31 05:55:18 INFO : A2: Making directory 2026/08/31 05:55:19 INFO : A1/B2: Making directory 2026/08/31 05:55:22 INFO : A1/B2/C2: Making directory 2026/08/31 05:55:22 INFO : A1/B1/C3: Making directory 2026/08/31 05:55:23 INFO : A3: Making directory 2026/08/31 05:55:23 INFO : A3/B3: Making directory 2026/08/31 05:55:24 INFO : A3/B3/C4: Making directory 2026/08/31 05:55:28 DEBUG : Waiting for deletions to finish 2026/08/31 05:55:29 INFO : A1/B1/C1/one: Deleted 2026/08/31 05:55:31 DEBUG : removing 2 level 3 directories 2026/08/31 05:55:31 INFO : A1/B1/C3: Removing directory 2026/08/31 05:55:31 INFO : A1/B1/C1: Removing directory 2026/08/31 05:55:31 DEBUG : removing 1 level 2 directories 2026/08/31 05:55:31 INFO : A1/B1: Removing directory 2026/08/31 05:55:34 DEBUG : Waiting for deletions to finish 2026/08/31 05:55:36 INFO : A1/two: Deleted 2026/08/31 05:55:38 DEBUG : removing 2 level 3 directories 2026/08/31 05:55:38 INFO : A3/B3/C4: Removing directory 2026/08/31 05:55:38 INFO : A1/B2/C2: Removing directory 2026/08/31 05:55:39 DEBUG : removing 2 level 2 directories 2026/08/31 05:55:39 INFO : A3/B3: Removing directory 2026/08/31 05:55:39 INFO : A1/B2: Removing directory 2026/08/31 05:55:39 DEBUG : removing 3 level 1 directories 2026/08/31 05:55:39 INFO : A3: Removing directory 2026/08/31 05:55:39 INFO : A1: Removing directory 2026/08/31 05:55:39 INFO : A2: Removing directory 2026/08/31 05:55:44 DEBUG : removing 1 level 0 directories 2026/08/31 05:55:44 INFO : Internxt root 'rclone-test-suyusog0konu': Removing directory 2026/08/31 05:55:44 ERROR : error listing: directory not found 2026/08/31 05:55:44 ERROR : error listing: directory not found 2026/08/31 05:55:44 DEBUG : Internxt root 'rclone-test-suyusog0konu': Purge dir "" 2026/08/31 05:55:45 NOTICE: purge failed to rmdir "": directory not found 2026/08/31 05:55:45 NOTICE: purge failed: directory not found --- PASS: TestPurge (43.29s) === RUN TestRmdirsNoLeaveRoot run.go:198: Remote "Internxt root 'rclone-test-qotimav9pisa'", Local "Local file system at /tmp/rclone543845900", Modify Window "876000h0m0s" 2026/08/31 05:55:48 INFO : A2: Making directory 2026/08/31 05:55:50 INFO : A1/B2: Making directory 2026/08/31 05:55:50 INFO : A1/B2/C2: Making directory 2026/08/31 05:55:51 INFO : A1/B1/C3: Making directory 2026/08/31 05:55:51 INFO : A3: Making directory 2026/08/31 05:55:52 INFO : A3/B3: Making directory 2026/08/31 05:55:54 INFO : A3/B3/C4: Making directory 2026/08/31 05:56:01 DEBUG : removing 1 level 3 directories 2026/08/31 05:56:01 INFO : A3/B3/C4: Removing directory 2026/08/31 05:56:09 DEBUG : removing 2 level 3 directories 2026/08/31 05:56:09 INFO : A1/B2/C2: Removing directory 2026/08/31 05:56:09 INFO : A1/B1/C3: Removing directory 2026/08/31 05:56:11 DEBUG : removing 2 level 2 directories 2026/08/31 05:56:11 INFO : A3/B3: Removing directory 2026/08/31 05:56:11 INFO : A1/B2: Removing directory 2026/08/31 05:56:11 ERROR : A1/B2: Failed to rmdir: directory not empty 2026/08/31 05:56:11 DEBUG : removing 2 level 1 directories 2026/08/31 05:56:11 INFO : A3: Removing directory 2026/08/31 05:56:11 INFO : A2: Removing directory 2026/08/31 05:56:11 ERROR : A3: Failed to rmdir: directory not empty operations_test.go:706: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:706 Error: Received unexpected error: failed to remove directories: 2 errors: last error: directory not empty Test: TestRmdirsNoLeaveRoot 2026/08/31 05:56:24 DEBUG : pacer: low level retry 1/10 (error delete folder: error code: 502 (status 502)) 2026/08/31 05:56:24 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/08/31 05:56:24 DEBUG : pacer: Reducing sleep to 15ms run.go:186: removing dir "A1" failed - try 1/5: sleeping 1s: directory not found 2026/08/31 05:56:25 DEBUG : pacer: Reducing sleep to 11.25ms 2026/08/31 05:56:26 DEBUG : pacer: Reducing sleep to 10ms run.go:186: removing dir "A1" failed - try 2/5: sleeping 1.5s: directory not found run.go:186: removing dir "A1" failed - try 3/5: sleeping 2.25s: directory not found run.go:186: removing dir "A1" failed - try 4/5: sleeping 3.375s: directory not found run.go:186: removing dir "A1" failed: directory not found --- FAIL: TestRmdirsNoLeaveRoot (49.89s) === RUN TestRmdirsLeaveRoot run.go:198: Remote "Internxt root 'rclone-test-qotimav9pisa'", Local "Local file system at /tmp/rclone543845900", Modify Window "876000h0m0s" 2026/08/31 05:56:35 INFO : A1: Making directory 2026/08/31 05:56:35 INFO : A1/B1: Making directory 2026/08/31 05:56:35 INFO : A1/B1/C1: Making directory fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:302: Flushing the directory cache fstest.go:328: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:328 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:751 Error: Not equal: expected: []string{"A1", "A1/B1", "A1/B1/C1"} actual : []string{} Diff: --- Expected +++ Actual @@ -1,5 +1,2 @@ -([]string) (len=3) { - (string) (len=2) "A1", - (string) (len=5) "A1/B1", - (string) (len=8) "A1/B1/C1" +([]string) { } Test: TestRmdirsLeaveRoot Messages: directories 2026/08/31 05:57:11 ERROR : Internxt root 'rclone-test-qotimav9pisa': Failed to list "A1": directory not found fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:302: Flushing the directory cache fstest.go:328: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:328 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:765 Error: Not equal: expected: []string{"A1"} actual : []string{} Diff: --- Expected +++ Actual @@ -1,3 +1,2 @@ -([]string) (len=1) { - (string) (len=2) "A1" +([]string) { } Test: TestRmdirsLeaveRoot Messages: directories --- FAIL: TestRmdirsLeaveRoot (71.09s) === RUN TestRmdirsWithFilter run.go:198: Remote "Internxt root 'rclone-test-qotimav9pisa'", Local "Local file system at /tmp/rclone543845900", Modify Window "876000h0m0s" 2026/08/31 05:57:46 INFO : A1: Making directory 2026/08/31 05:57:49 INFO : A1/B1: Making directory 2026/08/31 05:57:50 INFO : A1/B1/C1: Making directory 2026/08/31 05:57:53 DEBUG : removing 1 level 3 directories 2026/08/31 05:57:53 INFO : A1/B1/C1: Removing directory 2026/08/31 05:57:53 DEBUG : removing 1 level 2 directories 2026/08/31 05:57:53 INFO : A1/B1: Removing directory 2026/08/31 05:57:54 ERROR : A1/B1: error listing: directory not found fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:293: Sleeping for 6s just to make sure --- PASS: TestRmdirsWithFilter (18.24s) === RUN TestRemoveExisting run.go:198: Remote "Internxt root 'rclone-test-qotimav9pisa'", Local "Local file system at /tmp/rclone543845900", Modify Window "876000h0m0s" 2026/08/31 05:58:10 DEBUG : sub dir/test remove existing: TEST: renaming existing object to "sub dir/test remove existing.hanexav5" before starting 2026/08/31 05:58:10 DEBUG : sub dir/test remove existing.hanexav5: TEST: removing renamed existing file after operation 2026/08/31 05:58:11 DEBUG : sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890123456789: TEST: renaming existing object to "sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890.buvehoy1" before starting 2026/08/31 05:58:13 DEBUG : sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890.buvehoy1: TEST: renaming existing back after failed operation 2026/08/31 05:58:13 DEBUG : sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890123456789: TEST: renaming existing object to "sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890.zoginiy1" before starting 2026/08/31 05:58:14 DEBUG : sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890.zoginiy1: TEST: removing renamed existing file after operation --- PASS: TestRemoveExisting (13.50s) FAIL 2026/08/31 05:58:18 DEBUG : Internxt root 'rclone-test-qotimav9pisa': Purge dir "" "./operations.test -test.v -test.timeout 2h0m0s -remote TestInternxt: -list-retries 5 -verbose -test.run '^(TestCopyFileBackupDir|TestPurge|TestRemoveExisting|TestRmdirsLeaveRoot|TestRmdirsNoLeaveRoot|TestRmdirsWithFilter)$|^TestMultithreadCopy$/^(upload=true,size=62914560,streams=2|upload=true,size=62914561,streams=2)$'" - Finished ERROR in 4m19.809534869s (try 2/5): exit status 1: Failed [TestMultithreadCopy/upload=true,size=62914561,streams=2 TestRmdirsNoLeaveRoot TestRmdirsLeaveRoot]