"./operations.test -test.v -test.timeout 2h0m0s -remote TestInternxt: -list-retries 5 -verbose -test.run '^(TestCheckSumDownload|TestPurge|TestRmdirsNoLeaveRoot|TestRmdirsWithFilter)$|^TestMultithreadCopy$/^(upload=true,size=62914560,streams=2|upload=true,size=62914561,streams=2)$'" - Starting (try 2/5) 2026/09/01 02:00:50 DEBUG : Creating backend with remote "TestInternxt:rclone-test-cibenar7yali" 2026/09/01 02:00:50 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/01 02:00:50 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/01 02:00:50 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/01 02:00:50 DEBUG : TestInternxt: Saved new token in config file 2026/09/01 02:00:50 DEBUG : Internxt root 'rclone-test-cibenar7yali': Persisted rotated token from user info, expiry: 2026-09-08 02:00:50 +0000 UTC 2026/09/01 02:00:51 DEBUG : Creating backend with remote "/tmp/rclone1255950679" === RUN TestMultithreadCopy run.go:198: Remote "Internxt root 'rclone-test-cibenar7yali'", Local "Local file system at /tmp/rclone1255950679", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } 2026/09/01 02:00:59 NOTICE: Internxt root 'rclone-test-cibenar7yali': Multipart upload aborted for chunksize-probe === RUN TestMultithreadCopy/upload=true,size=62914560,streams=2 2026/09/01 02:01:05 DEBUG : test-multithread-copy-true-62914560-2: multi-thread copy: disabling buffering because source is local disk multithread_test.go:225: file too small for multipart upload: multi-thread copy: failed to open chunk writer: file too small for multipart upload (minimum 104857600 bytes) === RUN TestMultithreadCopy/upload=true,size=62914561,streams=2 fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:306: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:306 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:352 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:193 Error: Should be true Test: TestMultithreadCopy/upload=true,size=62914561,streams=2 Messages: listing wrong, want test-multithread-copy-true-62914561-2 (62914561) got test-multithread-copy-true-62914560-2 (62914560), test-multithread-copy-true-62914561-2 (62914561) fstest.go:192: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:192 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:309 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:352 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:193 /usr/local/go/src/runtime/asm_amd64.s:1771 Error: Should be true Test: TestMultithreadCopy/upload=true,size=62914561,streams=2 Messages: Unexpected file "test-multithread-copy-true-62914560-2" 2026/09/01 02:01:42 DEBUG : test-multithread-copy-true-62914561-2: multi-thread copy: disabling buffering because source is local disk multithread_test.go:225: file too small for multipart upload: multi-thread copy: failed to open chunk writer: file too small for multipart upload (minimum 104857600 bytes) --- FAIL: TestMultithreadCopy (54.03s) --- SKIP: TestMultithreadCopy/upload=true,size=62914560,streams=2 (5.99s) --- FAIL: TestMultithreadCopy/upload=true,size=62914561,streams=2 (36.80s) === RUN TestCheckSumDownload run.go:198: Remote "Internxt root 'rclone-test-cibenar7yali'", Local "Local file system at /tmp/rclone1255950679", Modify Window "876000h0m0s" 2026/09/01 02:01:45 DEBUG : Creating backend with remote "TestInternxt:rclone-test-cibenar7yali/data" 2026/09/01 02:01:45 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/01 02:01:45 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/01 02:01:45 DEBUG : TestInternxt: Saved new token in config file 2026/09/01 02:01:45 DEBUG : Internxt root 'rclone-test-cibenar7yali/data': Persisted rotated token from user info, expiry: 2026-09-08 02:01:45 +0000 UTC 2026/09/01 02:01:54 DEBUG : pacer: low level retry 1/10 (error failed to list all folders at offset 0: list folders: error code: 502 (status 502)) 2026/09/01 02:01:54 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/01 02:01:54 DEBUG : pacer: Reducing sleep to 15ms 2026/09/01 02:01:54 DEBUG : pacer: Reducing sleep to 11.25ms run.go:303: Failed to put "data/banana" to "Internxt root 'rclone-test-cibenar7yali'": failed to create file metadata: create meta: Folder not found (status 404) --- FAIL: TestCheckSumDownload (12.03s) === RUN TestPurge 2026/09/01 02:01:57 DEBUG : Creating backend with remote "TestInternxt:rclone-test-vubosus5doco" 2026/09/01 02:01:57 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/01 02:02:01 DEBUG : Internxt root 'rclone-test-vubosus5doco': getUserInfo transient error (attempt 1/3): failed to fetch user info: refresh token: error code: 502 (status 502) 2026/09/01 02:02:02 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/01 02:02:02 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/01 02:02:02 DEBUG : TestInternxt: Saved new token in config file 2026/09/01 02:02:02 DEBUG : Internxt root 'rclone-test-vubosus5doco': Persisted rotated token from user info, expiry: 2026-09-08 02:02:02 +0000 UTC 2026/09/01 02:02:02 DEBUG : Creating backend with remote "/tmp/rclone1286152960" run.go:198: Remote "Internxt root 'rclone-test-vubosus5doco'", Local "Local file system at /tmp/rclone1286152960", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } 2026/09/01 02:02:20 INFO : A2: Making directory 2026/09/01 02:02:21 INFO : A1/B2: Making directory 2026/09/01 02:02:21 INFO : A1/B2/C2: Making directory 2026/09/01 02:02:26 INFO : A1/B1/C3: Making directory 2026/09/01 02:02:26 INFO : A3: Making directory 2026/09/01 02:02:28 INFO : A3/B3: Making directory 2026/09/01 02:02:29 INFO : A3/B3/C4: Making directory 2026/09/01 02:02:34 DEBUG : Waiting for deletions to finish 2026/09/01 02:02:35 INFO : A1/B1/C1/one: Deleted 2026/09/01 02:02:38 DEBUG : pacer: low level retry 1/10 (error failed to list all folders at offset 0: list folders: error code: 502 (status 502)) 2026/09/01 02:02:38 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/01 02:02:39 DEBUG : pacer: Reducing sleep to 15ms 2026/09/01 02:02:39 DEBUG : pacer: Reducing sleep to 11.25ms 2026/09/01 02:02:40 DEBUG : pacer: Reducing sleep to 10ms 2026/09/01 02:02:41 DEBUG : removing 2 level 3 directories 2026/09/01 02:02:41 INFO : A1/B1/C3: Removing directory 2026/09/01 02:02:41 INFO : A1/B1/C1: Removing directory 2026/09/01 02:02:42 DEBUG : removing 1 level 2 directories 2026/09/01 02:02:42 INFO : A1/B1: Removing directory 2026/09/01 02:02:47 DEBUG : Waiting for deletions to finish 2026/09/01 02:02:51 INFO : A1/two: Deleted 2026/09/01 02:02:52 DEBUG : removing 2 level 3 directories 2026/09/01 02:02:52 INFO : A3/B3/C4: Removing directory 2026/09/01 02:02:52 INFO : A1/B2/C2: Removing directory 2026/09/01 02:02:54 DEBUG : removing 2 level 2 directories 2026/09/01 02:02:54 INFO : A3/B3: Removing directory 2026/09/01 02:02:54 INFO : A1/B2: Removing directory 2026/09/01 02:02:55 DEBUG : removing 3 level 1 directories 2026/09/01 02:02:55 INFO : A3: Removing directory 2026/09/01 02:02:55 INFO : A1: Removing directory 2026/09/01 02:02:55 INFO : A2: Removing directory 2026/09/01 02:02:55 ERROR : A3: Failed to rmdir: directory not empty 2026/09/01 02:02:55 ERROR : A1: Failed to rmdir: directory not empty 2026/09/01 02:02:55 DEBUG : removing 1 level 0 directories 2026/09/01 02:02:55 INFO : Internxt root 'rclone-test-vubosus5doco': Removing directory 2026/09/01 02:02:55 ERROR : Failed to rmdir: directory not empty operations_test.go:632: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:632 Error: Received unexpected error: failed to remove directories: 3 errors: last error: directory not empty Test: TestPurge 2026/09/01 02:02:56 DEBUG : Internxt root 'rclone-test-vubosus5doco': Purge dir "A3" 2026/09/01 02:02:58 DEBUG : Internxt root 'rclone-test-vubosus5doco': Purge dir "A2" 2026/09/01 02:02:58 NOTICE: purge failed to rmdir "A2": directory not found 2026/09/01 02:02:58 DEBUG : Internxt root 'rclone-test-vubosus5doco': Purge dir "A1" 2026/09/01 02:02:59 DEBUG : Internxt root 'rclone-test-vubosus5doco': Purge dir "" --- FAIL: TestPurge (62.68s) === RUN TestRmdirsNoLeaveRoot run.go:198: Remote "Internxt root 'rclone-test-cibenar7yali'", Local "Local file system at /tmp/rclone1255950679", Modify Window "876000h0m0s" 2026/09/01 02:03:06 INFO : A2: Making directory 2026/09/01 02:03:07 INFO : A1/B2: Making directory 2026/09/01 02:03:09 INFO : A1/B2/C2: Making directory 2026/09/01 02:03:10 INFO : A1/B1/C3: Making directory 2026/09/01 02:03:10 INFO : A3: Making directory 2026/09/01 02:03:10 INFO : A3/B3: Making directory 2026/09/01 02:03:12 INFO : A3/B3/C4: Making directory fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:302: Flushing the directory cache 2026/09/01 02:03:51 DEBUG : pacer: low level retry 1/10 (error failed to list all folders at offset 0: list folders: error code: 502 (status 502)) 2026/09/01 02:03:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2026/09/01 02:03:51 DEBUG : pacer: Reducing sleep to 15ms 2026/09/01 02:03:51 DEBUG : pacer: Reducing sleep to 11.25ms 2026/09/01 02:03:51 DEBUG : pacer: Reducing sleep to 10ms fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:302: Flushing the directory cache fstest.go:328: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:328 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:663 Error: Not equal: expected: []string{"A1", "A1/B1", "A1/B1/C1", "A1/B1/C3", "A1/B2", "A1/B2/C2", "A2", "A3", "A3/B3", "A3/B3/C4"} actual : []string{"A1", "A1/B1", "A1/B1/C1", "A1/B1/C3", "A1/B2", "A1/B2/C2", "A2", "A3", "A3/B3", "A3/B3/C4", "data"} Diff: --- Expected +++ Actual @@ -1,2 +1,2 @@ -([]string) (len=10) { +([]string) (len=11) { (string) (len=2) "A1", @@ -10,3 +10,4 @@ (string) (len=5) "A3/B3", - (string) (len=8) "A3/B3/C4" + (string) (len=8) "A3/B3/C4", + (string) (len=4) "data" } Test: TestRmdirsNoLeaveRoot Messages: directories 2026/09/01 02:04:32 DEBUG : removing 1 level 3 directories 2026/09/01 02:04:32 INFO : A3/B3/C4: Removing directory fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:302: Flushing the directory cache fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:302: Flushing the directory cache fstest.go:328: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:328 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:686 Error: Not equal: expected: []string{"A1", "A1/B1", "A1/B1/C1", "A1/B1/C3", "A1/B2", "A1/B2/C2", "A2", "A3", "A3/B3"} actual : []string{"A1", "A1/B1", "A1/B1/C1", "A1/B1/C3", "A1/B2", "A1/B2/C2", "A2", "A3", "A3/B3", "data"} Diff: --- Expected +++ Actual @@ -1,2 +1,2 @@ -([]string) (len=9) { +([]string) (len=10) { (string) (len=2) "A1", @@ -9,3 +9,4 @@ (string) (len=2) "A3", - (string) (len=5) "A3/B3" + (string) (len=5) "A3/B3", + (string) (len=4) "data" } Test: TestRmdirsNoLeaveRoot Messages: directories 2026/09/01 02:05:34 DEBUG : removing 2 level 3 directories 2026/09/01 02:05:34 INFO : A1/B2/C2: Removing directory 2026/09/01 02:05:34 INFO : A1/B1/C3: Removing directory 2026/09/01 02:05:35 DEBUG : removing 2 level 2 directories 2026/09/01 02:05:35 INFO : A3/B3: Removing directory 2026/09/01 02:05:35 INFO : A1/B2: Removing directory 2026/09/01 02:05:36 DEBUG : removing 3 level 1 directories 2026/09/01 02:05:36 INFO : data: Removing directory 2026/09/01 02:05:36 INFO : A2: Removing directory 2026/09/01 02:05:36 INFO : A3: Removing directory fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:302: Flushing the directory cache fstest.go:293: Sleeping for 6s just to make sure 2026/09/01 02:05:51 DEBUG : removing 1 level 3 directories 2026/09/01 02:05:51 INFO : A1/B1/C1: Removing directory 2026/09/01 02:05:51 DEBUG : removing 1 level 2 directories 2026/09/01 02:05:51 INFO : A1/B1: Removing directory 2026/09/01 02:05:52 DEBUG : removing 1 level 1 directories 2026/09/01 02:05:52 INFO : A1: Removing directory 2026/09/01 02:05:52 DEBUG : removing 1 level 0 directories 2026/09/01 02:05:52 INFO : Internxt root 'rclone-test-cibenar7yali': Removing directory 2026/09/01 02:05:54 ERROR : error listing: directory not found 2026/09/01 02:05:54 ERROR : error listing: directory not found --- FAIL: TestRmdirsNoLeaveRoot (174.54s) === RUN TestRmdirsWithFilter run.go:198: Remote "Internxt root 'rclone-test-cibenar7yali'", Local "Local file system at /tmp/rclone1255950679", Modify Window "876000h0m0s" 2026/09/01 02:06:09 INFO : A1: Making directory 2026/09/01 02:06:13 INFO : A1/B1: Making directory operations_test.go:787: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:787 Error: Received unexpected error: failed to make directory: can't create folder, create folder: Parent folder does not exist (status 404) Test: TestRmdirsWithFilter --- FAIL: TestRmdirsWithFilter (21.38s) FAIL 2026/09/01 02:06:15 DEBUG : Internxt root 'rclone-test-cibenar7yali': Purge dir "" 2026/09/01 02:06:16 NOTICE: purge failed to rmdir "": directory not empty "./operations.test -test.v -test.timeout 2h0m0s -remote TestInternxt: -list-retries 5 -verbose -test.run '^(TestCheckSumDownload|TestPurge|TestRmdirsNoLeaveRoot|TestRmdirsWithFilter)$|^TestMultithreadCopy$/^(upload=true,size=62914560,streams=2|upload=true,size=62914561,streams=2)$'" - Finished ERROR in 5m25.605660051s (try 2/5): exit status 1: Failed [TestMultithreadCopy/upload=true,size=62914561,streams=2 TestCheckSumDownload TestPurge TestRmdirsNoLeaveRoot TestRmdirsWithFilter]