"./operations.test -test.v -test.timeout 2h0m0s -remote TestInternxt: -list-retries 5 -verbose -test.run '^TestApplyTransforms$|^TestMultithreadCopy$/^(upload=true,size=62914560,streams=2|upload=true,size=62914561,streams=2)$'" - Starting (try 2/5) 2026/09/14 05:45:18 DEBUG : Creating backend with remote "TestInternxt:rclone-test-xeburuk3qefa" 2026/09/14 05:45:18 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/14 05:45:23 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/14 05:45:23 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/14 05:45:23 DEBUG : TestInternxt: Saved new token in config file 2026/09/14 05:45:23 DEBUG : Internxt root 'rclone-test-xeburuk3qefa': Persisted rotated token from user info, expiry: 2026-09-21 05:45:20 +0000 UTC 2026/09/14 05:45:34 DEBUG : Creating backend with remote "/tmp/rclone3162308404" === RUN TestMultithreadCopy run.go:198: Remote "Internxt root 'rclone-test-xeburuk3qefa'", Local "Local file system at /tmp/rclone3162308404", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } 2026/09/14 05:46:38 NOTICE: Internxt root 'rclone-test-xeburuk3qefa': Multipart upload aborted for chunksize-probe === RUN TestMultithreadCopy/upload=true,size=62914560,streams=2 2026/09/14 05:46:57 DEBUG : test-multithread-copy-true-62914560-2: multi-thread copy: disabling buffering because source is local disk multithread_test.go:226: file too small for multipart upload: multi-thread copy: failed to open chunk writer: file too small for multipart upload (minimum 104857600 bytes) === RUN TestMultithreadCopy/upload=true,size=62914561,streams=2 fstest.go:299: Sleeping for 1s for list eventual consistency: 1/5 fstest.go:299: Sleeping for 2s for list eventual consistency: 2/5 fstest.go:299: Sleeping for 4s for list eventual consistency: 3/5 fstest.go:299: Sleeping for 8s for list eventual consistency: 4/5 fstest.go:299: Sleeping for 16s for list eventual consistency: 5/5 fstest.go:306: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:306 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:352 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:194 Error: Should be true Test: TestMultithreadCopy/upload=true,size=62914561,streams=2 Messages: listing wrong, want test-multithread-copy-true-62914561-2 (62914561) got test-multithread-copy-true-62914560-2 (62914560), test-multithread-copy-true-62914561-2 (62914561) fstest.go:192: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:192 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:309 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:339 /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:351 /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:352 /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:194 /usr/local/go/src/runtime/asm_amd64.s:1264 Error: Should be true Test: TestMultithreadCopy/upload=true,size=62914561,streams=2 Messages: Unexpected file "test-multithread-copy-true-62914560-2" 2026/09/14 05:47:41 DEBUG : test-multithread-copy-true-62914561-2: multi-thread copy: disabling buffering because source is local disk multithread_test.go:226: file too small for multipart upload: multi-thread copy: failed to open chunk writer: file too small for multipart upload (minimum 104857600 bytes) --- FAIL: TestMultithreadCopy (143.48s) --- SKIP: TestMultithreadCopy/upload=true,size=62914560,streams=2 (18.68s) --- FAIL: TestMultithreadCopy/upload=true,size=62914561,streams=2 (43.76s) === RUN TestApplyTransforms 2026/09/14 05:47:58 DEBUG : Creating backend with remote "TestInternxt:rclone-test-sicuyor5cace" 2026/09/14 05:47:58 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/14 05:48:03 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/14 05:48:03 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/14 05:48:03 DEBUG : TestInternxt: Saved new token in config file 2026/09/14 05:48:03 DEBUG : Internxt root 'rclone-test-sicuyor5cace': Persisted rotated token from user info, expiry: 2026-09-21 05:48:03 +0000 UTC 2026/09/14 05:48:13 DEBUG : Creating backend with remote "/tmp/rclone143328426" run.go:198: Remote "Internxt root 'rclone-test-sicuyor5cace'", Local "Local file system at /tmp/rclone143328426", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } 2026/09/14 05:49:15 DEBUG : pacer: low level retry 1/1 (error failed to create file metadata: create meta: error code: 502 (status 502)) 2026/09/14 05:49:15 DEBUG : pacer: Rate limited, increasing sleep to 20ms run.go:299: Retry Put of "hello, world!" to Internxt root 'rclone-test-sicuyor5cace': 1/10 (failed to create file metadata: create meta: error code: 502 (status 502)) 2026/09/14 05:49:32 DEBUG : pacer: Reducing sleep to 15ms 2026/09/14 05:49:36 DEBUG : pacer: Reducing sleep to 11.25ms 2026/09/14 05:49:40 DEBUG : pacer: Reducing sleep to 10ms upper checkfile vs. lower remote (without normalization) 2026/09/14 05:49:48 ERROR : hello, world!: sum not found 2026/09/14 05:49:48 ERROR : HELLO, WORLD!: file not in Internxt root 'rclone-test-sicuyor5cace' 2026/09/14 05:49:48 NOTICE: Internxt root 'rclone-test-sicuyor5cace': 1 files missing 2026/09/14 05:49:48 NOTICE: 1 hashes missing 2026/09/14 05:49:48 NOTICE: Internxt root 'rclone-test-sicuyor5cace': 1 differences found 2026/09/14 05:49:48 NOTICE: Internxt root 'rclone-test-sicuyor5cace': 2 errors while checking upper checkfile vs. lower remote (with normalization) 2026/09/14 05:49:52 DEBUG : hello, world!: md5 = 65a8e27d8879283831b664bd8b7f0ad4 OK 2026/09/14 05:49:52 NOTICE: Internxt root 'rclone-test-sicuyor5cace': 0 differences found 2026/09/14 05:49:52 NOTICE: Internxt root 'rclone-test-sicuyor5cace': 1 matching files 2026/09/14 05:49:52 DEBUG : Creating backend with remote "TestInternxt:rclone-test-yecugap4maba" 2026/09/14 05:49:52 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/14 05:49:57 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/14 05:49:57 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/14 05:49:57 DEBUG : TestInternxt: Saved new token in config file 2026/09/14 05:49:57 DEBUG : Internxt root 'rclone-test-yecugap4maba': Persisted rotated token from user info, expiry: 2026-09-21 05:49:53 +0000 UTC 2026/09/14 05:50:05 DEBUG : Creating backend with remote "/tmp/rclone4059533813" run.go:198: Remote "Internxt root 'rclone-test-yecugap4maba'", Local "Local file system at /tmp/rclone4059533813", Modify Window "876000h0m0s" FileLimitsResponse { "maxUploadFileSize": 107374182400, "versioning": { "enabled": true, "maxFileSize": 20971520, "retentionDays": 30, "maxVersions": 20 } } lower checkfile vs. upper remote (without normalization) 2026/09/14 05:51:24 ERROR : HELLO, WORLD!: sum not found 2026/09/14 05:51:24 ERROR : hello, world!: file not in Internxt root 'rclone-test-yecugap4maba' 2026/09/14 05:51:24 NOTICE: Internxt root 'rclone-test-yecugap4maba': 1 files missing 2026/09/14 05:51:24 NOTICE: 1 hashes missing 2026/09/14 05:51:24 NOTICE: Internxt root 'rclone-test-yecugap4maba': 1 differences found 2026/09/14 05:51:24 NOTICE: Internxt root 'rclone-test-yecugap4maba': 2 errors while checking lower checkfile vs. upper remote (with normalization) 2026/09/14 05:51:30 DEBUG : HELLO, WORLD!: md5 = 65a8e27d8879283831b664bd8b7f0ad4 OK 2026/09/14 05:51:30 NOTICE: Internxt root 'rclone-test-yecugap4maba': 0 differences found 2026/09/14 05:51:30 NOTICE: Internxt root 'rclone-test-yecugap4maba': 1 matching files 2026/09/14 05:51:30 DEBUG : Creating backend with remote "TestInternxt:rclone-test-nawelal8poda" 2026/09/14 05:51:30 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/09/14 05:51:34 DEBUG : User info: rootFolderId=2661cf64-e104-478b-b737-48c3ac8984d2, bucket=697358bf16ceabcf39af85f2 2026/09/14 05:51:34 DEBUG : Saving config "token" in section "TestInternxt" of the config file 2026/09/14 05:51:34 DEBUG : TestInternxt: Saved new token in config file 2026/09/14 05:51:34 DEBUG : Internxt root 'rclone-test-nawelal8poda': Persisted rotated token from user info, expiry: 2026-09-21 05:51:33 +0000 UTC 2026/09/14 05:51:44 DEBUG : Creating backend with remote "/tmp/rclone1292624993" 2026/09/14 05:51:44 DEBUG : Config file has changed externally - reloading run.go:198: Remote "Internxt root 'rclone-test-nawelal8poda'", Local "Local file system at /tmp/rclone1292624993", Modify Window "876000h0m0s" 2026/09/14 05:51:54 DEBUG : pacer: low level retry 1/1 (error create folder: error code: 502 (status 502)) 2026/09/14 05:51:54 DEBUG : pacer: Rate limited, increasing sleep to 20ms run.go:255: Failed to mkdir "Internxt root 'rclone-test-nawelal8poda'": failed to make directory: can't create folder, create folder: error code: 502 (status 502) 2026/09/14 05:51:59 DEBUG : pacer: Reducing sleep to 15ms 2026/09/14 05:51:59 ERROR : error listing: directory not found 2026/09/14 05:51:59 DEBUG : Internxt root 'rclone-test-nawelal8poda': Purge dir "" 2026/09/14 05:52:05 DEBUG : pacer: Reducing sleep to 11.25ms 2026/09/14 05:52:05 NOTICE: purge failed to rmdir "": directory not found 2026/09/14 05:52:05 NOTICE: purge failed: directory not found 2026/09/14 05:52:22 DEBUG : Internxt root 'rclone-test-yecugap4maba': Purge object "HELLO, WORLD!" 2026/09/14 05:52:24 DEBUG : Internxt root 'rclone-test-yecugap4maba': Purge dir "" 2026/09/14 05:52:51 DEBUG : Internxt root 'rclone-test-sicuyor5cace': Purge object "hello, world!" 2026/09/14 05:52:53 DEBUG : Internxt root 'rclone-test-sicuyor5cace': Purge dir "" --- FAIL: TestApplyTransforms (309.96s) FAIL 2026/09/14 05:53:20 DEBUG : Internxt root 'rclone-test-xeburuk3qefa': Purge dir "" "./operations.test -test.v -test.timeout 2h0m0s -remote TestInternxt: -list-retries 5 -verbose -test.run '^TestApplyTransforms$|^TestMultithreadCopy$/^(upload=true,size=62914560,streams=2|upload=true,size=62914561,streams=2)$'" - Finished ERROR in 8m16.843583627s (try 2/5): exit status 1: Failed [TestMultithreadCopy/upload=true,size=62914561,streams=2 TestApplyTransforms]