"./operations.test -test.v -test.timeout 2h0m0s -remote TestDrime: -verbose -test.run '^(TestCopyURLDownloadHeaders|TestDirMove|TestMaxDelete|TestMaxDeleteSize|TestMoveFile|TestMultithreadCopyAbort|TestRemoveExisting|TestRmdirsNoLeaveRoot)$|^TestCheck$/^(1|2|3|4|5|6|7)$'" - Starting (try 2/5) 2026/05/04 04:36:51 DEBUG : Creating backend with remote "TestDrime:rclone-test-jiyilat5rawo" 2026/05/04 04:36:51 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2026/05/04 04:36:51 DEBUG : Creating backend with remote "/tmp/rclone1705160176" === RUN TestMultithreadCopyAbort run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure 2026/05/04 04:37:02 DEBUG : test-multithread-abort: multi-thread copy: disabling buffering because source is local disk 2026/05/04 04:37:02 DEBUG : test-multithread-abort: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 1 2026/05/04 04:37:02 DEBUG : test-multithread-abort: multi-thread copy: number of streams 4 was bigger than number of chunks 3 2026/05/04 04:37:02 DEBUG : test-multithread-abort: Starting multi-thread copy with 3 chunks of size 5Mi with 3 parallel streams 2026/05/04 04:37:02 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (5242880-10485760) size 5Mi starting 2026/05/04 04:37:02 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-5242880) size 5Mi starting 2026/05/04 04:37:02 DEBUG : Open with options = [RangeOption(0,5242879)] 2026/05/04 04:37:02 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 (10485760-10485761) size 1 starting 2026/05/04 04:37:02 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:02 DEBUG : Open with options = [RangeOption(5242880,10485759)] 2026/05/04 04:37:03 DEBUG : test-multithread-abort: multi-thread copy: chunk 1/3 (0-5242880) size 5Mi finished 2026/05/04 04:37:03 DEBUG : test-multithread-abort: multi-thread copy: chunk 2/3 (5242880-10485760) size 5Mi finished 2026/05/04 04:37:03 DEBUG : Returning error reader 2026/05/04 04:37:03 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:03 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 0/10: BOOM: simulated read failure 2026/05/04 04:37:03 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:04 DEBUG : Returning error reader 2026/05/04 04:37:04 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:04 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 1/10: BOOM: simulated read failure 2026/05/04 04:37:04 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:05 DEBUG : Returning error reader 2026/05/04 04:37:05 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:05 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 2/10: BOOM: simulated read failure 2026/05/04 04:37:05 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:06 DEBUG : Returning error reader 2026/05/04 04:37:06 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:06 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 3/10: BOOM: simulated read failure 2026/05/04 04:37:06 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:07 DEBUG : Returning error reader 2026/05/04 04:37:07 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:07 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 4/10: BOOM: simulated read failure 2026/05/04 04:37:07 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:08 DEBUG : Returning error reader 2026/05/04 04:37:08 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:08 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 5/10: BOOM: simulated read failure 2026/05/04 04:37:08 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:09 DEBUG : Returning error reader 2026/05/04 04:37:09 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:09 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 6/10: BOOM: simulated read failure 2026/05/04 04:37:09 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:10 DEBUG : Returning error reader 2026/05/04 04:37:10 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:10 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 7/10: BOOM: simulated read failure 2026/05/04 04:37:10 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:11 DEBUG : Returning error reader 2026/05/04 04:37:11 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:11 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 8/10: BOOM: simulated read failure 2026/05/04 04:37:11 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:12 DEBUG : Returning error reader 2026/05/04 04:37:12 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:12 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 9/10: BOOM: simulated read failure 2026/05/04 04:37:12 DEBUG : Open with options = [RangeOption(10485760,10485760)] 2026/05/04 04:37:13 DEBUG : Returning error reader 2026/05/04 04:37:13 DEBUG : BOOM: simulated read failure 2026/05/04 04:37:13 DEBUG : test-multithread-abort: Reopening on read failure after offset 0 bytes: retry 10/10: BOOM: simulated read failure 2026/05/04 04:37:13 DEBUG : test-multithread-abort: Reopen failed after offset 0 bytes read: failed to reopen: too many retries 2026/05/04 04:37:13 DEBUG : test-multithread-abort: multi-thread copy: chunk 3/3 failed: multi-thread copy: failed to write chunk: failed to upload part 3: Put "https://18b804eaa6bc9759e2278f8d5367f42f.r2.cloudflarestorage.com/drimestorage/uploads/17cfce5c-2fdf-4655-bb52-c14677ff3079/17cfce5c-2fdf-4655-bb52-c14677ff3079?uploadId=AEE3t-lK7cWaM_YHf3IjGwxwnJqrkPtor-_YwHsY3aHq-XSaxeJYekJBYhdluKJmA2fE0PafIvcrQxJqMJLIjLfIQakWfQsYbFKdDBexLZYppSOB6cqBh8TDfsXv7lZW70In5wSwVsnrmMkOkirxAAyGCk2B56qNtYNpD6n79_kLqdQxanuDnJHxQHpvDgY7rRy1gOVy8A-FNZcz1gJ1puBC-628PB6VhKtlBNc7mLNKHrTW_fv3RKAT4Xx9Bl2KAtoBk6u9BVutUs3wyyOPijOd-iDk7Pz0KG26ppf6LXaqJ5ZMagCzFp1ZZ27g9JgxIMwYx0BPDOsbfQTvoyPgykM&partNumber=3&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=6a6ba1504e777c9aad0cd55f765be59c%2F20260504%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20260504T043703Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Signature=3d082df4bbff6ae0e697863ac57edd688f7acbc88d89b453047171caa5dd1f10": BOOM: simulated read failure 2026/05/04 04:37:13 DEBUG : test-multithread-abort: multi-thread copy: cancelling transfer on exit --- PASS: TestMultithreadCopyAbort (23.34s) === RUN TestCheck run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" === RUN TestCheck/1 === RUN TestCheck/2 === RUN TestCheck/3 === RUN TestCheck/4 === RUN TestCheck/5 === RUN TestCheck/6 === RUN TestCheck/7 --- PASS: TestCheck (7.11s) --- PASS: TestCheck/1 (0.27s) --- PASS: TestCheck/2 (0.33s) --- PASS: TestCheck/3 (0.18s) --- PASS: TestCheck/4 (0.20s) --- PASS: TestCheck/5 (0.19s) --- PASS: TestCheck/6 (0.19s) --- PASS: TestCheck/7 (0.20s) === RUN TestMaxDelete run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure 2026/05/04 04:37:32 DEBUG : Waiting for deletions to finish 2026/05/04 04:37:32 ERROR : small: Got fatal error on delete: --max-delete threshold reached 2026/05/04 04:37:33 INFO : large: Deleted 2026/05/04 04:37:33 INFO : medium: Deleted operations_test.go:453: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:453 Error: Not equal: expected: 1 actual : 3 Test: TestMaxDelete --- FAIL: TestMaxDelete (12.33s) === RUN TestMaxDeleteSize run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" 2026/05/04 04:37:37 DEBUG : Waiting for deletions to finish 2026/05/04 04:37:37 ERROR : small: Got fatal error on delete: --max-delete-size threshold reached 2026/05/04 04:37:37 INFO : large: Deleted 2026/05/04 04:37:37 INFO : medium: Deleted operations_test.go:492: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:492 Error: Not equal: expected: 1 actual : 3 Test: TestMaxDeleteSize --- FAIL: TestMaxDeleteSize (4.44s) === RUN TestRmdirsNoLeaveRoot run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" 2026/05/04 04:37:42 INFO : A2: Making directory 2026/05/04 04:37:42 INFO : A1/B2: Making directory 2026/05/04 04:37:43 INFO : A1/B2/C2: Making directory 2026/05/04 04:37:43 INFO : A1/B1/C3: Making directory 2026/05/04 04:37:44 INFO : A3: Making directory 2026/05/04 04:37:44 INFO : A3/B3: Making directory 2026/05/04 04:37:44 INFO : A3/B3/C4: Making directory fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure 2026/05/04 04:37:56 DEBUG : removing 1 level 3 directories 2026/05/04 04:37:56 INFO : A3/B3/C4: Removing directory 2026/05/04 04:37:58 DEBUG : removing 2 level 3 directories 2026/05/04 04:37:58 INFO : A1/B2/C2: Removing directory 2026/05/04 04:37:58 INFO : A1/B1/C3: Removing directory 2026/05/04 04:37:59 DEBUG : removing 2 level 2 directories 2026/05/04 04:37:59 INFO : A3/B3: Removing directory 2026/05/04 04:37:59 INFO : A1/B2: Removing directory 2026/05/04 04:37:59 DEBUG : removing 2 level 1 directories 2026/05/04 04:37:59 INFO : A3: Removing directory 2026/05/04 04:37:59 INFO : A2: Removing directory 2026/05/04 04:38:02 DEBUG : removing 1 level 3 directories 2026/05/04 04:38:02 INFO : A1/B1/C1: Removing directory 2026/05/04 04:38:03 DEBUG : removing 1 level 2 directories 2026/05/04 04:38:03 INFO : A1/B1: Removing directory 2026/05/04 04:38:03 ERROR : A1/B1: Failed to rmdir: directory not empty operations_test.go:728: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:728 Error: Received unexpected error: failed to remove directories: directory not empty Test: TestRmdirsNoLeaveRoot --- FAIL: TestRmdirsNoLeaveRoot (26.45s) === RUN TestCopyURLDownloadHeaders run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure --- PASS: TestCopyURLDownloadHeaders (9.00s) === RUN TestMoveFile run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" 2026/05/04 04:38:14 DEBUG : file1: Need to transfer - File not found at Destination 2026/05/04 04:38:16 DEBUG : sub/file2: size = 14 OK 2026/05/04 04:38:16 INFO : file1: Copied (new) to: sub/file2 2026/05/04 04:38:16 INFO : file1: Deleted 2026/05/04 04:38:16 DEBUG : sub/file2: size = 14 OK 2026/05/04 04:38:16 DEBUG : file1: Sizes identical 2026/05/04 04:38:16 DEBUG : file1: Unchanged skipping 2026/05/04 04:38:16 INFO : file1: Deleted 2026/05/04 04:38:17 DEBUG : drime root 'rclone-test-jiyilat5rawo': don't need to copy/move sub/file2, it is already at target location run.go:130: removing dir "sub" failed - try 1/3: directory not empty --- PASS: TestMoveFile (5.36s) === RUN TestDirMove run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" 2026/05/04 04:38:25 INFO : A1/B2: Making directory 2026/05/04 04:38:26 INFO : A1/B1/C3: Making directory operations_test.go:1411: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1411 Error: Received unexpected error: Move rename leaf: failed to patch item "name" to A2: Error "520 (520): {\"type\":\"https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-520/\",\"title\":\"Error 520: Web server is returning an unknown error\",\"status\":520,\"detail\":\"The origin web server sent a response that Cloudflare could not parse. This may indicate the origin returned an empty response, malformed HTTP headers, or an otherwise invalid response.\",\"instance\":\"9f64d0e9d83cdbbd\",\"error_code\":520,\"error_name\":\"unknown_origin_error\",\"error_category\":\"origin\",\"ray_id\":\"9f64d0e9d83cdbbd\",\"timestamp\":\"2026-05-04T04:38:28Z\",\"zone\":\"app.drime.cloud\",\"cloudflare_error\":true,\"retryable\":true,\"retry_after\":60,\"owner_action_required\":true,\"what_you_should_do\":\"**Wait and retry.** Back off for at least 60 seconds. If the error persists, the website operator should check the origin server error logs and submit them to Cloudflare support.\",\"footer\":\"This error was generated by Cloudflare on behalf of the website owner.\"}" Test: TestDirMove --- FAIL: TestDirMove (13.39s) === RUN TestRemoveExisting run.go:185: Remote "drime root 'rclone-test-jiyilat5rawo'", Local "Local file system at /tmp/rclone1705160176", Modify Window "876000h0m0s" fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3 fstest.go:301: Flushing the directory cache fstest.go:292: Sleeping for 6s just to make sure 2026/05/04 04:38:44 DEBUG : sub dir/test remove existing: TEST: renaming existing object to "sub dir/test remove existing.kasaqum7" before starting operations_test.go:1981: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1981 Error: Not equal: expected: *fmt.wrapError(&fmt.wrapError{msg:"TEST: failed to rename existing file: Move rename leaf: failed to patch item \"name\" to test remove existing.kasaqum7: Error \"520 (520): {\\\"type\\\":\\\"https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-520/\\\",\\\"title\\\":\\\"Error 520: Web server is returning an unknown error\\\",\\\"status\\\":520,\\\"detail\\\":\\\"The origin web server sent a response that Cloudflare could not parse. This may indicate the origin returned an empty response, malformed HTTP headers, or an otherwise invalid response.\\\",\\\"instance\\\":\\\"9f64d151ba41dbbd\\\",\\\"error_code\\\":520,\\\"error_name\\\":\\\"unknown_origin_error\\\",\\\"error_category\\\":\\\"origin\\\",\\\"ray_id\\\":\\\"9f64d151ba41dbbd\\\",\\\"timestamp\\\":\\\"2026-05-04T04:38:44Z\\\",\\\"zone\\\":\\\"app.drime.cloud\\\",\\\"cloudflare_error\\\":true,\\\"retryable\\\":true,\\\"retry_after\\\":60,\\\"owner_action_required\\\":true,\\\"what_you_should_do\\\":\\\"**Wait and retry.** Back off for at least 60 seconds. If the error persists, the website operator should check the origin server error logs and submit them to Cloudflare support.\\\",\\\"footer\\\":\\\"This error was generated by Cloudflare on behalf of the website owner.\\\"}\"", err:(*fmt.wrapError)(0x3d35191fb980)}) actual : () Test: TestRemoveExisting operations_test.go:1989: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1989 Error: " 11 sub dir/test remove existing\n 21 sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890123456789\n" should not contain " 11 sub dir/test remove existing\n" Test: TestRemoveExisting operations_test.go:1990: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1990 Error: " 11 sub dir/test remove existing\n 21 sub dir/test remove existing with long name 123456789012345678901234567890123456789012345678901234567890123456789\n" does not contain " 11 sub dir/test remove existing." Test: TestRemoveExisting run.go:130: removing dir "sub dir" failed - try 1/3: directory not empty --- FAIL: TestRemoveExisting (14.64s) panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked] [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x21acb64] goroutine 1245 [running]: testing.tRunner.func1.2({0x2544d40, 0x461eb10}) /usr/local/go/src/testing/testing.go:1974 +0x232 testing.tRunner.func1() /usr/local/go/src/testing/testing.go:1977 +0x349 panic({0x2544d40?, 0x461eb10?}) /usr/local/go/src/runtime/panic.go:860 +0x13a github.com/rclone/rclone/fs/operations_test.TestRemoveExisting(0x3d351910cb48) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:1993 +0x8c4 testing.tRunner(0x3d351910cb48, 0x2bd3190) /usr/local/go/src/testing/testing.go:2036 +0xea created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:2101 +0x4c5 "./operations.test -test.v -test.timeout 2h0m0s -remote TestDrime: -verbose -test.run '^(TestCopyURLDownloadHeaders|TestDirMove|TestMaxDelete|TestMaxDeleteSize|TestMoveFile|TestMultithreadCopyAbort|TestRemoveExisting|TestRmdirsNoLeaveRoot)$|^TestCheck$/^(1|2|3|4|5|6|7)$'" - Finished ERROR in 1m56.748446362s (try 2/5): exit status 2: Failed [TestMaxDelete TestMaxDeleteSize TestRmdirsNoLeaveRoot TestDirMove TestRemoveExisting]