"./operations.test -test.v -test.timeout 1h0m0s -remote TestOneDrive: -verbose" - Starting (try 1/5) 2025/08/02 03:32:59 DEBUG : Creating backend with remote "TestOneDrive:rclone-test-resixiz9cipo" 2025/08/02 03:32:59 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/08/02 03:33:01 DEBUG : Creating backend with remote "/tmp/rclone766143044" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:180: Remote "OneDrive root 'rclone-test-resixiz9cipo'", Local "Local file system at /tmp/rclone766143044", Modify Window "1s" multithread_test.go:121: multithread writing not supported 2025/08/02 03:33:01 ERROR : error listing: directory not found --- SKIP: TestMultithreadCopy (0.51s) === RUN TestMultithreadCopyAbort run.go:180: Remote "OneDrive root 'rclone-test-resixiz9cipo'", Local "Local file system at /tmp/rclone766143044", Modify Window "1s" multithread_test.go:121: multithread writing not supported 2025/08/02 03:33:02 ERROR : error listing: directory not found --- SKIP: TestMultithreadCopyAbort (0.52s) === RUN TestSizeDiffers --- PASS: TestSizeDiffers (0.00s) === RUN TestReOpen === RUN TestReOpen/Normal === RUN TestReOpen/Normal/Basics 2025/08/02 03:33:02 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/ErrorAtStart === RUN TestReOpen/Normal/WithErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/Normal/TooManyErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/Normal/Seek 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Seek from 5 to 2 === RUN TestReOpen/Normal/AccountRead === RUN TestReOpen/Normal/AccountReadDelay 2025/08/02 03:33:02 DEBUG : potato: Seek from 10 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 10 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 10 to 0 === RUN TestReOpen/Normal/AccountReadError === RUN TestReOpen/WithRangeOption === RUN TestReOpen/WithRangeOption/Basics 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/ErrorAtStart === RUN TestReOpen/WithRangeOption/WithErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithRangeOption/TooManyErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithRangeOption/Seek 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Seek from 5 to 2 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 4 === RUN TestReOpen/WithRangeOption/AccountRead === RUN TestReOpen/WithRangeOption/AccountReadDelay 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 0 === RUN TestReOpen/WithRangeOption/AccountReadError === RUN TestReOpen/WithSeekOption === RUN TestReOpen/WithSeekOption/Basics 2025/08/02 03:33:02 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/ErrorAtStart === RUN TestReOpen/WithSeekOption/WithErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/WithSeekOption/TooManyErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/WithSeekOption/Seek 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Seek from 5 to 2 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 5 === RUN TestReOpen/WithSeekOption/AccountRead === RUN TestReOpen/WithSeekOption/AccountReadDelay 2025/08/02 03:33:02 DEBUG : potato: Seek from 8 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 8 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 8 to 0 === RUN TestReOpen/WithSeekOption/AccountReadError === RUN TestReOpen/UnknownSize === RUN TestReOpen/UnknownSize/Basics 2025/08/02 03:33:02 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/ErrorAtStart === RUN TestReOpen/UnknownSize/WithErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/10: test error === RUN TestReOpen/UnknownSize/TooManyErrors 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 1/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 2/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 6 bytes: retry 3/3: test error 2025/08/02 03:33:02 DEBUG : potato: Reopen failed after offset 6 bytes read: failed to reopen: too many retries === RUN TestReOpen/UnknownSize/Seek 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 2 bytes: retry 0/10: test error 2025/08/02 03:33:02 DEBUG : potato: Reopening on read failure after offset 3 bytes: retry 1/10: test error 2025/08/02 03:33:02 DEBUG : potato: Seek from 5 to 2 2025/08/02 03:33:02 DEBUG : potato: Seek from 7 to 6 === RUN TestReOpen/UnknownSize/AccountRead === RUN TestReOpen/UnknownSize/AccountReadDelay 2025/08/02 03:33:02 DEBUG : potato: Seek from 9 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 9 to 0 2025/08/02 03:33:02 DEBUG : potato: Seek from 9 to 0 === RUN TestReOpen/UnknownSize/AccountReadError --- PASS: TestReOpen (0.00s) --- PASS: TestReOpen/Normal (0.00s) --- PASS: TestReOpen/Normal/Basics (0.00s) --- PASS: TestReOpen/Normal/ErrorAtStart (0.00s) --- PASS: TestReOpen/Normal/WithErrors (0.00s) --- PASS: TestReOpen/Normal/TooManyErrors (0.00s) --- PASS: TestReOpen/Normal/Seek (0.00s) --- PASS: TestReOpen/Normal/AccountRead (0.00s) --- PASS: TestReOpen/Normal/AccountReadDelay (0.00s) --- PASS: TestReOpen/Normal/AccountReadError (0.00s) --- PASS: TestReOpen/WithRangeOption (0.00s) --- PASS: TestReOpen/WithRangeOption/Basics (0.00s) --- PASS: TestReOpen/WithRangeOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithRangeOption/WithErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithRangeOption/Seek (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountRead (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithRangeOption/AccountReadError (0.00s) --- PASS: TestReOpen/WithSeekOption (0.00s) --- PASS: TestReOpen/WithSeekOption/Basics (0.00s) --- PASS: TestReOpen/WithSeekOption/ErrorAtStart (0.00s) --- PASS: TestReOpen/WithSeekOption/WithErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/TooManyErrors (0.00s) --- PASS: TestReOpen/WithSeekOption/Seek (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountRead (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadDelay (0.00s) --- PASS: TestReOpen/WithSeekOption/AccountReadError (0.00s) --- PASS: TestReOpen/UnknownSize (0.00s) --- PASS: TestReOpen/UnknownSize/Basics (0.00s) --- PASS: TestReOpen/UnknownSize/ErrorAtStart (0.00s) --- PASS: TestReOpen/UnknownSize/WithErrors (0.00s) --- PASS: TestReOpen/UnknownSize/TooManyErrors (0.00s) --- PASS: TestReOpen/UnknownSize/Seek (0.00s) --- PASS: TestReOpen/UnknownSize/AccountRead (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadDelay (0.00s) --- PASS: TestReOpen/UnknownSize/AccountReadError (0.00s) === RUN TestCheck run.go:180: Remote "OneDrive root 'rclone-test-resixiz9cipo'", Local "Local file system at /tmp/rclone766143044", Modify Window "1s" 2025/08/02 03:33:02 DEBUG : Too many requests. Trying again in 45822 seconds. 2025/08/02 03:33:02 DEBUG : pacer: low level retry 1/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 12h43m42s) 2025/08/02 03:33:02 DEBUG : pacer: Rate limited, increasing sleep to 12h43m42s 2025/08/02 03:33:03 DEBUG : Too many requests. Trying again in 63822 seconds. 2025/08/02 03:33:03 DEBUG : pacer: low level retry 2/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 17h43m42s) 2025/08/02 03:33:03 DEBUG : pacer: Rate limited, increasing sleep to 17h43m42s 2025/08/02 04:12:49 DEBUG : OneDrive root 'rclone-test-resixiz9cipo': Token expired but no uploads in progress - doing nothing panic: test timed out after 1h0m0s running tests: TestCheck (59m59s) goroutine 135 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc000103dc0, {0x26a9cdc?, 0xc00006dad8?}, 0x27e2720) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc000103dc0) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc000103dc0, 0xc00006dc18) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc0009957b8, {0x3eb1080, 0x6d, 0x6d}, {0xee01f7ded?, 0x3ec7120?, 0x3ec7120?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc0000f8960) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc0000f8960) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:265 +0xa9 goroutine 36 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 37 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 38 [select, 21 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0000d92c0) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x86 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 130 [chan receive, 60 minutes]: github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall(0xc00039e540) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:161 +0x25 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc00039e540, 0xc000a82b00, 0xa) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:198 +0x5c github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc00039e540, 0xc000a82b00) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:218 +0x93 github.com/rclone/rclone/backend/onedrive.(*Fs).readMetaDataForPathRelativeToID(0xc000440480, {0x2b19e68, 0x3eea6e0}, {0xc00093d290, 0x25}, {0xc00015a2ad, 0x18}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:934 +0x2c9 github.com/rclone/rclone/backend/onedrive.(*Fs).FindLeaf(0xc000440480, {0x2b19e68, 0x3eea6e0}, {0xc00093d290, 0x25}, {0xc00015a2ad, 0x18}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1232 +0x77 github.com/rclone/rclone/lib/dircache.(*DirCache)._findDir(0xc000180b40, {0x2b19e68, 0x3eea6e0}, {0xc00015a2ad, 0x18}, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:234 +0x112 github.com/rclone/rclone/lib/dircache.(*DirCache)._findRoot(0xc000180b40, {0x2b19e68?, 0x3eea6e0?}, 0x20?) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:305 +0x45 github.com/rclone/rclone/lib/dircache.(*DirCache).FindDir(0xc000180b40, {0x2b19e68, 0x3eea6e0}, {0x0, 0x0}, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:201 +0xcd github.com/rclone/rclone/backend/onedrive.(*Fs).Mkdir(0x26c4df0?, {0x2b19e68?, 0x3eea6e0?}, {0x0?, 0x1?}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1566 +0x2f github.com/rclone/rclone/fstest.(*Run).ForceMkdir(0xc0004f8c40, {0x2b19e68?, 0x3eea6e0?}, {0x2b30a60, 0xc000440480}) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:235 +0x38 github.com/rclone/rclone/fstest.(*Run).Mkdir(0xc0004f8c40, {0x2b19e68, 0x3eea6e0}, {0x2b30a60, 0xc000440480}) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:245 +0x71 github.com/rclone/rclone/fstest.(*Run).WriteObjectTo(0xc0004f8c40, {0x2b19e68, 0x3eea6e0}, {0x2b30a60, 0xc000440480}, {0x26a75b4, 0x8}, {0x26a75bc, 0x8}, {0x0, ...}, ...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:258 +0x145 github.com/rclone/rclone/fstest.(*Run).WriteObject(...) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:292 github.com/rclone/rclone/fstest.(*Run).WriteBoth(0xc0004f8c40, {0x2b19e68, 0x3eea6e0}, {0x26a75b4, 0x8}, {0x26a75bc, 0x8}, {0x0, 0xec68faccf, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:303 +0x109 github.com/rclone/rclone/fs/operations_test.testCheck(0xc000a90380, 0x27e2ca0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:98 +0x145 github.com/rclone/rclone/fs/operations_test.TestCheck(0xc000a90380?) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/check_test.go:181 +0x1a testing.tRunner(0xc000a90380, 0x27e2720) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 133 [sleep, 60 minutes]: time.Sleep(0x29acc4086c00) /usr/local/go/src/runtime/time.go:338 +0x165 github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall.func1(0xc000995a70?) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:169 +0x1c created by github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall in goroutine 130 /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:168 +0xef goroutine 49 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc0000261e0, {0x2b19f80, 0xc000964140}) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:339 +0x145 created by github.com/rclone/rclone/fs/accounting.(*StatsInfo)._startAverageLoop in goroutine 73 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:376 +0x117 "./operations.test -test.v -test.timeout 1h0m0s -remote TestOneDrive: -verbose" - Finished ERROR in 1h0m1.244401116s (try 1/5): exit status 2: Failed []