"./operations.test -test.v -test.timeout 1h0m0s -remote TestOneDrive: -verbose" - Starting (try 1/5) 2025/11/15 03:21:47 DEBUG : Creating backend with remote "TestOneDrive:rclone-test-jacaheq6yonu" 2025/11/15 03:21:47 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/11/15 03:21:47 DEBUG : Too many requests. Trying again in 2010 seconds. 2025/11/15 03:21:47 DEBUG : pacer: low level retry 1/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 33m30s) 2025/11/15 03:21:47 DEBUG : pacer: Rate limited, increasing sleep to 33m30s 2025/11/15 03:21:48 DEBUG : Too many requests. Trying again in 2009 seconds. 2025/11/15 03:21:48 DEBUG : pacer: low level retry 2/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 33m29s) 2025/11/15 03:21:48 DEBUG : pacer: Rate limited, increasing sleep to 33m29s 2025/11/15 03:55:15 DEBUG : OneDrive root 'rclone-test-jacaheq6yonu': Background refresher detected expired token but no uploads in progress - doing nothing 2025/11/15 03:55:17 DEBUG : TestOneDrive: Token expired 2025/11/15 03:55:17 DEBUG : Config file has changed externally - reloading 2025/11/15 03:55:17 DEBUG : TestOneDrive: Loaded fresh token from config file 2025/11/15 03:55:17 DEBUG : TestOneDrive: Loaded new refresh token from config file 2025/11/15 03:55:17 DEBUG : TestOneDrive: Token refresh successful 2025/11/15 03:55:18 DEBUG : Too many requests. Trying again in 3600 seconds. 2025/11/15 03:55:18 DEBUG : pacer: low level retry 3/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 1h0m0s) 2025/11/15 03:55:18 DEBUG : pacer: Rate limited, increasing sleep to 1h0m0s 2025/11/15 04:28:47 DEBUG : Too many requests. Trying again in 3600 seconds. 2025/11/15 04:28:47 DEBUG : pacer: low level retry 4/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 1h0m0s) 2025/11/15 04:55:16 DEBUG : OneDrive root 'rclone-test-jacaheq6yonu': Background refresher detected expired token but no uploads in progress - doing nothing 2025/11/15 05:28:46 DEBUG : TestOneDrive: Token expired 2025/11/15 05:28:46 DEBUG : Config file has changed externally - reloading 2025/11/15 05:28:46 DEBUG : TestOneDrive: Loaded fresh token from config file 2025/11/15 05:28:46 DEBUG : TestOneDrive: Loaded new refresh token from config file 2025/11/15 05:28:46 DEBUG : TestOneDrive: Token refresh successful 2025/11/15 05:28:47 DEBUG : pacer: Reducing sleep to 45m0s 2025/11/15 05:55:16 DEBUG : OneDrive root 'rclone-test-jacaheq6yonu': Background refresher detected expired token but no uploads in progress - doing nothing 2025/11/15 06:28:46 DEBUG : TestOneDrive: Token expired 2025/11/15 06:28:46 DEBUG : Config file has changed externally - reloading 2025/11/15 06:28:46 DEBUG : TestOneDrive: Loaded fresh token from config file 2025/11/15 06:28:46 DEBUG : TestOneDrive: Loaded new refresh token from config file 2025/11/15 06:28:46 DEBUG : TestOneDrive: Token refresh successful 2025/11/15 06:28:47 DEBUG : pacer: Reducing sleep to 33m45s 2025/11/15 06:55:16 DEBUG : OneDrive root 'rclone-test-jacaheq6yonu': Background refresher detected expired token but no uploads in progress - doing nothing 2025/11/15 07:13:46 DEBUG : TestOneDrive: Token expired 2025/11/15 07:13:46 DEBUG : Config file has changed externally - reloading 2025/11/15 07:13:46 DEBUG : TestOneDrive: Loaded fresh token from config file 2025/11/15 07:13:46 DEBUG : TestOneDrive: Loaded new refresh token from config file 2025/11/15 07:13:46 DEBUG : TestOneDrive: Token refresh successful 2025/11/15 07:13:47 DEBUG : Too many requests. Trying again in 50400 seconds. 2025/11/15 07:13:47 DEBUG : pacer: low level retry 1/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 14h0m0s) 2025/11/15 07:13:47 DEBUG : pacer: Rate limited, increasing sleep to 14h0m0s 2025/11/15 07:47:32 DEBUG : Too many requests. Trying again in 173921 seconds. 2025/11/15 07:47:32 DEBUG : pacer: low level retry 2/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 48h18m41s) 2025/11/15 07:47:32 DEBUG : pacer: Rate limited, increasing sleep to 48h18m41s 2025/11/15 07:55:17 DEBUG : OneDrive root 'rclone-test-jacaheq6yonu': Background refresher detected expired token but no uploads in progress - doing nothing 2025/11/15 21:47:32 DEBUG : TestOneDrive: Token expired 2025/11/15 21:47:32 DEBUG : Config file has changed externally - reloading 2025/11/15 21:47:32 DEBUG : TestOneDrive: Loaded new refresh token from config file 2025/11/15 21:47:32 DEBUG : TestOneDrive: Token refresh successful 2025/11/15 21:47:32 DEBUG : Saving config "token" in section "TestOneDrive" of the config file 2025/11/15 21:47:32 DEBUG : TestOneDrive: Saved new token in config file 2025/11/15 21:47:32 DEBUG : pacer: Reducing sleep to 36h14m0.75s 2025/11/15 21:47:32 DEBUG : Creating backend with remote "/tmp/rclone2319473038" === RUN TestDoMultiThreadCopy --- PASS: TestDoMultiThreadCopy (0.00s) === RUN TestMultithreadCalculateNumChunks === RUN TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} === RUN TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} === RUN TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} === RUN TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} --- PASS: TestMultithreadCalculateNumChunks (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1_chunkSize:65536_wantNumChunks:1} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:1_wantNumChunks:1048576} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048576_chunkSize:2_wantNumChunks:524288} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048577_chunkSize:2_wantNumChunks:524289} (0.00s) --- PASS: TestMultithreadCalculateNumChunks/{size:1048575_chunkSize:2_wantNumChunks:524288} (0.00s) === RUN TestMultithreadCopy run.go:185: Remote "OneDrive root 'rclone-test-jacaheq6yonu'", Local "Local file system at /tmp/rclone2319473038", Modify Window "1s" multithread_test.go:119: multithread writing not supported 2025/11/15 22:47:31 DEBUG : OneDrive root 'rclone-test-jacaheq6yonu': Background refresher detected expired token but no uploads in progress - doing nothing panic: test timed out after 1h0m0s running tests: TestMultithreadCopy (1h0m0s) goroutine 149 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc0005048c0, {0x28a09dd?, 0xc00006bad8?}, 0x29ccfb0) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc0005048c0) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc0005048c0, 0xc00006bc18) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc000650648, {0x41efde0, 0x6e, 0x6e}, {0xee0aaebf4?, 0x42066a0?, 0x42066a0?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc0008057c0) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc0008057c0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:74 +0xa6 github.com/rclone/rclone/fs/operations_test.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/operations_test.go:61 main.main() _testmain.go:267 +0xa9 goroutine 9 [syscall, 1165 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 10 [chan receive, 1165 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 11 [select]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc0003b0fc0) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x86 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 132 [sleep, 60 minutes]: time.Sleep(0x9e2e236c8a00) /usr/local/go/src/runtime/time.go:338 +0x165 github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall.func1(0x0?) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:170 +0x1c created by github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:169 +0xf6 goroutine 169 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 168 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 167 [sync.WaitGroup.Wait, 60 minutes]: sync.runtime_SemacquireWaitGroup(0xc00009b860?) /usr/local/go/src/runtime/sema.go:110 +0x25 sync.(*WaitGroup).Wait(0x2d5f918?) /usr/local/go/src/sync/waitgroup.go:118 +0x48 github.com/rclone/rclone/fs/walk.walk({0x2d5f918, 0x4229ec0}, {0x2d77b70, 0xc00079c600}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0xc000114bd0, 0x29cd5e8) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:453 +0x355 github.com/rclone/rclone/fs/walk.walkListDirSorted(...) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:349 github.com/rclone/rclone/fs/walk.Walk({0x2d5f918, 0x4229ec0}, {0x2d77b70, 0xc00079c600}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0xc000114bd0) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:76 +0x2a5 github.com/rclone/rclone/fs/walk.listRwalk({0x2d5f918, 0x4229ec0}, {0x2d77b70, 0xc00079c600}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0x3, 0xc000114ab0) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:170 +0x10d github.com/rclone/rclone/fs/walk.ListR({0x2d5f918, 0x4229ec0}, {0x2d77b70, 0xc00079c600}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0x3, 0xc000114ab0) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:160 +0x236 github.com/rclone/rclone/fstest.newRunIndividual.func1() /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:155 +0x107 github.com/rclone/rclone/fstest.(*Run).Finalise(0xc0007bf1f0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:393 +0x1c testing.(*common).Cleanup.func1() /usr/local/go/src/testing/testing.go:1211 +0x106 testing.(*common).runCleanup(0xc0005056c0, 0xc000a02c40?) /usr/local/go/src/testing/testing.go:1445 +0xea testing.tRunner.func2() /usr/local/go/src/testing/testing.go:1786 +0x25 runtime.Goexit() /usr/local/go/src/runtime/panic.go:636 +0x5e testing.(*common).SkipNow(0xc0005056c0) /usr/local/go/src/testing/testing.go:1156 +0x45 testing.(*common).Skip(0xc0005056c0, {0xc00009be38?, 0x41bb05?, 0x200000000000010?}) /usr/local/go/src/testing/testing.go:1132 +0x54 github.com/rclone/rclone/fs/operations.skipIfNotMultithread({0x2d5f918, 0x4229ec0}, 0xc0005056c0, 0xc0007bf1f0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:119 +0xa5 github.com/rclone/rclone/fs/operations.TestMultithreadCopy(0xc0005056c0) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/multithread_test.go:148 +0x7e testing.tRunner(0xc0005056c0, 0x29ccfb0) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 170 [chan receive, 60 minutes]: github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall(0xc00039f500, 0x0) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:162 +0x26 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc00039f500, 0xc00070c540, 0xa) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:215 +0xb2 github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc00039f500, 0xc00070c540) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:235 +0x93 github.com/rclone/rclone/backend/onedrive.(*Fs).readMetaDataForPathRelativeToID(0xc00079c600, {0x2d5f918, 0x4229ec0}, {0xc000875b00, 0x25}, {0xc00070600d, 0x18}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:934 +0x2c9 github.com/rclone/rclone/backend/onedrive.(*Fs).FindLeaf(0xc00079c600, {0x2d5f918, 0x4229ec0}, {0xc000875b00, 0x25}, {0xc00070600d, 0x18}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1232 +0x77 github.com/rclone/rclone/lib/dircache.(*DirCache)._findDir(0xc0001906c0, {0x2d5f918, 0x4229ec0}, {0xc00070600d, 0x18}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:234 +0x112 github.com/rclone/rclone/lib/dircache.(*DirCache)._findRoot(0xc0001906c0, {0x2d5f918?, 0x4229ec0?}, 0x0?) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:305 +0x45 github.com/rclone/rclone/lib/dircache.(*DirCache).FindDir(0xc0001906c0, {0x2d5f918, 0x4229ec0}, {0x0, 0x0}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/lib/dircache/dircache.go:201 +0xcd github.com/rclone/rclone/backend/onedrive.(*Fs).ListP(0xc00079c600, {0x2d5f918, 0x4229ec0}, {0x0, 0x0}, 0x2?) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1398 +0x6f github.com/rclone/rclone/fs/list.WithListP({0x2d5f918, 0x4229ec0}, {0x0, 0x0}, {0x2d38d60, 0xc00079c600}) /home/rclone/go/src/github.com/rclone/rclone/fs/list/helpers.go:55 +0xb0 github.com/rclone/rclone/backend/onedrive.(*Fs).List(0xffffffffffffffff?, {0x2d5f918?, 0x4229ec0?}, {0x0?, 0x0?}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1380 +0x38 github.com/rclone/rclone/fs/list.DirSorted({0x2d5f918, 0x4229ec0}, {0x2d77b70, 0xc00079c600}, 0x1, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fs/list/list.go:26 +0x67 github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:403 +0x1e9 created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 172 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 171 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 173 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 174 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 175 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 167 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df "./operations.test -test.v -test.timeout 1h0m0s -remote TestOneDrive: -verbose" - Finished ERROR in 19h25m45.404875838s (try 1/5): exit status 2: Failed []