"go test -v -timeout 1h0m0s -remote TestOneDrive: -verbose" - Starting (try 3/5) 2025/11/15 05:45:42 DEBUG : Creating backend with remote "TestOneDrive:rclone-test-gubecap7vule" 2025/11/15 05:45:42 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/11/15 05:45:43 DEBUG : Too many requests. Trying again in 7200 seconds. 2025/11/15 05:45:43 DEBUG : pacer: low level retry 1/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 2h0m0s) 2025/11/15 05:45:43 DEBUG : pacer: Rate limited, increasing sleep to 2h0m0s 2025/11/15 05:45:43 DEBUG : pacer: Reducing sleep to 1h30m0s 2025/11/15 05:45:43 DEBUG : Creating backend with remote "/tmp/rclone962126160" === RUN TestOrderPermissions === RUN TestOrderPermissions/personal === RUN TestOrderPermissions/personal/empty === RUN TestOrderPermissions/personal/users_first,_then_group,_then_none === RUN TestOrderPermissions/personal/same_type_unsorted === RUN TestOrderPermissions/personal/all_user_identities === RUN TestOrderPermissions/personal/no_user_or_group_info === RUN TestOrderPermissions/business === RUN TestOrderPermissions/business/empty === RUN TestOrderPermissions/business/users_first,_then_group,_then_none === RUN TestOrderPermissions/business/same_type_unsorted === RUN TestOrderPermissions/business/all_user_identities === RUN TestOrderPermissions/business/no_user_or_group_info --- PASS: TestOrderPermissions (0.00s) --- PASS: TestOrderPermissions/personal (0.00s) --- PASS: TestOrderPermissions/personal/empty (0.00s) --- PASS: TestOrderPermissions/personal/users_first,_then_group,_then_none (0.00s) --- PASS: TestOrderPermissions/personal/same_type_unsorted (0.00s) --- PASS: TestOrderPermissions/personal/all_user_identities (0.00s) --- PASS: TestOrderPermissions/personal/no_user_or_group_info (0.00s) --- PASS: TestOrderPermissions/business (0.00s) --- PASS: TestOrderPermissions/business/empty (0.00s) --- PASS: TestOrderPermissions/business/users_first,_then_group,_then_none (0.00s) --- PASS: TestOrderPermissions/business/same_type_unsorted (0.00s) --- PASS: TestOrderPermissions/business/all_user_identities (0.00s) --- PASS: TestOrderPermissions/business/no_user_or_group_info (0.00s) === RUN TestOrderPermissionsJSON --- PASS: TestOrderPermissionsJSON (0.00s) === RUN TestIntegration fstests.go:438: Using remote "TestOneDrive:" 2025/11/15 05:45:43 DEBUG : Creating backend with remote "TestOneDrive:rclone-test-jifemox5jibi" 2025/11/15 05:45:43 DEBUG : Using config file from "/home/rclone/.rclone.conf" === RUN TestIntegration/FsCheckWrap fstests.go:479: Not a wrapping Fs === RUN TestIntegration/FsCommand fstests.go:507: No commands in this remote === RUN TestIntegration/FsRmdirNotFound 2025/11/15 05:45:45 DEBUG : Too many requests. Trying again in 104974 seconds. 2025/11/15 05:45:45 DEBUG : pacer: low level retry 1/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 29h9m34s) 2025/11/15 05:45:45 DEBUG : pacer: Rate limited, increasing sleep to 29h9m34s 2025/11/15 05:45:45 DEBUG : Too many requests. Trying again in 108574 seconds. 2025/11/15 05:45:45 DEBUG : pacer: low level retry 2/10 (error accessDenied: throttledRequest: Too Many Requests: trying again in 30h9m34s) 2025/11/15 05:45:45 DEBUG : pacer: Rate limited, increasing sleep to 30h9m34s 2025/11/15 05:55:16 DEBUG : OneDrive root 'rclone-test-gubecap7vule': Background refresher detected expired token but no uploads in progress - doing nothing 2025/11/15 05:55:16 DEBUG : OneDrive root 'rclone-test-jifemox5jibi': Background refresher detected expired token but no uploads in progress - doing nothing panic: test timed out after 1h0m0s running tests: TestIntegration (1h0m0s) goroutine 107 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 60 minutes]: testing.(*T).Run(0xc00040ea80, {0xd38e15?, 0xc00004fad8?}, 0xd9eb28) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc00040ea80) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc00040ea80, 0xc00004fc18) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc000535518, {0x13ff9c0, 0x4, 0x4}, {0xee0aa0a87?, 0x142ad40?, 0x142ad40?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc0003441e0) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc0003441e0) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:74 +0xa6 github.com/rclone/rclone/backend/onedrive.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive_internal_test.go:41 main.main() _testmain.go:53 +0xa9 goroutine 10 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 11 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 12 [select, 51 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc000360c80) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x86 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 98 [select, 59 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 42 [sleep, 60 minutes]: time.Sleep(0x68c61714000) /usr/local/go/src/runtime/time.go:338 +0x165 github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall.func1(0x0?) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:170 +0x1c created by github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:169 +0xf6 goroutine 66 [select, 51 minutes]: github.com/rclone/rclone/lib/oauthutil.(*Renew).renewOnExpiry(0xc00032c080) /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:43 +0x86 created by github.com/rclone/rclone/lib/oauthutil.NewRenew in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/lib/oauthutil/renew.go:32 +0xe5 goroutine 58 [sync.WaitGroup.Wait, 59 minutes]: sync.runtime_SemacquireWaitGroup(0xc00004f738?) /usr/local/go/src/runtime/sema.go:110 +0x25 sync.(*WaitGroup).Wait(0xe8c338?) /usr/local/go/src/sync/waitgroup.go:118 +0x48 github.com/rclone/rclone/fs/walk.walk({0xe8c338, 0x144cbc0}, {0xe925a0, 0xc0000d8480}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0xc0002d6fc0, 0xd9f920) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:453 +0x355 github.com/rclone/rclone/fs/walk.walkListDirSorted(...) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:349 github.com/rclone/rclone/fs/walk.Walk({0xe8c338, 0x144cbc0}, {0xe925a0, 0xc0000d8480}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0xc0002d6fc0) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:76 +0x2a5 github.com/rclone/rclone/fs/walk.listRwalk({0xe8c338, 0x144cbc0}, {0xe925a0, 0xc0000d8480}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0x3, 0xc00011d500) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:170 +0x10d github.com/rclone/rclone/fs/walk.ListR({0xe8c338, 0x144cbc0}, {0xe925a0, 0xc0000d8480}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0x3, 0xc00011d500) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:160 +0x236 github.com/rclone/rclone/fs/walk.GetAll({0xe8c338, 0x144cbc0}, {0xe925a0, 0xc0000d8480}, {0x0, 0x0}, 0x1, 0xffffffffffffffff) /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:634 +0x105 github.com/rclone/rclone/fstest.CheckListingWithRoot(0xc000103500, {0xe925a0, 0xc0000d8480}, {0x0, 0x0}, {0x144cbc0, 0x0, 0x0}, {0x0, 0x0, ...}, ...) /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:280 +0x405 github.com/rclone/rclone/fstest.CheckListingWithPrecision(...) /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:338 github.com/rclone/rclone/fstest.CheckListing(0xc000103500, {0xe925a0, 0xc0000d8480}, {0x144cbc0, 0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fstest/fstest.go:344 +0x8a github.com/rclone/rclone/fstest/fstests.Run(0xc000103500, 0xc0003a81a0) /home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:531 +0xc1c github.com/rclone/rclone/backend/onedrive.TestIntegration(0xc000103500) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive_test.go:14 +0x89 testing.tRunner(0xc000103500, 0xd9eb28) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 59 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 99 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 87 [sleep, 60 minutes]: time.Sleep(0x5f7929fb0c00) /usr/local/go/src/runtime/time.go:338 +0x165 github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall.func1(0x0?) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:170 +0x1c created by github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall in goroutine 103 /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:169 +0xf6 goroutine 81 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 80 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 79 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc000198960, {0xe8c418, 0xc000323400}) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:352 +0x145 created by github.com/rclone/rclone/fs/accounting.(*StatsInfo)._startAverageLoop in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:389 +0x11a goroutine 100 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 101 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 102 [select, 60 minutes]: github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:398 +0x16c created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df goroutine 103 [chan receive, 60 minutes]: github.com/rclone/rclone/lib/pacer.(*Pacer).beginCall(0xc000408660, 0x0) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:162 +0x26 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000408660, 0xc000323450, 0xa) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:215 +0xb2 github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc000408660, 0xc000323450) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:235 +0x93 github.com/rclone/rclone/backend/onedrive.(*Fs)._listAll(0xc0000d8480, {0xe8c338, 0x144cbc0}, {0x0?, 0x0?}, 0x0, 0x0, 0xc0000e7c60, 0xc0004b8b40, {0xbe1080, ...}, ...) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1287 +0x165 github.com/rclone/rclone/backend/onedrive.(*Fs).listAll(0xc0000d8480, {0xe8c338, 0x144cbc0}, {0xc000539fb0, 0x29}, 0x0, 0x0, 0xc00014ec60) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1339 +0x1cc github.com/rclone/rclone/backend/onedrive.(*Fs).ListP(0xc0000d8480, {0xe8c338, 0x144cbc0}, {0x0, 0x0}, 0x2?) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1402 +0x10c github.com/rclone/rclone/fs/list.WithListP({0xe8c338, 0x144cbc0}, {0x0, 0x0}, {0xe84b40, 0xc0000d8480}) /home/rclone/go/src/github.com/rclone/rclone/fs/list/helpers.go:55 +0xb0 github.com/rclone/rclone/backend/onedrive.(*Fs).List(0x47bfce?, {0xe8c338?, 0x144cbc0?}, {0x0?, 0xc00049ef60?}) /home/rclone/go/src/github.com/rclone/rclone/backend/onedrive/onedrive.go:1380 +0x38 github.com/rclone/rclone/fs/list.DirSorted({0xe8c338, 0x144cbc0}, {0xe925a0, 0xc0000d8480}, 0x1, {0x0, 0x0}) /home/rclone/go/src/github.com/rclone/rclone/fs/list/list.go:26 +0x67 github.com/rclone/rclone/fs/walk.walk.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:403 +0x1e9 created by github.com/rclone/rclone/fs/walk.walk in goroutine 58 /home/rclone/go/src/github.com/rclone/rclone/fs/walk/walk.go:395 +0x1df exit status 2 FAIL github.com/rclone/rclone/backend/onedrive 3600.904s "go test -v -timeout 1h0m0s -remote TestOneDrive: -verbose" - Finished ERROR in 1h0m1.905572271s (try 3/5): exit status 1: Failed []