"go test -v -timeout 1h0m0s -remote TestCompressDrive: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsChangeNotify$'" - Starting (try 4/5) === RUN TestIntegration fstests.go:418: Using remote "TestCompressDrive:" 2021/04/05 06:08:44 DEBUG : Creating backend with remote "TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8" === RUN TestIntegration/FsMkdir === RUN TestIntegration/FsMkdir/FsChangeNotify 2021/04/05 06:08:46 DEBUG : dir: Making directory 2021/04/05 06:08:47 DEBUG : dir/subdir1: Making directory 2021/04/05 06:08:47 DEBUG : dir/subdir3: Making directory 2021/04/05 06:08:48 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:48 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/subdir1" entryType 2 2021/04/05 06:08:48 DEBUG : doChangeNotify("dir/subdir1", 2) 2021/04/05 06:08:48 DEBUG : dir/subdir2: Making directory 2021/04/05 06:08:49 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:49 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/subdir3" entryType 2 2021/04/05 06:08:49 DEBUG : doChangeNotify("dir/subdir3", 2) 2021/04/05 06:08:50 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:50 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/subdir2" entryType 2 2021/04/05 06:08:50 DEBUG : doChangeNotify("dir/subdir2", 2) 2021/04/05 06:08:51 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:51 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file2.ZAAAAAAAAAA.gz" entryType 3 2021/04/05 06:08:51 DEBUG : doChangeNotify("dir/file2.ZAAAAAAAAAA.gz.json", 3) 2021/04/05 06:08:52 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:52 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file2.json" entryType 3 2021/04/05 06:08:52 DEBUG : doChangeNotify("dir/file2.json.json", 3) 2021/04/05 06:08:53 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:54 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:55 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:55 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file4.ZAAAAAAAAAA.gz" entryType 3 2021/04/05 06:08:55 DEBUG : doChangeNotify("dir/file4.ZAAAAAAAAAA.gz.json", 3) 2021/04/05 06:08:56 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:56 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file4.json" entryType 3 2021/04/05 06:08:56 DEBUG : doChangeNotify("dir/file4.json.json", 3) 2021/04/05 06:08:57 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:58 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:59 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:08:59 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file3.ZAAAAAAAAAA.gz" entryType 3 2021/04/05 06:08:59 DEBUG : doChangeNotify("dir/file3.ZAAAAAAAAAA.gz.json", 3) 2021/04/05 06:09:00 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:00 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file3.json" entryType 3 2021/04/05 06:09:00 DEBUG : doChangeNotify("dir/file3.json.json", 3) fstests.go:841: Try 1/10 waiting for dirChanges and objChanges 2021/04/05 06:09:01 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:02 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:03 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 2/10 waiting for dirChanges and objChanges 2021/04/05 06:09:04 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:05 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:06 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 3/10 waiting for dirChanges and objChanges 2021/04/05 06:09:07 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:08 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:09 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 4/10 waiting for dirChanges and objChanges 2021/04/05 06:09:10 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:11 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:12 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 5/10 waiting for dirChanges and objChanges 2021/04/05 06:09:13 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:14 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:15 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 6/10 waiting for dirChanges and objChanges 2021/04/05 06:09:16 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:17 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:18 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 7/10 waiting for dirChanges and objChanges 2021/04/05 06:09:19 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:20 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:21 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 8/10 waiting for dirChanges and objChanges 2021/04/05 06:09:22 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:23 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:24 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:841: Try 9/10 waiting for dirChanges and objChanges 2021/04/05 06:09:25 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:26 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:27 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote fstests.go:845: map[dir/subdir1:{} dir/subdir2:{} dir/subdir3:{}] does not contain [dir/subdir1 dir/subdir3 dir/subdir2] or map[dir/file2.ZAAAAAAAAAA.gz.json:{} dir/file2.json.json:{} dir/file3.ZAAAAAAAAAA.gz.json:{} dir/file3.json.json:{} dir/file4.ZAAAAAAAAAA.gz.json:{} dir/file4.json.json:{}] does not contain [dir/file2 dir/file4 dir/file3] 2021/04/05 06:09:28 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:29 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:29 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file2.json" entryType 3 2021/04/05 06:09:29 DEBUG : doChangeNotify("dir/file2.json.json", 3) 2021/04/05 06:09:29 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file2.ZAAAAAAAAAA.gz" entryType 3 2021/04/05 06:09:29 DEBUG : doChangeNotify("dir/file2.ZAAAAAAAAAA.gz.json", 3) 2021/04/05 06:09:30 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:30 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file4.json" entryType 3 2021/04/05 06:09:30 DEBUG : doChangeNotify("dir/file4.json.json", 3) 2021/04/05 06:09:30 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file4.ZAAAAAAAAAA.gz" entryType 3 2021/04/05 06:09:30 DEBUG : doChangeNotify("dir/file4.ZAAAAAAAAAA.gz.json", 3) 2021/04/05 06:09:31 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:31 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file3.json" entryType 3 2021/04/05 06:09:31 DEBUG : doChangeNotify("dir/file3.json.json", 3) 2021/04/05 06:09:31 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/file3.ZAAAAAAAAAA.gz" entryType 3 2021/04/05 06:09:31 DEBUG : doChangeNotify("dir/file3.ZAAAAAAAAAA.gz.json", 3) 2021/04/05 06:09:32 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:32 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/subdir1" entryType 2 2021/04/05 06:09:32 DEBUG : doChangeNotify("dir/subdir1", 2) 2021/04/05 06:09:32 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2021/04/05 06:09:32 DEBUG : pacer: Rate limited, increasing sleep to 1.966869627s 2021/04/05 06:09:32 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2021/04/05 06:09:32 DEBUG : pacer: Rate limited, increasing sleep to 2.080411503s 2021/04/05 06:09:33 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:35 DEBUG : pacer: Reducing sleep to 0s 2021/04/05 06:09:37 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/subdir3" entryType 2 2021/04/05 06:09:37 DEBUG : doChangeNotify("dir/subdir3", 2) 2021/04/05 06:09:37 NOTICE: Compressed: TestCompressDrive:rclone-test-nalibaq3hekasal4malaroc8: path "dir/subdir2" entryType 2 2021/04/05 06:09:37 DEBUG : doChangeNotify("dir/subdir2", 2) 2021/04/05 06:09:37 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "file3.json" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "file3.ZAAAAAAAAAA.gz" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "file4.json" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "file4.ZAAAAAAAAAA.gz" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "file2.json" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "file2.ZAAAAAAAAAA.gz" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "subdir2" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "subdir3" 2021/04/05 06:09:37 DEBUG : dir: Rmdir: contains trashed file: "subdir1" 2021/04/05 06:09:37 DEBUG : Google drive root 'rclone-test-nalibaq3hekasal4malaroc8': Checking for changes on remote 2021/04/05 06:09:37 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2021/04/05 06:09:37 DEBUG : pacer: Rate limited, increasing sleep to 1.65002672s 2021/04/05 06:09:37 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2021/04/05 06:09:37 DEBUG : pacer: Rate limited, increasing sleep to 2.233147654s 2021/04/05 06:09:39 DEBUG : pacer: Reducing sleep to 0s --- FAIL: TestIntegration (57.35s) --- FAIL: TestIntegration/FsMkdir (55.61s) --- FAIL: TestIntegration/FsMkdir/FsChangeNotify (51.09s) FAIL exit status 1 FAIL github.com/rclone/rclone/backend/compress 57.383s "go test -v -timeout 1h0m0s -remote TestCompressDrive: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsChangeNotify$'" - Finished ERROR in 1m2.807907162s (try 4/5): exit status 1: Failed [TestIntegration/FsMkdir/FsChangeNotify]