"go test -v -timeout 1h0m0s -remote TestCompressDrive: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsChangeNotify$'" - Starting (try 2/5) === RUN TestIntegration fstests.go:418: Using remote "TestCompressDrive:" 2022/01/17 05:25:39 DEBUG : Creating backend with remote "TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4" 2022/01/17 05:25:39 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2022/01/17 05:25:40 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:25:40 DEBUG : pacer: Rate limited, increasing sleep to 1.354277943s 2022/01/17 05:25:41 DEBUG : pacer: Reducing sleep to 0s === RUN TestIntegration/FsMkdir === RUN TestIntegration/FsMkdir/FsChangeNotify 2022/01/17 05:25:42 DEBUG : dir: Making directory 2022/01/17 05:25:42 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:25:42 DEBUG : pacer: Rate limited, increasing sleep to 1.797262594s 2022/01/17 05:25:43 DEBUG : pacer: Reducing sleep to 0s 2022/01/17 05:25:45 DEBUG : dir/subdir1: Making directory 2022/01/17 05:25:46 DEBUG : dir/subdir3: Making directory 2022/01/17 05:25:46 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:46 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:25:46 DEBUG : pacer: Rate limited, increasing sleep to 1.452273935s 2022/01/17 05:25:46 DEBUG : pacer: Reducing sleep to 0s 2022/01/17 05:25:47 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:25:47 DEBUG : pacer: Rate limited, increasing sleep to 1.743763873s 2022/01/17 05:25:49 DEBUG : pacer: Reducing sleep to 0s 2022/01/17 05:25:49 DEBUG : dir/subdir2: Making directory 2022/01/17 05:25:50 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir" entryType 0 2022/01/17 05:25:50 DEBUG : doChangeNotify("dir", 0) 2022/01/17 05:25:50 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/subdir1" entryType 0 2022/01/17 05:25:50 DEBUG : doChangeNotify("dir/subdir1", 0) 2022/01/17 05:25:50 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/subdir3" entryType 0 2022/01/17 05:25:50 DEBUG : doChangeNotify("dir/subdir3", 0) 2022/01/17 05:25:50 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:50 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:50 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:25:50 DEBUG : pacer: Rate limited, increasing sleep to 1.683309128s 2022/01/17 05:25:50 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:25:50 DEBUG : pacer: Rate limited, increasing sleep to 2.379970502s 2022/01/17 05:25:52 DEBUG : pacer: Reducing sleep to 0s 2022/01/17 05:25:52 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/subdir2" entryType 0 2022/01/17 05:25:52 DEBUG : doChangeNotify("dir/subdir2", 0) 2022/01/17 05:25:52 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:55 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:55 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:56 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:56 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file2.ZAAAAAAAAAA.gz" entryType 1 2022/01/17 05:25:56 DEBUG : doChangeNotify("dir/file2.ZAAAAAAAAAA.gz.json", 1) 2022/01/17 05:25:57 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:58 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:25:59 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:00 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file2.json" entryType 1 2022/01/17 05:26:00 DEBUG : doChangeNotify("dir/file2.json.json", 1) 2022/01/17 05:26:00 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:01 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:02 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:02 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file4.ZAAAAAAAAAA.gz" entryType 1 2022/01/17 05:26:02 DEBUG : doChangeNotify("dir/file4.ZAAAAAAAAAA.gz.json", 1) 2022/01/17 05:26:03 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:04 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file4.json" entryType 1 2022/01/17 05:26:04 DEBUG : doChangeNotify("dir/file4.json.json", 1) 2022/01/17 05:26:04 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:05 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:05 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:26:05 DEBUG : pacer: Rate limited, increasing sleep to 1.125786896s 2022/01/17 05:26:05 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:26:05 DEBUG : pacer: Rate limited, increasing sleep to 2.142214016s fstests.go:150: Put error: googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded - low level retry 1/10 2022/01/17 05:26:05 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:26:05 DEBUG : pacer: Rate limited, increasing sleep to 4.48036844s 2022/01/17 05:26:07 DEBUG : pacer: Reducing sleep to 0s 2022/01/17 05:26:07 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:11 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:12 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:13 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:13 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file3.ZAAAAAAAAAA.gz" entryType 1 2022/01/17 05:26:13 DEBUG : doChangeNotify("dir/file3.ZAAAAAAAAAA.gz.json", 1) 2022/01/17 05:26:14 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:15 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:16 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file3.json" entryType 1 2022/01/17 05:26:16 DEBUG : doChangeNotify("dir/file3.json.json", 1) fstests.go:846: Try 1/10 waiting for dirChanges and objChanges 2022/01/17 05:26:16 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:17 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:18 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 2/10 waiting for dirChanges and objChanges 2022/01/17 05:26:19 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:20 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:21 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 3/10 waiting for dirChanges and objChanges 2022/01/17 05:26:22 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:23 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:24 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 4/10 waiting for dirChanges and objChanges 2022/01/17 05:26:25 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:26 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:27 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 5/10 waiting for dirChanges and objChanges 2022/01/17 05:26:28 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:29 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:30 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 6/10 waiting for dirChanges and objChanges 2022/01/17 05:26:31 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:32 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:33 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 7/10 waiting for dirChanges and objChanges 2022/01/17 05:26:34 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:35 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:36 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 8/10 waiting for dirChanges and objChanges 2022/01/17 05:26:37 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:38 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:39 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:846: Try 9/10 waiting for dirChanges and objChanges 2022/01/17 05:26:40 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:41 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:42 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote fstests.go:850: map[dir:{} dir/subdir1:{} dir/subdir2:{} dir/subdir3:{}] does not contain [dir/subdir1 dir/subdir3 dir/subdir2] or map[dir/file2.ZAAAAAAAAAA.gz.json:{} dir/file2.json.json:{} dir/file3.ZAAAAAAAAAA.gz.json:{} dir/file3.json.json:{} dir/file4.ZAAAAAAAAAA.gz.json:{} dir/file4.json.json:{}] does not contain [dir/file2 dir/file4 dir/file3] 2022/01/17 05:26:43 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:44 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:44 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file2.json" entryType 1 2022/01/17 05:26:44 DEBUG : doChangeNotify("dir/file2.json.json", 1) 2022/01/17 05:26:45 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:46 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file2.ZAAAAAAAAAA.gz" entryType 1 2022/01/17 05:26:46 DEBUG : doChangeNotify("dir/file2.ZAAAAAAAAAA.gz.json", 1) 2022/01/17 05:26:46 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file4.json" entryType 1 2022/01/17 05:26:46 DEBUG : doChangeNotify("dir/file4.json.json", 1) 2022/01/17 05:26:46 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:46 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file4.ZAAAAAAAAAA.gz" entryType 1 2022/01/17 05:26:46 DEBUG : doChangeNotify("dir/file4.ZAAAAAAAAAA.gz.json", 1) 2022/01/17 05:26:46 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file3.json" entryType 1 2022/01/17 05:26:46 DEBUG : doChangeNotify("dir/file3.json.json", 1) 2022/01/17 05:26:47 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:48 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/file3.ZAAAAAAAAAA.gz" entryType 1 2022/01/17 05:26:48 DEBUG : doChangeNotify("dir/file3.ZAAAAAAAAAA.gz.json", 1) 2022/01/17 05:26:48 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:49 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/subdir1" entryType 0 2022/01/17 05:26:49 DEBUG : doChangeNotify("dir/subdir1", 0) 2022/01/17 05:26:49 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:49 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/subdir3" entryType 0 2022/01/17 05:26:49 DEBUG : doChangeNotify("dir/subdir3", 0) 2022/01/17 05:26:50 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=847840889997, userRateLimitExceeded) 2022/01/17 05:26:50 DEBUG : pacer: Rate limited, increasing sleep to 1.412327072s 2022/01/17 05:26:50 DEBUG : pacer: Reducing sleep to 0s 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "file3.ZAAAAAAAAAA.gz" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "file3.json" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "file4.ZAAAAAAAAAA.gz" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "file4.json" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "file2.json" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "file2.ZAAAAAAAAAA.gz" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "subdir2" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "subdir3" 2022/01/17 05:26:50 DEBUG : dir: Rmdir: contains trashed file: "subdir1" 2022/01/17 05:26:50 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote 2022/01/17 05:26:51 NOTICE: Compressed: TestCompressDrive:rclone-test-hutisus2bipuwel3pesucod4: path "dir/subdir2" entryType 0 2022/01/17 05:26:51 DEBUG : doChangeNotify("dir/subdir2", 0) 2022/01/17 05:26:51 DEBUG : Google drive root 'rclone-test-hutisus2bipuwel3pesucod4': Checking for changes on remote --- FAIL: TestIntegration (74.07s) --- FAIL: TestIntegration/FsMkdir (70.77s) --- FAIL: TestIntegration/FsMkdir/FsChangeNotify (69.32s) FAIL exit status 1 FAIL github.com/rclone/rclone/backend/compress 74.086s "go test -v -timeout 1h0m0s -remote TestCompressDrive: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsChangeNotify$'" - Finished ERROR in 1m15.070437191s (try 2/5): exit status 1: Failed [TestIntegration/FsMkdir/FsChangeNotify]