"./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose" - Starting (try 1/5) 2025/09/09 01:37:42 DEBUG : Creating backend with remote "TestGoFile:rclone-test-pawosik1timo" 2025/09/09 01:37:42 DEBUG : Using config file from "/home/rclone/.rclone.conf" 2025/09/09 01:37:42 DEBUG : Creating backend with remote "/tmp/rclone1266851857" === RUN TestPipe --- PASS: TestPipe (0.00s) === RUN TestPipeConcurrent --- PASS: TestPipeConcurrent (0.03s) === RUN TestPipeOrderBy === RUN TestPipeOrderBy/#00 === RUN TestPipeOrderBy/size === RUN TestPipeOrderBy/name === RUN TestPipeOrderBy/modtime === RUN TestPipeOrderBy/size,ascending === RUN TestPipeOrderBy/name,asc === RUN TestPipeOrderBy/modtime,ascending === RUN TestPipeOrderBy/size,descending === RUN TestPipeOrderBy/name,desc === RUN TestPipeOrderBy/modtime,descending === RUN TestPipeOrderBy/size,mixed,50 === RUN TestPipeOrderBy/size,mixed,51 --- PASS: TestPipeOrderBy (0.00s) --- PASS: TestPipeOrderBy/#00 (0.00s) --- PASS: TestPipeOrderBy/size (0.00s) --- PASS: TestPipeOrderBy/name (0.00s) --- PASS: TestPipeOrderBy/modtime (0.00s) --- PASS: TestPipeOrderBy/size,ascending (0.00s) --- PASS: TestPipeOrderBy/name,asc (0.00s) --- PASS: TestPipeOrderBy/modtime,ascending (0.00s) --- PASS: TestPipeOrderBy/size,descending (0.00s) --- PASS: TestPipeOrderBy/name,desc (0.00s) --- PASS: TestPipeOrderBy/modtime,descending (0.00s) --- PASS: TestPipeOrderBy/size,mixed,50 (0.00s) --- PASS: TestPipeOrderBy/size,mixed,51 (0.00s) === RUN TestNewLess === RUN TestNewLess/blankOK === RUN TestNewLess/tooManyParts === RUN TestNewLess/tooManyParts2 === RUN TestNewLess/badMixed === RUN TestNewLess/unknownComparison === RUN TestNewLess/unknownSortDirection === RUN TestNewLess/size === RUN TestNewLess/name === RUN TestNewLess/modtime === RUN TestNewLess/size,ascending === RUN TestNewLess/name,asc === RUN TestNewLess/modtime,ascending === RUN TestNewLess/size,descending === RUN TestNewLess/name,desc === RUN TestNewLess/modtime,descending === RUN TestNewLess/modtime,mixed === RUN TestNewLess/modtime,mixed,30 --- PASS: TestNewLess (0.00s) --- PASS: TestNewLess/blankOK (0.00s) --- PASS: TestNewLess/tooManyParts (0.00s) --- PASS: TestNewLess/tooManyParts2 (0.00s) --- PASS: TestNewLess/badMixed (0.00s) --- PASS: TestNewLess/unknownComparison (0.00s) --- PASS: TestNewLess/unknownSortDirection (0.00s) --- PASS: TestNewLess/size (0.00s) --- PASS: TestNewLess/name (0.00s) --- PASS: TestNewLess/modtime (0.00s) --- PASS: TestNewLess/size,ascending (0.00s) --- PASS: TestNewLess/name,asc (0.00s) --- PASS: TestNewLess/modtime,ascending (0.00s) --- PASS: TestNewLess/size,descending (0.00s) --- PASS: TestNewLess/name,desc (0.00s) --- PASS: TestNewLess/modtime,descending (0.00s) --- PASS: TestNewLess/modtime,mixed (0.00s) --- PASS: TestNewLess/modtime,mixed,30 (0.00s) === RUN TestRcCopy rc_test.go:16: Skipping test on non local remote --- SKIP: TestRcCopy (0.00s) === RUN TestRcMove rc_test.go:16: Skipping test on non local remote --- SKIP: TestRcMove (0.00s) === RUN TestRcSync rc_test.go:16: Skipping test on non local remote --- SKIP: TestRcSync (0.00s) === RUN TestCopyWithDryRun run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" 2025/09/09 01:37:43 DEBUG : Added delayed dir = "sub dir", newDst= 2025/09/09 01:37:43 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/09/09 01:37:43 NOTICE: sub dir/hello world: Skipped copy as --dry-run is set (size 11) 2025/09/09 01:37:43 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for checks to finish 2025/09/09 01:37:43 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for transfers to finish 2025/09/09 01:37:43 NOTICE: sub dir: Skipped set directory modification time as --dry-run is set --- PASS: TestCopyWithDryRun (0.63s) === RUN TestCopy run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" 2025/09/09 01:37:43 INFO : sub dir: Set directory modification time (using DirSetModTime) 2025/09/09 01:37:43 DEBUG : Added delayed dir = "sub dir", newDst= 2025/09/09 01:37:43 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/09/09 01:37:43 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for checks to finish 2025/09/09 01:37:43 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for transfers to finish 2025/09/09 01:39:53 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:39:53 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:39:53 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:39:53 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 0/10 2025/09/09 01:39:53 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:42:04 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:42:04 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:42:04 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:42:04 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 1/10 2025/09/09 01:42:04 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:44:15 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:44:15 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:44:15 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:44:15 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 2/10 2025/09/09 01:44:15 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:46:26 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:46:26 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:46:26 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:46:26 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 3/10 2025/09/09 01:46:26 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:48:37 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:48:37 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:48:37 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:48:37 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 4/10 2025/09/09 01:48:37 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:50:48 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:50:48 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:50:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:50:48 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 5/10 2025/09/09 01:50:48 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:52:59 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:52:59 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:52:59 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:52:59 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 6/10 2025/09/09 01:52:59 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:55:10 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:55:10 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:55:10 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:55:10 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 7/10 2025/09/09 01:55:11 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:57:21 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:57:21 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:57:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:57:21 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 8/10 2025/09/09 01:57:22 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 01:57:22 DEBUG : sub dir/hello world: md5 = 5eb63bbbe01eeed093cb22bb8f5acdc3 OK 2025/09/09 01:57:22 INFO : sub dir/hello world: Copied (new) 2025/09/09 01:57:22 INFO : sub dir: Set directory modification time (using DirSetModTime) --- PASS: TestCopy (1179.65s) === RUN TestCopyMetadata run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" sync_test.go:98: Skipping as metadata not supported --- SKIP: TestCopyMetadata (0.15s) === RUN TestCopyMetadataNoEmptyDirs run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" sync_test.go:98: Skipping as metadata not supported --- SKIP: TestCopyMetadataNoEmptyDirs (0.13s) === RUN TestCopyMissingDirectory run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" 2025/09/09 01:57:23 DEBUG : Creating backend with remote "/non-existing" 2025/09/09 01:57:23 DEBUG : Config file has changed externally - reloading 2025/09/09 01:57:23 ERROR : Local file system at /non-existing: error reading source root directory: directory not found 2025/09/09 01:57:23 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for checks to finish 2025/09/09 01:57:23 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for transfers to finish --- PASS: TestCopyMissingDirectory (0.28s) === RUN TestCopyNoTraverse run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" 2025/09/09 01:57:23 DEBUG : Added delayed dir = "sub dir", newDst= 2025/09/09 01:57:23 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/09/09 01:57:23 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for checks to finish 2025/09/09 01:57:23 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for transfers to finish 2025/09/09 01:59:33 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 01:59:33 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 01:59:33 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 01:59:33 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 0/10 2025/09/09 01:59:33 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:01:44 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:01:44 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:01:44 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:01:44 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 1/10 2025/09/09 02:01:44 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:01:44 DEBUG : sub dir/hello world: md5 = 5eb63bbbe01eeed093cb22bb8f5acdc3 OK 2025/09/09 02:01:44 INFO : sub dir/hello world: Copied (new) 2025/09/09 02:01:45 INFO : sub dir: Set directory modification time (using DirSetModTime) --- PASS: TestCopyNoTraverse (262.55s) === RUN TestCopyNoTraverseDeadlock run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" sync_test.go:222: Only runs on local --- SKIP: TestCopyNoTraverseDeadlock (0.20s) === RUN TestCopyCheckFirst run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" 2025/09/09 02:01:46 INFO : gofile root 'rclone-test-pawosik1timo': Running all checks before starting transfers 2025/09/09 02:01:46 DEBUG : Added delayed dir = "sub dir", newDst= 2025/09/09 02:01:46 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/09/09 02:01:46 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for checks to finish 2025/09/09 02:01:46 INFO : gofile root 'rclone-test-pawosik1timo': Checks finished, now starting transfers 2025/09/09 02:01:46 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for transfers to finish 2025/09/09 02:03:57 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:03:57 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:03:57 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:03:57 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 0/10 2025/09/09 02:03:57 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:06:08 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:06:08 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:06:08 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:06:08 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 1/10 2025/09/09 02:06:08 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:08:19 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:08:19 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:08:19 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:08:19 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 2/10 2025/09/09 02:08:19 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:10:30 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:10:30 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:10:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:10:30 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 3/10 2025/09/09 02:10:30 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:12:41 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:12:41 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:12:41 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:12:41 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 4/10 2025/09/09 02:12:41 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:14:52 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:14:52 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:14:52 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:14:52 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 5/10 2025/09/09 02:14:52 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:17:03 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:17:03 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:17:03 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:17:03 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 6/10 2025/09/09 02:17:03 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:19:14 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:19:14 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:19:14 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:19:14 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 7/10 2025/09/09 02:19:14 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:21:25 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:21:25 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:21:25 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:21:25 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 8/10 2025/09/09 02:21:26 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:23:36 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:23:36 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:23:36 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:23:36 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 9/10 2025/09/09 02:23:36 ERROR : sub dir/hello world: Failed to copy: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" 2025/09/09 02:23:37 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:23:37 INFO : sub dir: Set directory modification time (using DirSetModTime) sync_test.go:260: Error Trace: /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:260 Error: Received unexpected error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" Test: TestCopyCheckFirst --- FAIL: TestCopyCheckFirst (1311.20s) === RUN TestSyncNoTraverse run.go:180: Remote "gofile root 'rclone-test-pawosik1timo'", Local "Local file system at /tmp/rclone1266851857", Modify Window "1s" 2025/09/09 02:23:37 ERROR : Ignoring --no-traverse with sync 2025/09/09 02:23:37 DEBUG : Added delayed dir = "sub dir", newDst= 2025/09/09 02:23:37 DEBUG : sub dir/hello world: Need to transfer - File not found at Destination 2025/09/09 02:23:37 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for checks to finish 2025/09/09 02:23:37 DEBUG : gofile root 'rclone-test-pawosik1timo': Waiting for transfers to finish 2025/09/09 02:25:47 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:25:47 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:25:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:25:47 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 0/10 2025/09/09 02:25:48 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:27:59 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:27:59 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:27:59 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:27:59 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 1/10 2025/09/09 02:27:59 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:30:10 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:30:10 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:30:10 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:30:10 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 2/10 2025/09/09 02:30:10 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:32:21 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:32:21 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:32:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:32:21 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 3/10 2025/09/09 02:32:21 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:34:32 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:34:32 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:34:32 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:34:32 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 4/10 2025/09/09 02:34:32 DEBUG : pacer: Reducing sleep to 10ms 2025/09/09 02:36:43 DEBUG : Couldn't decode error response: invalid character 'E' looking for beginning of value 2025/09/09 02:36:43 DEBUG : pacer: low level retry 1/1 (error Error "502 Bad Gateway (502): Error forwarding request to upload server\n") 2025/09/09 02:36:43 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2025/09/09 02:36:43 DEBUG : sub dir/hello world: Received error: failed to upload file: Error "502 Bad Gateway (502): Error forwarding request to upload server\n" - low level retry 5/10 2025/09/09 02:36:43 DEBUG : pacer: Reducing sleep to 10ms panic: test timed out after 1h0m0s running tests: TestSyncNoTraverse (14m5s) goroutine 608 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2484 +0x394 created by time.goFunc /usr/local/go/src/time/sleep.go:215 +0x2d goroutine 1 [chan receive, 15 minutes]: testing.(*T).Run(0xc000840540, {0x276bbca?, 0xc000555ad8?}, 0x288b058) /usr/local/go/src/testing/testing.go:1859 +0x431 testing.runTests.func1(0xc000840540) /usr/local/go/src/testing/testing.go:2279 +0x37 testing.tRunner(0xc000840540, 0xc000555c18) /usr/local/go/src/testing/testing.go:1792 +0xf4 testing.runTests(0xc000a012f0, {0x3fb0dc0, 0x6c, 0x6c}, {0xee0517be6?, 0x3fc6f40?, 0x3fc6f40?}) /usr/local/go/src/testing/testing.go:2277 +0x4b4 testing.(*M).Run(0xc00044d180) /usr/local/go/src/testing/testing.go:2142 +0x64a github.com/rclone/rclone/fstest.TestMain(0xc00044d180) /home/rclone/go/src/github.com/rclone/rclone/fstest/run.go:69 +0xa5 github.com/rclone/rclone/fs/sync.TestMain(...) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:45 main.main() _testmain.go:261 +0xa9 goroutine 6 [syscall, 60 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:152 +0x1f goroutine 7 [chan receive, 60 minutes]: github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1() /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x27 created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xa7 goroutine 796 [sync.WaitGroup.Wait, 15 minutes]: sync.runtime_SemacquireWaitGroup(0x0?) /usr/local/go/src/runtime/sema.go:110 +0x25 sync.(*WaitGroup).Wait(0xc000688707?) /usr/local/go/src/sync/waitgroup.go:118 +0x48 github.com/rclone/rclone/fs/sync.(*syncCopyMove).stopTransfers(0xc000732008) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync.go:543 +0x6f github.com/rclone/rclone/fs/sync.(*syncCopyMove).run(0xc000732008) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync.go:978 +0x325 github.com/rclone/rclone/fs/sync.runSyncCopyMove({0x2bd62f0, 0xc000303c20}, {0x2bebe20, 0xc000356d00}, {0x2bec980, 0xc000a140c0}, 0x3, 0x0, 0x0, 0x0, ...) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync.go:1369 +0x225 github.com/rclone/rclone/fs/sync.Sync({0x2bd62f0, 0xc000303c20}, {0x2bebe20, 0xc000356d00}, {0x2bec980, 0xc000a140c0}, 0x0) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync.go:1375 +0x77 github.com/rclone/rclone/fs/sync.TestSyncNoTraverse(0xc000a3e000) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync_test.go:279 +0x198 testing.tRunner(0xc000a3e000, 0x288b058) /usr/local/go/src/testing/testing.go:1792 +0xf4 created by testing.(*T).Run in goroutine 1 /usr/local/go/src/testing/testing.go:1851 +0x413 goroutine 154 [select]: net/http.(*persistConn).writeLoop(0xc00008b560) /usr/local/go/src/net/http/transport.go:2590 +0xe7 created by net/http.(*Transport).dialConn in goroutine 228 /usr/local/go/src/net/http/transport.go:1945 +0x17a5 goroutine 153 [IO wait]: internal/poll.runtime_pollWait(0x7b00b7e0e4c0, 0x72) /usr/local/go/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc00051a180?, 0xc000499800?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00051a180, {0xc000499800, 0xc00, 0xc00}) /usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc00051a180, {0xc000499800?, 0x2000?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc000534000, {0xc000499800?, 0x5?, 0x5?}) /usr/local/go/src/net/net.go:194 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000aac000, {0xc000499800?, 0x7b0070142e18?, 0x7b00b7fc0108?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x29 crypto/tls.(*atLeastReader).Read(0xc000aac048, {0xc000499800?, 0xa?, 0xc00009b9c8?}) /usr/local/go/src/crypto/tls/conn.go:809 +0x3b bytes.(*Buffer).ReadFrom(0xc0007142b8, {0x2bb04a0, 0xc000aac048}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc000714008, {0x7b00703220f8, 0xc000aac000}, 0x4426d4?) /usr/local/go/src/crypto/tls/conn.go:831 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc000714008, 0x0) /usr/local/go/src/crypto/tls/conn.go:629 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:591 crypto/tls.(*Conn).Read(0xc000714008, {0xc0008aa000, 0x1000, 0xc00068f050?}) /usr/local/go/src/crypto/tls/conn.go:1385 +0x145 net/http.(*persistConn).Read(0xc00008b560, {0xc0008aa000?, 0x80d2e5?, 0x2bad1a0?}) /usr/local/go/src/net/http/transport.go:2122 +0x47 bufio.(*Reader).fill(0xc0002e77a0) /usr/local/go/src/bufio/bufio.go:113 +0x103 bufio.(*Reader).Peek(0xc0002e77a0, 0x1) /usr/local/go/src/bufio/bufio.go:152 +0x53 net/http.(*persistConn).readLoop(0xc00008b560) /usr/local/go/src/net/http/transport.go:2275 +0x172 created by net/http.(*Transport).dialConn in goroutine 228 /usr/local/go/src/net/http/transport.go:1944 +0x174f goroutine 981 [chan receive]: github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc000420140) /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:114 +0x3f created by github.com/rclone/rclone/backend/local.newFadvise in goroutine 870 /home/rclone/go/src/github.com/rclone/rclone/backend/local/fadvise_unix.go:74 +0x11f goroutine 797 [select]: github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc0005de1e0, {0x2bd6328, 0xc000688730}) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:339 +0x145 created by github.com/rclone/rclone/fs/accounting.(*StatsInfo)._startAverageLoop in goroutine 796 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/stats.go:376 +0x11a goroutine 982 [select]: github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc000800100) /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:221 +0xed created by github.com/rclone/rclone/fs/accounting.newAccountSizeName in goroutine 870 /home/rclone/go/src/github.com/rclone/rclone/fs/accounting/accounting.go:119 +0x436 goroutine 870 [select]: net/http.(*persistConn).roundTrip(0xc00008b560, 0xc0006882d0) /usr/local/go/src/net/http/transport.go:2826 +0x785 net/http.(*Transport).roundTrip(0xc0006169c0, 0xc000349040) /usr/local/go/src/net/http/transport.go:696 +0xb9a net/http.(*Transport).RoundTrip(...) /usr/local/go/src/net/http/roundtrip.go:30 github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc00038fbc0, 0xc000349040) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/http.go:496 +0x345 net/http.send(0xc000349040, {0x2baee40, 0xc00038fbc0}, {0x6fbe01?, 0x27ac8de?, 0x0?}) /usr/local/go/src/net/http/client.go:259 +0x5e2 net/http.(*Client).send(0xc00068ed80, 0xc000349040, {0xb?, 0x1?, 0x0?}) /usr/local/go/src/net/http/client.go:180 +0x91 net/http.(*Client).do(0xc00068ed80, 0xc000349040) /usr/local/go/src/net/http/client.go:728 +0x989 net/http.(*Client).Do(...) /usr/local/go/src/net/http/client.go:587 github.com/rclone/rclone/lib/rest.(*Client).Call(0xc000523ef0, {0x2bd6328, 0xc000688870}, 0xc000585260) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:353 +0xd67 github.com/rclone/rclone/lib/rest.(*Client).callCodec(0xc000523ef0, {0x2bd6328, 0xc000688870}, 0x419691?, {0x0?, 0x0?}, {0x22e0600, 0xc0006a20c0}, 0x41373d?, 0x288c658, ...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:550 +0x3f2 github.com/rclone/rclone/lib/rest.(*Client).CallJSON(...) /home/rclone/go/src/github.com/rclone/rclone/lib/rest/rest.go:493 github.com/rclone/rclone/backend/gofile.(*Object).Update.func2() /home/rclone/go/src/github.com/rclone/rclone/backend/gofile/gofile.go:1461 +0x88 github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc00054c038?) /home/rclone/go/src/github.com/rclone/rclone/fs/pacer.go:86 +0x32 github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc00038fc20, 0xc000690240, 0x1) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:199 +0x7c github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...) /home/rclone/go/src/github.com/rclone/rclone/lib/pacer/pacer.go:227 github.com/rclone/rclone/backend/gofile.(*Object).Update(0xc00017ebd0, {0x2bd6328, 0xc000688870}, {0x2badbc0, 0xc000800100}, {0x7b00702f1ef0, 0xc0009523c0}, {0xc000a9e140, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/backend/gofile/gofile.go:1460 +0x628 github.com/rclone/rclone/backend/gofile.(*Fs).PutUnchecked(0xc000356d00, {0x2bd6328, 0xc000688870}, {0x2badbc0, 0xc000800100}, {0x7b00702f1ef0, 0xc0009523c0}, {0xc000a9e140, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/backend/gofile/gofile.go:790 +0x14c github.com/rclone/rclone/backend/gofile.(*Fs).Put(0xc000356d00, {0x2bd6328, 0xc000688870}, {0x2badbc0, 0xc000800100}, {0x7b00702f1ef0, 0xc0009523c0}, {0xc000a9e140, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/backend/gofile/gofile.go:763 +0x12c github.com/rclone/rclone/fs/operations.(*copy).updateOrPut(0xc00065a3f0, {0x2bd6328, 0xc000688870}, {0x2bc1168?, 0xc00017c380?}, {0xc000a9e140, 0x1, 0x1}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy.go:221 +0x225 github.com/rclone/rclone/fs/operations.(*copy).manualCopy(0xc00065a3f0, {0x2bd6328, 0xc000688870}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy.go:274 +0x6d8 github.com/rclone/rclone/fs/operations.(*copy).copy(0xc00065a3f0, {0x2bd6328, 0xc000688870}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy.go:314 +0x170 github.com/rclone/rclone/fs/operations.Copy({0x2bd6328, 0xc000688870}, {0x2bebe20, 0xc000356d00}, {0x0, 0x0}, {0xc00054c030, 0x13}, {0x2bec9f0, 0xc0009523c0}) /home/rclone/go/src/github.com/rclone/rclone/fs/operations/copy.go:416 +0x4db github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc000732008, {0x2bd6328, 0xc000688870}, 0xc0006887d0, {0x2bebe20, 0xc000356d00}, 0x0, 0xc0007cc8c0?) /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync.go:505 +0x1fe created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers in goroutine 796 /home/rclone/go/src/github.com/rclone/rclone/fs/sync/sync.go:535 +0x7d goroutine 993 [IO wait]: internal/poll.runtime_pollWait(0x7b00b7e0e6f0, 0x72) /usr/local/go/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc00051a380?, 0xc000498000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00051a380, {0xc000498000, 0xc00, 0xc00}) /usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc00051a380, {0xc000498000?, 0x7ad?, 0xc000a5a340?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc0001483d0, {0xc000498000?, 0xc00049844e?, 0x5?}) /usr/local/go/src/net/net.go:194 +0x45 github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000aac210, {0xc000498000?, 0x7b0070143958?, 0x7b00b7fc0f30?}) /home/rclone/go/src/github.com/rclone/rclone/fs/fshttp/dialer.go:111 +0x29 crypto/tls.(*atLeastReader).Read(0xc000a000c0, {0xc000498000?, 0x7ad?, 0xc00038b088?}) /usr/local/go/src/crypto/tls/conn.go:809 +0x3b bytes.(*Buffer).ReadFrom(0xc0007149b8, {0x2bb04a0, 0xc000a000c0}) /usr/local/go/src/bytes/buffer.go:211 +0x98 crypto/tls.(*Conn).readFromUntil(0xc000714708, {0x7b00703220f8, 0xc000aac210}, 0x4426d4?) /usr/local/go/src/crypto/tls/conn.go:831 +0xde crypto/tls.(*Conn).readRecordOrCCS(0xc000714708, 0x0) /usr/local/go/src/crypto/tls/conn.go:629 +0x3cf crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:591 crypto/tls.(*Conn).Read(0xc000714708, {0xc0008ab000, 0x1000, 0xc00056fca0?}) /usr/local/go/src/crypto/tls/conn.go:1385 +0x145 bufio.(*Reader).Read(0xc000952e40, {0xc00017c4a0, 0x9, 0x3f33f20?}) /usr/local/go/src/bufio/bufio.go:245 +0x197 io.ReadAtLeast({0x2bad7a0, 0xc000952e40}, {0xc00017c4a0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:335 +0x91 io.ReadFull(...) /usr/local/go/src/io/io.go:354 net/http.http2readFrameHeader({0xc00017c4a0, 0x9, 0x68e3f0?}, {0x2bad7a0?, 0xc000952e40?}) /usr/local/go/src/net/http/h2_bundle.go:1805 +0x65 net/http.(*http2Framer).ReadFrame(0xc00017c460) /usr/local/go/src/net/http/h2_bundle.go:2072 +0x7d net/http.(*http2clientConnReadLoop).run(0xc00056ffa8) /usr/local/go/src/net/http/h2_bundle.go:9933 +0xda net/http.(*http2ClientConn).readLoop(0xc00072e380) /usr/local/go/src/net/http/h2_bundle.go:9812 +0x79 created by net/http.(*http2Transport).newClientConn in goroutine 992 /usr/local/go/src/net/http/h2_bundle.go:8334 +0xde5 goroutine 882 [chan receive, 15 minutes]: github.com/rclone/rclone/fs/march.(*March).Run.func2() /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:255 +0x3c created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 796 /home/rclone/go/src/github.com/rclone/rclone/fs/march/march.go:253 +0x378 "./sync.test -test.v -test.timeout 1h0m0s -remote TestGoFile: -verbose" - Finished ERROR in 1h0m0.285415977s (try 1/5): exit status 2: Failed [TestCopyCheckFirst]