mirror of https://github.com/citusdata/citus.git
5df1b49bed
When `master_update_node` is called to update a node's location it waits for appropriate locks to become available. This is useful during normal operation as new operations will be blocked till after the metadata update while running operations have time to finish. When `master_update_node` is called after a node failure it is less useful to wait for running operations to finish as they can't. The lock being held indicates an operation that once attempted to commit will fail as the machine already failed. Now the downside is the failover is postponed till the termination point of the operation. This has been observed by users to take a significant amount of time causing the rest of the system to be observed unavailable. With this patch it is possible in such situations to invoke `master_update_node` with 2 optional arguments: - `force` (bool defaults to `false`): When called with true the update of the metadata will be forced to proceed by terminating conflicting backends. A cancel is not enough as the backend might be in idle time (eg. an interactive session, or going back and forth between an appliaction), therefore a more intrusive solution of termination is used here. - `lock_cooldown` (int defaults to `10000`): This is the time in milliseconds before conflicting backends are terminated. This is to allow the backends to finish cleanly before terminating them. This allows the user to set an upperbound to the expected time to complete the metadata update, eg. performing the failover. The functionality is implemented by spawning a background worker that has the task of helping a certain backend in acquiring its locks. The backend is either terminated on successful execution of the metadata update, or once the memory context of the expression gets reset, eg. on a cancel of the statement. |
||
|---|---|---|
| .. | ||
| bin | ||
| data | ||
| expected | ||
| input | ||
| mitmscripts | ||
| output | ||
| specs | ||
| sql | ||
| .gitignore | ||
| Makefile | ||
| Pipfile | ||
| Pipfile.lock | ||
| base_schedule | ||
| failure_base_schedule | ||
| failure_schedule | ||
| isolation_schedule | ||
| log_test_times | ||
| multi_follower_schedule | ||
| multi_mx_schedule | ||
| multi_schedule | ||
| multi_task_tracker_extra_schedule | ||
| pg_regress_multi.pl | ||
| worker_schedule | ||