Increase the default value of citus.node_connection_timeout

The previous default was 5 seconds, and we change it to 30 seconds.
The main motivation for this is that for busy clusters, 5 seconds
can be too aggressive. Especially with connection throttling, the servers
might be kept busy for a really long time, and users may see the
connection errors more frequently.

We've done some sanity checks, for really quick queries (like
`SELECT count(*) from table`), 30 seconds is a decent value even
if users execute 300 distributed queries on the coordinator. We've
verified this on Hyperscale(Citus).

src/backend/distributed/connection/connection_management.c

@@ -41,7 +41,7 @@
 #include "utils/memutils.h"
 
 
-int NodeConnectionTimeout = 5000;
+int NodeConnectionTimeout = 30000;
 int MaxCachedConnectionsPerWorker = 1;
 
 HTAB *ConnectionHash = NULL;

src/backend/distributed/shared_library_init.c

@@ -450,7 +450,7 @@ RegisterCitusConfigVariables(void)
 		gettext_noop("Sets the maximum duration to connect to worker nodes."),
 		NULL,
 		&NodeConnectionTimeout,
-		5 * MS_PER_SECOND, 10 * MS, MS_PER_HOUR,
+		30 * MS_PER_SECOND, 10 * MS, MS_PER_HOUR,
 		PGC_USERSET,
 		GUC_UNIT_MS | GUC_STANDARD,
 		NULL, NULL, NULL);