Commit Graph

6779 Commits (tenant-schema-isolation-complete-view)

Author SHA1 Message Date
Nils Dijk 6a64c23e85
stab at having a complete view of the cluster when assigning nodes 2023-11-02 14:07:38 +00:00
Onur Tirtir bcdcd7f455
Merge branch 'main' into tenant-schema-isolation 2023-11-01 11:48:32 +03:00
Jelte Fennema-Nio a76a832553
Fix flaky validate_constraint test (#7293)
Sometimes validate constraint would fail like this:

```diff
  validatable_constraint_8000016 | t
 (10 rows)

 DROP TABLE constrained_table;
+ERROR:  deadlock detected
+DETAIL:  Process 16602 waits for ShareRowExclusiveLock on relation 56258 of database 16384; blocked by process 16601.
+Process 16601 waits for AccessShareLock on relation 56120 of database 16384; blocked by process 16602.
+HINT:  See server log for query details.
 DROP TABLE referenced_table CASCADE;
 DROP TABLE referencing_table;
 DROP SCHEMA validate_constraint CASCADE;
-NOTICE:  drop cascades to 3 other objects
+NOTICE:  drop cascades to 4 other objects
 DETAIL:  drop cascades to type constraint_validity
 drop cascades to view constraint_validations_in_workers
 drop cascades to view constraint_validations
+drop cascades to table constrained_table
 SET search_path TO DEFAULT;

```

Source:
https://github.com/citusdata/citus/actions/runs/6708383699?pr=7291

This change fixes that by not running together with the
foreign_key_to_reference_table test anymore. In passing it also
simplifies dropping of the test its resources.
2023-11-01 09:41:28 +01:00
Jelte Fennema-Nio 81aa660b31
Fix flaky test detection (#7291)
PR #7289 broke flaky test detction. This fixes that.
2023-10-31 15:59:16 +00:00
Gokhan Gulbiz ce58c04304
Disable CircleCI (#7276)
We are switching to Github Actions. In the test period it has worked well enough, so now we can stop using CircleCI.
2023-10-31 16:00:10 +01:00
Onur Tirtir 50848f92f9 fix builds 2023-10-31 16:38:22 +03:00
Jelte Fennema-Nio 83e3fb817d
Only put major Postgres version in CI task name (#7289)
Making tasks in CI required before merging to master is important and
useful. The way this works is by saving the exact names of the required
tasks in the admin interface of the repo. It has a search box to add
them so it's not completely horrible, but doing so is quite a hassle
since we have so many jobs. So limiting the amount of churn in this list
of required jobs is quite useful.

This changes the names of tasks to only include the major versions of
Postgres, not the minor ones. Otherwise the next time we bump the minor
versions we would have to remove and re-add each of the jobs.
2023-10-31 14:05:09 +01:00
Onur Tirtir 4c2d4ce982 Merge remote-tracking branch 'origin/main' into tenant-schema-isolation 2023-10-31 15:57:15 +03:00
Onur Tirtir deca9778db address feedback 2023-10-31 15:56:53 +03:00
Emel Şimşek ee8f4bb7e8
Start Maintenance Daemon for Main DB at the server start. (#7254)
DESCRIPTION: This change starts a maintenance deamon at the time of
server start if there is a designated main database.

This is the code flow:

1. User designates a main database:
   `ALTER SYSTEM SET citus.main_db =  "myadmindb";`

2. When postmaster starts, in _PG_Init, citus calls 
    `InitializeMaintenanceDaemonForMainDb`
  
This function registers a background worker to run
`CitusMaintenanceDaemonMain `with `databaseOid = 0 `

3. `CitusMaintenanceDaemonMain ` takes some special actions when
databaseOid is 0:
     - Gets the citus.main_db  value.
     - Connects to the  citus.main_db
     - Now the `MyDatabaseId `is available, creates a hash entry for it.
     - Then follows the same control flow as for a regular db,
2023-10-30 09:44:13 +03:00
Onur Tirtir d4391f2425
Merge branch 'main' into tenant-schema-isolation 2023-10-27 18:00:03 +03:00
Nils Dijk d0b093c975
automatically add a breakpoint that breaks on postgres errors (#7279)
When debugging postgres it is quite hard to get to the source for
`errfinish` in `elog.c`. Instead of relying on the developer to set a
breakpoint in the `elog.c` file for `errfinish` for `elevel == ERROR`,
this change adds the breakpoint to `.gdbinit`. This makes sure that
whenever a debugger is attached to a postgres backend it will break on
postgres errors.

When attaching the debugger a small banner is printed that explains how
to disable the breakpoint.
2023-10-27 16:57:51 +02:00
Benjamin O f9218d9780
Support replacing IPv6 Loopback in `normalize.sed` (#7269)
I had a test failure issue due to my machine using the IPv6 loopback
address. This change to the `normalize.sed` solves that issue.
2023-10-27 16:42:55 +02:00
Onur Tirtir ad2af65edb
Merge branch 'main' into tenant-schema-isolation 2023-10-27 15:22:34 +03:00
Gokhan Gulbiz 2bf1472c8e
Move GHA environment variables to workflow file (#7275)
Since GHA does not interpolate env variables in a matrix context, This
PR defines them in a separate job and uses them in other jobs.
2023-10-26 14:54:58 +03:00
Onur Tirtir 74e6002739 better 2023-10-24 18:35:16 +03:00
Onur Tirtir d9c1e992b2 remove WorkerNodeListGetNodeWithGroupId 2023-10-24 18:34:04 +03:00
Onur Tirtir 6af8a51065
Merge branch 'main' into tenant-schema-isolation 2023-10-24 14:27:45 +03:00
Naisila Puka 10198b18e8
Technical readme small fixes (#7261) 2023-10-23 13:43:43 +03:00
Naisila Puka 1fe16fa746
Remove unnecessary pre-fastpath code (#7262)
This code was here because we first implemented
`fast path planner` via
[#2606](https://github.com/citusdata/citus/pull/2606)
and then later `deferred pruning`
[#3369](https://github.com/citusdata/citus/pull/3369)
So, for some years, this code was useful.
2023-10-23 13:01:48 +03:00
zhjwpku 2d1444188c
Fix wrong comments around HasDistributionKey() (#7223)
HasDistributionKey & HasDistributionKeyCacheEntry returns true when the
corresponding table has a distribution key, the comments state the
opposite,
which should be fixed.

Signed-off-by: Zhao Junwang <zhjwpku@gmail.com>
Co-authored-by: Onur Tirtir <onurcantirtir@gmail.com>
2023-10-18 10:53:00 +02:00
Onur Tirtir db13afaa7b
Fix flaky columnar_create.sql test (#7266) 2023-10-17 16:58:17 +03:00
Gürkan İndibay 71a4633dad
Fixes typo and renames multi_process_utility (#7259) 2023-10-17 16:39:37 +03:00
Onur Tirtir aa8733faa7 comment 2023-10-16 16:47:11 +03:00
Onur Tirtir db43b6fdce improve & comment 2023-10-16 16:42:59 +03:00
Onur Tirtir ac97d54515 fix 2023-10-16 16:13:31 +03:00
Onur Tirtir 01b2bf5e3c fix 2023-10-16 14:55:29 +03:00
Onur Tirtir 26c27a9fbf fix tests 2023-10-16 14:46:25 +03:00
Onur Tirtir 09f0003ae1 Merge remote-tracking branch 'origin/main' into tenant-schema-isolation 2023-10-16 14:29:10 +03:00
Onur Tirtir 2d16b0fd9e address feedback 2023-10-16 14:28:25 +03:00
Onur Tirtir 5eaf6c221e
Fix flaky test detection job (#7256)
We were getting such errors in flaky-test detection job:
```
Unable to process file command 'output' successfully
```

Even though we don't seem to be writing multiple lines to
$GITHUB_OUTPUT, this seems to be the right fix.

https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
2023-10-16 14:20:55 +03:00
Jelte Fennema-Nio 788e09a39a
Add a test for citus_shards where table names have spaces (#7224)
There was a bug reported for previous versions of Citus where
shard\_size was returning NULL for tables with spaces in them. It works
fine on the main branch though, but I'm still adding a test for this to
the main branch because it seems a good test to have.
2023-10-16 11:38:24 +02:00
Onur Tirtir b0fa3d91bc fix flaky test detection job 2023-10-13 15:01:25 +03:00
Onur Tirtir a2fb92d8dd actually use the index 2023-10-13 15:01:25 +03:00
Onur Tirtir e9c05647af
Merge branch 'main' into tenant-schema-isolation 2023-10-13 14:04:21 +03:00
Nils Dijk fb08f9b198
Remove software-properties-common from dev container after use (#7255)
During the creation of the devcontainer we need to add a ppa repository,
which is easiest done via software-properies-common. As turns out this
installes pkexec into the container as a side effect.

When vscode tries to attach a debugger it first checks if pkexec is
installed as this gives a nicer popup asking for elevation of rights to
attach to the process. However, since dev containers don't have a
windowing system running pkexec isn't working as expected and thus
prevents the debugger from attaching.

Without pkexec in the container vscode 'falls back' to plain old sudo
which we can run passwordless in the container.

For pkexec to be removed we need to first purge
software-propertied-common as well as autoremove all packages that were
installed due to the installation of said package. By performing this
all in one step we minimize the size of the layer we are creating.
2023-10-12 17:47:44 +02:00
Onur Tirtir 89f13f038e
Merge branch 'main' into tenant-schema-isolation 2023-10-11 14:21:31 +03:00
Gokhan Gulbiz e0b0cdbb87
CircleCI to GHA migration (#7154)
Co-authored-by: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
2023-10-10 16:58:50 +03:00
Onur Tirtir 9ea89a0063 Update src/backend/distributed/operations/rebalancer_placement_isolation.c 2023-10-10 11:28:20 +03:00
Onur Tirtir 2fc1411da4 properly assign conflicting placements 2023-10-10 11:28:20 +03:00
Onur Tirtir 7e9a186fa2 comment 2023-10-10 11:28:20 +03:00
Onur Tirtir 77c5c882de properly handle the cases where rebalancer is called for specific table 2023-10-10 11:28:20 +03:00
Onur Tirtir 6e9fc45b97 improve 2023-10-10 11:28:20 +03:00
Onur Tirtir d541f64e3c improve test 2023-10-10 11:28:20 +03:00
Onur Tirtir faffeccc76 take shardAllowedOnNode udf into account when planning 2023-10-10 11:28:20 +03:00
Onur Tirtir d1a1ad0147 improve citus_shards 2023-10-10 11:28:20 +03:00
Onur Tirtir cc587101ed rename needsisolatednode to needsseparatenode 2023-10-10 11:28:19 +03:00
Onur Tirtir a58442d411 rename to has_separate_node 2023-10-10 11:28:02 +03:00
Onur Tirtir 7d68e655bc add own_node to citus_shards 2023-10-10 11:28:01 +03:00
Onur Tirtir 785296406f reindent 2023-10-10 11:27:19 +03:00