Commit Graph

3168 Commits (d1331055d457a0a14cc4998829a51c0a8d2b0b44)

Author SHA1 Message Date
Burak Velioglu d1331055d4
Improve and fix table creation tests 2021-09-01 13:39:43 +03:00
Burak Velioglu 3a9008a79c
Address reviews and improve tests 2021-08-31 16:10:18 +03:00
Burak Velioglu 47ec669a71
Update tests and comments 2021-08-30 12:36:33 +03:00
Burak Velioglu 868c522d1f
Return partition name from range finder function (without fixing tests) 2021-08-28 22:12:20 +03:00
Burak Velioglu fa3f661421
Address reviews except partition naming related ones 2021-08-27 23:44:54 +03:00
Burak Velioglu 8f179fbb83
Rename drop function to drop_timeseries_table_metadata 2021-08-27 22:33:03 +03:00
Burak Velioglu acc787fac3
Downgrade drop_timeseries_table properly 2021-08-27 22:05:58 +03:00
Burak Velioglu d23b8d3b8b
Add downgrade scripts 2021-08-27 17:12:24 +03:00
Burak Velioglu 22dcc251c3
Fix tests and adress reviews 2021-08-27 13:44:08 +03:00
Burak Velioglu e4fbcfe3cf
EditorConfig 2021-08-27 12:21:45 +03:00
Burak Velioglu ec05df9708
Fix include 2021-08-27 11:15:26 +03:00
Burak Velioglu 7a4293fe37
Update ordering on test and add new one for create_missing_partitions 2021-08-26 13:32:21 +03:00
Burak Velioglu dfdda48cbf
Add get_missing_partition_ranges tests 2021-08-26 02:06:21 +03:00
Burak Velioglu e1299ec72b
Add create_timeseries_table tests 2021-08-26 01:26:41 +03:00
Burak Velioglu aad741a2e2
Fix error checks and messages 2021-08-24 16:02:52 +03:00
Burak Velioglu c6705ebbd1
Address reviews 2021-08-24 13:31:32 +03:00
Burak Velioglu 06dd189c8f
Add start_from parameter to create_timeseries_table 2021-08-24 00:45:34 +03:00
Burak Velioglu 4754a86e31
Introduce drop_timeseries_table UDF to drop metadata 2021-08-23 15:36:33 +03:00
Burak Velioglu 993a2cbce9
Reorder checks to make them more actionable 2021-08-22 22:39:54 +03:00
Burak Velioglu 20471d2a38
Add TODOs 2021-08-22 22:29:37 +03:00
Burak Velioglu e7138bcd4c
Introduce get_missing_partition_ranges and use it as a part of create_missing_partitions 2021-08-22 22:18:53 +03:00
Burak Velioglu 787f76a952
citus_indent 2021-08-20 12:34:28 +03:00
Burak Velioglu ce12ab5570
Introduce timeseries_utils 2021-08-20 12:03:41 +03:00
Burak Velioglu e9bb732b22
Add truncating according to interval and minor fixes 2021-08-20 01:53:26 +03:00
Burak Velioglu ef724edf31
Introduce create_timeseries_table with related checks 2021-08-19 13:18:48 +03:00
Burak Velioglu 4355ba0a38
Add CREATE INDEX ... ON ONLY and ALTER INDEX ... ATTACH PARTITION (#4938 #4980)
- Add support for CRETE INDEX ... ON ONLY: Before that commit we were not sending "ONLY" option to the worker nodes at all. With this commit, "ONLY" parameter will be sent to the worker nodes if it is necessary. (#4938)

- Add support for ALTER INDEX ... ATTACH PARTITION: Attach child_index to parent_index by creating same inheritance on shard level in addition to table level. (#4980)
2021-08-13 13:12:45 +03:00
SaitTalhaNisanci 2ec4e37e45
Fix assert failure in FindReferencedTableColumn (#5175) 2021-08-12 18:21:45 +03:00
Ahmet Gedemenli 9e90894f21
Synchronize hasmetadata flag on mx workers (#5086)
* Synchronize hasmetadata flag on mx workers

* Switch to sequential execution

* Add test

* Use SetWorkerColumn

* Add test for stop_sync

* Remove usage of UpdateHasmetadataOnWorkersWithMetadata

* Remove MarkNodeMetadataSynced

* Fix test for metadatasynced

* Remove MarkNodeMetadataSynced

* Style

* Remove MarkNodeHasMetadata

* Remove UpdateDistNodeBoolAttr

* Refactor SetWorkerColumn

* Use SetWorkerColumnLocalOnly when setting up dependencies

* Use SetWorkerColumnLocalOnly in TriggerSyncMetadataToPrimaryNodes

* Style

* Make update command generator functions static

* Set metadatasynced before syncing

* Call SetWorkerColumn only if the sync is successful

* Try to sync all nodes

* Fix indexno

* Update metadatasynced locally first

* Break if a node fails to sync metadata

* Send worker commands optional

* Style & Rebase

* Add raiseOnError param to SetWorkerColumn

* Style

* Set metadatasynced for all metadata nodes

* Style

* Introduce SetWorkerColumnOptional

* Polish

* Style

* Dont send set command to not synced metadata nodes

* Style

* Polish

* Add test for stop_sync

* Add test for shouldhaveshards

* Add test for isactive flag

* Sort by placementid in the function verify_metadata

* Cover edge cases for failing nodes

* Add comments

* Add nodeport to isactive test

* Add warning if metadata out of sync

* Update warning message
2021-08-12 14:16:18 +03:00
Naisila Puka e5b32b2c3c
Acquire AccessShareLock before updating table statistics (#5155) 2021-08-12 13:58:15 +03:00
Onder Kalaci d4368ff2b3 Make sure that shouldhaveshards is synced to workers 2021-08-11 15:53:31 +02:00
Onder Kalaci 86bd28b92c Guard against hard WaitEvenSet errors
In short, add wrappers around Postgres' AddWaitEventToSet() and
ModifyWaitEvent().

AddWaitEventToSet()/ModifyWaitEvent*() may throw hard errors. For
example, when the underlying socket for a connection is closed by
the remote server and already reflected by the OS, however
Citus hasn't had a chance to get this information. In that case,
if replication factor is >1, Citus can failover to other nodes
for executing the query. Even if replication factor = 1, Citus
can give much nicer errors.

So CitusAddWaitEventSetToSet()/CitusModifyWaitEvent() simply puts
AddWaitEventToSet()/ModifyWaitEvent() into a PG_TRY/PG_CATCH block
in order to catch any hard errors, and returns this information to
the caller.
2021-08-10 09:35:03 +02:00
Onder Kalaci 5f02d18ef8 transactional metadata sync for maintanince daemon
As we use the current user to sync the metadata to the nodes
with #5105 (and many other PRs), there is no reason that
prevents us to use the coordinated transaction for metadata syncing.

This commit also renames few functions to reflect their actual
implementation.
2021-08-09 10:34:55 +02:00
Onder Kalaci 35964c6366 Dropped columns do not diverge distribution column for partitioned tables
Before this commit, creating a partition after a DROP column
on the parent (position before dist. key) was leading to
partition to have the wrong distribution column.
2021-08-06 13:36:12 +02:00
jeff-davis deb7ec605b
Columnar: fix misleading comments and useless types. (#5162)
CustomScan and CustomPath structures cannot be extended with
additional fields. Fix comments and type structure that implied that
they can.
2021-08-05 09:22:21 -07:00
Ahmet Gedemenli 51d410bb7b Add check for alphabetically sorted gucs
Move to a separate script

Add the new script to readme
2021-08-05 16:37:49 +03:00
naisila 798a7902bf Fix master_update_table_statistics scripts for 9.5 2021-08-03 18:15:56 +03:00
naisila f9fa5a3d69 Fix master_update_table_statistics scripts for 9.4 2021-08-03 18:15:56 +03:00
Onder Kalaci 482b8096e9 Introduce citus_internal_update_relation_colocation
update_distributed_table_colocation can be called by the relation
owner, and internally it updates pg_dist_partition. With this
commit, update_distributed_table_colocation uses an internal
UDF to access pg_dist_partition.

As a result, this operation can now be done by regular users
on MX.
2021-08-03 11:44:58 +02:00
Onur Tirtir 93ebbb0607 Re-cost SeqPath's as well for columnar tables 2021-08-02 11:32:25 +03:00
Onur Tirtir 453ac40725 Comment why we still remove non IndexPath's when custom scan is off 2021-08-02 11:25:18 +03:00
Onur Tirtir a87405b6ba Not adjust IndexPath cost if indexscan is off 2021-08-02 11:25:18 +03:00
Onur Tirtir 51691a8994 Rename RecostColumnarIndexPaths to RecostColumnarPaths 2021-08-02 11:25:18 +03:00
Onur Tirtir 297f59a70e Re-cost columnar table index paths 2021-08-02 11:16:37 +03:00
Onur Tirtir 8adcf2096b Multiply ColumnarCustomScan cost by tblspace.seqpage cost 2021-08-02 11:16:37 +03:00
Onur Tirtir dba8421453 Refactor ColumnarScanCost into ColumnarPerChunkGroupScanCost 2021-08-02 11:16:37 +03:00
Onur Tirtir d8f92697f2
Free memory used for last stripe read when re-scanning a columnar table (#5143)
Instead of setting stripeReadState to NULL, call ColumnarResetRead
before re-scanning a columnar table since this function is already
designed for doing the necessary clean up when finishing a stripe
read.

Note that this change shouldn't have a great effect on memory usage
since AdvanceStripe was already doing the clean-up for all the
stripes except the last one.
2021-08-02 11:16:01 +03:00
Onur Tirtir 73058d35cc Not free (stripe) chunk buffers after de-serializing
Previously, we were only using chunk group reader for sequential scan.
However, to support index scans on columnar tables, now we use very
same low level functions for index scan too.

Since those low-level functions were only used for sequential scan, it
was guaranteed that we would never read the same chunk group more than
once, so we were freeing chunk buffers after deserializing them into a
separate buffer.

Now that we use those low level functions for index scan, we cannot
free chunk buffers since it's possible to read the same chunk group
again, such that:

- read chunk group 1 of stripe 5
- read chunk group 2 of stripe 5
- read chunk group 1 of stripe 5 again

Here, when we decide to read chunk group 1 for a second time,
chunk group 1 is not cached. Plus, before this commit, we were
freeing the chunk buffers for chunk group 1 after the first
read and then we were getting segfault or errors from low-level
de-compression APIs.
2021-08-02 11:00:12 +03:00
Onur Tirtir 327ae43b83 Get rid of EndStripeRead, since we anyway reset mem cxt 2021-08-02 11:00:12 +03:00
Onur Tirtir 83f5d42365 Use long-lasting mem cxt & optimize correlated index scan 2021-08-02 11:00:12 +03:00
Onur Tirtir c021b82a43 Introduce CreateColumnarScanMemoryContext 2021-08-02 11:00:12 +03:00