DESCRIPTION: Add infrastructure to run long running management operations in background This infrastructure introduces the primitives of jobs and tasks. A task consists of a sql statement and an owner. Tasks belong to a Job and can depend on other tasks from the same job. When there are either runnable or running tasks we would like to make sure a bacgrkound task queue monitor process is running. A Task could be in running state while there is actually no monitor present due to a database restart or failover. Once the monitor starts it will reset any running task to its runnable state. To make sure only one background task queue monitor is ever running at once it will acquire an advisory lock that self conflicts. Once a task is done it will find all tasks depending on this task. After checking that the task doesn't have unmet dependencies it will transition the task from blocked to runnable state for the task to be picked up on a subsequent task start. Currently only one task can be running at a time. This can be improved upon in later releases without changes to the higher level API. The initial goal for this background tasks is to allow a rebalance to run in the background. This will be implemented in a subsequent PR. |
||
---|---|---|
.. | ||
bin | ||
citus_tests | ||
data | ||
expected | ||
mitmscripts | ||
spec | ||
sql | ||
.gitignore | ||
Makefile | ||
Pipfile | ||
Pipfile.lock | ||
README.md | ||
after_citus_upgrade_coord_schedule | ||
after_pg_upgrade_schedule | ||
base_isolation_schedule | ||
base_schedule | ||
before_citus_upgrade_coord_schedule | ||
before_pg_upgrade_schedule | ||
columnar_isolation_schedule | ||
columnar_schedule | ||
create_schedule | ||
enterprise_failure_schedule | ||
enterprise_isolation_logicalrep_1_schedule | ||
enterprise_isolation_logicalrep_2_schedule | ||
enterprise_isolation_logicalrep_3_schedule | ||
enterprise_isolation_schedule | ||
enterprise_schedule | ||
enterprise_split_schedule | ||
failure_base_schedule | ||
failure_schedule | ||
isolation_schedule | ||
json_table_select_only.out | ||
json_table_select_only_0.out | ||
log_test_times | ||
minimal_schedule | ||
mixed_after_citus_upgrade_schedule | ||
mixed_before_citus_upgrade_schedule | ||
multi_1_schedule | ||
multi_follower_schedule | ||
multi_mx_schedule | ||
multi_schedule | ||
multi_schedule_hyperscale | ||
multi_schedule_hyperscale_superuser | ||
mx_base_schedule | ||
mx_minimal_schedule | ||
operations_schedule | ||
pg_regress_multi.pl | ||
postgres_schedule | ||
split_schedule | ||
sql_base_schedule | ||
sql_schedule |
README.md
How our testing works
We use the test tooling of postgres to run our tests. This tooling is very
simple but effective. The basics it runs a series of .sql
scripts, gets
their output and stores that in results/$sqlfilename.out
. It then compares the
actual output to the expected output with a simple diff
command:
diff results/$sqlfilename.out expected/$sqlfilename.out
Schedules
Which sql scripts to run is defined in a schedule file, e.g. multi_schedule
,
multi_mx_schedule
.
Makefile
In our Makefile
we have rules to run the different types of test schedules.
You can run them from the root of the repository like so:
# e.g. the multi_schedule
make install -j9 && make -C src/test/regress/ check-multi
Take a look at the makefile for a list of all the testing targets.
Running a specific test
Often you want to run a specific test and don't want to run everything. You can use one of the following commands to do so:
# If your tests needs almost no setup you can use check-minimal
make install -j9 && make -C src/test/regress/ check-minimal EXTRA_TESTS='multi_utility_warnings'
# Often tests need some testing data, if you get missing table errors using
# check-minimal you should try check-base
make install -j9 && make -C src/test/regress/ check-base EXTRA_TESTS='with_prepare'
# Sometimes this is still not enough and some other test needs to be run before
# the test you want to run. You can do so by adding it to EXTRA_TESTS too.
make install -j9 && make -C src/test/regress/ check-base EXTRA_TESTS='add_coordinator coordinator_shouldhaveshards'
Normalization
The output of tests is sadly not completely predictable. Still we want to
compare the output of different runs and error when the important things are
different. We do this by not using the regular system diff
to compare files.
Instead we use src/test/regress/bin/diff
which does the following things:
- Change the
$sqlfilename.out
file by running it throughsed
using thesrc/test/regress/bin/normalize.sed
file. This does stuff like replacing numbers that keep changing across runs with anXXX
string, e.g. portnumbers or transaction numbers. - Backup the original output to
$sqlfilename.out.unmodified
in case it's needed for debugging - Compare the changed
results
andexpected
files with the systemdiff
command.
Updating the expected test output
Sometimes you add a test to an existing file, or test output changes in a way
that's not bad (possibly even good if support for queries is added). In those
cases you want to update the expected test output.
The way to do this is very simple, you run the test and copy the new .out file
in the results
directory to the expected
directory, e.g.:
make install -j9 && make -C src/test/regress/ check-minimal EXTRA_TESTS='multi_utility_warnings'
cp src/test/regress/{results,expected}/multi_utility_warnings.out
Adding a new test file
Adding a new test file is quite simple:
- Write the SQL file in the
sql
directory - Add it to a schedule file, to make sure it's run in CI
- Run the test
- Check that the output is as expected
- Copy the
.out
file fromresults
toexpected
Isolation testing
See src/test/regress/spec/README.md
Upgrade testing
See src/test/regress/citus_tests/upgrade/README.md
Failure testing
See src/test/regress/mitmscripts/README.md
Perl test setup script
To automatically setup a citus cluster in tests we use our
src/test/regress/pg_regress_multi.pl
script. This sets up a citus cluster and
then starts the standard postgres test tooling. You almost never have to change
this file.