Following changes are included in this commit:
1. Updated pgsm.pm to enable runtime loading of PG server version dependent
perl modules that are needed for TAP testing. Similarly removed unneeded
code from this file that is not needed right now.
2. Added generic settings and helper functions to pgsm.pm that could be used
across different test cases.
3.Updated following TAP test case to use pgsm.pm based global settings and helper
functions while making sure that we reduce the clutter and duplicate code in
test cases, where possible.
t/001_settings_default.pl
t/002_settings_pgsm_track_planning.pl
t/003_settings_pgms_extract_comments.pl
t/004_settings_pgsm_track.pl
t/005_settings_pgsm_enable_query_plan.pl
t/006_settings_pgsm_overflow_target.pl
t/007_settings_pgsm_query_shared_buffer.pl
t/008_settings_pgsm_histogram_buckets.pl
t/009_settings_pgsm_histogram_max.pl
t/010_settings_pgsm_histogram_min.pl
t/011_settings_pgsm_bucket_time.pl
t/012_settings_pgsm_max_buckets.pl
t/013_settings_pgsm_normalized_query.pl
t/014_settings_pgsm_track_utility.pl
t/015_settings_pgsm_query_max_len.pl
t/016_settings_pgsm_max.pl
t/017_execution_stats.pl
t/019_insufficient_shared_space.pl
t/020_buffer_overflow.pl
t/021_misc_1.pl
t/022_misc_2.pl
t/023_missing_queries.pl
t/024_check_timings.pl
t/025_compare_pgss.pl
t/026_shared_blocks.pl
t/027_local_blocks.pl
t/028_temp_block.pl
4. Removed following TAP test cases as these are no longer needed and are
covered by existing other test cases.
0001_settings_pgsm_track_planning.pl
0002_settings_pgsm_enable_query_plan.pl
5. Added more out files for histogram sql test cases to cover the behavior for
bucket_start_time and server versions.
regression/expected/histogram_3.out
regression/expected/histogram_4.out
regression/expected/histogram_5.out
regression/expected/histogram_6.out
6. Added following out file that is PG server version 15 specific.
t/expected/007_settings_pgsm_query_shared_buffer.out.15
* PG-475: Inconsistent behaviour of PGSM
Reverting the bucket locking mechanism to previous behavior. This has
a lot of room for improvement that needs to be part of a major refactoring
in the 2.x release.
* PG-481 Release notes 1.1.0 (#294)
modified: RELEASE_NOTES.md
* PG-500: Bump the version of pg_stat_monitor to 1.1.0 (#297)
* PG-501: Missing Buckets and incorrect calls count. (#298)
prev_bucket_sec holds the actual time at which the previous bucket was created
and it is used to compute if the previous bucket time has elapsed and when is
the time to create a new one. But since the bucket start time is rounded down
to logical time window start, that makes the prev_bucket_sec and bucket start
time out of sync with each other, and depending on the query arrival time there
is a high probability that a bucket gets missed especially when the last bucket
was created around the end of the bucket time window.
Solution is to keep the prev_bucket_sec and bucket start time in-sync.
Moreover, we are using the unint64 for storing the prev_bucket_sec which is kind
of an overkill and a simple uint should be good enough for the purpose. But that
change can be taken up as part of the create-bucket function refactoring task.
* PG-501: Missing Buckets and incorrect calls count.
Ensuring the outer bound for the bucket is an exclusive boundary and it
as it belongs to the next bucket. To explain the point further, a set of
five second bucket would be:
Bucket 1: 00:00:00.00 -> 00:00:04.99...
Bucket 2: 00:00:00.05 -> 00:00:09.99...
Bucket 3: 00:00:00.10 -> 00:00:14.99...
...
Co-authored-by: Ibrar Ahmed <ibrar.ahmed@percona.com>
Co-authored-by: Anastasia Alexandrova <anastasia.alexandrova@percona.com>
Co-authored-by: Muhammad Usama <m.usama@gmail.com>
Regression was failing in main branch after merging REL_1_STABLE. Changes to
stablise the regression are summarised below.
Updated following TAP testcase with 'where' clause for filtering GUC specific row
from settings view. Intention is to fetch only GUC value and test it.
1) t/001_settings_default.pl
2) t/002_settings_pgsm_track_planning.pl
3) t/003_settings_pgms_extract_comments.pl
4) t/004_settings_pgsm_track.pl
5) t/005_settings_pgsm_enable_query_plan.pl
6) t/006_settings_pgsm_overflow_target.pl
7) t/007_settings_pgsm_query_shared_buffer.pl
8) t/008_settings_pgsm_histogram_buckets.pl
9) t/009_settings_pgsm_histogram_max.pl
10) t/010_settings_pgsm_histogram_min.pl
11) t/011_settings_pgsm_bucket_time.pl
12) t/012_settings_pgsm_max_buckets.pl
13) t/013_settings_pgsm_normalized_query.pl
14) t/014_settings_pgsm_track_utility.pl
15) t/015_settings_pgsm_query_max_len.pl
16) t/016_settings_pgsm_max.pl
Updated following expected files for TAP testcase where testcase was updated
with 'where' clause to filter GUC specific row only from settings view.
1) t/expected/001_settings_default.out
2) t/expected/002_settings_pgsm_track.outanning.out
3) t/expected/003_settings_pgms_extract_comments.out
4) t/expected/004_settings_pgsm_track.out
5) t/expected/005_settings_pgsm_enable_query.out
6) t/expected/006_settings_pgsm_overflow_target.out
7) t/expected/007_settings_pgsm_query_shared_buffer.out
8) t/expected/008_settings_pgsm_histogram_buckets.out
9) t/expected/009_settings_pgsm_histogram_max.out
10) t/expected/010_settings_pgsm_histogram_min.out
11) t/expected/011_settings_pgsm_bucket_time.out
12) t/expected/012_settings_pgsm_max_buckets.out
13) t/expected/013_settings_pgsm_normalized_query.out
14) t/expected/014_settings_pgsm_track_utility.out
15) t/expected/015_settings_pgsm_query_max_len.out
16) t/expected/016_settings_pgsm_max.out
17) t/expected/017_execution_stats.out
Updated expected files for following sql based testcases due to changed default
value of pgsm_normalized_query.
1) error_insert_1.out
2) guc_1.out
3) top_query_1.out
Enabled GUC pgsm_normalized_query to 'yes' for following testcases.
1) t/024_check_timings.pl
2) t/025_compare_pgss.pl
3) t/026_shared_blocks.pl
Added the output file for histogram testcase to accomodate behaviour.
Changes suggested by Ibrar to make regression more modular (PG-440), are also
made part of regression so those these can used for future improvements.
Removed commented/unneeded steps from t/017_execution_stats.pl.
Updated testcase tags.sql and output file tags.out to accomodate enabling of
pgsm_extract_comments to yes.
Updated error_1.out output file with required output, it was overwritten in
one of previous commits.
Updated test condition (count of calls and bucket) in t/023_missing_queries.pl
as per comment from Ibrar in PG-461 where PGSM does not track the pg_sleep in
some cases.
pg_stat_monitor.pgsm_normalized_query needs to be 0 by default because
people will get Query examples. This is one of the main user-faced
advantages of pgsm over pgss (pg_stat_statements).
There was no maximum limit set for the number of maximum histograms
bucket, which can lead to a crash in case of higher value.
PG-382: Adjust the maximum value for histogram buckets.
Fix the regression issue related to GUC and set the maximum
buckets value correctly.
PG-382: Adjust the maximum value for histogram buckets.
Fix the TAP test cases.
There was no maximum limit set for the number of maximum histograms
bucket, which can lead to a crash in case of higher value.
PG-382: Adjust the maximum value for histogram buckets.
Fix the regression issue for PostgreSQL-12.
PG-382: Adjust the maximum value for histogram buckets.
Fix the TAP test cases.
This commit brings following changes to this branch:
1) Port changes/additions TAP testing from main branch to this branch, under PG-292.
2) Changes to test cases due to GUCs change, under PG-331.
3) Call counts verfications, under PG-338.
4) Changes to github workflows to accomodate automation for TAP testing, under PG-343.
After fixing the problem with utility statements, this whole block:
do $$
declare
n integer:= 1;
begin
loop
PERFORM a,b,c,d FROM t1, t2, t3, t4
WHERE t1.a = t2.b AND t3.c = t4.d ORDER BY a;
exit when n = 1000;
n := n + 1;
end loop;
end $$;
Is only processed once, as those are nested statements, in order to
match the 1000 statements the GUC pg_stat_monitor.track must be set to
'all' and then back to the default of 'top' when done testing it.
This commit adds following three sql based testcases:
1) Test unique application name set by user.
2) Histogram function is working properly as desired.
3) Error on insert is shown with proper message.
As the query normalization and query cleaning is always done in the
right place (pgss_store), no more parsed queries have a trailling comma
';' at the end.
Also, on error regression test, after fixing some problems with utility
related queries, we now have two entries for the RAISE WARNING case, the
first entry is the utility query itself, the second entry is the error
message logged by emit_log_hook.
Some queries have the order adjusted due to the fix introduced by the
previous commits.
The regression tests required some adjustmentes as they were based on a
wrong behavior in pg_stat_monitor that was fixed in the last commits.
The problem was that pg_stat_monitor_reset() was not properly clearing
the query buffers, as such, some garbage queries were residing in the
buffers after calling pg_stat_monitor_reset().
One example of a problem, a query such as "SELECT 1 AS num" and the
same query with comments such as:
SELECT $1 AS num /* { "application", psql_app, "real_ip", 192.168.1.3)
*/
Are evaluated to the same query ID, if a test issue the first query, call
pg_stat_monitor_reset() to clear query buffer, then issue the second
query with comments, the result in pg_stat_monitor view would still contain
the first query without comments, this was leading to tests expecting
the wrong output, which is now fixed.
1) Enabled configure and build with proper flags and environment to make sure that
built server is aligned with pg and ppg package distribution in terms of features
and configurations. Earlier build (configure) was not aligned with pg community standard
configuration that are used for community builds and distribution.
2) Enabled installcheck-world regression test suites of the pg server, to verify
the stability and compatibility of pg server after loading pg_stat_monitor in
server.
3) Change in expected files of error.out and error_1.out for error testacase.
4) Disbaled and removed coverage using coveralls.io, as it was not serving the purpose.
(Note: installcheck-world was failing on pg-14 due to some changes that are done in pg14
or upstream PGSS, and that is causing additional line (Query Identifier) is output of some
of the test cases of installcheck-world so it is not enabled in this commit. Problem with
pg14 server installcheck regression is output of an extra line (Query identifier ****)
in some of the test cases after loading extension PGSM and that causes regression to fail
for server after library load.)
Changed the default value of pgsm_bucket_time from 300 to 60. Now neither PMM users will need to adjust the default value, nor
restart of PG server will be required for this purpose.
Kai Wagner
Naeem Akhter
Hamid Akhtar
Hamid Akhtar
Ibrar Ahmed
Ibrar Ahmed
Ibrar Ahmed
Naeem Akhter
Diego Fronza
Evgeniy Patlan
Naeem Akhter