* Denormalize prepared statement queries
Added support for extracting query arguments for prepared statements
when `pg_stat_monitor.pgsm_normalized_query` is off.
Previously pg_stat_monitor was unable to extract the arguments for
prepared statements, thus leaving queries with placeholders $1
.. $N instead of the actual arguments.
* Optmize query denormalization
Instead of copying original query text byte by byte, copy data between
query placeholders in chunks, example:
`INSERT INTO foo(a, b, c) VALUES('test', 100, 'test again)'`
Would result in normalized query:
`INSERT INTO foo(a, b, c) VALUES($1, $2, $3)`
The original patch would copy the parts between placeholders byte by
byte, e.g. `INSERT INTO foo(a, b, c) VALUES(`, instead we can copy this
whole block at once, 1 function call and maybe 1 buffer re-allocation
per call.
Also make use of `appendBinaryStringInfo` to avoid calculating string
length as we have this info already.
* Optmize query denormalization(2)
Avoid allocating an array of strings for extracting query argument
values, instead append the current parameter value directly in the
buffer used to store the denormalized query.
This avoids not only unnecessary memory allocations, but also copying
data between temporary memory and the buffer.
* Store denormalized query only under certain constraints
This commit introduces a little optimization along with a feature, it
stores the query in denormalized form only under the circumstances
below:
- The psgm_normalized_query GUC is disabled (off).
- The query is seem for the first time, or the query total
execution time exceeds the mean execution time calculated for
the previous queries.
Having the query which took most execution time along with it's
arguments could help users in further investigating performance issues.
* Fix regression tests
When query normalization is disabled utility queries like SELECT 10+20
are now stored as is, instead of SELECT $1+$2.
Also when functions or sub queries are created the arguments used
internally by the function or subqueries will be replaced by NULL instead
of $1..$N. The actual arguments will be displayed when the function or
subquery is actually invoked.
* Add query denormalization regression test for prepared statements
Ensures that the denormalization of prepared statements is working, also
ensure that a query which takes more time to execute replaces the
previous denormalized query.
* Updated pgsm_query_id regression tests
With the query dernomalization feature, having integer literals used in
sql like 1, or 2 could create some confusion on whether those are
placeholders or constant values, thus this commit updates the
pgsm_query_id regression test to use different integer literals to avoid
confusion.
* Improve query denormalization regression test
Add a new test case:
1. Execute a prepared statement with larger execution time first.
2. Execute the same prepared statement with cheap execution time.
3. Ensures that the denormalized heavy query is not replaced by the
cheaper.
* Format source using pgindent
* Fix top query regression tests on PG 12,13
On PG 12, 13, the internal return instruction in the following function:
```
CREATE OR REPLACE FUNCTION add(int, int) RETURNS INTEGER AS
$$
BEGIN
return (select $1 + $2);
END; $$ language plpgsql;
```
Is stored as SELECT (select expr1 + expr2)
On PG 14 onward it's stored just as SELECT (expr1 + expr2)
* PG-592: Treat queries with different parent queries as separate entries
1. Previously pg_stat_monitor had a `topquery` and `topqueryid` field, but it was only a sample:
it showed one of the top queries executing the specific query.
With this change, the same entry executed by two different functions will result in two entries in the statistics table.
2. This also fixes a bug where the content of these field disappeared for every second query executed:
previously the update function changed topqueryid to `0` if it was non zero, and changed it to the proper id when it was 0.
This resulted in an alternating behavior, where for every second executed query the top query disappeared.
After these changes, the top query is always shown.
3. The previous implementation also leaked dsa memory used to store the parent queries. This is now also fixed.
* PG-502: Fixing review comments
* dsa_free changed to assert as it can never happen
* restructured the ifs to be cleaner
Note: kept the two-level ifs, as that makes more sense with the assert
Note: didn't convert nested_level checks to macro, as it is used differently at different parts of the code
* PG-502: Fixing review comments
* PG-592 Add regression test
* Make test compatible with PG12
* Remove redundant line
---------
Co-authored-by: Artem Gavrilov <artem.gavrilov@percona.com>
* Cache application name for every backed instance
* Improve pg_get_backend_status performance for PG16 and PG17
* Fix
* Make application_name tracking disabled by default
* Meke app name tracking opt-out
* Format newly added code with pgindent
* Fix build for PG17
* Fix
* Temporary disable all workflows
* Add build workflow with PG17
* Fix incompatibilities
* Fix 007_settings_pgsm_query_shared_buffer.pl test
* Fix 018_column_names.pl
* Fix 025_compare_pgss.pl
* Remove tuplestore_donestoring usage at all
* Rename I/O timing statistics columns to shared_blk_{read|write}_time
* Fix comments with fileds numbers
* Fix format
* Revert "Temporary disable all workflows"
This reverts commit 12e75beb63.
* Disable all workflows except check and build for PG 15, 16 and 17
* Fix
* Fix comments
* Fix migration
* Use REL_17_BETA1 in CI
* Add timers tests to 028_temp_block.pl
* Add local blocks timing statistics columns local_blk_{write|read}_time
* Fix t/027_local_blocks.pl test for older PG versions
* Fix
* Add jit_deform_{count|time} metrics
* Fix
* Add stats_since and minmax_stats_since fields
* Revert "Disable all workflows except check and build for PG 15, 16 and 17"
This reverts commit 73febf3aee.
* Fix t/028_temp_block.pl for PG14 and below
* Fix build for PG12
* Add pgdg workflow for PG17
* Try to fix PG pgdg workflow
* Fixes and formatting
* Format code
* Add level tracking regression test
* Fix nesting level tracking
* Format code
* Add level tracking test expected result for PG13
* Fix for PG12
* Skip level tracking regression test for PG version less than 14
* Fix toplevel calculation for older PG version
* Fix level tracking test results
* Fix nesting level counting for older PG version
* Revert "Fix nesting level counting for older PG version"
This reverts commit 3e91da8010.
* Fix level tracking for older PG versions once again
* Set REL_17_BETA2 tag for PG
* Add CI badge for PG17
* Use PG17 for examples in readme
* PG-607: Allow histogram to track queries in sub-ms time brackets
Updated the GUC configuration and the relevant histogram functionality
to track queries in lower cardinality than ms. This is done by saving
the GUC values for histogram min and max values in real (double) type.
All test cases except for the 030 tap test are passing. The test case
needs an update.
* Fixing regression issues for v12 and below because of histogram changes.
* PG-606: New GUC required for enabling/disabling of pgsm_query_id calculation
Adds a new GUC pg_stat_monitor.pgsm_enable_pgsm_query_id to enable/disable
pgsm query id calculation. Apart from that patch also refactors the GUC-related
code to match PostgreSQL conventions.
Moreover, the commit also changes the pgsm_enable_overflow GUC to boolean
instead of enum.
This bug uncovered serious issues with how the data was being stored by PSGM.
So it require a complete redesign.
pg_stat_monitor now stores the data locally within the backend process's local
memory. The data is only stored when the query completes. This reduces the
number of lock acquisitions that were previously needed during various stages
of the execution. Also, this avoids data loss in case the current bucket
changes during execution. Also, the unavailability of jumble state during later
stages of executions was causing pg_stat_monitor to save non-normalized query.
This was a major problem as well.
pg_stat_monitor specific memory context is implemented. It is used for saving
data locally. The context memory callback helps us clear the locally saved data
so that we do not store it multiple times in the shared hash.
As part of this major rewrite, pgss reference in function and variable names
is changed to pgsm. Memory footprint for the entries is reduced, data types
are corrected where needed, and we've removed unused variables, functions and
macros.
This patch was mutually created by:
Co-authored-by: Hamid Akhtar <hamid.akhtar@percona.com>
Co-authored-by: Muhammad Usama <muhammad.usama@percona.com>
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
The view now carries all the columns as pg_stat_statements. This required fixing
data types of some of the columns, renaming a few, as well inclusion of new
columns to make the view fully compatible with pg_stat_statements.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating the upgrade sql file from 1.0 to 2.0 version linked with this issue
changes.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating datum calls to use UInt64 rather than Int64.
Regardless of the database or the user, the same query will yield the
same query ID. As part of this, a new column, 'pgsm_query_id', is added.
* pgsm_query_id:
pgsm_query_id has the same data type of int8 as the queryid column. If
the incoming SQL command includes any constants, it internally normalizes
the query to remove those constant values with placeholders. Otherwise,
it uses the query directly to generate the query hash.
Since we no longer depend on the server's parse tree mechanism, we can
generate the same hash for the same query text for all server versions.
Also, it is important to note that the hash being calculated is a database,
schema and user independent. So same query text in different databases
will generate the same hash.
This column is not part of the key; rather, for observability purposes only.
* Regression
SQL test case pgsm_query_id.sql is added to the SQL regression.
As part of this PR, also updated regression test cases that are related to
following JIRA issues as well.
PG-354 pg_stat_monitor: Remove pg_stat_monitor_settings view
Now we not using pg_stat_monitor_settings view, due to this change majority of
TAP testcase requried output changes.
PG-558: Create test case to verify the function names and count in PGSM.
Added additional output file for SQL test case.
PG-554: Remove redundant expected output files from regression.
Removed unnecessary output files in TAP testcases where these were not needed.
Removing the view for 2.0. Updating the required SQL files to manage
the upgrade. Downgrade from 2.x to 1.x is not supported.
Also part of this fix is the SQL regression. This does not update the
tap test cases.
PG-554: Remove reduntant files and fix regression.
Removed old files with same name and add these files to fix sql regression
on PG 14 & 15.
1- regression/expected/error_1.out
2- regression/expected/error_insert_1.out
3- regression/expected/top_query_1.out
The query status monitoring code was used to track the current query state, for example,
parsing, executing and finishing. After careful review, we have figured out that
it does not make sense while a lot of time same query is running. Therefore it
is also consuming resources. This commit will remove that feature. The upgrade
SQL from 1.0 - 2.0 is also updated.
Following changes are included in this commit:
1. Updated pgsm.pm to enable runtime loading of PG server version dependent
perl modules that are needed for TAP testing. Similarly removed unneeded
code from this file that is not needed right now.
2. Added generic settings and helper functions to pgsm.pm that could be used
across different test cases.
3.Updated following TAP test case to use pgsm.pm based global settings and helper
functions while making sure that we reduce the clutter and duplicate code in
test cases, where possible.
t/001_settings_default.pl
t/002_settings_pgsm_track_planning.pl
t/003_settings_pgms_extract_comments.pl
t/004_settings_pgsm_track.pl
t/005_settings_pgsm_enable_query_plan.pl
t/006_settings_pgsm_overflow_target.pl
t/007_settings_pgsm_query_shared_buffer.pl
t/008_settings_pgsm_histogram_buckets.pl
t/009_settings_pgsm_histogram_max.pl
t/010_settings_pgsm_histogram_min.pl
t/011_settings_pgsm_bucket_time.pl
t/012_settings_pgsm_max_buckets.pl
t/013_settings_pgsm_normalized_query.pl
t/014_settings_pgsm_track_utility.pl
t/015_settings_pgsm_query_max_len.pl
t/016_settings_pgsm_max.pl
t/017_execution_stats.pl
t/019_insufficient_shared_space.pl
t/020_buffer_overflow.pl
t/021_misc_1.pl
t/022_misc_2.pl
t/023_missing_queries.pl
t/024_check_timings.pl
t/025_compare_pgss.pl
t/026_shared_blocks.pl
t/027_local_blocks.pl
t/028_temp_block.pl
4. Removed following TAP test cases as these are no longer needed and are
covered by existing other test cases.
0001_settings_pgsm_track_planning.pl
0002_settings_pgsm_enable_query_plan.pl
5. Added more out files for histogram sql test cases to cover the behavior for
bucket_start_time and server versions.
regression/expected/histogram_3.out
regression/expected/histogram_4.out
regression/expected/histogram_5.out
regression/expected/histogram_6.out
6. Added following out file that is PG server version 15 specific.
t/expected/007_settings_pgsm_query_shared_buffer.out.15
* PG-475: Inconsistent behaviour of PGSM
Reverting the bucket locking mechanism to previous behavior. This has
a lot of room for improvement that needs to be part of a major refactoring
in the 2.x release.
* PG-481 Release notes 1.1.0 (#294)
modified: RELEASE_NOTES.md
* PG-500: Bump the version of pg_stat_monitor to 1.1.0 (#297)
* PG-501: Missing Buckets and incorrect calls count. (#298)
prev_bucket_sec holds the actual time at which the previous bucket was created
and it is used to compute if the previous bucket time has elapsed and when is
the time to create a new one. But since the bucket start time is rounded down
to logical time window start, that makes the prev_bucket_sec and bucket start
time out of sync with each other, and depending on the query arrival time there
is a high probability that a bucket gets missed especially when the last bucket
was created around the end of the bucket time window.
Solution is to keep the prev_bucket_sec and bucket start time in-sync.
Moreover, we are using the unint64 for storing the prev_bucket_sec which is kind
of an overkill and a simple uint should be good enough for the purpose. But that
change can be taken up as part of the create-bucket function refactoring task.
* PG-501: Missing Buckets and incorrect calls count.
Ensuring the outer bound for the bucket is an exclusive boundary and it
as it belongs to the next bucket. To explain the point further, a set of
five second bucket would be:
Bucket 1: 00:00:00.00 -> 00:00:04.99...
Bucket 2: 00:00:00.05 -> 00:00:09.99...
Bucket 3: 00:00:00.10 -> 00:00:14.99...
...
Co-authored-by: Ibrar Ahmed <ibrar.ahmed@percona.com>
Co-authored-by: Anastasia Alexandrova <anastasia.alexandrova@percona.com>
Co-authored-by: Muhammad Usama <m.usama@gmail.com>
Regression was failing in main branch after merging REL_1_STABLE. Changes to
stablise the regression are summarised below.
Updated following TAP testcase with 'where' clause for filtering GUC specific row
from settings view. Intention is to fetch only GUC value and test it.
1) t/001_settings_default.pl
2) t/002_settings_pgsm_track_planning.pl
3) t/003_settings_pgms_extract_comments.pl
4) t/004_settings_pgsm_track.pl
5) t/005_settings_pgsm_enable_query_plan.pl
6) t/006_settings_pgsm_overflow_target.pl
7) t/007_settings_pgsm_query_shared_buffer.pl
8) t/008_settings_pgsm_histogram_buckets.pl
9) t/009_settings_pgsm_histogram_max.pl
10) t/010_settings_pgsm_histogram_min.pl
11) t/011_settings_pgsm_bucket_time.pl
12) t/012_settings_pgsm_max_buckets.pl
13) t/013_settings_pgsm_normalized_query.pl
14) t/014_settings_pgsm_track_utility.pl
15) t/015_settings_pgsm_query_max_len.pl
16) t/016_settings_pgsm_max.pl
Updated following expected files for TAP testcase where testcase was updated
with 'where' clause to filter GUC specific row only from settings view.
1) t/expected/001_settings_default.out
2) t/expected/002_settings_pgsm_track.outanning.out
3) t/expected/003_settings_pgms_extract_comments.out
4) t/expected/004_settings_pgsm_track.out
5) t/expected/005_settings_pgsm_enable_query.out
6) t/expected/006_settings_pgsm_overflow_target.out
7) t/expected/007_settings_pgsm_query_shared_buffer.out
8) t/expected/008_settings_pgsm_histogram_buckets.out
9) t/expected/009_settings_pgsm_histogram_max.out
10) t/expected/010_settings_pgsm_histogram_min.out
11) t/expected/011_settings_pgsm_bucket_time.out
12) t/expected/012_settings_pgsm_max_buckets.out
13) t/expected/013_settings_pgsm_normalized_query.out
14) t/expected/014_settings_pgsm_track_utility.out
15) t/expected/015_settings_pgsm_query_max_len.out
16) t/expected/016_settings_pgsm_max.out
17) t/expected/017_execution_stats.out
Updated expected files for following sql based testcases due to changed default
value of pgsm_normalized_query.
1) error_insert_1.out
2) guc_1.out
3) top_query_1.out
Enabled GUC pgsm_normalized_query to 'yes' for following testcases.
1) t/024_check_timings.pl
2) t/025_compare_pgss.pl
3) t/026_shared_blocks.pl
Added the output file for histogram testcase to accomodate behaviour.
Changes suggested by Ibrar to make regression more modular (PG-440), are also
made part of regression so those these can used for future improvements.
Removed commented/unneeded steps from t/017_execution_stats.pl.
Updated testcase tags.sql and output file tags.out to accomodate enabling of
pgsm_extract_comments to yes.
Updated error_1.out output file with required output, it was overwritten in
one of previous commits.
Updated test condition (count of calls and bucket) in t/023_missing_queries.pl
as per comment from Ibrar in PG-461 where PGSM does not track the pg_sleep in
some cases.
pg_stat_monitor.pgsm_normalized_query needs to be 0 by default because
people will get Query examples. This is one of the main user-faced
advantages of pgsm over pgss (pg_stat_statements).
There was no maximum limit set for the number of maximum histograms
bucket, which can lead to a crash in case of higher value.
PG-382: Adjust the maximum value for histogram buckets.
Fix the regression issue related to GUC and set the maximum
buckets value correctly.
PG-382: Adjust the maximum value for histogram buckets.
Fix the TAP test cases.
There was no maximum limit set for the number of maximum histograms
bucket, which can lead to a crash in case of higher value.
PG-382: Adjust the maximum value for histogram buckets.
Fix the regression issue for PostgreSQL-12.
PG-382: Adjust the maximum value for histogram buckets.
Fix the TAP test cases.
This commit brings following changes to this branch:
1) Port changes/additions TAP testing from main branch to this branch, under PG-292.
2) Changes to test cases due to GUCs change, under PG-331.
3) Call counts verfications, under PG-338.
4) Changes to github workflows to accomodate automation for TAP testing, under PG-343.
Diego Fronza
Zsolt Parragi
Artem Gavrilov
Hamid Akhtar
Naeem Akhter
Muhammad Usama
Naeem Akhter
Ibrar Ahmed
Kai Wagner
Hamid Akhtar
Ibrar Ahmed
Ibrar Ahmed
Naeem Akhter
Diego Fronza