Saving the client IP address once per the lifetime of a backend. This avoid
the expensive operation multiple times, and hence improving performance
significantly.
This bug uncovered serious issues with how the data was being stored by PSGM.
So it require a complete redesign.
pg_stat_monitor now stores the data locally within the backend process's local
memory. The data is only stored when the query completes. This reduces the
number of lock acquisitions that were previously needed during various stages
of the execution. Also, this avoids data loss in case the current bucket
changes during execution. Also, the unavailability of jumble state during later
stages of executions was causing pg_stat_monitor to save non-normalized query.
This was a major problem as well.
pg_stat_monitor specific memory context is implemented. It is used for saving
data locally. The context memory callback helps us clear the locally saved data
so that we do not store it multiple times in the shared hash.
As part of this major rewrite, pgss reference in function and variable names
is changed to pgsm. Memory footprint for the entries is reduced, data types
are corrected where needed, and we've removed unused variables, functions and
macros.
This patch was mutually created by:
Co-authored-by: Hamid Akhtar <hamid.akhtar@percona.com>
Co-authored-by: Muhammad Usama <muhammad.usama@percona.com>
Removed use of String::Util perl module from TAP test cases, and now using
Text::Trim module instead, as that is more stable. Also removed the
Data::Str2Num perl module as it was not needed any more.
Disallow V1 API to be used with V2.0 lib and remove pg_stat_monitor--1.0.sql
as part of that. A few adjustments to 1.x to 2.0 upgrade script are also
part of the commit
Resolved the issue with histogram outlier buckets. Also updated
the printing of bucket ranges to be in correct set notation with
reference to brackets. The lower bounds of buckets always have an
exclusive range except for the first bucket, and the upper bounds
always have an inclusive value.
( or ) => exclusive
{ or } => inclusive
The entire range is enclosed within the {} brackets.
for utility statements as well
Added the necessary capture of resource usage in the process
utility function. We are now storing CPU and user timings for a
utility statement.
As an observability tool that serves data to other tools, data must be output without any loss. So rounding off causes data loss and rounding off errors when comparing different columns.
Therefore, it was decided to eliminate rounding off when outputting values. Any consumer of this data should round off data to whatever precision it prefers.
This behaviour is also consistent with pg_stat_statements.
1) Added the Group by clause to make sure that bucket change doesn't have any
impact on aggregates of queries.
2) Updated column names where required.
3) Updated pgbench parameters to reduce the time taken by test case, around
70-80% decrease in time taken by test case.
1) Added the Group by clause to make sure that bucket change doesn't have any
impact on aggregates of queries.
2) Updated column names where required.
3) Updated pgbench parameters to reduce the time taken by test case, around
70-80% decrease in time taken by test case.
There is no specific test case where I can either reproduce or validate
the fix. Though, one of the suspects is this condition in pgss_store.
Therefore removed, and it requires verification.
Replaced the error on server start with a warning. The functionality
now handles "pgsm_histogram_buckets" as the maximum number of histogram
buckets to be created. On init, pg_stat_monitor calculates the max
number of buckets that can be created within the given min/max time
range. If the number is below the user configuration, it emits a
warning in the log file stating the number of max buckets set.
Added buckets for queries that take less than minimum histogram time
and one for the ones taking more than the max value specified.
Also, in case the buckets end up overlapping, on server start, an
error will be thrown informing the user of this issue and requesting
a rectification.
Refactored the code to consolidate the calculations in a single
function.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
The view now carries all the columns as pg_stat_statements. This required fixing
data types of some of the columns, renaming a few, as well inclusion of new
columns to make the view fully compatible with pg_stat_statements.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating the upgrade sql file from 1.0 to 2.0 version linked with this issue
changes.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating datum calls to use UInt64 rather than Int64.
* PG-582: blk_read_time and blk_write_time are not being rounded.
Added the round off within the internal function so that values for
blk_read_time, blk_write_time are rounded off to 4 decimal places.
Additionally, added rounding off for the PG15+ columns of
temp_blk_read_time and temp_blk_write_time.
* PG-582: blk_read_time and blk_write_time are not being rounded.
Added rounding off for four JIT related columns introduced for PG15.
The bucket start time reported by pg_stat_monitor does not match the PG time and
timezone. The fix is to use TimestampTz for recording the bucket start time.
pgsm_get_ss() must only be called when pg_stat_monitor.so is loaded.
Fix is to move the pgsm_get_ss() call after checking if the pg_stat_monotor
library is loaded or not.
* PG-488: pg_stat_monitor: Overflow management.
Reimplement the storage mechanism of buckets (for PG-15 onward) and query texts
using Dynamic shared memory. Since the dynamic shared memory can grow into a
swap area, so we get the overflow out of the box.
As PostgreSQL versions prior to V15 does not support sequence scan on dynamic
shared memory hashes, so older versions has to live with the classic shared
memory hash for storing the buckets.
Another noteworthy change with the new design is: it saves the query pointer
inside the bucket, and eventually, the query text gets evicted with the bucket
recycle.
Finally, the dynamic shared memory hash has a built-in locking mechanism, so we
can revisit the whole locking in pg_stat_monitor has the potential for lots of
performance improvements.
* Fixing tap test reported issues and also disabling dynamic hash for all versions
* Updating the expected out file for top_query test case
Co-authored-by: Hamid Akhtar <hamid.akhtar@percona.com>
As we are using compute_query_id on pg14 onwards for PGSM and it
causes the server installcheck-world to fail (same behaviour with PGSS).
To test installcheck-world on pg14 onwards we need to disable compute_query_id
and run server installcheck-world. But for PGSM regression we will still have
compute_query_id on.
queryid creation mechanism.
Resolving the crash identified by regression and reported by Naeem.
This fix resolves the issue with incorrect query length in case of
normalized query when the query length exceeds PGSM_QUERY_MAX_LEN.
Resolving the crash identify by regression and reported by Naeem.
