This is the initial framework for locally maintaining hash entries so that we can insert the data in one go in the hash table being maintained in the shared memory.
Pending issues:
This causes the regression to fail (and crash) from the counters test case
The top query test case is failing,
pgsm_store function is not saving all the data at the moment, especially the buffers, JIT and WAL information.
The total time needs to be stored separately for planning and execution.
Removed use of String::Util perl module from TAP test cases, and now using
Text::Trim module instead, as that is more stable. Also removed the
Data::Str2Num perl module as it was not needed any more.
Disallow V1 API to be used with V2.0 lib and remove pg_stat_monitor--1.0.sql
as part of that. A few adjustments to 1.x to 2.0 upgrade script are also
part of the commit
Resolved the issue with histogram outlier buckets. Also updated
the printing of bucket ranges to be in correct set notation with
reference to brackets. The lower bounds of buckets always have an
exclusive range except for the first bucket, and the upper bounds
always have an inclusive value.
( or ) => exclusive
{ or } => inclusive
The entire range is enclosed within the {} brackets.
for utility statements as well
Added the necessary capture of resource usage in the process
utility function. We are now storing CPU and user timings for a
utility statement.
As an observability tool that serves data to other tools, data must be output without any loss. So rounding off causes data loss and rounding off errors when comparing different columns.
Therefore, it was decided to eliminate rounding off when outputting values. Any consumer of this data should round off data to whatever precision it prefers.
This behaviour is also consistent with pg_stat_statements.
1) Added the Group by clause to make sure that bucket change doesn't have any
impact on aggregates of queries.
2) Updated column names where required.
3) Updated pgbench parameters to reduce the time taken by test case, around
70-80% decrease in time taken by test case.
1) Added the Group by clause to make sure that bucket change doesn't have any
impact on aggregates of queries.
2) Updated column names where required.
3) Updated pgbench parameters to reduce the time taken by test case, around
70-80% decrease in time taken by test case.
There is no specific test case where I can either reproduce or validate
the fix. Though, one of the suspects is this condition in pgss_store.
Therefore removed, and it requires verification.
Replaced the error on server start with a warning. The functionality
now handles "pgsm_histogram_buckets" as the maximum number of histogram
buckets to be created. On init, pg_stat_monitor calculates the max
number of buckets that can be created within the given min/max time
range. If the number is below the user configuration, it emits a
warning in the log file stating the number of max buckets set.
Added buckets for queries that take less than minimum histogram time
and one for the ones taking more than the max value specified.
Also, in case the buckets end up overlapping, on server start, an
error will be thrown informing the user of this issue and requesting
a rectification.
Refactored the code to consolidate the calculations in a single
function.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
The view now carries all the columns as pg_stat_statements. This required fixing
data types of some of the columns, renaming a few, as well inclusion of new
columns to make the view fully compatible with pg_stat_statements.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating the upgrade sql file from 1.0 to 2.0 version linked with this issue
changes.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating datum calls to use UInt64 rather than Int64.
* PG-582: blk_read_time and blk_write_time are not being rounded.
Added the round off within the internal function so that values for
blk_read_time, blk_write_time are rounded off to 4 decimal places.
Additionally, added rounding off for the PG15+ columns of
temp_blk_read_time and temp_blk_write_time.
* PG-582: blk_read_time and blk_write_time are not being rounded.
Added rounding off for four JIT related columns introduced for PG15.
The bucket start time reported by pg_stat_monitor does not match the PG time and
timezone. The fix is to use TimestampTz for recording the bucket start time.
pgsm_get_ss() must only be called when pg_stat_monitor.so is loaded.
Fix is to move the pgsm_get_ss() call after checking if the pg_stat_monotor
library is loaded or not.
* PG-488: pg_stat_monitor: Overflow management.
Reimplement the storage mechanism of buckets (for PG-15 onward) and query texts
using Dynamic shared memory. Since the dynamic shared memory can grow into a
swap area, so we get the overflow out of the box.
As PostgreSQL versions prior to V15 does not support sequence scan on dynamic
shared memory hashes, so older versions has to live with the classic shared
memory hash for storing the buckets.
Another noteworthy change with the new design is: it saves the query pointer
inside the bucket, and eventually, the query text gets evicted with the bucket
recycle.
Finally, the dynamic shared memory hash has a built-in locking mechanism, so we
can revisit the whole locking in pg_stat_monitor has the potential for lots of
performance improvements.
* Fixing tap test reported issues and also disabling dynamic hash for all versions
* Updating the expected out file for top_query test case
Co-authored-by: Hamid Akhtar <hamid.akhtar@percona.com>
Muhammad Usama
Hamid Akhtar
Naeem Akhter
Vadim Yalovets
Muhammad Usama
Kai Wagner
Ibrar Ahmed
Naeem Akhter