As an observability tool that serves data to other tools, data must be output without any loss. So rounding off causes data loss and rounding off errors when comparing different columns.
Therefore, it was decided to eliminate rounding off when outputting values. Any consumer of this data should round off data to whatever precision it prefers.
This behaviour is also consistent with pg_stat_statements.
1) Added the Group by clause to make sure that bucket change doesn't have any
impact on aggregates of queries.
2) Updated column names where required.
3) Updated pgbench parameters to reduce the time taken by test case, around
70-80% decrease in time taken by test case.
1) Added the Group by clause to make sure that bucket change doesn't have any
impact on aggregates of queries.
2) Updated column names where required.
3) Updated pgbench parameters to reduce the time taken by test case, around
70-80% decrease in time taken by test case.
There is no specific test case where I can either reproduce or validate
the fix. Though, one of the suspects is this condition in pgss_store.
Therefore removed, and it requires verification.
Replaced the error on server start with a warning. The functionality
now handles "pgsm_histogram_buckets" as the maximum number of histogram
buckets to be created. On init, pg_stat_monitor calculates the max
number of buckets that can be created within the given min/max time
range. If the number is below the user configuration, it emits a
warning in the log file stating the number of max buckets set.
Added buckets for queries that take less than minimum histogram time
and one for the ones taking more than the max value specified.
Also, in case the buckets end up overlapping, on server start, an
error will be thrown informing the user of this issue and requesting
a rectification.
Refactored the code to consolidate the calculations in a single
function.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
The view now carries all the columns as pg_stat_statements. This required fixing
data types of some of the columns, renaming a few, as well inclusion of new
columns to make the view fully compatible with pg_stat_statements.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating the upgrade sql file from 1.0 to 2.0 version linked with this issue
changes.
* PG-543: pg_stat_monitor: PostgreSQL's pg_stat_statements compatible view.
Updating datum calls to use UInt64 rather than Int64.
* PG-582: blk_read_time and blk_write_time are not being rounded.
Added the round off within the internal function so that values for
blk_read_time, blk_write_time are rounded off to 4 decimal places.
Additionally, added rounding off for the PG15+ columns of
temp_blk_read_time and temp_blk_write_time.
* PG-582: blk_read_time and blk_write_time are not being rounded.
Added rounding off for four JIT related columns introduced for PG15.
The bucket start time reported by pg_stat_monitor does not match the PG time and
timezone. The fix is to use TimestampTz for recording the bucket start time.
pgsm_get_ss() must only be called when pg_stat_monitor.so is loaded.
Fix is to move the pgsm_get_ss() call after checking if the pg_stat_monotor
library is loaded or not.
* PG-488: pg_stat_monitor: Overflow management.
Reimplement the storage mechanism of buckets (for PG-15 onward) and query texts
using Dynamic shared memory. Since the dynamic shared memory can grow into a
swap area, so we get the overflow out of the box.
As PostgreSQL versions prior to V15 does not support sequence scan on dynamic
shared memory hashes, so older versions has to live with the classic shared
memory hash for storing the buckets.
Another noteworthy change with the new design is: it saves the query pointer
inside the bucket, and eventually, the query text gets evicted with the bucket
recycle.
Finally, the dynamic shared memory hash has a built-in locking mechanism, so we
can revisit the whole locking in pg_stat_monitor has the potential for lots of
performance improvements.
* Fixing tap test reported issues and also disabling dynamic hash for all versions
* Updating the expected out file for top_query test case
Co-authored-by: Hamid Akhtar <hamid.akhtar@percona.com>
As we are using compute_query_id on pg14 onwards for PGSM and it
causes the server installcheck-world to fail (same behaviour with PGSS).
To test installcheck-world on pg14 onwards we need to disable compute_query_id
and run server installcheck-world. But for PGSM regression we will still have
compute_query_id on.
queryid creation mechanism.
Resolving the crash identified by regression and reported by Naeem.
This fix resolves the issue with incorrect query length in case of
normalized query when the query length exceeds PGSM_QUERY_MAX_LEN.
Resolving the crash identify by regression and reported by Naeem.
Regardless of the database or the user, the same query will yield the
same query ID. As part of this, a new column, 'pgsm_query_id', is added.
* pgsm_query_id:
pgsm_query_id has the same data type of int8 as the queryid column. If
the incoming SQL command includes any constants, it internally normalizes
the query to remove those constant values with placeholders. Otherwise,
it uses the query directly to generate the query hash.
Since we no longer depend on the server's parse tree mechanism, we can
generate the same hash for the same query text for all server versions.
Also, it is important to note that the hash being calculated is a database,
schema and user independent. So same query text in different databases
will generate the same hash.
This column is not part of the key; rather, for observability purposes only.
* Regression
SQL test case pgsm_query_id.sql is added to the SQL regression.
As part of this PR, also updated regression test cases that are related to
following JIRA issues as well.
PG-354 pg_stat_monitor: Remove pg_stat_monitor_settings view
Now we not using pg_stat_monitor_settings view, due to this change majority of
TAP testcase requried output changes.
PG-558: Create test case to verify the function names and count in PGSM.
Added additional output file for SQL test case.
PG-554: Remove redundant expected output files from regression.
Removed unnecessary output files in TAP testcases where these were not needed.
Reimplement the storage mechanism of buckets and query texts
using Dynamic shared memory. Since the dynamic shared memory
can grow into a swap area, so we get the overflow out of the box.
oreover the new design saves the query pointer inside the bucket
and eventually, the query text gets evicted with the bucket recycle.
Finally, the dynamic shared memory hash has a built-in locking
mechanism so we can revisit the whole locking in pg_stat_monitor
has potential for lots of performance improvements
Removing the view for 2.0. Updating the required SQL files to manage
the upgrade. Downgrade from 2.x to 1.x is not supported.
Also part of this fix is the SQL regression. This does not update the
tap test cases.
In line with pg_stat_statments for PG15, This commit adds eight new cumulative
counters for jit operations, making it easier to diagnose how JIT is used in an
installation. And two new columns, temp_blk_read_time, and temp_blk_write_time,
respectively, show the time spent reading and writing temporary file blocks
on disk.
Moreover, The commit also contains a few indentations and API adjustments.
Muhammad Usama
Naeem Akhter
Naeem Akhter
Muhammad Usama
Hamid Akhtar
Ibrar Ahmed