external
/
pg_stat_monitor
mirror of https://github.com/percona/pg_stat_monitor.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Major refactoring to support multiple new features. 

Issue - (#16): PG-112: Change the column name "ip" to "client_ip" for readability purpose.
Issue - (#17): PG-111: Show all the queries from complete and incomplete buckets.
Issue - (#18): PG-108: Log the bucket start time.
Issue - (#19): PG-99: Response time histogram.
Issue - (#20): PG-97: Log CPU time for a query.
Issue - (#21): PG-96: Show objects(tables) involved in the query.
Issue - (#22): PG-93: Retain the bucket, and don't delete the bucket automatically.
Issue - (#23): PG-91: Log queries of all the databases.
Issue - (#24): PG-116: Restrict the query size.
Issue - (#3) : README file update.
pull/25/head
Ibrar Ahmed 2020-02-27 19:27:05 +00:00
parent 0e47677552
commit 179e99d692
4 changed files with 933 additions and 1634 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
Makefile
View File

@ -4,12 +4,11 @@ MODULE_big = pg_stat_monitor
OBJS = pg_stat_monitor.o $(WIN32RES) OBJS = pg_stat_monitor.o $(WIN32RES)
EXTENSION = pg_stat_monitor EXTENSION = pg_stat_monitor
DATA = pg_stat_monitor--1.0.sql DATA = pg_stat_monitor--1.0.sql
PGFILEDESC = "pg_stat_monitor - execution statistics of SQL statements" PGFILEDESC = "pg_stat_monitor - execution statistics of SQL statements"
LDFLAGS_SL += $(filter -lm, $(LIBS)) LDFLAGS_SL += $(filter -lm, $(LIBS))
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_stat_monitor/pg_stat_monitor.conf REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_stat_monitor/pg_stat_monitor.conf
REGRESS = pg_stat_monitor REGRESS = pg_stat_monitor

208
README.md
View File

@ -18,11 +18,11 @@ There are two ways to install pg_stat_monitor. The first is by downloading the p
##### Download and compile ##### Download and compile
The latest release of pg_stat_monitor can be downloaded from this GitHub page: The latest release of pg_stat_monitor can be downloaded from this GitHub page:
https://github.com/Percona-Lab/pg_stat_monitor/releases https://github.com/Percona/pg_stat_monitor/releases
or it can be downloaded using the git: or it can be downloaded using the git:
git clone git://github.com/Percona-Lab/pg_stat_monitor.git git clone git://github.com/Percona/pg_stat_monitor.git
After downloading the code, set the path for the [PostgreSQL][1] binary: After downloading the code, set the path for the [PostgreSQL][1] binary:
@ -57,46 +57,46 @@ There are four views, and complete statistics can be accessed using these views.
##### pg_stat_monitor ##### pg_stat_monitor
This is the main view which stores per query-based statistics, similar to pg_stat_statment with some additional columns. This is the main view which stores per query-based statistics, similar to pg_stat_statment with some additional columns.
\d pg_stat_monitor; postgres=# \d pg_stat_monitor;
View "public.pg_stat_monitor" View "public.pg_stat_monitor"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
---------------------+------------------+-----------+----------+--------- ---------------------+--------------------------+-----------+----------+---------
userid | oid | | | bucket | oid | | |
dbid | oid | | | bucket_start_time | timestamp with time zone | | |
queryid | bigint | | | userid | oid | | |
query | text | | | dbid | oid | | |
calls | bigint | | | queryid | text | | |
total_time | double precision | | | query | text | | |
min_time | double precision | | | calls | bigint | | |
max_time | double precision | | | total_time | double precision | | |
mean_time | double precision | | | min_time | double precision | | |
stddev_time | double precision | | | max_time | double precision | | |
rows | bigint | | | mean_time | double precision | | |
shared_blks_hit | bigint | | | stddev_time | double precision | | |
shared_blks_read | bigint | | | int8 | bigint | | |
shared_blks_dirtied | bigint | | | shared_blks_hit | bigint | | |
shared_blks_written | bigint | | | shared_blks_read | bigint | | |
local_blks_hit | bigint | | | shared_blks_dirtied | bigint | | |
local_blks_read | bigint | | | shared_blks_written | bigint | | |
local_blks_dirtied | bigint | | | local_blks_hit | bigint | | |
local_blks_written | bigint | | | local_blks_read | bigint | | |
temp_blks_read | bigint | | | local_blks_dirtied | bigint | | |
temp_blks_written | bigint | | | local_blks_written | bigint | | |
blk_read_time | double precision | | | temp_blks_read | bigint | | |
blk_write_time | double precision | | | temp_blks_written | bigint | | |
host | integer | | | blk_read_time | double precision | | |
hist_calls | text | | | blk_write_time | double precision | | |
hist_min_time | text | | | host | bigint | | |
hist_max_time | text | | | client_ip | inet | | |
hist_mean_time | text | | | resp_calls | text[] | | |
slow_query | text | | | cpu_user_time | double precision | | |
cpu_user_time | double precision | | | cpu_sys_time | double precision | | |
cpu_sys_time | double precision | | | tables_names | text[] | | |
These are new column added to have more detail about the query. These are new column added to have more detail about the query.
host: Client IP or Hostname client_ip: Client IP or Hostname
hist_calls: Hourly based 24 hours calls of query histogram hist_calls: Hourly based 24 hours calls of query histogram
hist_min_time: Hourly based 24 hours min time of query histogram hist_min_time: Hourly based 24 hours min time of query histogram
Hist_max_time: Hourly based 24 hours max time of query histogram Hist_max_time: Hourly based 24 hours max time of query histogram
@ -106,84 +106,72 @@ These are new column added to have more detail about the query.
cpu_sys_time: CPU System time for that query. cpu_sys_time: CPU System time for that query.
regression=# \d pg_stat_agg_database postgres=# \d pg_stat_agg_database
View "public.pg_stat_agg_database" View "public.pg_stat_agg_database"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
----------------+--------------------------+-----------+----------+--------- ---------------+------------------+-----------+----------+---------
queryid | bigint | | | bucket | oid | | |
dbid | bigint | | | queryid | text | | |
userid | oid | | | dbid | bigint | | |
host | inet | | | userid | oid | | |
total_calls | integer | | | client_ip | inet | | |
min_time | double precision | | | total_calls | integer | | |
max_time | double precision | | | min_time | double precision | | |
mean_time | double precision | | | max_time | double precision | | |
hist_calls | text[] | | | mean_time | double precision | | |
hist_min_time | text[] | | | resp_calls | text[] | | |
hist_max_time | text[] | | | cpu_user_time | double precision | | |
hist_mean_time | text[] | | | cpu_sys_time | double precision | | |
first_log_time | timestamp with time zone | | | query | text | | |
last_log_time | timestamp with time zone | | | tables_names | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
slow_query | text | | |
# \d pg_stat_agg_user postgres=# \d pg_stat_agg_user
View "public.pg_stat_agg_user" View "public.pg_stat_agg_user"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
----------------+--------------------------+-----------+----------+--------- ---------------+------------------+-----------+----------+---------
queryid | bigint | | | bucket | oid | | |
dbid | bigint | | | queryid | text | | |
userid | oid | | | dbid | bigint | | |
host | inet | | | userid | oid | | |
total_calls | integer | | | client_ip | inet | | |
min_time | double precision | | | total_calls | integer | | |
max_time | double precision | | | min_time | double precision | | |
mean_time | double precision | | | max_time | double precision | | |
hist_calls | text[] | | | mean_time | double precision | | |
hist_min_time | text[] | | | resp_calls | text[] | | |
hist_max_time | text[] | | | cpu_user_time | double precision | | |
hist_mean_time | text[] | | | cpu_sys_time | double precision | | |
first_log_time | timestamp with time zone | | | query | text | | |
last_log_time | timestamp with time zone | | | tables_names | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
slow_query | text | | |
# \d pg_stat_agg_host postgres=# \d pg_stat_agg_ip
View "public.pg_stat_agg_host" View "public.pg_stat_agg_ip"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
----------------+--------------------------+-----------+----------+--------- ---------------+------------------+-----------+----------+---------
queryid | bigint | | | bucket | oid | | |
dbid | bigint | | | queryid | text | | |
userid | oid | | | dbid | bigint | | |
host | inet | | | userid | oid | | |
total_calls | integer | | | client_ip | inet | | |
min_time | double precision | | | host | bigint | | |
max_time | double precision | | | total_calls | integer | | |
mean_time | double precision | | | min_time | double precision | | |
hist_calls | text[] | | | max_time | double precision | | |
hist_min_time | text[] | | | mean_time | double precision | | |
hist_max_time | text[] | | | resp_calls | text[] | | |
hist_mean_time | text[] | | | cpu_user_time | double precision | | |
first_log_time | timestamp with time zone | | | cpu_sys_time | double precision | | |
last_log_time | timestamp with time zone | | | query | text | | |
cpu_user_time | double precision | | | tables_names | text[] | | |
cpu_sys_time | double precision | | |
query | text | | |
slow_query | text | | |
Examples Examples
1 - In this query we are getting the exact value of f1 which is '05:06:07-07' in the case of slow queries. 1 - In this query we are getting the exact value of f1 which is '05:06:07-07' (special setting is required for this).
# select userid, queryid, query, slow_query, max_time, total_calls from pg_stat_agg_user; # select userid, queryid, query, max_time, total_calls from pg_stat_agg_user;
-[ RECORD 1 ]---------------------------------------------------------------------------------------- -[ RECORD 1 ]----------------------------------------------------------------------------------------
userid | 10 userid | 10
queryid | -203926152419851453 queryid | -203926152419851453
query | SELECT f1 FROM TIMETZ_TBL WHERE f1 < $1 query | SELECT f1 FROM TIMETZ_TBL WHERE f1 < '05:06:07-07';
slow_query | SELECT f1 FROM TIMETZ_TBL WHERE f1 < '05:06:07-07';
max_time | 1.237875 max_time | 1.237875
total_calls | 8 total_calls | 8
@ -206,6 +194,6 @@ Copyright (c) 2006 - 2019, Percona LLC.
See [`LICENSE`][2] for full details. See [`LICENSE`][2] for full details.
[1]: https://www.postgresql.org/ [1]: https://www.postgresql.org/
[2]: https://github.com/Percona-Lab/pg_stat_monitor/blob/master/LICENSE [2]: https://github.com/Percona/pg_stat_monitor/blob/master/LICENSE
[3]: https://github.com/Percona-Lab/pg_stat_monitor/issues/new [3]: https://github.com/Percona/pg_stat_monitor/issues/new
[4]: CONTRIBUTING.md [4]: CONTRIBUTING.md

108
pg_stat_monitor--1.0.sql
View File

@ -1,4 +1,4 @@
/* contrib/pg_stat_monitor/pg_stat_monitor--1.4.sql */ /* contrib/pg_stat_monitor/pg_stat_monitor--1.1.sql */
-- complain if script is sourced in psql, rather than via CREATE EXTENSION -- complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "CREATE EXTENSION pg_stat_monitor" to load this file. \quit \echo Use "CREATE EXTENSION pg_stat_monitor" to load this file. \quit
@ -10,10 +10,13 @@ AS 'MODULE_PATHNAME'
LANGUAGE C PARALLEL SAFE; LANGUAGE C PARALLEL SAFE;
CREATE FUNCTION pg_stat_monitor(IN showtext boolean, CREATE FUNCTION pg_stat_monitor(IN showtext boolean,
OUT bucket oid,
OUT userid oid, OUT userid oid,
OUT dbid oid, OUT dbid oid,
OUT queryid bigint,
OUT queryid text,
OUT query text, OUT query text,
OUT bucket_start_time timestamptz,
OUT calls int8, OUT calls int8,
OUT total_time float8, OUT total_time float8,
OUT min_time float8, OUT min_time float8,
@ -33,109 +36,126 @@ CREATE FUNCTION pg_stat_monitor(IN showtext boolean,
OUT temp_blks_written int8, OUT temp_blks_written int8,
OUT blk_read_time float8, OUT blk_read_time float8,
OUT blk_write_time float8, OUT blk_write_time float8,
OUT host bigint, OUT client_ip bigint,
OUT hist_calls text, OUT resp_calls text,
OUT hist_min_time text,
OUT hist_max_time text,
OUT hist_mean_time text,
OUT slow_query text,
OUT cpu_user_time float8, OUT cpu_user_time float8,
OUT cpu_sys_time float8 OUT cpu_sys_time float8,
OUT tables_names text
) )
RETURNS SETOF record RETURNS SETOF record
AS 'MODULE_PATHNAME', 'pg_stat_monitor_1_3' AS 'MODULE_PATHNAME', 'pg_stat_monitor'
LANGUAGE C STRICT VOLATILE PARALLEL SAFE; LANGUAGE C STRICT VOLATILE PARALLEL SAFE;
CREATE FUNCTION pg_stat_agg( CREATE FUNCTION pg_stat_agg(
OUT queryid bigint, OUT queryid text,
OUT id bigint, OUT id bigint,
OUT type bigint, OUT type bigint,
OUT total_calls int, OUT total_calls int)
OUT first_call_time timestamptz,
OUT last_call_time timestamptz)
RETURNS SETOF record RETURNS SETOF record
AS 'MODULE_PATHNAME', 'pg_stat_agg' AS 'MODULE_PATHNAME', 'pg_stat_agg'
LANGUAGE C STRICT VOLATILE PARALLEL SAFE; LANGUAGE C STRICT VOLATILE PARALLEL SAFE;
-- Register a view on the function for ease of use. -- Register a view on the function for ease of use.
CREATE VIEW pg_stat_monitor AS CREATE VIEW pg_stat_monitor AS SELECT
SELECT * FROM pg_stat_monitor(true); bucket,
bucket_start_time,
userid,
dbid,
queryid,
query,
calls,
total_time,
min_time,
max_time,
mean_time,
stddev_time,
rows int8,
shared_blks_hit,
shared_blks_read,
shared_blks_dirtied,
shared_blks_written,
local_blks_hit,
local_blks_read,
local_blks_dirtied,
local_blks_written,
temp_blks_read,
temp_blks_written,
blk_read_time,
blk_write_time,
client_ip as host,
'0.0.0.0'::inet + client_ip AS client_ip,
(string_to_array(resp_calls, ',')) resp_calls,
cpu_user_time,
cpu_sys_time,
(string_to_array(tables_names, ',')) tables_names
FROM pg_stat_monitor(true);
GRANT SELECT ON pg_stat_monitor TO PUBLIC; GRANT SELECT ON pg_stat_monitor TO PUBLIC;
CREATE VIEW pg_stat_agg_database AS CREATE VIEW pg_stat_agg_database AS
SELECT SELECT
ss.bucket,
agg.queryid, agg.queryid,
agg.id AS dbid, agg.id AS dbid,
ss.userid, ss.userid,
'0.0.0.0'::inet + ss.host AS host, client_ip,
agg.total_calls, agg.total_calls,
ss.min_time, ss.min_time,
ss.max_time, ss.max_time,
ss.mean_time, ss.mean_time,
(string_to_array(hist_calls, ',')) hist_calls, ss.resp_calls,
(string_to_array(hist_min_time, ',')) hist_min_time,
(string_to_array(hist_max_time, ',')) hist_max_time,
(string_to_array(hist_mean_time, ',')) hist_mean_time,
agg.first_call_time AS first_log_time,
agg.last_call_time AS last_log_time,
ss.cpu_user_time, ss.cpu_user_time,
ss.cpu_sys_time, ss.cpu_sys_time,
ss.query, ss.query,
ss.slow_query ss.tables_names
FROM pg_stat_agg() agg FROM pg_stat_agg() agg
INNER JOIN (SELECT DISTINCT queryid, dbid, userid, query, host, min_time, max_time, mean_time, hist_calls, hist_min_time, hist_max_time,hist_mean_time,slow_query,cpu_user_time,cpu_sys_time INNER JOIN (SELECT DISTINCT bucket, queryid, dbid, userid, query, client_ip, min_time, max_time, mean_time, resp_calls, tables_names, cpu_user_time,cpu_sys_time
FROM pg_stat_monitor) ss FROM pg_stat_monitor) ss
ON agg.queryid = ss.queryid AND agg.type = 0 AND id = dbid; ON agg.queryid = ss.queryid AND agg.type = 0 AND id = dbid;
CREATE VIEW pg_stat_agg_user AS CREATE VIEW pg_stat_agg_user AS
SELECT SELECT
ss.bucket,
agg.queryid, agg.queryid,
agg.id AS dbid, agg.id AS dbid,
ss.userid, ss.userid,
'0.0.0.0'::inet + ss.host AS host, client_ip,
agg.total_calls, agg.total_calls,
ss.min_time, ss.min_time,
ss.max_time, ss.max_time,
ss.mean_time, ss.mean_time,
(string_to_array(hist_calls, ',')) hist_calls, ss.resp_calls,
(string_to_array(hist_min_time, ',')) hist_min_time,
(string_to_array(hist_max_time, ',')) hist_max_time,
(string_to_array(hist_mean_time, ',')) hist_mean_time,
agg.first_call_time AS first_log_time,
agg.last_call_time AS last_log_time,
ss.cpu_user_time, ss.cpu_user_time,
ss.cpu_sys_time, ss.cpu_sys_time,
ss.query, ss.query,
ss.slow_query ss.tables_names
FROM pg_stat_agg() agg FROM pg_stat_agg() agg
INNER JOIN (SELECT DISTINCT queryid, userid, query, host, min_time, max_time, mean_time, hist_calls, hist_min_time, hist_max_time,hist_mean_time,slow_query,cpu_user_time,cpu_sys_time FROM pg_stat_monitor) ss INNER JOIN (SELECT DISTINCT bucket, queryid, userid, query, client_ip, min_time, max_time, mean_time, resp_calls, tables_names, cpu_user_time,cpu_sys_time FROM pg_stat_monitor) ss
ON agg.queryid = ss.queryid AND agg.type = 1 AND id = userid; ON agg.queryid = ss.queryid AND agg.type = 1 AND id = userid;
CREATE VIEW pg_stat_agg_host AS CREATE VIEW pg_stat_agg_ip AS
SELECT SELECT
ss.bucket,
agg.queryid, agg.queryid,
agg.id AS dbid, agg.id AS dbid,
ss.userid, ss.userid,
'0.0.0.0'::inet + ss.host AS host, ss.client_ip,
ss.host,
agg.total_calls, agg.total_calls,
ss.min_time, ss.min_time,
ss.max_time, ss.max_time,
ss.mean_time, ss.mean_time,
(string_to_array(hist_calls, ',')) hist_calls, ss.resp_calls,
(string_to_array(hist_min_time, ',')) hist_min_time,
(string_to_array(hist_max_time, ',')) hist_max_time,
(string_to_array(hist_mean_time, ',')) hist_mean_time,
agg.first_call_time AS first_log_time,
agg.last_call_time AS last_log_time,
ss.cpu_user_time, ss.cpu_user_time,
ss.cpu_sys_time, ss.cpu_sys_time,
ss.query, ss.query,
ss.slow_query ss.tables_names
FROM pg_stat_agg() agg FROM pg_stat_agg() agg
INNER JOIN (SELECT DISTINCT queryid, userid, query, host, min_time, max_time, mean_time, hist_calls, hist_min_time, hist_max_time,hist_mean_time,slow_query,cpu_user_time,cpu_sys_time FROM pg_stat_monitor) ss INNER JOIN (SELECT DISTINCT bucket, queryid, userid, query, client_ip, host, min_time, max_time, mean_time, resp_calls, tables_names, cpu_user_time,cpu_sys_time FROM pg_stat_monitor) ss
ON agg.queryid = ss.queryid AND agg.type = 2 AND id = host; ON agg.queryid = ss.queryid AND agg.type = 2 AND id = host;
GRANT SELECT ON pg_stat_agg_user TO PUBLIC;
GRANT SELECT ON pg_stat_agg_ip TO PUBLIC;
GRANT SELECT ON pg_stat_agg_database TO PUBLIC; GRANT SELECT ON pg_stat_agg_database TO PUBLIC;
-- Don't want this to be available to non-superusers. -- Don't want this to be available to non-superusers.

2248
pg_stat_monitor.c
View File

File diff suppressed because it is too large Load Diff