external
/
pg_stat_monitor
mirror of https://github.com/percona/pg_stat_monitor.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of https://github.com/percona/pg_stat_monitor

pull/44/head
Ibrar Ahmed 2020-08-17 15:04:35 +00:00
parent 6700c9686e 949d2048dc
commit b3df3064fb
6 changed files with 665 additions and 304 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

707
README.md
View File

@ -1,199 +1,554 @@
# pg_stat_monitor - Statistics collector for PostgreSQL.
### pg_stat_monitor - Statistics collector for [PostgreSQL][1].
The pg_stat_monitor is the statistics collection tool based on PostgreSQL's contrib module ``pg_stat_statements``. PostgreSQL’s ``pg_stat_statements`` provides the basic statistics, which is sometimes not enough. The major shortcoming in ``pg_stat_statements`` is that it accumulates all the queries and their statistics and does not provide aggregated statistics nor histogram information. In this case, a user needs to calculate the aggregate which is quite expensive.
The pg_stat_monitor is statistics collector tool based on PostgreSQL's contrib module "pg_stat_statements". PostgreSQL’s “pg_stat_statments” provides the basic statistics which is sometimes not enough. The major shortcoming in pg_stat_statment is that it accumulates all the queries and its statistics and does not provide aggregate statistics or histogram information. in that case, the user needs to calculate the aggregate which is quite expensive.
``pg_stat_monitor`` is developed on the basis of ``pg_stat_statements`` as its more advanced replacement. It provides all the features of ``pg_stat_statements`` plus its own feature set.
pg_stat_monitor is developed on the basis of pg_stat_statments as more advanced replacement for pg_stat_statment. It provides all the features of pg_stat_statment plus its own feature set.
``pg_stat_monitor`` collects and aggregates data on a bucket basis. The size of a bucket and the number of buckets should be configured using GUC (Grand Unified Configuration). The flow is the following:
#### Supported PostgreSQL Versions.
Pg_stat_monitor should work on the latest version of PostgreSQL but only tested with these versions of PostgreSQL.
* PostgreSQL Version 11
* PostgreSQL Version 12
* Percona Distribution for PostgreSQL
#### Installation
There are two ways to install pg_stat_monitor. The first is by downloading the pg_stat_monitor source code and compiling it. The second is to download the deb or rpm packages.
##### Download and compile
The latest release of pg_stat_monitor can be downloaded from this GitHub page:
https://github.com/Percona/pg_stat_monitor/releases
or it can be downloaded using the git:
git clone git://github.com/Percona/pg_stat_monitor.git
After downloading the code, set the path for the [PostgreSQL][1] binary:
###### Compile and Install extension
cd pg_stat_monitor
make USE_PGXS=1
make USE_PGXS=1 install
###### Enable and Create Extension
This extension needs to be loaded at the start time. Which requires adding the pg_stat_monitor extension shared_preload_libraries and restart the PostgreSQL Instance.
postgres=# alter system set shared_preload_libraries=pg_stat_monitor;
ALTER SYSTEM
sudo systemctl restart postgresql-11
Create the extension in the desired database.
postgres=# create extension pg_stat_monitor;
CREATE EXTENSION
* ``pg_stat_monitor`` collects the statistics and aggregates it in a bucket.
* When a bucket time elapses, ``pg_stat_monitor`` resets all the statistics and switches to the next bucket.
* After the last bucket elapses, ``pg_stat_monitor`` goes back to the first bucket. All the data on the first bucket will vanish; therefore, users must read the buckets before that to not lose the data.
#### Usage
There are four views, and complete statistics can be accessed using these views.
## Supported PostgreSQL Versions
* pg_stat_monitor
* pg_stat_agg_database
* pg_stat_agg_user
* pg_stat_agg_host
The ``pg_stat_monitor`` should work on the latest version of PostgreSQL but is only tested with these PostgreSQL versions:
##### pg_stat_monitor
This is the main view which stores per query-based statistics, similar to pg_stat_statment with some additional columns.
postgres=# \d pg_stat_monitor;
View "public.pg_stat_monitor"
Column | Type | Collation | Nullable | Default
---------------------+--------------------------+-----------+----------+---------
bucket | oid | | |
bucket_start_time | timestamp with time zone | | |
userid | oid | | |
dbid | oid | | |
queryid | text | | |
query | text | | |
calls | bigint | | |
total_time | double precision | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
stddev_time | double precision | | |
int8 | bigint | | |
shared_blks_hit | bigint | | |
shared_blks_read | bigint | | |
shared_blks_dirtied | bigint | | |
shared_blks_written | bigint | | |
local_blks_hit | bigint | | |
local_blks_read | bigint | | |
local_blks_dirtied | bigint | | |
local_blks_written | bigint | | |
temp_blks_read | bigint | | |
temp_blks_written | bigint | | |
blk_read_time | double precision | | |
blk_write_time | double precision | | |
host | bigint | | |
client_ip | inet | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
tables_names | text[] | | |
* PostgreSQL Version 11
* PostgreSQL Version 12
* Percona Distribution for PostgreSQL 11 and 12
These are new column added to have more detail about the query.
# Documentation
client_ip: Client IP or Hostname
hist_calls: Hourly based 24 hours calls of query histogram
hist_min_time: Hourly based 24 hours min time of query histogram
Hist_max_time: Hourly based 24 hours max time of query histogram
hist_mean_time: Hourly based 24 hours mean time of query histogram
slow_query: Slowest query with actual parameters.
cpu_user_time: CPU user time for that query.
cpu_sys_time: CPU System time for that query.
1. [Installation](#installation)
2. [Configuration](#configuration)
3. [Setup](#setup)
4. [User Guide](#user-guide)
5. [Contribution](#contribution)
6. [Limitation](#limitation)
7. [License](#license)
**Copyright notice**
Copyright (c) 2006 - 2020, Percona LLC.
postgres=# \d pg_stat_agg_database
View "public.pg_stat_agg_database"
Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+---------
bucket | oid | | |
queryid | text | | |
dbid | bigint | | |
userid | oid | | |
client_ip | inet | | |
total_calls | integer | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
tables_names | text[] | | |
## Installation
postgres=# \d pg_stat_agg_user
View "public.pg_stat_agg_user"
Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+---------
bucket | oid | | |
queryid | text | | |
dbid | bigint | | |
userid | oid | | |
client_ip | inet | | |
total_calls | integer | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
tables_names | text[] | | |
There are two ways to install ``pg_stat_monitor``:
- by downloading the pg_stat_monitor source code and compiling it.
- by downloading the ``deb`` or ``rpm`` packages.
postgres=# \d pg_stat_agg_ip
View "public.pg_stat_agg_ip"
Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+---------
bucket | oid | | |
queryid | text | | |
dbid | bigint | | |
userid | oid | | |
client_ip | inet | | |
host | bigint | | |
total_calls | integer | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
tables_names | text[] | | |
**Compile from the source code**
Examples
1 - In this query we are getting the exact value of f1 which is '05:06:07-07' (special setting is required for this).
The latest release of ``pg_stat_monitor`` can be downloaded from [this GitHub page](https://github.com/Percona/pg_stat_monitor/releases) or it can be downloaded using the git:
# select userid, queryid, query, max_time, total_calls from pg_stat_agg_user;
-[ RECORD 1 ]----------------------------------------------------------------------------------------
userid | 10
queryid | -203926152419851453
query | SELECT f1 FROM TIMETZ_TBL WHERE f1 < '05:06:07-07';
max_time | 1.237875
total_calls | 8
```
git clone git://github.com/Percona/pg_stat_monitor.git
```
2 - Collect all statistics based on the user.
After downloading the code, set the path for the PostgreSQL binary.
# select usename, query, max_time, total_calls from pg_stat_agg_user au, pg_user u where au.userid = u.usesysid;
usename | query | max_time | total_calls
---------+---------------------------------------------------------+----------+-------------
vagrant | select userid, query, total_calls from pg_stat_agg_user | 0.268842 | 7
foo | select userid, query, total_calls from pg_stat_agg_user | 0.208551 | 1
vagrant | select * from pg_stat_monitor_reset() | 0.0941 | 1
(3 rows)
Compile and Install the extension
```
cd pg_stat_monitor
make USE_PGXS=1
make USE_PGXS=1 install
```
**Installing from rpm/deb packages**
TBD
#### Limitation
There are some limitations and Todos.
## Configuration
Here is the complete list of configuration parameters.
```
postgres=# select name,value, minimum, maximum from pg_stat_monitor_settings;
name | value | minimum | maximum
-----------------------------------------------+--------+---------+------------
pg_stat_monitor.pgsm_max | 5000 | 5000 | 2147483647
pg_stat_monitor.pgsm_query_max_len | 1024 | 1024 | 2147483647
pg_stat_monitor.pgsm_enable | 1 | 0 | 0
pg_stat_monitor.pgsm_track_utility | 1 | 0 | 0
pg_stat_monitor.pgsm_normalized_query | 1 | 0 | 0
pg_stat_monitor.pgsm_max_buckets | 10 | 1 | 10
pg_stat_monitor.pgsm_bucket_time | 60 | 1 | 2147483647
pg_stat_monitor.pgsm_object_cache | 500000 | 5 | 10
pg_stat_monitor.pgsm_respose_time_lower_bound | 5 | 1 | 2147483647
pg_stat_monitor.pgsm_respose_time_step | 1 | 1 | 2147483647
pg_stat_monitor.pgsm_query_shared_buffer | 1 | 500000 | 2147483647
(11 rows)
```
Some configuration parameters require the server restart and should be set before the server startup. These must be set in the ``postgresql.conf`` file. Other parameters do not require server restart and can be set permanently either in the ``postgresql.conf`` or from the client (``psql``).
The table below shows set up options for each configuration parameter and whether the server restart is required to apply its value:
<table>
<tr>
<td rowspan="2" ><strong>Name</strong>
</td>
<td colspan="3" ><strong>Setup options</strong>
</td>
<td colspan="2" ><strong>Restart methods</strong>
</td>
</tr>
<tr>
<td>edit postgresql conf
</td>
<td>set
</td>
<td>alter system set
</td>
<td>server restart
</td>
<td>configuration reload (via pg_reload_conf)
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_max
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_query_max_len
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_enable
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_track_utility
</td>
<td>YES
</td>
<td>YES
</td>
<td>YES
</td>
<td>NO
</td>
<td>YES
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_normalized_query
</td>
<td>YES
</td>
<td>YES
</td>
<td>YES
</td>
<td>NO
</td>
<td>YES
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_max_buckets
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_bucket_time
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_object_cache
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_respose_time_lower_bound
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_respose_time_step
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_query_shared_buffer
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
</table>
### Parameters description:
- **pg_stat_monitor.pgsm_max**: This parameter defines the limit of shared memory for ``pg_stat_monitor``. This memory is used by buckets in a circular manner. The memory is divided between the buckets equally, at the start of the PostgreSQL.
- **pg_stat_monitor.pgsm_query_max_len**: Sets the maximum size of the query. This parameter can only be set at the start of PostgreSQL. For long queries, the query is truncated to that particular length. This is to avoid unnecessary usage of shared memory.
- **pg_stat_monitor.pgsm_enable**: This parameter enables or disables the monitoring. "Disable" means that ``pg_stat_monitor`` will not collect the statistics for the whole cluster.
- **pg_stat_monitor.pgsm_track_utility**: This parameter controls whether utility commands are tracked by the module. Utility commands are all those other than ``SELECT``, ``INSERT``, ``UPDATE`` and ``DELETE``.
- **pg_stat_monitor.pgsm_normalized_query**: By default, query shows the actual parameter instead of the placeholder. It is quite useful when users want to use that query and try to run that query to check the abnormalities. But in most cases users like the queries with a placeholder. This parameter is used to toggle between the two said options.
- **pg_stat_monitor.pgsm_max_buckets**: ``pg_stat_monitor`` accumulates the information in the form of buckets. All the aggregated information is bucket based. This parameter is used to set the number of buckets the system can have. For example, if this parameter is set to 2, then the system will create two buckets. First, the system will add all the information into the first bucket. After its lifetime (defined in the pg_stat_monitor.pgsm_bucket_time parameter) expires, it will switch to the second bucket, reset all the counters and repeat the process.
- **pg_stat_monitor.pgsm_bucket_time**: This parameter is used to set the lifetime of the bucket. System switches between buckets on the basis of ``pg_stat_monitor.pgsm_bucket_time``.
- **pg_stat_monitor.pgsm_object_cache**: This parameter is used to store information about the objects in the query. ``pg_stat_monitor`` saves the information used by the objects in the query. To limit that information, you can set the length of that memory.
- **pg_stat_monitor.pgsm_respose_time_lower_bound**: ``pg_stat_monitor`` also stores the execution time histogram. This parameter is used to set the lower bound of the histogram.
- **pg_stat_monitor.pgsm_respose_time_step:** This parameter is used to set the steps for the histogram.
## Setup
``pg_stat_monitor`` cannot be installed in your running PostgreSQL instance. It should be set in the ``postgresql.conf`` file.
```
# - Shared Library Preloading -
shared_preload_libraries = 'pg_stat_monitor' # (change requires restart)
#local_preload_libraries = ''
#session_preload_libraries = ''
```
Or you can do from `psql` terminal using the ``alter system`` command.
``pg_stat_monitor`` needs to be loaded at the start time. This requires adding the ``pg_stat_monitor`` extension for the ``shared_preload_libraries`` parameter and restart the PostgreSQL instance.
```
postgres=# alter system set shared_preload_libraries=pg_stat_monitor;
ALTER SYSTEM
sudo systemctl restart postgresql-11
```
Create the extension using the ``create extension`` command.
```
postgres=# create extension pg_stat_monitor;
CREATE EXTENSION
```
After doing that change, we need to restart the PostgreSQL server. PostgreSQL will start monitoring and collecting the statistics.
## User Guide
To view the statistics, there are multiple views available.
```
postgres=# \d pg_stat_monitor
View "public.pg_stat_monitor"
Column | Type | Default
---------------------+--------------------------+-----------
bucket | int |
bucket_start_time | timestamp with time zone |
userid | oid |
dbid | oid |
queryid | text |
query | text |
calls | bigint |
total_time | double precision |
min_time | double precision |
max_time | double precision |
mean_time | double precision |
stddev_time | double precision |
rows | bigint |
shared_blks_hit | bigint |
shared_blks_read | bigint |
shared_blks_dirtied | bigint |
shared_blks_written | bigint |
local_blks_hit | bigint |
local_blks_read | bigint |
local_blks_dirtied | bigint |
local_blks_written | bigint |
temp_blks_read | bigint |
temp_blks_written | bigint |
blk_read_time | double precision |
blk_write_time | double precision |
host | bigint |
client_ip | inet |
resp_calls | text[] |
cpu_user_time | double precision |
cpu_sys_time | double precision |
tables_names | text[] |
wait_event | text |
wait_event_type | text |
```
**`bucket`**: ``pg_stat_monitor`` accumulates the statistics per bucket. All the information and aggregate reset for each bucket. The `bucket` will be a number showing the number of buckets for which this record belongs.
**`bucket_start_time`**: `bucket_start_time` shows the start time of the bucket.
```
postgres=# select bucket, bucket_start_time, query from pg_stat_monitor;
bucket | bucket_start_time | query
--------+-------------------------------+--------------------------------------------------------
2 | 2020-05-23 13:24:44.652415+00 | select * from pg_stat_monitor_reset()
3 | 2020-05-23 13:45:01.55658+00 | select bucket, bucket_start_time, query from pg_stat_monitor
2 | 2020-05-23 13:24:44.652415+00 | SELECT * FROM foo
(3 rows)
```
**`userid`**: An ID of the user whom that query belongs. ``pg_stat_monitor`` is used to collect queries from all the users; therefore, `userid` is used to segregate the queries based on different users.
**`dbid`**: The database ID of the query. ``pg_stat_monitor`` accumulates queries from all the databases; therefore, this column is used to identify the database.
**`queryid`**: ``pg_stat_monitor`` generates a unique ID for each query (queryid).
**`query`**: The `query` column contains the actual text of the query. This parameter depends on the **`pg_stat_monitor.pgsm_normalized_query`** configuration parameters, in which format to show the query.
**`calls`**: Number of calls of that particular query.
```
postgres=# select userid, dbid, queryid, query, calls from pg_stat_monitor;
userid | dbid | queryid | query | calls
--------+-------+------------------+--------------------------------------------------------------+-------
10 | 12696 | 56F12CE7CD01CF2C | select * from pg_stat_monitor_reset() | 1
10 | 12696 | 748D9EC1F4CECB36 | select userid, dbid, queryid, query from pg_stat_monitor | 1
10 | 12696 | 85900141D214EC52 | select bucket, bucket_start_time, query from pg_stat_monitor | 2
10 | 12696 | F1AC132034D5B366 | SELECT * FROM foo | 1
```
**`total_time`**, **`min_time`**, **`max_time`**, **`mean_time`**: The total / minimum / maximum and mean time spent for the same query.
```
postgres=# select userid, total_time, min_time, max_time, mean_time, query from pg_stat_monitor;
userid | total_time | min_time | max_time | mean_time | query
--------+--------------------+--------------------+--------------------+--------------------+------------------------------------------------------------------
10 | 0.14| 0.14 | 0.14 | 0.14 | select * from pg_stat_monitor_reset()
10 | 0.19 | 0.19 | 0.19 | 0.19 | select userid, dbid, queryid, query from pg_stat_monitor
10 | 0.30 | 0.13 | 0.16 | 0.15 | select bucket, bucket_start_time, query from pg_stat_monitor
10 | 0.29 | 0.29 | 0.29 | 0.29 | select userid, dbid, queryid, query, calls from pg_stat_monitor
10 | 11277.79 | 11277.79 | 11277.79 | 11277.79| SELECT * FROM foo
```
**`client_ip`**: The IP address of the client that originated the query.
```
postgres=# select client_ip, query from pg_stat_monitor;
client_ip | query
-----------+---------------------------------------------------------------------------------------
127.0.0.1 | select * from pg_stat_monitor_reset()
127.0.0.1 | select userid, dbid, queryid, query from pg_stat_monitor
127.0.0.1 | select bucket, bucket_start_time, query from pg_stat_monitor
127.0.0.1 | select userid, total_time, min_time, max_time, mean_time, query from pg_stat_monitor
127.0.0.1 | select userid, dbid, queryid, query, calls from pg_stat_monitor
127.0.0.1 | SELECT * FROM foo
(6 rows)
```
**`resp_calls`**: Call histogram
```
postgres=# select resp_calls, query from pg_stat_monitor;
resp_calls | query
--------------------------------------------------+---------------------------------------------- {1," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"} | select client_ip, query from pg_stat_monitor
{3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | select * from pg_stat_monitor_reset()
{3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | SELECT * FROM foo
```
There are 10 timebase buckets of the time **`pg_stat_monitor.pgsm_respose_time_step`** in the field ``resp_calls``. The value in the field shows how many queries run in that period of time.
**`tables_names`**: The list of tables involved in the query
```
postgres=# select tables_names, query from pg_stat_monitor;
tables_names | query
--------------------------+---------------------------------------------------------------------------------------
{public.pg_stat_monitor} | select client_ip, query from pg_stat_monitor
| select * from pg_stat_monitor_reset()
{public.pg_stat_monitor} | select userid, dbid, queryid, query from pg_stat_monitor
{public.pg_stat_monitor} | select bucket, bucket_start_time, query from pg_stat_monitor
{public.pg_stat_monitor} | select userid, total_time, min_time, max_time, mean_time, query from pg_stat_monitor
{public.pg_stat_monitor} | select userid, dbid, queryid, query, calls from pg_stat_monitor
{public.foo} | SELECT * FROM foo
{public.pg_stat_monitor} | select resp_calls, query from pg_stat_monitor
(8 rows)
```
**`wait_event`**: Current state of the query
```
postgres=# select wait_event_type, query from pg_stat_monitor;
wait_event | query
-----------------+-------------------------------------------------------------------------------
| select client_ip, query from pg_stat_monitor
ClientRead | select wait_event_type, query from pg_stat_monitor
| select * from pg_stat_monitor_reset()
```
**`pg_stat_agg_database`**: Shows the aggregated value per database.
```
postgres=# \d pg_stat_agg_database;
View "public.pg_stat_agg_database"
Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+---------
bucket | oid | | |
queryid | text | | |
dbid | bigint | | |
userid | oid | | |
client_ip | inet | | |
total_calls | integer | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
tables_names | text[] | | |
```
**`pg_stat_agg_user`**: Shows the aggregated value per user.
```
postgres=# \d pg_stat_agg_user;
View "public.pg_stat_agg_user"
Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+---------
bucket | oid | | |
queryid | text | | |
dbid | bigint | | |
userid | oid | | |
client_ip | inet | | |
total_calls | integer | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
tables_names | text[] | | |
```
**`pg_stat_agg_ip`**: Shows the aggregated value per IP / host.
```
postgres=# \d pg_stat_agg_ip;
View "public.pg_stat_agg_ip"
Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+---------
bucket | oid | | |
queryid | text | | |
dbid | bigint | | |
userid | oid | | |
client_ip | inet | | |
host | bigint | | |
total_calls | integer | | |
min_time | double precision | | |
max_time | double precision | | |
mean_time | double precision | | |
resp_calls | text[] | | |
cpu_user_time | double precision | | |
cpu_sys_time | double precision | | |
query | text | | |
tables_names | text[] | | |
```
## License
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
#### Licence
Copyright (c) 2006 - 2019, Percona LLC.
See [`LICENSE`][2] for full details.
[1]: https://www.postgresql.org/
[2]: https://github.com/Percona/pg_stat_monitor/blob/master/LICENSE
[3]: https://github.com/Percona/pg_stat_monitor/issues/new
[4]: CONTRIBUTING.md

4
percona-packaging/debian/control
View File

@ -6,10 +6,10 @@ Build-Depends:
debhelper (>= 9),
percona-postgresql-server-dev-all (>= 153~),
Package: percona-pg-stat-monitor
Package: percona-pg-stat-monitor@@PG_REL@@
Architecture: any
Depends:
percona-postgresql-11,
percona-postgresql-@@PG_REL@@,
${misc:Depends},
${shlibs:Depends},
Description: The pg_stat_monitor is statistics collector tool

4
percona-packaging/debian/control.in
View File

@ -6,10 +6,10 @@ Build-Depends:
debhelper (>= 9),
percona-postgresql-server-dev-all (>= 153~),
Package: percona-pg-stat-monitor
Package: percona-pg-stat-monitor@@PG_REL@@
Architecture: any
Depends:
percona-postgresql-11,
percona-postgresql-@@PG_REL@@,
${misc:Depends},
${shlibs:Depends},
Description: The pg_stat_monitor is statistics collector tool

2
percona-packaging/debian/rules
View File

@ -18,7 +18,7 @@ override_dh_auto_test:
# nothing to do here
override_dh_auto_install:
+pg_buildext install build-%v percona-pg-stat-monitor
+pg_buildext install build-%v percona-pg-stat-monitor@@PG_REL@@
override_dh_installdocs:
dh_installdocs --all README.*

16
percona-packaging/rpm/pg-stat-monitor.spec
View File

@ -1,17 +1,19 @@
%global sname percona-pg-stat-monitor
%global pginstdir /usr/pgsql-11/
%global pgrel @@PG_REL@@
%global rpm_release @@RPM_RELEASE@@
%global pginstdir /usr/pgsql-@@PG_REL@@/
%global pgstatmonmajver 1
%global pgstatmonmidver 0
%global pgstatmonminver 0
Summary: Statistics collector for PostgreSQL
Name: %{sname}
Name: %{sname}%{pgrel}
Version: %{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}
Release: 1%{?dist}
Release: %{rpm_release}%{?dist}
License: Apache 2.0
Source0: %{sname}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}.tar.gz
Source0: %{sname}%{pgrel}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}.tar.gz
URL: https://github.com/Percona-Lab/pg_stat_monitor
BuildRequires: percona-postgresql11-devel
BuildRequires: percona-postgresql%{pgrel}-devel
Requires: postgresql-server
@ -25,11 +27,11 @@ It provides all the features of pg_stat_statment plus its own feature set.
%prep
%setup -q -n %{sname}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}
%setup -q -n %{sname}%{pgrel}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}
%build
sed -i 's:PG_CONFIG = pg_config:PG_CONFIG = /usr/pgsql-11/bin/pg_config:' Makefile
sed -i 's:PG_CONFIG = pg_config:PG_CONFIG = /usr/pgsql-%{pgrel}/bin/pg_config:' Makefile
%{__make} USE_PGXS=1 %{?_smp_mflags}

236
percona-packaging/scripts/pg_stat_monitor_builder.sh
View File

@ -10,15 +10,19 @@ Usage: $0 [OPTIONS]
The following options may be given :
--builddir=DIR Absolute path to the dir where all actions will be performed
--get_sources Source will be downloaded from github
--build_src_rpm If it is set - src rpm will be built
--build_src_deb If it is set - source deb package will be built
--build_rpm If it is set - rpm will be built
--build_deb If it is set - deb will be built
--install_deps Install build dependencies(root privilages are required)
--build_src_rpm If it is 1 src rpm will be built
--build_source_deb If it is 1 source deb package will be built
--build_rpm If it is 1 rpm will be built
--build_deb If it is 1 deb will be built
--build_tarball If it is 1 tarball will be built
--install_deps Install build dependencies(root previlages are required)
--branch Branch for build
--repo Repo for build
--rpm_release RPM version( default = 1)
--deb_release DEB version( default = 1)
--pg_release PPG version build on( default = 11)
--help) usage ;;
Example $0 --builddir=/tmp/BUILD --get_sources=1 --build_src_rpm=1 --build_rpm=1
Example $0 --builddir=/tmp/test --get_sources=1 --build_src_rpm=1 --build_rpm=1
EOF
exit 1
}
@ -27,7 +31,7 @@ append_arg_to_args () {
args="$args "$(shell_quote_string "$1")
}
parse_arguments() {
parse_arguments() {
pick_args=
if test "$1" = PICK-ARGS-FROM-ARGV
then
@ -38,15 +42,20 @@ parse_arguments() {
for arg do
val=$(echo "$arg" | sed -e 's;^--[^=]*=;;')
case "$arg" in
# these get passed explicitly to mysqld
--builddir=*) WORKDIR="$val" ;;
--build_src_rpm=*) SRPM="$val" ;;
--build_src_deb=*) SDEB="$val" ;;
--build_source_deb=*) SDEB="$val" ;;
--build_rpm=*) RPM="$val" ;;
--build_deb=*) DEB="$val" ;;
--get_sources=*) SOURCE="$val" ;;
--build_tarball=*) TARBALL="$val" ;;
--install_deps=*) INSTALL="$val" ;;
--branch=*) BRANCH="$val" ;;
--repo=*) REPO="$val" ;;
--install_deps=*) INSTALL="$val" ;;
--rpm_release=*) RPM_RELEASE="$val" ;;
--deb_release=*) DEB_RELEASE="$val" ;;
--pg_release=*) PG_RELEASE="$val" ;;
--help) usage ;;
*)
if test -n "$pick_args"
@ -74,34 +83,13 @@ check_workdir(){
}
add_percona_yum_repo(){
if [ ! -f /etc/yum.repos.d/percona-dev.repo ]
then
wget http://jenkins.percona.com/yum-repo/percona-dev.repo
mv -f percona-dev.repo /etc/yum.repos.d/
if [ ! -f /etc/yum.repos.d/percona-dev.repo ]; then
curl -o /etc/yum.repos.d/percona-dev.repo https://jenkins.percona.com/yum-repo/percona-dev.repo
sed -i 's:$basearch:x86_64:g' /etc/yum.repos.d/percona-dev.repo
fi
yum -y install https://repo.percona.com/yum/percona-release-latest.noarch.rpm
percona-release disable all
percona-release enable ppg-11 release
return
}
add_percona_apt_repo(){
if [ ! -f /etc/apt/sources.list.d/percona-dev.list ]; then
cat >/etc/apt/sources.list.d/percona-dev.list <<EOL
deb http://jenkins.percona.com/apt-repo/ @@DIST@@ main
deb-src http://jenkins.percona.com/apt-repo/ @@DIST@@ main
EOL
sed -i "s:@@DIST@@:$OS_NAME:g" /etc/apt/sources.list.d/percona-dev.list
fi
wget -qO - http://jenkins.percona.com/apt-repo/8507EFA5.pub | apt-key add -
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
dpkg -i percona-release_latest.generic_all.deb
percona-release disable all
rm -f percona-release_latest.generic_all.deb
percona-release enable ppg-11 release
return
}
get_sources(){
cd "${WORKDIR}"
if [ "${SOURCE}" = 0 ]
@ -109,14 +97,19 @@ get_sources(){
echo "Sources will not be downloaded"
return 0
fi
PRODUCT=percona-pg-stat-monitor
echo "PRODUCT=${PRODUCT}" > pg-stat-monitor.properties
PRODUCT=percona-pg-stat-monitor${PG_RELEASE}
PRODUCT_FULL=${PRODUCT}-${VERSION}
echo "PRODUCT=${PRODUCT}" > pg-stat-monitor.properties
echo "PRODUCT_FULL=${PRODUCT_FULL}" >> pg-stat-monitor.properties
echo "VERSION=${PSM_VER}" >> pg-stat-monitor.properties
echo "VERSION=${VERSION}" >> pg-stat-monitor.properties
echo "BRANCH_NAME=$(echo ${BRANCH} | awk -F '/' '{print $(NF)}')" >> pg-stat-monitor.properties
echo "BUILD_NUMBER=${BUILD_NUMBER}" >> pg-stat-monitor.properties
echo "BUILD_ID=${BUILD_ID}" >> pg-stat-monitor.properties
echo "BRANCH_NAME=$(echo ${BRANCH} | awk -F '/' '{print $(NF)}')" >> pg-stat-monitor.properties
echo "PG_RELEASE=${PG_RELEASE}" >> pg-stat-monitor.properties
echo "RPM_RELEASE=${RPM_RELEASE}" >> pg-stat-monitor.properties
echo "DEB_RELEASE=${DEB_RELEASE}" >> pg-stat-monitor.properties
git clone "$REPO" ${PRODUCT_FULL}
retval=$?
if [ $retval != 0 ]
@ -136,13 +129,20 @@ get_sources(){
rm -rf rpm debian
cp -r percona-packaging/rpm ./
cp -r percona-packaging/debian ./
EDITFILES="debian/control debian/control.in debian/rules rpm/pg-stat-monitor.spec"
for file in $EDITFILES; do
sed -i "s:@@PG_REL@@:${PG_RELEASE}:g" "$file"
done
sed -i "s:@@RPM_RELEASE@@:${RPM_RELEASE}:g" rpm/pg-stat-monitor.spec
cd ${WORKDIR}
#
source pg-stat-monitor.properties
#
tar --owner=0 --group=0 --exclude=.* -czf ${PRODUCT_FULL}.tar.gz ${PRODUCT_FULL}
echo "UPLOAD=UPLOAD/experimental/BUILDS/${PRODUCT}/${PRODUCT_FULL}/${PSM_BRANCH}/${REVISION}/${BUILD_ID}" >> pg-stat-monitor.properties
echo "UPLOAD=UPLOAD/experimental/BUILDS/${PRODUCT}/${PRODUCT_FULL}/${BRANCH}/${REVISION}/${BUILD_ID}" >> pg-stat-monitor.properties
mkdir $WORKDIR/source_tarball
mkdir $CURDIR/source_tarball
cp ${PRODUCT_FULL}.tar.gz $WORKDIR/source_tarball
@ -154,14 +154,16 @@ get_sources(){
get_system(){
if [ -f /etc/redhat-release ]; then
RHEL=$(rpm --eval %rhel)
ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g')
OS_NAME="el$RHEL"
OS="rpm"
GLIBC_VER_TMP="$(rpm glibc -qa --qf %{VERSION})"
export RHEL=$(rpm --eval %rhel)
export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g')
export OS_NAME="el$RHEL"
export OS="rpm"
else
ARCH=$(uname -m)
OS_NAME="$(lsb_release -sc)"
OS="deb"
GLIBC_VER_TMP="$(dpkg-query -W -f='${Version}' libc6 | awk -F'-' '{print $1}')"
export ARCH=$(uname -m)
export OS_NAME="$(lsb_release -sc)"
export OS="deb"
fi
return
}
@ -178,61 +180,68 @@ install_deps() {
exit 1
fi
CURPLACE=$(pwd)
if [ "x$OS" = "xrpm" ]; then
yum -y install wget
add_percona_yum_repo
wget http://jenkins.percona.com/yum-repo/percona-dev.repo
mv -f percona-dev.repo /etc/yum.repos.d/
yum clean all
RHEL=$(rpm --eval %rhel)
if [ ${RHEL} = 8 ]; then
dnf -y module disable postgresql
dnf config-manager --set-enabled codeready-builder-for-rhel-8-x86_64-rpms
dnf clean all
rm -r /var/cache/dnf
dnf -y upgrade
yum -y install perl
else
until yum -y install centos-release-scl; do
echo "waiting"
sleep 1
done
yum -y install epel-release
INSTALL_LIST="llvm-toolset-7-clang llvm5.0-devel"
yum -y install ${INSTALL_LIST}
source /opt/rh/devtoolset-7/enable
source /opt/rh/llvm-toolset-7/enable
INSTALL_LIST="clang-devel git clang llvm-devel rpmdevtools vim wget"
yum -y install ${INSTALL_LIST}
yum -y install binutils gcc gcc-c++
fi
INSTALL_LIST="percona-postgresql-common clang-devel llvm-devel percona-postgresql11-devel git rpm-build rpmdevtools wget gcc make autoconf"
yum -y install ${INSTALL_LIST}
if [ "$OS" == "rpm" ]
then
yum install -y https://repo.percona.com/yum/percona-release-latest.noarch.rpm
add_percona_yum_repo
if [[ ${PG_RELEASE} == "11" ]]; then
percona-release enable ppg-11 release
elif [[ $PG_RELEASE == "12" ]]; then
percona-release enable ppg-12 release
fi
yum -y install git wget
PKGLIST+=" clang-devel git clang llvm-devel rpmdevtools vim wget"
PKGLIST+=" perl binutils gcc gcc-c++"
PKGLIST+=" percona-postgresql-common clang-devel llvm-devel percona-postgresql${PG_RELEASE}-devel git rpm-build rpmdevtools wget gcc make autoconf"
if [[ "${RHEL}" -eq 8 ]]; then
dnf -y module disable postgresql
elif [[ "${RHEL}" -eq 7 ]]; then
PKGLIST+=" llvm-toolset-7-clang llvm-toolset-7-llvm-devel llvm5.0-devel"
until yum -y install epel-release centos-release-scl; do
yum clean all
sleep 1
echo "waiting"
done
until yum -y makecache; do
yum clean all
sleep 1
echo "waiting"
done
fi
until yum -y install ${PKGLIST}; do
echo "waiting"
sleep 1
done
else
export DEBIAN=$(lsb_release -sc)
export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g')
apt-get -y install gnupg2
add_percona_apt_repo
apt-get update || true
LLVM_EXISTS=$(grep -c "apt.llvm.org" /etc/apt/sources.list)
if [ ${LLVM_EXISTS} = 0 ]; then
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
echo "deb http://apt.llvm.org/${DEBIAN}/ llvm-toolchain-${DEBIAN}-7 main" >> /etc/apt/sources.list
echo "deb-src http://apt.llvm.org/${DEBIAN}/ llvm-toolchain-${DEBIAN}-7 main" >> /etc/apt/sources.list
apt-get --allow-unauthenticated update
fi
apt-get update || true
INSTALL_LIST="build-essential percona-postgresql-11 debconf debhelper clang-7 devscripts dh-exec dh-systemd git wget libkrb5-dev libssl-dev percona-postgresql-common percona-postgresql-server-dev-all"
DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated install ${INSTALL_LIST}
INSTALL_LIST="build-essential debconf debhelper devscripts dh-exec dh-systemd git wget libxml-checker-perl libxml-libxml-perl libio-socket-ssl-perl libperl-dev libssl-dev libxml2-dev txt2man zlib1g-dev libpq-dev"
until DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated install ${INSTALL_LIST}; do
sleep 1
echo "waiting"
done
apt-get -y --allow-unauthenticated install libmysqlclient-dev || true
apt-get -y --allow-unauthenticated install default-libmysqlclient-dev || true
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y install lsb-release gnupg git wget
wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb && dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
if [[ "${PG_RELEASE}" == "11" ]]; then
percona-release enable ppg-11 release
elif [[ "${PG_RELEASE}" == "12" ]]; then
percona-release enable ppg-12 release
fi
apt-get update
if [[ "${OS_NAME}" != "focal" ]]; then
LLVM_EXISTS=$(grep -c "apt.llvm.org" /etc/apt/sources.list)
if [ "${LLVM_EXISTS}" == 0 ]; then
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
echo "deb http://apt.llvm.org/${OS_NAME}/ llvm-toolchain-${OS_NAME}-7 main" >> /etc/apt/sources.list
echo "deb-src http://apt.llvm.org/${OS_NAME}/ llvm-toolchain-${OS_NAME}-7 main" >> /etc/apt/sources.list
apt-get update
fi
fi
PKGLIST+=" percona-postgresql-${PG_RELEASE} debconf debhelper clang-7 devscripts dh-exec dh-systemd git wget libkrb5-dev libssl-dev percona-postgresql-common percona-postgresql-server-dev-all"
PKGLIST+=" build-essential debconf debhelper devscripts dh-exec dh-systemd git wget libxml-checker-perl libxml-libxml-perl libio-socket-ssl-perl libperl-dev libssl-dev libxml2-dev txt2man zlib1g-dev libpq-dev"
until DEBIAN_FRONTEND=noninteractive apt-get -y install ${PKGLIST}; do
sleep 1
echo "waiting"
done
fi
return;
}
@ -350,8 +359,8 @@ build_rpm(){
source /opt/rh/devtoolset-7/enable
source /opt/rh/llvm-toolset-7/enable
fi
export LIBPQ_DIR=/usr/pgsql-11/
export LIBRARY_PATH=/usr/pgsql-11/lib/:/usr/pgsql-11/include/
export LIBPQ_DIR=/usr/pgsql-${PG_RELEASE}/
export LIBRARY_PATH=/usr/pgsql-${PG_RELEASE}/lib/:/usr/pgsql-${PG_RELEASE}/include/
rpmbuild --define "_topdir ${WORKDIR}/rpmbuild" --define "dist .$OS_NAME" --define "version ${VERSION}" --rebuild rpmbuild/SRPMS/$SRC_RPM
return_code=$?
@ -389,7 +398,7 @@ build_source_deb(){
mv ${TARFILE} percona-pg-stat-monitor_${VERSION}.orig.tar.gz
cd ${BUILDDIR}
dch -D unstable --force-distribution -v "${VERSION}-${RELEASE}" "Update to new percona-pg-stat-monitor version ${VERSION}"
dch -D unstable --force-distribution -v "${VERSION}-${DEB_RELEASE}" "Update to new percona-pg-stat-monitor${PG_RELEASE} version ${VERSION}"
dpkg-buildpackage -S
cd ../
mkdir -p $WORKDIR/source_deb
@ -422,19 +431,17 @@ build_deb(){
cd $WORKDIR
rm -fv *.deb
#
export DEBIAN=$(lsb_release -sc)
export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g')
#
echo "DEBIAN=${DEBIAN}" >> pg-stat-monitor.properties
echo "DEBIAN=${OS_NAME}" >> pg-stat-monitor.properties
echo "ARCH=${ARCH}" >> pg-stat-monitor.properties
#
DSC=$(basename $(find . -name '*.dsc' | sort | tail -n1))
#
dpkg-source -x ${DSC}
#
cd percona-pg-stat-monitor-${VERSION}
dch -m -D "${DEBIAN}" --force-distribution -v "1:${VERSION}-${RELEASE}.${DEBIAN}" 'Update distribution'
dch -m -D "${OS_NAME}" --force-distribution -v "1:${VERSION}-${DEB_RELEASE}.${OS_NAME}" 'Update distribution'
unset $(locale|cut -d= -f1)
dpkg-buildpackage -rfakeroot -us -uc -b
mkdir -p $CURDIR/deb
@ -442,10 +449,9 @@ build_deb(){
cp $WORKDIR/*.*deb $WORKDIR/deb
cp $WORKDIR/*.*deb $CURDIR/deb
}
#main
CURDIR=$(pwd)
VERSION_FILE=$CURDIR/pg-stat-monitor.properties
VERSION_FILE=$CURDIR/percona-pg-stat-monitor.properties
args=
WORKDIR=
SRPM=0
@ -453,21 +459,19 @@ SDEB=0
RPM=0
DEB=0
SOURCE=0
TARBALL=0
OS_NAME=
ARCH=
OS=
REVISION=0
BRANCH="master"
INSTALL=0
RPM_RELEASE=1
DEB_RELEASE=1
REVISION=0
BRANCH="master"
REPO="https://github.com/Percona/pg_stat_monitor.git"
PRODUCT=percona-pg-stat-monitor
DEBUG=0
VERSION="1.0.0"
PG_RELEASE=11
parse_arguments PICK-ARGS-FROM-ARGV "$@"
VERSION='1.0.0'
RELEASE='1'
PRODUCT_FULL=${PRODUCT}-${VERSION}-${RELEASE}
check_workdir
get_system