external
/
pg_stat_monitor
mirror of https://github.com/percona/pg_stat_monitor.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of https://github.com/percona/pg_stat_monitor

pull/44/head
Ibrar Ahmed 2020-08-17 15:04:35 +00:00
parent 6700c9686e 949d2048dc
commit b3df3064fb
6 changed files with 665 additions and 304 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

553
README.md
View File

@ -1,112 +1,480 @@
# pg_stat_monitor - Statistics collector for PostgreSQL.
### pg_stat_monitor - Statistics collector for [PostgreSQL][1]. The pg_stat_monitor is the statistics collection tool based on PostgreSQL's contrib module ``pg_stat_statements``. PostgreSQL’s ``pg_stat_statements`` provides the basic statistics, which is sometimes not enough. The major shortcoming in ``pg_stat_statements`` is that it accumulates all the queries and their statistics and does not provide aggregated statistics nor histogram information. In this case, a user needs to calculate the aggregate which is quite expensive.
The pg_stat_monitor is statistics collector tool based on PostgreSQL's contrib module "pg_stat_statements". PostgreSQL’s “pg_stat_statments” provides the basic statistics which is sometimes not enough. The major shortcoming in pg_stat_statment is that it accumulates all the queries and its statistics and does not provide aggregate statistics or histogram information. in that case, the user needs to calculate the aggregate which is quite expensive. ``pg_stat_monitor`` is developed on the basis of ``pg_stat_statements`` as its more advanced replacement. It provides all the features of ``pg_stat_statements`` plus its own feature set.
pg_stat_monitor is developed on the basis of pg_stat_statments as more advanced replacement for pg_stat_statment. It provides all the features of pg_stat_statment plus its own feature set. ``pg_stat_monitor`` collects and aggregates data on a bucket basis. The size of a bucket and the number of buckets should be configured using GUC (Grand Unified Configuration). The flow is the following:
#### Supported PostgreSQL Versions. * ``pg_stat_monitor`` collects the statistics and aggregates it in a bucket.
Pg_stat_monitor should work on the latest version of PostgreSQL but only tested with these versions of PostgreSQL. * When a bucket time elapses, ``pg_stat_monitor`` resets all the statistics and switches to the next bucket.
* After the last bucket elapses, ``pg_stat_monitor`` goes back to the first bucket. All the data on the first bucket will vanish; therefore, users must read the buckets before that to not lose the data.
## Supported PostgreSQL Versions
The ``pg_stat_monitor`` should work on the latest version of PostgreSQL but is only tested with these PostgreSQL versions:
* PostgreSQL Version 11 * PostgreSQL Version 11
* PostgreSQL Version 12 * PostgreSQL Version 12
* Percona Distribution for PostgreSQL * Percona Distribution for PostgreSQL 11 and 12
#### Installation
There are two ways to install pg_stat_monitor. The first is by downloading the pg_stat_monitor source code and compiling it. The second is to download the deb or rpm packages.
##### Download and compile # Documentation
The latest release of pg_stat_monitor can be downloaded from this GitHub page:
https://github.com/Percona/pg_stat_monitor/releases 1. [Installation](#installation)
2. [Configuration](#configuration)
3. [Setup](#setup)
4. [User Guide](#user-guide)
5. [Contribution](#contribution)
6. [Limitation](#limitation)
7. [License](#license)
or it can be downloaded using the git: **Copyright notice**
Copyright (c) 2006 - 2020, Percona LLC.
## Installation
There are two ways to install ``pg_stat_monitor``:
- by downloading the pg_stat_monitor source code and compiling it.
- by downloading the ``deb`` or ``rpm`` packages.
**Compile from the source code**
The latest release of ``pg_stat_monitor`` can be downloaded from [this GitHub page](https://github.com/Percona/pg_stat_monitor/releases) or it can be downloaded using the git:
```
git clone git://github.com/Percona/pg_stat_monitor.git git clone git://github.com/Percona/pg_stat_monitor.git
```
After downloading the code, set the path for the [PostgreSQL][1] binary: After downloading the code, set the path for the PostgreSQL binary.
###### Compile and Install extension Compile and Install the extension
```
cd pg_stat_monitor cd pg_stat_monitor
make USE_PGXS=1 make USE_PGXS=1
make USE_PGXS=1 install make USE_PGXS=1 install
```
###### Enable and Create Extension **Installing from rpm/deb packages**
This extension needs to be loaded at the start time. Which requires adding the pg_stat_monitor extension shared_preload_libraries and restart the PostgreSQL Instance. TBD
## Configuration
Here is the complete list of configuration parameters.
```
postgres=# select name,value, minimum, maximum from pg_stat_monitor_settings;
name | value | minimum | maximum
-----------------------------------------------+--------+---------+------------
pg_stat_monitor.pgsm_max | 5000 | 5000 | 2147483647
pg_stat_monitor.pgsm_query_max_len | 1024 | 1024 | 2147483647
pg_stat_monitor.pgsm_enable | 1 | 0 | 0
pg_stat_monitor.pgsm_track_utility | 1 | 0 | 0
pg_stat_monitor.pgsm_normalized_query | 1 | 0 | 0
pg_stat_monitor.pgsm_max_buckets | 10 | 1 | 10
pg_stat_monitor.pgsm_bucket_time | 60 | 1 | 2147483647
pg_stat_monitor.pgsm_object_cache | 500000 | 5 | 10
pg_stat_monitor.pgsm_respose_time_lower_bound | 5 | 1 | 2147483647
pg_stat_monitor.pgsm_respose_time_step | 1 | 1 | 2147483647
pg_stat_monitor.pgsm_query_shared_buffer | 1 | 500000 | 2147483647
(11 rows)
```
Some configuration parameters require the server restart and should be set before the server startup. These must be set in the ``postgresql.conf`` file. Other parameters do not require server restart and can be set permanently either in the ``postgresql.conf`` or from the client (``psql``).
The table below shows set up options for each configuration parameter and whether the server restart is required to apply its value:
<table>
<tr>
<td rowspan="2" ><strong>Name</strong>
</td>
<td colspan="3" ><strong>Setup options</strong>
</td>
<td colspan="2" ><strong>Restart methods</strong>
</td>
</tr>
<tr>
<td>edit postgresql conf
</td>
<td>set
</td>
<td>alter system set
</td>
<td>server restart
</td>
<td>configuration reload (via pg_reload_conf)
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_max
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_query_max_len
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_enable
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_track_utility
</td>
<td>YES
</td>
<td>YES
</td>
<td>YES
</td>
<td>NO
</td>
<td>YES
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_normalized_query
</td>
<td>YES
</td>
<td>YES
</td>
<td>YES
</td>
<td>NO
</td>
<td>YES
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_max_buckets
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_bucket_time
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_object_cache
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_respose_time_lower_bound
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_respose_time_step
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
<tr>
<td>pg_stat_monitor.pgsm_query_shared_buffer
</td>
<td>YES
</td>
<td>NO
</td>
<td>NO
</td>
<td>YES
</td>
<td>NO
</td>
</tr>
</table>
### Parameters description:
- **pg_stat_monitor.pgsm_max**: This parameter defines the limit of shared memory for ``pg_stat_monitor``. This memory is used by buckets in a circular manner. The memory is divided between the buckets equally, at the start of the PostgreSQL.
- **pg_stat_monitor.pgsm_query_max_len**: Sets the maximum size of the query. This parameter can only be set at the start of PostgreSQL. For long queries, the query is truncated to that particular length. This is to avoid unnecessary usage of shared memory.
- **pg_stat_monitor.pgsm_enable**: This parameter enables or disables the monitoring. "Disable" means that ``pg_stat_monitor`` will not collect the statistics for the whole cluster.
- **pg_stat_monitor.pgsm_track_utility**: This parameter controls whether utility commands are tracked by the module. Utility commands are all those other than ``SELECT``, ``INSERT``, ``UPDATE`` and ``DELETE``.
- **pg_stat_monitor.pgsm_normalized_query**: By default, query shows the actual parameter instead of the placeholder. It is quite useful when users want to use that query and try to run that query to check the abnormalities. But in most cases users like the queries with a placeholder. This parameter is used to toggle between the two said options.
- **pg_stat_monitor.pgsm_max_buckets**: ``pg_stat_monitor`` accumulates the information in the form of buckets. All the aggregated information is bucket based. This parameter is used to set the number of buckets the system can have. For example, if this parameter is set to 2, then the system will create two buckets. First, the system will add all the information into the first bucket. After its lifetime (defined in the pg_stat_monitor.pgsm_bucket_time parameter) expires, it will switch to the second bucket, reset all the counters and repeat the process.
- **pg_stat_monitor.pgsm_bucket_time**: This parameter is used to set the lifetime of the bucket. System switches between buckets on the basis of ``pg_stat_monitor.pgsm_bucket_time``.
- **pg_stat_monitor.pgsm_object_cache**: This parameter is used to store information about the objects in the query. ``pg_stat_monitor`` saves the information used by the objects in the query. To limit that information, you can set the length of that memory.
- **pg_stat_monitor.pgsm_respose_time_lower_bound**: ``pg_stat_monitor`` also stores the execution time histogram. This parameter is used to set the lower bound of the histogram.
- **pg_stat_monitor.pgsm_respose_time_step:** This parameter is used to set the steps for the histogram.
## Setup
``pg_stat_monitor`` cannot be installed in your running PostgreSQL instance. It should be set in the ``postgresql.conf`` file.
```
# - Shared Library Preloading -
shared_preload_libraries = 'pg_stat_monitor' # (change requires restart)
#local_preload_libraries = ''
#session_preload_libraries = ''
```
Or you can do from `psql` terminal using the ``alter system`` command.
``pg_stat_monitor`` needs to be loaded at the start time. This requires adding the ``pg_stat_monitor`` extension for the ``shared_preload_libraries`` parameter and restart the PostgreSQL instance.
```
postgres=# alter system set shared_preload_libraries=pg_stat_monitor; postgres=# alter system set shared_preload_libraries=pg_stat_monitor;
ALTER SYSTEM ALTER SYSTEM
sudo systemctl restart postgresql-11 sudo systemctl restart postgresql-11
```
Create the extension in the desired database.
Create the extension using the ``create extension`` command.
```
postgres=# create extension pg_stat_monitor; postgres=# create extension pg_stat_monitor;
CREATE EXTENSION CREATE EXTENSION
```
After doing that change, we need to restart the PostgreSQL server. PostgreSQL will start monitoring and collecting the statistics.
#### Usage ## User Guide
There are four views, and complete statistics can be accessed using these views.
* pg_stat_monitor To view the statistics, there are multiple views available.
* pg_stat_agg_database
* pg_stat_agg_user
* pg_stat_agg_host
##### pg_stat_monitor
This is the main view which stores per query-based statistics, similar to pg_stat_statment with some additional columns.
postgres=# \d pg_stat_monitor; ```
postgres=# \d pg_stat_monitor
View "public.pg_stat_monitor" View "public.pg_stat_monitor"
Column | Type | Collation | Nullable | Default Column | Type | Default
---------------------+--------------------------+-----------+----------+--------- ---------------------+--------------------------+-----------
bucket | oid | | | bucket | int |
bucket_start_time | timestamp with time zone | | | bucket_start_time | timestamp with time zone |
userid | oid | | | userid | oid |
dbid | oid | | | dbid | oid |
queryid | text | | | queryid | text |
query | text | | | query | text |
calls | bigint | | | calls | bigint |
total_time | double precision | | | total_time | double precision |
min_time | double precision | | | min_time | double precision |
max_time | double precision | | | max_time | double precision |
mean_time | double precision | | | mean_time | double precision |
stddev_time | double precision | | | stddev_time | double precision |
int8 | bigint | | | rows | bigint |
shared_blks_hit | bigint | | | shared_blks_hit | bigint |
shared_blks_read | bigint | | | shared_blks_read | bigint |
shared_blks_dirtied | bigint | | | shared_blks_dirtied | bigint |
shared_blks_written | bigint | | | shared_blks_written | bigint |
local_blks_hit | bigint | | | local_blks_hit | bigint |
local_blks_read | bigint | | | local_blks_read | bigint |
local_blks_dirtied | bigint | | | local_blks_dirtied | bigint |
local_blks_written | bigint | | | local_blks_written | bigint |
temp_blks_read | bigint | | | temp_blks_read | bigint |
temp_blks_written | bigint | | | temp_blks_written | bigint |
blk_read_time | double precision | | | blk_read_time | double precision |
blk_write_time | double precision | | | blk_write_time | double precision |
host | bigint | | | host | bigint |
client_ip | inet | | | client_ip | inet |
resp_calls | text[] | | | resp_calls | text[] |
cpu_user_time | double precision | | | cpu_user_time | double precision |
cpu_sys_time | double precision | | | cpu_sys_time | double precision |
tables_names | text[] | | | tables_names | text[] |
wait_event | text |
wait_event_type | text |
```
These are new column added to have more detail about the query. **`bucket`**: ``pg_stat_monitor`` accumulates the statistics per bucket. All the information and aggregate reset for each bucket. The `bucket` will be a number showing the number of buckets for which this record belongs.
client_ip: Client IP or Hostname **`bucket_start_time`**: `bucket_start_time` shows the start time of the bucket.
hist_calls: Hourly based 24 hours calls of query histogram
hist_min_time: Hourly based 24 hours min time of query histogram
Hist_max_time: Hourly based 24 hours max time of query histogram
hist_mean_time: Hourly based 24 hours mean time of query histogram
slow_query: Slowest query with actual parameters.
cpu_user_time: CPU user time for that query.
cpu_sys_time: CPU System time for that query.
postgres=# \d pg_stat_agg_database ```
postgres=# select bucket, bucket_start_time, query from pg_stat_monitor;
bucket | bucket_start_time | query
--------+-------------------------------+--------------------------------------------------------
2 | 2020-05-23 13:24:44.652415+00 | select * from pg_stat_monitor_reset()
3 | 2020-05-23 13:45:01.55658+00 | select bucket, bucket_start_time, query from pg_stat_monitor
2 | 2020-05-23 13:24:44.652415+00 | SELECT * FROM foo
(3 rows)
```
**`userid`**: An ID of the user whom that query belongs. ``pg_stat_monitor`` is used to collect queries from all the users; therefore, `userid` is used to segregate the queries based on different users.
**`dbid`**: The database ID of the query. ``pg_stat_monitor`` accumulates queries from all the databases; therefore, this column is used to identify the database.
**`queryid`**: ``pg_stat_monitor`` generates a unique ID for each query (queryid).
**`query`**: The `query` column contains the actual text of the query. This parameter depends on the **`pg_stat_monitor.pgsm_normalized_query`** configuration parameters, in which format to show the query.
**`calls`**: Number of calls of that particular query.
```
postgres=# select userid, dbid, queryid, query, calls from pg_stat_monitor;
userid | dbid | queryid | query | calls
--------+-------+------------------+--------------------------------------------------------------+-------
10 | 12696 | 56F12CE7CD01CF2C | select * from pg_stat_monitor_reset() | 1
10 | 12696 | 748D9EC1F4CECB36 | select userid, dbid, queryid, query from pg_stat_monitor | 1
10 | 12696 | 85900141D214EC52 | select bucket, bucket_start_time, query from pg_stat_monitor | 2
10 | 12696 | F1AC132034D5B366 | SELECT * FROM foo | 1
```
**`total_time`**, **`min_time`**, **`max_time`**, **`mean_time`**: The total / minimum / maximum and mean time spent for the same query.
```
postgres=# select userid, total_time, min_time, max_time, mean_time, query from pg_stat_monitor;
userid | total_time | min_time | max_time | mean_time | query
--------+--------------------+--------------------+--------------------+--------------------+------------------------------------------------------------------
10 | 0.14| 0.14 | 0.14 | 0.14 | select * from pg_stat_monitor_reset()
10 | 0.19 | 0.19 | 0.19 | 0.19 | select userid, dbid, queryid, query from pg_stat_monitor
10 | 0.30 | 0.13 | 0.16 | 0.15 | select bucket, bucket_start_time, query from pg_stat_monitor
10 | 0.29 | 0.29 | 0.29 | 0.29 | select userid, dbid, queryid, query, calls from pg_stat_monitor
10 | 11277.79 | 11277.79 | 11277.79 | 11277.79| SELECT * FROM foo
```
**`client_ip`**: The IP address of the client that originated the query.
```
postgres=# select client_ip, query from pg_stat_monitor;
client_ip | query
-----------+---------------------------------------------------------------------------------------
127.0.0.1 | select * from pg_stat_monitor_reset()
127.0.0.1 | select userid, dbid, queryid, query from pg_stat_monitor
127.0.0.1 | select bucket, bucket_start_time, query from pg_stat_monitor
127.0.0.1 | select userid, total_time, min_time, max_time, mean_time, query from pg_stat_monitor
127.0.0.1 | select userid, dbid, queryid, query, calls from pg_stat_monitor
127.0.0.1 | SELECT * FROM foo
(6 rows)
```
**`resp_calls`**: Call histogram
```
postgres=# select resp_calls, query from pg_stat_monitor;
resp_calls | query
--------------------------------------------------+---------------------------------------------- {1," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"} | select client_ip, query from pg_stat_monitor
{3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | select * from pg_stat_monitor_reset()
{3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | SELECT * FROM foo
```
There are 10 timebase buckets of the time **`pg_stat_monitor.pgsm_respose_time_step`** in the field ``resp_calls``. The value in the field shows how many queries run in that period of time.
**`tables_names`**: The list of tables involved in the query
```
postgres=# select tables_names, query from pg_stat_monitor;
tables_names | query
--------------------------+---------------------------------------------------------------------------------------
{public.pg_stat_monitor} | select client_ip, query from pg_stat_monitor
| select * from pg_stat_monitor_reset()
{public.pg_stat_monitor} | select userid, dbid, queryid, query from pg_stat_monitor
{public.pg_stat_monitor} | select bucket, bucket_start_time, query from pg_stat_monitor
{public.pg_stat_monitor} | select userid, total_time, min_time, max_time, mean_time, query from pg_stat_monitor
{public.pg_stat_monitor} | select userid, dbid, queryid, query, calls from pg_stat_monitor
{public.foo} | SELECT * FROM foo
{public.pg_stat_monitor} | select resp_calls, query from pg_stat_monitor
(8 rows)
```
**`wait_event`**: Current state of the query
```
postgres=# select wait_event_type, query from pg_stat_monitor;
wait_event | query
-----------------+-------------------------------------------------------------------------------
| select client_ip, query from pg_stat_monitor
ClientRead | select wait_event_type, query from pg_stat_monitor
| select * from pg_stat_monitor_reset()
```
**`pg_stat_agg_database`**: Shows the aggregated value per database.
```
postgres=# \d pg_stat_agg_database;
View "public.pg_stat_agg_database" View "public.pg_stat_agg_database"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+--------- ---------------+------------------+-----------+----------+---------
@ -124,8 +492,12 @@ These are new column added to have more detail about the query.
cpu_sys_time | double precision | | | cpu_sys_time | double precision | | |
query | text | | | query | text | | |
tables_names | text[] | | | tables_names | text[] | | |
```
postgres=# \d pg_stat_agg_user **`pg_stat_agg_user`**: Shows the aggregated value per user.
```
postgres=# \d pg_stat_agg_user;
View "public.pg_stat_agg_user" View "public.pg_stat_agg_user"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+--------- ---------------+------------------+-----------+----------+---------
@ -143,8 +515,13 @@ These are new column added to have more detail about the query.
cpu_sys_time | double precision | | | cpu_sys_time | double precision | | |
query | text | | | query | text | | |
tables_names | text[] | | | tables_names | text[] | | |
```
postgres=# \d pg_stat_agg_ip
**`pg_stat_agg_ip`**: Shows the aggregated value per IP / host.
```
postgres=# \d pg_stat_agg_ip;
View "public.pg_stat_agg_ip" View "public.pg_stat_agg_ip"
Column | Type | Collation | Nullable | Default Column | Type | Collation | Nullable | Default
---------------+------------------+-----------+----------+--------- ---------------+------------------+-----------+----------+---------
@ -163,37 +540,15 @@ These are new column added to have more detail about the query.
cpu_sys_time | double precision | | | cpu_sys_time | double precision | | |
query | text | | | query | text | | |
tables_names | text[] | | | tables_names | text[] | | |
```
Examples ## License
1 - In this query we are getting the exact value of f1 which is '05:06:07-07' (special setting is required for this).
# select userid, queryid, query, max_time, total_calls from pg_stat_agg_user; Apache License
-[ RECORD 1 ]---------------------------------------------------------------------------------------- Version 2.0, January 2004
userid | 10 http://www.apache.org/licenses/
queryid | -203926152419851453
query | SELECT f1 FROM TIMETZ_TBL WHERE f1 < '05:06:07-07';
max_time | 1.237875
total_calls | 8
2 - Collect all statistics based on the user. TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
# select usename, query, max_time, total_calls from pg_stat_agg_user au, pg_user u where au.userid = u.usesysid;
usename | query | max_time | total_calls
---------+---------------------------------------------------------+----------+-------------
vagrant | select userid, query, total_calls from pg_stat_agg_user | 0.268842 | 7
foo | select userid, query, total_calls from pg_stat_agg_user | 0.208551 | 1
vagrant | select * from pg_stat_monitor_reset() | 0.0941 | 1
(3 rows)
#### Limitation
There are some limitations and Todos.
#### Licence
Copyright (c) 2006 - 2019, Percona LLC.
See [`LICENSE`][2] for full details.
[1]: https://www.postgresql.org/
[2]: https://github.com/Percona/pg_stat_monitor/blob/master/LICENSE
[3]: https://github.com/Percona/pg_stat_monitor/issues/new
[4]: CONTRIBUTING.md

4
percona-packaging/debian/control
View File

@ -6,10 +6,10 @@ Build-Depends:
debhelper (>= 9), debhelper (>= 9),
percona-postgresql-server-dev-all (>= 153~), percona-postgresql-server-dev-all (>= 153~),
Package: percona-pg-stat-monitor Package: percona-pg-stat-monitor@@PG_REL@@
Architecture: any Architecture: any
Depends: Depends:
percona-postgresql-11, percona-postgresql-@@PG_REL@@,
${misc:Depends}, ${misc:Depends},
${shlibs:Depends}, ${shlibs:Depends},
Description: The pg_stat_monitor is statistics collector tool Description: The pg_stat_monitor is statistics collector tool

4
percona-packaging/debian/control.in
View File

@ -6,10 +6,10 @@ Build-Depends:
debhelper (>= 9), debhelper (>= 9),
percona-postgresql-server-dev-all (>= 153~), percona-postgresql-server-dev-all (>= 153~),
Package: percona-pg-stat-monitor Package: percona-pg-stat-monitor@@PG_REL@@
Architecture: any Architecture: any
Depends: Depends:
percona-postgresql-11, percona-postgresql-@@PG_REL@@,
${misc:Depends}, ${misc:Depends},
${shlibs:Depends}, ${shlibs:Depends},
Description: The pg_stat_monitor is statistics collector tool Description: The pg_stat_monitor is statistics collector tool

2
percona-packaging/debian/rules
View File

@ -18,7 +18,7 @@ override_dh_auto_test:
# nothing to do here # nothing to do here
override_dh_auto_install: override_dh_auto_install:
+pg_buildext install build-%v percona-pg-stat-monitor +pg_buildext install build-%v percona-pg-stat-monitor@@PG_REL@@
override_dh_installdocs: override_dh_installdocs:
dh_installdocs --all README.* dh_installdocs --all README.*

16
percona-packaging/rpm/pg-stat-monitor.spec
View File

@ -1,17 +1,19 @@
%global sname percona-pg-stat-monitor %global sname percona-pg-stat-monitor
%global pginstdir /usr/pgsql-11/ %global pgrel @@PG_REL@@
%global rpm_release @@RPM_RELEASE@@
%global pginstdir /usr/pgsql-@@PG_REL@@/
%global pgstatmonmajver 1 %global pgstatmonmajver 1
%global pgstatmonmidver 0 %global pgstatmonmidver 0
%global pgstatmonminver 0 %global pgstatmonminver 0
Summary: Statistics collector for PostgreSQL Summary: Statistics collector for PostgreSQL
Name: %{sname} Name: %{sname}%{pgrel}
Version: %{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver} Version: %{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}
Release: 1%{?dist} Release: %{rpm_release}%{?dist}
License: Apache 2.0 License: Apache 2.0
Source0: %{sname}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}.tar.gz Source0: %{sname}%{pgrel}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}.tar.gz
URL: https://github.com/Percona-Lab/pg_stat_monitor URL: https://github.com/Percona-Lab/pg_stat_monitor
BuildRequires: percona-postgresql11-devel BuildRequires: percona-postgresql%{pgrel}-devel
Requires: postgresql-server Requires: postgresql-server
@ -25,11 +27,11 @@ It provides all the features of pg_stat_statment plus its own feature set.
%prep %prep
%setup -q -n %{sname}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver} %setup -q -n %{sname}%{pgrel}-%{pgstatmonmajver}.%{pgstatmonmidver}.%{pgstatmonminver}
%build %build
sed -i 's:PG_CONFIG = pg_config:PG_CONFIG = /usr/pgsql-11/bin/pg_config:' Makefile sed -i 's:PG_CONFIG = pg_config:PG_CONFIG = /usr/pgsql-%{pgrel}/bin/pg_config:' Makefile
%{__make} USE_PGXS=1 %{?_smp_mflags} %{__make} USE_PGXS=1 %{?_smp_mflags}

216
percona-packaging/scripts/pg_stat_monitor_builder.sh
View File

@ -10,15 +10,19 @@ Usage: $0 [OPTIONS]
The following options may be given : The following options may be given :
--builddir=DIR Absolute path to the dir where all actions will be performed --builddir=DIR Absolute path to the dir where all actions will be performed
--get_sources Source will be downloaded from github --get_sources Source will be downloaded from github
--build_src_rpm If it is set - src rpm will be built --build_src_rpm If it is 1 src rpm will be built
--build_src_deb If it is set - source deb package will be built --build_source_deb If it is 1 source deb package will be built
--build_rpm If it is set - rpm will be built --build_rpm If it is 1 rpm will be built
--build_deb If it is set - deb will be built --build_deb If it is 1 deb will be built
--install_deps Install build dependencies(root privilages are required) --build_tarball If it is 1 tarball will be built
--install_deps Install build dependencies(root previlages are required)
--branch Branch for build --branch Branch for build
--repo Repo for build --repo Repo for build
--rpm_release RPM version( default = 1)
--deb_release DEB version( default = 1)
--pg_release PPG version build on( default = 11)
--help) usage ;; --help) usage ;;
Example $0 --builddir=/tmp/BUILD --get_sources=1 --build_src_rpm=1 --build_rpm=1 Example $0 --builddir=/tmp/test --get_sources=1 --build_src_rpm=1 --build_rpm=1
EOF EOF
exit 1 exit 1
} }
@ -38,15 +42,20 @@ parse_arguments() {
for arg do for arg do
val=$(echo "$arg" | sed -e 's;^--[^=]*=;;') val=$(echo "$arg" | sed -e 's;^--[^=]*=;;')
case "$arg" in case "$arg" in
# these get passed explicitly to mysqld
--builddir=*) WORKDIR="$val" ;; --builddir=*) WORKDIR="$val" ;;
--build_src_rpm=*) SRPM="$val" ;; --build_src_rpm=*) SRPM="$val" ;;
--build_src_deb=*) SDEB="$val" ;; --build_source_deb=*) SDEB="$val" ;;
--build_rpm=*) RPM="$val" ;; --build_rpm=*) RPM="$val" ;;
--build_deb=*) DEB="$val" ;; --build_deb=*) DEB="$val" ;;
--get_sources=*) SOURCE="$val" ;; --get_sources=*) SOURCE="$val" ;;
--build_tarball=*) TARBALL="$val" ;;
--install_deps=*) INSTALL="$val" ;;
--branch=*) BRANCH="$val" ;; --branch=*) BRANCH="$val" ;;
--repo=*) REPO="$val" ;; --repo=*) REPO="$val" ;;
--install_deps=*) INSTALL="$val" ;; --rpm_release=*) RPM_RELEASE="$val" ;;
--deb_release=*) DEB_RELEASE="$val" ;;
--pg_release=*) PG_RELEASE="$val" ;;
--help) usage ;; --help) usage ;;
*) *)
if test -n "$pick_args" if test -n "$pick_args"
@ -74,31 +83,10 @@ check_workdir(){
} }
add_percona_yum_repo(){ add_percona_yum_repo(){
if [ ! -f /etc/yum.repos.d/percona-dev.repo ] if [ ! -f /etc/yum.repos.d/percona-dev.repo ]; then
then curl -o /etc/yum.repos.d/percona-dev.repo https://jenkins.percona.com/yum-repo/percona-dev.repo
wget http://jenkins.percona.com/yum-repo/percona-dev.repo sed -i 's:$basearch:x86_64:g' /etc/yum.repos.d/percona-dev.repo
mv -f percona-dev.repo /etc/yum.repos.d/
fi fi
yum -y install https://repo.percona.com/yum/percona-release-latest.noarch.rpm
percona-release disable all
percona-release enable ppg-11 release
return
}
add_percona_apt_repo(){
if [ ! -f /etc/apt/sources.list.d/percona-dev.list ]; then
cat >/etc/apt/sources.list.d/percona-dev.list <<EOL
deb http://jenkins.percona.com/apt-repo/ @@DIST@@ main
deb-src http://jenkins.percona.com/apt-repo/ @@DIST@@ main
EOL
sed -i "s:@@DIST@@:$OS_NAME:g" /etc/apt/sources.list.d/percona-dev.list
fi
wget -qO - http://jenkins.percona.com/apt-repo/8507EFA5.pub | apt-key add -
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
dpkg -i percona-release_latest.generic_all.deb
percona-release disable all
rm -f percona-release_latest.generic_all.deb
percona-release enable ppg-11 release
return return
} }
@ -109,14 +97,19 @@ get_sources(){
echo "Sources will not be downloaded" echo "Sources will not be downloaded"
return 0 return 0
fi fi
PRODUCT=percona-pg-stat-monitor PRODUCT=percona-pg-stat-monitor${PG_RELEASE}
echo "PRODUCT=${PRODUCT}" > pg-stat-monitor.properties
PRODUCT_FULL=${PRODUCT}-${VERSION} PRODUCT_FULL=${PRODUCT}-${VERSION}
echo "PRODUCT=${PRODUCT}" > pg-stat-monitor.properties
echo "PRODUCT_FULL=${PRODUCT_FULL}" >> pg-stat-monitor.properties echo "PRODUCT_FULL=${PRODUCT_FULL}" >> pg-stat-monitor.properties
echo "VERSION=${PSM_VER}" >> pg-stat-monitor.properties echo "VERSION=${VERSION}" >> pg-stat-monitor.properties
echo "BRANCH_NAME=$(echo ${BRANCH} | awk -F '/' '{print $(NF)}')" >> pg-stat-monitor.properties
echo "BUILD_NUMBER=${BUILD_NUMBER}" >> pg-stat-monitor.properties echo "BUILD_NUMBER=${BUILD_NUMBER}" >> pg-stat-monitor.properties
echo "BUILD_ID=${BUILD_ID}" >> pg-stat-monitor.properties echo "BUILD_ID=${BUILD_ID}" >> pg-stat-monitor.properties
echo "BRANCH_NAME=$(echo ${BRANCH} | awk -F '/' '{print $(NF)}')" >> pg-stat-monitor.properties
echo "PG_RELEASE=${PG_RELEASE}" >> pg-stat-monitor.properties
echo "RPM_RELEASE=${RPM_RELEASE}" >> pg-stat-monitor.properties
echo "DEB_RELEASE=${DEB_RELEASE}" >> pg-stat-monitor.properties
git clone "$REPO" ${PRODUCT_FULL} git clone "$REPO" ${PRODUCT_FULL}
retval=$? retval=$?
if [ $retval != 0 ] if [ $retval != 0 ]
@ -136,13 +129,20 @@ get_sources(){
rm -rf rpm debian rm -rf rpm debian
cp -r percona-packaging/rpm ./ cp -r percona-packaging/rpm ./
cp -r percona-packaging/debian ./ cp -r percona-packaging/debian ./
EDITFILES="debian/control debian/control.in debian/rules rpm/pg-stat-monitor.spec"
for file in $EDITFILES; do
sed -i "s:@@PG_REL@@:${PG_RELEASE}:g" "$file"
done
sed -i "s:@@RPM_RELEASE@@:${RPM_RELEASE}:g" rpm/pg-stat-monitor.spec
cd ${WORKDIR} cd ${WORKDIR}
# #
source pg-stat-monitor.properties source pg-stat-monitor.properties
# #
tar --owner=0 --group=0 --exclude=.* -czf ${PRODUCT_FULL}.tar.gz ${PRODUCT_FULL} tar --owner=0 --group=0 --exclude=.* -czf ${PRODUCT_FULL}.tar.gz ${PRODUCT_FULL}
echo "UPLOAD=UPLOAD/experimental/BUILDS/${PRODUCT}/${PRODUCT_FULL}/${PSM_BRANCH}/${REVISION}/${BUILD_ID}" >> pg-stat-monitor.properties echo "UPLOAD=UPLOAD/experimental/BUILDS/${PRODUCT}/${PRODUCT_FULL}/${BRANCH}/${REVISION}/${BUILD_ID}" >> pg-stat-monitor.properties
mkdir $WORKDIR/source_tarball mkdir $WORKDIR/source_tarball
mkdir $CURDIR/source_tarball mkdir $CURDIR/source_tarball
cp ${PRODUCT_FULL}.tar.gz $WORKDIR/source_tarball cp ${PRODUCT_FULL}.tar.gz $WORKDIR/source_tarball
@ -154,14 +154,16 @@ get_sources(){
get_system(){ get_system(){
if [ -f /etc/redhat-release ]; then if [ -f /etc/redhat-release ]; then
RHEL=$(rpm --eval %rhel) GLIBC_VER_TMP="$(rpm glibc -qa --qf %{VERSION})"
ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g') export RHEL=$(rpm --eval %rhel)
OS_NAME="el$RHEL" export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g')
OS="rpm" export OS_NAME="el$RHEL"
export OS="rpm"
else else
ARCH=$(uname -m) GLIBC_VER_TMP="$(dpkg-query -W -f='${Version}' libc6 | awk -F'-' '{print $1}')"
OS_NAME="$(lsb_release -sc)" export ARCH=$(uname -m)
OS="deb" export OS_NAME="$(lsb_release -sc)"
export OS="deb"
fi fi
return return
} }
@ -178,61 +180,68 @@ install_deps() {
exit 1 exit 1
fi fi
CURPLACE=$(pwd) CURPLACE=$(pwd)
if [ "$OS" == "rpm" ]
if [ "x$OS" = "xrpm" ]; then then
yum -y install wget yum install -y https://repo.percona.com/yum/percona-release-latest.noarch.rpm
add_percona_yum_repo add_percona_yum_repo
wget http://jenkins.percona.com/yum-repo/percona-dev.repo if [[ ${PG_RELEASE} == "11" ]]; then
mv -f percona-dev.repo /etc/yum.repos.d/ percona-release enable ppg-11 release
yum clean all elif [[ $PG_RELEASE == "12" ]]; then
RHEL=$(rpm --eval %rhel) percona-release enable ppg-12 release
if [ ${RHEL} = 8 ]; then fi
yum -y install git wget
PKGLIST+=" clang-devel git clang llvm-devel rpmdevtools vim wget"
PKGLIST+=" perl binutils gcc gcc-c++"
PKGLIST+=" percona-postgresql-common clang-devel llvm-devel percona-postgresql${PG_RELEASE}-devel git rpm-build rpmdevtools wget gcc make autoconf"
if [[ "${RHEL}" -eq 8 ]]; then
dnf -y module disable postgresql dnf -y module disable postgresql
dnf config-manager --set-enabled codeready-builder-for-rhel-8-x86_64-rpms elif [[ "${RHEL}" -eq 7 ]]; then
dnf clean all PKGLIST+=" llvm-toolset-7-clang llvm-toolset-7-llvm-devel llvm5.0-devel"
rm -r /var/cache/dnf until yum -y install epel-release centos-release-scl; do
dnf -y upgrade yum clean all
yum -y install perl sleep 1
else echo "waiting"
until yum -y install centos-release-scl; do done
until yum -y makecache; do
yum clean all
sleep 1
echo "waiting"
done
fi
until yum -y install ${PKGLIST}; do
echo "waiting" echo "waiting"
sleep 1 sleep 1
done done
yum -y install epel-release else
INSTALL_LIST="llvm-toolset-7-clang llvm5.0-devel" apt-get update
yum -y install ${INSTALL_LIST} DEBIAN_FRONTEND=noninteractive apt-get -y install lsb-release gnupg git wget
source /opt/rh/devtoolset-7/enable
source /opt/rh/llvm-toolset-7/enable
INSTALL_LIST="clang-devel git clang llvm-devel rpmdevtools vim wget"
yum -y install ${INSTALL_LIST}
yum -y install binutils gcc gcc-c++
fi
INSTALL_LIST="percona-postgresql-common clang-devel llvm-devel percona-postgresql11-devel git rpm-build rpmdevtools wget gcc make autoconf"
yum -y install ${INSTALL_LIST}
else wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb && dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
export DEBIAN=$(lsb_release -sc) if [[ "${PG_RELEASE}" == "11" ]]; then
export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g') percona-release enable ppg-11 release
apt-get -y install gnupg2 elif [[ "${PG_RELEASE}" == "12" ]]; then
add_percona_apt_repo percona-release enable ppg-12 release
apt-get update || true
LLVM_EXISTS=$(grep -c "apt.llvm.org" /etc/apt/sources.list)
if [ ${LLVM_EXISTS} = 0 ]; then
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
echo "deb http://apt.llvm.org/${DEBIAN}/ llvm-toolchain-${DEBIAN}-7 main" >> /etc/apt/sources.list
echo "deb-src http://apt.llvm.org/${DEBIAN}/ llvm-toolchain-${DEBIAN}-7 main" >> /etc/apt/sources.list
apt-get --allow-unauthenticated update
fi fi
apt-get update || true apt-get update
INSTALL_LIST="build-essential percona-postgresql-11 debconf debhelper clang-7 devscripts dh-exec dh-systemd git wget libkrb5-dev libssl-dev percona-postgresql-common percona-postgresql-server-dev-all"
DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated install ${INSTALL_LIST} if [[ "${OS_NAME}" != "focal" ]]; then
INSTALL_LIST="build-essential debconf debhelper devscripts dh-exec dh-systemd git wget libxml-checker-perl libxml-libxml-perl libio-socket-ssl-perl libperl-dev libssl-dev libxml2-dev txt2man zlib1g-dev libpq-dev" LLVM_EXISTS=$(grep -c "apt.llvm.org" /etc/apt/sources.list)
until DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated install ${INSTALL_LIST}; do if [ "${LLVM_EXISTS}" == 0 ]; then
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
echo "deb http://apt.llvm.org/${OS_NAME}/ llvm-toolchain-${OS_NAME}-7 main" >> /etc/apt/sources.list
echo "deb-src http://apt.llvm.org/${OS_NAME}/ llvm-toolchain-${OS_NAME}-7 main" >> /etc/apt/sources.list
apt-get update
fi
fi
PKGLIST+=" percona-postgresql-${PG_RELEASE} debconf debhelper clang-7 devscripts dh-exec dh-systemd git wget libkrb5-dev libssl-dev percona-postgresql-common percona-postgresql-server-dev-all"
PKGLIST+=" build-essential debconf debhelper devscripts dh-exec dh-systemd git wget libxml-checker-perl libxml-libxml-perl libio-socket-ssl-perl libperl-dev libssl-dev libxml2-dev txt2man zlib1g-dev libpq-dev"
until DEBIAN_FRONTEND=noninteractive apt-get -y install ${PKGLIST}; do
sleep 1 sleep 1
echo "waiting" echo "waiting"
done done
apt-get -y --allow-unauthenticated install libmysqlclient-dev || true
apt-get -y --allow-unauthenticated install default-libmysqlclient-dev || true
fi fi
return; return;
} }
@ -350,8 +359,8 @@ build_rpm(){
source /opt/rh/devtoolset-7/enable source /opt/rh/devtoolset-7/enable
source /opt/rh/llvm-toolset-7/enable source /opt/rh/llvm-toolset-7/enable
fi fi
export LIBPQ_DIR=/usr/pgsql-11/ export LIBPQ_DIR=/usr/pgsql-${PG_RELEASE}/
export LIBRARY_PATH=/usr/pgsql-11/lib/:/usr/pgsql-11/include/ export LIBRARY_PATH=/usr/pgsql-${PG_RELEASE}/lib/:/usr/pgsql-${PG_RELEASE}/include/
rpmbuild --define "_topdir ${WORKDIR}/rpmbuild" --define "dist .$OS_NAME" --define "version ${VERSION}" --rebuild rpmbuild/SRPMS/$SRC_RPM rpmbuild --define "_topdir ${WORKDIR}/rpmbuild" --define "dist .$OS_NAME" --define "version ${VERSION}" --rebuild rpmbuild/SRPMS/$SRC_RPM
return_code=$? return_code=$?
@ -389,7 +398,7 @@ build_source_deb(){
mv ${TARFILE} percona-pg-stat-monitor_${VERSION}.orig.tar.gz mv ${TARFILE} percona-pg-stat-monitor_${VERSION}.orig.tar.gz
cd ${BUILDDIR} cd ${BUILDDIR}
dch -D unstable --force-distribution -v "${VERSION}-${RELEASE}" "Update to new percona-pg-stat-monitor version ${VERSION}" dch -D unstable --force-distribution -v "${VERSION}-${DEB_RELEASE}" "Update to new percona-pg-stat-monitor${PG_RELEASE} version ${VERSION}"
dpkg-buildpackage -S dpkg-buildpackage -S
cd ../ cd ../
mkdir -p $WORKDIR/source_deb mkdir -p $WORKDIR/source_deb
@ -422,19 +431,17 @@ build_deb(){
cd $WORKDIR cd $WORKDIR
rm -fv *.deb rm -fv *.deb
# #
export DEBIAN=$(lsb_release -sc)
export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g') export ARCH=$(echo $(uname -m) | sed -e 's:i686:i386:g')
# #
echo "DEBIAN=${DEBIAN}" >> pg-stat-monitor.properties echo "DEBIAN=${OS_NAME}" >> pg-stat-monitor.properties
echo "ARCH=${ARCH}" >> pg-stat-monitor.properties echo "ARCH=${ARCH}" >> pg-stat-monitor.properties
# #
DSC=$(basename $(find . -name '*.dsc' | sort | tail -n1)) DSC=$(basename $(find . -name '*.dsc' | sort | tail -n1))
# #
dpkg-source -x ${DSC} dpkg-source -x ${DSC}
# #
cd percona-pg-stat-monitor-${VERSION} cd percona-pg-stat-monitor-${VERSION}
dch -m -D "${DEBIAN}" --force-distribution -v "1:${VERSION}-${RELEASE}.${DEBIAN}" 'Update distribution' dch -m -D "${OS_NAME}" --force-distribution -v "1:${VERSION}-${DEB_RELEASE}.${OS_NAME}" 'Update distribution'
unset $(locale|cut -d= -f1) unset $(locale|cut -d= -f1)
dpkg-buildpackage -rfakeroot -us -uc -b dpkg-buildpackage -rfakeroot -us -uc -b
mkdir -p $CURDIR/deb mkdir -p $CURDIR/deb
@ -442,10 +449,9 @@ build_deb(){
cp $WORKDIR/*.*deb $WORKDIR/deb cp $WORKDIR/*.*deb $WORKDIR/deb
cp $WORKDIR/*.*deb $CURDIR/deb cp $WORKDIR/*.*deb $CURDIR/deb
} }
#main
CURDIR=$(pwd) CURDIR=$(pwd)
VERSION_FILE=$CURDIR/pg-stat-monitor.properties VERSION_FILE=$CURDIR/percona-pg-stat-monitor.properties
args= args=
WORKDIR= WORKDIR=
SRPM=0 SRPM=0
@ -453,21 +459,19 @@ SDEB=0
RPM=0 RPM=0
DEB=0 DEB=0
SOURCE=0 SOURCE=0
TARBALL=0
OS_NAME= OS_NAME=
ARCH= ARCH=
OS= OS=
REVISION=0
BRANCH="master"
INSTALL=0 INSTALL=0
RPM_RELEASE=1 RPM_RELEASE=1
DEB_RELEASE=1 DEB_RELEASE=1
REVISION=0
BRANCH="master"
REPO="https://github.com/Percona/pg_stat_monitor.git" REPO="https://github.com/Percona/pg_stat_monitor.git"
PRODUCT=percona-pg-stat-monitor VERSION="1.0.0"
DEBUG=0 PG_RELEASE=11
parse_arguments PICK-ARGS-FROM-ARGV "$@" parse_arguments PICK-ARGS-FROM-ARGV "$@"
VERSION='1.0.0'
RELEASE='1'
PRODUCT_FULL=${PRODUCT}-${VERSION}-${RELEASE}
check_workdir check_workdir
get_system get_system