Merge branch 'master' of github.com:xstevens/decoderbufs

pull/1/head
Xavier Stevens 2015-01-07 10:59:34 -08:00
commit ec818d7d58
2 changed files with 22 additions and 18 deletions

View File

@ -12,7 +12,7 @@ Version: 0.0.1
**decoderbufs** is released under the MIT license (See LICENSE file).
**Shoutouts:**
- [The PostgreSQL Team](https://postgresql.org) for adding logical decoding support. This is a immensely useful feature.
- [The PostgreSQL Team](https://postgresql.org) for adding [logical decoding](http://www.postgresql.org/docs/9.4/static/logicaldecoding.html) support. This is a immensely useful feature.
- [Michael Paquier](https://github.com/michaelpq) for his [decoder_raw](https://github.com/michaelpq/pg_plugins/tree/master/decoder_raw)
project and blog posts as a guide to teach myself how to write a PostgreSQL logical decoder output plugin.
- [Martin Kleppmann](https://github.com/ept) for making me aware that PostgreSQL was working on logical decoding.
@ -35,7 +35,7 @@ This code is built with the following assumptions. You may get mixed results if
To build you will need to install PostgreSQL (for pg_config) and PostgreSQL server development packages. On Debian
based distributions you can usually do something like this:
apt-get install -y postgresql postgresql-server-dev-9.2
apt-get install -y postgresql postgresql-server-dev-9.4
You will also need to make sure that protobuf-c and it's header files have been installed. See their Github
page for further details.
@ -56,7 +56,7 @@ Once the extension has been installed you just need to enable it and logical rep
#wal_sender_timeout = 60s # in milliseconds; 0 disables
max_replication_slots = 4 # max number of replication slots (change requires restart)
In addition, permissions will have to be added for the user that astrochicken uses to connect to be able to replicate. This can be modified in _pg\_hba.conf_ like so:
In addition, permissions will have to be added for the user that connects to the DB to be able to replicate. This can be modified in _pg\_hba.conf_ like so:
local replication <youruser> trust
host replication <youruser> 127.0.0.1/32 trust
@ -67,16 +67,20 @@ And restart PostgreSQL.
### Usage
-- can use SQL for demo purposes
select * from pg_create_logical_replication_slot('decoderbufs_demo', 'decoderbufs');
-- DO SOME TABLE MODIFICATIONS
-- DO SOME TABLE MODIFICATIONS (see below about UPDATE/DELETE)
-- peek at WAL changes using decoderbufs debug mode for SQL console
select data from pg_logical_slot_peek_changes('decoderbufs_demo', NULL, NULL, 'debug-mode', '1');
-- get WAL changes using decoderbufs to update the WAL position
select data from pg_logical_slot_get_changes('decoderbufs_demo', NULL, NULL, 'debug-mode', '1');
-- check the WAL position of logical replicators
select * from pg_replication_slots where slot_type = 'logical';
The binary format uses simple frame encoding, which uses an 8-byte length (uint64_t) followed by that number of bytes for the Protocol Buffer payload. The
easy way to test check this out is to use _pg\_recvlogical_ like so:
If you're performing an UPDATE/DELETE on your table and you don't see results for those operations from logical decoding, make sure you have set [REPLICA IDENTITY](http://www.postgresql.org/docs/9.4/static/sql-altertable.html#SQL-CREATETABLE-REPLICA-IDENTITY) appropriately for your use case.
The binary format uses simple frame encoding, which uses an 8-byte length (uint64\_t) followed by that number of bytes for the Protocol Buffer payload. The easy way to test check this out is to use _pg\_recvlogical_ like so:
pg_recvlogical -h localhost -d <yourdb> -U <youruser> -w -S decoderbufs_demo -P decoderbufs -f decoderbuf.frames -s 1 -F 1 --start

View File

@ -1,6 +1,6 @@
package decoderbufs;
option java_package="dedoderbufs.proto";
option java_package="decoderbufs.proto";
option java_outer_classname = "PgldProto";
option optimize_for = SPEED;
@ -11,20 +11,20 @@ enum Op {
}
message DatumMessage {
optional string column_name = 1;
optional int64 column_type = 2;
optional int32 datum_int32 = 3;
optional int64 datum_int64 = 4;
optional float datum_float = 5;
optional double datum_double = 6;
optional bool datum_bool = 7;
optional string datum_string = 8;
optional bytes datum_bytes = 9;
optional string column_name = 1;
optional int64 column_type = 2;
optional int32 datum_int32 = 3;
optional int64 datum_int64 = 4;
optional float datum_float = 5;
optional double datum_double = 6;
optional bool datum_bool = 7;
optional string datum_string = 8;
optional bytes datum_bytes = 9;
}
message RowMessage {
optional sint64 commit_time = 1;
optional string table = 2;
optional sint64 commit_time = 1;
optional string table = 2;
optional Op op = 3;
repeated DatumMessage new_tuple = 4;
repeated DatumMessage old_tuple = 5;