From aebc1ba67252e85fdf2a0914724e448ff17f91ad Mon Sep 17 00:00:00 2001 From: Xavier Stevens Date: Tue, 30 Sep 2014 12:09:45 -0700 Subject: [PATCH] Updating README --- README.md | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/README.md b/README.md index 502f5f4..5ae8fef 100644 --- a/README.md +++ b/README.md @@ -2,3 +2,97 @@ decoderbufs =========== A PostgreSQL logical decoder output plugin to deliver data as Protocol Buffers + +# decoderbufs + +Version: 0.0.1 + +**decoderbufs** is a PostgreSQL logical decoder output plugin to deliver data as Protocol Buffers. + +**decoderbufs** is released under the MIT license (See LICENSE file). + +**Shoutouts:** +- [The PostgreSQL Team](https://postgresql.org) for adding logical decoding support. This is a immensely useful feature. +- [Michael Paquier](https://github.com/michaelpq) for his [decoder_raw](https://github.com/michaelpq/pg_plugins/tree/master/decoder_raw) +project and blog posts as a guide to teach myself how to write a PostgreSQL logical decoder output plugin. +- [Martin Kleppmann](https://github.com/ept) for making me aware that PostgreSQL was working on logical decoding. +- [Magnus Edenhill](https://github.com/edenhill) for letting me know that protobuf-c existed and just in general for writing librdkafka. + +### Version Compatability +This code is built with the following assumptions. You may get mixed results if you deviate from these versions. + +* [PostgreSQL](http://www.postgresql.org) 9.4+ +* [Protocol Buffers](https://developers.google.com/protocol-buffers) 2.5.0 +* [protobuf-c](https://github.com/protobuf-c/protobuf-c) 1.0.2 + +### Requirements +* PostgreSQL +* Protocol Buffers +* protobuf-c + +### Building + +To build you will need to install PostgreSQL (for pg_config) and PostgreSQL server development packages. On Debian +based distributions you can usually do something like this: + + apt-get install -y postgresql postgresql-server-dev-9.2 + +You will also need to make sure that protobuf-c and it's header files have been installed. See their Github +page for further details. + +If you have all of the prerequisites installed you should be able to just: + + make && make install + +Once the extension has been installed you just need to enable it and logical replication in postgresql.conf: + + # MODULES + shared_preload_libraries = 'decoderbufs' + + # REPLICATION + wal_level = logical # minimal, archive, hot_standby, or logical (change requires restart) + max_wal_senders = 8 # max number of walsender processes (change requires restart) + wal_keep_segments = 4 # in logfile segments, 16MB each; 0 disables + #wal_sender_timeout = 60s # in milliseconds; 0 disables + max_replication_slots = 4 # max number of replication slots (change requires restart) + +In addition, permissions will have to be added for the user that astrochicken uses to connect to be able to replicate. This can be modified in _pg\_hba.conf_ like so: + + local replication trust + host replication 127.0.0.1/32 trust + host replication ::1/128 trust + +And restart PostgreSQL. + +### Usage + -- can use SQL for demo purposes + select * from pg_create_logical_replication_slot('decoderbufs_demo', 'decoderbufs'); + -- DO SOME TABLE MODIFICATIONS + -- peek at WAL changes using decoderbufs debug mode for SQL console + select data from pg_logical_slot_peek_changes('decoderbufs_demo', NULL, NULL, 'debug-mode', '1'); + -- get WAL changes using decoderbufs to update the WAL position + select data from pg_logical_slot_get_changes('decoderbufs_demo', NULL, NULL, 'debug-mode', '1'); + -- check the WAL position of logical replicators + select * from pg_replication_slots where slot_type = 'logical'; + +The binary format uses simple frame encoding, which uses an 8-byte length (uint64_t) followed by that number of bytes for the Protocol Buffer payload. The +easy way to test check this out is to use _pg\_recvlogical_ like so: + + pg_recvlogical -h localhost -d -U -w -S decoderbufs_demo -P decoderbufs -f decoderbuf.frames -s 1 -F 1 --start + +For something a bit more useful, I am looking to implement a custom PostgreSQL logical replication client that publishes to something like Apache Kafka. + +### Support + +File bug reports, feature requests and questions using +[GitHub Issues](https://github.com/xstevens/decoderbufs/issues) + +### Notes + +This approach is the one we wanted when we first started kicking around ideas for how to replicate our Postgres DBs in near-realtime. It should provide much better +resiliency in the face of network outages and unplanned downtime than a push mechanism (like using pg_kafka with a trigger) would. + +The PostgreSQL docs are pretty good and are definitely worth a read. + +**NOT ALL OID TYPES ARE SUPPORTED CURRENTLY**. I really want to iterate this point. There are lots of OIDs in Postgres. Right now I'm translating the ones that I see used a lot, but it is by no means comprehensive. I hope to update this project to support most if not all types at some point. +