changelog

2.0.18 - 31 March 2022

  • Support the ON DELETE and ON UPDATE clause when creating the foreign keys in PostgreSQL
  • change logic for index and foreign key names by managing only duplicates within same schema
  • use mysql-replication<0.27 as new versions crash when receiving queries
  • add copy_schema method for copying only the schema without data (EXPERIMENTAL)
  • change type for identifiers in replica schema to varchar(64)

2.0.17 - 30 January 2022

  • Remove argparse from the requirements
  • Add the collect for unique constraints when keep_existing_schema is Yes
  • Fix wrong order in copy data/create indices when keep_existing_schema is No
  • Remove check for log_bin we are replicating from Aurora MySQL
  • Manage different the different behaviour in pyyaml to allow pg_chameleon to be installed as rpm in centos 7

2.0.16 - 23 September 2020

  • Fix for issue #126 init_replica failure with tables on transactional engine and invalid data

2.0.15 - 20 September 2020

  • Support for reduced lock if MySQL engine is transactional, thanks to @rascalDan
  • setup.py now requires python-mysql-replication to version 0.22 which adds support for PyMySQL >=0.10.0
  • removed PyMySQL requirement <0.10.0 from setup.py
  • prevent pg_chameleon to run as root

2.0.14 - 26 July 2020

  • Add support for spatial data types (requires postgis installed on the target database)
  • When keep_existing_schema is set to yes now drops and recreates indices, and constraints during the init_replica process
  • Fix for issue #115 thanks to @porshkevich
  • setup.py now forces PyMySQL to version <0.10.0 because it breaks the python-mysql-replication library (issue #117)

2.0.13 - 05 July 2020

  • EXPERIMENTAL support for Point datatype - @jovankricka-everon
  • Add keep_existing_schema in MySQL source type to keep the existing scema in place instead of rebuilding it from the mysql source
  • Change tabs to spaces in code

2.0.12 - 11 Dec 2019

  • Fixes for issue #96 thanks to @daniel-qcode
  • Change for configuration and SQL files location
  • Package can build now as source and wheel
  • The minimum python requirements now is 3.5

2.0.11 - 25 Oct 2019

  • Fix wrong formatting for yaml example files. @rebtoor
  • Make start_replica run in foreground when log_file == stdout . @clifff
  • Travis seems to break down constantly, Disable the CI until a fix is found. Evaluate to use a different CI.
  • Add the add loader to yaml.load as required by the new PyYAML version.

2.0.10 - 01 Sep 2018

  • Fix regression in new replay function with PostgreSQL 10
  • Convert to string the dictionary entries pulled from a json field
  • Let enable_replica to disable any leftover maintenance flag
  • Add capture in CHANGE for tables in the form schema.table

2.0.9 - 19 Aug 2018

  • Fix wrong check for the next auto maintenance run if the maintenance wasn’t run before
  • Improve the replay function’s speed
  • Remove blocking from the GTID operational mode

2.0.8 - 14 Jul 2018

  • Add support for skip events as requested in issue #76. Is now possible to skip events (insert,delete,update) for single tables or for entire schemas.
  • EXPERIMENTAL support for the GTID. When configured on MySQL or Percona server pg_chameleon will use the GTID to auto position the replica stream. Mariadb is not supported by this change.
  • ALTER TABLE RENAME is now correctly parsed and executed
  • Add horrible hack to ALTER TABLE MODIFY. Previously modify with default values would parse wrongly and fail when translating to PostgreSQL dialect
  • Disable erroring the source when running with --debug switch enabled
  • Add cleanup for logged events when refreshing schema and syncing tables. previously spurious logged events could lead to primary key violations when syncing single tables or refreshing single schemas.

2.0.7 - 19 May 2018

  • Fix for issue #71, make the multiprocess logging safe. Now each replica process logs in a separate file
  • Fix the --full option to store true instead of false. Previously the option had no effect.
  • Add auto_maintenance optional parameter to trigger a vacuum over the log tables after a specific timeout
  • Fix for issue #75, avoid the wrong conversion to string for None keys when cleaning up malformed rows during the init replica and replica process
  • Fix for issue #73, fix for wrong data type tokenisation when an alter table adds a column with options (e.g. ADD COLUMN foo DEFAULT NULL)
  • Fix wrong TRUNCATE TABLE tokenisation if the statement specifies the table with the schema.

2.0.6 - 29 April 2018

  • fix for issue #69 add source’s optional parameter on_error_read: to allow the read process to continue in case of connection issues with the source database (e.g. MySQL in maintenance)
  • remove the detach partition during the maintenance process as this proved to be a very fragile approach
  • add switch --full to run a VACUUM FULL during the maintenance
  • when running the maintentenance execute a VACUUM instead of a VACUUM FULL
  • fix for issue #68. fallback to binlog_row_image=FULL if the parameter is missing in mysql 5.5.
  • add cleanup for default value NOW() when adding a new column with ALTER TABLE
  • allow enable_replica to reset the source status in the case of a catalogue version mismatch

2.0.5 - 25 March 2018

  • fix wrong exclusion when running sync_tables with limit_tables set
  • add run_maintenance command to perform a VACUUM FULL on the source’s log tables
  • add stop_all_replicas command to stop all the running sources within the target postgresql database

2.0.4 - 04 March 2018

  • Fix regression added in 2.0.3 when handling MODIFY DDL
  • Improved handling of dropped columns during the replica

2.0.3 - 11 February 2018

  • fix regression added by commit 8c09ccb. when ALTER TABLE ADD COLUMN is in the form datatype DEFAULT (NOT) NULL the parser captures two words instead of one
  • Improve the speed of the cleanup on startup deleting only for the source’s log tables instead of the parent table
  • fix for issue #63. change the field i_binlog_position to bigint in order to avoid an integer overflow error when the binlog is largher than 2 GB.
  • change to psycopg2-binary in install_requires. This change will ensure the psycopg2 will install using the wheel package when available.
  • add upgrade_catalogue_v20 for minor schema upgrades

2.0.2 - 21 January 2018

  • Fix for issue #61, missing post replay cleanup for processed batches.
  • add private method _swap_enums to the class pg_engine which moves the enumerated types from the loading to the destination schema.

2.0.1 - 14 January 2018

  • Fix for issue #58. Improve the read replica performance by filtering the row images when limit_tables/skip_tables are set.
  • Make the read_replica_stream method private.
  • Fix read replica crash if in alter table a column was defined as character varying

2.0.0 - 01 January 2018

  • Add option --rollbar-level to set the maximum level for the messages to be sent to rollbar. Accepted values: “critical”, “error”, “warning”, “info”. The Default is “info”.
  • Add command enable_replica used to reset the replica status in case of error or unespected crash
  • Add script alias chameleon along with chameleon.py

2.0.0.rc1 - 24 December 2017

  • Fix for issue #52, When adding a unique key the table’s creation fails because of the NULLable field
  • Add check for the MySQL configuration when initialising or refreshing replicated entities
  • Add class rollbar_notifier for simpler message management
  • Add end of init_replica,refresh_schema,sync_tables notification to rollbar
  • Allow --tables disabled when syncing the tables to re synchronise all the tables excluded from the replica

2.0.0.beta1 - 10 December 2017

  • fix a race condition where an unrelated DDL can cause the collected binlog rows to be added several times to the log_table
  • fix regression in write ddl caused by the change of private method
  • fix wrong ddl parsing when a column definition is surrounded by parentheses e.g. ALTER TABLE foo ADD COLUMN(bar varchar(30));
  • error handling for wrong table names, wrong schema names, wrong source name and wrong commands
  • init_replica for source pgsql now can read from an hot standby but the copy is not consistent
  • init_replica for source pgsql adds “replicated tables” for better show_status display
  • check if the source is registered when running commands that require a source name

2.0.0.alpha3 - 03 December 2017

  • Remove limit_tables from binlogreader initialisation, as we can read from multiple schemas we should only exclude the tables not limit
  • Fix wrong formatting for default value when altering a field
  • Add upgrade procedure from version 1.8.2 to 2.0
  • Improve error logging and table exclusion in replay function
  • Add stack trace capture to the rollbar and log message when one of the replica daemon crash
  • Add on_error_replay to set whether the replay process should skip the tables or exit on error
  • Add init_replica support for source type pgsql (EXPERIMENTAL)

2.0.0.alpha2 - 18 November 2017

  • Fix wrong position when determining the destination schema in read_replica_stream
  • Fix wrong log position stored in the source’s high watermark
  • Fix wrong table inclusion/exclusion in read_replica_steam
  • Add source parameter replay_max_rows to set the amount of rows to replay. Previously the value was set by replica_batch_size
  • Fix crash when an alter table affected a table not replicated
  • Fixed issue with alter table during the drop/set default for the column (thanks to psycopg2’s sql.Identifier)
  • add type display to source status
  • Add fix for issue #33 cleanup NUL markers from the rows before trying to insert them in PostgreSQL
  • Fix broken save_discarded_row
  • Add more detail to show_status when specifying the source with –source
  • Changed some methods to private
  • ensure the match for the alter table’s commands are enclosed by word boundaries
  • add if exists when trying to drop the table in swap tables. previously adding a new table failed because the table wasn’t there
  • fix wrong drop enum type when adding a new field
  • add log error for storing the errors generated during the replay
  • add not functional class pgsql_source for source type pgsql
  • allow type_override to be empty
  • add show_status command for displaying the log error entries
  • add separate logs for per source
  • change log line formatting inspired by the super clean look in pgbackrest (thanks you guys)

2.0.0.alpha1 - 11 November 2017

  • Python 3 only development
  • Add support for reading from multiple MySQL schemas and restore them it into a target PostgreSQL database. The source and target schema names can be different.
  • Conservative approach to the replica. Tables which generate errors are automatically excluded from the replica.
  • Daemonised init_replica process.
  • Daemonised replica process with two separated subprocess, one for the read and one for the replay.
  • Soft replica initialisation. The tables are locked when needed and stored with their log coordinates. The replica damon will put the database in a consistent status gradually.
  • Rollbar integration for a simpler error detection.