Month: May 2015

MySQL + Galera + Xtrabackup (Errors and Fixes)

MySQL 5.6.21
Galera 25.3.5
Percona-xtrabackup-21 2.1.9-746-1.precise

Got error: 1047

Fix:
Database is probably in ‘initialized’ state. need to restart the service and check configuration.

 

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20150325 09:17:28.755)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150325 09:17:28.756)
WSREP_SST: [INFO] Removing the sst_in_progress file (20150325 09:17:28.759)
2015-03-25 09:17:28 6459 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address 'ip_address' --auth 'user:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '6459' --binlog 'binlog' : 32 (Broken pipe)
2015-03-25 09:17:28 6459 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-03-25 09:17:28 6459 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-03-25 09:17:28 6459 [ERROR] Aborting
2015-03-25 09:17:28 6459 [Warning] WSREP:State transfer failed: -2 (No such file or directory)
2015-03-25 09:17:28 6459 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.

Fix:
Check innodb_data_file_path=ibdata1:256M:autoextend does not exist in my.cnf
Check that xtrabackup isn’t still running orphaned on either node “ps aux | grep mysql” on donor
Check apt-get install persona-xtrabackup-21 is installed (All servers should be running the same version)
Check ssh keys are correct and servers can ssh freely between each other
Make sure sst_in_progress file does not exist on the joining server
Check that the node joining has an /etc/hosts file entry for the donor (or DNS is good)
Check the wsrep_sst_method on the donor at runtime is xtrabackup-v2
Make sure the root:password in /etc/mysql/my.cnf is the same as what the localhost is or xtrabackup can’t use it and will bomb

 

[ERROR] Unknown/unsupported storage engine: InnoDB 2015-03-25 14:33:48 31707 [ERROR] Aborting

Fix:
ibdata and log files are probably not the same size as the ones on the donor server. If innodb_data_file_path is set, this could be causing the problem.

 

 Other interesting facts:

Syncing with xtrabackup:
– creating or dropping a database during sync will cause the node syncing to the cluster to drop

envconsul and Docker ….. soo long config files

As Docker continues to grow in popularity there are quite a few things that become readily apparent. Fortunately I’m only going to address one of them. Enter envconsul to retrieve application config data at run time.

This post assumes you already have a running Consul server with some data you wish to retrieve.

envconsul was written by Hashicorp, a great company that I personally respect. In everything I’ve touched made by this great little company, I’ve yet to be disappointed. Their applications are rock solid.

Github Link:

https://github.com/hashicorp/envconsul

 

envconsul utilizes a key/value stored called Consul to retrieve configuration data and present them as environment variables to the application at runtime. This concept offers up a lot of opportunity around dynamic configuration, centralized configuration management and security because there aren’t free text usernames and passwords hanging around the file system. Not that any respectable company would ever do that right? No way. Never. Ok maybe it kinda happens almost always. With envconsul, we can solve that.

 

Build envconsul:

Currently there is no package in the general package managers for envconsul so I like to pull the repo, make the binary and copy it into /usr/bin which places the binary in the path and makes it immediately executable.

git clone https://github.com/hashicorp/envconsul.git
cd envconsul
make

If you decide you like envconsul, bake it right into your vm or container and you’ll always have it available.

 

Create a envconsul.cnf file:

Basically this file tells envconsul where the consul server exists.

consul = "consul.mydomain.com:8500"
timeout = "5s"

 

Add it to your Dockerfile:

I mentioned this had to do with Docker right? Well in the Dockerfile when you build your images you can bake envconsul right into to run command with something like the following:

CMD /usr/sbin/apache2ctl -k start && envconsul -config=/etc/envconsul.cnf -sanitize=false -upcase=false myblog env /usr/local/tomcat/bin/catalina.sh run

Let’s imagine I have an Tomcat container with Apache Web Server running in front of it. In the command above I’m starting apache and then executing envconsul to call the consul server.
So what have I really done here?

I’ve set sanitize to false otherwise envconsul will replace “invalid” characters as underscores
I’ve referenced the envconsul.cnf with -config
I’ve set upcase to false cause being a Linux nut, I know some devs like to ingest environment variables that aren’t just uppercase
I’ve specified the key myblog to get data back from consul
I’ve added env so envconsul presents the results from consul as environment variables to catalina.sh

 

One thing I love about envconsul is when it provides the environment variables to the application, it is ONLY to the application. Logging in as root and running printenv won’t even provide the variables envconsul presents to the application.

 

This has been a very basic “get it up and running” scenario around envconsul. There are other things to explore like ssl, authentication and consul API Tokens so head over to the Github page dig in.

 

And if you have found this valuable, Tweet it please.