Howto Configure High Availability Prelude Central Services

This is an example configuration utilizing two machines to provide a high availability pair for the central Prelude services, which include: Prelude-Manager Prelude-Correlator, Prewikka, and MySQL.

Table of Contents

Heartbeat will control the failing over of hard failures, such as a machine going down or total loss of connectivity. Both servers will maintain an up-to-date set of databases, and either can take over at a moments notice as the primary. You will need to use some form of monitoring, such as Nagios to handle service failures - thus alerting you when a manual failover is required (such as if a particular service dies, but the hardware remains operational and reachable).

Things that are assumed:
  • You will need at least two ethernet interfaces per server (and two servers).
  • Assumes you have eth0 configured with IP/hostnames on both boxes. This example uses preludecentral_1 and preludecentral_2, and a VIP (virtual IP) with associated hostname, such as preludecentral. Make sure you also add the VIP/hostname to DNS/host/etc., as this is what you expose to your clients/relays/agents/etc.
  • This document assumes high availability for fault tolerance, not for performance. Although you could stagger which services are offered where, etc.

MySQL Multi-Master Replication Configuration

You must have installed MySQL v5.x or above. Some features to avoid collisions in multi-master replication are only available in MySQL v5.x

1. Setup secondary ethernet interfaces
  • Setup eth1 on each server to be an unused private network for use by your HA pair.
  • Connect with a crossover cable between them.

2. Make the following additions on both servers in /etc/my.cnf, under [mysqld] section:


      log-bin=<$HOSTNAME>-mysql-bin  #change hostname to be each machines hostname

      server-id={1,2}  #set one server for 1 and the other for 2

      auto_increment_offset={1,2}  #set one server for 1 and the other for 2

      # Timeouts and max connections have been increased to handle a client disconnect issue.

      # Various other settings can be adjusted to the specs of your machine, such as buffer sizes, log sizes, etc.

3. Change datadir in /etc/my.cnf to be on a partition of its own of acceptable size, edit /etc/fstab, and add "async,noatime" as options for the partition used for the MySQL database directory.

4. Run mysql on each server:
  • Enter at prompt: GRANT REPLICATION SLAVE on *.* TO some_replication_user IDENTIFIED by 'putreplicationpasswordhere';
5. Add to the end of the mysql binary line in the start section of /etc/init.d/mysqld
  • --relay-log=<$HOSTNAME>-relay-bin #make sure to specify the correct hostname

6. From each server's mysql command-line: show master status;

7. From each server's mysql command-line: change master to master_host='<other hosts eth1 private ip address>', master_user='<username setup earlier for replication>', master_password='<password used earlier for replication>', master_log_file='<output of 1st column from above step on other server', master_log_pos=<output of 2nd column from above step on other server>;

8. Restart mysqld on each of the servers

9. Configure a root mysql password from the mysql command-line (only needs to be done on one server):
  • SET password for root@localhost=password('putmysqlrootpasswordhere');

Prelude Central Components

1. You must install Libprelude LibpreludeDB Prelude-Manager Prelude-Correlator and Prewikka on both machines.

2. All of the central Prelude services should have the same configuration files on both servers (ie. prelude-manager, prelude-correlator, apache, etc.)

3. When importing the prelude and prewikka schemas to their respective databases, it only needs to be done on one of the servers, it will be replicated to the other automatically.

4. Copy over profiles (if you have a Prelude Central already setup) or register for all profiles needed: correlator, manager, prewikka. Do this on one server, then copy those profiles/directories under /usr/local/etc/prelude/profile/ of the other machine in the HA pair.

5. Edit /usr/local/etc/prelude/default/global.conf, make 'Node Name' the name of the VIPs hostname on both. Copy over to other HA server.

6. Your web server and MySQL should be set to start automatically on boot. Prelude Manager and Correlator should NOT be set to start automatically on boot, as Heartbeat will handle these two services.

Setting up the Heartbeat Pair

1. Heartbeat Configuration
  • Install heartbeat, and all packages necessary such as stonih, pils, etc. that are required by your package manager.
  • Configure heartbeat to start on boot.
  • Create /etc/ha.d/
      debugfile /var/log/ha-debug
      logfile /var/log/ha-log
      logfacility     local0
      keepalive 2
      auto_failback off
      deadtime 30
      bcast eth1 eth0
      node    preludecentral_1
      node    preludecentral_2
  • Create /etc/ha.d/haresources:
preludecentral_1 IPaddr:: prelude-manager prelude-correlator

Use your primary hostname, assumes is your VIP, and that you called your init scripts prelude-manager and prelude-correlator

  • Create /etc/ha.d/authkeys; make sure to chmod 600 it
      auth 1
      1 sha1 some_password

2. Set the Heartbeat service to start on boot automatically.

Service Monitoring

1. Setup SNMP monitoring with something such as Nagios
  • For the VIP hostname/IP
    • Ping
      • Apache
      • Prelude Manager
      • Prelude Correlator
      • Prewikka Ctl
      • TCP Port 4690
    • For each hostname/IP of the pair
      • Ping
      • Diskspace
      • MySQL
      • Heartbeat

2. Set the snmpd service to start on boot automatically.

Clients / Relays / Agents

Make sure all clients, agents, relays, etc. connect to your VIP hostname/IP.