Adding an On-Premise HA Search Server

There are two basic steps to adding (or removing) a search node in an already existing HA search environment: introducing the new search node and then configuring the ingress replicators to recognize the new node. The following example assumes you have two search nodes and you will add a third.

Adding a New Search Node to the Configuration

Here's how to add a new search node to your configuration:

  1. Take all ingress replicators out of the load balancer rotation.
  2. Wait for all ingress replicators to deliver pending activities. To do this, from the command line run: curl http://ir0x.yourdomain.com:29000/logging/metric/listCounters | grep InQueue
  3. This should return something like the following, which indicates that you are replicating to search01.yourdomain.com:30000 and search02.yourdomain.com:30000, and that the InQueue metric is 0.0, which means that all activities have been delivered:
    {"key":"counter>com.jivesoftware.service.activity.stream.replicator.ActivityStreamReplicator>search01.yourdomain.com_30000>InQueue","metric":0.0},
    {"key":"counter>com.jivesoftware.service.activity.stream.replicator.ActivityStreamReplicator>search02.yourdomain.com_30000>InQueue","metric":0.0}
  4. Shut down the ingress replicators (service jive-ingress-replicator-service stop).
  5. Install the third search service 3 (search03.yourdomain.com). For instructions, see Installing One or More Search Servers.
  6. Copy the active search index from search service 1 or 2 to the new search service 3. You can find the active search index by looking in /usr/local/jive/services/search-service/main-args.properties file for CONTENT_SEARCH_HOME_DIRECTORY=var/data/contentSearch/. This property lists the location of the search indexes.
  7. Start search service 3 (service jive-search-service start).
  8. Add the third search node to your load balancer.
  9. Next, you'll point the ingress replicators at the new search node as described below.

Diagram of HA On-prem Search: How to add or remove a Search node

Point to the New Search Node

After you've successfully added the new node as described above, you'll need the other nodes to point to the new search service node. Here's how to do that:

  1. In /usr/local/jive/services/ingress-replicator-service/, re-configure ingress replicator 1 and 2 to include search service 3. Set the load balancer to forward the ingress service to all machines holding the ingress replicator, and the ingress replicators to forward to all machines where the search service runs. The relevant parameters in the main-args.properties file should look something like this:
    PORT=29000
    REPLICATE_INGRESS_TO_THESE_COMMA_SEPARATED_HOST_COLON_PORT_TUPLES=search01.yourdomain.com:30000,search02.yourdomain.com:30000,search03.yourdomain.com:30000
  2. Restart the ingress replicator services (service jive-ingress-replicator-service restart).
  3. Add the ingress replicator services back into the load balancer configuration.

Diagram of HA On-prem Search: How to configure the newly added Search node