How To: Renew the expired Search Guard certificates

Applies To:

GroupID 9 or above.

Business Case:

By default, Search Guard is installed with the default certificates that were provided to Imanami by its admins at the time of GroupID 10 release. As the certificates carry a date of expiry, you might face a situation where the Search Guard certificates may be expired. Consequently, replication will behave erratically and you may get the “Search Guard not initialized” error when you try to connect with Elasticsearch using its URL. Moreover, in case of multiple GroupID servers, the cluster may break, with every server assuming the role of a primary server.

This can also happen when you install a new instance of GroupID 10 and replication does not work or when you run the GroupID Configuration Tool in a running environment and encounter replication issues afterwards.

Symptoms:

When confronted with any of the above-mentioned issues, look for the following errors in the Elasticsearch log file, GroupIDCluster10.log:

[ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node-10-HQDVGRPID103] SSL Problem Received fatal alert: certificate_unknown Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun May 16 00:07:04 PDT 2021 ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node-10-HQDVGRPID103] SSL Problem General SSLEngine problem javax.net.ssl.SSLHandshakeException: General SSLEngine problem

Another possible error is:

[2021-05-18T11:45:42,221][ERROR][c.f.s.a.BackendRegistry  ] Not yet initialized (you may need to run sgadmin) [2021-05-18T11:45:44,107][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node-10-OMAPI-ADAPP04] SSL Problem Received fatal alert: certificate_unknown javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

The log file may contain all or any one of these errors. In any case, this indicates that the Search Guard certificates have expired and you need to renew them.

Solution:

The certificates can be updated via two different methods.

• By using the Certificate Deployment Utility (only valid for GroupID version 10).
• By manually replacing the Certificates (valid for GroupID version 9 or above).

Using the Certificate Deployment Utility:

Follow the steps provided below to use the Certificate Deployment utility to update the Search Guard certificates. 

1.  Download and extract the attached "CertificateDeployment.zip" file and copy files C-Drive root directory.
2.  Open the "CertificatesDeployment1.0.0.exe" as an administrator. A command prompt window will open and the utility will automatically update the certificates.

           4.  Once the utility has completed, press "Enter" to exit the command prompt window. 

Manually replacing and updating the Certificates:

Follow these steps to manually renew the Search Guard certificates.

1. Download and extract the attached file content and copy files to:
   [GroupID installation folder]\ElasticSearch\elasticsearch-6.2.4\config\
   
   The default installation folder is:
   C:\Program Files\Imanami|GroupID 10.0
   
   (The zip file contains three certificates and two key files.)

2. Remove these configurations from the elasticsearch.yml file:

   searchguard.ssl.transport.keystore_filepath: CN=imanami-keystore.jks searchguard.ssl.transport.keystore_password: f72d717f3d965e62b1a5 searchguard.ssl.transport.truststore_filepath: truststore.jks searchguard.ssl.transport.truststore_password: a2d13de70437c76d1c5

3. Add the following lines at the same location you removed the lines from:

   searchguard.ssl.transport.pemcert_filepath: imanami.pem searchguard.ssl.transport.pemkey_filepath: imanami.key searchguard.ssl.transport.pemkey_password: aGsCBjIeXYfy searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem searchguard.nodes_dn: - CN=imanami

   

4. Remove these files from the location:
   [GroupID installation folder]\ElasticSearch\elasticsearch-6.2.4\config\

   • CN=imanami-keystore.jks
   • CN=sgadmin-keystore.jks
   • truststore.jks
      
5. Restart the Elasticsearch service from Windows services.
    

   Note:

   For multiple GroupID servers in a master-slave configuration, carry out the above steps on all the GroupID servers before you proceed further.

6. Check your Elasticsearch cluster health and make sure it is green.
   
   To view the health status, run the following command in your browser (preferably Chrome) on the master server:
   http://[machinename]:9200/_cluster/health
   If prompted for username and password, enter admin in both the fields.
   
   If you run the command in IE, it will prompt you to download a JSON file. Save it and open with Notepad or Notepad++. 
   
   Cluster health is displayed as:
   
   
    
7. Next, replace foldername, cluster name, and hostname in the following command.
    

   "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\plugins\search-guard-6\tools\sgadmin.bat" -cd "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\plugins\search-guard-6\sgconfig" -cacert "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\config\root-ca.pem" -cert "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\config\sgadmin.pem" -key "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\config\sgadmin.key" -keypass bmEWfgPU2acY -nhnv -h GID10SR1 -cn GroupidCluster10 -p 9305

   
   Note that the foldername points to the default GroupID installation path. Change it if you have a different path.
   You can get the hostname and cluster name from the elasticsearch.yml file.
   
   
8. Launch command prompt as an admin user and execute the above command. 
   On execution, the following sequence should occur:
   
   
   
   Check the cluster log file and the error(s) should be gone.
    

   Note:

   For multiple GroupID servers in a master-slave configuration, run the command on all the GroupID servers, starting with the primary server and then the secondary servers.

There may be a case where the following occurs instead of the above sequence:



It implies that the command has not been executed because an Elasticsearch index (most probably the Search Guard index) is locked.

Next, you need to identify the locked index. Use the following command for each index to check if it is locked:

http://[machine-name]:9200/domainname_index/_settings?pretty
To unlock the Elasticsearch indices, follow this article.

• For a locked index, it returns the following:
  "read_only_allow_delete" : True
• If this entry is not present, it indicates that the index is not locked.

When done, check the cluster log file again and the error(s) should be gone.