Views:

Applies To:

GroupID 10 only

Business Case:

By default, Search Guard is installed with the default certificates that were provided to Imanami by its admins at the time of GroupID 10 release. As the certificates carry a date of expiry, you might face a situation where the Search Guard certificates may be expired. Consequently, replication will behave erratically and you may get the “Search Guard not initialized” error when you try to connect with Elasticsearch using its URL. Moreover, in case of multiple GroupID servers, the cluster may break, with every server assuming the role of a primary server.

This can also happen when you install a new instance of GroupID 10 and replication is not working or if you run the GroupID Configuration Tool in a running environment and encounter replication issues afterwards.

Symptoms:

When confronted with any of the above-mentioned issues, look for the following errors in the Elasticsearch log file, GroupIDCluster10.log:

  • [ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node-10-HQDVGRPID103] SSL Problem Received fatal alert: certificate_unknown
  • Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun May 16 00:07:04 PDT 2021
  • ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node-10-HQDVGRPID103] SSL Problem General SSLEngine problem javax.net.ssl.SSLHandshakeException: General SSLEngine problem

Another possible error is:

[2021-05-18T11:45:42,221][ERROR][c.f.s.a.BackendRegistry  ] Not yet initialized (you may need to run sgadmin)
[2021-05-18T11:45:44,107][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node-10-OMAPI-ADAPP04] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

The log file may contain all errors or any one of these. In any case, this indicates that the Search Guard certificates have expired and you need to renew them.

Solution:

Follow these steps to renew the Search Guard certificates.

  1. Download and extract the attached file content and copy files to:
    [GroupID installation folder]\ElasticSearch\elasticsearch-6.2.4\config\

    The default installation folder is:
    C:\Program Files\Imanami|GroupID 10.0

    (The zip file contains three certificates and two key files.)
  2. Remove these configurations from the ElasticSearch yml file:

    searchguard.ssl.transport.keystore_filepath: CN=imanami-keystore.jks
    searchguard.ssl.transport.keystore_password: f72d717f3d965e62b1a5
    searchguard.ssl.transport.truststore_filepath: truststore.jks
    searchguard.ssl.transport.truststore_password: a2d13de70437c76d1c5
  3. Add the following lines at the same location you removed the lines from:

    searchguard.ssl.transport.pemcert_filepath: imanami.pem
    searchguard.ssl.transport.pemkey_filepath: imanami.key
    searchguard.ssl.transport.pemkey_password: aGsCBjIeXYfy
    searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
    searchguard.nodes_dn:
    - CN=imanami

  4. Remove these files from the location:
    [GroupID installation folder]\ElasticSearch\elasticsearch-6.2.4\config\

    • CN=imanami-keystore.jks
    • CN=sgadmin-keystore.jks
    • truststore.jks
       
  5. Restart the Elasticsearch service from Windows services.
     
    Note: For multiple GroupID servers in a master-slave configuration, carry out the above steps on all the GroupID servers before you proceed further.

  6. Check your Elasticsearch cluster health and make sure it is green.

    To view the health status, run the following command in your browser (preferably Chrome) on the master server:
    http://[machinename]:9200/_cluster/health
    If prompted for username and password, enter admin in both the fields.

    If you run the command in IE, it will prompt you to download a JSON file. Save it and open with Notepad or Notepad++. 

    Cluster health is displayed as:


     
  7. Next, replace foldername, cluster name, and hostname in the following command.
     
    "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\plugins\search-guard-6\tools\sgadmin.bat" -cd "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\plugins\search-guard-6\sgconfig" -cacert "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\config\root-ca.pem" -cert "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\config\sgadmin.pem" -key "C:\Program Files\Imanami\GroupID 10.0\ElasticSearch\elasticsearch-6.2.4\config\sgadmin.key" -keypass bmEWfgPU2acY -nhnv -h GID10SR1 -cn GroupidCluster10 -p 9305

    Note that the foldername points to the default GroupID installation path. Change it if you have a different path.
    You can get the hostname and cluster name from the elasticsearch.yml file.


  8. Launch command prompt as an admin user and execute the above command. 
    On execution, the following sequence should occur:



    Check the cluster log file and the error(s) should be gone.
     
    Note: For multiple GroupID servers in a master-slave configuration, run the command on all the GroupID servers, starting with the primary server and then the secondary servers.

  9. There may be a case where the following occurs instead of the above sequence:



    It implies that the command has not been executed because an Elasticsearch index (most probably the Search Guard index) is locked.

    Next, you need to identify the locked index. Use the following command for each index to check if it is locked:
    http://[machine-name]:9200/domainname_index/_settings?pretty
    • For a locked index, it returns the following:
      "read_only_allow_delete" : True
    • If this entry is not present, it indicates that the index is not locked.

    To unlock the Elasticsearch indices, follow this article.

  10. When done, check the cluster log file again and the error(s) should be gone.
     
    Note: In case you run into problems, contact support@imanami.com.

Related Attachments (1)

Comments (0)