How to resolve the error: The remote server returned an error: (403) Forbidden. Call: Status code 403 from: POST /extest.lab_identity_1/identity/extest.lab/_update?retry_on_conflict=15. ServerError: Type: cluster_block_exception Reason: "block“ during rep

Applies To:

GroupID 9 and 10

Symptoms:

Based on observation from different environments, this type of issue occurs when you have low disk space on the drive where GroupID is installed and other hardware resources are running low. Or for some reason, available resources are being used up by other processes and services. As a result, the Elasticsearch service stops working and starts behaving abnormally. The Elastic indices get locked.

Affected Modules:

This error can occur in GroupID Automate and Self-Service while performing different functions, like group creation, advanced search, etc. Following is an example of the error while creating a group in Automate:

Solution:

We can unlock the Elastic indices using the Elasticsearch utility. Once these indices are unlocked, replication starts working again.

Steps to resolve the issue:

1. Check the replication logs to confirm that the error is the same as stated above; then proceed further.

2. Stop the GroupID Replication service.

3. Download the Elasticsearch head (which is an extension of Google Chrome) from the following link:
   https://mobz.github.io/elasticsearch-head/
   
   As the extension has been removed the Chrome web store, you can download the Chrome extension file (.crx) at the end of the KB article or as an alternate you can use the Multi Elasticsearch Head extension, from the link below.

   Multi Elasticsearch Head - Chrome Web Store (google.com)

4. Once downloaded and added in the browser, open Elasticsearch using its URL (http://localhost:9200/). Use any browser other than IE for this purpose; else you will not be able to see the indices. You can look for this URL in the registry file under the following key:
   
   

5. Now connect the Elasticsearch head with this URL. Make sure it prompts for the user name and password. Enter ‘admin’ in both fields.

6. Once opened, navigate to the Indices tab and observe the indices. Make sure all your clusters are in yellow or green color.
   
   

7. Now copy the first index and navigate to any request. Paste the index in the search box and add ‘/_settings’ (e.g., dc1.local_computer_1/_settings). In the query box, type the following command line:

   {"index.blocks.read_only_allow_delete" : false }

   
   

8. Change the query input to PUT and click the Request button. On the right side of the page, you should see a confirmation.
   
   

9. Repeat steps 7 and 8 for every index under Indices. Leave the Searchguard index as it doesn’t need to be unlocked.

10. Once done, close the browser and restart the services.

11. Now restart the Replication service from GroupID Management Console (navigate to the Replication node; restart the Replication service, and click Replicate Now). Replication should complete without any issue.