Stratusphere Collector Does Not Register - Error "Key material cert.txt not found" – Liquidware Customer Support

Symptoms

When registering a new collector the status column in the Collector Administration page displays "NO" on a red background.
Last Audit Receipt is blank
Edit Properties page shows an error message "Key material cert.txt not found'

6.7 Revision

Collector unable to register with the HUB receiving a 503 error in the lwl console upon first setup.
HUB does not contain a cert.txt and is unable to generate one.

Guide

Summary

Cert.txt file that is missing, is the file responsible form creating a secured connection to the HUB.

No matter the version of the collector, the registration depends on availability of that cert.txt file on the HUB.

To troubleshoot, you may identify if the cert.txt file exists on the hub, why it's missing or if the collector has the means to communicate with the HUB to acquire it?

Logging on to Hub/DB or Collector and helpful commands.

Putty on to HUB/Collector as friend and then su to root.
You will be using the following commands to troubleshoot
- df -h
- ls -ltr /opt/vdesktools/grd/*.txt
- cat /opt/vdesktools/grd/imgrcomm.txt
- ls -ltr /opt/tnt/var/backend/activation.crt
  - cat /opt/tnt/var/backend/activation.crt
- /etc/init.d/vs-helper restart (stop & start)
- touch /opt/vdesktools/grd/cert.txt

Diagnosing certificate file availability

On the HUB check the following files exist and have a most recent date.
- ls -ltr /opt/tnt/var/backend/activation.crt
- ls -ltr /opt/vdesktools/grd/*.txt
Confirm the cert.txt file has the most recent date in above queries or if it exists? If not, generate a new one on the collector.
- /etc/init.d/vs-helper restart
  - This should update the time stamp on the cert.txt file or generate a new one.
- If the file fails to update the time stamp or fails to generate if missing, you will need to check the following
  - Is there an error in imgrcomm.txt?
    - cat /opt/vdesktools/grd/imgrcomm.txt
  - Can the collector resolve the hostname (CID Key Callback DNS) in mgrcert.pem?
    - head /opt/vdesktools/grd/key_material/public/mgrcert.pem
    - getent hosts <hubDNS>
  - Is the database full? Run this command on the appliance with the database on it:
    - df -h | grep /var/lib/pgsql
      - If full expand the disk 3 & 4 accordingly in vsphere and restart the appliance.
    - On the HUB restart vs-helper.
  - Is the cert.txt file stale? Run this command to refresh cert.txt file & restart vs helper and rejoin or re-register the collector in the WebUI.
    - touch /opt/vdesktools/grd/cert.txt
    - /etc/init.d/vs-helper restart
    - cat /opt/vdesktools/grd/imgrcomm.txt
Is the activation.crt current? Check the address in the activation.crt file hash. Due to admin configuration error, DNS, IP or CID Callback address entry. This file can be written with the wrong address therefore not allowing Collectors to register correctly.
- cat /opt/tnt/var/backend/activation.crt
  - Make sure the address is the host name of the hub or the IP address.
  - If not, then in the Stratusphere webui or lwl menu on the HUB change the cid key callback address to the HUB hostname or IP address and the restart the vs-helper service. Then re-register the collectors
Lastly, can the collector communicate to the HUB?
- Ping the HUB from the collector
- Ping the Collector from the HUB
- Check the Network configuration using lwl menu or ssconsole user on the Collector
  - Are all the DNS servers similar as they are for the HUB and DB
  - Is the Gateway the same?
  - Are the VMs on the same network in Vsphere?
  - Are the IP and HOSTNAMES defined in DNS and and DHCP?

6.7+ Additional Diagnose steps for Trobleshooting

HUB fails to generate cert.txt file after deployment.
- Collector unable to register with the HUB receiving a 503 error in the lwl console upon first setup.
- Collector receives a 503 error communicating to the HUB.
  - The error is written to the lwl-backend.log on the HUB or Collector imgrcomm.txt file.
- HUB did not generate its own cert.txt out of the box, failed on lwl-backend starting
  - Error found in the lwl-backend.log
- NATS queue consumer issue, error creating the natsInspectionReportMessageListener
  - Fixed by rebuilding the consumer on the listener side on the hub and restarting lwl-bckend
  - ```
  /opt/lwl/nats-server/bin/client/nats -s localhost:4223 consumer rm -f cid-reports NatsInspectionReportMessageListener > /dev/null 2>&1 && systemctl restart lwl-backend
```
- Once the lwl-backend service restarts, cert.txt should be in the proper place.
- Continue with registering the collector via the WEBUI
HUB appliance Collector under Collector Administration receives cert.txt not found
.
- HUB appliance imgrcomm.txt file displays "Unable to unwrap (1402)" in its imgrcomm.txt file.
  - It is a cipher mismatch on the certificate file (This can occur during a migration of appliances with separate database appliances)
  - It is supposed to be regenerated and fails.
- Resolve the issue by querying the new Database appliance for the SID .
  - ```
  select sid from tnt_config.node nde join tnt_config.policynode pn on (nde.id = pn.id and pn.type =3 and pn.system_object is true and pn.deletE_time is null);
```
- Regenerate the SID on the HUB by running the following command, replace $SID with the DB value:
  - ```
  /opt/tnt/bin/showactcert true $SID > /opt/vdesktools/grd/key_material/public/mgrcert.pem
```
- Check for the cert.txt or reboot.
- Steps to regenerate the consumer for the NATS queue may be required
  - Rebuild the consumer on the listener side on the hub and restarting lwl-bckend
  - ```
  /opt/lwl/nats-server/bin/client/nats -s localhost:4223 consumer rm -f cid-reports NatsInspectionReportMessageListener > /dev/null 2>&1 && systemctl restart lwl-backend
```
- Once the lwl-backend service restarts, cert.txt should be in the proper place.
- Continue with registering the collector via the WEBUI

Related articles