Product: Stratusphere UX - Hub, Collectors
Product Version: 6.5.1
Expires on: 365 days from publish date
Updated: March 14, 2022
Summary
In 6.5.1, Collectors and hub are set to process a maximum of 1024 report files at a time. If pending DB inserts start piling up on a hub or collector, the lwl-smx service may run out of memory. /opt/tnt/var/ filesystem approaching or going over 70% is a good indicator of this issue. You will see something like this in /var/log/messages if lwl-smx runs out of memory:
Oct 20 14:50:56 Collector01 start-smx[2121]: java.lang.OutOfMemoryError: Java heap space
Oct 20 14:50:56 Collector01 start-smx[2121]: Dumping heap to "/opt/tnt/var/backend/dumps/lwl-smx.hprof" ...
Oct 20 14:50:56 Collector01 start-smx[2121]: Unable to create "/opt/tnt/var/backend/dumps/lwl-smx.hprof": No such file or directory
Oct 20 14:50:56 Collector01 start-smx[2121]: Terminating due to java.lang.OutOfMemoryError: Java heap space
Oct 20 14:50:57 Collector01 systemd[1]: lwl-smx.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 20 14:50:57 Collector01 systemd[1]: lwl-smx.service: Failed with result 'exit-code'.
To check max inflight value use this curl command as root, it should output ""max_inflight": 1024," before making a change:
curl 'localhost:8222/streaming/channelsz?channel=processed-inspection-reports&subs=1' 2>&1 |grep inflight
Possible Resolution:
Upload the attached jar file to /home/friend/ on the hub and collectors, then log in as friend run the following as root to backup/replace the jar and restart lwl-smx. For AWS appliances, replace friend with ec2-user. For Azure appliances, replace friend with azureuser or whatever username was set for the appliance:
cp /opt/lwl/servicemix/deploy/lwl-inspection-data-processor-copy-0.229.jar /opt/lwl/servicemix/deploy/lwl-inspection-data-processor-copy-0.229.jar.BACKUP
cp /home/friend/lwl-inspection-data-processor-copy-0.234-SNAPSHOT.jar /opt/lwl/servicemix/deploy/lwl-inspection-data-processor-copy-0.229.jar
systemctl restart lwl-smx
Check the max inflight value again, it should be 4 for a hub or 2 for a collector:
curl 'localhost:8222/streaming/channelsz?channel=processed-inspection-reports&subs=1' 2>&1 |grep inflight
Lastly, there may have crash files from when lwl-smx ran out of memory. To check for and remove them:
ls -lrt /var/crash/*
rm -rf /var/crash/*