Q&A
Thread error
Vaughan Data Systems
1-4-22
Operating system: Windows
We recently had this error in the log followed by the Hansa server crashing.

Does anyone know what this could mean?

/THREAD(31) out of file-descriptors
Zilvinas Suliauskas
1-4-22
Hello,

What version of Standard ERP your customer is using?
Reply
Lee D Papworth
1-6-22
Created byZilvinas Suliauskas16:30 4 Jan 2022
Hello, What version of Standard ERP your customer is using?
Hi Zilvinas,

SERP 8.5 2021-04-18 Build 85360453

Running on Centos Linux release 7.9.2009

Not had this issue in the previous version of SERP.

We have recently upgrade customer to this build


Thanks

Lee

Reply
Zilvinas Suliauskas
1-6-22
Hi Lee,

I spoke to one of our programmers. It could be version specific issue. Could you please test if same problem appears in the latest released version as well.
Reply
Lee D Papworth
1-6-22
Created byZilvinas Suliauskas14:38 6 Jan 2022
Hi Lee, I spoke to one of our programmers. It could be version specific issue. Could you please test if same problem appears in the latest released version as well.
Hi Zilvinas,

I am afraid that will be quite difficult to prove. You see we tested SERP with the customer before the upgrade and this issue did not arise. I suspect it has now come to light because in the live environment hundreds more transactions are being processed that cannot be replicated in a test setup.

Can any techie please review this article and see if it makes sense?

https://docs.pingidentity.com/bundle/pingdirectory-73/page/ygy1564011416825-1.html

I am wondering if it can be solved by changing some Linux settings


Thanks

Lee
Reply
Lee D Papworth
1-14-22
Created byLee D Papworth15:35 6 Jan 2022
Hi Zilvinas, I am afraid that will be quite difficult to prove. You see we tested SERP with the customer before the upgrade and this issue did not arise. I suspect it has now come to light because in the live environment hundreds more transactions are be
Hi Zilvinas, any feedback from the techies?

Thanks

Lee
Reply
Nejc R.
1-17-22
Hello,

Try updating to a later version 8.5 2021-04-18 is pretty old and its likely the issue is resolved in newer versions.


what version are you updating from?


are you running any halcust?
Reply
Vaughan Data Systems
1-19-22
Hi Nejc,

They were on 85330538 and are now on 85360453

All of our other customers are on this release and are working fine.

The reason we settled on this release is because of issues with later versions of Standard ERP that I believe still haven't been resolved yet.

Are you able to confirm this was a known issue that has been fixed in newer builds now?

The customer who is getting these crashes is a huge company with lots of files in their halcust, so updating them takes hours of testing and taking time out of their schedule.

Do you know what this error actually means? And is there anything we can try before attempting an update? E.g. changing thread settings or server settings etc
Reply
Nejc R.
1-20-22
Hello,

So this was just a build upgrade and the version did not actually change?

If the halcust is not closing files correctly the system can run out of file descriptors.


File descriptors are pointers/handles to opened files in Linux, whenever you want to read/write to a file you first need to open it and then close it when you are done. The amount of files you can have open at once is limited (usually a pretty high number). But if you open a file and do not close it, it will just remain open. Do that enough times and you will run out of file descriptors and the program will crash.
Reply
Vaughan Data Systems
1-20-22
Hi Nejc,

No, we went from 2020-01-07 to 2021-04-18

Do you have anything we can try/look for in order to resolve this?
Reply
Nejc R.
1-21-22
If there are any OpenFile("filename"); that do not CloseFile; after that definitely can cause it.


Has this error occurred again since then. And is there anything specific being done on the system when it crashes?
Reply
Lee D Papworth
1-24-22
Created byNejc R.16:24 21 Jan 2022
If there are any OpenFile("filename"); that do not CloseFile; after that definitely can cause it. Has this error occurred again since then. And is there anything specific being done on the system when it crashes?
Hi Nejc,

there are no OpenFile instances.

We use xdata = ParseXMLFile(filename);

So I am assuming ParseXMLFile actually opens the file for reading and should also terminate with a CloseFile?

Can you come back to me on that please?


Nothing out of the ordinary is running on the system when this crashes. Our import routines have been working without any issues in the last two versions of SERP. It is all happening since the upgrade

Thanks

Lee
Reply
Aldevinas Katkus
1-25-22
Try to run on Linux command prompt after you files have been imported:
ps -aux | grep StandardERPServer64

then
lsof -p serp_pid

where serp_pid is PID of the StandardERPServer64 (second column of the result of the previous command)

Or try


lsof -p serp_pid +D 'path_to_your_xml_files_directory'
Reply
Lee D Papworth
1-25-22
Hi Aldevinas,

Can you please tell me what I am looking for when I run those commands?

Thanks

Lee
Reply
Lee D Papworth
1-27-22
It appears that it might be to do with web functions.

When running the command lsof -p serp_pid there are literally thousands of open files

/home/hansa/StandardERPServer/webcust/styles/HHIStyle24.css
/home/hansa/StandardERPServer/webcust/js/HHIMainapp24.js
/home/hansa/StandardERPServer/webcust/sdlogo.png


We have a set of bespoke web functions that we use for bar code scanning and put away in the Warehouse. These functions have been running perfectly for the last couple of years but since this new release we hit the out of file descriptors snag.

Could it be the web engine is not terminating these files when fresh pages are loaded?

Please please get to me asap on this. Happy to provide any more info.

My only action at the moment is to stop and restart SERP every night for the customer which can only be a short time solution

Many thanks in advance

Lee Papworth

Reply
Aldevinas Katkus
1-27-22
I have tested webshop and our custom code which refreshes the page every minute with v. 2021-11-24 and didn't notice anything wrong, lsof did not show any increase in open files.
Reply
Lee D Papworth
3-4-22
Hi,

I am really beyond desperate with this now. I am having to stop and restart SERP every evening just so my customer does not get this error/crash during the day.

So just to recap. I have basic web functions to assist my customer with picking and dispatch. I have a stlyesheet, a small javascript file and a company logo. They have a webshop licence but I have written these routines using global webpublic hal functions. Not had any issues with these functions in the last 2 years. Only since we upgraded in December 2021. Have similar functions at other customer sites that do not get this problem.

When I run this command "lsof -p serp_pid" after a restart of SERP I immediately can see a growing number of these entries when they start to use the web functions

/home/hansa/StandardERPServer/webcust/styles/HHIStyle24.css
/home/hansa/StandardERPServer/webcust/js/HHIMainapp24.js
/home/hansa/StandardERPServer/webcust/sdlogo.png

This list of open files just continues to grow and grow and I believe this then causes the "out of file-descriptors" error and crashes SERP

Any ideas please?

Thanks

Lee







Reply
Aldevinas Katkus
3-10-22
Do you see the same files are listed multiple times in the lsof output?
Some crazy ideas:

Compare system>>settings>>optional features on the crashing site and other sites.
Are your clients using the same web browser or different ones?
Reply
Aldevinas Katkus
3-10-22
Created byAldevinas Katkus15:37 10 Mar 2022
Do you see the same files are listed multiple times in the lsof output? Some crazy ideas: Compare system>>settings>>optional features on the crashing site and other sites. Are your clients using the same web browser or different ones?
try to use AddFileToArea then ParseXMLArea instead.
Reply
Lee D Papworth
3-10-22
Hi Aldevinas,

yes the three files I listed previously are constantly repeated when I look at the output of lsof. I have other links in my html to google fonts and a jquery and they do not show in lsof. These are external links. It just seems to be the files I am serving from the webcust folder.

I have compared optional features with other customers. All the same except this server that has the issue is a very high spec so we have more report slots and HTTP threads specified

Lee

Reply
Jennifer
4-4-22
Created byLee D Papworth15:42 10 Mar 2022
Hi Aldevinas, yes the three files I listed previously are constantly repeated when I look at the output of lsof. I have other links in my html to google fonts and a jquery and they do not show in lsof. These are external links. It just seems to b
Hi Lee,

Did you manage to get to the bottom of this?

Jen
Reply
Lee D Papworth
4-4-22
Created byJennifer11:15 4 Apr 2022
Hi Lee, Did you manage to get to the bottom of this? Jen
Hi Jen,

unfortunately not :(

I had to embed the css and javascript into the pages rather than using an external link.


Not ideal but at least it is now not crashing SERP

Lee
Reply
Leave Comment
You can subscribe to notifications for this post by selecting the 'star' icon on the top right corner of the post.
Back to the list
Latest Posts
Re: Re: Periodic Customer Statement
Hello, Kindly please report the bug so that we can further investigate and resolve it. Thank you. Best regards, Benjamin...
11:31 3 May 2024
Re: Client Hansa.HDB
Hi, No changes should have been made that would increase client's HDB file. Please try to delete HANSA.HDB from client folder and reconnect to server and check if it will grow to 30MB again. ...
10:14 30 Apr 2024