critical issue with sybase ASE allocating memory

[suznCB]

Dears, kindly help in this critical issue,

we use Sybase ASE 15.7 for our production database, suddenly, for some users while they are executing statements on the database by our company's applications, they got the following message:

select error ct_cmd_alloc() user api layer external error the connection has been marked dead.

I've searched for this error and found maybe the reason is lack of adequate memory

P.S other users complain about slowing the applications that connected to database

please recommend me how to enhance some configuration parameters:

they are listed here with their current values:

max memory: 21000000

heap memory per user 24576 /* kindly note the max number of connections is 1000 , and about 750 user connected daily */

memory per worker process 1024

total logical memory 15586362

total physical memory 16109188

procedure cache size 393216

statement cache size 157184

total data cache size 28835650

p.s the windows server which host ASE has a 64 GB ram,

and during the working hours, memory doesn't exceed 34 GB as shown in windows task manager.

please advice.

Best Regards

[suznCB]

Dear Mark, Thanks for replying

1 - do all users of the application experience this problem? [wondering if the issue can be traced to a specific group of users]:

No, not all users experience this problem, also no specified users: every time a random users experienced the problem.

2 - do all users (of the application) on a particular network segment see this problem? [wondering if the issue could be network (segment) related]

It is not a network issue, I check with network admin, also some users who disconnected use WAN network (our users in the same building where is the data center use wan )

in addition, when some user disconnected from the database because of network issue, there will be a record in the ASE error log.

BUT for this problem, there are no related entries in the log.

3 - has this always been a problem with the application, and if not, what may have changed lately (eg, new version of the app was recently released, new set of users were added and said users are seeing problems, etc)?

no upgrade for applications, they have used for one and a half year.

4 - has anyone checked the application's logs to see if there are any details listed there? [wondering if this could be more of an application issue and not so much a database issue]

there is no application's log, we can run tracing on the application to export trace file for its behavior, but as the problem caused in randoms period we cannot run this trace cause it loads a lot of unusefull data.

5 - you mention some searches showing a possible issue with lack of adequate memory; can you provide links to some of your search results? [wondering about the context in which others have seen the error, and whether there's any resemblance to your environment]

http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.mainframeconnect_12.6.occprc/html...

I conducted that ct_cmd_alloc() fails so it returned the error message

I run sp_sysmon '00:00:30' , noclear, 'cache wizard' last night, at the same time of problem's time in a day before,

please find the attached file, but kindly know yesterday night we didn't experienced the problem, so I don't know if the attached file will be useful.

sp-sysmon1.txt

I will rerun it when the problem returns again.

Thanks in advance for your help

[Mark_A_Parsons]

I'm not a front-end/application developer so I'm not sure if that infocenter.sybase.com link is referring to a memory issue with the (data)server or the client/application.

------------

Where does the application run?

Is a copy of the application running on each user's PC, and each user connects directly to ASE? [user/PC -> ASE]

Or is the application running on a machine separate from the users (eg, an application/middleware server), and users connect to this application server, which in turn connects to ASE? [user/PC -> appserver -> ASE]

If the application is running on its own server ... wondering if there could be some intermittent memory (allocation) issues on the application server?

------------

I didn't see any sort of glaring issues in the sp_sysmon report.

[suznCB]

Dear Mark,

yes, a copy of the application running on each user's PC, and each user connects directly to ASE

[suznCB]

for this time until finding a solution, I'm thinking to restart an ASE service, not windows server .

is there any risk by doing this, I 'a afraid that the service will not restarted sommthly, is there any propability to that.

Regards

[Former Member]

Could you please provide below details.

(1) Select @@maxpagesize

(2) Could you please give output of sybase Configuration parameters "default network packet size" and "max network packet size"

[suznCB]

Dear, here are the results

@@maxpagesize = 8192

Parameter Name Default Memory Used Config Value Run Value Unit Type
default network packet size 2048 #6830 2048 2048 bytes static

Parameter Name Default Memory Used Config Value Run Value Unit Type
max network packet size 2048 0 4096 4096 bytes static

[former_member188958]

Any messages in the ASE errorlog when clients get this message?

Client-fatal errors often correlate with an error with a stack trace in the ASE errorlog.

-bret

[suznCB]

Dear Bret, Thanks for replying

but there is no messages in ASE error log