question about Rep Agent, 'memory_limit' and message

[jmtorres]

Hello,

Having  Warm Standby pair  ASE to ASE I'm getting this error :

45. WARNING #7038 REP AGENT(dba.machi) - de/generic/mem/mem.c(2763)

WARNING: Memory usage is above 80 percent. Increase 'memory_limit' or reduce cache sizes to avoid repserver threads from sleeping due to lack of memory.

- This means that I should increase memory_limit in RS ? ( surely)

- What caches are supposed to be reduced .? This message is  kind of confusing..

Thanks

Regards

Jose

[terry_penna · Accepted solution]

Jose

I apologize that the warning message is confusing let me try and help clear up some of the confusion?

You are correct that the warning message is telling you that the RS is close to exceeding its memory_limit and that you should increase it if necessary.  It may hit this briefly and you will be okay or it may continue and the RS will either stop replicating or it could shutdown?  In a lot of cases your host should have enough RAM and you can increase the parameter.  In other cases RAM is limited on your host which brings me to the second part of the message.

This is the suggestion to reduce 'cache sizes', this means any RS parameter that uses memory from the memory_limit. This could be sqt_max_cache_size, dsi_sqt_cache_size, dist_sqt_max_cache_size, sts_cache, exec_nrm_request_limit etc.  The most common and biggest users of the memory_limit are the sqt_max_cache_size, dsi_sqt_cache_size and dist_sqt_max_cache_size and you could look at those to see if they could be reduced, even temporarily to allow a big replication job to finish and if RAM is limited on your host?

Another way to see if you can reduce some of the RS parameters that use memory from the memory_limit is to tune it using the RS Monitors and Counters package that can be found on the Replication Server Wiki page.  You can download the package and it contains documentation on how to set it up, collect the counter data and run the reports to evaluate.  You may find that some configuration parameters that you have set are to big and can be reduced to free up more of your memory_limit.

I hope this helps with some of the confusion regarding this warning.

Regards

Terry

[jmtorres · Accepted solution]

Thanks  a lot Terry,

Currently we cannot decrease some of :sqt_max_cache_size, dsi_sqt_cache_size because there is one DB which is heavy in transactions , so the SD and sqt caches are having a lot of contention and latency.

I've attached a sample image form SCC.

RS is running on a  Linux Virtual machine so I guess we should increase  RAM available in order to bump up max_memory and sqts caches.

Regards

Jose

[terry_penna · Accepted solution]

Jose

I would first increase the memory_limit and monitor what is happening.  You may have enough sqt cache and everything will be fine. 

If you are also seeing warning messages similar to this:

W. 2014/04/20 19:12:46. WARNING #24068 SQT(103:1  DIST PDS.pdb) - t/sqtint.c(1380) SQT cache size is too low to load more than one transaction into the cache.

or

W. 14/04/20 18:12:46. WARNING #24057: _sqt_remove_largest_tran(252:1 PDS.pdb): No candidate found for removal. Memory limit will be exceeded by SQM/TI thread.

then you would need to increase your sqt cache size after you increased your memory_limit.

Regards

Terry

[jmtorres · Accepted solution]

Terry, Thanks a lot!

Regards

Jose