Apache proxy getting 502 errors talking to portal

[Former Member]

We went live with our SAP project approximately 4 weeks ago. We use Apache 2.2 as a reverse proxy with mod_balancer splitting traffic between two SAP portal systems at version 7.0 ehp 1 service pack 7.

In the first 10 days we had two incidents where communication from the Apache reverse proxy to one SAP portal system was consistently failing with a 502 return code, but communication with the other portal system was fine. We do not see anything abnormal in the SAP logs.

The first incident was resolved by restarting just the Apache server while the second required restarting the SAP portal application on the affected system as well.

We are currently examining some leads found on the SAP forum and support site but are looking for input from others to help guide our investigation. The SAP Note 929082 describes a sporadic problem similar to ours except that ours was consistently failing for over 30 minutes.

[Former Member · Accepted solution]

Hello David,

I checked the SAP Note 929082. It seems that whilst you are using load balancing on your Apache Servers, the keep alive timeouts on both your Portal Systems are conflicting. Since you mentioned that the problem has occurred for more than 30 mins I think that the traffic is getting routed to one Portal System only. Can you please use HTTPWatch to trace the urls and the instances where the hits are being made. Also, as the note suggests can you please disable the keep alive timeouts on the Portal Systems and check the results.

Regards,

Pinac

[Former Member · Accepted solution]

Pinac,

Thank you for your reply.

We have a moderate level of logging turned on in the Apache server, so we can see that connections are being evenly distributed between the two portal nodes. To clarify the original statement, we consistently get the problem on connections that are using the portal node with which communications have gone bad, while connections to the other node continue to work properly.

We are reluctant to try the solution suggested in the note, because the KeepAlive value in the Apache proxy relates to communications between the proxy and the client browser, while the communication problem we are having is between the proxy and the portal nodes.

One option we are looking into involves disabling all connection caching between Apache and SAP by setting the DisableReuse flag. None of the examples we have seen have involved doing that but most examples appear to be using Apache 1.3 or 2.0 and so probably are not caching connections. Has anyone needed to do this with their Apache 2.2 reverse proxies?

The problem occurred again for us last Friday. We tested by connecting to the portal directly using a browser from the system on which the Apache proxy runs. Connections to the affected portal node on the SSL port were problematic, while connections to the non-SSL port worked. The same tests performed from a desktop on the portal's subnet yielded the same results.

[Former Member · Accepted solution]

Hello David,

Thank you for the update. Have tried checking your load balancing settings on the Apache Server. Also check the settings in Visual Administrator for HTTP/s cache for both your portal systems. If you require you can troubleshoot the mod_proxy_balancer module.

Other alternatives to your problem using a SAP Web Dispatcher 7.2 which will help you load balance between multiple SAP systems. You may still need to use Apache to support the reverse proxy scenarios but atleast the problem with one node might be resolved.

I suggest that you have a look at Alan Weinnstein's blog on using Apache as a reverse proxy.

/people/sap.user72/blog/2005/02/16/the-reverse-proxy-series--part-1-introduction

Try performing the steps mentioned in the link below under the section Load Balancer Stickyness. See if it helps.

http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html

Regards,

Pinac

Edited by: Pinac Bendre on Aug 2, 2011 10:12 AM

[Former Member · Accepted solution]

Pinac, thank you for your reply.

We are familiar with Alon's blog, and we are already using stick session load balancing.

We can consider the SAP web dispatcher for a longer term solution, but would still like to find something that will address our stability issues in a shorter time frame.

Your thought about changing the cache size led us to some documentation on the [HTTP provider service|http://help.sap.com/saphelp_nw70ehp1/helpdata/en/52/46f6a089754e3a964a5d932eb9db8b/frameset.htm]. Is that what you were referring to? If so changing the CacheSize doesn't seem all that interesting, since that is just the number of static content files which will be cached, but there are some other interesting knobs. Does anyone have experience with tuning these?

[Former Member · Accepted solution]

Hi David,

I agree with you on the use of Web Dispatcher as a long term solution. With regards to the Cache sizing on the portals, I did go through the HTTP Provider service.

In your particular scenario, it seems we need to consider a couple of cache settings, which I am sure you must have already checked on. A) The Portal Cache Settings B) The Server Cache Settings and C) The Browser/Client-side Cache and Cookie settings.

I have a quick question for you. In your scenario, what happens when the Portal where things are working fine is shutdown ? Is it possible for you to try this scenario and let me know the results. Considering that both the systems use different protocols to get routed (HTTP and HTTPS) it will help you in isolating the errors.

I'm looking for other solutions as well. In the meanwhile if you are able to get an answer to this please do share it.

Regards,

Pinac

[Former Member · Accepted solution]

Hi Pinac,

Because of the business expense of having SAP down, I don't have the luxury of doing the sort of diagnosis you suggest unless I can make a strong case that it will provide a critical piece of information. I do know that the problem can still occur with only one portal available, because we have had an occurrence where restarting a problematic portal application server shifted the problem to the other one.

[Former Member · Accepted solution]

Hello David,

Sorry for the delayed reply. I was travelling on account of business.

As mentioned by you, I am precisely worried about the same problem. In the event of downtime on one of your Portal Servers, the other server may get affected. This however seems a extremely peculiar case. Have you raised an OSS message with SAP regarding this problem ?

Also I suggest asking some tips from either Mark Nicholls, Alon Weinstein and other Portal Gurus wouldn't be a bad idea.

I'm still trying to find some solutions and this issue being difficult to replicate has me racking my brains for alternatives.

Regards,

Pinac Bendre

Edited by: Pinac Bendre on Aug 8, 2011 4:12 PM