Skip to Content
author's profile photo Former Member
0
Former Member
Sep 03, 2015 at 12:38 PM

Problem getting database connection

2445 Views

Hi,

Last night, the following errors appeared on my console logs website.

To fix the problem, the MySQL service has been restarted and the problem no longer appears but I want to understand how this problem could appear.

 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | ERROR [hybrisHTTP95] [81.250.91.222] [JDBCConnectionPool] ----------------------------------------
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | Wed Sep 02 18:45:55 CEST 2015: problem getting database connection
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | Exception message thrown was: Timeout waiting for idle object
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | (enable 'db.pool.dumpThreadsOnConnectionError' to see complete thread dump)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | ----------------------------------------
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | sept. 02, 2015 6:45:55 PM org.apache.catalina.core.StandardWrapperValve invoke
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | GRAVE: Servlet.service() for servlet [DispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is de.hybris.platform.persistence.hjmp.HJMPException: illegal exception type: java.util.NoSuchElementException] with root cause
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | java.util.NoSuchElementException: Timeout waiting for idle object
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.JDBCConnectionPool.borrowConnection(JDBCConnectionPool.java:83)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.ConnectionErrorCheckingJDBCConnectionPool.borrowConnection(ConnectionErrorCheckingJDBCConnectionPool.java:87)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.DataSourceImpl.getFromPool(DataSourceImpl.java:422)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.DataSourceImpl.doGetConnection(DataSourceImpl.java:397)
 ...

And another example,

 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | ERROR [hybrisHTTP87] [Registry] cannot activate tenant <<master>> since its database connection is currently lost
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | sept. 02, 2015 6:45:56 PM org.apache.catalina.core.StandardWrapperValve invoke
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | GRAVE: "Servlet.service()" pour la servlet DispatcherServlet a généré une exception
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | java.lang.IllegalStateException: could not switch tenant to <<master>>
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.core.Registry.activateTenant(Registry.java:634)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.core.Registry.setCurrentTenant(Registry.java:476)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.core.Registry.setCurrentTenantByID(Registry.java:608)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.activateTenantFromContext(AbstractPlatformFilterChain.java:178)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.doFilterInternal(AbstractPlatformFilterChain.java:148)
 ...

The database connection was working.

The database instance was up, running and accessible.

I did not find any error in the error.log of MySQL.

The max_connections was not reached.

The connection to the database from the website servers was working properly (via telnet).

I had no apparent problem on the system side.

Below, you'll find mysqltunner stats before restarting :

 root@GAPIX1:~# mysqltuner
 
 >>  MySQLTuner 1.0.1 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
 
 -------- General Statistics --------------------------------------------------
 [--] Skipped version check for MySQLTuner script
 [OK] Currently running supported MySQL version 5.5.38-0ubuntu0.12.04.1-log
 [OK] Operating on 64-bit architecture
 
 -------- Storage Engine Statistics -------------------------------------------
 [--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
 [--] Data in InnoDB tables: 8G (Tables: 432)
 [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
 [!!] Total fragmented tables: 54
 
 -------- Performance Metrics -------------------------------------------------
 [--] Up for: 70d 0h 58m 27s (2B q [340.837 qps], 3M conn, TX: 2080B, RX: 465B)
 [--] Reads / Writes: 89% / 11%
 [--] Total buffers: 34.4G global + 10.2M per thread (1000 max threads)
 [OK] Maximum possible memory usage: 44.4G (71% of installed RAM)
 [OK] Slow queries: 0% (59K/2B)
 [!!] Highest connection usage: 87%  (879/1000)
 [OK] Key buffer size / total MyISAM indexes: 1.0G/94.0K
 [OK] Key buffer hit rate: 100.0% (4B cached / 7 reads)
 [!!] Query cache is disabled
 [OK] Sorts requiring temporary tables: 0% (47K temp sorts / 215M sorts)
 [!!] Joins performed without indexes: 118495
 [OK] Temporary tables created on disk: 0% (14K on disk / 86M total)
 [OK] Thread cache hit rate: 99% (4K created / 3M connections)
 [OK] Table cache hit rate: 24% (4K open / 16K opened)
 [OK] Open file limit used: 0% (20/9K)
 [OK] Table locks acquired immediately: 100% (1B immediate / 1B locks)
 [OK] InnoDB data size / buffer pool: 8.1G/32.0G
 
 -------- Recommendations -----------------------------------------------------
 General recommendations:
     Run OPTIMIZE TABLE to defragment tables for better performance
     Reduce or eliminate persistent connections to reduce connection usage
     Adjust your join queries to always utilize indexes
 Variables to adjust:
     max_connections (> 1000)
     wait_timeout (< 600)
     interactive_timeout (< 600)
     query_cache_size (>= 8M)
     join_buffer_size (> 1.0M, or always use indexes with joins)

I strongly feel that it'sa lock but no action could have completely blocked the database at this time.

I'd like your opinion on this topic and also for one that speaks of the option 'db.customsessionsql' in https://wiki.hybris.com/display/release5/Include+-+projectproperties+-+MySQL+parameters

I'll enable the option 'db.pool.dumpThreadsOnConnectionError' to see complete thread dump for the next time but I wish it no longer happens.

I hope someone can help me.

Kind regards,