cancel
Showing results for 
Search instead for 
Did you mean: 

Problem getting database connection

Former Member
0 Kudos

Hi,

Last night, the following errors appeared on my console logs website.

To fix the problem, the MySQL service has been restarted and the problem no longer appears but I want to understand how this problem could appear.

 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | ERROR [hybrisHTTP95] [81.250.91.222] [JDBCConnectionPool] ----------------------------------------
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | Wed Sep 02 18:45:55 CEST 2015: problem getting database connection
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | Exception message thrown was: Timeout waiting for idle object
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | (enable 'db.pool.dumpThreadsOnConnectionError' to see complete thread dump)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | ----------------------------------------
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | sept. 02, 2015 6:45:55 PM org.apache.catalina.core.StandardWrapperValve invoke
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | GRAVE: Servlet.service() for servlet [DispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is de.hybris.platform.persistence.hjmp.HJMPException: illegal exception type: java.util.NoSuchElementException] with root cause
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 | java.util.NoSuchElementException: Timeout waiting for idle object
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.JDBCConnectionPool.borrowConnection(JDBCConnectionPool.java:83)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.ConnectionErrorCheckingJDBCConnectionPool.borrowConnection(ConnectionErrorCheckingJDBCConnectionPool.java:87)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.DataSourceImpl.getFromPool(DataSourceImpl.java:422)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:55.994 |     at de.hybris.platform.jdbcwrapper.DataSourceImpl.doGetConnection(DataSourceImpl.java:397)
 ...

And another example,

 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | ERROR [hybrisHTTP87] [Registry] cannot activate tenant <<master>> since its database connection is currently lost
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | sept. 02, 2015 6:45:56 PM org.apache.catalina.core.StandardWrapperValve invoke
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | GRAVE: "Servlet.service()" pour la servlet DispatcherServlet a généré une exception
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 | java.lang.IllegalStateException: could not switch tenant to <<master>>
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.core.Registry.activateTenant(Registry.java:634)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.core.Registry.setCurrentTenant(Registry.java:476)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.core.Registry.setCurrentTenantByID(Registry.java:608)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.activateTenantFromContext(AbstractPlatformFilterChain.java:178)
 INFO   | jvm 8    | main    | 2015/09/02 18:45:56.096 |     at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.doFilterInternal(AbstractPlatformFilterChain.java:148)
 ...

The database connection was working.

The database instance was up, running and accessible.

I did not find any error in the error.log of MySQL.

The max_connections was not reached.

The connection to the database from the website servers was working properly (via telnet).

I had no apparent problem on the system side.


Below, you'll find mysqltunner stats before restarting :

 root@GAPIX1:~# mysqltuner
 
 >>  MySQLTuner 1.0.1 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
 
 -------- General Statistics --------------------------------------------------
 [--] Skipped version check for MySQLTuner script
 [OK] Currently running supported MySQL version 5.5.38-0ubuntu0.12.04.1-log
 [OK] Operating on 64-bit architecture
 
 -------- Storage Engine Statistics -------------------------------------------
 [--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
 [--] Data in InnoDB tables: 8G (Tables: 432)
 [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
 [!!] Total fragmented tables: 54
 
 -------- Performance Metrics -------------------------------------------------
 [--] Up for: 70d 0h 58m 27s (2B q [340.837 qps], 3M conn, TX: 2080B, RX: 465B)
 [--] Reads / Writes: 89% / 11%
 [--] Total buffers: 34.4G global + 10.2M per thread (1000 max threads)
 [OK] Maximum possible memory usage: 44.4G (71% of installed RAM)
 [OK] Slow queries: 0% (59K/2B)
 [!!] Highest connection usage: 87%  (879/1000)
 [OK] Key buffer size / total MyISAM indexes: 1.0G/94.0K
 [OK] Key buffer hit rate: 100.0% (4B cached / 7 reads)
 [!!] Query cache is disabled
 [OK] Sorts requiring temporary tables: 0% (47K temp sorts / 215M sorts)
 [!!] Joins performed without indexes: 118495
 [OK] Temporary tables created on disk: 0% (14K on disk / 86M total)
 [OK] Thread cache hit rate: 99% (4K created / 3M connections)
 [OK] Table cache hit rate: 24% (4K open / 16K opened)
 [OK] Open file limit used: 0% (20/9K)
 [OK] Table locks acquired immediately: 100% (1B immediate / 1B locks)
 [OK] InnoDB data size / buffer pool: 8.1G/32.0G
 
 -------- Recommendations -----------------------------------------------------
 General recommendations:
     Run OPTIMIZE TABLE to defragment tables for better performance
     Reduce or eliminate persistent connections to reduce connection usage
     Adjust your join queries to always utilize indexes
 Variables to adjust:
     max_connections (> 1000)
     wait_timeout (< 600)
     interactive_timeout (< 600)
     query_cache_size (>= 8M)
     join_buffer_size (> 1.0M, or always use indexes with joins)

I strongly feel that it'sa lock but no action could have completely blocked the database at this time.

I'd like your opinion on this topic and also for one that speaks of the option 'db.customsessionsql' in https://wiki.hybris.com/display/release5/Include+-+projectproperties+-+MySQL+parameters

I'll enable the option 'db.pool.dumpThreadsOnConnectionError' to see complete thread dump for the next time but I wish it no longer happens.

I hope someone can help me.

Kind regards,

Accepted Solutions (0)

Answers (5)

Answers (5)

0 Kudos
former_member537989
Contributor
0 Kudos

Next time the error occurs you should get a thread dump - if there is resource scarcity situation ( lack of any available connection) then the dump should contain a lot of threads actively using connections. Do you have custom transaction handling in your code?

Former Member
0 Kudos

HI Matthieu,

I'm also encountering the same problem, did you able to find out work around solution for this and it would be helpful if you can share the information.

Thanks & Regards, Teja

Former Member
0 Kudos

Bad track. No network/firewall issue. The problem hasn't reappeared.

I'll keep you informed if I have news.

former_member602476
Active Participant
0 Kudos

A network issue between the hybris app and db server could cause this, or maybe firewall dropping idle connections. Make sure, if you have a firewall between the DB and app server, that the firewall does not drop idle connections