Timer already cancelled error in yellowfin.log
9 February, 2011
Hi
Taupo DC was upgraded to YF 5.1 on Monday 7 Feb. Since then they have experienced intermittent problems where one of the Database connections shows an exclamation mark rather than a tick, beside it, and reports are not accessible.
Looking at the logs, 'Timer already cancelled' features when this problem arises. Google searches indicate perhaps this is a Tomcat-related problem.
I've attached a recent yellowfin.log.
What is perplexing is that, when I compare processes on our other YF5.1 client sites, and indeed Taupo's test server, there is no tomcat process running, while there IS on the main Taupo server. IS Tomcat actually required?
Thank you
Paula
e.g. on Taupo main server (where the problem is):
[ncspaulak@taupodc ~]$ ps -ef |grep [y]ellow
root 2978 1 0 01:00 ? 00:02:16 /usr/java/jdk1.6.0_01/jre/bin/java -Djava.util.logging.config.file=/usr/local/yellowfin/appserver/conf/logging.properties -Xms64m -Xmx256m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/local/yellowfin/appserver/common/endorsed -classpath :/usr/local/yellowfin/appserver/bin/bootstrap.jar:/usr/local/yellowfin/appserver/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/yellowfin/appserver -Dcatalina.home=/usr/local/yellowfin/appserver -Djava.io.tmpdir=/usr/local/yellowfin/appserver/temp org.apache.catalina.startup.Bootstrap start
[ncspaulak@taupodc ~]$ ps -ef |grep [t]omcat
tomcat 13420 1 0 Feb09 ? 00:00:03 /usr/lib/jvm/java/bin/java -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar:/usr/share/java/mx4j/mx4j-impl.jar:/usr/share/java/mx4j/mx4j-jmx.jar -Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start
On Taupo's test server:
[root@taupo-tst logs]# ps -ef |grep [y]ellow
root 8084 1 0 Jan28 ? 00:19:22 /usr/java/jdk1.6.0_01/jre/bin/java -Djava.util.logging.config.file=/usr/local/yellowfin/appserver/conf/logging.properties -Xms64m -Xmx256m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/local/yellowfin/appserver/common/endorsed -classpath :/usr/local/yellowfin/appserver/bin/bootstrap.jar:/usr/local/yellowfin/appserver/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/yellowfin/appserver -Dcatalina.home=/usr/local/yellowfin/appserver -Djava.io.tmpdir=/usr/local/yellowfin/appserver/temp org.apache.catalina.startup.Bootstrap start
[root@taupo-tst logs]# ps -ef |grep [t]omcat
Taupo DC was upgraded to YF 5.1 on Monday 7 Feb. Since then they have experienced intermittent problems where one of the Database connections shows an exclamation mark rather than a tick, beside it, and reports are not accessible.
Looking at the logs, 'Timer already cancelled' features when this problem arises. Google searches indicate perhaps this is a Tomcat-related problem.
I've attached a recent yellowfin.log.
What is perplexing is that, when I compare processes on our other YF5.1 client sites, and indeed Taupo's test server, there is no tomcat process running, while there IS on the main Taupo server. IS Tomcat actually required?
Thank you
Paula
e.g. on Taupo main server (where the problem is):
[ncspaulak@taupodc ~]$ ps -ef |grep [y]ellow
root 2978 1 0 01:00 ? 00:02:16 /usr/java/jdk1.6.0_01/jre/bin/java -Djava.util.logging.config.file=/usr/local/yellowfin/appserver/conf/logging.properties -Xms64m -Xmx256m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/local/yellowfin/appserver/common/endorsed -classpath :/usr/local/yellowfin/appserver/bin/bootstrap.jar:/usr/local/yellowfin/appserver/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/yellowfin/appserver -Dcatalina.home=/usr/local/yellowfin/appserver -Djava.io.tmpdir=/usr/local/yellowfin/appserver/temp org.apache.catalina.startup.Bootstrap start
[ncspaulak@taupodc ~]$ ps -ef |grep [t]omcat
tomcat 13420 1 0 Feb09 ? 00:00:03 /usr/lib/jvm/java/bin/java -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar:/usr/share/java/mx4j/mx4j-impl.jar:/usr/share/java/mx4j/mx4j-jmx.jar -Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start
On Taupo's test server:
[root@taupo-tst logs]# ps -ef |grep [y]ellow
root 8084 1 0 Jan28 ? 00:19:22 /usr/java/jdk1.6.0_01/jre/bin/java -Djava.util.logging.config.file=/usr/local/yellowfin/appserver/conf/logging.properties -Xms64m -Xmx256m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/local/yellowfin/appserver/common/endorsed -classpath :/usr/local/yellowfin/appserver/bin/bootstrap.jar:/usr/local/yellowfin/appserver/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/yellowfin/appserver -Dcatalina.home=/usr/local/yellowfin/appserver -Djava.io.tmpdir=/usr/local/yellowfin/appserver/temp org.apache.catalina.startup.Bootstrap start
[root@taupo-tst logs]# ps -ef |grep [t]omcat
Hi Paula,
The "exclamation mark" next to the data source just means that a connection has not yet been attempted to the data source. There are now three status icons for data sources:
OK (blue tick) - connection successful
Undetermined (yellowfin exclamation mark) - connection has not yet been attempted
Unsuccessful (red cross) - connection failed
The yellowfin exclamation mark icon shouldn't be a problem - if you want to check if the connection is ok, edit the data source and click the Test Connection link.
In regards to these Tomcat messages:
On the Taupo main server, there are two java processes - the first one is Yellowfin (/usr/local/yellowfin/...). The second looks like another tomcat installation on that server, located in /usr/share/tomcat5. This doesn't have anything to do with Yellowfin, and may be from another application installed on the server.
On the test server there is only the Yellowfin process, so it looks like the additional tomcat application is not installed (or at least is not running).
There should be no problem having multiple instances of Tomcat on a server, as long as they are listening on different ports.
Regards,
David
The "exclamation mark" next to the data source just means that a connection has not yet been attempted to the data source. There are now three status icons for data sources:
OK (blue tick) - connection successful
Undetermined (yellowfin exclamation mark) - connection has not yet been attempted
Unsuccessful (red cross) - connection failed
The yellowfin exclamation mark icon shouldn't be a problem - if you want to check if the connection is ok, edit the data source and click the Test Connection link.
In regards to these Tomcat messages:
On the Taupo main server, there are two java processes - the first one is Yellowfin (/usr/local/yellowfin/...). The second looks like another tomcat installation on that server, located in /usr/share/tomcat5. This doesn't have anything to do with Yellowfin, and may be from another application installed on the server.
On the test server there is only the Yellowfin process, so it looks like the additional tomcat application is not installed (or at least is not running).
There should be no problem having multiple instances of Tomcat on a server, as long as they are listening on different ports.
Regards,
David
Hi David
Thanks for that. On the strength of your reply, and being aware that tomcat isn't used for anything else on the server concerned, I've stopped tomcat.
From what I can see, in fact it is the yellowfin catalina logs that tell us that Tomcat is being started. Can you perhaps advise on this?:
Feb 11, 2011 1:00:05 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.27
Feb 11, 2011 1:00:05 AM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Feb 11, 2011 1:01:42 AM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8081
Feb 11, 2011 1:01:42 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 96975 ms
(For performance reasons the yellowfin service is being restarted at 1am each morning. Our client has reported a happy improvement in performance with YF5.1, so this restart may be redundant now.)
There are two areas where the client has made conf changes:
/etc/java/java.conf:
# Added to increase performance of Yellowfin/Java
# Removed again for YF v5.1 Upgrade as 'Out of Memory' errors occuring
# FORCE_IPADDR_LOOKUP=0
# INITIAL_MEMORY=72M
# MAX_MEMORY=512M
/etc/my.cnf (MySQL)
[mysqld]
max_allowed_packet=32M
# query_cache_size=32M
read_buffer_size=2M
Thanks and kind regards
Paula
Thanks for that. On the strength of your reply, and being aware that tomcat isn't used for anything else on the server concerned, I've stopped tomcat.
From what I can see, in fact it is the yellowfin catalina logs that tell us that Tomcat is being started. Can you perhaps advise on this?:
Feb 11, 2011 1:00:05 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.27
Feb 11, 2011 1:00:05 AM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Feb 11, 2011 1:01:42 AM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8081
Feb 11, 2011 1:01:42 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 96975 ms
(For performance reasons the yellowfin service is being restarted at 1am each morning. Our client has reported a happy improvement in performance with YF5.1, so this restart may be redundant now.)
There are two areas where the client has made conf changes:
/etc/java/java.conf:
# Added to increase performance of Yellowfin/Java
# Removed again for YF v5.1 Upgrade as 'Out of Memory' errors occuring
# FORCE_IPADDR_LOOKUP=0
# INITIAL_MEMORY=72M
# MAX_MEMORY=512M
/etc/my.cnf (MySQL)
[mysqld]
max_allowed_packet=32M
# query_cache_size=32M
read_buffer_size=2M
Thanks and kind regards
Paula
Hi Paula,
That is expected behavior.
As Tomcat is still needed for Yellowfin, though now it seems you only have 1 instance running.
That's great new about the performance issues.
Let me know if you have any other issues.
Regards,
David
That is expected behavior.
As Tomcat is still needed for Yellowfin, though now it seems you only have 1 instance running.
That's great new about the performance issues.
Let me know if you have any other issues.
Regards,
David