broadcast email failure – DB commit exception
9 August, 2012
Hi,
I have a report we are broadcasting that is failing consistently.
The Schedule Management screen shows the reason as "DBAction.commit() exception"
This started this week. No changes have been made to the environment.
The report runs fine when done manually. Only the scheduled job fails.
Log file snippet attached.
Thanks,
Chris
I have a report we are broadcasting that is failing consistently.
The Schedule Management screen shows the reason as "DBAction.commit() exception"
This started this week. No changes have been made to the environment.
The report runs fine when done manually. Only the scheduled job fails.
Log file snippet attached.
Thanks,
Chris
Hi Chris,
The errors in the logs seem to say that the DB connection itself has been closed.
When you say 'done manually' do you mean going into schedule management and manually running the broadcast?
The most common cause of these issues is a DB timeout when sending out the request.
Are you able to send across the following log files;
-JDBC.log (this shows the Yellowfin DB connection)
-Source....log (you will have a few of these files, each one is allocated to a data source) .
Also if you can note the data source that this report is based of it will help us identify which source log is relevant.
Thanks,
David
The errors in the logs seem to say that the DB connection itself has been closed.
When you say 'done manually' do you mean going into schedule management and manually running the broadcast?
The most common cause of these issues is a DB timeout when sending out the request.
Are you able to send across the following log files;
-JDBC.log (this shows the Yellowfin DB connection)
-Source....log (you will have a few of these files, each one is allocated to a data source) .
Also if you can note the data source that this report is based of it will help us identify which source log is relevant.
Thanks,
David
Hi David,
When you say 'done manually' do you mean going into schedule management and manually running the broadcast?
The most common cause of these issues is a DB timeout when sending out the request.
[color="#FF0000"> No, I meant running the report from the Report List works. Manually kicking off the job through schedule management fails.[/color]
Are you able to send across the following log files;
-JDBC.log (this shows the Yellowfin DB connection)
[color="#FF0000">Attached in zip.[/color]
-Source....log (you will have a few of these files, each one is allocated to a data source). [color="#FF0000">Where are these located. I found nothing like mentioned on the app server.[/color]
Also if you can note the data source that this report is based of it will help us identify which source log is relevant.
[color="#FF0000">The source is QTPM Prod[/color]
Regards,
Chris
When you say 'done manually' do you mean going into schedule management and manually running the broadcast?
The most common cause of these issues is a DB timeout when sending out the request.
[color="#FF0000"> No, I meant running the report from the Report List works. Manually kicking off the job through schedule management fails.[/color]
Are you able to send across the following log files;
-JDBC.log (this shows the Yellowfin DB connection)
[color="#FF0000">Attached in zip.[/color]
-Source....log (you will have a few of these files, each one is allocated to a data source). [color="#FF0000">Where are these located. I found nothing like mentioned on the app server.[/color]
Also if you can note the data source that this report is based of it will help us identify which source log is relevant.
[color="#FF0000">The source is QTPM Prod[/color]
Regards,
Chris
Hi Chris,
The JDBC.log that was sent across is over 2 years old and not relevant to your current issue.
The Stdout..log file shows a number of connection issues, though not the cause of them.
When you ran the broadcast via the task management window, what error did you get?
It's worth noting that when a report is run via a broadcast, it is run for each user, so running it via the report list once, may be fine, though running it 20 times (if the broadcast is sent to 20 users) requires the DB connection to stay open while it runs the report, which is where we usually see errors, as the DB may close the connection before the broadcast is finished.
Are you able to send the relevant log files under YellowfinappserverLOGS ;
-JDBC.log
-Stdout..log
-Yellowfin.log
-Source...log files (e.g. source.37000.log)
Once we have more information we should be able to home in on what is causing the connection error.
Regards,
David
The JDBC.log that was sent across is over 2 years old and not relevant to your current issue.
The Stdout..log file shows a number of connection issues, though not the cause of them.
When you ran the broadcast via the task management window, what error did you get?
It's worth noting that when a report is run via a broadcast, it is run for each user, so running it via the report list once, may be fine, though running it 20 times (if the broadcast is sent to 20 users) requires the DB connection to stay open while it runs the report, which is where we usually see errors, as the DB may close the connection before the broadcast is finished.
Are you able to send the relevant log files under YellowfinappserverLOGS ;
-JDBC.log
-Stdout..log
-Yellowfin.log
-Source...log files (e.g. source.37000.log)
Once we have more information we should be able to home in on what is causing the connection error.
Regards,
David
Hi David,
Along the upgrade path, it looks like we did not turn on the new logging options available (Believe it was with v5.0). So we do not have the newer logs available.
I have updated the log4j.properties files to use the expanded log options.
When you ran the broadcast via the task management window, what error did you get?
[color="#FF0000">We have seen 2 different messages. 'DBAction.commit() exception', and 'Invalid state, the Connection object is closed.'[/color]
I have attached the stdout.log and the Yellowfin.log files (zipped).
Once I get some data in the new logs, I'll post it.
Thanks,
Chris
Along the upgrade path, it looks like we did not turn on the new logging options available (Believe it was with v5.0). So we do not have the newer logs available.
I have updated the log4j.properties files to use the expanded log options.
When you ran the broadcast via the task management window, what error did you get?
[color="#FF0000">We have seen 2 different messages. 'DBAction.commit() exception', and 'Invalid state, the Connection object is closed.'[/color]
I have attached the stdout.log and the Yellowfin.log files (zipped).
Once I get some data in the new logs, I'll post it.
Thanks,
Chris
Thanks Chris
I have the new logs but they are over the 2MB attachment size. I'll email them to the Support email address.
Chris
Chris
Hi Chris,
Thanks for the logs.
The JDBC.log does in fact show some timeout issues, which means the connection to the Yellowfin DB is open too long, see error below which appears not long before your broadcast error, so I suspect they are related;
2012-08-14 09:20:57 ERROR: Connection[6] checkout timeout exceeded
2012-08-14 09:20:57 ERROR: Connection[6] was checked out at 2012-08-14 09:15:38
I have tried to find corrosponding errors in the 'Source..log' though it looks like this is for a data source 'QTPM_PerformanceMetrics' which I don't believe is related.
You should have other 'Source..log' files in the appserverLOGS folder?
Can you please send them all across.
Either way to resolve your timeout issue you will need to increase the timeout on the Yellowfin DB, as mentioned in this post:
How to configure the timeout settings for a database connection.
I would also suggest increasing the timeout for the data source conneciton.
Also just a few quick questions;
How long does the report take to run, and how many users are you sending this to?
If you broadcast this to 1 user only does it work ok?
Please let me know your findings.
Regards,
David
Thanks for the logs.
The JDBC.log does in fact show some timeout issues, which means the connection to the Yellowfin DB is open too long, see error below which appears not long before your broadcast error, so I suspect they are related;
2012-08-14 09:20:57 ERROR: Connection[6] checkout timeout exceeded
2012-08-14 09:20:57 ERROR: Connection[6] was checked out at 2012-08-14 09:15:38
I have tried to find corrosponding errors in the 'Source..log' though it looks like this is for a data source 'QTPM_PerformanceMetrics' which I don't believe is related.
You should have other 'Source..log' files in the appserverLOGS folder?
Can you please send them all across.
Either way to resolve your timeout issue you will need to increase the timeout on the Yellowfin DB, as mentioned in this post:
How to configure the timeout settings for a database connection.
I would also suggest increasing the timeout for the data source conneciton.
Also just a few quick questions;
How long does the report take to run, and how many users are you sending this to?
If you broadcast this to 1 user only does it work ok?
Please let me know your findings.
Regards,
David
Hi David,
Thanks for looking at the logs and your input. I have added the timeout for the Yellowfin DB to the web.xml file. I set it and the data source timeout to 45 mins (2700 secs).
I was successful in running one of the failed schedules after the restart.
[color="#FF0000">How long does the report take to run, and how many users are you sending this to?
If you broadcast this to 1 user only does it work ok?[/color]
The report by itself takes 3 mins to run (fairly large report).
There are 9 users it is getting sent to.
Yes, running it for 1 user does work.
The other source logs are attached.
I'll run the other scheduled jobs manually and see how they do with the new settings.
Best Regards,
Chris
Thanks for looking at the logs and your input. I have added the timeout for the Yellowfin DB to the web.xml file. I set it and the data source timeout to 45 mins (2700 secs).
I was successful in running one of the failed schedules after the restart.
[color="#FF0000">How long does the report take to run, and how many users are you sending this to?
If you broadcast this to 1 user only does it work ok?[/color]
The report by itself takes 3 mins to run (fairly large report).
There are 9 users it is getting sent to.
Yes, running it for 1 user does work.
The other source logs are attached.
I'll run the other scheduled jobs manually and see how they do with the new settings.
Best Regards,
Chris
Hi Chris,
That is good news, it looks like we have confirmed the issue, and should be able to get around it by increasing the time out period.
I don't see any time out errors in the attached source logs, so I guess the timeout will be purely on the Yellowfin DB side.
Please let me know your findings and if you run into any other issues.
Regards,
David
That is good news, it looks like we have confirmed the issue, and should be able to get around it by increasing the time out period.
I don't see any time out errors in the attached source logs, so I guess the timeout will be purely on the Yellowfin DB side.
Please let me know your findings and if you run into any other issues.
Regards,
David
Hi David,
The broadcasts have run for a bit over a week and we have had no failures. Thanks for the assistance.
Best Regards,
Chris
The broadcasts have run for a bit over a week and we have had no failures. Thanks for the assistance.
Best Regards,
Chris