Login attempt is deadlocking

This forum is now locked, since Gold Support is no longer offered.

Moderator: SourceGear

Locked
ajj3085
Posts: 228
Joined: Thu Feb 16, 2006 11:39 am

Login attempt is deadlocking

Post by ajj3085 » Wed Feb 22, 2006 7:09 am

Hi,

I'm running the latest Vault (3.1.7) and have been pleased with it so far.

I have two projects setup under CruiseControl.Net and for the most part things work fine... except that sometimes a build fails, and when I check the Vault log file, it indicates there was a deadlock during the login process.

I'm guessing that the two projects are trying to get a history of changes at once, and one is blocking the other somehow. Any ideas on how to resolve this?

Thanks
Andy

Log file contains this:

Code: Select all

----2/22/2006 3:56:07 AM     andyj--AndyJ.sta.medvt.us(172.18.1.32)--SSL Enabled	Login 
----2/22/2006 3:56:07 AM     andyj--AndyJ.sta.medvt.us(172.18.1.32)--SSL Enabled	System.Data.SqlClient.SqlException: Transaction (Process ID 53) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at VaultServiceSQL.VaultSqlSystem.LoginUser(VaultSqlConn conn, String strLogin, String strDbHashPwd, Guid& gSession)    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at VaultServiceSQL.VaultSqlSystem.LoginUser(VaultSqlConn conn, String strLogin, String strDbHashPwd, Guid& gSession)
----2/22/2006 3:56:07 AM     andyj--AndyJ.sta.medvt.us(172.18.1.32)--SSL Enabled	Login failed:  FailDBUpdate 
----2/22/2006 3:56:08 AM     andyj--AndyJ.sta.medvt.us(172.18.1.32)--SSL Enabled	Logout
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Wed Feb 22, 2006 8:50 am

Hmm. This was a known issue that I thought we resolved in 3.1.7. Can you:

1) Verify that the server is running 3.1.7. (The GUI client will tell you upon login, unless you've turned that off, in which case, Help->Technical Support will tell you.)

Assuming that you are, you can help us find the problem by:

2) Enable deadlock trace logging in SQL Server by running this script:

Code: Select all

dbcc traceon(1204)
The next time you encounter the problem, your SQL error log will have some useful information for us. (By default, it's located at C:\Program Files\Microsoft SQL Server\MSSQL\LOG\ERRORLOG.)
Ian Olsen
SourceGear
Top
ajj3085
Posts: 228
Joined: Thu Feb 16, 2006 11:39 am

Post by ajj3085 » Wed Feb 22, 2006 1:38 pm

ian_sg wrote:Hmm. This was a known issue that I thought we resolved in 3.1.7. Can you:

1) Verify that the server is running 3.1.7. (The GUI client will tell you upon login, unless you've turned that off, in which case, Help->Technical Support will tell you.)
Version is reported as 3.1.7.3719
ian_sg wrote:Assuming that you are, you can help us find the problem by:

2) Enable deadlock trace logging in SQL Server by running this script:

Code: Select all

dbcc traceon(1204)
The next time you encounter the problem, your SQL error log will have some useful information for us. (By default, it's located at C:\Program Files\Microsoft SQL Server\MSSQL\LOG\ERRORLOG.)
Ok, I'll try that. One thing that I forgot to mention was that I'm running the vault server with a Sql Server 2005 Express database. Maybe that's causing something to come up? Ill let you know the trace results.
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Wed Feb 22, 2006 1:42 pm

That might be relevant, though I doubt it. Would you mind sending me the entire contents of your Help->Technical Support?
Ian Olsen
SourceGear
Top
ajj3085
Posts: 228
Joined: Thu Feb 16, 2006 11:39 am

Post by ajj3085 » Wed Feb 22, 2006 1:46 pm

ian_sg wrote:That might be relevant, though I doubt it. Would you mind sending me the entire contents of your Help->Technical Support?
Sure:

Client Information
Vault Client Version: 3.1.7.3719
.Net Framework Version: 2.0.50727.42
Operating System: Microsoft Windows XP Professional
Service Pack: 2.0
OS Version: 5.1.2600
Total Physical Memory: 502.73 MB
Time Zone: (GMT-05:00) Eastern Time (US & Canada)

Server Information
Vault Server Version: 3.1.7.3719
.Net Framework Version: 2.0.50727.42
Operating System: Microsoft Windows XP Professional
Service Pack: 2.0
OS Version: 5.1.2600
Timezone: (GMT-05:00) Eastern Time (US & Canada)
SQL Version: Microsoft SQL Server 2005 - 9.00.1399.06 (Intel X86)
Oct 14 2005 00:33:37
Copyright (c) 1988-2005 Microsoft Corporation
Express Edition on Windows NT 5.1 (Build 2600: Service Pack 2)

License Information
1 serial number(s):
1 of 1: 1 users, permanent
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Wed Feb 22, 2006 2:45 pm

Okay thanks, I just wanted to rule out anything strange. We'll have to wait and see what the SQL log reveals next time it happens.
Ian Olsen
SourceGear
Top
ajj3085
Posts: 228
Joined: Thu Feb 16, 2006 11:39 am

Post by ajj3085 » Thu Feb 23, 2006 10:41 am

ian_sg wrote:Okay thanks, I just wanted to rule out anything strange. We'll have to wait and see what the SQL log reveals next time it happens.
Just wanted to update.. I did get the exception again, but no log was wriTten from SQL. I executed the dbcc command again, hopefully it will catch it next time.

Andy
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Thu Feb 23, 2006 10:44 am

Just as an FYI, the dbcc command should "stick" for as long as SQL server is running. You should only need to re-run it if you restart SQL Server.
Ian Olsen
SourceGear
Top
gsmalter
Posts: 115
Joined: Sat Jul 09, 2005 11:13 am

I'm experience the same issue

Post by gsmalter » Sun Mar 05, 2006 12:47 pm

I just went to make a post about receiving deadlocks when using CruiseControl and Vault 3.1.7, but I did a search first and came across this. Our problem is exactly as Andy described it. We have 5 Cruise Control projects.
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Sun Mar 05, 2006 9:37 pm

I'm pretty sure I've duplicated this over the weekend. I'll put the logs under a microscope tomorrow and see what that reveals.
Ian Olsen
SourceGear
Top
ajj3085
Posts: 228
Joined: Thu Feb 16, 2006 11:39 am

Post by ajj3085 » Mon Mar 06, 2006 6:55 am

Hey,

Unfortunatly I haven't been able to reproduce it, but that may be because I've changed the setup a bit.

CCNet and vault are now running on a server; previously they were running on my workstation, and it seemed as though the problem came up more during startup (when CCNet, vault et al) were starting, although there were times it happened overnight while my workstation was kept on.

Also, I moved from Sql Express to Sql Developer Ed.

I'll keep an eye out for it and post logs if it happens again.

Andy
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Mon Mar 06, 2006 1:43 pm

We have a fix for this that will be included in the next maintenance release of Vault (3.1.8), which is scheduled to ship next week. (We're still testing, and pushing that date back is possible but unlikely at this point.)
Ian Olsen
SourceGear
Top
ian_sg
Posts: 787
Joined: Wed May 04, 2005 10:55 am
Location: SourceGear
Contact:
Contact ian_sg

Post by ian_sg » Tue Mar 07, 2006 1:33 pm

The bad news: my fix doesn't completely eliminate these. There's some security-related code in here that we don't want to overhaul in a maintenance release that prevents me from squashing them altogether.

The good news: I added retry code to CruiseControl.NET to work around this, and it has worked beautifully under a pretty ridiculous load test. It's included as of CC.NET build 1.1.0.2217. The two new configuration settings, pollRetryAttempts and pollRetryWait are documented: http://confluence.public.thoughtworks.o ... trol+Block
Ian Olsen
SourceGear
Top
ajj3085
Posts: 228
Joined: Thu Feb 16, 2006 11:39 am

Post by ajj3085 » Tue Mar 07, 2006 1:34 pm

ian_sg wrote:The bad news: my fix doesn't completely eliminate these. There's some security-related code in here that we don't want to overhaul in a maintenance release that prevents me from squashing them altogether.

The good news: I added retry code to CruiseControl.NET to work around this, and it has worked beautifully under a pretty ridiculous load test. It's included as of CC.NET build 1.1.0.2217. The two new configuration settings, pollRetryAttempts and pollRetryWait are documented: http://confluence.public.thoughtworks.o ... trol+Block
Great, thanks for making a workaround!
Top
ismangil
Posts: 197
Joined: Wed Jun 30, 2004 10:49 am
Location: Sheffield, UK
Contact:
Contact ismangil

Post by ismangil » Fri Apr 07, 2006 9:24 am

ian_sg wrote:The bad news: my fix doesn't completely eliminate these. There's some security-related code in here that we don't want to overhaul in a maintenance release that prevents me from squashing them altogether.
Ian,

Would this deadlock problem be reduced or gone if, to be extreme, we have a separate username for each project?

In other words is this specific to using the same username by multiple threads/client?
Top
Locked

Return to “Gold Support (Vault) -- Read-only”