Slowness during check outs

If you are having a problem using Vault, post a message here.
Post Reply
joshp8a
Posts: 14
Joined: Thu Mar 27, 2014 11:34 am

Slowness during check outs

Post by joshp8a » Mon Apr 06, 2020 1:16 pm

Hi SG team! Hope you all well.

Sorry to bother you guys. I am having an issue when doing check outs from the Vault client or VS. From the messages in the client, it takes 7 minutes for check out; 10 min in some cases. Other operations like view history, compares, undo check-ins, get are quick. Only check-out drags.

RAM is high.. 98% use right now, only process taking CPU is SQL server. I bounced the IIS services in hopes of correcting it but no luck, while IIS was down, SQL server process was using same amount of CPU. Brought IIS up, and process stood about the same. Checking the sgvault.log I don't see much. Vault Pro is set to quiet so I am not doing deep logging at the moment.

Any other ideas on what to check? Thinking about bouncing SQL services as well, but thought asking for help first.

We are running 7.2.1, Windows 2012R2, 3.2GHz, 1 Core, 2 processors, 4 Gb RAM, 6 GB page file.

DB size is: sgvault is 7.82 GB. LDF is 2.26 GB.

Thanks in advance!

Tonya
Posts: 862
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Slowness during check outs

Post by Tonya » Tue Apr 07, 2020 8:08 am

Hello,

Are other Vault user's experiencing the performance issue when checking out files? Does the issue occur for all files or only certain ones? Are the files large in size?

Is this issue new? Has anything changed on your network?

I have copied below a KB article we have that may help (particularly #7 Speeding up Client-side Operations):

viewtopic.php?t=4206

How you are doing well as well.

Thanks,

Tonya

joshp8a
Posts: 14
Joined: Thu Mar 27, 2014 11:34 am

Re: Slowness during check outs

Post by joshp8a » Thu Apr 09, 2020 4:26 pm

Thanks for the reply. All users were experiencing the issue. It occurred for all files. This is something new, at least the first time it was noticed. Files were not large at all, I was testing with a file that uses 1KB.

I fixed this by bouncing the whole server. I believe something was happening on the SQL Server end so with the restart, the SQL services bounced as well and the issue went away. As reported initially, at the time of the slowness, the sqlserver.exe process was the only one using most of the memory. Will check other stuff in the article if this happens again.

Thanks for the help!

Tonya
Posts: 862
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Slowness during check outs

Post by Tonya » Fri Apr 10, 2020 7:28 am

Glad to hear the problem has been resolved. Thank you for the update as well.

If the problem returns and you need further assistance, please get back in touch to let us know.

Tonya

joshp8a
Posts: 14
Joined: Thu Mar 27, 2014 11:34 am

Re: Slowness during check outs

Post by joshp8a » Fri May 08, 2020 12:05 am

This issue happened again. CPU and Memory high. Memory up to 95% in the server. IIS process and SQL Server processes are the most active ones. SQL Server process is the most active one. Vault log showing timeouts when attempting operations, a lot of rollbacks. Issue went away after bouncing the server.

Is there another way to be able to troubleshoot this? Any ideas on what to do to pinpoint the problem?

Tonya
Posts: 862
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Slowness during check outs

Post by Tonya » Fri May 08, 2020 6:28 am

Can you please email your Vault Server log to support@Sorucegear.com? Check your SQL Server log as well for any errors.

Thanks,

Tonya

joshp8a
Posts: 14
Joined: Thu Mar 27, 2014 11:34 am

Re: Slowness during check outs

Post by joshp8a » Thu Jun 04, 2020 10:25 am

Tonya, I just saw your reply. Sorry for the delay. I will send over the log once it happens again.

Sadly, I cannot access the SQL DB. After Vault was setup in this new server, the admin pass I selected it is not letting me access it (nor login as admin to the vault page) but I am hoping the vault log will suffice.

Thanks!

Tonya
Posts: 862
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Slowness during check outs

Post by Tonya » Fri Jun 05, 2020 7:25 am

Thank you for the most recent update. When the problem reoccurs, please email us your Vault Server log and refer to this forum post.

Tonya

joshp8a
Posts: 14
Joined: Thu Mar 27, 2014 11:34 am

Re: Slowness during check outs

Post by joshp8a » Wed Jul 15, 2020 8:18 pm

Hi Tonya. I started encountering the issue again. Here are some of the errors in the log. Which email should I send the log to?

Error (replaced the username and server info with USER1 and SERVER fpr the server info, USER2 is a client). Script fired up and started doing getlatest on some of the files (first 4 lines) with a specific user (USER1) then it went into timeout and errors with USER2 session. USER2 was logged in but not doing anything same with other users.

----7/15/2020 6:03:03 PM USER1--SERVER--SSL Disabled Login
----7/15/2020 6:03:08 PM USER1--SERVER--SSL Disabled Logout
----7/15/2020 6:03:10 PM USER1--SERVER--SSL Disabled Login
----7/15/2020 6:03:20 PM USER1--SERVER--SSL Disabled Logout
----7/15/2020 6:34:51 PM USER1--SERVER--SSL Disabled System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.CreateDBLockObject(VaultSqlConn conn, SqlCommand cmd, Int32 nDefaultRetVal)
ClientConnectionId:e2b0df13-d46b-4210-ae26-9e8adc2cb9a1
Error Number:-2,State:0,Class:11 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.CreateDBLockObject(VaultSqlConn conn, SqlCommand cmd, Int32 nDefaultRetVal)
----7/15/2020 6:34:51 PM USER1--SERVER--SSL Disabled Rolling Back a transaction at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
at VaultServiceAPILib.VaultServiceAPI.GetCheckoutListChanges(Int32 nUserID, Boolean bAdminMode, Boolean bStrictCheckoutListFiltering, Int32 nRepID, Int64& nBaseCheckoutListRevision, VaultDateTime& dtLastSecurityChange, Boolean& bRefreshList, VaultCheckOutItemColl& coList, VaultCheckOutReleaseColl& coRelList)
at VaultService.VaultService.GetCheckOutListChanges(Int32 nRepID, Int64& nCheckoutListRevision, VaultDateTime& dtLastLockDate, Boolean& bRefreshList, VaultCheckOutItem[]& coList, VaultCheckOutRelease[]& coRelList)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
at System.Web.Services.Protocols.WebServiceHandler.Invoke()
at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context)
at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

----7/15/2020 6:34:51 PM USER1--SERVER--SSL Disabled GetCheckOutListChanges returned: Failure
----7/15/2020 6:36:24 PM USER2--IP_ADDRESS--SSL Disabled System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.CreateDBLockObject(VaultSqlConn conn, SqlCommand cmd, Int32 nDefaultRetVal)
ClientConnectionId:f623ec10-81f7-4c0f-8c66-cee01453ba88
Error Number:-2,State:0,Class:11 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.CreateDBLockObject(VaultSqlConn conn, SqlCommand cmd, Int32 nDefaultRetVal)
----7/15/2020 6:36:24 PM USER2--IP_ADDRESS--SSL Disabled Rolling Back a transaction at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
at VaultServiceAPILib.VaultServiceAPI.GetCheckoutListChanges(Int32 nUserID, Boolean bAdminMode, Boolean bStrictCheckoutListFiltering, Int32 nRepID, Int64& nBaseCheckoutListRevision, VaultDateTime& dtLastSecurityChange, Boolean& bRefreshList, VaultCheckOutItemColl& coList, VaultCheckOutReleaseColl& coRelList)
at VaultService.VaultService.GetCheckOutListChanges(Int32 nRepID, Int64& nCheckoutListRevision, VaultDateTime& dtLastLockDate, Boolean& bRefreshList, VaultCheckOutItem[]& coList, VaultCheckOutRelease[]& coRelList)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
at System.Web.Services.Protocols.WebServiceHandler.Invoke()
at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context)
at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

Tonya
Posts: 862
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Slowness during check outs

Post by Tonya » Thu Jul 16, 2020 7:05 am

Please send the Vault server log to support@sourcegear.com. Refer to this forum post and if you have any new details to provide us with, please include those as well.

Thanks,

Tonya

Post Reply