Critical: Fortress/Vault do not start anymore

[Xavier]

Hello

here is a snippet of the log file
----02/10/2009 16:22:01 sgvaultsystem--()--
System Started
Version 2.0.1.18729
Cache Level = 1
DataBase Buffer Size (KB) = 256
LogFile Path = C:\Windows\Temp\sgvault
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = True
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 10080
SGVault Working Directory = C:\Windows\Temp
SGVault Server URL = https://----
Identity = AUTORITE NT\SERVICE RÉSEAU
----02/10/2009 16:27:01 --()-- Uncaught Exception: Could not continue scan with NOLOCK due to data movement.
----02/10/2009 16:27:01 --()-- à System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
à System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
à System.Data.SqlClient.SqlDataReader.HasMoreRows()
à System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
à System.Data.SqlClient.SqlDataReader.Read()
à VaultServiceSQL.VaultSqlSCC.GetRepositories(VaultSqlConn conn, Int32 nUserID, Boolean bAdminMode, VaultRepositoryInfoColl& vrArray)
à VaultServiceAPILib.VaultServiceAPI.GetRepositories(Int32 nUserID, Boolean bAdminMode, VaultRepositoryInfoColl& reps)
à VaultServiceAPILib.VaultRepositoryInfoCache.Reset()
à VaultServiceAPILib.VaultRepositoryInfoCache..ctor()
à VaultService.Global.Application_Start(Object sender, EventArgs e)

Any idea ???

This occured after an unsuccesful shelve: vault was no more accessible (either WEB or GUI client), we used iisreset and got the above error
Below is the shelve error:
----02/10/2009 16:04:07 Eric--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled System.Data.SqlClient.SqlException: Time-out occurred while waiting for buffer latch type 2 for page (1:369), database ID 5.
à System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
à System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
à System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
à System.Data.SqlClient.SqlDataReader.get_MetaData()
à System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
à System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
à VaultServiceSQL.VaultSqlSCC.ShelveSetExists(VaultSqlConn conn, Int32 userid, Int32 repid, String strShelvesetName, Int64& ssid) à System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
à System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
à System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
à System.Data.SqlClient.SqlDataReader.get_MetaData()
à System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
à System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
à VaultServiceSQL.VaultSqlSCC.ShelveSetExists(VaultSqlConn conn, Int32 userid, Int32 repid, String strShelvesetName, Int64& ssid)
----02/10/2009 16:07:29 Sébastien--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled Login
----02/10/2009 16:08:01 Eric--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled Login
----02/10/2009 16:09:08 Eric--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled System.Data.SqlClient.SqlException: Time-out occurred while waiting for buffer latch type 2 for page (1:369), database ID 5.
à System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
à System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
à System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
à System.Data.SqlClient.SqlDataReader.get_MetaData()
à System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
à System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
à VaultServiceSQL.VaultSqlSCC.SaveShelveSet(VaultSqlConn conn, ShelvesetDetails ss, Int32 nUserID, String strXml, Int64& ssid) à System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
à System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
à System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
à System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
à System.Data.SqlClient.SqlDataReader.get_MetaData()
à System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
à System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
à System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
à System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
à VaultServiceSQL.VaultSqlSCC.SaveShelveSet(VaultSqlConn conn, ShelvesetDetails ss, Int32 nUserID, String strXml, Int64& ssid)
----02/10/2009 16:09:08 Eric--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled Rolling Back a transaction à VaultServiceSQL.VaultSqlConn.RollbackTransaction()
à VaultServiceAPILib.VaultServiceAPI.EndSaveShelveSet(VaultIntShelve vis, Int32 nUserID, Int64& ssid)
à VaultService.VaultService.EndSaveShelveSet(String strShelveID, Int64& ssid)
à System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
à System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
à System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
à System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
à System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
à System.Web.Services.Protocols.WebServiceHandler.Invoke()
à System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
à System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context)
à System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context)
à System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
à System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
à System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
à System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
à System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
à System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
à System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
à System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
à System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
à System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

----02/10/2009 16:09:08 Eric--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled EndSaveShelveSet returned: FailDBReader
----02/10/2009 16:09:08 Eric--LRouen-152-82-7-62.w80-13.abo.wanadoo.fr(80.13.102.62)--SSL Enabled EndSaveShelveSet returned: FailDBReader


Xavier

[lbauer]

We're looking into these errors, as we haven't had reports of this before. Please don't make any changes or restart any Vault/SQL services until we get back with you.

[Xavier]

I know I shouldn't have (as I have called you), but I tried to restart SQLServer and it worked after some minutes (after the cache was built I guess)

Is it possible to do sanity checks to know if we've lost some data ?

Thanks

[lbauer]

I emailed you some instructions for a DBCC CHECKDB query, to check database integrity. Although at this point, the database is in a different state, so this may or may not tell us anything useful.

[Xavier]

dbcc CheckDB ('sgvault')
=>
Msg 5030, Level 16, State 12, Line 1
The database could not be exclusively locked to perform the operation.
Msg 7926, Level 16, State 1, Line 1
Check statement aborted. The database could not be checked as a database snapshot could not be created and the database or table could not be locked. See Books Online for details of when this behavior is expected and what workarounds exist. Also see previous errors for more details.

Vault starts, but db is still locked somewhere....

[Xavier]

Now the server fails to start with:
----07/10/2009 10:02:39 --SSL Enabled System.Data.SqlClient.SqlException: Cannot open database "sgvault" requested by the login. The login failed.
Login failed for user 'AUTORITE NT\SERVICE RÉSEAU'.

That seems logical as I changed the service under which SQLServer runs...

I will try to switch back to the former user (instead of local account as you suggested) and may be reinstall fortress and specify the former account. I'm getting lost.

[lbauer]

SourceGear staff worked with you online to restore Vault operations. Let us know if you need further assistance.