Shadow Folders causing ASP to restart

[jstarbird]

Vault 5.0.2.18753

I've setup a Shadow Folder and periodically it is causing ASP.Net to crash with an unhandled exception. It's giving the KeySet Does Not Exist error in the Event logs. I've verified all the permissions settings and it is updating files correctly for the Shadow Folder that is setup, I've only setup one so far.
This is causing the Vault service to be restarted fairly frequently so I really need to get this fixed but cannot figure out where the issue is.
The Vault server logs is not showing anything and the Shadow Folder service log is showing this at about the time of the crash:

3/1/2010 12:45:51 PM <generic>: [<No Name>:3520] GetEntryAssembly() returned null; not logging assembly name
3/1/2010 12:45:51 PM <generic>: [<No Name>:3520] Starting Shadow Folder Thread for repository 15.
3/1/2010 12:45:51 PM <generic>: [<No Name>:3520] Shadow Folder Thread for repository 15 started.
3/1/2010 12:45:51 PM <generic>: [<No Name>:3304] Shadow Folder is using the folder, C:\WINDOWS\TEMP\, as a TEMP directory.

And this is from just prior to the above msg:

3/1/2010 12:43:53 PM <generic>: [<No Name>:3376] Starting Shadow Folder Thread for repository 2.
3/1/2010 12:43:53 PM <generic>: [<No Name>:3376] Shadow Folder Thread for repository 2 started.
3/1/2010 12:43:53 PM <generic>: [<No Name>:3228] Shadow Folder is using the folder, C:\WINDOWS\TEMP\, as a TEMP directory.
3/1/2010 12:43:57 PM <generic>: [<No Name>:3228] Vault's Shadow Folders attempting get (by Delta) for Development
3/1/2010 12:43:58 PM <generic>: [<No Name>:3228] Setting last write time on \\pathname\dir1\dir2\dir3\dir4\filename.asp to 3/1/2010 12:00:24 PM

As I said I have only 1 Shadow Folder setup for any repository so I'm not to sure why it's listing two different repositoriesin the log like it shows above.

Some addtional info. Usually when a crash occurs I get 2 and 3 below only but every now and then I get 1 first. Also, note for the 1, the login for the Vault server and for Shadow Folders services have full control over the c:\vaulttemp folder.

This is what is in the Event logs, 3 messges in this order:

1)
Event Type: Error
Event Source: Vault Client Simple Logger
Event Category: None
Event ID: 0
Date: 3/1/2010
Time: 2:30:51 PM
User: N/A
Computer:
Description:
Error writing message "SimpleLogger finalized" to log file c:\vaultTemp\vaultshadowfolderservice.txt. Exception was: Access to the path 'c:\vaultTemp\vaultshadowfolderservice.txt' is denied.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

2)
Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1334
Date: 3/1/2010
Time: 2:30:51 PM
User: N/A
Computer:
Description:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/Root/VaultShadowFolder
Process ID: 3032
Exception: System.Security.Cryptography.CryptographicException
Message: Keyset does not exist

StackTrace: at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr)
at System.Security.Cryptography.SafeProvHandle._FreeCSP(IntPtr pProvCtx)
at System.Security.Cryptography.SafeProvHandle.ReleaseHandle()
at System.Runtime.InteropServices.SafeHandle.InternalFinalize()
at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
at System.Runtime.InteropServices.SafeHandle.Finalize()

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

3)
Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 5000
Date: 3/1/2010
Time: 2:30:54 PM
User: N/A
Computer: LG-VAULT
Description:
EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d691cc, P4 mscorlib, P5 2.0.0.0, P6 4a7cb192, P7 4e1a, P8 6, P9 udta330idobh2roz2ayvlcelag5agtls, P10 NIL.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.



Thanks,
J

[Beth]

Does the behavior change if completely open up permissions on c:\vaultTemp? You might try setting the Everyone account to Full Control just for one test to see. If no failure occurs, then you will know where to tweak the security settings.

Can you open IIS Manager, expand the website that Vault was installed to, right-click VaultShadwoFolder and select properties? Go to the Directory Security tab and click the top Edit button. Verify that you have Anonymous Authentication allowed. If it's not set there, then check that option and click ok. Then check in a change to be shadowed. Any errors come up?

Are you using certificates?

If you aren't using certificates and the first two ideas didn't help, then go to C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys on the server. You might try setting the Everyone account to Full Control just for one test. Any changes in behavior?

On the areas where you set the Everyone account, you can change that back to the settings it had before starting.

While troubleshooting, you could change the settings of the VaultAppPool in the IIS Manager to get it to not recycle on errors. If you look in the VaultAppPool properties, I think you want to go to the Health Tab and uncheck the Enable Rapid-Fail protection, or bump up the errors to a much higher number to cut down the number of recycles.

[jstarbird]

Beth,
I am going to try your suggestions but need to wait until a quieter time, probably this weekend. I just wanted to point out that a permissions issue wouldn't make any sense as the Shadow Folder function does update correctly when I check in a changed file and that the only time I"m getting these errors is when it seems to be checking for any checkins but there is nothing changed. Also, the services are writing to the logs successfully, even the shadow folder log which is the one it's erroring on in one case. The main Vault service and the Shadow Folder service are using the same creditinals as well so it failing to use the C:\vaulttemp is odd as well.
We do not use certificates and the IIS app pool is set to allow anonymous connections.

Anyway, just wanted to post some info just in case some other ideas come to mind over there.


Thanks,
J

[jstarbird]

Beth,
got a chance to try this tonight as it turns out. It's very odd behavior but it does seem to be a permissions issue on the C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys folder. Once I set Everyone to Full control the crashes stopped.
I tried removing and then re-adding the login we're using for the service but still had the same results even though I would give it full control.
I don't think we want to open this folder up to Everyone so is there some reason this is happening that we can adjust something to fix it?


Thanks,
J

[Beth]

If you choose the machine process model, you might give permissions to Network Service and ASPNET. If you chose the custom process model, then give permissions to the same user you have Vault running under.

You might also check what Application Pool the shadow folder virtual directory is set to use. It should be set to the VaultAppPool. Then check what VaultAppPool is running and add that account.

[jstarbird]

Beth,
giving the Network Service access on the App Pool solved that one then oddly it caused the C:\VaultTemp one to fail again for the log file so I added Network Service there as well and now it seems to be working fine.

Thanks for the assistance.


J

[Beth]

Thank you for the update. The additional information might be useful to others as well.