Unable to upload large files.

[unreal]

Our company is using Vault 4.1.4.18402 server and client for source control. The server is hosted on an SBS 2008 machine running IIS 7. I have 4 files ranging from 25 to 55MB in a VS2008 project under source control.

Originally the files would just timeout and fail to upload (sorry, I don't have the original messages.) I tried to turn on chunk encoding (per the KB) but resulted in the Vault client hanging while waiting for a response. I left it there for 45 minutes to see if it would finally complete but it never did.

Eventually turning off chunk encoding allowed one file to transmit but the rest failed after that. The successful file wasn't even the smallest either. The Vault server is setup correctly via the web interface and has the limit set at 100MB. This is all on a local network; 50MB uploads take a matter of seconds. In addition, I am able to check-in all other files except for these big ones.

I verified that IIS wasn't recycling the application process per the IIS log and confirmed that IIS request limits were not an issue by adding this to the web.config file for the VaultService directory...

<system.webServer>
<security>
<authentication>
<windowsAuthentication enabled="true" />
</authentication>
<requestFiltering>
<requestLimits maxAllowedContentLength="1000000" />
</requestFiltering>
</security>
</system.webServer>

The latest error that the Vault client is reporting is...

[3/18/2009 5:45:57 PM] Preparing data to begin transaction
[3/18/2009 5:45:58 PM] Beginning transaction
[3/18/2009 5:45:58 PM] Add $/Palo/Installer/Resources/SQLEXPR.EXE
[3/18/2009 5:46:15 PM] Upload for item $/Palo/Installer/Resources/SQLEXPR.EXE failed, retrying...
[3/18/2009 5:46:33 PM] Upload for item $/Palo/Installer/Resources/SQLEXPR.EXE failed, retrying...
[3/18/2009 5:46:50 PM] Upload for item $/Palo/Installer/Resources/SQLEXPR.EXE failed, retrying...
[3/18/2009 5:46:50 PM] Ending the transaction
[3/18/2009 5:46:56 PM] Transaction failed
[3/18/2009 5:46:50 PM] Upload for item $/Palo/Installer/Resources/SQLEXPR.EXE failed too many times, aborting transaction.
Please verify your network settings using the Options dialog under the Tools menu in the Vault GUI Client.
The specific error was "The server had an error processing the file upload but did not return an error header. The remote server returned an error: (404) Not Found."
An exception was encountered during the transaction. Exception: The server had an error processing the file upload but did not return an error header. The remote server returned an error: (404) Not Found. at VaultClientOperationsLib.ClientInstance.UploadItem(ChangeSetItem item, String txID, Byte[]& streamBuffer, Int32& bytesWrittenThisFile, Boolean bIsImport)
at VaultClientOperationsLib.UploadThread.ProcessCommand(UploadThreadCommand command, UploadThreadCommandResult& outputResult)
[3/18/2009 5:46:56 PM] Transaction failed

Here is the result of the server log as well; I altered the user column and repository path for this public post...

----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled ASMX: BeginTx started.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled ASMX: DoBeginTx started.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled BeginTx beginning transaction
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled DB: GetUserRepositoryLastChange started.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled DB: GetUserRepositoryLastChange finished.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled DB: GetLatestRepositoryInfo started.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled DB: GetLatestRepositoryInfo finished.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled TreeManager: cache matches repository revision and folder security hasn't changed. Returning cached tree, revID 7317
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled BeginTx: Applying AddFile to $/Palo/Installer/Resources/SQLEXPR.EXE
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled BeginTx: Finished Applying AddFile to $/Palo/Installer/Resources/SQLEXPR.EXE
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled (523b8870-2b4b-497e-a290-38fd9d9b9db3) BeginTx returned: Success
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled (523b8870-2b4b-497e-a290-38fd9d9b9db3) AddFile: $/Palo/Installer/Resources/SQLEXPR.EXE returned: Success
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled ASMX: DoBeginTx finished.
----3/18/2009 5:45:58 PM user--user(192.168.10.17)--SSL Disabled ASMX: BeginTx finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled Ending transaction
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled ASMX: EndTx started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled EndTx(): Client aborted the transaction.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: DeleteSessionBranchSandbox started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: DeleteSessionBranchSandbox finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled (523b8870-2b4b-497e-a290-38fd9d9b9db3) EndTx (Revision - 0) returned: Success
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled ASMX: EndTx finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled ASMX: GetRepositoryStructure started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled Getting repository Structure-> Rep ID: 2 Base: 7317 Target: -1
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetUserRepositoryLastChange started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetUserRepositoryLastChange finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetLatestRepositoryInfo started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetLatestRepositoryInfo finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled TreeManager: cache matches repository revision and folder security hasn't changed. Returning cached tree, revID 7317
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetLastTreeSecurityChange started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetLastTreeSecurityChange finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled VaultServiceAPI::GetRepositoryTreeDelta() UserID:6 RepID:2 Base:7317 Target:7317 Calling VaultRepUtil.DiffRepTrees() - in-memory tree diff.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled ASMX: GetRepositoryStructure finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled GetRepositoryStructure returned: Success
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled Getting list of checkout changes.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled ASMX: GetCheckOutListChanges started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetUserRepositoryLastChange started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetUserRepositoryLastChange finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled Beginning SQL transaction 48253981
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled GetCheckoutListChanges: Transaction Started
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: LockCheckoutList started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: CreateDBLockObject started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: CreateDBLockObject finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: LockCheckoutList finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetCurrentCheckoutInfo started.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled DB: GetCurrentCheckoutInfo finished.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled SQL transaction 48253981 successfully committed.
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled GetCheckoutListChanges: Transaction Committed
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled GetCheckOutListChanges returned: Success
----3/18/2009 5:46:56 PM user--user(192.168.10.17)--SSL Disabled ASMX: GetCheckOutListChanges finished.
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled ASMX: SetUserOptions started.
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled DB: SetUserOptions started.
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled Beginning SQL transaction 52638671
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled SQL transaction 52638671 successfully committed.
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled DB: SetUserOptions finished.
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled SetUserOptions returned: Success
----3/18/2009 5:46:59 PM user--user(192.168.10.17)--SSL Disabled ASMX: SetUserOptions finished.

I thought this might be an issue with memory space on the server but the performance monitor the server did not show any maxing out of memory or disk I/O. I searched all over the forum and KB for an answer but none of them seem to work.

[tjruska]

I had similar issues too (v3.5.3). Try checking the box under Network settings to "Do not use a proxy". Also I had to up the values in the Vault service web.config for executionTimeout and maxRequestLength to 86400 and 1048576 respectively.

Now we are able to upload files up to 400M in size. Don't ask why

HTH, Tom

[unreal]

Thanks for the suggestions. I turned off the default proxy settings in the client and upped the values in my Web.config, restarted IIS and tried again, but to no avail. Any other ideas?

[Beth]

In the network options of a Vault GUI client, try flipping the options in the radial buttons, of course, except chunked encoding. Try turning off the other items in network options or just one at a time. Then let me know your results.

Are you trying to load those items through VS or just using the Vault GUI client?

[unreal]

I have tried through the IDE and the Vault client with the same result. I mainly use the client to troubleshoot this issue since it is the most verbose.

I turned off all the options and set proxy to both "Do not use a proxy." and "Use the system default proxy." in two tests, both failed. I tried this same combination but with "Use Expect..." and
"Request database delta..." turned on, still failed. I then did the same combinations as before but with chunked encoding turned on, same result.

The progress indicator goes all the way to 100% but then reports the failure and retries two more times before aborting. This has always been the behavior.

[Beth]

Did you try once with all the network options off?

[unreal]

With the proxy radio buttons, I have to select an option. Although, as stated before, I tried with all other options turned off and a combination of different proxy selections each time. All yielded the same result.

[Beth]

Could you send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread? I think we need to look a little closer.

[jmlange]

We had this same problem back in the November/December timeframe. The fix was to change a registry value that allowed longer than 10 minutes for an upload.

Hope this helps.

[lbauer]

This registry setting is HKEY_CURRENT_USER\Software\SourceGear\Vault\Client\Settings. Look for the key ServiceRequestTimeoutSeconds. This was implemented in Vault 4.1.4.

[unreal]

Thanks for the suggestion. I checked this entry and it is currently set to 2 hours (7200.)

[lbauer]

Sorry, I gave you the wrong item to change. Same registry key, HKEY_CURRENT_USER\Software\SourceGear\Vault\Client\Settings, but the item is UploadTimeoutSeconds. The default is 600, so try increasing it.

Can you upload smaller files? Is it just the large files that are failing?

Can you upload the file(s) from a Vault Client on the server machine using "localhost" for the server name? This will bypass any network issues.

[unreal]

I doubled the registry value to 1200 and still received the same error. This issue is only with large files (see my first post for details.) I attempted to upload the same files from the server itself using Vault Client (as localhost) and it reported the same exact error as before.

[Beth]

All the information I can find on IIS7 suggests a very low upload limit. I think you were getting close with what you were trying initially. Here's a list of the things I could find for increasing the IIS7 upload limit.

1) Increase the limits properties on the website.

• -Open Up IIS Manager
  -Click on ASP
  -Expand Limits Properties
  -Change the Maximum Requesting Entity Bpdy Limit to a much larger value. I would suggest adding 4 more 0's.

The rest will be in the next post.

[Beth]

2) Edit the IIS metabase. I know that there are tools included in the IIS7 Resource Kit that can do this.

You might give the suggestion found here a try: http://www.chrisbunk.com/archives/2005/ ... g_iis.html

Overcoming IIS http post file upload limit

• -To solve this problem go to C:\windows\system32\inetsvr and edit metabase.xml after stopping IIS.
  -The default setting of AspMaxRequestEntityAllowed is set to 204800. To
  allow for larger upload, increase this setting. Example: To increase to 1MB
  then 5 * 200kb = 1MB. Multiply 204800 * 5 =1024000 so change the setting
  to 1024000.

3) Return to your original idea of adding the following to your web.config, but add about 3-4 more 0's to the end of the number. Some other website that I don't recall now suggested this was in bits instead of bytes, which could make a difference. Make sure you perform an iisreset to force the change to be loaded.

<requestFiltering>
<requestLimits maxAllowedContentLength="2000000000" />
</requestFiltering>