Slow on Multiprocessor machine

[cityone]

This problem is that Vault Server running on a 4-CPU 3 GHz 4GB Ram machine performs much slower than a less sophiscated Test machine (1-CPU, 3GHz, 1GB Ram).

I have done some tests downloading the same amount of source from the same VaultDB. The 4-CPU machine takes a min while the 1-CPU machine takes 22s.

By increasing the Max Work Processes from 1 to 4 in IIS6 on the MP machine, the download speed has improved to about 22s. However, this caused existing users not able to checkin muliple sources at the same time. Also, there are 4 w3wp.exe running and each taking up 400MB+ of RAM.

Is it a known problem that Vault Server runs slow on a MP machine with 1 w3wp.exe worker process? If the number of Max Work Processes is increased on a MP, does it caused checkin of multiple sources and increasing the memory of each w3wp.exe greatly?

Our developers have been complaining Vault is getting very slow. Sometimes, checking in a file takes 5-10 mins. We have about 40users. On avergae 20 are logged on.



Thanks!

[jclausius]

Is it a known problem that Vault Server runs slow on a MP machine with 1 w3wp.exe worker process?

Not really. Our own internal testing takes place on a duo SMP machine, and there have been no indications of performance problems.

We have customers running Vault on quad processors, and nothing has been reported by them either.


There will be a couple of different things you may want to look at:

1) Database / machine maintenance. Is regular maintenance being performed on the sgvault database?

2) What is the configuration of your setup? For larger installations, I would recommend a smaller front end running IIS / Vault Server, and then a second machine using SQL Server. In larger environments, this will be a more responsive environment because the Vault Server is no overly taxed with other requests incur some sort of database interaction.

3) IIS restarts. Check your Vault Server Log for multiple restarts. If your server is restarting unexpectedly, that would explain things.

4) Vault Refreshes. Sometimes if the average transactions / refresh ratio is too high, a slow part of the Vault server is triggered. Currently, the only work-around is to make sure your <TreeManagerSize> in vault.config is set to a value high enough to avoid these cases. With a 4GB machine something like 300 should be a good starting point (-1 is the default). After changing the value, you'll need to restart the Vault server (iisreset.exe).

[cityone]

(1) Yes, DB maintenance is carried out weekly. Also, as I use two Vault Server machines tested against the same VaultDB machine, I would expect the faster Multiprocessor machine to run quicker. But it turns out my 4-CPU machine runs slower.

(2) My configuration is:

Production: Client -> 4-CPU Machine [Vault Server/IIS6/W2P3 SP1] ->{ VaultDB/SQL Server 2000 }
Test: Client -> 1-CPU Machine [Vault Server/IIS6/W2P3 SP1] -> { same box as above }

Both machines runs W2K3 SP1.

(3) Restarted IIS. But seen nothing unexpected:


----14/08/2006 12:12:10 sgvaultsystem--()--
System Started
Version 3.1.7.3719
Cache Level = 1
DataBase Buffer Size (KB) = 256
LogFile Path = C:\WINDOWS\Temp\sgvault
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
Active Directory Domain =
SGVault Working Directory = C:\WINDOWS\Temp
SGVault Server URL = http://BOB/VaultService
Identity = NT AUTHORITY\NETWORK SERVICE



(4) I will try changing TreeManagerSize to 300. Given my setup, Would changing any of the following settings make my 4-CPU machine run Vault Server faster?

<DBBufferSizeKB>256</DBBufferSizeKB>
<TreeManagerSize>-1</TreeManagerSize>
<WebBufferSizeBytes>32768</WebBufferSizeBytes>
<ChainLimit>20</ChainLimit>


The other thing is w3wp.exe run in high CPU (>80%) when there is a client request, which makes Vault cpu-bound on the MP machine. Any idea?

[jclausius]

Nothing looks out of the ordinary. Note, after the change to vault.config's treemanagersize, you'll need to restart the Vault server.

In all our internal testing, and external installations, the Vault Server itself is not CPU bound.

When you say a client request, what kind of request? Is it a commit? Are you by chance running Vault Shadow Folders? Is the answer is yes to both of these, try temporarily removing shadowing from the equation to see what kind of effect that has on your system.

[cityone]

Tried changing TreeManagerSize to 300. Does not seem to make any difference. Downloading a project source takes about 1 min as before compared with about 30s on the test machine. The only way I found would improve the performance on the MP machine is by increasing Max Worker Processes from 1 to say 4. But this cause w3wp.exe eating up lot of memory (400-500MB each). Also, this has caused problems doing multiple checkins. Can you check this on your MP machine? Thanks!

[jclausius]

I just noticed something. Your server is running Vault 3.1.7.

Is it possible, you have a mixture of 1.1 .Net Frameworks vs. 2.0 .Net Frameworks on clients / servers?


Also, you didn't say anything about Shadow Folders? Are there any Shadow Folders configured on this machine?

[cityone]

The server got both .NET 1.1 and 2.0 on it. All clients and the server are 3.1.7. But what do you mean by mixing .NET 1.1 and 2.0 with clients and server? How do I find out?

Yes, it has got shadowing on it. But it cannot be turned off now. Also, it's slow even when there is no activity to the VaultDB (for example after working hours). I have done a timing test for 24 hours and the result shows that it's always slower than my test machine.

[jclausius]

The server got both .NET 1.1 and 2.0 on it. All clients and the server are 3.1.7. But what do you mean by mixing .NET 1.1 and 2.0 with clients and server? How do I find out?

Are you using VS.Net integration or the standard Vault GUI?

Vault GUI -> Help -> Technical support will give you the info on that client as well as the server.

With IDE based clients within Visual Studio, it is a little bit easier:
VS.Net 2003 Integrated Vault client - .Net 1.1
VS.Net 2005 Integrated Vault client - .Net 2.0

----------
Basically there was a bug that Microsoft changed the way datetimes are sent in SOAP in the 2.0 Framework. They changed it in a way that was not backwards compatible with 1.1 .Net Frameorks. This problem can sometimes cause repositories to be re-downloaded on every refresh when machines are using a mixture of .Net Frameworks.

To see if this is the problem, you can place your Vault server in Debug mode. Then the log will contain info if the server is sending a Repository Delta on each and every refresh.

There is a fix for this in Vault 3.5. Another possible work around is to ensure every client/server is on the same .Net Framework version.

[jclausius]

I have done a timing test for 24 hours and the result shows that it's always slower than my test machine.

Slow performance when doing a file GET could also be caused by any active anti-virus or disk fragmentation.

Is it possible the quad SMP machine has anti-virus scanners or severely fragmented disks where the single processor machine does not?


I'll try to see what effect Max Worker Processes has on our SMP machine as well.

[cityone]

No, it has nothing to do with the disk i/o or any virus scanning.

If you can do some performance testing on a quad SMP, comparing with a single processor machine. That's will be great. Also, try doing multiple checkins on it while you have multiple Worker Processes running for VaultAppPool. Our experience has been that the would cause checkin failures. I suspect because w3wp does not share Vault information or locks.

[cityone]

Also, is there any performance improvement in 3.1.9 over 3.1.7. By reading the release note, there does not seem any change that suggests upgrading will result in better performance.

[jclausius]

There were some performance gains in move.

Nothing changed that would affect GET.

Can I get some more info on your two test. Can you place both Vault Servers in Debug Log mode. Do a couple of tests, and send me the results of both logs. I'd like to look at this as a quick profiler to see what is going on.

[jclausius]

Do you happen to have Intel based chips? If so, have you turned OFF hyper threading?

In our testing here, hyper-threading is off due to some problem we've encountered with SQL Server and the Vault stored procedures.

[jclausius]

However, this caused existing users not able to checkin muliple sources at the same time.

What is the Vault Server error message in this case?

Scratch that - Web Gardens are not a supported Vault Server configuration, so any type of error would be expected.

[jclausius]

OK. Some preliminary results.

The Max Processor setting is a part of IIS' Web Garden feature. This will not work as Cached Application information is not shared between processes. This value needs to be configured to 1, so the Web Garden is not used.