Vault Performance / Hardware

If you are having a problem using Vault, post a message here.

Moderator: SourceGear

Post Reply
hwapdc
Posts: 35
Joined: Thu Dec 09, 2010 6:11 am

Vault Performance / Hardware

Post by hwapdc » Fri Jun 15, 2018 4:22 am

Hi
Due to increasing activity CPU usage sometimes skyrocks to 98% for several hours. Vault Server is running at Windows Server (VMWARE Guest with 4 vCPU). I'm considering adding 4 or 8 vCPU. Do I need to reconfigure IIS or reinstall Vault Server after adding more vCPU?

Tonya
Posts: 866
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Vault Performance / Hardware

Post by Tonya » Fri Jun 15, 2018 8:33 am

Hi there,

Before you proceed with addition, we have some questions we'd like to ask. The problem may be resolved without adding more vCPU.

What are your maintenance plans for SQL Server? Are they working correctly? Sometimes tweaking a maintenance plan will improve overall performance.

How many repositories are active and how large (number of files/folders, versions are irrelevant) are in each repository? How many transactions do you commit a minute? We may be able to apply some changes in your vault.config that may help.

Do you know if ALL of your users have *unchecked* the "Request database delta on repository cache miss" on the "Network Settings" options?

Have you been able to determine the source of the CPU load? Vault Server (w3wp.exe) or SQL Server? Is SQL Server on the same virtual machine as Vault?

Thanks,

Tonya

hwapdc
Posts: 35
Joined: Thu Dec 09, 2010 6:11 am

Re: Vault Performance / Hardware

Post by hwapdc » Fri Jun 15, 2018 9:06 am

Hi
The issue is not the SQL Server. The SQL server is running at dedicated Windows Server. We have carefully monitored the performance of the SQL server. The SQL is waiting to deliver the results to the Vault Server.
Vault server is running at dedikated Windows Server only running Vault Web Service. Server has 4vCPU and 32 GB RAM (only 10 GB is used). When Vault Service is busy proces w3wp.exe is consuming most of the CPUs.


I just need to know if I need to reconfigure IIS or other setting to improve the performance.

vault.config:
<?xml version="1.0" encoding="utf-8"?>
<!-- The path to this file must be set in web.config -->
<appSettings>
<WorkingDirectory>E:\SGVault\Temp</WorkingDirectory>
<CachingStrategy>1</CachingStrategy>
<FileCacheEnabled>False</FileCacheEnabled>
<FileCachePath>E:\SGVault\Temp\sgvault\filecache</FileCachePath>
<FileCacheDiskSizeMb>100</FileCacheDiskSizeMb>
<AuthHeader>_authHdr</AuthHeader>
<VaultContentType>application/octet-stream</VaultContentType>
<DownloadQryParam>DownloadToken</DownloadQryParam>
<TxQryParam>TxToken</TxQryParam>
<FileQryParam>FileToken</FileQryParam>
<ShelveQryParam>ShelveToken</ShelveQryParam>
<DBBufferSizeKB>512</DBBufferSizeKB>
<TreeManagerSize>-1</TreeManagerSize>
<WebBufferSizeBytes>32768</WebBufferSizeBytes>
<ChainLimit>20</ChainLimit>
<FullText>50</FullText>
<VaultContentLengthHeader>Content-Length</VaultContentLengthHeader>
<LogLevel>Quiet</LogLevel>
<OverWriteLogOnStartup>False</OverWriteLogOnStartup>
<ArchiveLog>Daily</ArchiveLog>
<LogFilePath>E:\sgvault\Log</LogFilePath>
<LogReverseDNSLookup>True</LogReverseDNSLookup>
<SqlCommandTimeout>3600</SqlCommandTimeout>
<SaveFullTreeInterval>2000</SaveFullTreeInterval>
<VerifyTreeIntegrity>False</VerifyTreeIntegrity>
<TreeDeltaCompresionThreshold>1000</TreeDeltaCompresionThreshold>
<MaxDownload>10240000</MaxDownload>
<GetFileCheckBatchSize>256</GetFileCheckBatchSize>
<DelayThreshold>10</DelayThreshold>
<DelayDurationSeconds>30</DelayDurationSeconds>
<CheckoutListDaysToExpire>5</CheckoutListDaysToExpire>
<AllowRepositories>*</AllowRepositories>
<FindinFilesWordChars>A-Za-z0-9_</FindinFilesWordChars>
<MaxFileSizeBytes>-1</MaxFileSizeBytes>
<ServerURL>https://Vault.pdc.xx:443/VaultService</ServerURL>
</appSettings>

Tonya
Posts: 866
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Vault Performance / Hardware

Post by Tonya » Fri Jun 15, 2018 1:47 pm

Thank you for the details. In order to provide you with further suggestions, we are going need answers to the following questions:

1) How many repositories are active and how large (number of files/folders, versions are irrelevant) are in each repository?
2) How many transactions are committed a minute?
3) Perhaps temporarily turning on debug logging on the server, and taking a look at the log file may give us an indication where Vault Server is spending its time.
4) Do you know if ALL of your users have *unchecked* the "Request database delta on repository cache miss" on the "Network Settings" options?

We're unsure that changing the number of virtualized CPUs won't require IIS re-configuration, this is sort of out of our expertise. Searching the internet may provide additional information to confirm any behavior.

Thanks,

Tonya

hwapdc
Posts: 35
Joined: Thu Dec 09, 2010 6:11 am

Re: Vault Performance / Hardware

Post by hwapdc » Mon Jun 18, 2018 7:27 am

Yesterday I added more vCPU to the Windows Server running the Vault Web Server, so far it seems to be running OK. I'l look

Tonya
Posts: 866
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Vault Performance / Hardware

Post by Tonya » Mon Jun 18, 2018 7:51 am

Glad to hear. Thank you for the update.

Tonya

hwapdc
Posts: 35
Joined: Thu Dec 09, 2010 6:11 am

Re: Vault Performance / Hardware

Post by hwapdc » Tue Jun 19, 2018 8:19 am

Hi
SG Support did ask:
4) Do you know if ALL of your users have *unchecked* the "Request database delta on repository cache miss" on the "Network Settings" options?

At PDC some developers has this option checked some hasn't. What is the recommended setting of this option?

Tonya
Posts: 866
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Vault Performance / Hardware

Post by Tonya » Tue Jun 19, 2018 10:26 am

Since you are experiencing performance issues, we recommend the option "Request database delta on repository cache miss" to be disabled.

Thanks,

Tonya

Hans Olav Nymand
Posts: 55
Joined: Wed Sep 29, 2004 8:09 am
Location: Denmark, Copenhagen
Contact:

Re: Vault Performance / Hardware

Post by Hans Olav Nymand » Wed Jun 20, 2018 1:11 am

Hi Tonya,

Below is some additional info in regard to the original post from Henrik:
"Due to increasing activity CPU usage sometimes skyrocks to 98% for several hours."

We seem to have tracked this - often or always - correspondinng to one or more clients "hanging" during get latest with a CPU utilisation well above 90% all the time (for that one CPU used by that client). I am working to get a specific example with client log and server log.

But - should I post this as an independent issue (an error) to avoid cluttering this specific discussion on performance analysis and hardware scoping. Or should I post my findings in this thread - what do you prefer ?

Best regards, Hans Olav

Hans Olav Nymand
Posts: 55
Joined: Wed Sep 29, 2004 8:09 am
Location: Denmark, Copenhagen
Contact:

Re: Vault Performance / Hardware

Post by Hans Olav Nymand » Wed Jun 20, 2018 6:49 am

Does the "Request database delta on repository cache miss" option in the UI also apply to the command line client (on the same machine)

/Hans Olav

Tonya
Posts: 866
Joined: Thu Jan 20, 2005 1:47 pm
Location: SourceGear

Re: Vault Performance / Hardware

Post by Tonya » Wed Jun 20, 2018 8:39 am

Hi Hans,

The answer to your most recent question, Does the "Request database delta on repository cache miss" option in the UI also apply to the command line client (on the same machine), is yes ONLY if the user is logged into their machine under a windows account.

Now on to the performance issue. Did adding the additional vCPU not resolve the problem? Why don't you email me directly with your new findings along with the log files. Please respond to support@sourcegear.com and reference this forum post.

Thanks,

Tonya

Post Reply