Calls To Add and Rename Take a Long/Are Too Slow

Examples of programs integrating with Vault and Fortress API. Also, information on integrating with NAnt, CruiseControl.Net and other third-party tools

Moderator: SourceGear

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Tue Aug 04, 2009 5:24 pm

When I make calls to ProcessCommandRename or ProcessCommandAdd, it can take up to a minute per call for the call to return. These same calls via the command line (e.g. vault.exe add and vault.exe rename) also take up to 60 seconds per file being added renamed.

My code is pretty simple:

Code: Select all

	ServerOperations.ProcessCommandAdd(parentDir, files);
and

Code: Select all

	ServerOperations.ProcessCommandRename(vaultPath, newName);
What can I do to speed this up?

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Wed Aug 05, 2009 8:23 am

To enable logging, put this code at the start of your program:

Code: Select all

VaultLib.SimpleLogger.Log.EnableClass("all");
VaultLib.SimpleLogger.Log.Enabled = true;
VaultLib.SimpleLogger.Log.LogFilePath = "c:\\logfile.txt"
Looking at the log file will help determine where the slowdown is happening.
Subscribe to the Fortress/Vault blog

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Wed Aug 05, 2009 1:50 pm

Here is the log file. There appears to be a lot of mutex action going on.
Attachments
VaultLog.zip
(1.68 MiB) Downloaded 5182 times

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Wed Aug 05, 2009 3:10 pm

Wow, you're right it's taking a very long time. During the long stretch of <wf> and <mutex> messages, it's scanning all of your working folders for edited files. There are a couple of things you can do to avoid this working folder scan (which happens in the background in the GUI client)

1. Set less working folders. It looks like you have $/ mapped to C:\Build. If you set the workingfolders lower down, you will be able to avoid scanning those directories.
2. Set the "Require Checkout before Checkin" option for the user.
Subscribe to the Fortress/Vault blog

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Wed Aug 05, 2009 3:40 pm

Neither of these are viable solutions for me. I can only move my working directory one level down, as below $/ are all our branches (e.g., $/Current, $/Jul09, etc.). As for working in Check-out mode, well, we don't use Visual Studio integration so I would have to check-out each file by hand, in the Vault GUI. This is unworkable.

I don't understand why vault.exe scans my entire working directory; I don't ask it to. These are the commands I'm issuing:

Code: Select all

// I'm passing in a *specific* list of files to see if there are any pending updates on them
var changeSet = ServerOperations.ProcessCommandListChangeSet(pathsToFiles);

// ...snip...

// checking if a specific file exists in Vault or not
ServerOperations.ProcessCommandListObjectProperties(path);

// ...snip...

// adding specific items to Vault
ServerOperations.ProcessCommandAdd(hashItem.Key, files);

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Thu Aug 06, 2009 10:20 am

I've got a temp build which updates only the items that you're requesting in the ListChangeSet. Please email support at sourcegear.com ATTN: jeremy with a link to this thread, and I'll send it to you.
Subscribe to the Fortress/Vault blog

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Thu Aug 06, 2009 3:11 pm

The ProcessCommandListChangeSet command now works much faster. I'm using it to see if an item has already been added, so I have to call ListChangeSet with the parent directory of the file being added. It also throws an exception when passed a file that doesn't exist in the repository whereas before I think it returned an empty change set.

But the ProcessCommandAdd still looks like it does a full repository scan. I've attached a new log file.
Attachments
VaultLog2.zip
(1.15 MiB) Downloaded 5074 times

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Thu Aug 06, 2009 3:56 pm

Please add this line to the above logging setup lines:

VaultLib.SimpleLogger.Log.IncludeStackTraces = true;

This will generate a huge log file, but will help me track this down better. Thanks.
Subscribe to the Fortress/Vault blog

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Thu Aug 06, 2009 5:13 pm

Here you go. File was too big, so I had to split it in half.
Attachments
Vault20090806153006.002.zip
(1.77 MiB) Downloaded 5044 times
Vault20090806153006.001.zip
(1.96 MiB) Downloaded 5047 times

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Mon Aug 10, 2009 6:49 pm

I'm now having a hard time committing add files, both via the command line and the API. These are my command line invocations:

Code: Select all

> vault.exe add -host vault -repository Repo c:\WorkingFolder c:\WorkingFolder\d69e28dc-c58b-487c-a83c-e118f46ee2c4.txt

> vault.exe commit -host vault -repository Repo c:\WorkingFolder\d69e28dc-c58b-487c-a83c-e118f46ee2c4.txt
The commit fails with the following output:

Code: Select all

<vault>
  <error>
    <exception>System.Exception: No object was found at the repository path: $/d69e28dc-c58b-487c-a83c-e118f46ee2c4.txt
   at VaultClientIntegrationLib.RepositoryUtil.FindVaultTreeObjectAtReposOrLocalPath(String testPath)
   at VaultClientIntegrationLib.RepositoryUtil.MatchWildcardToTreeObjects(String repositoryPath)
   at VaultClientIntegrationLib.ServerOperations.ProcessCommandCommit(String[] objectPaths, UnchangedHandler unchanged, Boolean keepCheckedOut, LocalCopyType localCopy, Boolean resolveMerge, ChangeSetItemColl& csicRemove)
   at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
   at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)</exception>
  </error>
  <result>
    <success>False</success>
  </result>
</vault>
If I commit on the parent directory, however, the commit succeeds:

Code: Select all

> vault.exe commit -host vault -repository core C:\WorkingFolder\
Am I not allowed to commit a specific file? Must I always commit a directory? If I can commit a specific file, what is the proper incantation to do so?

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Tue Aug 11, 2009 7:16 am

Add files are a special case, since there is no object at the repository path yet. Probably the easiest thing to do is to add the -commit flag to your vault.exe add command. That will commit the add file immediately.
Subscribe to the Fortress/Vault blog

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Tue Aug 11, 2009 10:12 am

What about via the API? I don't see any flags on the ProcessCommandAdd that allows me to auto-commit.

Code: Select all

ServerOperations.ProcessCommandAdd(@"c:\WorkingFolder", new string[] { @"c:\WorkingFolder\d69e28dc-c58b-487c-a83c-e118f46ee2c4.txt" })

ServerOperations.ProcessCommandCheckIn(new string[] { @"c:\WorkingFolder\d69e28dc-c58b-487c-a83c-e118f46ee2c4.txt" }, UnchangedHandler.UndoCheckout, false, LocalCopyType.Replace, false);
The second call fails with the same error: unable to find path in repository.

I would prefer to not auto-commit additions. In my case, I have a bunch of modifications, deletions and additions that should get committed/checked-in together.

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Tue Aug 11, 2009 11:42 am

In the API, every ProcessCommand* that could result in a commit (if the ServerOperations.Client.AutoCommit flag is set to true) returns a ChangeSetItemColl collection. You can commit that collection later (with ProcessCommandCommit) or add the items to a ChangeSetItemColl that you keep, and then commit all at once.
Subscribe to the Fortress/Vault blog

splatteredbits
Posts: 18
Joined: Fri Mar 06, 2009 12:27 pm

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by splatteredbits » Tue Aug 11, 2009 11:49 am

I've re-written my commit logic to commit a specific directory. But when I do that, my commit fails with a generic System.Exception: Commit failed message.

Here are the two calls I'm making:

Code: Select all

ServerOperations.ProcessCommandAdd(@"C:\WorkingDirectory\", new string[] { @"C:\WorkingDirectory\newfile.txt" } );

// ...snip...

ChangeSetItemColl committedItems;
ServerOperations.ProcessCommandCommit(new[] {@"C:\WorkingDirectory" }, UnchangedHandler.UndoCheckout, false,		                                      LocalCopyType.Replace, false, out committedItems);
This results in the following exception:

Code: Select all

System.Exception: Commit failed
at VaultClientIntegrationLib.ServerOperations.ProcessCommandCommit(ChangeSetItemColl csic, UnchangedHandler unchanged, Boolean keepCheckedOut, LocalCopyType localCopy, Boolean removeLocalCopy, Boolean resolveMerge, ref ChangeSetItemColl csicRemove)
at VaultClientIntegrationLib.ServerOperations.ProcessCommandCommit(String[] objectPaths, UnchangedHandler unchanged, Boolean keepCheckedOut, LocalCopyType localCopy, Boolean resolveMerge, ref ChangeSetItemColl csicRemove)
at WebMD.Vault.Facade.VaultFacade.Commit(String pathToFolder) in VaultFacade.cs: line 34
at WebMD.Vault.Facade.Tests.VaultFacadeTests.Commit() in VaultFacadeTests.cs: line 99 

In addition, it looks like the call to commit is also doing a scan of the entire working directory. Should it be doing this if I give it a specific directory of changes to commit? I've attached a log file for both the exception and entire working directory scan.
Attachments
Vault20090811103032.zip
(602.16 KiB) Downloaded 4909 times

jeremy_sg
Posts: 1821
Joined: Thu Dec 18, 2003 11:39 am
Location: Sourcegear
Contact:

Re: Calls To Add and Rename Take a Long/Are Too Slow

Post by jeremy_sg » Tue Aug 11, 2009 12:06 pm

The code that I was imagining goes something like this:

Code: Select all

ChangeSetItemColl mycollection = new ChangeSetItemColl();

ChangeSetItemColl temp = ProcessCommandAdd(...);
foreach (ChangeSetItem item in temp)
   mycollection.Add(item);

//Pend the second add.
temp = ProcessCommandAdd(...);
foreach (ChangeSetItem item in temp)
   mycollection.Add(item);

ServerOperations.ProcessCommandCommit(mycollection,....);
The ProcessCommandCommit does not do a full working folder scan. It only refreshes the items that it already knows about.

If you want information on why the commit failed, you need to add a listener to the ServerOperations.UserMessage event.
Subscribe to the Fortress/Vault blog

Post Reply