Vault Pro & Teamcity Server Hang

If you are having a problem using Vault, post a message here.
Post Reply
PeteM
Posts: 12
Joined: Wed Oct 22, 2008 7:10 pm

Vault Pro & Teamcity Server Hang

Post by PeteM » Tue Sep 11, 2012 6:06 pm

Hello,

We currently have Teamcity 7.1 & Vault Pro 6.0 integration. After Upgrading both to these versions we started to see a TeamCity Server hang with Out Of Memery errors. I logged a call with Teamcity & supplied them with several memory dumps. ( This is the call http://youtrack.jetbrains.com/issue/TW-23565 ) They have said that it appears to be the Java CLC that is used as part of the integration that is causing the issue.

Yesterday I noticed there was a version 6.0.1 available and installed this version of the Java CLC but we are still experiencing the problem.

Can I send some memory dumps to you to help solve this issue?? Do you have any advice regarding this problem?

Many thanks,

Pete.

Beth
Posts: 8550
Joined: Wed Jun 21, 2006 8:24 pm
Location: SourceGear
Contact:

Re: Vault Pro & Teamcity Server Hang

Post by Beth » Wed Sep 12, 2012 8:02 am

We haven't tried integrating with Teamcity, and it's not a supported product. You can send information, but there's no guarantee that we'll have a solution.

There needs to be some background information about this situation.

Try some of the source management actions with either the Java CLC alone or a Vault GUI client alone. Do they behave any differently than when you have TeamCity and Vault integrated?

Do any other machines have problems performing similar actions as the machine that is hanging?

At what actions does it seem to hang?

What is the OS of the TeamCity server?
Beth Kieler
SourceGear Technical Support

PeteM
Posts: 12
Joined: Wed Oct 22, 2008 7:10 pm

Re: Vault Pro & Teamcity Server Hang

Post by PeteM » Wed Sep 12, 2012 8:08 pm

Beth, thanks for the reply, here is some more detail.

We had Teamcity (TC) workinf well with version 5 of Vault Pro. No memory issues until the upgrade to version 6.

When I began seeing the Out of memory issue in TC, I began monitoring the problem there as it appeared the upgrade to 7.1 was the issue. It just so happened the I upgraded to Vault 6 at the same time.

So after issuing TC with the memory dumps, the have said the issue is with the Java CLC. Their Vault API did not change between the prior and upgraded version. I believe this to be a wrapper for the Java CLC.

When I was monitoring the memory usage, it crept up during the day at a steady rate, even when it was not building projects. After they said it was vault, it kinda made sense as TC checks for source changes every 60 seconds. This is the default. So the leak maybe small, but at 1440 time a day, it will hurt!

So for the time being, I have changed this period and given the TC Service some more memory, but the server still needs a restart every day to free up the allocated memory.

To answer your questions :-

Try some of the source management actions with either the Java CLC alone or a Vault GUI client alone. Do they behave any differently than when you have TeamCity and Vault integrated?

We will have to find out how the Java CLC works and write something to make calls several thousand times in order to get a true reflection.

Do any other machines have problems performing similar actions as the machine that is hanging?

No, we havent seen this anywhere else, but it's also unlikely to make the same volume of calls.

At what actions does it seem to hang?

Difficult to say as it all works fine until it runs out of memory, so not a single action that is causing the crash.

What is the OS of the TeamCity server?

The OS is Server 2008 R2

Hope this helps,

Pete.

Beth
Posts: 8550
Joined: Wed Jun 21, 2006 8:24 pm
Location: SourceGear
Contact:

Re: Vault Pro & Teamcity Server Hang

Post by Beth » Thu Sep 13, 2012 7:41 am

Is the Vault server and the Teamcity server on the same machine?

Can you check your Vault server log on the Vault server at %windir%\temp\sgvault\sgvault.log? Are any errors showing up there?
Beth Kieler
SourceGear Technical Support

PeteM
Posts: 12
Joined: Wed Oct 22, 2008 7:10 pm

Re: Vault Pro & Teamcity Server Hang

Post by PeteM » Thu Sep 13, 2012 6:48 pm

No, the Vault & TeamCity servers are on different machines.

No errors in the log, just a whole lot of login / logout.

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Re: Vault Pro & Teamcity Server Hang

Post by lbauer » Mon Sep 17, 2012 9:38 pm

We're aware that Team City created a plugin for Vault a year or two ago. They contacted our developers. with a few questions. However, we've never tested Vault with Team City and are not familiar with its design.

Do you know whether TeamCity is using our Java CLC or our Java API?

For Vault 6, we updated our third party .NET to Java libraries to run with .NET 2.0. Initial testing showed that those Java commands still worked. It could be that the memory allocation tree is handled differently by the .NET to Java bridge for .NET 2.0 and that is causing TeamCity to run out of memory.

Do you know the memory configuration allocated by Team City during the run of the CLC?

Since you're using a Windows server, is it possible to configure TeamCity to run the .NET (Windows) version of the Vault Command Line Client instead of the Java CLC?
Linda Bauer
SourceGear
Technical Support Manager

PeteM
Posts: 12
Joined: Wed Oct 22, 2008 7:10 pm

Re: Vault Pro & Teamcity Server Hang

Post by PeteM » Tue Sep 18, 2012 12:45 am

Linda,

Thanks for the reply.

I believe its using the CLC according to thier install instructions here http://confluence.jetbrains.net/display/TW/Vault

In terms of allocating memory, we can change the settings for the TeamCity service, but not what is allocated to the calls to Vault.

I do not believe its possible to use the windows version rather than the Java one.

I still have the memory dumps that I can send if it helps.

Thanks again,

Peter.

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Re: Vault Pro & Teamcity Server Hang

Post by lbauer » Tue Sep 18, 2012 7:58 am

Go ahead and send the memory dumps. We can't provide support for third-party plugins like TeamCity, but we can at least take a look to see if there's a Vault issue that's easy to fix.

It would also be a good idea for Team City to test their plugin with Vault 6 to make sure any incompatibilties are resolved.
Linda Bauer
SourceGear
Technical Support Manager

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Re: Vault Pro & Teamcity Server Hang

Post by lbauer » Tue Sep 18, 2012 8:41 am

Send the dumps zipped up to support at sourcegear.com, Attn: Linda. Please include a link to this forum post.
Linda Bauer
SourceGear
Technical Support Manager

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Re: Vault Pro & Teamcity Server Hang

Post by lbauer » Mon Sep 24, 2012 10:28 am

It may be possible to change the memory allocation used by the Vault Java Command Line Client.

There are two places where memory can be set.

Using notepad, open the "vault" file in the Java CLC directory, and look for this text: -Xmx512M
This is the memory allocation, in this case 512 MB. Try changing it to something larger, like -Xmx1024M

There's a similar setting in the "vault.cmd" file (open in notepad) : java -Xmx512M

I would suggest changing both these settings and see if that helps with the TeamCity memory issues.
Linda Bauer
SourceGear
Technical Support Manager

PeteM
Posts: 12
Joined: Wed Oct 22, 2008 7:10 pm

Re: Vault Pro & Teamcity Server Hang

Post by PeteM » Mon Oct 08, 2012 7:47 pm

Hello Linda,

Apologies for the delay, I have been on leave.

While I was away, the guys at Teamcity said this to my earlier post http://youtrack.jetbrains.com/issue/TW-23565

"In a while we'll install Vault 6 on out test server and investigate the issue. We'll contact guys from SourceGear if necessary."

So I guess they have taken this issue on and will be in contact with you if required.

Many thanks for your assistance,

Take care,

Pete.

bjin
Posts: 18
Joined: Tue Aug 28, 2007 10:13 am

Re: Vault Pro & Teamcity Server Hang

Post by bjin » Tue Oct 09, 2012 6:32 am

Hi,

I've just updated to Vault 6 and use TeamCity 7.1.1 and I also suffer from these memory leaks.
The only change made to TeamCity installation is the change of Java CLC (Vault Pro 6.0.1) .

Our TC server runs for a just couple of hours before grinding to a halt. I've turned up the JVM memory to 1 Gb without any luck (-Xmx1024M). This is getting very serious indeed and I keep my fingers crossed you guys at SourceGear can resolve this one asap.

Our TC server runs on Win2K8 R2 with 6 Gb of RAM.

Regards
Bjarne

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Re: Vault Pro & Teamcity Server Hang

Post by lbauer » Tue Oct 09, 2012 8:22 am

Team City did contact us more than two weeks ago, but we haven't heard any more since then. You may want to follow up with them, since this involves their product.
Linda Bauer
SourceGear
Technical Support Manager

bjin
Posts: 18
Joined: Tue Aug 28, 2007 10:13 am

Re: Vault Pro & Teamcity Server Hang

Post by bjin » Sun Nov 04, 2012 12:28 pm

I can happily confirm that this problem is resolved in JavaCLC 6.0.1 30613 :D

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Re: Vault Pro & Teamcity Server Hang

Post by lbauer » Mon Nov 05, 2012 12:27 pm

Thanks for the update. That's good news.

One of our developers worked closely with Team City to determine what changes needed to be made to the Java CLC. Your copy is a pre-release build, and the final fix will be in Vault 6.1, due out later this month.
Linda Bauer
SourceGear
Technical Support Manager

Post Reply