Scalability

[pshort]

Anyone out there having major problems with a large amount of users?

We have dozens of developers hitting vault (not all at once), and it's slow and unreliable. We have to reboot our server several times a day, checkout statuses gets wiped out, everyone waits for "Retriving Repository from the Server...", logins takes forgever, and the merge tool is so frustrating and unreliable that everyone merges by hand now. (It's always fun to watch an overly optimistic newbie try to use the tool and fail.) We've stuck with the product for a couple years hoping that our problems will be patched away, but nothing really seems to help.

Anyway, our only hope is to migrate away from this product. We've lost countless hours of productivity, and it's the butt of countless jokes and parodies at the office. Imagine programmers banging their heads at their desks, or shaking their fists and yelling "Vault!!!" a-la Cpt Kirk, Wrath of Khan. For a good laugh we play the voiced-overed vault marketing video while the server is down.

[ericsink]

I see this is your first post. Have you spoken with anyone here at SourceGear about these problems? If you'd like to have a constructive conversation, we are certainly willing to help.

Vault scalability has steadily improved with each release. What version are you using?

[ismangil]

We don't have any performance problems at all. Before deciding on Vault we specifically trialled the scenario we want to use.

We have used Vault since 3.0.6, installed on March 2005. Recently upgraded to 3.1.1. The DB server and and Vault are installed together on a 4x2.8GHz Xeon with 4GB of RAM, with a total of 53 listed users. The server has never been rebooted because of Vault/SQL Server (it was rebooted once for security hotfixes to W2K3 though).

We have development teams in UK (HQ), India and Eastern Europe using VPN, so because of the time difference it is in use most of the day. There are a mixture of pure Vault GUI client users, pure VS2003 IDE integration users, and mixed users.

We also do continuous integration, so there is pretty much constant traffic for history queries, labeling, and gets.

We have also tied Vault into our development process with a plugin to our issues log.

Occasional problems do happen when one of the overseas team tries to submit a large (6MB or over) file, usually due to Internet traffic fluctuations. We get around that by getting the UK team to check it in.

There is also a subtle API history query issue that is affecting the plugin which I am still trying to track down (with fast and excellent support from Sourcegear).

So in summary we do not have performance problems with our current setup.

[ismangil]

the merge tool is so frustrating and unreliable that everyone merges by hand now.

One more thing to add: we do not use merging so can not comment on Vault merging facility.

[pshort]

I see this is your first post. Have you spoken with anyone here at SourceGear about these problems? If you'd like to have a constructive conversation, we are certainly willing to help.

Vault scalability has steadily improved with each release. What version are you using?

Actually, our company has been using SourceGear Vault for several years, back when it was beta. Our build people have been regularly communicating with you guys for years. Our baseline is several million lines of code, and our development staff is large but not humongous. We upgrade regularly and are currently only a patch or two behind. But after years of patching and performance problems, most of us are convinced the product isn't suited for medium or large scale development (50-200 developers).

But we've also noticed reliability problems with lots of code on isolated vault servers with hardly any load (just five people), but the baseline was large.

I suspect mosy of your "happy" cusomers are development teams under 50 people, or baselines that are small (when you branch parts of baselines that are 1, 2, or 3MLOC, 2-5 year histories, labels almost daily, things get big).

Anyway, here are some constructive questions:

1. What are your metrics for scalability testing (concurrent hits, hits against a LARGE baseline, etc.)?

2. Are there any SQL Server indexing or performance things that can be done to improve performance? (We are using the server license on a fast server, not MSDE or the Desktop Edition).

3. Any LARGE customers (our size or bigger) faring any better?

[ismangil]

I suspect mosy of your "happy" cusomers are development teams under 50 people, or baselines that are small (when you branch parts of a baseline that is almost 1.5 MLOC, 2 year history, labels almost daily, things get big).

Yep, you're right, our baseline is not that large.

[ericsink]

1. What are your metrics for scalability testing (concurrent hits, hits against a LARGE baseline, etc.)?

2. Are there any SQL Server indexing or performance things that can be done to improve performance? (We are using the server license on a fast server, not MSDE or the Desktop Edition).

3. Any LARGE customers (our size or bigger) faring any better?

We have several tests which specifically address scalability. Our own repository has over 200,000 files. Our largest test repository has 500,000 files.

We have a "Crowd Test" which simulates large numbers of concurrent users beating on the server. We used this test during the 3.1 dev cycle to make some dramatic improvements in scalability. Folks here tell me that you guys are still back on 3.0.7. That would explain a lot.

However, in your particular situation, even after the 3.1 upgrade there will be room for improvement. We do have a number of customers larger than you which are quite happy, but none of them use Vault in quite the same way as you. Right now, Vault simply doesn't cope very well with your particular situation, which I understand to be:

1. You have lots of users

2. Your repository is really large

3. You branch a LOT. Like, all the time.

1 and 2 are okay, but add 3, and this combination catches Vault at its worst. In Vault 3.x, the branch command is a lot slower than we want it to be. When somebody is branching, all the other users are affected. And, the node count of the repository grows with every branch. It's kind of a vicious cycle.

You'd be a lot happier on Vault 3.1. Or at least, you'd be less unhappy.

Improvements to the branch command are coming in Vault 4.0. I'm hoping you guys can hold out that long. Most all of our unhappy customers have been converted into happy ones. It would be quite satisfying to eventually add you to that list.