w3wp.exe high cpu usage
Moderator: SourceGear
We're experiencing exactly the same problems (top-CPU usage with shadowing enabled) here - independent of shadowing on a network drive or locally. CPU usage hits the fan mostly when the structure changes, i.e. when files or folders are being removed or added. As long as single files merely are being checked in, everything runs smoothly. Studying the process performance, what is interesting is, that the amount of I/O operations that are not "read" or "write" (e.g. "stat" and the like) are enormous (see picture). This count easily quadruples the similar mark of McAfee with on-access scanning enabled, which is monitoring the entire filesystem and all accesses. My guess ist that some optimization in comparing the folder/directory structures could be done.
Another symptom seems to be, that after a restart of the IIS-worker process for sgVault, the CPU-usage when changing folder structure tends to be not so high and not so lengthy. We do not have specific measurements and comparisons to this, that is currently just an impression.
Another symptom seems to be, that after a restart of the IIS-worker process for sgVault, the CPU-usage when changing folder structure tends to be not so high and not so lengthy. We do not have specific measurements and comparisons to this, that is currently just an impression.
- Attachments
-
- performance.jpg (254.51 KiB) Viewed 33726 times
Turning off shadow folders also stops the enormous amount of creating and deleting mutants/mutexes named WF_something. Possibly some internal cache created for shadow folder usage steadily grows and causes the increasíngly bad performance. Objective data/measurements for this increase in performance can be found in this topic, where the horrendous performance was also caused by shadow folders - not on a network drive, but locally!
-
- Posts: 71
- Joined: Tue Aug 17, 2004 7:59 pm
- Location: West Coast, USA
There seems to be a correlation not with the number of checkins, but with the number of structure changes, say: adds/removes of files and folders. This seems to trigger a different kind of scan/sync in the shadowfolder service. So, after a restart, things may go well for a long time despite a lot of checkins, until structure changes happen.
We are seeing the same issue on our server. When I shut down shadow folders, things run smoothly. When they are turned on, the server locks (sits there busy and doesn't respond to anyone for up to 45 minutes) about three or four times a day. We're working with the support team, which directed me to this thread, so I'm hoping that when an answer is available, it'll be posted here.
Shadow folders do put an additional load on the Vault server. The shadow folder service is a type of Vault client, which updates the shadow folder contents whenever the shadowed folders are updated.
It is possible to run the Vault Shadow Folder Service on a different machine from the Vault Server, and this can improve performance somewhat. If you're interested in trying this configuration, email linda at sourcegear.com.
It is possible to run the Vault Shadow Folder Service on a different machine from the Vault Server, and this can improve performance somewhat. If you're interested in trying this configuration, email linda at sourcegear.com.
Linda Bauer
SourceGear
Technical Support Manager
SourceGear
Technical Support Manager
I can do that, but I have a couple of observations:
The load on the server comes in bursts. It seems to be fine for a long time and then suddently lock up for a very long time. Our server isn't a slouch; it's a good dual processor machine with lots of ram. Also, the CPU use isn't really that high, so I don't understand why the server process is totally locked up.
We didn't have this problem until version 3.5. Then suddenly (that day) this became an issue. I suspect things. I sort of hate to mess with another server for shadow folders when they seemed to take so little load in past. I wonder if I'll work to get them moved somewhere and then you'll find a bug and fix it, making my work redundant. Are you sure this isn't a bug? It sure looks like it. I can patiently wait a few weeks to see if you get a fix, but if you really think that this is a bedrock performance issue, then I'll go ahead an move things around. So what do you think it is?
The load on the server comes in bursts. It seems to be fine for a long time and then suddently lock up for a very long time. Our server isn't a slouch; it's a good dual processor machine with lots of ram. Also, the CPU use isn't really that high, so I don't understand why the server process is totally locked up.
We didn't have this problem until version 3.5. Then suddenly (that day) this became an issue. I suspect things. I sort of hate to mess with another server for shadow folders when they seemed to take so little load in past. I wonder if I'll work to get them moved somewhere and then you'll find a bug and fix it, making my work redundant. Are you sure this isn't a bug? It sure looks like it. I can patiently wait a few weeks to see if you get a fix, but if you really think that this is a bedrock performance issue, then I'll go ahead an move things around. So what do you think it is?
Acknowledge Problem?
Is SourceGear acknowledging that this problem exists? We see it too
(CPU spike when checking in files) and are checking this forum and the site daily for an update.
Our current setup (shadow folders) is the best solution for us now and
we're not interested in a temporary fix involving moving sourcegear to
addl servers.
Please let us know that SourceGear has replicated or at least
acknowledges that this problem exists so we know you're working on it.
Thank you.
(CPU spike when checking in files) and are checking this forum and the site daily for an update.
Our current setup (shadow folders) is the best solution for us now and
we're not interested in a temporary fix involving moving sourcegear to
addl servers.
Please let us know that SourceGear has replicated or at least
acknowledges that this problem exists so we know you're working on it.
Thank you.
To avoid any confusion, I'll clarify my last post.
We are aware of a problem with Shadow Folders in certain types of repository configurations. We are actively working on this, but have yet to track down the cause of the CPU utilization problem. Once we know what the problem is there will be a fix in the next maintenance release.
If anyone would like to help out we're looking for volunteers to send us some debug logs from Shadow Folders.
To enable Shadow Folder debug logging you will need to
After we have the log file, you can feel free to remove the Debug entries so the Shadow Folder log does not grow too big.
We are aware of a problem with Shadow Folders in certain types of repository configurations. We are actively working on this, but have yet to track down the cause of the CPU utilization problem. Once we know what the problem is there will be a fix in the next maintenance release.
If anyone would like to help out we're looking for volunteers to send us some debug logs from Shadow Folders.
To enable Shadow Folder debug logging you will need to
- Alter web config in the Shadow Folder installation directory to use the following lines. Note, the configurations should already be present, it is just a matter of changing the values:
<add key="enableLogging" value="true" />
<add key="classesToLog" value="shadowfolderdebug,refresh" /> - Run a transaction which triggers Shadow Folders.
After we have the log file, you can feel free to remove the Debug entries so the Shadow Folder log does not grow too big.
Jeff Clausius
SourceGear
SourceGear
We haven't received logs from anyone yet. If you're having problems with high cpu usage when using Shadow Folders, please let us know. Enable Shadow Folder debug logging as described in the previous post and send your Shadow Folder log to support@sourcegear.com.
Thanks.
Thanks.
Linda Bauer
SourceGear
Technical Support Manager
SourceGear
Technical Support Manager