WhiteSites Blog
WhiteSites Blog
Over the past few days I ran into issues where my servers would become slow or unresponsive while running backups. After some digging around I am pretty sure it has to do with the Windows File System Cache. Fortunately Microsoft has a Module to help control this issue.
Rather than having to retrive files from the disk every time windows needs to access them. Windows will instead cache files to the servers Memory. In the old days when memory was expensive, this was not something windows could take advantage of. But today with many computers have more than 4GB of memory it makes sense to cache files for performance.
The biggest problem I found was its not very transparent. If you open task manager you can see a list of processes running and how much memory each is using. But when added up they often do not equal the Total Physical Memory usage. Some people would say its bad software that is leaking memory, and in some cases this could be true. But more commonly the memory in use that you don't see is being used by Windows File System Cache.
Even Today Windows has a small flaw in its design. When you use more than 50% of your systems memory, windows will start to use your page file more heavily. The page file is a porition of your hard drive that is reserved to be used as a scratch pad for the system. Think of it as Fake Memory. When your memory usage gets over 50%, windows goes "Ohh Crap glass half empty", better start shuffling stuff off to the the Page File. At this point your system performance degrades, as your hard drive is 1/100 the speed of your memory.
Almost always when I do help desk support for clients that are complaining about their computers slow, its tied to high memory usage over 50%. The solution is most cases is to add more memory to the machine, and or an SSD drive.
In my case the issue is with a few colocated servers. During your scheduled backups its normal to take a performance hit, but what I didn't expect was the impact Windows Server backup would have on the Windows File System Cache.
One of my servers sits with 24 GB of RAM. It runs DB, Mail Server, Stats Server, Web Server, and normally sits treading water at about 20% Memory Usage. At times it might get as high as 30%, but not much more than that. Today while the system was running a little slow, I logged in and noticed the Memory usage was at 55%. But in task manager none of the processes were using anywhere close to an amount to total that. I started to clear out processes, in case there was some kind of memory leak. The only thing running was Windows Server Backup ( wbengine.exe) Using about 13% CPU, and only 200 MB of memory didn't see like a red flag, but it was the only thing running so it had to be tied to the issue. Another red flag was that the backup which started at 10PM was still running and it was 4PM the following day. So I cancelled the backup, and afterwards the memory usuage dropped from 55% down to 18%. The server retuned to normal performance.
To help prevent Windows Server Backup from pushing the file cache too high I have installed a module from microsoft. I set my max cache size to 25%.
Keep in mind this only works for Windows Server 2008 R2 and older.
https://www.microsoft.com/en-us/download/details.aspx?id=9258
UPDATE 4/28/2016
The update seems to have fixed the problem with high memory usage, but I still was having problems with slow system performance. Today the sites on one of my servers was extremely sluggish. I logged into the server the only thing actively using CPU was wbengine.exe Memory usage was only at 32%, so it couldn't be an issue with the page file. I tired to load up the control panel for Windows Server Backup and the problem failed to load. It got to the point where I was forced to reboot the server to get it responsive again. At this point I suspected the problem was with Windows Server Backup.
I already installed this program on another server and it seems to do a much better job than the build in Backup Solution Microsoft provides. Incremental backups are completed in 30 minutes, plus they provide the option to make an additional backup to FTP. I have setup a disk backup locally, plus a file backup of the more information files, plus FTP backup option is enabled for the file backup.
I have 2 seperate 2TB WD green drives the backups are bring written to. the disk/partitian backup to one drive, and the file backup to the second drive.
The first full backup is running now. I will keep an eye on this and report back how it works out.