Thursday, October 29, 2009

Case of rogue audit log

We are very lucky to have System Center Operations Manager in our shop. I can easily access tons of information that is typically hidden from DBA. At some point, I was able to find this strange phenomenon. One of my servers had low disk space and I started researching this. Turns out, msdb grew to 16gb. I started looking into this problem and found a table with a name Apex in msdb database. The table alone was 16gb. I googled the named of the table and found out that this is a log auditing application. It was removed awhile ago but add remove by itself won't cut it. There are server-side traces and events enabled as well. The funny thing is that this application is only installed on one node and not the other. Extended Stored Procedures were called every time SQL would start on this node. I followed instructions I found on Google to remove the extended procedures, DLL, exe and some left-over stored procedures, then I truncated the table and deleted it. My msdb is now 35mb. Whew.

Watch out, this application gets installed in msdb database, if you have it in your environment, monitor the size of your msdb!

Good luck.
Post a Comment