We have been having issues with Visual Studio Lab Management 2012 and SCVMM 2012 SP1 where the SCVMM service crashes when MTM queues asynchronous database jobs (e.g. creating/reverting a snapshot of an entire environment).
The error logged in SCVMM (on the first failed database job) is:
Error (2604)
Database operation failed.
Recommended Action
Ensure that the SQL Server is running and configured correctly, and try the operation again.
The error logged in Windows is:
Virtual Machine Manager (vmmservice:5036) has encountered an error and needed to exit the process. Windows generated an error report with the following parameters:
Event:VMM20
P1(appName):vmmservice
P2(appVersion):3.1.6020.0
P3(assemblyName):Utils
P4(assemblyVer):3.1.6020.0
P5(methodName):M.V.U.ImgLibCatalog.GetAppLock
P6(exceptionType):M.V.DB.NonFatalDbException
P7(callstackHash):3729
And the stack trace of the exception (found at C:\Users\<scvmm service account>\AppData\Local\Temp\SCVMM.{GUID}\report.txt) goes something like:
Microsoft.VirtualManager.DB.NonFatalDbException: Database operation failed.
Ensure that the SQL Server is running and configured correctly, and try the operation again.
at Microsoft.VirtualManager.Utils.ImgLibCatalog.GetAppLock(String lockName)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.GetResourceAppLock(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.SharableLibObject.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateChildObjects[T](List`1 childIds, List`1 childContainer, ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.VMBase.UpdateChildObjects(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.CommonVMBase.AddOrUpdateObjectInDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.VM.AddOrUpdateObjectInDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.SharableLibObject.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(Boolean audit)
at Microsoft.VirtualManager.Engine.VmOperations.CheckpointSubtask.Recover(Stack`1 childExecutionStack)
at Microsoft.VirtualManager.Engine.TaskRepository.SubtaskBase.Recover(ITaskContext context, Stack`1 childExecutionStack)
at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.RecoverLiveTask()
at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
*** Carmine error was: DatabaseNonFatalError (2604); 0
--- Caller StackTrace -----------------
FileName:; Method:GetAppLock(); lineNo:0; ilOffset:104.
FileName:; Method:GetResourceAppLock(); lineNo:0; ilOffset:111.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:120.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:25.
FileName:; Method:UpdateChildObjects(); lineNo:0; ilOffset:42.
FileName:; Method:UpdateChildObjects(); lineNo:0; ilOffset:227.
FileName:; Method:AddOrUpdateObjectInDB(); lineNo:0; ilOffset:65.
FileName:; Method:AddOrUpdateObjectInDB(); lineNo:0; ilOffset:563.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:241.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:25.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:26.
FileName:; Method:Recover(); lineNo:0; ilOffset:313.
FileName:; Method:Recover(); lineNo:0; ilOffset:93.
FileName:; Method:RecoverLiveTask(); lineNo:0; ilOffset:96.
FileName:; Method:SubtaskRun(); lineNo:0; ilOffset:147.
FileName:; Method:RunInternal(); lineNo:0; ilOffset:112.
FileName:; Method:Run(); lineNo:0; ilOffset:0.
FileName:; Method:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem(); lineNo:0; ilOffset:35.
FileName:; Method:Dispatch(); lineNo:0; ilOffset:162.
We have spoken with MS about this and apparently it's a known issue, this is a quote from an SCVMM tech:
This is a new bug. VMM is taking an Applock on the Snapshot object, and the AppLock being taken only consists of the ID of the Host machine where the snapshot resides and the snapshot’s name. It seems that across your multiple VMs, many of the snapshots share the same name. So when you try to restore multiple snapshots at the same time, multiple of these jobs try to take the same lock and this consequently results in a failure.
The difference between SCVMM 2008 R2 and SCVMM 2012 SP1 is that we extended the length of time the AppLock is taken to encompass the entire SQL transaction; as the old code, even with the Applock, could lead to race conditions which would cause multiple entries of the same object to get added to the DB. Consequently, the user will be much more likely to see this issue in SCVMM 2012 SP1.
So, it's a known bug that is supposed to be fixed with the next update (whatever that means). Inquiries into whether a hotfix (whether for MTM [to make the call to SCVMM without the -RunAsynchronously tag] or for SCVMM [to fix the AppLock issue] or for SQL
[to change the procedure for accepting/responding to calls]) can be downloaded upon request have gone unanswered.
I was told, by the tech, to create this post so that any updates from the MS side can be made on here. Also, it should serve to help others who are also having this issue--you're not alone!