Hello All,
I am running SCVMM 2019 on a Server 2019 vm. I have SQl 2016 running on a separate dedicated VM. I have 19 hosts connected to my scvmm and all of these hosts are hosting 210 vm's. I was running SCVMM 2012 and I upgraded to SCVMM 2016. Everything was
running fine before and after the upgrade. I have since upgraded Scvmm to 2019. All of my hosts have been upgraded to 2019 or 2016. My vm hosting scvmm has 8 virtual processors and dynamic memory allowing a max memory of 20GB. Since I have done the upgrade
to 2019 the service maxes out the memory and the client will not run. It is a slow progression to 100% memory. It takes about 10 min or so to reach max mem. I have added the reg setting for TaskGC
and I have ran the following:
DECLARE @CutoffDateTime datetime
SELECT @CutoffDateTime = '10/10/2019'
EXECUTE [VirtualMachineManagerDB].[dbo].prc_TR_GarbageCollect @CutoffDateTime
to clear out the old log data and this has not changed the results.
The result of the SCVMM log file are below:
----------------------------------------------------
------------------- Error Report -------------------
----------------------------------------------------
Error report created 10/25/2019 9:14:13 AM
CLR is not terminating
----------------------------------------------------
--------------- Bucketing Parameters ---------------
----------------------------------------------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=10.19.1013.0
P3(assemblyName)=unknown
P4(assemblyVer)=0.0.0.0
P5(methodName)=unknown
P6(exceptionType)=System.TimeoutException
P7(callstackHash)=9c9b
SCVMM Version=10.19.1013.0
SCVMM flavor=C-buddy-RTL-AMD64
Default Assembly Version=10.19.1013.0
Executable Name=vmmservice.exe
Executable Version=10.19.1013.0
Base Exception Target Site=unknown
Base Exception Assembly name=unknown
Base Exception Method Name=unknown
Exception Message=Sending to via http://www.w3.org/2005/08/addressing/anonymous timed out after 00:05:30. The time allotted to this operation may have been a portion of a longer timeout.
EIP=0x00007ff85c9e9129
Build bit-size=64
----------------------------------------------------
------------ exceptionObject.ToString() ------------
----------------------------------------------------
System.TimeoutException: Sending to via http://www.w3.org/2005/08/addressing/anonymous timed out after 00:05:30. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: Cannot claim lock within the allotted
timeout of 00:05:30. The time allotted to this operation may have been a portion of a longer timeout.
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.TransportDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.DuplexRequestContext.OnReply(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)
----------------------------------------------------
------------------- Scope Trace --------------------
----------------------------------------------------
[s#3b6faea] ClientConnection.GetLibraryObjectByType(this: (ClientConnection#[3]) { sessionIdentifier = (string) "uuid:2a2c7bed-881e-4e15-bf7b-af0ef0bdbbc6;id=2", carmineCallback = (IVirtualManagerServiceCallback#a68b97) Microsoft.VirtualManager.Remoting.IVirtualManagerServiceCallback,
resultChannel = (EventClientConnection#3a56f27) Microsoft.VirtualManager.Engine.EventClientConnection, instanceContext = (InstanceContext#3a02d23) System.ServiceModel.InstanceContext, connectAsUser = (UserOrGroup#[9fa155]) sessionUserRole = (UserRoleData#[9fb6b4])
{ id = 75700cd5-893e-4f68-ada7-50ef4668acc6 }, sessionProfile = Administrator, rolesThatUserIsMemberOf = (List`1<BaseUserRoleData>#2d11273) { n: 1[4] (UserRoleData#[9fad1f]) { id = 75700cd5-893e-4f68-ada7-50ef4668acc6 } }, userRoles = (UserRoleCollection#2cdfbeb)
Microsoft.VirtualManager.Engine.UserRoleCollection, objectAuthorizationManager = (AdminObjectAuthorizationManager#14aec2b) Microsoft.VirtualManager.Engine.AuthorizationManager.AdminObjectAuthorizationManager, filter = Administrator, DelegatedAdmin, TenantAdmin,
VMAdmin, SelfServiceUser, ReadOnlyAdmin, forOnBehalfOf = False, sessionUser = (UserOrGroup#[9fa155]) { sidStr = (string) , accountName = (string) sessionUserRole = (UserRoleData#[9fb6b4]) { id = 75700cd5-893e-4f68-ada7-50ef4668acc6 }, sessionProfile = Administrator,
oboSupport = OBOCloned, fetcher = (AuthorizationDataFetcher#17c0c6e) Microsoft.VirtualManager.Engine.AuthorizationManager.AuthorizationDataFetcher, abortDelegate = (ConnectionAbortDelegate#f64af8) Microsoft.VirtualManager.Engine.EventManager+ConnectionAbortDelegate
}, clientVersion = (Version#20100fff) 2.1.0, authorizedRoles = (List`1<BaseUserRoleData>#cf005e) { n: 1[4] (UserRoleData#[9fad1f]) { id = 75700cd5-893e-4f68-ada7-50ef4668acc6 } } }, objType: CarmineObjectType.ComputerTier, allObjects: ReturnAllObjects.No,
obo: (OnBehalfOf) null) in s:\btvsts\25562\private\product\engine\IndigoAccessLayer\Library\LibraryImpl_pp.cs:line 352
-------- exception.InnerException.ToString() -------
System.TimeoutException: Cannot claim lock within the allotted timeout of 00:05:30. The time allotted to this operation may have been a portion of a longer timeout.
----------------------------------------------------
--------------- exception.StackTrace ---------------
----------------------------------------------------
at System.ServiceModel.Channels.TransportDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.DuplexRequestContext.OnReply(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)
----------------------------------------------------
------------- StackTrace from handler --------------
----------------------------------------------------
This is the call stack from where the exception was caught, not where it was thrown.
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonReport.WriteReportTextFile(TextWriter reportFile)
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonReport.Send()
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonExceptionReport.Send()
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonCenter.ReportException(Exception e, WERReportOptions options, String& localReportPath)
at Microsoft.VirtualManager.Engine.Remoting.ClientConnection.HandleError(Exception error)
at System.ServiceModel.Dispatcher.ErrorBehavior.HandleErrorCommon(Exception error, ErrorHandlerFaultInfo& faultInfo)
at System.ServiceModel.Dispatcher.ErrorBehavior.HandleError(Exception error)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage9(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage8(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)
at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Any help would be greatly appreciated,
Thanks,
Scott