Hello again Theopenem!
This is going to be lengthy, but as thorough as possible:
I have been analyzing client behavior at just over 300 endpoints at this juncture, however, I have noticed the majority of the clients begin to fail checking in.
Please note today's date as 3/19/2021: at ~0933 -5 UTC
The end of the log file of one computer looks like:
2021-03-18 14:41:39,520 INFO [249] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 14:41:41,536 INFO [249] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 14:41:41,536 INFO [249] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 15:41:39,609 INFO [97] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 15:41:41,516 INFO [97] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 15:41:41,516 INFO [97] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 16:41:39,583 INFO [115] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 16:41:41,536 INFO [115] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 16:41:41,536 INFO [115] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 17:41:39,586 INFO [50] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 17:41:41,851 INFO [50] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 17:41:41,851 INFO [50] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 18:41:42,239 INFO [982] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 19:41:42,215 INFO [4589] ServiceTriggerAction - Checking For Checkin Policies
It just simply stopped...
Also please note that the amount of memory that the Toec.exe is consuming on this specific workstation is now at 212MB.
Often times, attempting to stop the service fails, so I will run a taskkill /f /im toec.exe; taskkill /f /im toec-ui.exe; net start toec
as a catchall to restart the service on the workstation and it, of course, begins checking in again for about a day or so.
Another workstation pulled at random from further down the list contains this in its log file:
2021-03-17 16:44:03,452 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:56 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,452 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:56 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [25] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:56 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [75] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [75] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:56 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [75] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [22] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [22] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:56 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [22] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [74] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [74] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:56 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [74] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [48] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [48] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:55 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [48] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:55 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:55 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:55 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:55 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,483 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [76] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [76] ServiceSocket - Microsoft.AspNet.SignalR.Client.Infrastructure.StartException: Error during start request. Stopping the connection. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:54 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
2021-03-17 16:44:03,483 INFO [76] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:54 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:54 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:53 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:53 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:52 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:52 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:52 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:52 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:52 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:51 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:51 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:51 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,505 INFO [24] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,505 ERROR [24] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:51 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [24] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,505 INFO [17] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,505 ERROR [17] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Wed, 17 Mar 2021 23:43:50 GMT
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Content-Length: 27
Content-Type: text/html
}
at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [17] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,636 ERROR [53] ApiRequest - Invalid Reponse, Signature Missing: ProvisionedComm/Policy/GetClientPolicies/
2021-03-17 16:44:03,636 ERROR [53] PolicyRunner - Error Trying To Parse Policies. Aborting Trigger: Checkin
2021-03-17 16:44:03,636 INFO [53] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-17 17:43:01,658 INFO [3551] ServiceTriggerAction - Checking For Checkin Policies
2021-03-17 22:43:06,711 INFO [21282] ServiceTriggerAction - Checking For Checkin Policies
2021-03-17 23:43:07,721 INFO [24309] ServiceTriggerAction - Checking For Checkin Policies
This workstation attempts to reconnect every 30 seconds, and finally successfully found a spot in my server to regain its link. The computer's last checkin time was 3/18 at ~1000 -5 UTC, however you are viewing the tail end of the log.
This workstation's toec.exe process is sitting at 247MB.
I am not going to post from the end of another log, but it is very similar to the one above, wherein it attempts to establish a connection. In its case, the time stamp of it's last check in (from the server's perspective) predates when the log begins to show instances of it "Checking for Checkin Policies" - assuming it "thought" it connected to the server?
Toems (server) itself is running on a windows server, and should not have any sort of limitation on the number of concurrent connections it is able to handle - not sure how the Toems itself is built, but I'm assuming there is an array which captures incoming connections in the socket, and hands off to a separate process, etc etc... so I would assume within the server soft, it isn't capped off.
Very excellent work on the system overall guys! I can set a routine task to have the computers restart the agent for now, but I was hoping this information would be helpful for you!