Toec client stops checking in / builds in memory / random failure

Support
Log in to reply
  • J

    Hello again Theopenem!

    This is going to be lengthy, but as thorough as possible:

    I have been analyzing client behavior at just over 300 endpoints at this juncture, however, I have noticed the majority of the clients begin to fail checking in.

    Please note today's date as 3/19/2021: at ~0933 -5 UTC

    The end of the log file of one computer looks like:

    2021-03-18 14:41:39,520 INFO [249] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 14:41:41,536 INFO [249] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 14:41:41,536 INFO [249] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 15:41:39,609 INFO [97] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 15:41:41,516 INFO [97] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 15:41:41,516 INFO [97] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 16:41:39,583 INFO [115] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 16:41:41,536 INFO [115] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 16:41:41,536 INFO [115] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 17:41:39,586 INFO [50] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 17:41:41,851 INFO [50] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 17:41:41,851 INFO [50] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 18:41:42,239 INFO [982] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 19:41:42,215 INFO [4589] ServiceTriggerAction - Checking For Checkin Policies

    It just simply stopped...

    Also please note that the amount of memory that the Toec.exe is consuming on this specific workstation is now at 212MB.

    Often times, attempting to stop the service fails, so I will run a taskkill /f /im toec.exe; taskkill /f /im toec-ui.exe; net start toec as a catchall to restart the service on the workstation and it, of course, begins checking in again for about a day or so.

    Another workstation pulled at random from further down the list contains this in its log file:

    2021-03-17 16:44:03,452 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,452 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [25] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [75] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [75] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [75] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [22] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [22] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [22] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [74] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [74] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [74] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [48] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [48] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [48] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,483 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [76] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [76] ServiceSocket - Microsoft.AspNet.SignalR.Client.Infrastructure.StartException: Error during start request. Stopping the connection. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:54 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
2021-03-17 16:44:03,483 INFO [76] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:54 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:54 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:53 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:53 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,505 INFO [24] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,505 ERROR [24] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [24] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,505 INFO [17] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,505 ERROR [17] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:50 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [17] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,636 ERROR [53] ApiRequest - Invalid Reponse, Signature Missing: ProvisionedComm/Policy/GetClientPolicies/
2021-03-17 16:44:03,636 ERROR [53] PolicyRunner - Error Trying To Parse Policies. Aborting Trigger: Checkin
2021-03-17 16:44:03,636 INFO [53] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-17 17:43:01,658 INFO [3551] ServiceTriggerAction - Checking For Checkin Policies
2021-03-17 22:43:06,711 INFO [21282] ServiceTriggerAction - Checking For Checkin Policies
2021-03-17 23:43:07,721 INFO [24309] ServiceTriggerAction - Checking For Checkin Policies

    This workstation attempts to reconnect every 30 seconds, and finally successfully found a spot in my server to regain its link. The computer's last checkin time was 3/18 at ~1000 -5 UTC, however you are viewing the tail end of the log.

    This workstation's toec.exe process is sitting at 247MB.

    I am not going to post from the end of another log, but it is very similar to the one above, wherein it attempts to establish a connection. In its case, the time stamp of it's last check in (from the server's perspective) predates when the log begins to show instances of it "Checking for Checkin Policies" - assuming it "thought" it connected to the server?

    Toems (server) itself is running on a windows server, and should not have any sort of limitation on the number of concurrent connections it is able to handle - not sure how the Toems itself is built, but I'm assuming there is an array which captures incoming connections in the socket, and hands off to a separate process, etc etc... so I would assume within the server soft, it isn't capped off.

    Very excellent work on the system overall guys! I can set a routine task to have the computers restart the agent for now, but I was hoping this information would be helpful for you!

    0
  • T

    Is there anything in your server logs? I run over 1000 clients with a single server without issue.

    0
  • T

    How is your memory on the server?

    J 1 Reply 0
  • J

    @theopenem_admin
    Just got tasked with a huge VPN swap project, however, I checked the memory and it's sitting very low. I allocated 16GB, and it's within 3. 12 cores - Xenon, shouldn't be a physical thing.

    I'm getting this error periodically, I don't think it matters, just a lack of terminating a DB connection i'm assuming in one of the routines:

    Aborted connection 1954 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (Got an error reading communication packets)

    I'll dig through the logs in more depth later, but I have to finish many profile kick-overs before next week -.-

    Thanks again! I'll offer up more information as soon as I have a window.

    0
  • T

    I haven't been able to replicate this, but I think I see what's happening. Each time a client checks in, it creates a new connection to the server but never closes the old one. I've released 1.3.3 which I hope will resolve this.

    J 1 Reply 0
  • J

    @theopenem_admin
    Wow, thank you so much for following up on this. Public thank you to this team, as you guys do not present that "pulling teeth" feeling that is so frequent elsewhere 😃

    I am going to deploy this and see what comes of it - I'll report back.

    As an added note, I noticed that in every case of this occurring, I attempt a

    net stop toec; net start toec;

    and I receive a This service is currently stopping or starting
    This very well may be related to an established start/stop connection as well, as if the server "crashes" and attempts to restart, the socket itself may have been hanging it.

    Thanks again and I will let you know!

    0