@theopenem_admin
Wow, thank you so much for following up on this. Public thank you to this team, as you guys do not present that "pulling teeth" feeling that is so frequent elsewhere

I am going to deploy this and see what comes of it - I'll report back.

As an added note, I noticed that in every case of this occurring, I attempt a

net stop toec; net start toec;

and I receive a This service is currently stopping or starting
This very well may be related to an established start/stop connection as well, as if the server "crashes" and attempts to restart, the socket itself may have been hanging it.

Thanks again and I will let you know!

@theopenem_admin
Just got tasked with a huge VPN swap project, however, I checked the memory and it's sitting very low. I allocated 16GB, and it's within 3. 12 cores - Xenon, shouldn't be a physical thing.

I'm getting this error periodically, I don't think it matters, just a lack of terminating a DB connection i'm assuming in one of the routines:

Aborted connection 1954 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (Got an error reading communication packets)

I'll dig through the logs in more depth later, but I have to finish many profile kick-overs before next week -.-

Thanks again! I'll offer up more information as soon as I have a window.

Hello again Theopenem!

This is going to be lengthy, but as thorough as possible:

I have been analyzing client behavior at just over 300 endpoints at this juncture, however, I have noticed the majority of the clients begin to fail checking in.

Please note today's date as 3/19/2021: at ~0933 -5 UTC

The end of the log file of one computer looks like:

2021-03-18 14:41:39,520 INFO [249] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 14:41:41,536 INFO [249] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 14:41:41,536 INFO [249] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 15:41:39,609 INFO [97] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 15:41:41,516 INFO [97] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 15:41:41,516 INFO [97] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 16:41:39,583 INFO [115] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 16:41:41,536 INFO [115] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 16:41:41,536 INFO [115] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 17:41:39,586 INFO [50] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 17:41:41,851 INFO [50] PolicyRunner - No Policies For Trigger Checkin Were Found For This Computer
2021-03-18 17:41:41,851 INFO [50] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-18 18:41:42,239 INFO [982] ServiceTriggerAction - Checking For Checkin Policies
2021-03-18 19:41:42,215 INFO [4589] ServiceTriggerAction - Checking For Checkin Policies

It just simply stopped...

Also please note that the amount of memory that the Toec.exe is consuming on this specific workstation is now at 212MB.

Often times, attempting to stop the service fails, so I will run a taskkill /f /im toec.exe; taskkill /f /im toec-ui.exe; net start toec as a catchall to restart the service on the workstation and it, of course, begins checking in again for about a day or so.

Another workstation pulled at random from further down the list contains this in its log file:

2021-03-17 16:44:03,452 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,452 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,452 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [25] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [25] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [25] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [75] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [75] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [75] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [22] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [22] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [22] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [74] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [74] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:56 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [74] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [48] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [48] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [48] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,467 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,467 INFO [29] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [29] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:55 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,483 INFO [29] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [76] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [76] ServiceSocket - Microsoft.AspNet.SignalR.Client.Infrastructure.StartException: Error during start request. Stopping the connection. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:54 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
2021-03-17 16:44:03,483 INFO [76] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:54 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,483 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,483 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:54 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:53 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:53 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:52 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,489 INFO [26] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,489 ERROR [26] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [26] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,505 INFO [24] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,505 ERROR [24] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:51 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [24] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,505 INFO [17] ServiceSocket - Could Not Connect To Web Socket
2021-03-17 16:44:03,505 ERROR [17] ServiceSocket - Microsoft.AspNet.SignalR.Client.HttpClientException: StatusCode: 503, ReasonPhrase: 'Service Unavailable', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 17 Mar 2021 23:43:50 GMT
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Content-Length: 27
  Content-Type: text/html
}
   at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass6_0.<Post>b__1(HttpResponseMessage responseMessage)
   at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t)
2021-03-17 16:44:03,505 INFO [17] ServiceSocket - Server Push Events Will Not Be Available
2021-03-17 16:44:03,636 ERROR [53] ApiRequest - Invalid Reponse, Signature Missing: ProvisionedComm/Policy/GetClientPolicies/
2021-03-17 16:44:03,636 ERROR [53] PolicyRunner - Error Trying To Parse Policies. Aborting Trigger: Checkin
2021-03-17 16:44:03,636 INFO [53] ServiceTriggerAction - Trigger Action: Checkin. Complete.
2021-03-17 17:43:01,658 INFO [3551] ServiceTriggerAction - Checking For Checkin Policies
2021-03-17 22:43:06,711 INFO [21282] ServiceTriggerAction - Checking For Checkin Policies
2021-03-17 23:43:07,721 INFO [24309] ServiceTriggerAction - Checking For Checkin Policies

This workstation attempts to reconnect every 30 seconds, and finally successfully found a spot in my server to regain its link. The computer's last checkin time was 3/18 at ~1000 -5 UTC, however you are viewing the tail end of the log.

This workstation's toec.exe process is sitting at 247MB.

I am not going to post from the end of another log, but it is very similar to the one above, wherein it attempts to establish a connection. In its case, the time stamp of it's last check in (from the server's perspective) predates when the log begins to show instances of it "Checking for Checkin Policies" - assuming it "thought" it connected to the server?

Toems (server) itself is running on a windows server, and should not have any sort of limitation on the number of concurrent connections it is able to handle - not sure how the Toems itself is built, but I'm assuming there is an array which captures incoming connections in the socket, and hands off to a separate process, etc etc... so I would assume within the server soft, it isn't capped off.

Very excellent work on the system overall guys! I can set a routine task to have the computers restart the agent for now, but I was hoping this information would be helpful for you!

@jikeidan I definitely had an issue with my itptables ruleset - had an adapter "eth-" entered instead of "eth0" - was preventing

Thanks again!

On this note:

During the setup there are a few different interface entries. I am seeking clarification on which are local and which are public, considering the change of network configuration.

Using https://docs.theopenem.com/getting-started/configure/ as a reference

Local or Public?
Com Server:
Multicast:
tftp:

I ask because I am receiving the following error from the client:

2021-03-14 13:48:02,785 ERROR [4] ApiRequest DOMAIN\User - Error Retrieving API Response: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond PROXY_IP_ADDRESS:8888
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at RestSharp.Http.GetRawResponse(HttpWebRequest request)
   at RestSharp.Http.GetResponse(HttpWebRequest request)
2021-03-14 13:48:02,801 ERROR [4] ServiceActiveComServer DOMAIN\User - Could Not Connect To Any Client Com Servers.  Delaying 30 Seconds Before Next Retry.

However, at my DMZ proxy I am seeing in the logs the traffic inbound, masquerading to my server, and a response being sent back out.

I am limited to a single server in my environment, but have decided to better fortify my network. I currently have everything filtered and am whitelisting traffic through a DMZ proxy inbound to my Toec server.

Thus far I have only been experimenting with Toec on my local network, and it works beautifully, however, I was wondering what ports are used for the clients to register with the server.

I'm looking to forward those ports from a DMZ proxy into my server, whilst keeping the management console locked to the local net.

I can't spin up a new machine due to resource limitations at this point, but have come across the need to manage endpoints off the network.

Is this possible given my chosen design?

Thanks!

Hello @theopenem_admin,
Works beautifully. I really have very little else to say on it...

I install the agent, and once windows updates are done processing (all msiexec is blocked while it is updating), my provisioning script renames the computers and joins them to my domain. Immediately after restarting, the software pushes in.

Very well done, I am currently auto-adding after provisioning directly to a Toec Group - haven't tried an AD OU yet...

Thank you so much!

I will upgrade and check it out! Let you know how it goes

@theopenem_admin
Just in case you or anyone else were interested and hadn't tried it:
I set the LDAP sync time to 4 minutes, and my provisioning scripts renames the computer and joins it to the domain.

Typically by the time the computer reboots, the OpenEM has resync'd with AD and the default computer OU contains the computer. Having set the provision policy for that, it pushes down all the things I need... which is awesome.

A point to note, currently with the recent version I downloaded (1.3.0.0), none of the policies will take hold until windows has updated to 20H2. Once it receives the cumulative 20H2 update and reboots, all policies launch immediately - without fail.

Good tidings, and thanks again! - and if anyone has anything to offer in regard to the updates, I'm happy to hear it!

@theopenem_admin
I just noticed yesterday where the scheduler had it entered (was setting up my second instance). I will try to play around with that and see what we get.

And yeah, a check-in init where a series of blanket policies themselves can be activated (or auto assigning to a group and forcing a re-checkin). The pieces are all there.

Thank you, again!