Can't push modules/policies or send messages to computers.
-
Hello,
I'm pulling my hair out... any help would be greatly appreciated. All of my endpoints are checking in normally. Over 1000 endpoints. Theopenem is running on Windows Server 2016. Running well for the most part but I cannot seem to send messages to computers, gather inventory, install modules or really perform any functions at this time. Not sure what happened exactly, however I did recently upgrade to the 4.8.8.8 and updated the TOEC Client... all agents seem to be on 4.5... I did the update installer not the full installer. Everything went smoothly but although messages say they sent successfully, nothing shows up. Modules fail to deploy. Can you point me in the right direction? I will post my com server and cluster settings as well as some screenshots. Thank you so much for you help in advance. Everything is on one server. I have a DNS entry to access theopenem from outside my network. Maybe I have a setting wrong but I can't seem to find it. I'm seeing references to certificate issues from a few days back... and a few other errors referencing socket issues.
-
And if I need to adjust the config file to allow generating new certificates, generate new certificates, delete the existing ones from the server and then install the newly generated ones... Can I do this without losing my endpoints?
-
This is a new one for me. Can you verify the CA and intermediate are installed on the endpoints, don't see how they would get uninstalled but you never know.
-
Could you verify on the server also?
-
@theopenem_admin Yes. Checking now. Thanks.
-
Is it me or does the Server ID not match from what I have on the server to the certificates? Or am I looking at it incorrectly?
-
It's the thumbprint. They appear to match. Everything looks good there. I'll need to think about this some more. Like I said, I haven't seen this issue before. Was there any recent Windows updates on the server when this started happening?
-
Also, I'm assuming you tried rebooting the server?
-
I did reboot the server several times as well as a few endpoints. Take your time. I appreciate it. So everything that you see above looks to be in order? I will check for recent windows updates now.
-
@anx98049 KB5019964 Security update installed on 11/9. Uninstalling for good measure.
-
The only other thing that comes to mind is the server load.
-
Should I try deleting the certificates from the server and generating new ones and adding them back? Should I try to maybe restore my server from a backup? I am really concerned with losing all of these endpoints though. It's weird because we've pushed some policies without issue within the last few months but if you look at some of the logs on the server... these api errors and certificates errors go far back to even april. So confused.
-
I don't think regenerating the certs will help. I'll try to see if I can replicate this, but I'm not exactly sure how at the moment.
-
This thread is the closest thing I can find to something similar.
https://social.msdn.microsoft.com/Forums/en-US/3d581bdb-ccaa-43c7-bbaa-ae22fce06b32/bug-in-cng-rsa-key-generation?forum=windowssdkIt's the same error and also with the Windows Cryptography library. It seemed to be related to load and Windows version. Not sure where that leaves us.
-
@theopenem_admin I really do appreciate your help on this as I'm not sure where to go with it. Let me ask you this... I have this TOEMS server running on a VM, before I installed the latest update to 4.8 and clicked on prepare toec clients to push out the 4.5 version of toec on all my endpoints, I took a snapshot... If I restore the vm to that snapshot, which will bring me back to 4.4 I believe (and of course it wouldn't restore the toec clients) will I still be able to communicate with my endpoints? (And is this even worth it since some of these errors predate the update) I'm not sure why because I just pushed a software/policy module in september with no issues whatsoever... the message feature never worked for me but just about everything else did. Should I maybe build a new vm with a fresh install of theopenem and use the current server ID and fingerprint etc? just a fresh IP?
-
I wouldn't do that. I would create a new server, Server 2019 if possible, since you are currently using 2016. You can keep the database on the existing server for now, and just setup a new application / com server. Theopenem supports multiple com and application servers, so nothing would be lost
-
@theopenem_admin Building new vm with server 19 now, Will report back. Thanks again!
-
@theopenem_admin Ok new VM is up and running. TheOpenEM 4.8 installed on Server 2019. I've done just about everything except the certificates... I kind of expected to have to log in with toemsadmin and toemsadmin but it carried over my login credentials from the primary server... all endpoints are listed. I have this secondary server listed as passive right now within the com server settings of the main server. Where do I go from here? Did I do something wrong that it doesn't have it's own seperate credentials for this seperate server? Do I log into this new server, log in to toems and generate,export and install the certificates on the new server now? Then should I test sending a message to one of the endpoints? They are all there but I feel like this is more of a duplicate or mirroring. Thanks for your guidance.
-
@anx98049 I grabbed the connection string from the main server config file, grabbed the db encryption key from there also, then just updated the ip address in the connection string to point to the primary database/server. I did get the com server unique ID from the new com server entry and updated that within the new config file as well.
-
Do NOT generate the certificates. Everything is already set from the database, you just need to export the existing ones and install them on the new server. You'll want to set both com servers as active, then eventually switch the old one to passive, then just remove the old com server all together, if the new seems to be working. If everything works, finally we can migrate the database to the new server and shutdown the old one.
