Having multiple servers

Support
Log in to reply
  • T

    I have a single machine hosting Toec API, Toems API, Toems WebUI, and MariaDB, I would like to add a new sever, later on remove the first server, and still later add a second server which uses a different fileserver. I've been thinking about this for a while and I'm not making headway on figuring out what of this is possible, what needs to be done, and what order to do it in.

    My current understanding

    1. There needs to be a single database(?)

    My current uncertainties

    1. Does every Toems API node need it's own Toec API node?
    2. Does every Toems API node need it's own WebUI node?
    3. Is it possible to have more than one storage location?
      • I'm currently using SMB but I will be able to instead use "local" storage via iSCSI targets
    4. Does anything in the config.web files need to be duplicated to link with the additional nodes?
    5. Is Theopenem able to run on ASP.NET Core?
      6.a. If it can, has anybody tried to containerize the nodes?
      6.b. If it can, has anybody tried to run Theopenem on Linux?
      • I have Theopenem running on a windows 10 desktop currently, I'm moving it to a windows server VM on a linux based server, but I'd much rather move it to kubernetes and not have a full GUI VM at all.
    6. What is the intended use of having multiple clusters?
    7. Is there a load balancing feature already?

    Issues I've run into so far

    When I copy the production database to the new installation, it is larger (original: 49.8MiB, copy: 53.9MiB, the difference appears to be primarily in computer_logs, audit_logs, state, job, and jobparameter. Is this normal?

    0
  • T

    @tyler said in Having multiple servers:

    There needs to be a single database(?)

    Correct, only 1 database.

    Does every Toems API node need it's own Toec API node?

    No, they are independent

    Does every Toems API node need it's own WebUI node?

    No, they are independent

    Is it possible to have more than one storage location?

    There can only be 1 SMB storage total. If you are using local storage, each com server can only have 1 local storage location, if you setup an additional com server, it can also have it's own local storage, which could be an iscsi target.

    Does anything in the config.web files need to be duplicated to link with the additional nodes?

    Yes, the database connection string and encryption key

    Is Theopenem able to run on ASP.NET Core?

    No, it's a complete rewrite to move to .NET core. I was planning on doing this, but as time goes by and Theopenem remains limited in popularity, it's really not worth investing the time in this move. I'll probably just ride out .NET 4.8 for as long as possible, Microsoft still hasn't even announced an EOL for it, so it should be good for a while.

    What is the intended use of having multiple clusters?

    It gives you control for which computers communicate with specific clusters. For example, if you wanted to put a com server in a specific building and only have the computers in that building communicate with that server, that's how you would do it.

    Is there a load balancing feature already?

    Yes, adding more than 1 com server to a cluster automatically enables load balancing if they are both set to active

    When I copy the production database to the new installation, it is larger (original: 49.8MiB, copy: 53.9MiB, the difference appears to be primarily in computer_logs, audit_logs, state, job, and jobparameter. Is this normal?

    I have no idea why it would be larger.

    There is a video titled multiple com servers that you can watch for reference. It's slightly more complicated than what's needed for most people because it ties together 2 networks that can't talk to each other, but the concepts remain the same.

    T 1 Reply 0
  • T

    Thanks for pointing out this video, it's not listed and I forgot to check the channel directly.

    I suspect being attached to IIS is a contributing factor in Theopenem's popularity. Running Windows Server costs a lot of money and is not the most efficient utilization of server resources, while running a critical service on a user facing edition of Windows 10+ subjects you to mandatory periodic updates. As far as I can tell containers are the direction everything is headed in and platform agnostic containers open up the potential for cloud hosting, though I don't know exactly how far of a reach that is.

    This is getting into feedback, but I really do think that opening up the platform and hosting options could alter the nature of Theopenem's popularity. I'm going to go take notes on that video now, thanks again.

    T 1 Reply 0
  • T

    @tyler
    I agree it would be nice. When Theopenem was CloneDeploy and I supported both Windows and Linux it just became too much to support as each OS had their own issues and about 90% of my users were installing on Windows anyway. I haven't ruled out the move completely but I don't believe that imaging / management will be needed for too much longer into the future anyway, but I guess we'll see.

    0
  • T

    Also, I'm just an old school kinda guy. Not really interested in containers or newer tech.

    0
  • T

    The industry I work in will always need imaging, I don't really use the management part, you may be right about that. Theopenem is the only solution I have found that is actually good at imaging in large quantities, everything else is single user solutions maybe with a hack that lets them scale a tiny bit.

    0
  • T

    Is there a code sheet for verifydb?

    It's giving me "5"

    T 1 Reply 0
  • T

    5 is fine, really any number other than an error message. It's just your checkin time.

    0
  • T

    What is the format for Storage Path with local storage? I don't have the default value anymore

    I have the iSCSI target mounted as D:\ and this target is just for Theopenem

    0
  • T

    It would just be d:\ then.

    0
  • T

    Alright so,

    Since storage location is configured globally I'm taking the old server down until I retrofit it to an iSCSI target (and install the recent update)

    I have:

    • copied the database over as discussed
    • Configured the Theopenem sites on the new server
    • Set Storage Type Local, Storage Path D:\
    • Disabled the sites in the old server's IIS
    • Configured the Com Server entry
    • Added it to the default cluster
    • Removed the old server from the default cluster

    When I go to the Dashboard, it takes 20+ seconds to load, and the whole site locks up

    When the dashboard loads, there's nothing listed there.

    This feels like something is wrong, but I don't actually know that

    T 1 Reply 0
  • T

    Did you give IIS permissions to the local storage?

    0
  • T

    I matched the permissions on C:\toems_local_storage, no change, is there somewhere in IIS I have to configure permissions as well?

    T 1 Reply 0
  • T

    Definitely seems related to storage location. Anything in the application log?

    T 1 Reply 0
  • T

    @theopenem_admin Just spent a while finding my own mistakes, so now I do have the new server listed correctly on the dashboard and the dashboard loads promptly. I have cleared up a lot of errors, but I'm still getting this one when the dashboard is loaded

    FrontEnd.log

    2024-03-06 11:53:16,772 [9] ERROR Toems_ApiCalls.ApiRequest Response Data Was Null For Resource: FileSystem/GetSMBFreeSpace/

    I suspect something is still configured to think about SMB?
    Maybe I'll find it as I regenerate WIE.

    ClientAPI.log and Application.log are no longer accruing errors.

    This is already solved but for future problem havers, I was getting:

    2024-03-06 10:29:21,042 [44] ERROR Toems_ApiCalls.ApiRequest Could Not Complete API Request. The Response Produced An Error.Storage/GetFreeSpace
    2024-03-06 10:29:21,074 [44] ERROR Toems_ApiCalls.ApiRequest

    {
	"Message":"An error has occurred.",
	"ExceptionMessage":"One or more errors occurred.",
	"ExceptionType":"System.AggregateException",
	"StackTrace":"
		at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
		at Toems_ClientApi.Controllers.Authorization.InterComAuth.AuthenticateAsync(HttpAuthenticationContext context, CancellationToken cancellationToken)
		at System.Web.Http.Controllers.AuthenticationFilterResult.<ExecuteAsync>d__5.MoveNext()
		--- End of stack trace from previous location where exception was thrown ---
		at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
		at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
		at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()",
	"InnerException":
	{
		"Message":"An error has occurred.",
		"ExceptionMessage":"An error occurred accessing the database. This usually means that the connection to the database failed. Check that the connection string is correct and that the appropriate DbContext constructor is being used to specify it or find it in the application's config file. See http://go.microsoft.com/fwlink/?LinkId=386386 for information on DbContext and connections. See the inner exception for details of the failure.",
	"ExceptionType":"System.Data.Entity.Core.ProviderIncompatibleException",
	"StackTrace":"
		at System.Data.Entity.Utilities.DbProviderServicesExtensions.GetProviderManifestTokenChecked(DbProviderServices providerServices, DbConnection connection)
		at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
		at System.Data.Entity.Utilities.DbConnectionExtensions.GetProviderInfo(DbConnection connection, DbProviderManifest& providerManifest)
		at System.Data.Entity.DbModelBuilder.Build(DbConnection providerConnection)
		at System.Data.Entity.Internal.LazyInternalContext.CreateModel(LazyInternalContext internalContext)
		at System.Data.Entity.Internal.RetryLazy`2.GetValue(TInput input)
		at System.Data.Entity.Internal.LazyInternalContext.InitializeContext()
		at System.Data.Entity.Internal.InternalContext.Initialize()
		at System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType)
		at System.Data.Entity.Internal.Linq.InternalSet`1.Initialize()
		at System.Data.Entity.Internal.Linq.InternalSet`1.get_InternalContext()
		at System.Data.Entity.Infrastructure.DbQuery`1.System.Linq.IQueryable.get_Provider()
		at System.Linq.Queryable.Where[TSource](IQueryable`1 source, Expression`1 predicate)
		at Toems_DataModel.GenericRepository`1.GetFirstOrDefault(Expression`1 filter, Func`2 orderBy)
		at Toems_Service.Entity.ServiceSetting.GetSetting(String settingName)
		at Toems_Service.Entity.ServiceSetting.GetSettingValue(String settingName)
		at Toems_ClientApi.Controllers.Authorization.InterComAuth.<isValidRequest>d__9.MoveNext()",
	"InnerException":
	{
		"Message":"An error has occurred.",
		"ExceptionMessage":"The provider did not return a ProviderManifestToken string.",
		"ExceptionType":"System.Data.Entity.Core.ProviderIncompatibleException",
		"StackTrace":"
			at System.Data.Entity.Core.Common.DbProviderServices.GetProviderManifestToken(DbConnection connection)
			at System.Data.Entity.Utilities.DbProviderServicesExtensions.GetProviderManifestTokenChecked(DbProviderServices providerServices, DbConnection connection)",
		"InnerException":
		{
			"Message":"An error has occurred.",
			"ExceptionMessage":"Unable to connect to any of the specified MySQL hosts.",
			"ExceptionType":"MySql.Data.MySqlClient.MySqlException",
			"StackTrace":"
				at MySql.Data.MySqlClient.NativeDriver.Open()
				at MySql.Data.MySqlClient.Driver.Open()
				at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
				at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()
				at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
				at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
				at MySql.Data.MySqlClient.MySqlPool.GetConnection()
				at MySql.Data.MySqlClient.MySqlConnection.Open()
				at MySql.Data.MySqlClient.MySqlProviderServices.GetDbProviderManifestToken(DbConnection connection)
				at System.Data.Entity.Core.Common.DbProviderServices.GetProviderManifestToken(DbConnection connection)",
			"InnerException":
				{
				"Message":"An error has occurred.",
				"ExceptionMessage":"No such host is known",
				"ExceptionType":"System.Net.Sockets.SocketException",
				"StackTrace":"
					at System.Net.Dns.GetAddrInfo(String name)
					at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
					at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
					at MySql.Data.Common.MyNetworkStream.CreateStream(MySqlConnectionStringBuilder settings, Boolean unix)
					at MySql.Data.Common.StreamCreator.GetStream(MySqlConnectionStringBuilder settings)
					at MySql.Data.MySqlClient.NativeDriver.Open()"
				}
			}
		}
	}
}
2024-03-06 10:29:21,074 [44] ERROR Toems_Service.Workflows.ComServerFreeSpace Com server returned null for status. Check your com server URL!

    The problem was that in toec-api\web.config I had pasted incorrectly and had connectionString="server=server=localhost; [...], this kind of error is hard for my eyes to pick up on.

    There was also an interim error where an entire html page was dumped in the logs saying the com server's credentials were not authorized to connect from that host, I'll need to go in HeidiSQL and add a GRANT when the next server is coming up.

    Tangent on that, for MariaDB, do you recommend sticking with the root account for all servers, or setting different credentials for each server?

    0
  • T

    If it's the only database that's running on there, then it really doesn't matter.

    0