Welcome to the community forums for Theopenem. All responses are provided by other users that wish to help out. Theopenem will not respond to these posts. If you require more assistance than what the community can provide, we offer various paid support options.


  • I was able to set up a share to prep for a 2nd com server after updating my existing COM server from 1.5.0 to 1.5.2. It is able to replicate only the images folder from local storage to the newly created share.

    Below is the error that is thrown on replication task execution:

    2022-12-13 16:22:37,406 [Worker #19] ERROR Toems_ApiCalls.ApiRequest Could Not Complete API Request. The Response Produced An Error.Storage/Sync
    2022-12-13 16:22:37,421 [Worker #19] ERROR Toems_ApiCalls.ApiRequest {"Message":"An error has occurred.","ExceptionMessage":"One or more errors occurred.","ExceptionType":"System.AggregateException","StackTrace":" at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.Task.Wait()\r\n at Toems_Service.Workflows.FolderSync.Sync()\r\n at Toems_ClientApi.Controllers.StorageController.Sync()\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\r\n at Toems_ClientApi.Controllers.Authorization.ResultWithChallenge.<ExecuteAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.AuthenticationFilterResult.<ExecuteAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()","InnerException":{"Message":"An error has occurred.","ExceptionMessage":"A task was canceled.","ExceptionType":"System.Threading.Tasks.TaskCanceledException","StackTrace":null}}
    2022-12-13 16:22:37,421 [Worker #19] INFO Toems_Service.Workflows.ImageSync Starting Image Replication From Com Servers
    2022-12-13 16:22:37,468 [Worker #19] INFO Toems_Service.Workflows.ImageSync No Images Found To Replicate
    2022-12-13 16:22:37,484 [Worker #19] INFO Toems_Service.Workflows.ImageSync Starting Image Replication To Com Servers
    2022-12-13 16:22:37,593 [Worker #19] INFO Toems_Service.Workflows.ImageSync Com Server Images Are All Up To Date. Skipping Replication.
    

    COM Server Bitrate:
    2eeeac75-985d-4d9f-8595-ba81baf9e88e-image.png
    COM Server Local Storage:
    ac71e66a-f7a0-4bd0-b40d-10c18a49f221-image.png
    UNC Path of SMB Share:
    430d93a4-8e5e-4df6-b651-91abf6e5468b-image.png
    About:
    c04809e9-4a02-4117-9618-38691bb2606c-image.png


  • Does replication work for your first com server or are your share and first com server the same location?


  • @theopenem_admin Right now this is the first and only com server. It was initially set up for local storage during my testing, and I was attempting to move to SMB storage in preparation for a 2nd com server.

    The images folder replicates as expected, but software_uploads and client_versions directories do not.


  • Are there any errors in the com server log? I think you just posted the application log.


  • @theopenem_admin Not from the same day, after looking. I attempted the change from local to SMB 12/13 and there aren't any events logged for that date on com server logs.

    I did find one error in the com server ClientApi.log dated 12/07, but did not notice an impact to deployment testing from that error:

    2022-12-07 16:19:29,075 [46] ERROR Toems_Service.FilesystemServices Could not find a part of the path 'C:\toems_local_storage\images\Win10_x64_Edu_21H2'.
    

  • Difficult to say. The error seems to state it's being cancelled, but not sure why. Are there any errors in the Windows event viewer under application?


  • @theopenem_admin Looks like it's trying and failing to load a 2FA assembly?

    Could not load file or assembly 'TwoFactorAuth.Net' or one of its dependencies. The process cannot access the file because it is being used by another process. (Exception from HRESULT: 0x80070020) at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) at System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory() at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai) at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig) at System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies() at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) at System.Web.Compilation.BuildManager.ExecutePreAppStart() at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) Could not load file or assembly 'TwoFactorAuth.Net' or one of its dependencies. The process cannot access the file because it is being used by another process. (Exception from HRESULT: 0x80070020) at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection) at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection) at System.Reflection.Assembly.Load(String assemblyString) at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)</Data> 
    
    

  • @tgilpin And also this event:

    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="ASP.NET 4.0.30319.0" /> 
      <EventID Qualifiers="16384">1315</EventID> 
      <Level>4</Level> 
      <Task>3</Task> 
      <Keywords>0x80000000000000</Keywords> 
      <TimeCreated SystemTime="2022-12-14T14:26:26.525917700Z" /> 
      <EventRecordID>16180</EventRecordID> 
      <Channel>Application</Channel> 
      <Computer>[REDACTED]</Computer> 
      <Security /> 
      </System>
    - <EventData>
      <Data>4005</Data> 
      <Data>Forms authentication failed for the request. Reason: The ticket supplied has expired.</Data> 
      <Data>12/14/2022 8:26:26 AM</Data> 
      <Data>12/14/2022 2:26:26 PM</Data> 
      <Data>c6b5532bc393458793974d0434e1ef02</Data> 
      <Data>4</Data> 
      <Data>1</Data> 
      <Data>50202</Data> 
      <Data>/LM/W3SVC/4/ROOT-1-133155004994963843</Data> 
      <Data>Full</Data> 
      <Data>/</Data> 
      <Data>C:\Program Files\Theopenem\Toems-UI\</Data> 
      <Data>[REDACTED]</Data> 
      <Data /> 
      <Data>1356</Data> 
      <Data>w3wp.exe</Data> 
      <Data>NT AUTHORITY\NETWORK SERVICE</Data> 
      <Data>[REDACTED]/views/admin/chooser.aspx</Data> 
      <Data>/views/admin/chooser.aspx</Data> 
      <Data>[REDACTED]</Data> 
      <Data /> 
      <Data>False</Data> 
      <Data /> 
      <Data>NT AUTHORITY\NETWORK SERVICE</Data> 
      <Data /> 
      </EventData>
      </Event>
    

  • Those are most likely unrelated.


  • @theopenem_admin I'll just wipe and start from the top. I feel like something may have gone wrong with my TOEMS updates.


  • @theopenem_admin I created a new VM and freshly installed TOEMS latest version, however this replication issue persists in the same manner.

    Only the Images folder will sync to SMB storage and TOEMS reports a successful sync task. All previously reported logged events are reoccurring on the new install. Com server logs from the web interface generated no data related to the sync.

    I have verified that the SMB location is able to be reached and is accepting authentication as configured in TOEMS. I am unsure what would cause TOEMS to successfully sync images but fail to sync software uploads and generated files.


  • Pushing ahead and manually copying from local to SMB seems to clear the hurdle and produce expected behavior once completed.

    I had followed instruction to first set up as local and then move to SMB share on adding COM servers, however it seems like in this version this order of operations is what caused the issue.

    The failure I was experiencing was being unable to replicate up from an active COM server local storage location to a new SMB share location after changing the server storage from local to SMB share in Admin Settings>Storage Location, and leaving the COM server local path unchanged. However, once they are set in a shared storage configuration, and after a manual sync, new uploads are first stored in the SMB storage location which has no issues replicating down to the COM server local storage(s).

    I'd be curious to see if this issue is able to be reproduced on an install that was set as SMB from the start, to confirm this theory.


  • Thanks for your research into this. I was not able to replicate the issue and couldn't say where to start. Now it seems you've narrowed it down, I can look into it.


  • Unsure if it is related to forcing the initial sync, but I am now receiving "Could not download core scripts" with response code 403 when attempting to boot with USB ISO.

    I have attempted an ISO configured for all available clusters including default. Below is what we are seeing on both com server logs.

    2023-01-18 13:10:02,533 [25] ERROR Toems_Service.StringManipulationServices Authorization Base64 Decoding Failed. Value cannot be null.
    Parameter name: s
    

    USB ISO boot was working prior to adding this 2nd com server into the environment.


  • @tgilpin Did you ever figure this out?

    I finally got the LIE boot to hit my server (was trying WIE, but for some reason can't get the NIC driver to load, even side loading the .ini), and am now getting this.

    When I try to hit my server at ip:8888 (same as the LIE environment), I get an Apache looking 403 screen as well.

    But I'm currently going to Toems Local Storage, so I don't have the SMB complication.

    Asking here as you seem to be the only kindered soul so far.


  • @palehbt Fixed it. Turn off "console Tasks require login" in Admin -> security