Issue:  UI locks up when the user click on the program form whilst an automated session is active.

Description
This has been a long standing issue, and whilst it had largely disappeared for session between x and x, the issue has reappeared again during recent sessions. It means that it that there is a regular requirement to park the telescope, kill the program and restart observatory software and the session to continue the session.  The issue is being managed by setting the program to automated mode and to then leave the program and indeed the whole machine totally alone in order to not interact with the program and to thereby not cause the issue to occur.  This however means that user can't directly check upon chart displays and data in the program or to interact with the running program by change various settings.     Chart Displays and dat can fortunately be checked by examining displays and data uploaded from the program to the Current Session WebPage or by opening picture files and document files (graphs, log files etc) output to various folders on the observatory computer.

When it works the UI is a pleasure to use (having been adpated and streamlined over several years of use and fine tuning),  but has because the program's achilles heel, something that one is scared to use for a real fear of causing it to lock up with consequent impact to the smooth running of the session

Historically the UI lockups are associated with stalled / frozen programs threads, typically Obs.Monitor and Obs.Worker, but sometime also extending to Job Queue and Obs.Manager.   A separate thread (Obs.Overseer)  was added to the program a while ago with the sole task of keeping tabs on what threads are working and which are frozen/stalled, with the capability to make the observatory safe after a certain amount of time has passed.

S863

In S863 (v3.37.2) the AstroProgram UI hung shortly after restarting Live Session (19:38 & 19:55).  Two similar indicidents. Scope had to be parked , the AstroMain program killed and suite of observatory software & session restarted. Program hang happened when the main program form was clicked upon to bring it to the front. Opportunity was taken on the second occasion to install a revised version of the program(3.37.3.1) to fix an issue where the log file was recording UNIQUE constraint failed exceptions from multiple attempts to retrospectively Insert SessionEvents for 'Program Hung  - AstroMain UI appears to have stopped responding..."

In first case UI hung up at 19:38:46 and remained in a non-responsive state till 19:46:23 when the program was killed (a period of some484s)   The Obs.Manager froze at 19:44:36 its unclear what caused it to freeze at this time, logfile provides no indication.

Throughout the period there exceptions every 5s reporting UNIQUE constraint failure by trying to Insert Session Event recording the Program was hung. This was caused by a bug (fixed in 3.37.3) in recently modified Obs.Overseer code and was exposed as a consequence of the underlying issue but  was not the cause of the UI lockup.

Throughout the period ObsWorker reported every 5s that its calls to UpdateSynopsisFields were taking > 1s. (this means that ObsWorker stopped waiting after 1s  for the BeginInvoke to be completed ). There is a high probability that the UpdateSynopsis fields delegation weren't being completed and there was a gradual build up of tasks for UI

Last message from the Job Queue was 19:38:45.32  (Convert FitsToJPG Ok | Converting Image 863003 to JPG), but there were already 'UpdateSynopsisFields has taken > 1s' message going back to 19:38:43,  19:38:37, 19:38:33, 19:38:31, 19:38:27, 19:38:23, 19:22:47 (* two messages lines produced at this time)

Throughout the period ObsMonitor/ObsViewer continued running as evidenced by uninterrupted series of ObsPics, every 30s.

Analysis
The UI is presumed to lock up because it becomes too overloaded with taks that it is unable to respond to the user and windows steps in to flag that the program is 'Not Responding' or the program UI becomes involved in an infinite loop or some form of blocking loop.

Actions already taken
SInce AstroMain 3.37.1 (2021-02-24) calls from threads to change the UI  have largely been all changed over to use BeginInvoke calls, (instead of Invoke call or Direct calls to form components). This has allowed the threads like Obs.Monitor & Obs.Worker to continue working without having to wait indefinately for the UI to respond. 

New Actions

• Eliminate calls from Business Code to read values in form (ie from textboxes, checkbox etc)
    - Form components to write validated changes to SU class parameters
    - Business code to read from SU class parameters
    - Examples     threshold  = SU.TooFewStarsThreshold   instead of    threshold = cint(objconsole.txtTooFewStarsThreshold.text)
    - Priority to be given to where monitoring loops /threads are making regular/frequent calls to form values
  
   - Done:  RainTrim, TooFewStarsThreshold,  
     AutoDewControl, ContinueWithNewPlan, DropOutToManual
     SimDome
  
  
• Move Business/Utility methods and functions  from ObsConsole class to globals or other modules
  
  - Done GetCameraTemperature(),  GetAmbientTemperature()

Update 2021-03-03  (AstroMain 3.38)
Several changes have now been implemented to try to solve the UI Hang-Up issue

• A 'Virtual Model' (VM object) framework has been put in place between the UI (Form Fields) and the Business Code.   
  
  Business Code (like ObservatoryMonitoring Loop) now makes calls such as  VM.DomeAzimuth = Dome.Azimuth rather than writing to objconsole.txtDomeAzmiuth.text.  (as in  'txtDomeAzmuth.text = cstr(DomeAzimuth)'   )  This has several benefits :
  
  •  all modifications to txtDomeAzimuth.text & .backcolor etc are now only made from the VM.DomeAzimuth's set property
     (this provides consistency and greater ease of maintainence)
     
  • by keeping a private parameters (e.g. _DomeAzimuth _DomeAzimuthColor ) the VM Set Property now only needs to invoke a change to a textbox when the value actually changes, significantly reducing the load on the UI thread.
     
  •  access to properties (achieved using property Get to return the private variable value) is significantly faster as there is no inherent load on the UI thread.
     
  • update of Form fields when required can be made using an Invoke (immediate/wait for completion) or BeginInvoke (don't wait for completion) as appropriate. 
     
  •  for Read/Write text boxes the Form's Text Box is responsible for validating the input (using AllowedCharacters) and updating either SU object fields or VM object fields.
     
  • the VM object does noy yet handle all fields but the main field that are associated with the operation of live sessions have been.
    
• UpdateSynopsisFields() is now a routine comprising a series of VM field assignments (called by ObservatoryMonitoring  Loop) rather than a routine comprising a series of Form field assignments run on the UI Thread.   The UpdateFoc1Fields() routine has been changed in a similar way.   (UpdateFoc2Fields has yet to be changed)
  
• The ObsWorker that requests Image Capture of AstroMain Tab/Panels for Web Upload on the (ObsConsole / UI thread)  only permits one live operation at a time
   
• Certain code has been been moved from the obsConsole Form into new code modules to simplify the Form Code.  So far this work has focussed on 'Reduction' Tasks,  but should in time be widened to encompass other tasks.    Aim is to more clearly separate the forms role as a dispatcher of tasks  from the tasks themselves. Since interaction between tasks and fields on the Form is increasingly made via the VM (Virtual) object there are no benefits in keeping the tasks in the Form Class  (previously taking the 'txtDomeAzimuth.text = "90"' reference out of the Form class meant that the field had to be referred by the longer 'objConsole.DomeAzimuth.text="90.0" ' reference or more recently the even longer reference 'objConsole.Invoke (Sub () objConsole.DomeAzimuth.text = "90.0")', but since this now shorterned to 'VM.DomeAzimuth = 90' things have become much simplier.
  
• The number of cross-thread calls has been significantly reduced.    This has been done by setting CheckForIllegalCrossThreadCalls = True as start of program on Development Computer and running the program through the debugger.   On Observatory Computer the 'CheckForIllegalCrossThreadCalls' parameter is set to False to prevent unexpected fatal exceptions /crashes during Live Sessions.
• 
  A backdoor option has been added to inject a DoEvents to the objconsole every 2s. By default this is off, but the option to switch it on via a checkbox is available to at least test whether is stops the UI hang-up and learn what unwanted sideeffects it might have.

• Dome Software & Drivers were updated today (2021-02-26) using the  'Pulsar_Observatories_Package 1.0.37.0.exe'  installer
  (Dome software was last updated 2018-12-22 using 'Pulsar_Observatories_Package 1.0.32.0' installer)
• Dome Rotation Driver updated from 1.39 to 1.41
• Shutter Driver remained at 1.22 despite several attempts to install software with the USB cable connected.
  
• Detailed Log:
  • Started by recording all existing settings and versions (Rotation Unit : 1.39 B, Shutter 1.22 C)
  • Software was installed first without the UBB connection to shutter. Software version in Rotation Unit updated to 1.41 without incident.
  •  USB cable was then connected between shutter and computer (cable connection recognised on computer as a COM15 connection)
  •  Initially there were some messages about installing driver / ready to use.
  •  Software Installer was rerun and the 'Repair' option used.
  • After installation cable was disconnected from shutter and Shutter Info checked (shutter still reporting Version 1.22 ~C)
  •  Tried re-powering the Rotation Unit, and installing again (see seeing V1.22 ~C)
  •  Tried installation again with cable connected, then removed fuse on Shutter unit,  then reinserted fuse, and repowered the Rotation Unit
      (again still seeing V1.22 ~C)
     
    
  • Gave up trying to update the Shutter  (pending contact with Matthew Player (Electrical Engineer Support for Pulsar Observatories) )
  • Re-entered relay and timeout settings in Pulsar Software as several settings had changed/reset as a result of installation
  •  Recalibrated the Dome as it seemed to have lost knowledge of its previous position as a result of installation
  •  Restarted ASCOM connection to Dome and tested functionality, everything seems ok.
    
  •  Examining Pulsar ASCOM Trace log:
     - shows  'DriverInfo Get               Pulsar Observatories Dome Drive Version: 6.3'  (this is noted to be unchanged from previous)
     - still shows  'Compleate' text for several Get statement (should be spelt 'Completed',  reported previously but not fixed)
     - still shows   'Slewing Get     True'   when dome/shutter are stationary (should be False, reported previously but not fixed)

Update 2021-03-04:

A new attempt to update the shutter software to v1.23 was made on 2021-03-04 (see notes).