Fig 1.   PowerBox Error Dialog "Disconnected from device"


 

Fig 2.   TheSky Error Dialog : "ASCOM DeviceHub - MeadeGeneric.Telescope: The port is closed"

Fig 3. 'Restart DeviceHub' attempt (06:44)

Stop Hub Services                       2024-10-11 06:43  (Local, BST)  
 Dome:                       Ok        Stopped
 Telescope:                  Ok        Stopped  

Restart Device Hub                      2024-10-11 06:44  (Local, BST)   
  Close Shutter               Info      Commanding a Dome Closure as a precaution. Standby   Close Dome                  Fail      Dome (objDome) not found   Disconnect TheSky Scope...  Ok        The telescope is not connected to TheSky6  

  Disconnect AstroGuard Services        
  Disconnect AstroGuard ...   Info      AstroGuard isn't running   

  Stop Hub Services                     2024-10-11 06:45  (Local, BST)
  Dome:                       Ok        Stopped  
  Telescope:                  Ok        Stopped already  
  Stopping Process            Ok        Stopped           (ASCOM.DeviceHub)
  DeviceHub:                  Ok        Started (06:46)   (ASCOM.DeviceHub 6.6.1.1)

  Start Hub Services                    2024-10-11 06:46  (Local, BST)   DeviceHub:                  Ok        Started already   (ASCOM.DeviceHub, 6.6.1.1)   
  Dome:             (COM3)    Ok        Started (06:46)   (ASCOM.Pulsar_Observatories_Dome.Dome, 6.4.0.0)

  Dome                                  2024-10-11 06:46  (Local, BST)  
  Azimuth                     Ok        Az: 90.0°
  Shutter                     Ok        Closed            (Zigbee: Closed)  
  Pulsar Log File             Ok        2024-10-11\ASCOM.Pulsar_Observatories_Dome.0646.090470.txt

  Test for Stalled Dome                 2024-10-11 06:46  (Local, BST)   Dome Driver/Dome Slaving are suspected of being stalled   Running a quick test by slewing dome 10 degs from its current azimuth (90.0 degs)     Start Slew                Info      Dome has started slewing  Az 90.0 degs > Dome Park                   Info      Dome has unparked             (06:46:13, Zigbee sensor)     End Slew                  Info      Dome has finished slewing. Az 100.0 degs     Result                    Info      Dome has moved 10.0 degs in 7.2s (speed: 1.39°/s)   Conclusion                  Ok        Dome Driver is working normally   

  Running second test to check that dome slaving is working
  by seeing if Dome Azimuth changes when slaving is turned back on   Turn on Dome Slaving...     Fail      Property write ASCOM.DeviceHub.Dome Slaved is not implemented in this driver.     Time Out                  Info      30s timeout reached. Az 100.0 degs     Result                    Info      Dome has moved 0.0 degs in 30.3s (speed: 0.00°/s)   Conclusion                  Fail      Dome Slaving has stalled

  Restart DeviceHub           Warning   Exception whilst running Restart DeviceHub() > Dome Alert                  Alert     2024-10-11 06:46  (Local, BST) > Dome Alert                  Alert     Dome is confirmed to be stalled

Issue
A number of issues causing or relating to the loss of telescope connection & control during last third of the S1253 session, and the failure of the automated system to correctly diagnose & resolve the problem.

1. Connection to Pegasus UPB Powerbox was lost affecting information on and the control of Scope Power, DewHeater and Foc2 Focuser (03:14)
2. Job Executor unable to make slew to Target 23/30 (SN 2024wak) after failing Telescope Coord Consistency check (03:30)
3. Multiple repeating Suspend/Resume cycles with each one suffering a 'CheckTheSky6  Scope' failure & a Restart TheSky6 operation (03:30 to 05:57)
4. Dome Opened and Closed 15 times during the last third of the session (03:30 to 05:57)
5. Telescope wasn't parked at the end of the session (06:22)
6. TheSky6 showing a ASCOM.DeviceHub(.Telescope) error dialog with the message "The port is closed" (06:29).
7. AstroVOE crashed with UnauthorizedAccessException Exception (~03:55)
   
   Post-Session
8. Attempt to 'Restart DeviceHub' back door facility unexpectedly included a 'Test For Stalled Dome' operation (06:46) 

Description

1. 03:14.  Upon checking the results of the S1232 Session, and noting multiple alerts & target failures from 03:30 onwards and the indication that the Telescope hadn't parked, a 'lockdown' was initiated at around 06:29 in order to understand the precise state of equipment and software on the Observatory Computer.   This identified that a 'Ultimate Powerbox' error dialog (probably from AstroMain) was displayed on the computer with the message "Disconnected from device" (see Fig 1).  
    
   Examination of flags on ObsPics recorded every 30s through the session showed  'Scope Power', ' Dew Heater' , 'Foc2 Focuser' & 'UPB Temperature' all dropped out at 03:14 and stayed out for the remainder of the session.  During later post-session recovery the connection with UPB Powerbox was reestablished by using the backdoor button facility "Restart UPB & Services". 
    
2. 03:30. After completing Target 22/30, the Job Executor proceeded with Target 23/30 (SN 2024wak).  During pre-slew Target Checks, AstroMain reported a 'Checking TheSky Coords' alert with the message "70.5 discrepency between TheSky6 and ASCOM Telescope Coords" and made a diagnosis that "Coordinates from TheSky6 are likely frozen" and proceeded to Restart TheSky on the basis that TheSky6 was unresponsive, but this didn't resolve the issue, and issues with telescope connection continued for the remainder of the session.
   
   The log file doesn't record the TheSky6 and DeviceHub telescope coordinates to fully diagnose the problem, but in this case this is likely not to be a significant loss, as in is just a sympton of a bigger issue, caused by loss of connection to Telescope.
   
3. 03:30 to 05:57. Following a "70.5 discrepency between TheSky6 and ASCOM Telescope Coords" for Target 23/30 (SN 2024wak) at 03:30,  ObsManager/Job Executor proceeded with 15 repeated Suspend/Resume cycles between 03:31 and the end of the session. Each cycle suffererd a 'CheckTheSky6  Scope' failure / Restart TheSky6, a failure to resolve the issue and a re-suspension of the session.  The failure in each case was accompanied by the messages  Given that failure in each case was the same two messages "GetRaDec failed - is the telescope connected ?" and "Ra and Dec coordinates are both 0.0. This is likely a problem". Each cycle tried the same unsuccessful fix which involved restarting TheSky6. Each cycle was unsuccessful. was to Restand that each dome opening/closing operation was proving to be fruitless, and is merely presenting potential wear/tear to shutter equipment & shutter battery and exposures the observatory to unnecessary risk to rain etc, the ObsManager should have recognised that it had an issue on its hands that was not solveable without user involvement and should have probably left the session suspended / dome closed till the morning, when user could resolve the problem.  Stopping and Restarting DeviceHub (without Test For Stalled Dome) should probably have been tried by the ObsManager, but wasn't. 
    
4. 03:30 to 05:57. The Observatory Dome was opened and closed 15 times between 03:30 and the end of the session. Only one of the suspensions was directly related to deteriorating weather conditions.  
    
   
5. 06:22. AstroMain's Closing Session operations were unable to park the telescope at the end of the session. During the Park Telescope operation at 06:22, AstroMain reported "TheSky Telescope is either not connected or there is an issue (0)" during the 'Slew Scope to PrePark' task, issued an Alert with message "Telescope may have failed To park correctly (attempt 120s)" following the attempted Park Telescope task, and reported a System.InvalidOperationException: "The port is closed" when attempt to get the Telescope's Handbox screen message. 
    
6. 06:29. Upon checking the results of the S1232 Session, and noting multiple alerts & target failures from 03:30 onwards and the indication that the Telescope hadn't parked, a 'lockdown' was initiated at around 06:29 in order to understand the precise state of equipment and software on the Observatory Computer.   This identified that TheSky6 was showing a DeviceHub Error Dialog on the computer with the message "CheckDotNetExceptions ASCOM.MeadeGeneric.Telescope RightAscension Get System.InvalidOperationException: The port is closed"
    
7. 06:31 Upon checking the results of the S1232 Session, and noting multiple alerts & target failures from 03:30 onwards and the indication that the Telescope hadn't parked, a 'lockdown' was initiated at around 06:29 in order to understand the precise state of equipment and software on the Observatory Computer.  This identifed a Just In-Time Debugger dialog indicating there had been an unhandled exception in AstroVOE.exe program. Opening Visual Studio showed an unhandled exception had occured  "System.UnauthorizedAccessException: 'Access to the path 'C:\Data\CCD Imaging\RestartTheSkyFlag.dat' is denied.' " in AstroVOE.globals.TWatcher.OnChanged() in globals.vb at the code line highlighted below:
   Private Sub OnChanged(source As Object, e As FileSystemEventArgs)
    Select Case Action                    '     added 2022-12-19 (CI S1072)
            Case "RestartTheSky" ' added 2022-12-19 (CI S1072)
                RefreshTheSkyObjects()
                Thread.Sleep(1000)
                File.Delete(WatchFolder + "\" + WatchFile)
     End Select
End Sub
    
   
8. Following the problems towards end of the session and the realisation that Telescope hadn't been parked and that DeviceHub wasn't communicating with the Telescope, the backdoor facility 'Restart DeviceHub' was used. This was done after first checking that AstroGuard, TheSky6 and PHD2 were no longer connected to DeviceHub/Telescope and after restarting UPB Software/Service and stopping Dome/Telescope Services in AstroMain.  
   
   It was hoped/expected that the backdoor facility would simply kill/stop DeviceHub and restart it.   Whilst it seemingly did this, the operation unexpectedly continued on to run 'Test For Stalled Dome' tasks (see Fig 1). The first part concluded by concluding that the Dome Driver was working normally (as the dome slewed 10 deg when requested by the test).  The second part of the test checked to see if Dome Slaving was working, and it got a "Property write ASCOM.DeviceHub.Dome Slaved is not implemented in this driver" error, and concluded that Dome Slaving has stalled, and raised two Red Alerts  "Dome Driver/Dome Slaving found to be stalled" and "Dome driver/Dome slaving is stalled". 
   
   Whilst the Test is employed for historic reasons and might still have a role, it was inappropriate in this particular case. The dome had already been parked, and the reason to for the exception was that the Telescope wasn't connected and Dome.Slaving was bound to failed. The reason for restarting DeviceHub was purely to manually go on restart Telescope connection and park the telescope, not to test for an issue with the Dome.  The RestartDeviceHub() routine needs to be modified to disable the call to TestForStalledDome()
   

Analysis

1. 03:14. ObsPics showed that 'Scope Power', ' Dew Heater' and 'Foc2 Focuser' all dropped out at 03:14 and remained out for the remainder of the session.  The Scope COM port goes missing at this same time (shown by Red Flag with a 'COM -' text), but comes back at 03:17 (shown by Green Flag with 'COM 6' text) . The Foc1 focuser appears to remain connected, and changes Focus Position at various times between 3:17 and 03:25, but then remains on 1699 position till the end of the session.  Its unclear what triggered the event at 03:14 but it seems to have the consequence that it knocked out MeadeGeneric Telescope Driver's connection with the Telescope (which goes via COM6 / USB Port on the Pegasus Powerbox).
    
   Examination of the session log file shows the following:
   
   03:14:37 -'Check PowerBoxPort'  line at 03:14:37  with the message
      "Powerbox port seems to have been lost at 03:14:37"
   
    03:15:07 - 'Check DewHeater' failures starting 03:15:07 and continuing to end of session with the message
      "objPowerBox2 is not connected".
   
    03:17:37- 'Check Focuser1' info line is present at 03:17:37 with the message
      "Powerbox port (COM4) is connected again. Outage was 180s" .
   
   COM4 port to the Foc1 Focusers goes via the UPB Powerbox.  The outage of 3 minutes is similar to the outage in the COM6 port, indicating a common problem, a blip in the Computer, USB Hub or the UPB Powerbox.   Did the Powerbox reset itself ?
   
    03:17:59 - 'Get CCDCamera property failures starting at 03:17:59 and continuing till 03:25:40 with the messages
     "Exception in CCDCamera.GetProperty IsExposureComplete"
     "Error: SBIG driver: Operating system error".
   
   - A 'Check Network_Router' message is present at  03:19:05 with message "Router connection is ok again (ping 89ms)", however there was 32 occasions spread-out through the entire session with messages ""Router ping timed out after 200ms", followed by "Router connection is ok again" , so this is likely just a red herring.
    
2. 03:30. Examination of the MeadeGeneric (Telescope Driver) Log for the session shows a "Get Altitude Error: The I/O operation has been aborted because of either a thread exit or an application request" at 03:14:32. Followed by multiple Get Propery Errors : "The port is closed" from 03:14:32 to 06:44:20.. At 06:44:21 there is a 'Disconnecting from port COM6' message, whereupon the log ends.  A new log is opened at 06:51 and the connections with the telescope are all normal and show the telescope reconnection and the final Park operation.
   
   Examination of  DeviceHub logs shows normal communications up to 03:14:33 at which point property calls start showing " (failed)".  03:14 is the same time that ObsPics show a loss of connection to UPB with 'Scope Power', 'Dew Heaters', Foc2 Focuser and 'UPB Temperature' all dropping at. Scope COM port also goes missing at this same time, but comes back (COM6) at 03:17.
   
   It is almost certain that the problems with Telescope Connection stem from the lost of connection with the UPB Powerbox, and to a temporary (180s long ) interruption to the COM6 Port (which serves the Telescope).
   
   The reporting of a discrepency between TheSky Coordinates and ASCOM Telescope Coordinates and the associated raising of a "TheSky6 Scope Coords" red alert only occurs when the discrepency in position is greater than 3 arc deg.
    
3. 03:30 to 05:57. The issue with Target 23/30 (SN 2024wak) at 03:30 was followed by a succession of resume/suspend cycles   Each cycle was unsuccessful.  ObsManager's diagnosis for each of the 15 cycle failures was the put down to being a likely Frozen TheSky6 Program/Telescope and the same attempted fix to  'Restart TheSky6' was used in each case.  This was fruitless and simply a waste time after the 3rd attempt.
   
   The RestartTheSky6 operation results from a call to the CheckTheSkyTelescopeCoords() routine, which after encountering an exception from calling TheSky6.Telescope.GetRaDec (giving the log message "GetRaDec failed - is the telescope connected ?")
   left both TheSkyRA & TheSkyDec with the value 0 (leading to log message "Ra and Dec coordinates are both 0.0. This is likely a problem" and to setting bError = True).   It would be interesting to know what the precise exception message was for the Telescope.GetRaDec calls and whether they contain "The port is lost" or not.  but unfortunately these messages weren't logged in AstroMain 3.71.5 (this is now added to AstroMain 3.71.6)
   
   As part of pre-slew in GotoTarget() routine the CheckTheSkyTelescopeCoords() routine is called.
    
4. 03:30 to 05:57. Of the 15 opening & closing of the Shutter between 03:30 and the end of the session, only one of the shutter closures was directly related to deteriorating weather conditions.   The other cases were ordered by ObsManager after failure to get fresh information from the TheSky6 following a restart of TheSky program. 
    
   Each Resume Session Cycle with Unsuccessful Target Slew is just presenting potential wear/tear to shutter equipment, shutter battery and exposures the observatory to unnecessary risk from rain etc, if a fault develops The ObsManager should have recognised that it had an issue on its hands that was not solveable and after 3 attempts with the same fix should have probably left the session suspended / dome closed till the morning, when user could resolve the problem.  Stopping and Restarting DeviceHub (without Test For Stalled Dome) should probably have been tried by the ObsManager, but wasn't included in the AstroMain 3.71.5 design. 
   
   The issue with Telescope connection was finally resolved by the user during post-session recovery when Device/Hub and MeadeGeneric Driver were restarted at 06:51 but it is unclear if success at this time depended on UPB Powerbox Software/Services being restarted by the user at 06:33 (ObsPics show that Scope Power, Dew Heaters and Foc2 Focuser remained unavailable to AstroMain until 06:33). AstroMain wouldn't have permitted a Telescope Reconnection at 06:51 if the Power State wasn't available, but that wouldn't have prevented a Telescope Reconnection by the user using DeviceHub.
    
5. 06:22. The failure to Park Telescope operation at end of session is almost certainly due to the continuation of the Telescope Connectivity problems that began at 03:30, and persisted till the end of the Job Queue / Session at 05:57 or so. 
    
6. 06:29. The ASCOM.DeviceHub error dialog with message "The port is closed" shown by TheSky6 when computer was checked at 06:39 is quite likely associated with the failed 'Slew To Near Park' attempt at 06:22.   Earlier error dialog may have previously popped up but would have been lost / closed when TheSky6 was restarted on multiple occassions between 03:30 and 05:57.
   
   The error dialog again confirms that the MeadeGeneric driver was still affected by the earlier 3 minute outage of the COM6 port between 03:14 and 03:17.
   
7. 06:31. It is isn't possible to say when exactly the unhandled exception in AstroVOE occured other than it was before 06:31.  The last VOE message was logged by the program was made at 03:55 and it therefore likely that the program crashed with the unhandled exception occurred shortly after this.  AstroMain made its first call to RestartTheSky6 at 03:30 with the last of a multiple calls to RestartTheSky6 being made at 05:47:20. It could have been associated with the call to RestartTheSky6 that was made at 03:56:54.

It seems clear that Obs Manager should have tried the Restart DeviceHub/Restart Telescope Service at an earlier point between 03:30 and 06:22 in order to have reestablished a working connection with the telescope before it came to the Park Telescope operation during the Session's Closing Stage. 

Conclusion

• The various problems appear to all stem from an event at 03:14 that knocked out connection with the UPB Powerbox / ASCOM Switches, which also saw the COM6 Port to the Telescope go down for around 3 minutes.
   
• The lost of COM6 upset the MeadeGeneric Telescope Driver and it couldn't regain connection with the Telescope, even though the COM Port possibly came back after 3 minutes. Connection to the Telescope could only be recovered by restarting the MeadeGeneric Driver which required stopping/restarting DeviceHub. 
  
• The Connection to UPB Powerbox (and to information on / control of Scope Power, Dew Heaters, Foc2 Focuser & UPB Temperature) wasn't reestablished until action by the user during post-session recovery.  The ObsManger could/should have automatically perfomed the action to 'Restart UPB & Services'.
   
• ObsManager action to try Restart TheSky6'  on 15 successive 'resume session/suspend session cycles was fruitless, and after 2-3 attempts to restart TheSky6.  The ObsManager should have taken the action of restarting DeviceHub (with the aim of restarting the MeadeGeneric Telescope Driver).   This action would likely to have allowed additional targets observations/images to be taken in the remainder of the session and would have allowed the Telescope to be successfully parked at the end of the session.
   
• The call to TestForStalledDome() within the RestartDeviceHub() routine is inappropriate and should be disabled.
   
•  It's unclear why the UnauthorizedAccessException occured in AstroVOE but the program clearly needs to catch & handle the exception without crashing.
  
• There is insufficient logging of exceptions from calls to TheSky6.Telescope.GetRaDec().

Actions:

•   AstroMain / Restart DeviceHub . Modify RestartDeviceHub() to remove call to Dome.TestForStalledDome(0)
  Done 2024-10-12 (AstroMain 3.71.6)
   
•   AstroMain / ObsManager.  Modify ObsManager to call Restart UPB, followed by RestartDeviceHub() in situation where connection to Telescope has been lost (which is shown to not be due to a frozen TheSky6).
  To Do 2024-10-12 (AstroMain 3.71.6 / 3.72)
   
• AstroMain. Job Executor to write TheSky6 & ASCOM telescope coordinates when it detects a discrepency that would require TheSky6 to be restarted. 
   
• AstroMain Logging. Exceptions encountered when call Telescope.GetRaDec should be logged 
   
•   AstroVOE.  Add a Try..Catch..End Try Block to AstroVOE.globals.TWatcher.OnChanged() to catch an unhandled exceptions when deleting the RestartTheSkyFlag.dat. The same protection has been added to equivalent routine in AstroPlan as a precaution.
  Done 2024-10-12 (AstroVOE 1.15.1, AstroPlan 1.42.3)

AstroMain design is being modified to identify scope port issues producing 'The port is closed' exceptions & resultant problems seen recently in the Session S1253  (2024-10-10) and to automatically resolve the issue, allowing the session observing activities to be successfully continued and for the telescope to be parked at the end of the session.

Issue Identification

RunObsMonitor()

• Existing routine
• Modification to 'Monitor Scope Port' section to
  - record & report time that scope port closed  (Scope.PortClosedAtTime)
  - record & report time that scope port reopened (Scope.PortOpenedAtTime)
• Constantly running (loop every 2s)

CheckTheSkyTelescopeCoords()

• Existing routine
• Modification to raise bResolveScopePortIssueRequired flag upon encountering a 'The port is closed' exception.

CheckAscomTelescopeCoords()

• New routine
• New routine to check telescope coordinates via DeviceHub (instead of via TheSky6)
• Raise bResolveScopePortIssueRequired) flag upon encountering a 'The port is closed' exception.

ObsManager MainLoop()

• Existing Routine
• Modification to look out for issues with Scope.Port being lost and refound and if neecessary to raise the bResolveScopePortIssueRequired flag

Issue Resolution

ResolveScopePortIssue()

•  New routine
• Carries out actions to resolve Scope Port Issue (where MeadeGeneric driver has been affected by closure / temporary loss of COM Port to the Telescope)
• Closes Dome (if it is open)
• Calls RestartDeviceHub to
     - communicate the restart with AstroVOE and AstroGuard
     - closes Dome/Scope connection
     - stops/closes DeviceHub & MeadeGeneric Driver
• Calls Dome.Connect  to reconnect to dome
• Calls Scope.Connect to reconnect to telescope
• Calls TheSky to reconnect to telescope
• Checks Dome, Checks Telescope
• Lowers relevant flags

ResolveScopePortIssueByThread()

• New routine
• Starts ResolveScopePortIssue() as a threaded operation. 

ObsManager MainLoop()

• Existing routine
  
• Modification to check status of bResolveScopePortIssueIsRequired flag and if required & conditions are suitable intervene to call ResolveScopePortIssueByThread() routine to activate ResolveScopePortIssue().  This takes place at 3 points
  
  - Towards the end of the 'Session Starting'  stage, before Shutter is open and before moving to 'Session Running'.
  -  During the 'Session Suspended' stage
  -  At the beginning of the 'Session Closing' stage
  
  At each point there is the execution of 2 new stages, the first stage calls ResolveScopePortIssueByThread() if appropriate, whilst the second waits until the ScopePortIssue has been resolved if appropriate.

Job Executor ()

• Various routines
• Modification required to exit the current target in event of a raised bResolveScopePortIssueRequired flag
  (unable to slew), but leave the Job Queue in state that the current target can be re-tried after Session Suspension & Resolution of ScopePortIssue (provided end of time slot hasn't been reached).

Actions:

•   Add new FailureState for recording Scope Port Issue.  FailureState ScopePortIssue = 34