Issue:  Dome failed to finish opening after 90s

Description

  

Timeline

The Timeline associated with the issue

Time		Event	Message / Data	Source	Note
20:15:01		Event (Session Equilibration)	Session ready to Open Dome	Events table
20:15:01.77		Open Shutter	Call OpenShutterByThread / Call objDome.OpenShutter	Log file	Opened by ObsManager
20:15:01.94		"	Open Shutter request  (Pulsar)	Pulsar log
20:15:02.78		Shutter opening	Observatory Shutter has begun opening	Log file / Zigbee
20:15:03.15		"	Shutter Opening (Pulsar)	Pulsar log
20:15:06		"	Shutter Opening (Relay Safe / Sensor Open)	ObsPic
20:15:36		"	Shutter Opening (Relay Safe / Sensor Open)	ObsPic
20:16:01.86		"	Dome opening hasn't finished after 60s	Log file
20:16:09		"	Shutter Opening (Relay Safe / Sensor Open)	ObsPic
20:16:30.36		"	Dome opening hasn't finished after 90s	Log file
20:16:31		"	Dome failed to finish opening after 90s (ShutterState=Opening)	Alerts table	Notifcation of the Issue
20:16:31.94		"	Setting bDomeOpenOpeningIssue=True	Log file
20:16:32.18		"	Shutter still Opening (Pulsar)	Pulsar log
20:16:39		"	Shutter Opening (Relay Safe / Sensor Open)	ObsPic
20:18:25 ~		"  (Ambiguous)	Shutter has opened (203s)	AstroGuard	Uncertain time (time is inferred)
20:19:44		"	Shutter Opening (Relay Safe / Sensor Open)	ObsPic
20:19:50		" + Observatory Door Opened	Observatory Door has opened	Events table / Zigbee	User enters Observatory
20:19:51		"  (Ambiguous)	User observes that shutter is open and its drive motors are off	User
20:20:15		"  (Ambiguous)	Shutter Opening / Relay Safe / Sensor Open	ObsPic
20:20:17.19		"	Shutter still Opening (Pulsar)	Pulsar log
20:20:18.16		Dome Closing	Shutter Closing (Pulsar)	Pulsar log	No 'Close Shutter' request logged !
20:20:22.35		"                         (unexpected)	Dome is unexpectedly closing (Good Conditions)	Log file & Alerts table	Closed by who/what ?
20:20:22.35		Open Dome	Opening Dome (AstroMain)	Log file	Opened by ObsManager ?
20:20:22.49		"	Open Dome Request (Pulsar)	Pulsar log
20:20:33.29		Dome Open	Shutter is open (Pulsar)	Pulsar log
20:20:37.37		"	Dome is open (15s)	Log file
20:20:37		"	Dome opened (opening time 15s)	Events table	15s is anomalously short
20:20:45		"	Shutter Open (Relay Safe / Sensor Open)	ObsPic
20:20:49.63		Dome Slewing (Az)	Calling Telescope.SlewToRaDec (Near Zenith Focus Point)	Log file
20:20:57.18		Close Dome	User Request to Close Dome	Log file	User trying to resolve
20:20:57.20		"	Close Shutter request  (Pulsar)	Pulsar log
20:20:58.49		Dome Closing	Shutter Closing (Pulsar)	Pulsar log
20:21:15		"	Shutter Closing / Slewing Az (Relay Safe / Sensor Open)	ObsPic
20:21:41.23		Dome Closed	Observatory Shutter has closed (20:21:41, open time 6.6 min)	Log file / Zigbee
20:21:41.96		"	Shutter Closed (Pulsar)	Pulsar log
20:21:45		"	Shutter Closed / Slewing Az (Relay Safe / Sensor Closed )	ObsPic
20:21:47.53		"	ShutterStatus: Closed (T+50.3s) / Dome is Closed	Log file & Events table
20:21:51		Open Dome	Open Shutter (DeviceHub ?)	User	User using DeviceHub
20:21:51.51		"	Open Shutter Request (Pulsar)	Pulsar log
20:21:52.07		Dome Opening	Shutter Opening (Pulsar)	Pulsar log
20:21:52.35		"	Observatory Shutter has opened (20:21:52, close time 11 sec)	Log file / Zigbee
20:21:52		"	Dome is unexpectedly opening (Good Conditions) - Red Alert	Alerts table	Opened by DeviceHub/User
20:22:33.51		Dome Open	Dome open	Pulsar log
20:22:37		"	Dome opened (opening time 45s)	Events table
20:23:46		Observatory Door Closed	Observatory Door has closed (open time 3.9 min)	Events table	User leaves Observatory




Analysis

•  Timeline raises the following questions:
  • what does the raising of bDomeOpenOpeningIssue flag lead to ? (AstroMain)
  • when did the Shutter physically finish opening ? (we can't tell, supports getting any extra zigbee sensor)
  • any connection with previous cases (e.g
     - Investigation:  Shutter Opening failed with ShutterStatus=5 (Error), 2021-01-08
    -  Investigation - Dome not reporting correct ShutterState, 2021-11-28
    -  Investigation - Commands to open Observatory shutter failed, 2022-04-10
    
• Examing Obs.Manager's 'Equilibrating' code section we see that provided than conditions are suitable (determined by calling CanResume() routine) and we haven't already made 3 attempts to open the dome,  an 'Open Door" Subtitle is reported and the Dome.OpenShutter (CalMode.Auto) routine is called. which calls OpenShutterByThread() which initiates Dome.OpenShutterTask() as a threaded operation.
  
  The OpenShutterTask() calls objDome.OpenShutter()  and then watches for the shutter to start opening before exiting.
  
  Once the shutter is moving the 'PerformDomeMonitoring()' routine is activated and monitors the shutter opening.
  If Shutter hasn't opened with 90s,  the routine raises a red alarm, raises bDomeOpeningIssue flag (setting it to True) and increments the nDomeOpeningAttempts counter by 1, lowers the MonitorShutterOpening flag (setting it to False).
  
  If there had been a Dome Closing Issue the routine would have then entered 'Dome Closing Diagnosis & Recovery' section of code, but there isn't an equivalent for a Dome Opening Issue and the
  
  ObsManager execution continues after 5s with the next cycle trhough MainLoop.  Provided that conditions are still suitable,  the 'Equilibrating' code regognises that Dome.ShutterIsOpening, but as per current design it does nothing and 'waits' until a later cycle when the shutter is open.   But as Dome.ShutterStatus remains 'stuck' on 'ShutterState.shutterOpening nothing happens.   Potentially ObsManager would continously cycle through this point until 'PlanFinalTime', was reached.  This is evidentally a 'design flaw', the ObsManager should take a more proactive approach when Shutter is 'stuck' on 'Opening'.
  
  In the event that the dome had stayed closed, two further attempts would have been made to Open Shutter, after which if the dome was still not open, the ObsManager would drop back to Manual.
  
  In the actual event,  in S1162 Session there is an unexplained change from ShutterStatus 2 (Opening) at 20:20:17.19 to ShutterStatus 3 (Closing) at 20:20:18.16.  The Pulsar ASCOM Log doesn't log any  'CloseShutter' request at around this time (does the supposed request fail to be logged as Dome was opening at the time, and it is only logged when the Dome is Open. ? a test would validate or invalidate this hypothesis.
  
  What ever the reason ShutterStatus changed to ShutterStatus 3 (Closing), the equilibrating section in ObsManager picked up on this state change and called the Dome.OpenShutter() routine again.
   
  
• It is impossible to say exactly when the Shutter finished physically opening.
  - it started opening at 20:15:02
  - normally the Shutter takes 50s to open, and it is most-likely that shutter finished opening at around 20:15:52 unless there was a physical hold-up)
  - Astroguard reporting infer that dome was open 20:18:25 (from its reported 203s opening time)
  - User visit to the observatory records that dome was physically open by 20:19:51 (latest)
  - Pulsar Log records that dome was open at 20:20:33.29 (corresponding to a gross opening time of 331s)
   
  It would have been helpful if there was an independant means of indicating when shutter has physically finished opening
  
  It is proposed to plug this information gap in future by ordering and installing an Zigbee Open/Closed sensor at the top of the shutter and program into AstroMain's reporting.
   
• The 3 earlier cases in 2021-2022 were examined but neither are a close match for the current case.

Conclusion

•  

Actions

• Order and install a zigbee 'Door/Window Sensor' to provide independant indication that the Dome's shutter is fully open
  Done 2023-11-09
   
• Investigate what does the raising of bDomeOpenOpeningIssue flag lead to ? (AstroMain)
  Done 2023-11-08
   
• Add a Dome Opening Diagnosis & Recovery section to AstroMain's PerformDomeMonitoring() or otherwise prevent 'Equilibrating' become stuck in a persistent loop waiting in vain for 'ShutterStatus=ShutterState.Opening' to 'clear'.
  To Do.