Issue:  

• Dome failed to fully shut during session suspension and safe-recovery attempts (23:48)

Description

• In response to Critical Cloud Conditions (Clarity 23.8 ie < 24) a HardSuspend call was made at 23:46:27, which led to the current target (18/37, AT2023wvl) being aborted and the dome begin to close. AstroMain's ObsOverseer reported a Red Alert/Alarm at 23:48:07 with message "Dome closure hasn't finished after 90s (ShutterState=Closing)", followed at 23:49:02 by a red alert with the message "Dome hasn't reached 'Closed' state after 120s during HardSuspend".  The session moved to Suspended State at 23:48:32 with the shutter still not fully closed.
  
  The shutter's new Top Zigbee Sensor records and event that that indicated that the shutter began closing at 23:46:33, but there is no corresponding event from the Bottom Zigbee Sensor to indicate that it had closed.
  
• Attempts at safe recovery in order to finish closing the dome also failed producing similar red alarms/alerts at 23:50:37 & 23:51:07. 
   
• Observatory was visited at 23:52:47 and confirmed that the Dome's Shutter was stationary and in a position that was around 75% closed (ie it was still partially open).  There was no obvious reason why the shutter has stopped closing at this point. 
   
• Observatory Tab & ObsPicture continued to show the Dome as 'Closing' .
  
• Attempts to close/reopen the Dome via  in AstroMain at 23:55:13 didn't reset the Shutter State.  DeviceHub.Dome tab showed that the shutter was busy 'Closing Shutter' .  'AstroMain reported 
     "Check Shutter  Ok  Hub reports dome is Closing (3)
  
  The Pulsar Dome's ascom log indicates that the Shutter was Closing (3)'.  Either the driver or the Dome Unit itself is stuck on 'Closing' .
   
• The shutter was eventually closed though a combination of manually pushing red button on the Shutter Drive Unit and using open/close shutter options from the Dome Control Unit. The Shutter Sensors reported it to be finally closed at 23:54:28.  
  
• At 23:55 the 'Restart Pulsar Dome' facility was activated which stopped and restarted the Pulsar Dome, after which the shutter was opened and then closed using the Dome Control Unit. The zigbee shutter sensors picked up the opening operation from 23:57:01 to 23:57:42 (41s) and the closing operation from 23:57:54 to 23:58:38 (44s). The Dome Status in ObsViewer/ObsPics which was previously showing 'Dome is Closing'  with a Yellow Flag then became 'Dome is vcNull' with a white flag after re-connection.
   
• Finally it was decided to Stop Observatory Services and Close AstroMain, Close AstroGuard and Stop/Close Device Hub and then Restart AstroMain (after upgrading to AstroMain 3.64.4), reconnnect to Dome/DeviceHub and restart the session. After some quick checks and finally happy that the shutter situation was under control again, the session was put back under Automated Operation (using AutoStart to do this).
  
  The incident should to be investigated & reviewed and any lessons and/or actions identified.   
  
  [  It should be noted that this issue occurred prior to Pulsar Dome Software/ASCOM Driver Update on 2023-11-15
   (see Pulsar Dome - Software Update (Firmware 1.51 & ASCOM Driver 6.3.10), 2023-11-15). ]

Analysis

• Question - What caused shutter to stop moving at point around 75% of the way through the act of closing ?
   
  - Was it caused by a mechancial hold-up ?
  - Was it an electrical / software glitch in the Shutter Drive Unit ?
  - Did The Dome Controller have a glitch and command Shutter to abort ?
   
  There is no firm evidence that particularly favour one these possibilities more than the others.
   
  - the problem doesn't seem to be due to a mechanical issue, as the Shutter was eventually closed and later Open/Close actions all proceeded without further incident.
   
   - based on Shutter starting to close at 23:46:33 and normally taking 44s to fully close, it is estimated that the shutter stopped closing after roughly 33s (44 * 0.75) which would be at around 23:47:06.
  
• The Pulsar Ascom Log shows Volatile records changing at the times listed below.  These show the shutter data was refreshing ok between 23:46:33 and 23:47:09, but then the data was seemingly frozen from 23:47:09 to 23:55:08 (when the Pulsar Dome was restarted).  
  
  The time of last fresh data at 23:47:09 is a close match to the time that the shutter is estimated to have stopped closing (23:47:06).
   
  23:46:33    312.8 0 0.000000 312.7 0 3 950 14937 -3 -9447 282 2513 0
  23:46:43    312.8 0 0.000000 312.7 0 3 950 14937 -3 -9446 282 2513 0
  23:46:47    312.8 0 0.000000 312.7 0 3 950 14937 -3 -9447 282 2513 0
  23:46:50    312.8 0 0.000000 312.7 0 3 950 14937 -3 -9446 282 2513 0
  23:46:52    312.8 0 0.000000 312.7 0 3 950 14365 -576 -9446 1029 2502 0
  23:46:55    312.8 0 0.000000 312.7 0 3 950 14372 -642 -9446 1029 2502 0
  23:47:09    312.8 0 0.000000 312.7 0 3 950 14548 -371 -9446 1278 2496 0
  23:55:08    312.8 0 0.000000 312.7 0 3 950 14548 -371 -9446 1278 2496 0
  
•  Why did the Dome Status in AstroMain's ObsViewer/ObsPics become ' Dome is vcNull' with a white flag after Restart Pulsar Dome operation ?
  
  The Pulsar Ascom Log shows shutter Status changes at the following times.   Between 23:55:41 & 22:55:49 the Pulsar driver was reporting a Shutter Status of 6.  This is not a standard ASCOM Shutter Status value, and is believed to be one that Pulsar Observatories finds useful to use to describe the specific case were the Pulsar Dome has just been connected after a reboot and it hasn't yet received an initial status value from the Shutter.
  
  23:55:41         connected
  23:55:41   6    booting up
  23:55:50   1   closed
  23:57:02   2   opening
  23:57:43   0   open
  23:57:54   3   closing
  23:58:39   1   closed
  00:01:26        disconnected
  00:02:56        connected
  00:02:56   1   closed
  00:13:09   2   opening
  00:13:51   0   open
   
  DeviceHub.Dome's Ascom logs (for AstroMain and AstroGuard connections to Dome, show unexpected records.  Normally these logs and indeed logs from the night prior to Get ShutterStatus: shutterClosing as pre- shows records
    Get CanSetShutter:  True
    Get ShutterStatus: shutterOpen
   
  but in the logs after the PulsarDome Restart ( 23:55+),  records shows
    Get CanSetShutter: False
    Get ShutterStatus: CanSetShutter = False prevents the operation.
   
  This explains why ObsPics produced by AstroMain continued to show 'Dome is vcNull ' even after Pulsar Driver logs shows dome as closed, opening, open & closing through the period 23:55 to 23:58.   It would seem the DeviceHub.Dome is having a glicth.
   
  DeviceHub.Dome logs show normal records after Dome was reconnected at 00:03.   ObsPics resume showing the expected ShutterStatus after the reconnection.
   

Conclusion

• The 'Red Button' method of shutting the dome is a last resort and is clearly not one that can be conducted remotely or via automated programming.
    
• Automated recovery routines should proceed to calling 'Restart Pulsar Dome' when other attempts to close the dome have failed
   
• In the event of continued problems after restarting the Dome, DeviceHub should be restarted.

Actions

• Modify ObsManager/ObsOverseer to 'Restart Pulsar Dome' when other attempts to close the dome have failed, and in event of continuing problems to also restart DeviceHub
  To Do  (AstroMain)
   

Since Oct 2022 the observatory has had a zigbee sensor at the bottom of Dome's shutter that provides independant indication of when the shutter is fully closed, and this has worked well .  Following an incident in the S1162 session where the shutter was reported to have not finished opening after 90s,  and unable to tell when the shutter physically become fully open it would be useful to have an extra zigbee sensor at the top of the shutter to indicate when the shutter is fully open.

A new Zigbee Sensor of the same type as previous (Aqara MCCGQ11LM Door & Window Sensor, https://www.amazon.co.uk/Aqara-MCCGQ11LM-Door-Window-Sensor/dp/B07D37VDM3) was ordered 2023-11-08. The sensor arrived 2023-11-09 and was the two parts fixed to the shutter and dome roof and installed on the observatory's zigbee network (AstroGW) the same day.  The sensor properties record type=ZHAOpenClose, manufacturer=LUMI, modelid=lumi.sensor_magnet.aq, swversion 20161128, and took sensor id = 12.

Network Diagram showing Observatory's Zigbee Network after the addition of the 'Shutter Top' Sensor.

 

AstroMain 3.64.3 was updated to access the new sensor ' as new device 'ShutterTopSensor' and used during session S1164 Session (2023-11-09)

First results from using the new device are shown by the following report messages that were produced whilst opening the dome at the start of the session.

Open Dome                               2023-11-09 18:18  (Local, GMT)
  Check Conditions            Ok        Conditions are ok (Conditions are suitable for opening dome)
> Observatory Shutter         Info      Shutter has opened            (18:18:05, close time 6.5 min)
  Open Dome...                Ok        Dome is opening   (Clarity: 37.5, Stars: 33, Rain: -0.9, Wind: 0.0)
> Observatory Shutter2        Info      Shutter2 is fully open        (18:18:45,  > 7.8 min)
  Open Dome...                Ok        Dome is open      (Opening time 45s)

Later results showing the associated reporting after futher development and fine tuning is shown below

Open Dome                               2023-11-11 14:01  (Local, GMT)
  Open Dome...                Ok        Dome is opening   (Clarity: 40.5, Stars: 0, Rain: -1.2, Wind: 5.5)
> Observatory Shutter         Info      Shutter is opening            (14:01:46, close time > 1.6 min)
> Observatory Shutter         Info      Shutter is open               (14:02:26, opening time 40s)
  Open Dome...                Ok        Dome is open      (Opening time 46s)

Close Dome                              2023-11-11 14:03  (Local, GMT)
> Observatory Shutter         Info      Shutter is closing            (14:03:02, open time 36s)
  Close Dome...               Ok        Dome is closing   (Clarity: 41.9, Stars: 0, Rain: -1.2, Wind: 6.3)
> Observatory Shutter         Info      Shutter has closed            (14:03:44, closing time 43s)
  Close Dome...               Ok        Dome is closed    (Closing time 51s)

Update 2023-11-11
The small block of wood with the attached sensor magnet came away from the shutter last night. (probably because the wood block + magnet was in an awkward position and couldn't be held in place whilst the adhesive set and gravity simply caused it to eventually fall away).  A more secure method of fitting the wood block + magnet using 'nano gel tape' (suggested by my wife from the beginning, https://www.creativeproducts.ltd.uk/nano-gel-tape/ ) was employed and is holding firm.

Photo showing the new 'Top Sensor' with the shutter in its fully open position. 



Notes:
- The smal block of wood is used to bring the small magnet unit up to the same level as the larger main sensor unit.
- Adjacent to the Top Sensor magnet is the Magnet that is associated with the Bottom Sensor
- Below the new Top Sensor, is the Wired Top Sensor used by the Pulsar Dome for detecting the fully open shutter state.