Issue.    Closing Session task stalled whilst attempting to the close the Observatory Shutter / Dome not reporting correct Shutter State.

Description.   Checking Session status via web page in morning indicated that Observatory Shutter was Open. A look from the house window showed that the Dome wasn't parked. The shutter couldn't be fully seen but enough could be seen to show that it was 95% closed or 100% closed.     Remote connection to the observatory computer showed Device Hub was displaying the Dome at Az 255 , and  'slewing' (alternating between closing and opening states) .  Visiting the Observatory showed that the shutter was closed, but the Device Hub was still showing the Dome as slewing and alternating between closing and opening states.   Although Shutter was closed the shutter was tight against the lower lip of the shutter aperture. 

The Dome Control Unit's Shutter Information showed that the Shutter Battery was at 91% (so ok).   Dome was parked (some slipping noticed), Dome connections were closed and  Pulsar Dome Unit rebooted (power turned off / turned on).  This seems to have resolved things.    Telescope Parked via Device Hub and Powered off.  Session ended using 'Close/Finish Session' button. 

Analysis

Shutter first opened at 18:49 and then underwent 6 close/reopen cycles due to short cloud/snow shower events with a 7th shutter closing at 01:34 when thicker high cloud developed.  Shutter remained closed until 05:37 when condition slightly improved and shutter was commanded to open.  Shutter was reported to not have become open after 90s with the message there 'may be a mechannical, electrical of software issue' with reporting of 'Dome will be close'.   Session resumption failed and the session returned to Suspended State. 

From 06:25 onwards (when Clarity values dropped below 24) the Obs.Manager started issuing Critical Cloud and Critical RainThreat Alerts every 5s/10s with the message 'Observatory Manager will close the Dome'.

Pulsar Log shows following :-

05:37:19  Shutter 1 (Closed)
05:37:20  OpenShutter
05:37:21  ShutterStatus 2 (Opening)
05:37:22  Shutter 2 (Opening)
05:38:51  CloseShutter
05:38:51  ShutterStatus 3 (Closing)
05:38:51  Shutter 2 (Opening)
05:38:51  ShutterStatus 2 (Opening)
05:39:01  CloseShutter

   etc    Shutter 2 (Opening)
06:25:20  CloseShutter
06:25:30  CloseShutter
06:25:40  CloseShutter
  etc     ShutterStatus 2 (Opening)
  ect     ShutterStatus 3 (Closing)
  etc     Shutter 2 (Opening)
  etc     CloseShutter
08:40:27  CloseShutter
08:42:50  CloseShutter
08:53:16  Shutter 2 (Opening)

AstroGuard and DeviceHub show the Dome Status alternating between Opening and Closed states.
Pulsar logs confirm this. After each CloseShutter command, the Pulsar sets ShutterStatus to 3 (Closing), but this isn't heard by or acted upon by the Shutter, which a moment later returns Shutter State 2 (Opening).

Conclusion
This would seem to be a glitch in the Dome/Shutter System, which resolved itself after being rebooted.  

It is possible, but not certain, that the problem related to a hypothesized mechanical issue opening the shutter at 05:37 whereby the shutter was tight against the lip of the lower aperture due to a bulge in panel below lower aperture (something which stopped the shutter from closing in S932, see Investigation - Shutter failed to fully close (2021-11-10) ), coupled with the particular orientation of the shutter & possible icing,  and as a consequence insufficient torque could be generated to open the shutter, which stalled and left the shutterstatus on 'opening'.

AstroMain acted according to its design, and kept trying to Close the shutter every 5/10s when it reported that it was 'opening' and weather conditions indicated that it wasn't safe to be open.

AstroGuard acted according to its design and monitoring the situation. It didn't intervene as Shutter state kept switching back to Closing.

Key thing is that shutter was closed (not open) through this time and the observatory was therefore physically safe.

Should there have been a point when after trying a certain number of times that Obs.Manager should given up trying to Close Shutter ?

After a certain number of cycles Obs.Manager should probably have monitored the dome for a while and determined that the 'Opening' state never turned into an 'Open' State, and therefore there must be some showstopping issue with Dome Driver or Dome itself that made it non-operational.  Obs.Manger should then issued an Alert and reduced the frequency at which further close attempts are made.   

AstroGuard needs to also have similar behaviour or communication with AstroMain otherwise it will continue making Dome Close requests.

Update 2023-11-09

Since Oct 2022 the observatory has had a zigbee sensor at the bottom of Dome's shutter that provides independant indication of when the shutter is fully closed, and this has worked well .  Following an incident in the S1162 session where the shutter was reported to have not finished opening after 90s,  and unable to tell when the shutter physically become fully open it would be useful to have an extra zigbee sensor at the top of the shutter to indicate when the shutter is fully open.

A new Zigbee Sensor of the same type was fitted to the top of the shutter on 2023-11-09 and AstroMain 3.64.5 modified to report the open/opening/closing/closed status of the shutter independant of the Pulsar Dome driver (see New Zigbee Sensor for Dome's Shutter (Top), 2023-11-09)

This review summarises the routines and safeguards associated with opening and closing the Observatory Dome. 

It has been made in response to an investigation into a dome opening issue during sesssion S938 (2021-11-28), and other dome opening/closing issues over the past 2 years.

Overview.  
Being able to operate the Observatory Shutter remotely and also robotically  is critical to the safe and effective functioning of the Observatory.   Preventing the opening of the dome or closing it in the event of inclement weather (especially rain or snow) is particularly critical.

Whilst the Shutter can be manually operated via the Main Dome Control Unit,  a Red Button on the Shutter Drive Unit, or via Pulsar Dome Software in practice the shutter is normally operated via an ASCOM connection from the AstroMain Observatory Control Program to the Pulsar Dome via DeviceHub & Pulsar ASCOM Driver.   This ASCOM driven operation of the shutter may from manually initiated requests to open/close the shutter or more usually in response to automated operation of the observatory.

Shutter Mechanism
Shutter is opened and closed by a Shutter Drive Unit using a motor with toothed sprocket that turns a heavy-duty plastic chain attached to the shutter. This is powered by a 15V lithium battery which automatically recharges via an induction charger when the Dome is at its Park position.  Limit sensors are depressed when the shutter is fully open or fully closed to tell the Drive Drive the state of the shutter. The Shutter Drive Unit communicates with the Main Dome Control Unit via radio/bluetooth, informing the Control Unit about the status of the shutter and of the battery and responding to requests to open or close the shutter.

The shutter mechanism is generally fairly reliable, and operates 98% of the time witthout an issue,  but there are some critical weaknesses which have given problems in the past and could do so again.   (discussed further below)

• Shutter Chain.  Shutter Chain parted and left the Drive Unit unable to close the shutter  (date )
  (Status:  Successfully resolved by repositoning the Shutter Drive Unit)
   
• Shutter Software.   Shutter Unit failed to close shutter on 1st attempt as it had gone into a sleep state
  (Status: Successfully resolved by updated Shutter Software that remains permanently awake)
   
• Shutter Hang-Up.  Shutter can potentially hang-up on the lip of the lower aperture and thereby not fully close due to an outward bulge in the panel  below the lower aperture (shutter left is a 'closing state'). 
  (Status : Resolution attempted by removing rubber strip on lower aperure, which make it thinner/less likely to cause a hang-up)
  (Update : Shutter failed to open and one occasion since,  Shutter may have been jammed against the panel below the lower aperature after a closure and wouldn't reopen (shutter left in an 'opening' state).
   
• Limit Switches.  Shutter left physically still partly open even though Shutter State was showing Closed. Possible glitch with limit switches may be the cause.  (Status:  One early case, related to a potential short circuit due to an exposed bare wire, was resolved.  A later case however has no firm explanation (may be a cat jumped up and pressed the limit switch ?)

Safeguards
The following safeguards are present to ensure that dome is closed during bad weather or other critical circumstance.

• AstroMain
  •  program will monitor weather data provided by AstroWeather/AstroAllSky and recognise conditions deemed to be  'bad weather'
     
  • program will prevent dome opening in the event of bad weather
     
  • program will command closure of the dome in the event of bad weather
     
  • program will ensure that AstroGuard (safety backup) is running whenever the dome is open
     
  • program will monitor the state of the Shutter Drive battery
     
  • program will prevent dome opening or close dome in the event of Shutter having a low battery level
     
  • program will monitor the Dome connection and attempt reconnection in event of unexpected disconnection or COM Port change
     
  • program will self monitor itself and initiate a MakeObservatory Safe mode in event of issue with the primary monitoring & control
     
  • program will issue an alert to user if there is a critical issue that cannot be resolved or made safe.
    
    (at the moment this alert system isn't fail-safe as it depends on the user logging in remotely or using observatory/session web page to see the alert, audible alerts can be provided but IPad has to be active and open. Need a way of sounding an audible alarm in the house or making a phone call to user mobile)
     
• AstroGuard
  •  program will monitor weather data provided by AstroWeather/AstroAllSky
    
  • program to command closure of the dome in the event of bad weather where there is no response from AstroMain
     
• Cloud Sensor / Dome Control Unit
  • Cloud Sensor will monitor weather conditions continuously and in the event of Clarity, Rain or Light exceeding set thresholds will activate relay output to the connected Dome Control Unit
     
  • Dome Control Unit will command closure of the dome after a set time (30s) in the event of the relay being activated
     
• Shutter Drive Unit
  • Shutter Drive Unit will automatically close after a set time (5 mins) in the event of loss of connection to Main Dome Control Unit
     which could happen due a Power Cut for example.
     
• AstroLaunch
  • program should automatically connect to dome and command closure of the dome in the event that the Observatory Computer crashes and reboots  (this process is not currently operational, 2021-12-01  and needs to be investigated)

Potential Points of Failure
Points of potential residual failure are as follows:

 

AstroMain

Principal software control of the shutter lies with the AstroMain observatory control program. Key components are

• 'Dome'
  A 'Dome' object manages dome functions and communicates with the physical dome using an ASCOM link (via a ascom dome object 'objDome' connected to DeviceHub.Dome that is connected to Pulsar Observatories Dome).  It includes routines for Commanding the Opening and Closing of the Dome, which may be called in response to user requests (manual mode) or automated and protective requests from the Obs.Manager.
   
• 'ObservatoryManager'. 
  An Obs.Manager daemon both pilots and safeguards the observatory when in robotic (automated) mode or simply safeguards the observatory when in manual mode.   A 'heartbeat' function allows Obs.Overseer and AstroGuard to judge whether Obs.Manager is in full control of operations or has failed or stalled.
   
• 'PerformDomeMonitoring'
  Dome Monitoring is called by the Obs.Manager to monitor status of the Dome and of particularly functtions such as MonitorDomeOpening, MonitoringDomeClosure.
   
• 'Obs.Overseer'
  This daemon monitors the running of Obs.Manager and in event of it stalling will intervene and call a MakeObservatorySafe routine that includes closure of the dome.

Open Shutter (Detail)

Opening the Shutter is performed by calling Dome.OpenShutter() routine.  This makes final checks that it is safe to open the Shutter. It is safe to open the dome and opening operations aren't already in progress it calls Dome.OpenShutterByThread() routine. This starts Dome.OpenShutterTask() as a threaded operation (to prevent any chance of it locking up the calling process) , which calls objDome.OpenShutter to perform the actual action.  The OpenShutterTask routine checks progress after 10s and if the Dome hasn't started to open it makes a second call to objDome.OpenShutter, provided that weather/environmental conditons are still ok for opening the dome.

Dome.OpenShutter () is called from 4 places :

- from User Interface
   - Open Shutter button (btnToggleDomeShutter.Click) when requested by user (manual mode)

- from Obs.Manager daemon
    -  SetState_Resuming() when session is resuming following a previous suspension
    -  MainLoop()  during Session.Equilibrating when conditions ok and shutter not aleady open
    -  MainLoop()  during Session.Continuing when conditions ok and shutter not aleady open

Close Shutter (Detail)

Opening the Shutter is performed by calling Dome.CloseShutter() routine. Provided that closing operations aren't already in progress it calls Dome.CloseShutterByThread().  This starts Dome.CloseShutterTask() as a threaded operation (to prevent any chance of it locking up the calling process) , which calls objDome.CloseShutter to perform the actual action.  The CloseShutterTask routine checks progress after 10s and if the Dome hasn't started to close  it makes a second call to objDome.CloseShutter.

Dome.CloseShutter () is called from 18 places:

- from Buttons on User Interface
       - Close Dome button (btnToggleDomeShutter.Click)  when operating in manual mode
       - Park Dome button   (btnParkDome.Click)
      -  Restart DeviceHub button (btnRestartDeviceHub.Click / RestartDeviceHub()

- from Dome Object
    -  DomeCloseShutterAndWait()
    -  Dome.CloseShutter2()  (2 places)
    -  Dome.ParkDome (2 places)

- from Obs.Manager daemon
    - SetState_Suspended()  when session is being suspended due to bad conditions
    - MainLoop() during Session.Suspended when conditions ok and shutter not already open
    - MainLoop() during 'End of Session/Catch All rule
    - MainLoop() during Review Dome Lock rule
    - MainLoop() during Snow Protection rule
    - MainLoop() during Ensure Shutter is closed is session is suspended (backup) rule
    - MainLoop() during MustClose / Session.Running / 'Initiate Close Dome immediately in event of rain or thick cloud'
    - MainLoop() during MustClose / Other State (Automated)
    - MainLoop() during MustClose / Manual Mode
    - PerformDomeMonitoring()  when trying to reverse a failed OpenShutter operation

PerformDomeMonitoring (Detail)

The normal flow path in PerformDomeMonitoring is that the progress of a dome opem/close operation (open shutter or close shutter) is monitored, that movement of the shutter is reported when first seen (dome is opening or dome is closing) and reported when the operation has completed (dome is open or dome is closed).   When the operation is interrupted, such as when dome closing is started because cloud thickens or it starts raining while dome is opening, this is reported and monitoring switches to monitoring the closure.  If an open or close operation hasn't completed within a certain time (90s) then there is deemed to be an error somewhere (either mechanical, electrical, comms or driver) and this is flagged as an alert requiring interventation.

Self diagnosis & actions for self resolution are still at an early stage.    Program has the capability to a) read / understand the ASCOM Log from the Dome and this can show up discrepencies between what the Pulsar Dome considers the status of the shutter to be versus what ASCOM driver is saying.   b) Restart the Dome Driver (disconnects Dome from AstroMain and sends a message to AstroGuard from the Dome, which should mean that all clients to Dome are now closed, connections to Dome are then automatically remade).  These can be helpful in some circumstances.  

Control Issues

• Bounching between Open and Closed Dome States
  This is a particular issue associated with marginal observing conditions or highly variable/rapidly changing conditions.   The issue is particular relevent for the typical highly variable weather conditions that we have here in NE Scotland, rather the more stable conditions that would be more typically for a mountain top professional observatory.
  
  Where a single specific value (threshold) is used to define the boundary between Good and Bad Flags then if a given parameter (say Clarity) bounches around this threshold value, it can lead to the dome being repeatdly opened and closed in quick succession leading to interrupted / incomplete observing of targets,  increased drawdown on the shutter battery and over time the increased rate of wear on the dome shutter components & motor.
  
  One solution is employ two additional parameters for each condition category, a bad delay and a good delay.  These are the time that the 'conditions daemon' waits between an input going bad/good and the setting of the condition flag accordingly which has the effect of smoothing out any sudden spikes in a value and help ensure that the dome will not be opening & closing too often.
  
  Another solution is to use a different threshold for setting a Must_Close flag from setting a Can_Open flag, with the threshold to open the dome being set at a higher (stricter) level than the threshold for closing the dome.   This automatically reduces any tendancy to bounce between open and closed dome states too often.   In this solution when the dome is closed the bias is towards keeping the dome closed until conditions are really suited for observing. When the dome is open the bias is towards keeping the dome open and the scope still taking images even in deterioting conditions until those conditions eventually either improve and observing continues with the next target (Continue), doesn't improve and the dome is closed/session suspended at end of the current target (SoftSuspend) or the conditions rapidly deterioriate to the point that dome must be closed and the dome is immediately closed (HardSuspend).
  
  (In case of a 'must-close' Rain flag the dome closure is immediately triggered and only then are scope/dome rotation/camera operations aborted whilst in the case of a 'must-close' Cloud_Flag the normal operation is to abort scope/dome rotation/camera operations and wait for Job Executor to enter a standby state, before proceeding to closing the dome.  This is to present more ordered reporting of operations)
  
  On some nights where there are passing shower clouds which may or may not drop some rain or snow on the observatory,  but have clear skies in between, there is no real option other than to expect that there may be some 6 of 7 dome close/dome opening cycles in a single night. 
   
• Interference between the main observatory control program and other safeguards
  This is a potential issue associated with multi-layer protection system whereby a particular safeguard can interfer with the functioning of the main observatory control program/script.  
  
  One example where this occurred was where the main safeguard program (AstroGuard) using a more resticted set of information and potentially slightly different thresholds deemed that weather conditions were Bad and commanded the dome to be closed, whilst the main observatory control routine (AstroMain Obs.Manager) working with more complete/more integrated information deemed that the weather conditions were still Good and commanded the dome to be opened, which ended up in a 'tug of war' involving 8 - 10 close/open cycles until a user intervention was made.
  
  This is to be resolved by ensuring consistency between data used and to use heatbeat communications to inform the safeguard program that the main control program was still operating and could be safely presumed to have lead control over the dome shutter, whilst retaining independant automomy with the power to command the immediate closure of the dome in the event of rain or thick cloud.
  (in respect to cloud conditions the safeguard program might consider the just cloud sensor's clarity (sky temp based) value , but the main observatory control program may also consider the number of stars visible in images from the observatory's AllSky Camera, or consider a moving average clarity rather than single values).
  
  Other examples have occurred when the Relay Safeguard was either triggering a dome closure based on a single rogue value or was closing the dome based on say deteriorating cloud values before the main observatory control program could halt current operations and close the dome in a coordinated way.   Solution was to employ a delay (30s) in Dome's Relay Settings so that a single rogue value won't trigger a closure, provide time for main observatory control program to control the halt current operations and close the dome in the manner it considers most appropriate.   The threshold for 'must-close' Cloud_Flag  in the main observatory program is set to a slightly more conservative value that the one in the Cloud Sensor so that the main control program will normally act to close the dome, before it cloud thickens enough to trigger the Cloud Sensor relay.
   
• Repetition of close/open commands which aren't working.
  When the shutter opening has been attempted but theshutter continues reporting Opening (as being the last state before it had some form of error), then the natural thing to attempt is to close the dome to make it safe pending investigation by the user at some point which may not be until the following morning. How many attempts should be made ?    In the event of bad weather (cloud, wind, rain threat based on radar data) the observatory manager will see 'Opening' status from the ASCOM Dome each cycle (every 5-10s) and make repeated attempts to close the shutter shutter and the ASCOM Shutter status will continuously flip between Closing (the immediate status following a CloseShutter command) and Opening (when the dome status refreshes with the status from the Dome Control Unit or with the staus that the Dome driver holds).  When this happens there is confusion over what exactly is happening and the Dome Event tables, program logs and program reports fill up rapidly with each attempted operation/failure.

Code Changes (2021-12-03)

AstroMain 3.48 contains a number of cosmetic and small technical improvements and enforcements of the design described above.
 

Tests (2021-12-03)  - ShutterStatus 4 / ShutterError

When Shutter is opening , regardless of how it was commanded to open (DeviceHub or Client),  the title of the 'Shutter' button in Device Hub changes to 'Opening in Progress' and the button is grayed/disabled. The only way to reverse the movement when it is moving and close the shutter in Device Hub is to click the red square 'Abort' button and then click the  'Close Button' which has retitled itself and become enabled again.

When the Shutter has a ShutterError state (meaning unknown position), the only follow-up movement that is possible in DeviceHub is to Close the Shutter (an Open Button is never presented as an option when shutter has the error state).

When using with Dome Simulator,  and with Shutter in ShutterError state, it is found that clicking on the Simulator's 'Open'  button produces the message 'Shutter must be Closed to clear the error'. 
  
It would seem a prudent course of action - if there is an error, then closing the shutter is a safer bet than trying to open the shutter. 

When DeviceHub's Dome Abort button is used (which calls Dome.AbortSlew) the ShutterStatus reported by the DeviceHub.Dome connected to Pulsar Observatories Dome is '4' (ShutterError).  (Due to a bug in the Pulsar ASCOM Dome Driver the error state is written as '5' in the Pulsar ASCOM log file, however it probably fires the correct value (4) back to DeviceHub.

During S939 the Pulsar ASCOM log did actual report a ShutterStatus 4 during open closing shutter attempt.  Suggests that Pulsar's state 4 and stare 5 might mean different things ?   (The ASCOM Standard set of ShutterStates only includes State 4, it doesn't have a State 5)

Update 2023-11-09

Since Oct 2022 the observatory has had a zigbee sensor at the bottom of Dome's shutter that provides independant indication of when the shutter is fully closed, and this has worked well .  Following an incident in the S1162 session where the shutter was reported to have not finished opening after 90s,  and unable to tell when the shutter physically become fully open it would be useful to have an extra zigbee sensor at the top of the shutter to indicate when the shutter is fully open.

A new Zigbee Sensor of the same type was fitted to the top of the shutter on 2023-11-09 and AstroMain 3.64.5 modified to report the open/opening/closing/closed status of the shutter independant of the Pulsar Dome driver (see New Zigbee Sensor for Dome's Shutter (Top), 2023-11-09)