Issue: Images taken over several hours with shutter closed (wasting useful observing time), and possible related issues (Obs.Monitor and Job Queue unexplainably freeze at 01:56 and unexplainably restarted at 03:02 (approx 1 hour later)), 

Description.
Observatory was taking series of Targets under relatively clear (but lowered transparency skies).   The job executor had completed Target 26/42 (KUG 0830+278 w/SN2020adow) at 01:45 and was performing target checks for Target 27/42 (UGC 5245) at 01:46.   These checks don't seem to have been completed.

Logs show that the  Obs.Monitor had become frozen at 01:47.   Then at 01:49 there were alerts reporting that the LAN and link to AllSky/Weather Computer had both been inactive for 3 minutes.    ( Last Successful Pings were at 01:46:52).

At 01:57 the Observatory Overseer determined the Job Queue had become frozen (last IamAlive message at 01:46) and proceeded to call the 'Make Observatory Safe' routine, which closed the Shutter (50s), turned off Telescope tracking and slewed the dome to Park position (Az 090 deg).

At 03:02 the LAN and Link to AllSky/Weather Computer became active again (outage of 75 minutes).   After testing for Stalled Dome condition (automatic response to Dome appearing stationary for 78 minutes, and large difference between dome and scope positions), the Job Queue resumed by completing target checks for Target 27/42 (UGC 5245) and slewing to the target at 03:03. After attempting to center the target, the main images were then all skipped as the target's End Time slot had been passed. 

The next 3 targets were skipped as their planned start times had been missed.    Next 9 targets were then fully executed (ie slew to target , attempted centering, attempted guiding/focusing, acquisition of main frames) with the observatory shutter closed.  Dome Position checks (including reporting the Shutter Status) were made but no action was taken  (to suspend/stop session or open the shutter)

Session was checked remotely at around 06:20. Seeing that the Shutter was closed, an attempt was made to Open the shutter using DeviceHub at 06:26:49.    AstroMain reported that Shutter State was Error  (See  Investigation:  Shutter Opening failed after 10-37s with ShutterStatus=5 (Error) ) and the Shutter was then commanded to close using DeviceHub at 06:29:20.   A second attempt to open observatory also failed and a User Interventation as therefore made to close the Session at 06:32.

Time		Event	Detail
01:25:06		Target Started (26/42)	Target started (26/42, KUG 0830+278 w/SN2020adow)
01:45:03		Target Completed	Target completed (26/42, KUG 0830+278 w/SN2020adow)
01:46:59		Target Started (27/42)	Target started (27/42, UGC 5245)
01:47:29		Job Queue Frozen	Job Queue appears to have frozen at around 01:46 - 01:47
01:58:21		Dome Closed	Dome closed (closing time 50s)
01:59:09		Dome Parked	Dome moved to Az 90 deg (Park Position)
03:02:10		Dome	Running test to see Dome Driver is stalled
03:02:52		Dome	Dome Driver/Dome Slaving found to be ok.
03:08:59		Target Completed	Target partially completed (27/42, UGC 5245)
03:09:01		Target Missed (28/42)	Target's time slot was missed (28/42, UGC 5267)
03:09:03		Target Missed (29/42)	Target's time slot was missed (29/42, UGC 5760 w/SN2020zoq)
03:09:05		Target Missed (30/42)	Target's time slot was missed (30/42, GCVS AM CVn)
03:09:07		Target Started (31/42)	Target started (31/42, UGC 5459)
03:30:55		Target Completed	Target completed (31/42, UGC 5459)
06:18:32		Target Started (40/42)	Target started (40/42, NGC 5331 w/AT2020abir)
06:29:39		Dome Closed	Dome closed (closing time 15s)
06:29:44		Target Aborted (40/42)	Target aborted (40/42, NGC 5331 w/AT2020abir) due to cloud
06:29:46		Queue Paused	Queue paused for Session Suspension
06:29:49		Session Suspended	Session suspended
06:30:04		Dome Closed	Dome closed (closing time 15s)
06:31:54		Session Resuming	Session resuming
06:32:04		User Intervention	User request to close and finish session
06:32:06		Queue Aborted	Job Queue aborted
06:32:09		Session Closing	Session closing
06:32:29		Dome Closed	Dome closed (closing time 15s)
06:33:20		Dome Parked	Dome parked (parking time 43s), Az: 90.0 deg
06:36:29		Session Finished	Session Finished

Time		Alert	Detail
01:49:55		LAN	Local Area Network has been inactive for more than 3 mins
01:49:57		AllSky/Weather Computer	Link to AllSky/Weather Computer (Laptop7) has been inactive for more 3 mins
01:56:31		Dome	Dome has been unexpectedly stationary for more than 12 mins
01:57:30		Job Queue	Job Queue appears to be frozen (last IamAlive message at 01:46)
03:02:12		LAN	Local Area Network is active again (outage 75.3 mins)
03:02:14		AllSky/Weather Computer	Link to AllSky/Weather Computer (Laptop7) is active again (outage 75.3 mins)
03:02:22		Dome	Dome Movements have begun again (outage 77.9 mins)
03:02:52		Dome	Dome Driver/Dome Slaving found to be ok following Slew Tests.

No ObsPics were generated for the period 01:47 to 03:01.  These are produced by the DrawDome() routine which is called by the Observatory Monitoring Thread (RunObservatoryMonitoring() routine)

Obs.Manager seemed to operate ok throughout . Arguably not enough action taken by Obs.Manager to understand the actual status of the observatory operations and intervene.

Investigation

Q.  What was cause and what was effect ?   Did a) the LAN/AllSky Computer link became inactive, and this somehow caused the Obs.Monitor and Job Queue to freeze  or  b) the LAN/AllSky Computer were simply percieved to have become inactive due to a frozen Obs.Monitor ?.   How to test whether LAN was working or not ?

One way might be to see if ObsEnv data was getting passed from Obs. Computer through to AllSky/Weather computer between 01:56 and 03:02

On the Observatory Computer the OTA Temp became 'stuck' on -5.6 degC between 01:56 and 03:02,  but Pulsar Shutter health data continued to record real values. e.g. at around 01:59-02:00 there are distinct changes in all voltage, battery % and  milliamp values  as the Shutter is closed.   These changes are seen in the ObsEnv file held on the AllSky/Weather computer indicating that the LAN and AllSky Computer link were working throughout.

AstroPlan's log shows 'Live' session files being uploaded to Website throughout this period (with the exception of the ObsPic.gif file). This again indicates than the LAN was operational thoughout and the Alerts were a by-product of a stalled Observatory Monitoring thread.

Q. What caused Obs.Monitor and Job Queue to both freeze at 01:56 and then start working again at 03:02 ?.  

The last reported message from Job Queue was
   Checking Coordinates ...
with the results (Ok     J2000 Coords RA: 09 47 34.441, Dec: -02 01 58.44) only being reported some 10 minutes later.

This would point towards a stall point within  LookupKeyTargetInfo() routine (or alternatively in LookupKeyTargetInfo2() )  however it isn't obvious what would have caused it.  It is conceivable that the problem was caused by an issue with TheSky6 program or its COM objects.

It isn't possible to determine where the stall point lay within the RunObservatoryMonitoring's loop.

Issues / Actions

• Job Queue should not have continued executing multiple targets with a closed Observatory Shutter.   Shutter should have either have been opened to allow Targets to be properly executed (using usual checks prior to opening dome) or targets should have been skipped.
   
• Observatory should have been placed into Suspended Mode following execution of the MakeObservatorySafe routine.   Made need to review how this should be done (is it responsibility of Obs.Overseer or Obs.Manager ?)
   
• Add Failure type 'Closed Shutter' for failures were shutter isn't open.  Current failure modes (Hardware, Software etc) are not useful to describe the issue.
   
• Slew to Target/Centering should be skipped if Target is beyond its End Time Slot.  Target 27/42 (UGC 5245) was inadvertently delayed between 01:46 and 03:02 due to frozen job queue (frozon Obs.Monitor). hutter should have either have been opened to allow Targets to be properly executed (using usual checks prior to opening dome) or targets should have been skipped.
   
• Inactive Lan / Inactive AllSky Computer Alerts should be qualified if Obs.Monitor is frozen. Alerts are produced by the Obs.Manager by looking at LastRouterPingTime and LastAllSkyPingTime,  but if Obs.Monitor is frozen then these LastPingTimes will be 'stuck' . Obs.Manager should look at ObsMonitorCycleTimePoint or LastPingChecksTime to decide whether to change the alert message to
        LAN/AllSky Computer Ping checks have stopped.

These Issues /actions have been added as Operational Issues or Continuous Improvement items for the session (see S847 Operational Issues)

Issue: a) OpenShutter request at  06:26 failed after 27s with ShutterStatus=5 (Error). 
              b) OpenShutter request at  06:29 failed after 10s with ShutterStatus=5 (Error).

Description.

Open/Close Shutter activity recorded by DeviceHub:
06:26:49.040: Dome - Commands: OpenShutter: (done)
06:29:20.051: Dome - Commands: CloseShutter: (done)
06:29:39.863: Dome - Commands: OpenShutter: (done)
06:29:49.567: Dome - Commands: CloseShutter: (done)
06:31:54.499: Dome - Commands: OpenShutter: (done)
06:32:04.420: Dome - Commands: AbortSlew: (done)
06:32:14.402: Dome - Commands: CloseShutter: (done)

Pulsar Trace Log shows the following:

1) dome driver received (Manually Initiated) request to OpenShutter at 06:26:49 after which ShutterStatus changes to 2 (Opening) at 06:26:50.
Shutter and continues to show as 'Opening' until 06:27:16 (27s later), when it changes to 5 (Error).  

2) dome driver received (Manually Initiated) request  to CloseShutter at 06:29:20 after which ShutterStatus changes to 3 (Closing) at 06:29:20
Shutter and to 1 (Closed) at 06:29:37 (closing time 17s)

3) dome driver received (Manually Initiated) request to OpenShutter at 06:29:39 after which ShutterStatus changes to 2 (Opening) at 06:29:40.
Shutter and continues to show as 'Opening' until 06:29:49 (9s later). At 06:29:49 there is a (Obs.Manager/Session Suspension) request  to CloseShutter after which the ShutterStatus changes to 3 (Closing), before changing to 1 (Closed) at 06:30:00 (closing time 10s).

4) dome driver received a (Obs.Manager/Session Resuming) request to OpenShutter at 06:31:54 and ShutterStatus changes to 2 (Opening) at 06:31:55.
Shutter, changing to 5 (Error) at 06:32:05 (10s later).  At 06:32:14 there is a (Obs.Manager/Session Closing) request  to CloseShutter,  and the ShutterStatus changes to 3 (Closing), before changing to 1 (Closed) at 06:32:25 (closing time 10s).

Conditions at Time
  Frosty/Freezing.
  Outside Temperature -7 degC

Investigation

Dome Open/Closing was tested the following day in daylight conditions.  The shutter opened ok but the stutters as the shutter/chain connector jumps over the pulley wheels were again very noticeable (as they usually are).  Examining the higher guide wheel shown that it has a slight notch worn it to it where the screw head on the shutter/chain connector hits the wheel.  It is theorised that the resistance at the guide wheel points, cold temperature and ice combined to cause shutter opening to stall and produce the Shutterstatus=5 (Error).   It is assumed that the stall after 27 sec corresponded with the upper guide wheel, whilst the later stall after 10 sec corresponded with lowermost wheel.  The higher pulley wheel was rotated to offer a fresh surface for the chain and the Shutter/Chain Connector spacing was adjusted in an attempt to reduce the resistance of the connector coming over the wheels, but there didn't seem to be any improvement.       

Photo showing shutter chain passing over upper guide wheel (chain is too slack / i.e. not tight enough) (photo taken 2021-01-09)		Photo showing chain/shutter connector passing over upper guide wheel (chain bolt spacing is 2 nuts + 1 washer) (photo taken 2021-01-09)

Action:
Pulsar Observatories were contacted for advice, and after receiving advice,  a couple for adjustments were made to the Observatory's Shutter Chain. The shutter chain was tightened by removing one link and the spacing on Chain bolt was reduced by taking out one bolt and replacing with a couple of washers (see Shutter Chain Adjustements, 2021-01-15).

Photo showing shutter chain after tightening (photo taken 2021-01-15)		Photo showing upper pulley wheel with tightened chain and reduced chain bolt spacing (chain bolt spacing is 1 nut + 3 washers) (photo taken 2021-01-15)