After finishing the S642 session and believing the Dome Shutter was Closed (Shutter flagged as close in POTH Hub and in my control program), it was by chance that I later looked out and saw that there was darkness showing where the white closed shutter would normally sit with dome parked - was the shutter still open ? .  Investigating closer I found that the Shutter was indeed open. Quite a shock and not expected.  I made  a new remote desktop connection to the observatory computer and even though the dome was flagged as 'closed' in several places including POTH.Hub.  I again commanded the dome to close which this time it did as confirmed  by going out to the observatory again.  
(This was a near miss since later in the night it rained.  Incidents like this are a potential showstopper for unattended operation of the observatory and need to be fully investigated & resolved)

Time Line

Time		Activity
19:01		objscope connected to POTH.Dome (COM9)
19:02		Job Queue restarted
19:59		Obs. Manager closed Shutter due to critical cloud conditions (44s to close) Session suspended
20:01		Obs. Manager reopened Shutter, conditions improved (45s to open)
20:02		Job Queue resumed
20:15		Obs. Manager closed Shutter due to critical cloud conditions (4s to close)
20:32		Obs. Manager reopened Shutter (conditions improved) (5s to open)
20:40		Multiple Messages from Obs. Manager that it will Close Shutter due to critical cloud conditions  (4s to close)
20:42		Obs.Manager set to manual, Queue Stopped
20:42		Closed Shutter, manual via AstroMain (5s to close)
22:16		Correctly Closed Shutter (manual, via POTH.Hub window)

A summarised version of the trace logs covering the critical time intervals is shown below.  Non-relevant lines are removed for brevity.

Time		Key	Value
19:59 (control)			Obs. Manager closed Shutter due to critical cloud conditions
19:59:19.527		Volatile	119.1 0 0.000000 119.1 0 0 51 12813 89688 -21256 11488 2418 0
19:59:23.583		ShutterStatus	0
19:59:23.786		CloseShutter	Completed  (completed mean that the command has been initiated)
19:59:24.488		ShutterStatus	3
20:00:06.749		ShutterStatus	3
20:00:06.359		Volatile	119.1 0 0.000000 119.1 0 3 51 12813 89688 -21256 11488 2418 0
20:00:09.853		ShutterStatus	1     (46s to close shutter)
20:00:08.340		Volatile	119.1 0 0.000000 119.1 0 1 51 12813 89688 -21256 11488 2418 0
20:01 (Control)			Obs. Manager reopened Shutter (conditions improved)
20:01:27.666		OpenShutter	Completed
20:01:27.682		ShutterStatus	2
20:02:07.618		ShutterStatus	2
20:02:09.037		ShutterStatus	0   (42s to Open Shutter)
20:15 (Review)			Hypothesis:  Obs. Manager closed Shutter due to critical cloud conditions (4s)
20:15:19.866		ShutterStatus	0     (shutter is open)
20:15:20.318		CloseShutter	Completed    (Close Shutter command has been sent /received )
20:15:20.490		ShutterStatus	3     (shutter is closing)
20:15:20.490		ShutterStatus	3     (shutter is closing)
20:15:21.207		ShutterStatus	3     (shutter is closing)
20:15:23.672		ShutterStatus	1     (shutter is closed)     (3s too close the shutter is abnormal  ! )
20:32 (Review)			Obs. Manager reopened Shutter (conditions improved)
20:32:10.351		ShutterStatus	1     (shutter is closed)
20:32:10.366		OpenShutter	Completed   (Open Shutter command has been sent & received )
20:32:10.694		Volatile	218.5 0 0.000000 218.5 0 1 51 12813 89688 -14286 11488 2418 0
20:32:11.708		Volatile	218.5 0 0.000000 218.5 0 2 51 12813 89688 -14286 11488 2418 0
20:32:12.722		Volatile	218.5 0 0.000000 218.5 0 0 51 12813 89688 -14286 11488 2418 0
20:32:12.738		ShutterStatus	0   (1s to Open Shutter is abnormal)
20:42 (Review)			Closed Shutter (manual, via AstroMain)
20:42:11.548		ShutterStatus	0     (shutter is open)
20:42:11.610		CloseShutter	Completed    (Close Shutter command has been sent & received )
20:42:11.782		ShutterStatus	3     (shutter is closing)
20:42:13.654		ShutterStatus	3     (shutter is closing)
20:42:14.012		ShutterStatus	1     (shutter is closed)     (2s too close shutter is abnormal  ! )
22:16 (Review)			Correctly Closed Shutter (manual, via POTH.Hub window)
22:10:16.685		ShutterStatus	0 (shutter is open)
22:10:16.763		CloseShutter	Completed    (Close Shutter command has been sent & received )
22:10:18.385		ShutterStatus	3     (shutter is closing)
22:11:00.084		ShutterStatus	3     (shutter is closing)
22:11:00.974		ShutterStatus	1     (shutter is closed     (44s to close shutter is normal)

It's unclear why problem occurred.      The approximate time to open and close the shutter is recorded and these demonstrated that the Shutter was not taking the normal 44-45s to close / reopen the shutter, but was taking 4-5s instead at operations at 20:15, 20:32 and 20:40.  The ObsManager runs on a loop with 5s intervals, so the time taken may have been even less.

- It doesn't immediately appear to be a hardware problem.
  (shutter was working ok at 19:59 & 20:01,   it worked when shutter was finally closed at 22:16 and was working ok when it was tested the following day)

- it doesn't appear to be a connection issue between Dome Controller and Shutter Drive Unit,
 (shutter was working ok at 19:59 & 20:01 and it worked ok at 22:16 when shutter was finally closed.  The trace logs don't suggest any outage in communications)

- it doesn't appear to be a dome disconnection / Com Port issue
 (slaved dome rotating ok to targets beforehand and dome parked ok at 20:42, this suggests POTH.Hub was working and connected to Pulsar Dome.  Logs confirm that the Pulsar Dome was on COM9 throughout the session and didn't change port)

- this might then suggest a software issue either with my objDome object disconnected from POTH.Dome or an issue within my Obs.Manager program. However this can be ruled out as the ASCOM Trace Logs from Pulsar Dome Unit are consistent with the commands sent and the responses seen.

My best hypothesis is that shutter closure sensor had either become temporarily stuck in the depressed/closed state or there was a short across the wires leading to the sensor. The wires have little nicks in them from day one at the points where they join the sensor which may have shorted if a drop of water had fallen onto and enveloped the sensor wires. Basically the controller reported that the shutter had gone from fully open to fully closed in 2-3 seconds when the normal travel time is 40-45s.  

The only flaw in the hypothesis is that with the shutter open then with close limit stuck  it would be expected that the Shutter Controller would put out an error state (State 4) when there was a command to close it because it can see both limits are activated. However no such Error State = 4 are present in the Trace Log.

Analysis of the available data suggests that AstroMain and POTH.Hub were working fine, but there was a issue with the Dome System whereby it believed it was moved into a shut position when it in fact hadn't.   Could this be the result of a temporarily stuck relay switch ?

Whilst my AstroMain program couldn't know the cause in anyway it had information to say that Open/Shut Operations were taking significantly less time than expected and could have highlighted this fact in someway.  In particular the final Manually initiated Close Shutter operation could have put up a Message Dialog to alert the user that there was an abnormality and to recommend that Shutter is physically examined.

Way Forward:
>> Communicate to Pulsar Observatories technician for comment
>> Examine relay switch & wires and repair
>> Attempt to emmulate a shorted relay wires.
>> Modify code to highlight any abnormalities in the Open/Shut Closure times 

Update 2021-04-13
There were no similar issues in the following sessions, until session S884 (2021-04-13), when the dome was left only 80% closed at the end of the session despite ObsViewer, ObsManager, Device Hub & Pulsar Trace Log all reporting that Dome was 'Closed'.

This later incident was investigated but could not be explained.  (See Investigation - Shutter didn't fully close at end of session, 2021-04-13 )

• Dome Software & Drivers were updated today using the  'Pulsar_Observatories_Package 1.0.32.0.exe'  installer
• Dome Rotation Driver updated from 1.30 to 1.39
• Shutter Driver updated from 1.17  to 1.22
  

- Started Pulsar program which proceeded to update the Dome Controller
   Installer progress indicator proceeded to 100% and hung with the message 'Don't Turn off Computer or Devices' still showing.
- After 3 hours with no apparent activity (no CPU usage) it was decided to intervene and kill the installer program and power cycled the Pulsar Dome Controller.  The  Dome Controller powererd up ok and checking Settings / Info showed that the controller had been duly updated to 1.39

AstroProtect is a new addition to the AstroSuite set of programs for operating the Clair3 Observatory. It's purpose is to protect the observatory in the event that the Observatory PC unexpectedly restarts whilst the observatory is in operation.   Upon computer restart the program connects to the Dome Controller amd commands the Shutter to Close (if Open) and the Dome to Park itself (if Unparked).    The program is executed upon computer start-up and doesn't require the user to have first logged in or for the login process to be circumvented risking security.

After initital development and testing on the User PC, the program was installed on the Observatory PC on 2018-12-23.

The program was added as a startup script by running the Group Policy Editor gpedit.msc (available on Windows Pro) and then opening Windows /  Start-up & Shutdown Scripts, and then clicking Add (Note: If the program was added to user StartUp or General Startup it wouldn't  execute immediately upon Computer restart but would require user login to occur first)

Adding AstroProtect to Windows Startup Scripts using Local Group Policy Editor

The AstroProtect program was then tested  to ensure that it worked on the Observatory Computer. Test was conducted by opening the dome shutter, rotating the dome away from its parked position and then commanding Windows to Restart (to simulate a crash/auto restart cycle). After a short delay during which the computer restarted the dome began to automatically close and then rotate to it's Park Position.

Report Log from AstroProtect Test, 2018-12-23

An UPS (Uninterrupted Power Supply) review has been in progress over recent days (Dec 2018), but hasn't yet been written up.   

At some stage 1 or 2 UPS systems will be ordered for the observatory, but this is currently on hold.

Update Mar 2024
A further review of UPS systems was carried out in Mar 2024 and it was decided to order and install a UPS System in the Observatory Computing bay for back-up battery power to the Observatory Telescope.
(see Review - Off Line Uninterruptible Power Supply (UPS) for the Observatory Telescope, 2024-03-03
  &  Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply System, 2024-03-03 )