David's Astronomy - Notes 2019-02-17 (S00664)

Following a shutter opening incident in the last session on 2019-02-13 S663 and following investigation into why the Pulsar Dome Controller couldn't detect that shutter had finished opening and allowed the shutter motor for up to 40 mins before it was noticed new code was added to AstroMain to intervene where shutter opening appears anomalous and prevent the motors from running ad-infinitum.

Whilst the new modify shutter opening routines had been tested this session (S664) was the first live full-system test of the new code.

After entering 'Session Equilibrating' state, the Shutter began opening under command of the Obs.Manager, but within seconds the dome unexpectedly began to close which was noted as an Alert . After a while the new ('opening') code kicked in with the Error message that the Shutter hadn't finished after 90s and would have issued a new command to close the shutter. The shutter (which was already closing) finished closing 5s later.

Some 5 minutes later dome was opened using the POTH.Hub panel, but again there was the message dome is unexpectedly closing (even though conditions were good). Finally the Obs.Manager was set to Auto, and the Observatory visited to check the status of the the Dome and opened via POTH.Hub without further incident.

Session Equilibrating                   2019-02-17 18:28  (Local)
  Plan Start Time             Ok        2019-02-17 18:36  (Local)
  Open Shutter...             Opening
  Close Shutter...            Closing
Dome Closure:                 Alert     Dome is unexpectedly closing (Good Conditions), 2019-02-17 18:28
  Shutter Safety              Error     Shutter opening hasn't finished after 90s
  Shutter Safety              Info      There may be a mechannical or electrical issue.
  Shutter Safety              Info      Shutter will be closed.
  Close Shutter...            Ok        Shutter is Closed (5s)
  Close Shutter...            Closing
Dome Closure:                 Alert     Dome is unexpectedly closing (Good Conditions), 2019-02-17 18:35
  Close Shutter...            Ok        Shutter is Closed (6s)
  Close Shutter...            Ok        Shutter is Closed (5s)
  Close Shutter...            Closing
Dome Closure:                 Alert     Dome is unexpectedly closing (Good Conditions), 2019-02-17 18:35
  Close Shutter...            Ok        Shutter is Closed (5s)
  Close Shutter...            Closing
Dome Closure:                 Alert     Dome is unexpectedly closing (Good Conditions), 2019-02-17 18:36

Observatory Mode 2019-02-17 18:36 (Local)
Setting Mode ... Ok Manual

Back in Jaunary when there were several cases of the dome shutter unexpectedy closing, but it was shown then that the commands to close weren't coming from the Obs. Computer via ASCOM but were glitches in the Cloud-Sensor - Dome Controller relay system. However this new run of 'unexpected/non-designed' closures are clearly caused by commands sent via ASCOM, as the following trace log abstract shows

18:28:11.741 ShutterStatus             1     << Shutter is closed

18:28:13.452 ShutterStatus             1

18:28:15.621 OpenShutter           << 
Initial Command recieved to Open Shutter 
18:28:15.621 ShutterStatus             2     << Shutter is opening

18:28:15.621 ShutterStatus             2



18:28:18.678 ShutterStatus             2   

18:28:20.628 ShutterStatus             2

18:28:20.753 CloseShutter          << 
Command recieved to Close Shutter 
18:28:21.814 ShutterStatus             3     << Shutter is closing

18:28:24.996 ShutterStatus             3

18:28:25.776 ShutterStatus             3

18:28:25.792 OpenShutter           << Command recieved to Open Shutter

18:28:27.913 ShutterStatus             2     << Shutter is opening

18:28:31.143 ShutterStatus             2

18:28:31.158 ShutterStatus             2

18:28:31.720 ShutterStatus             2

18:28:31.876 CloseShutter        << 
Command recieved to Close Shutter 
18:28:32.235 ShutterStatus             3     << Shutter is closing

18:29:25.868 OpenShutter         << Command recieved to Open Shutter

18:29:26.726 ShutterStatus             3

18:29:29.534 ShutterStatus             1     << Shutter is closed

18:29:30.735 ShutterStatus             1

Examining the entire set of Open/Close Shutter Commands it is evident that commands are generally being issued on a 5s or 10s interval that corresponds to the cycle time of the Obs.Manger loop (5s). On one cycle of the loop there is a command to open the shutter, whilst on the next round there is a countermanding command to close the shutter. This points clearly to a bug in the Obs.Manager with two different control processes interfering with each other.

18:28:15.621 OpenShutter     << Initial Command to 
Close Shutter at end of session
18:28:20.753 CloseShutter             

18:28:25.792 OpenShutter              

18:28:31.876 CloseShutter             

18:28:35.885 OpenShutter              

18:28:40.799 CloseShutter             

18:28:45.697 OpenShutter              

18:28:50.674 CloseShutter             

18:28:56.025 OpenShutter              

18:29:00.876 CloseShutter 
18:29:05.946 OpenShutter              

18:29:10.704 CloseShutter             

18:29:15.712 OpenShutter              

18:29:20.720 CloseShutter             

18:29:25.868 OpenShutter              

18:34:53.017 OpenShutter              

18:34:56.449 CloseShutter             

18:35:01.550 OpenShutter              

18:35:06.324 CloseShutter             

18:35:11.285 OpenShutter              

18:35:16.479 CloseShutter             

18:35:22.719 OpenShutter              

18:35:26.292 CloseShutter             

18:35:31.611 OpenShutter              

18:35:36.697 CloseShutter             

18:35:41.533 OpenShutter              

18:35:46.353 CloseShutter             

18:35:51.377 OpenShutter              

18:35:57.227 CloseShutter             

18:36:01.423 OpenShutter              

18:36:02.141 OpenShutter              

18:36:06.353 CloseShutter             

18:36:09.270 OpenShutter              

18:36:25.041 CloseShutter             


22:54:56.393 CloseShutter  << Final Command to Close Shutter at 
end of session

Code Review

Review showed

Bug found in 'Monitor Shutter Opening' whereby on the second cycle through the loop a wrongly placed CloseShutter command was caused the shutter to quickly begining closing. (close shutter should have been inside a 'If ShutterSecs > 90 ... endif block but lay outside it).

This bug explains why the shutter begins to Close.

Code in the 'State is Equilibrating' commands an Open Shutter if the Shutter is Closed or Closing, DomeCanOpen (ie conditions are safe) and DomeOpening is being monitored. So this explains why the shutter begins to Open again.

But what causes the 'Dome Is Unexpectedly Closing' Alerts if the Shutter is Closing due to calls to CloseShutter routine which sets the MonitorShutterClosure flag to true, which is used by 'Check for Unexpected Closure' code to ignore programmed closure.

It seem that in the OpenShutter routine, the MonitorShutterClosure flag is set to false. (as a prelude to commanding the Shutter to Open, and setting MonitorShutterOpening flag to true).

Actions :

Examine AstroMain/Obs.Manager code to locate the sections of code with bug or otherwise interfere and resolve.
Fixed Bug in AstroMain/Obs.Manager. (>> Monitor for similar issues howvere as there remains a potential issue that may have been caused by the original code modification where dome could also go into a cycle of opening/closing but with a different period and different cause)


	Session Aims & Highlights
	- Observing Result - Session Event Log - Night Summary Plot

	Operational Issues
	- Critical Issues (0), Major Issues (1), Minor Issues (2), Continuous Improvement (4)
	Investigation - Shutter unexpected closed during initial dome opening

	Images from 2019-02-17 >>

This Web Page:	Notes - Session 664 (2019-02-17)
Last Updated :	2023-11-26
Site Owner :	David Richards
Home Page :	David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 664 (2019-02-17)

Session Aims & Highlights (2019-02-17)

Operational Issues (2019-02-17, S664)

Investigation - Shutter unexpected closed during initial dome opening

David's Astronomy Pages Notes - Session 664 (2019-02-17)

Session Aims & Highlights (2019-02-17)

Operational Issues (2019-02-17, S664)

Investigation - Shutter unexpected closed during initial dome opening

David's Astronomy Pages
Notes - Session 664 (2019-02-17)