The session was scheduled to acquire 22 targets between 23:09 and 03:00, however only 6 targets were acquired. After acquiring the first four targets, the session was suspended at 23:36 (due to cloud) the session resumed at 00:32 and two further CI Cyg and SS Cyg were acquired. At 00:53 there was a critical cloud alert. Instead of just aborting the next target (CY Aqr) and putting the session  into a suspended state to wait until conditions improved (as it did do at 01:42), the Obs Manager quickly aborted each of the remaining 9 targets (UGC 118 to C/2018 W2 (Africano)) . Then finding itself at the end of the plan, the session began closing at 00:54 (with shutter closing at 00:55 and telescope parking at 00:56).

Observing Plan (2019-08-03, S679)

Session Event Log (2019-08-03, S679)

Time   Event Detail 23:45     Target Started (5/22) Target started (5/22, GCVS SS Her) 23:45 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 23:45     Target Aborted (5/22) Target aborted (5/22, GCVS SS Her) due to Critical Conditions 23:46 Session Suspended Session suspended 23:46 Shutter Closed Shutter closed (closing time 49s) 00:31 Session Resuming Session resuming 00:32 Shutter Opened Shutter opened (opening time 45s) 00:32 Queue Resumed Observing Queue resumed 00:32 Queue Resumed Observing Queue resumed 00:36     Target Started (11/22) Target started (11/22, GCVS CI Cyg) 00:43     Target Completed Target completed (11/22, GCVS CI Cyg) 00:38   Focusing Started (Foc1) Foc1 Focusing Started (TCF-S) 00:40   Focusing Completed Foc1 AutoFocus Completed (Profile No 4) 00:44     Target Started (12/22) Target started (12/22, GCVS SS Cyg) 00:52     Target Completed Target completed (12/22, GCVS SS Cyg) 00:53 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:53     Target Aborted (13/22) Target aborted (13/22, GCVS CY AQR) due to Critical Conditions 00:53 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:53     Target Aborted (14/22) Target aborted (14/22, UGC 118) due to Critical Conditions 00:53 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (15/22) Target aborted (15/22, GCVS RX And) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (16/22) Target aborted (16/22, GCVS BL Lac) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (17/22) Target aborted (17/22, IC 5217) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (18/22) Target aborted (18/22, GCVS EV Lac) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (19/22) Target aborted (19/22, GCVS Z And) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (20/22) Target aborted (20/22, WDS GRB 34) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (21/22) Target aborted (21/22, GSC 1224:29) due to Critical Conditions 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:54     Target Aborted (22/22) Target aborted (22/22, C/2018 W2 (Africano)) due to Critical Conditions 00:54 Queue Completed Job Queue completed 00:54 Session Closing Session closing 00:54 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter 00:55 Shutter Closed Shutter closed (closing time 40s) 00:55 Dome Parked Dome parked (parking time 33s) 00:56 Telescope Parked Telescope parked (parking time 42s) 03:40 Services Stopped Observatory Services stopped

Why did suspend operations work for the first cloudy sky event  at 23:45,  but not for the second cloudy sky event at 00:53 ?

One possible difference is that in the first case operations were already underway on the SS Her target when the critical cloud alert was raised, whilst in second case operations where on hold whilst waiting for the target to start. 

First Case (23:45)

GCVS SS Her  (5/22)                     2019-08-03 23:44  (Local)
  Target:                     Ok        GCVS SS Her       (ID: 172)

  Planned Start..             Ok        2019-08-03 23:46  (Scheduled)
  Planned End..               Ok        2019-08-03 23:54  (Scheduled) Duration : 8.1 mins

  Details                               
  Constellation:              Ok        Hercules          (Her)
  Observation Type:           Ok        Monitor
  Filters:                    Ok        VR (nFrames 3)
  Comment:                    Ok        Her SS

  Start                                 
  Actual Start                Ok        2019-08-03 23:44  (Local)

  Target Checks                         
  Checking Target ...         Ok        Sidereal Target   (GCVS SS Her)
  
  Slewing                               
  Slewing ...                 Ok        

  Centering                             
  Selecting Pointing Mode ... Ok        Higher Precision Pointing (up to 4 locate images)

  Centering 1                           
  Taking 10s image (C)...     Ok        [00679037] 22:45:41 (UT)  Full Locate C

  Centering 2                           
  Taking 20s image (C)...     

Observatory Alert                       2019-08-03 23:45  (Local)
Critical Cloud Conditions:    Alert     Observatory Manager will close the Dome Shutter
Reason To Suspend             Info      FailureState.0
Failed    ()

  Target Aborting ...         OK        Target aborting due to Critical Conditions

HIP 87937  (6/22)                       2019-08-03 23:45  (Local)
  Target:                     Ok        HIP 87937         (ID: 551)

  Planned Start..             Ok        2019-08-03 23:54  (Scheduled)
  Planned End..               Ok        2019-08-04 00:02  (Scheduled) Duration : 7.9 mins
  Waiting to start...         Hold      Wait for 3.8 mins

Session Suspended                       2019-08-03 23:46  (Local)
  Session: 2019-08-03         Info      Session suspended
  Close Shutter...            Closing   
  Close Shutter...            Ok        Shutter is Closed (49s)

  Continue..                  Ok        2019-08-03 23:54  (Local)

  Session Status:             Fail      Session is suspended due to cloud
  Target Result:              Fail      Failed (cloud)

 

Second Case (00:53)

GCVS CY AQR  (13/22)                    2019-08-04 00:52  (Local)
  Target:                     Ok        GCVS CY AQR       (ID: 614)

  Planned Start..             Ok        2019-08-04 01:03  (Scheduled)
  Planned End..               Ok        2019-08-04 01:17  (Scheduled) Duration : 13.6 mins
  Waiting to start...         Hold      Wait for 6.0 mins

Observatory Alert                       2019-08-04 00:53  (Local)
Critical Cloud Conditions:    Alert     Observatory Manager will close the Dome Shutter
Reason To Suspend             Info      FailureState.0

  Continue..                  Ok        2019-08-04 00:53  (Local)

UGC 118  (14/22)                        2019-08-04 00:53  (Local)
  Target:                     Ok        UGC 118           (ID: 538)

  Planned Start..             Ok        2019-08-04 01:17  (Scheduled)
  Planned End..               Ok        2019-08-04 01:38  (Scheduled) Duration : 21.9 mins
  Waiting to start...         Hold      Wait for 18.2 mins

Observatory Alert                       2019-08-04 00:53  (Local)
Critical Cloud Conditions:    Alert     Observatory Manager will close the Dome Shutter
Reason To Suspend             Info      FailureState.0

  Continue..                  Ok        2019-08-04 00:53  (Local)

GCVS RX And  (15/22)                    2019-08-04 00:53  (Local)
  Target:                     Ok        GCVS RX And       (ID: 20)

  Planned Start..             Ok        2019-08-04 01:38  (Scheduled)
  Planned End..               Ok        2019-08-04 01:46  (Scheduled) Duration : 7.5 mins
  Waiting to start...         Hold      Wait for 40.0 mins

Observatory Alert                       2019-08-04 00:53  (Local)

Critical Cloud Conditions:    Alert     Observatory Manager will close the Dome Shutter
Reason To Suspend             Info      FailureState.0

  Continue..                  Ok        2019-08-04 00:53  (Local)
  Current AllSky Meas...      Ok        Clear             (23)

It would seem there is a bug associated with forcing a suspension where the active target has not yet started.  More usually the active target has actually started when the critical cloud alert comes.    On this particular case the shutter was closed when the execution manager had aborted targets through to the end of the schedule,  but because there is a potential failure to close the shutter at earliest opportunity the issue is judged to be a critical one.

Actions:

>>  Perform a code review, examining Adv_Main code to identify the origin and location of the bug
>>  Fix bug .

Code Review

Code review shows that   'jobAbortTarget' is being repeatedly set to True by the critical event, which causes the TakeFrameJobsQSO to immediately exit out of the Target Wait Loop, to not proceed with Target Acquisition (Proceed=False) and move to the next target, which also then exits and so on until the end of the target list is quickly reached.

jobAbortTarget' is set to True by calls to 'AbortTargetExplicit' routine. This is called by Obs Manager in response to Critical Alert when session state is 'running'

     ' Initiate Hard Suspend (Abort Current Target without waiting for it to finish)
    ' -----------------------------------------------------------------------------
    bHardSuspend = True
    HardSuspendStartTime = Now()

    ReasonToAbort = AbortReason.CriticalConditions

    objConsole.AbortTargetExplicit()

However it is only appropriate to abort the target if it has been started (i.e. telescope and/or imaging operations are underway).    

Code Fix

- A new global variable 'TargetInHold'  is used to signify when a target is held in the loop waiting for its start time or conversely when a target has been started.

- The current state of 'TargetInHold' is used to decide when to abort the target or not when an Critical Alert occurs.

 ' Initiate Hard Suspend (abort Current Target without waiting for it to finish), unless it is 'in hold'
' ------------------------------------------------------------------------------------------------------------------------------
If TargetInHold = False Then

   bHardSuspend = True
   HardSuspendStartTime = Now()

   ' ReasonToAbort = AbortReason.CriticalConditions ' Not needed here as it is set by UpdateReasonToSuspend() above

   objConsole.AbortTargetExplicit()

Else
   ' place session is suspended mode
   ' -------------------------------
   SetState_Suspended()
End If

Large differences were noted between the tube temperature and outdoor temperature during the session. Tube was 3 degC higher at start of session, increasing to 4 degC higher by end of session.   This warrants a small investigation as it may be a factor in poor CCD cooling performance observed in this and previous session.

Dew Heater was intentionally left on at end of last session (S578) due to high humidity / low lying fog. and set to Auto during session itself (c. ~90%). 
Observatory temperatures were also noticeably higher than outdoor temperature.

Temperatures - This Session (2019-08-03, S679)

Temperatures - Last Session (2019-08-02, S678)

Temperatures - Earlier Session (2019-03-04, S671)

Windows 10 version on the Development Computer (Laptop10) was upgraded from 1803 to 1903 on 2018-08-05 as support for the 1803 version of windows 10 is to end shortly.

The upgrade appeared to proceed ok, but upon attempting to open CCDSoft a number of system errors occurred because a certain .dll files could not be found, including
APOGEE32.dll, CFWAPI.dll, DSS_DLL.dll, ipl.dll etc.

The required .dll files were copied to the C:\Program Files (x86)\Common Files\System folder from the equivalent folder on the Observatory Computer (Laptop9), until CCDSoft was able to be successfully started.

It is recalled that a similar problem occurred before after installing Windows10 on AllSky Computer (Laptop7)

Many other Common DLL files on Laptop 9 (Win7) are also not present on Laptop10 including TheSky6 dlls. 

TeleApi

Sometime later (2019-09-07) an attempt was made to connect and control a simulated telescope in POTH / TheSky6.  This was the first time that this had been attempted  for several months, and an error message was generated saying that a required driver was not available. It was immediatly suspected that there was a problem with TeleApi and its properties could not be open.

The TeleApi setup (TeleAPI(5.0.4)Setup.exe) was rerun and this fixed the problem. It's believed the necessary driver file(s) went missing at the time of the Windows 10 upgrade.

UPB Software on Observatory and Development Computer was updated to the latest version (1.4.15, Jul/19) today, 2019-08-05.

The 1.4.15 update still doesn't include the fix to allow the UPB Software to work with the ASCOM UPB Switch Drivers. This is now promised at end August..

[ Pegasus Astro Support Page:  https://pegasusastro.com/support/ ]