David's Astronomy Pages Notes - Session 788 (2020-05-31)

Notes (S787)

Notes Main

Home Page

Notes (S789)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (2),  Minor Issues (7),  Continuous Improvement (6)
	POTH.Dome / Pulsar Driver Glitch (23:43 to 00:06)
	Images from 2020-05-31 >>

Session Aims & Highlights (2020-05-31)

Main aims

1. Software Testing: Test 3.24.3 version of AstroMain which has updated routines for MustClose, ConditionsDeteriorating and CanResume following issue in previous session, where Protective Relay forced dome closure before AstroMain acted.
2. C/2020 F8 (SWAN): Image this comet using multiple 10s exposures in an attempt to overcome the bright NE sky associated with the developing dawn.
3. Targets: Acquire images of a number of targets if conditions allowed.  

Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• CCDApp2, TheSky6, CCDSoft5, PHD2,  POTH Hub
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.8
   
• AstroMain 3.24.3,  AstroGuard 1.3,   AstroPlan 1.15.1,  AstroShCap 1.2.22, AstroVOE 1.3.6, UPB 1.5 
  (AstroWeather 1.10, AstroAllSky 3.5.1)
  

Highlights

•   Sky Condition Alerts
  23:46:31 Sky Conditions Alert | Sky conditions have deteriorated (Clarity 23.8, Stars 0), Soft Suspend is called (during Near Zenith focusing)
  00:18:09 Sky Conditions Alert | Sky conditions have deteriorated (Clarity 27.0, Stars 5), Soft Suspend is called (during GRB 200529A)
  Sky conditions have improved (Clarity 26.9, Stars 6)
  00:19:09 Sky Conditions Alert | Sky conditions have deteriorated (Clarity 25.5, Stars 3), Soft Suspend is called (during GRB 200529A)
  
  Sky conditions have improved (Clarity 28.5, Stars 5)
  00:27:30 Sky Conditions Alert | Sky conditions have deteriorated (Clarity 27.8, Stars 3), Soft Suspend is called (during GRB 200529A)
  Sky conditions have improved (Clarity 29.4, Stars 3)
  
  00:49:18 Sky Conditions Alert | Sky conditions have deteriorated (Clarity 24.7, Stars 3), Soft Suspend is called (IC 1222)

Lowlights

• C/2020 F8 (SWAN).  Failed to acquire images due to cloud / fog.
• Dome Glitch. Glitch with POTH.Dome or Pulsar Driver caused 20 min outage and necessiated visit to Observatory and software/session restart

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Pre-Session Weather Forecast (API data from darksky.net) Pre-Session Weather forecast showed variable skies range from 5 to 50% cloud cover,  light winds (wind 7mph) and a minimum temperature of 6°C. In the event conditions were clear and calm through the whole night, with a minimum temperature of around 4°C

Session Event Log

Time     Event Detail 23:27:38 Session Created Live Session Created (2020-05-31 S00788, ImageSaveNum: 788001) 23:27:56   Obs.Manager Started Obs.Manager started 23:27:58   Obs.Overseer Started Obs.Overseer started 23:28:02   Services Started Observatory Services started 23:28:14   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 23:30:04   Services Started Observatory Services started 23:30:36 Observatory (Auto) Observatory placed in Fully-Automated Mode 23:30:38 Session Pending Session pending (2020-05-31) 23:30:40 Session Initiating Session initiating (2020-05-31) 23:30:42   Plan Requested Observing Plan requested from AstroPlan (1.15.2) 23:31:08   Plan Loaded Observing Plan loaded to queue (2020-05-31 V1 BEST PLAN, Plan ID: 517) 23:31:13   Camera1 Connected SBIG Camera Connected (set point -15°C) 23:31:21   Telescope Connected Telescope Connected (TheSky6) 23:31:34 Session Equilibration Session ready for dome & camera equilibration 23:32:19   Shutter Opened Shutter opened (opening time 46s) 23:32:22   Equilibration Started Dome Equilibration started 23:43:15   Equilibration Ended Dome Equilibration ended (10.9 mins total) 23:43:17 Session Running Session running 23:43:19   Queue Started Observing Queue started (11 targets selected) 23:44:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:46:31   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 23:46:33 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter) 23:46:35   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud) 23:47:20       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 23:47:22     Target Aborted (1/11) Target aborted (1/11, C/2019 Y1 (ATLAS)) due to Unknown Reason 23:47:24   Queue Paused Queue paused for SoftSuspend 23:47:26 Session Suspended Session suspended 23:47:28   Queue Paused Queue pausing for HardSuspend 23:54:31 Session Resuming Session resuming 23:56:07   Shutter Opening Shutter opening hasn't finished after 90s 23:56:12 Session Suspended Session suspended 23:58:16 Session Resuming Session resuming 23:59:46   Shutter Opening Shutter opening hasn't finished after 90s 00:07:16 Session Resumed Live Session Resumed (2020-05-31 S00788, ImageSaveNum: 788001) 00:07:34   Obs.Manager Started Obs.Manager started 00:07:36   Obs.Overseer Started Obs.Overseer started 00:07:40   Services Started Observatory Services started 00:08:04   Scope Switched Off Telescope Power has been switched off via UPB Powerbox. 00:08:24   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 00:09:56   Services Started Observatory Services started 00:10:24 Observatory (Auto) Observatory placed in Fully-Automated Mode 00:10:26 Session Pending Session pending (2020-05-31) 00:10:28 Session Initiating Session initiating (2020-05-31) 00:10:30   Plan Loaded Observing Plan loaded to queue (2020-05-31 V1 ACTIVE PLAN S788, Plan ID: 517) 00:10:33   Camera1 Connected SBIG Camera Connected (set point -15°C) 00:10:39   Telescope Connected Telescope Connected (TheSky6) 00:10:56 Session Equilibration Session ready for dome & camera equilibration 00:11:42   Shutter Opened Shutter opened (opening time 46s) 00:11:45   Equilibration Started Dome Equilibration started 00:11:47   Equilibration Ended Dome Equilibration ended (0.0 mins total) 00:11:49 Session Running Session running 00:11:51   Queue Started Observing Queue started (11 targets selected) 00:11:53     Target Missed (1/11) Target's time slot was missed (1/11, C/2019 Y1 (ATLAS)) 00:11:55     Target Started (2/11) Target started (2/11, GRB 200529A) 00:18:09   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 00:18:29   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 00:19:09   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 00:27:00   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 00:27:30   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 00:27:35   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 00:30:03     Target Completed Target partially completed (2/11, GRB 200529A) 00:30:05     Target Started (3/12) Target started (3/12, IC 1222, ToO) 00:49:18   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 00:49:37 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter) 00:49:39   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud) 00:49:43     Target Aborted (3/12) Target aborted (3/12, IC 1222, ToO) due to cloud 00:49:45   Queue Paused Queue paused for SoftSuspend 00:49:48 Session Suspended Session suspended 00:49:50   Queue Paused Queue pausing for HardSuspend 00:50:33   Shutter Closed Shutter closed (closing time 44s) 00:50:35     Target Missed (4/12) Target's time slot was missed (4/12, GCVS U Her) 00:50:37     Target Missed (5/12) Target's time slot was missed (5/12, GCVS SS Her) 00:56:01     Target Cancelled Target cancelled (6/12, HIP 87937) due to cloud 01:05:21     Target Cancelled Target cancelled (7/12, GCVS CY Lyr) due to cloud 01:14:47     Target Cancelled Target cancelled (8/12, GCVS R CrB) due to cloud 01:24:58     Target Cancelled Target cancelled (9/12, GCVS AM Her) due to cloud 01:38:33     Target Cancelled Target cancelled (10/12, GCVS S Cyg) due to cloud 01:59:54     Target Cancelled Target cancelled (11/12, NGC 582) due to cloud 02:29:56     Target Cancelled Target cancelled (12/12, C/2020 F8 (SWAN)) due to cloud 02:30:01   Queue Completed Job Queue completed 02:30:06 Session Closing Session closing 02:30:12   Shutter Closed Shutter already closed (closed at 00:50, closing time 44s) 02:32:03   Dome Parked Dome parked (parking time 111s) 02:33:07   Telescope Parked Telescope parked (parking time 60s) 02:33:37   Scope Switched Off Telescope Power has been switched off via UPB Powerbox. 02:34:23   Services Stopped Night Services stopped 02:34:25 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files) 02:34:26 Session Finished Session Finished

Session Alerts

Time     Alert Detail -- No Alerts                --

Back to Top

Operational Issues (2020-05-31, S788)

• Critical Issues (0)
• Major Issues (2)
• Minor Issues (7)
• Continuous Improvement (6)

[ Prev | Next ]

Critical Issues

• None

Major Issues

• POTH.Hub apparently stuck and not supplying valid data to Ascom Clients (AstroMain, AstroGuard). Dome apparently closing and opening for 90s+ without completing operation.  Problem investigated (see POTH.Dome / Pulsar Driver Glitch (23:43 to 00:06) ). Unclear what initiated the glitch but it throughly confused AstroMain's Obs.Manager (e.g  log message at 23:57:59.29 | Close Shutter - Call to Close Shutter, but shutter is already closing/closed.)    It did however respond by calling MakeObservatorySafe() routine, and although it couldn't see a successful resut to this action and the program UI eventually froze, the Observatory Dome was in fact successfully closed and parked as per design for this situation as proven by a physical visit to the Observatory.  May worth seeing if there were any similarities with case from a year ago, reported to Pulsar at the time.  Issue closed after observatory became fully switch over to using DeviceHub 6.5.1 (2020-12-30).  POTH no longer used,
  Closed 2020-12-30  (DeviceHub)
   
• AstroMain program froze at 23:59.  AstroMain UI and Obs.Worker froze at 23:59. Problem may possibly be related to ongoing problems with POTH.Dome /Pulsar Driver than began at 23:43 and saw the Shutter apparently stuck on 'Closing'.   Last IamAlive message was from ExecuteQueue/QueuePaused_ForHardSuspend/ at 23:47.   Issue closed on 2021-01-04 following a review that showed that the AstroMain was essentially stable (following a series of improvements and fixes), with an absence of any unexplained program hangs or crashes during the 35 sessions between  2020-09-18 & 2021-01-03.
  Closed 2021-01-04 (AstroMain)

Minor Issues

• Amount of Time recorded as 'Slewing' seems excessive. Looking back at breakdown in Active Time, the amount allocated to Slewing always seems excessive.   This has been noticed before.  Examining Session S760 (2020-03-08) where the total active time was 487 mins the amount of slewing time was 215 mins (44.2%) - more than the 163.3 mins spent imaging (33.5%) , 29.6 mins spent centering (6.1%) and 79.0 mins spent on focusing (16.2%). Of the total imaging time (including downloads etc) the total exposure time from FITS summary was 144mins (of which 15 mins was spent on Locate Frames and 128.9 mins on Image Frames).  
  During the session the scope was slewed to 40 targets, which implies the average slewing time per target was 5.37 mins. Again this seems very excessive. The Timings File indicates that the total slewing time as 29.6 mins, giving a much more reasonable 0.74 minutes per target. The Timings file also calls into question the other elements onf the Active Time breakdown, indicating 457.3 min Target Time (vs 487 active time),  185.3 mins imaging (vs 163.3), 109.3 mins focusing (vs 79.0), 75.4 mins centering (vs 29.6) , 177.2 mins waiting (vs 124.7 inactive).   Bug identified and fixed.
  Fixed 2020-06-08 (AstroMain 3.25.1)
   
• Session's Observing Plan appears to be have been created using Must-Include targets from the previous session rather than the Must-Include targets that were set up shortly before creating the plan.   2020-05-31 V1 Plan contained  GRB 200529A and  C/2020 F8 (SWAN) which were the two Must-Include targets from S787 session. Whilst C/2020 F8 (SWAN) was also a Must-Include target in the new list, 3 other  targets in the list  (including IC 1222, NGC 3243) weren't included. Is it a coincidence and the reason that GRB 200529A was included is because it had a Priority 1 rank and still marked as a ToO target ?.  Issue deemed to have been resolved or no longer significant in review made on 2021-06-04.
  Closed 2021-06-04  (Session / AstroAllSky)
   
• After downloading Targets Database to Analysis Computer it wasn't possible to see more than 1 target in the Session's V1 Executed Plan in AstroPlan. After restarting AstroPlan all 11 targets in the plan became visible. This behaviour hasn't been seen before.  Need to review code and understand if a button can be added to 'reset/fix' issue without having to restart AstroPlan.
  To investigate and provide workaround or fix (AstroAllSky)
   
• It wasn't possible to Insert IC1222 as a 'Do Immediately ToO target' as hoped.   After GRB 200529A was included in the nights' observing plan and IC 1222 wasn't it was decided to insert IC 1222 (with SN) instead of GRB 200529A which had already been imaged  in last two sessions. Attempting to insert IC 1222 as a Do Immediately ToO produced the AstroMain message  'Target (IC 1222, priority 1) inserted in job queue' followed by a second message 'Target (IC 1222) is deferred due to a current ToO'.     Problem was that whilst GRB 200529A was originally  created as a Priority 1 Rapid Response ToO, its special status was no longer appropriate when doing follow-up observations.   User needs to toggle-off  the target's 'Target of Opportunity' checkbox.
  Closed 2020-06-01 (AstroMain)
   
• When inserting a ToO the first message unexplainable says 'Target ... has timed-out' before putting a second message saying target has been inserted.
  Fixed 2020-06-01 (AstroMain 3.24.4)
   
• Log file element 'QueuePaused_ForHardSuspend' is too long and overruns allowed space and needs to be shortened.
  Fixed 2020-06-01 (AstroMain 3.24.4)
   
• Current Session Web Page display issue (page constantly flashing) when viewed on IPad/Safari. Problem is seen once in every 8-10 sessions, but it has proven impossible to identify the cause of the problem so far..  Problem doesn't occur with most sessions, and never happens when viewed on Windows.  Next time problem occurs a snapshot of Live Files should be taken, and later used to test the impact of adding each file back into the website in turn hoping that the problem file/instance can be identified.
  Fixed 2020-09-12 (AstroMain 3.28.7)

Continuous Improvement

• When Conditons Improve it would be useful to report  Clarity and Stars.
  Checking Conditions... Ok    Conditions have improved, and allow the session to be resumed
  Observatory Manager   Info   Sky conditions have improved (Clarity 26.9, Stars 6)
  Report number in similar way to that done when condition deteriorate
  Clarity should be shown with one decimal place , use FormatNumber() rather than cstr().  Would be useful to have reportline saying when Soft Suspend is cancelled with time of cancellation e.g. (00:20:09)
  Done 2020-06-01 (AstroMain 3.24.4)
   
• Ensure HardSuspends are clearly reported. Following  a Soft Suspend at 23:46:31 Wwhen Sky conditions deteriorated (Clarity 23.8, Stars 0) there was a report message 'Critical Cloud Conditions  Alert   Observatory Manager will close the Dome Shutter' but it wasn't clear that this was a Hard Suspend. Code has been modified  so that HardSuspend is now explicitly reported.   ExplicitTarget Aborts are also reported.   When Take Image exposure time does
  Done 2020-06-01 (AstroMain 3.24.4)
   
• When 'Take Image' actual exposure times don't match then intended exposure time, the images are marked 'Fail' .   The details are suffixed like '- aborted (59.21s) '
  Done 2020-06-01 (AstroMain 3.24.4)
   
• Add a 'Protective Relay' column to NightSummary6 plot using relay flags data on the recently extended Cloud Sensor data record'. Column would be thin line when Relay is Unsafe and a thick column when Relay is Safe.  Column would be blank when there is no current data (cloud sensor record missing or out of date)
  Done 2020-06-01 (AstroAllSky 3.6)
   
• Open Dome shouldn't be allowed in Automated Mode when Shutter Battery Level is < 80%
  Done 2020-06-02 (AstroMain 3.24.5)
   
• Extend ObsEnv to monitor Pulsar's ASCOM Trace Log for Volatile Exceptions that could point to a problem with reading reliable data from POTH.Dome
  RunPulsarDomeMonitor has been extended to capture number of Volatile Exceptions  (Public nVolatileExceptions as integer) and the Time/Date of Last Volatile exception (LastVolatileExceptionTime As Date).  Code to use this information has still to be written.
  Done 2020-06-01 (AstroMain 3.24.4)

[ Prev | Next ]

Back to Top

POTH.Dome / Pulsar Driver Glitch (23:43 to 00:06)

AstroMain Report Logs

23:27 POTH.Dome Started (COM9)
23:31 Session Equilibration - Shutter Opened
23:32 Dome moved to Az 198 (into Wind)
23:39 Dome moved to Az 171 (into Wind)
23:43 Session Running
23:43 Dome Slaving set to On
23:46 Sky Conditions Deteriorating (Clarity 23.8, Stars 0). SoftSuspend called
23:47 Queue Suspended for SoftSuspend
23:47 Session Suspended
23:47 Closing Shutter  (no completion ?)
23:54 Conditions Improved. Session resuming
23:54  Opening Shutter (no completion ?) - information message saying that shutter opening has finished after 90s
23:55  Safety Message given saying that Shutter will be closed, but information message saying that shutter hasn't closed after 100s
23:57  Message from Obs.Overseerer saying that Job Queue appears to be frozen
23:57  'MakeObservatorySafe' routine running (turn off scope tracking, slew dome to Park Position, close shutter,
23:58   Condition Improved. Session resuming (attempt to open shutter, whilst MakeObservatorySafe is trying to close shutter ?)
23:59  Session Suspended
23:59  Dome slew aborted after 180s (Dome at Az 171.0 deg)

Physical Check
00:03  Observatory physically checked. The dome was found to be closed and at its Park position (90 deg Az)

AstroMain ObsPics
Obs Pics show Dome moving into Wind at 23:32 and 23:39 but don't show Dome slaved to Scope at 23:43.  ObsPics show show Shutter in 'Closing' state from 23:47:40 to 23:54.29, at which point the shutter was in an opening state ntil 23:56:33 when it started to close again, but then started to open again at 23:58:40. When AstroMain restarted the ObsPics show the Shutter Closed and Dome at Park Position.

AstroGuard
AstroGuard records the Dome closing at 23:47 but taking >68s to close.   At 23:54 the Dome is recorded as Opening, but taking >68s to open. At 23:56 the dome is recorded as Closing, but taking >68s to close. At 23:58 dome is recorded as Opening  but taking >67s to open. Then at 23:59 dome is recorded as taking >68s to close.

Ascom Trace Logs
ASCOM Trace Log for Dome shows multiple  Command<> and Volatile Exceptions  beginning at 23:43:22 through to 00:06:04. These started around the time the session began running, way before the point where problem was percieved . Messages were :

Command<> Exception
System.FormatException    Input string was not in a correct format.[0D][0A] at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)[0D][0A] at System.Number.ParseUInt32(String value, NumberStyles options, NumberFormatInfo numfmt)[0D][0A] at System.UInt16.Parse(String s, NumberStyles style, NumberFormatInfo info)[0D][0A] at System.Convert.ToUInt16(String value, IFormatProvider provider)[0D][0A] at ASCOM.Pulsar_Observatories_Dome.Dome.Command[T](String command, Boolean raw)

Volatile Exception
System.FormatException: Input string was not in a correct format.[0D][0A] at ASCOM.Pulsar_Observatories_Dome.Dome.Command[T](String command, Boolean raw)[0D][0A] at ASCOM.Pulsar_Observatories_Dome.Dome.CommandBool(String command, Boolean raw)[0D][0A] at ASCOM.Pulsar_Observatories_Dome.Dome.UpdateAtHome()[0D][0A] at ASCOM.Pulsar_Observatories_Dome.Dome.Volatile()

After the exceptions appear there are no more Volatile records containing Shutter Health information.   The logs record shutter closing at 23:47:21,   opening at 23:54:31, closing at 23:56:07, opening at 23:58:16, closing at 23:59:47, but in none of these cases does the operation finish (status is always either 2 for Opening or 3 for Closing). In case the period is greater than the normal 50s or so for opening or closing the shutter.    For some of this time the number of ShutterStatus messages sent to Trace Log was truely excessive reaching some 43 lines every 0.015s .  During this time Slewing Get, Azimuth Get, AtPark Get are all present as normal.

After restarting the Session at 00:07 the problems with Command/Volatile exceptions has gone and the ASCOM trace log is normal.

ObsEnvData
ObsEnv Data File which includes the results of searching ASCOM log file for Battery Health Info shows vcNull values for Batery Voltage, Battery Voltage, Currrent and Temperature between 23:43:46 and 00:05:33 / 00:07:40.

Conclusion
Data seems to be consistent in showing that POTH.Hub was apparently stuck and not supplying valid Dome data to Ascom Clients (AstroMain, AstroGuard). Dome apparently closing and opening for 90s+ without completing operation.  Unclear what initiated the glitch but it throughly confused AstroMain's Obs.Manager.   It did however respond by calling MakeObservatorySafe() routine, and although it couldn't see a successful resut to this action and the program UI eventually froze, the Observatory Dome was in fact successfully closed and parked as per design for this situation as proven by a physical visit to the Observatory. 

May be worth seeing if there were any similarities between this case and a case from a year ago (reported to Pulsar Observatories at the time 2019-12-10).

May need to consider whether further action should be taken by AstroMain's Obs.Manager, for example preventing further attempt to resume/reopen shutter after initaiting MakeObservatorySafe routine.

Back to Top