David's Astronomy Pages Notes - Session 832 (2020-11-22)

Notes (S831)

Notes Main

Home Page

Notes (S833)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (1),  Minor Issues (4),  Continuous Improvement (1)
	Images from 2020-11-22 >>

Session Aims & Highlights (2020-11-22)

Main aims -

1. Targets.  Acquire images of a selection of deep sky targets

Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• CCDApp2, TheSky6, CCDSoft5, PHD2,  DeviceHub 4.5.0.1, UPB 1.5 
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.9
   
• AstroMain 3.32.7,  AstroGuard 1.4.6,   AstroPlan 1.17.5  AstroShCap 1.3.6, AstroVOE 1.5, AstroProtect 1.4.1
  (AstroWeather 1.10.13, AstroAllSky 3.6)

Highlights

• Targets.  x targets were observed.
• Safety Protection.  After recognising that job queue had becoming frozen Obs.Management protection operated to make the observatory safe (closing the shutter, parking the dome and turning off telescope tracking).
•   Frozen Job Queue. Job queue froze at 03:19 and prevented further observations (1.5 hours of clear sky observing time were lost)

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Pre-Session Weather Forecast (API data from darksky.net) Pre-Session Weather forecast showed a clear spell in middle of evening (cloud cover 0-23%) but clouding at 22:00 to 02:00 (cloud cover 54-78%) , becoming totally overcast from 02:00 onwards.  Minimum temperature 6 degC. Wind becoming strong onwards dawn with potential rain (heavy rain forecast from 08:00 onwards) In the event there was a clear spell from 18:00 to 20:10 but transparency and seeing were fairly poor, and sky was bright due to moonlight and cloud reflection. Cloud thickened at 21:45 and session suspended.   User intervention made to finish session at 23:30 due to  cloud and rain risk.

Session Event Log

Time     Event Detail 19:07:22 Session Monitoring AutoStart monitoring for Live Session opportunity starting between 19:07 & 06:03 19:07:25 Session AutoStarting Session autostarting (19:07) 19:08:04 Session Created Live Session Created (2020-11-22 S00832, ImageSaveNum: 832001) 19:08:07   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 19:09:56   Services Started Observatory Services started 19:10:05 Observatory (Auto) Observatory placed in Fully-Automated Mode 19:10:07 Session Pending Session pending (2020-11-22) 19:10:09 Session Initiating Session initiating (2020-11-22) 19:10:11   Plan Requested Observing Plan requested from AstroPlan (1.17.5) 19:10:51   Plan Loaded Observing Plan loaded to queue (2020-11-22 V1 BEST PLAN, Plan ID: 567) 19:11:03   Camera1 Connected SBIG Camera Connected (set point -20°C) 19:11:09   Telescope Connected Telescope Connected (TheSky6) 19:11:19 Session Equilibration Session ready for dome & camera equilibration 19:12:51   Dome Opened Dome opened (opening time 45s) 19:13:06 Session Running Session running 19:13:09   Queue Started Observing Queue started (38 targets selected) 19:14:22       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 19:17:47       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 19:20:24       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 19:20:26       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 19:22:40       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 19:24:27       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 19:24:44       Focusing Checked Foc2 Focusing Check Succeeded 19:25:05     Target Started (1/38) Target started (1/38, NGC 596) 19:28:03       Focusing Failed Foc1 Focusing Failed - star is too dim (TCF-S) 19:45:40     Target Completed Target completed (1/38, NGC 596) 19:47:27     Target Started (2/38) Target started (2/38, GCVS RX And) 19:50:25       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 19:53:15       Focusing Completed Foc1 AutoFocus Completed (Profile No 3) 19:55:55     Target Completed Target completed (2/38, GCVS RX And) 19:55:57     Target Started (3/38) Target started (3/38, C/2019 L3 (ATLAS)) 20:02:01       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 20:05:10       Focusing Completed Foc1 AutoFocus Completed (Profile No 4) 20:12:04     Target Completed Target completed (3/38, C/2019 L3 (ATLAS)) 20:12:06     Target Started (4/39) Target started (4/39, 156P/Russell-LINEAR, ToO) 20:14:54       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 20:17:56       Focusing Completed Foc1 AutoFocus Completed (Profile No 5) 20:25:38     Target Completed Target completed (4/39, 156P/Russell-LINEAR, ToO) 20:25:40     Target Missed (5/39) Target's time slot was missed (5/39, WDS GRB 34) 20:25:42     Target Started (6/39) Target started (6/39, GCVS U And) 20:35:33     Target Completed Target partially completed (6/39, GCVS U And) 20:35:35     Target Started (7/39) Target started (7/39, GCVS SS Cyg) 20:38:50       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 20:41:40       Focusing Completed Foc1 AutoFocus Completed (Profile No 6) 20:45:14     Target Completed Target completed (7/39, GCVS SS Cyg) 20:46:02     Target Started (8/39) Target started (8/39, UGC 1924) 20:49:01       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 20:51:51       Focusing Completed Foc1 AutoFocus Completed (Profile No 7) 21:07:45     Target Completed Target completed (8/39, UGC 1924) 21:07:48     Target Started (9/40) Target started (9/40, NGC 7635, ToO) 21:11:18       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:13:51       Focusing Completed Foc1 AutoFocus Completed (Profile No 8) 21:42:02   Dome Running test to see Dome Driver is stalled 21:42:21   Dome Dome Driver/Dome Slaving found to be ok. 21:42:31     Target Completed Target completed (9/40, NGC 7635, ToO) 21:42:33     Target Started (10/41) Target started (10/41, PGC 2121, ToO) 21:44:57       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:47:53       Focusing Completed Foc1 AutoFocus Completed (Profile No 9) 22:03:34     Target Completed Target completed (10/41, PGC 2121, ToO) 22:03:36     Target Started (11/42) Target started (11/42, NGC 246, ToO) 22:06:51       Focusing Failed Foc1 Focusing Failed - star is too dim (TCF-S) 22:23:05     Target Completed Target completed (11/42, NGC 246, ToO) 22:23:07     Target Missed (12/42) Target's time slot was missed (12/42, GCVS BL Lac) 22:23:09     Target Missed (13/42) Target's time slot was missed (13/42, GCVS HH And) 22:23:11     Target Missed (14/42) Target's time slot was missed (14/42, GCVS DY Per) 22:23:13     Target Missed (15/42) Target's time slot was missed (15/42, GCVS GK Per) 22:23:15     Target Missed (16/42) Target's time slot was missed (16/42, GSC 1224:29) 22:23:17     Target Missed (17/42) Target's time slot was missed (17/42, GCVS Z And) 22:23:19     Target Missed (18/42) Target's time slot was missed (18/42, GCVS TT Ari) 22:23:21     Target Started (19/42) Target started (19/42, UGCA 229) 22:44:22     Target Completed Target completed (19/42, UGCA 229) 22:44:24     Target Started (20/42) Target started (20/42, UGC 6930) 22:47:26       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 22:50:28       Focusing Completed Foc1 AutoFocus Completed (Profile No 10) 23:01:33     Target Completed Target completed (20/42, UGC 6930) 23:01:35     Target Started (21/42) Target started (21/42, NGC 514) 23:04:30       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:07:19       Focusing Completed Foc1 AutoFocus Completed (Profile No 11) 23:18:25     Target Completed Target completed (21/42, NGC 514) 23:18:27     Target Started (22/42) Target started (22/42, UGCA 47) 23:20:36       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:23:25       Focusing Completed Foc1 AutoFocus Completed (Profile No 12) 23:39:05     Target Completed Target completed (22/42, UGCA 47) 23:39:07     Target Started (23/42) Target started (23/42, NGC 1253) 23:41:01       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:44:13       Focusing Completed Foc1 AutoFocus Completed (Profile No 13) 00:00:11     Target Completed Target completed (23/42, NGC 1253) 00:00:13     Target Started (24/42) Target started (24/42, C/2020 M3 (Atlas)) 00:02:16       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:04:54       Focusing Completed Foc1 AutoFocus Completed (Profile No 14) 00:10:10     Target Completed Target completed (24/42, C/2020 M3 (Atlas)) 00:10:30     Target Started (25/42) Target started (25/42, IC 402) 00:12:48       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:15:51       Focusing Completed Foc1 AutoFocus Completed (Profile No 15) 00:31:37     Target Completed Target completed (25/42, IC 402) 00:32:51     Target Started (26/42) Target started (26/42, UGCA 104) 00:35:28       Focusing Failed Foc1 Focusing Failed - star is too dim (TCF-S) 00:52:47     Target Completed Target completed (26/42, UGCA 104) 00:55:13     Target Started (27/42) Target started (27/42, NGC 1843) 00:57:19       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:00:11       Focusing Completed Foc1 AutoFocus Completed (Profile No 16) 01:16:14     Target Completed Target completed (27/42, NGC 1843) 01:17:35     Target Started (28/42) Target started (28/42, GCVS CN Ori) 01:19:51       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:22:39       Focusing Completed Foc1 AutoFocus Completed (Profile No 17) 01:26:04     Target Completed Target completed (28/42, GCVS CN Ori) 01:26:47     Target Started (29/42) Target started (29/42, UGC 4171) 01:29:03       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:31:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 18) 01:47:35     Target Completed Target completed (29/42, UGC 4171) 01:49:08     Target Started (30/42) Target started (30/42, UGCA 206) 01:51:49       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:54:50       Focusing Completed Foc1 AutoFocus Completed (Profile No 19) 02:10:39     Target Completed Target completed (30/42, UGCA 206) 02:11:30     Target Started (31/42) Target started (31/42, UGC 4186) 02:13:27       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 02:16:28       Focusing Completed Foc1 AutoFocus Completed (Profile No 20) 02:24:16   Dome Running test to see Dome Driver is stalled 02:24:36   Dome Dome Driver/Dome Slaving found to be ok. 02:24:46     Target Completed Target completed (31/42, UGC 4186) 02:33:52     Target Started (32/42) Target started (32/42, UGC 4599) 02:36:04       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 02:39:05       Focusing Failed Foc1 Focusing Failed - failed QC check (Quadratic) 03:07:19     Target Completed Target completed (32/42, UGC 4599) 03:08:22     Target Started (33/42) Target started (33/42, GCVS U Gem) 03:10:27       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 03:13:29       Focusing Completed Foc1 AutoFocus Completed (Profile No 22) 03:16:16     Target Completed Target completed (33/42, GCVS U Gem) 03:17:55     Target Started (34/42) Target started (34/42, NGC 2782) 03:20:12 Job Queue Frozen Job Queue appears to have frozen at around 03:19 - 03:20 03:31:02   Dome Closed Dome closed (closing time 46s) 03:31:22   Dome Parked Dome moved to Az 90 deg (Park Position) 03:31:32 Job Queue Frozen Job Queue appears to have frozen at around 03:31 - 03:31 04:10:45   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 04:10:50   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 04:11:00   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 04:11:10   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 04:11:15   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 04:13:56 Job Queue Frozen Job Queue appears to have frozen at around 04:13 - 04:13 04:20:56   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 04:24:31 Job Queue Frozen Job Queue appears to have frozen at around 04:24 - 04:24        Multiple rows removed   08:26:54   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions 08:27:09   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 08:29:30 User Intervention User request to close and finish session 08:32:34 Session Closing Session closing 08:32:36   Dome Closed Dome already closed (closed at 03:31, closing time 46s) 08:32:38   Dome Parked Dome already parked (parked at 06:42, parking time 41s) 08:33:00   Telescope Parked Telescope parked (parking time 19s) 08:33:30   Telescope Switched Off Telescope Power has been switched off via UPB Powerbox. 08:34:19   Services Stopped Night Services stopped 08:34:21 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files) 08:35:07 Session Finished Session Finished

Session Alerts

Time     Alert Detail 21:09:29 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (NGC 7635) 21:20:57 Dome Dome has been unexpectedly stationary for more than 12 mins 21:29:13 Dome Dome Movements have begun again (outage 20.3 mins) 21:41:09 Dome Dome has been unexpectedly stationary for more than 12 mins 21:42:14 Dome Dome Movements have begun again (outage 13.1 mins) 21:42:21 Dome Dome Driver/Dome Slaving found to be ok following Slew Tests. 00:26:39 AllSky/Synopsis AllSky Star & Synopsis data has been inactive for more than 5 mins 00:26:44 Cloud Sensor Cloud Sensor data has been inactive for more than 5 mins 00:31:30 Cloud Sensor Cloud Sensor data is active again (outage 9.8 mins) 00:31:32 AllSky/Synopsis AllSky Star & Synopsis data is active again (outage 9.9 mins) 02:12:18 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (UGC 4186) 02:19:30 Dome Dome has been unexpectedly stationary for more than 12 mins 02:24:31 Dome Dome Movements have begun again (outage 17.1 mins) 02:24:36 Dome Dome Driver/Dome Slaving found to be ok following Slew Tests. 03:30:15 Job Queue Job Queue appears to be frozen (last IamAlive message at 03:19) 03:41:36 Job Queue Job Queue appears to be frozen (last IamAlive message at 03:31)     Multiple rows removed

Back to Top

Operational Issues (2020-11-22 S832)

• Critical Issues (0)
• Major Issues (1)
• Minor Issues (4)
• Continuous Improvement (1)

[ Prev | Next ]

Critical Issues

•  None

Major Issues

• Job Queue became frozen at 3:19 during Target 34/42 (NGC 2782),  effectively ending the session 3 hours earlier than planned.   Obs.Overseer issued alert of 03:30 saying that JobQueue had become frozen (Last IamAlive message was from ExecuteTarget/CenterTarget/ at 03:19).   The MakeObservatorySafe() routine was called at 03:30 which closed the shutter, parked the dome and turned off telescope tracking. With sky clouding over at 17:00 the probem caused the loss of 1.5 hours of clear sky observing.  Problem may have begun at 03:19 when an exception occurred whilst trying to set PHD2 Profile, and there was a failure to capture a Guiding Reference Frame.   Log file shows a number of trapped exceptions from 03:19:50  (Exception in PHD2.GetBinning, conversion from string "Error: device not connected" to type 'Boolean' is not valid.).    Code needs to be examined to see how the Job queue became frozen, regardless of  the underlying cause of the problem in this case appropriate protection/resilience should be in place to stop it esculating to the point that the job queue became frozen.   Extra code added to better trace exceptions whilst trying to set PHD2 Profile. Extra code added to repeat attempt to connect to PHD2 equipment
  Examining logs shows that exception in PHD2.GetBinning and PHD2.GetCalibrationData failed at 03:19:50.30 & 03:19:50.63 was due to PHD2 error {"code":1,"message":"camera not connected"},  capture_single_frame failed at 03:19:54.92 due to PHD2 error {"code":2,"message":"failed to start exposure"} , whilst guide request failed at 03:19:55.99 due to {"code":1,"message":"all equipment must be connected first"} and this is the last PHD2 message logged,
  So guiding clearly failed to get going. Target was ahead of schedule so Foc1 focusing would have been initiated but there is no indication that focusing was started.
  It can be concluded that Guider.StartGuider. was called but execution stalled within the StartPHD2Guiding() routine.
  The last report message was  Send Guide Request    Ok    Normal Mode: Settle 10s, TimeOut 15s.  Execution most like hung during the "Do While bGuideMethodFinished" . The loop has a timeout at 2 x PHD's timeout but is hidden inside a condition test  if PHD2.GetEvent <> "", but since guiding didn't start there were no guide events being generated.   Timeout has been moved outside of conditions test. Also a test for HardSuspend/SoftSuspend is also added which was missing before.
  Fixed 2020-11-24 (AstroMain 3.32.8)
   
• PHD2 lost connection to camera at 03:19 which led to Job Queue freezing.  Problem is documented with above issue. Reason for loss of connection is unclear. Problem is categorised as major as it led to the premature end of the session on this occasion and would have in case led to autoguiding to be unavailable.  Closed following review on 2021-05-25 as there haven't been an repeat occurences of the problem in sessions S833-S898.
  Closed but monitor for any future reoccurence  2021-05-25  (AstroMain / PHD2 / Session)

Minor Issues

• Session was allowed to go beyond the planned end job time. Even though the job queue froze, Obs.Manager should have still controlled things by killing the job queue and ending the session on time.
  Fixed 2020-11-24 (AstroMain 3.32.8)
   
• Too many alerts, error messages and MakeObservatorySafe calls were made between 03:41 and 08:27 following the original problem at 03:19.  Even though job queue was first recognised to be froze at 03:30 and the Observatory was successfully made safe at this time,  there were 28 further calls to Make Observatory Safe between 03:41 and 08:27 when a user intervention was made to finish the session. There was also a continuous string of SoftSupend Calls and Cancellations between 04:10 and 08:27.     Multiple MakeObservatorySafe calls is principally caused by SetTrackingOff() routine (indirectly called by MakeObservatorySafe() routine) resetting the IamAlive flag which makes it appear that JobQueue has resumed when in fact hasn't. This ends up resetting the .bFrozenJobQueueReported which repeats the identification of a frozen Job Queue every 10 minutes.  SetTrackingOff() is called from 11 different places, making it hard to either move or prevent specific IamALive called.  To get around the problem a new global flag bIgnoreIamAlive is used to temporarily avoid IamAlive() routine from functioning whilst the MakeObservatorySafe routine is operating.
  Fixed 2020-11-24 (AstroMain 3.32.8)
   
• Logfile contains warning message "Profile TS80/178MC: Mount is not calibrated" for each target" Get Phd2Calibration gave this warning message on 6 occasions during session. This message would occur if calls to GetCalibrationData() returned calibration = false, but this would only ok if PHD2 returned false (but no indication of this from PHD2 logfile), or there was an exception whilst parsing Response returned by Phd2.SendRequest. There were 8 such exceptions during the session which can more than explain the warnings.   Closed after review 2023-02-21 (problem hasn't reoccured. Issue deemed to be resolved).
  Closed after review 2023-02-21 (AstroMain, PHD2)
   
• Logfile contains several error messages "Exception in PHD2.GetCalibrationData, handling 'calibrated /  Conversion from string "" to type 'Boolean' is not valid." Get Calibration Data. e.g 22:05:35.48 NGC 246.   PHD2 log file shows request going out for calibration data, but no response is received.
  22:05:32.43 | 875 REQUEST >> {"method": "get_calibration_data", "params": ["Mount"], "id": 875}
  22:05:35.48 |
  22:05:35.48 | 876 REQUEST >> {"method": "set_guide_output_enabled", "params": [true],"id": 876}
  22:05:35.50 | 876 Response << {"jsonrpc":"2.0","result":0,"id":876}
  22:05:35.59 |
  But there is a follow up request a little later which does get a response
  22:05:36.15 | 882 REQUEST >> {"method": "get_calibration_data", "params": ["Mount"], "id": 882}
  22:05:36.17 | 882 Response << {"jsonrpc":"2.0",   "result":{"calibrated":true,"xAngle":    0.6,"xRate":10.561,"xParity":"+
                                                           ","yAngle":89.4, "yRate":12.764,"yParity":"+","declination":0.534445},"id":882}
  In total there were 52 calls to get_calibration_data method of which 8 calls ended up giving exceptions
  Question (AstroMain)

Continuous Improvement 

• A reminder to make fix AstroMain so that after Obs.Manager Makes Observatory Safe (due to closing dome or other issue) the Obs.Manager shouldn't have reopened  the dome without a user reset.  See issue in session S808 (2020-09-13).
  To fix (AstroMain)

[ Prev | Next ]

Back to Top