David's Astronomy Pages Notes - Session 805 (2020-09-04)

Notes (S804)

Notes Main

Home Page

Notes (S806)

	2020-09-04 (Session 805)
	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (2),  Minor Issues (6),  Continuous Improvement (1)
	Images from 2020-09-04 >>
	Replacement AllSky/Weather Computer - Goes Live

Session Aims & Highlights (2020-09-04)

Main aims

1. Targets. Acquire images of a number of targets if conditions allow.
2. AllSky/Weather Computer. Conduct live session test of new replacement AllSky/Weather Computer, ensuring it fufills it role in providing data to support Observatory operations and identify/resolve any teething issues.

Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• CCDApp2, TheSky6, CCDSoft5, PHD2,  DeviceHub 4.6.1.10, UPB 1.5 
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.9
   
• AstroMain 3.28.3,  AstroGuard 1.4.3,   AstroPlan 1.16,  AstroShCap 1.2.23, AstroVOE 1.3.6
  (AstroWeather 1.10.11, AstroAllSky 3.6)
  

Highlights

• AutoStart. AutoStart was successfully used to kick off the session.
• AutoFocusing. Foc1 & Foc2 Focusing (including wide focus search) successfully performed on Near-Zenith Target at start of session
•  AllSky/Weather Computer. Once a teething issue related to number of AllSky Stars was resolved the new replacement AllSky/Weather Computer operated successfully and provided the required data to support the session.

Lowlights

• Session Gremlins. AstroMain froze at 23:25, AstroMain crashed at 00:06. CCD Images not being saved from 00:37 onwards
• Dome/ Device Hub. Dome appeared to become non-responsive from 01:26 onwards.  
        Problem appears to have been caused by an issue in DeviceHub.
• Targets.  Due to Session Gremlins (in particular not saving CCD images to disk) & cloudy skies no targets were successfully observed
• CCD Images. No Target Centering or CCD images from 23:37 onwards, CCDSoft stopped saving images to disk.
• Observatory Shutdown. Observatory shutdown became stuck waiting for dome to become closed, when it was in fact already closed but incorrecly reported as 'Closing' by DeviceHub.

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Pre-Session Weather Forecast (API data from darksky.net) Pre-Session Weather forecast showed dry conditons for the night with minimum temperature of 13°C,  but fairly cloudy in first part of night (52 to 72% cloud cover) but gradually clearing towards dawn (37 - 4% cloud cover). Wind was forecast to be moderate (20 mph), and it was not expected that conditions would be suitable for an observing session. In the event conditions were less windy that expected and after seeing sky clearing at 01:00 - 01:30 a session was initiated via AutoStart around 01:30. Cloud unfortunately developed around 02:30 which persisted till Dawn. Minimum outside temperature was 13.5°C.

Session Event Log

Time     Event Detail 23:04:41 Session Created Live Session Created (2020-09-04 S00805, ImageSaveNum: 805001) 23:04:43   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 23:06:30   Services Started Observatory Services started 23:06:39 Observatory (Auto) Observatory placed in Fully-Automated Mode 23:06:41 Session Monitoring AutoStart monitoring for Live Session opportunity starting between 21:44 & 04:17 23:06:43   AutoStart Delayed Session delayed due to cloud 23:06:45 Session AutoStarting Session autostarting (23:04) 23:06:47 Session Pending Session pending (2020-09-04) 23:06:49 Session Initiating Session initiating (2020-09-04) 23:06:51   Plan Requested Observing Plan requested from AstroPlan (1.16) 23:07:29   Plan Loaded Observing Plan loaded to queue (2020-09-04 V1 BEST PLAN, Plan ID: 539) 23:07:41   Camera1 Connected SBIG Camera Connected (set point -15°C) 23:07:47   Telescope Connected Telescope Connected (TheSky6) 23:07:59 Session Equilibration Session ready for dome & camera equilibration 23:08:44   Dome Opened Dome opened (opening time 45s) 23:08:47   Equilibration Started Dome Equilibration started 23:10:39   Equilibration Ended Dome Equilibration ended (1.9 mins total) 23:10:41 Session Running Session running 23:10:43   Queue Started Observing Queue started (32 targets selected) 23:11:43       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:15:55       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 23:18:34       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 23:18:37       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 23:21:10       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 23:23:11       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 23:23:26       Focusing Failed Foc2 Focusing Check Failed 23:23:29     Target Missed (1/32) Target's time slot was missed (1/32, GCVS BL Lac) 23:23:31     Target Started (2/32) Target started (2/32, GCVS FG Sge) 23:25:00 Program Hung AstroMain program hung 23:29:36 Session Resumed Live Session Resumed (2020-09-04 S00805, ImageSaveNum: 805002) 23:29:51   Obs.Manager Started Obs.Manager started 23:29:53   Obs.Overseer Started Obs.Overseer started 23:29:55   Services Started Observatory Services started 23:30:01   UPB Port Lost UPB Powerbox Port Lost 23:30:27   UPB.Switch Restart UPB.Switch has been restarted. 23:31:09   Services Started Observatory Services started 23:31:57 Observatory (Auto) Observatory placed in Fully-Automated Mode 23:32:01 Session Pending Session pending (2020-09-04) 23:32:03 Session Initiating Session initiating (2020-09-04) 23:32:05   Plan Loaded Observing Plan loaded to queue (2020-09-04 V1 ACTIVE PLAN S805, Plan ID: 539) 23:32:07   Camera1 Connected SBIG Camera Connected (set point -15°C) 23:32:12   Telescope Connected Telescope Connected (TheSky6) 23:32:27 Session Equilibration Session ready for dome & camera equilibration 23:32:30   Equilibration Started Dome Equilibration started 23:32:32   Equilibration Skipped Dome Equilibration skipped by user 23:32:34 Session Running Session running 23:32:36   Queue Started Observing Queue started (30 targets selected) 23:32:38     Target Started (3/32) Target started (3/32, GCVS CI Cyg) 23:40:36     Target Completed Target completed (3/32, GCVS CI Cyg) 23:40:38     Target Started (4/32) Target started (4/32, GCVS S Cyg) 23:55:43     Target Completed Target completed (4/32, GCVS S Cyg) 23:55:45     Target Started (5/33) Target started (5/33, NGC 2782, ToO) 23:59:20   Camera1 Connected SBIG Camera Connected (set point -15°C) 00:00:37     Target Aborted (5/33) Target aborted (5/33, NGC 2782, ToO) due to GRB Alert 00:00:39     Target Started (6/34) Target started (6/34, UGC 6930, ToO) 00:02:50     Target Failed Target failed due to centering error (6/34, UGC 6930, ToO) 00:02:51 User Intervention User intervention to restart Obs.Software (attempt to cure issue saving CCD images) 00:02:52   Queue Aborted Job Queue aborted 00:03:12   Services Stopped Observatory Services stopped 00:03:18 Program Closed Program closed by User 00:03:20 Program Closed Program closed by User 00:04:31 Session Resumed Live Session Resumed (2020-09-04 S00805, ImageSaveNum: 805005) 00:04:48   Camera1 Connected SBIG Camera Connected (set point -15°C) 00:04:51   Obs.Manager Started Obs.Manager started 00:04:53   Obs.Overseer Started Obs.Overseer started 00:04:57   Services Started Observatory Services started 00:05:54   Services Started Observatory Services started 00:06:00 Program Crashed AstroMain program crashed with unhandled InvalidCastException 00:16:04 Session Resumed Live Session Resumed (2020-09-04 S00805, ImageSaveNum: 805005) 00:16:25   Obs.Manager Started Obs.Manager started 00:16:27   Obs.Overseer Started Obs.Overseer started 00:16:31   Services Started Observatory Services started 00:17:19   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 00:18:59   Services Started Observatory Services started 00:19:19 Observatory (Auto) Observatory placed in Fully-Automated Mode 00:19:22 Session Pending Session pending (2020-09-04) 00:19:24 Session Initiating Session initiating (2020-09-04) 00:19:26   Plan Loaded Observing Plan loaded to queue (2020-09-04 V1 ACTIVE PLAN S805, Plan ID: 539) 00:19:28   Camera1 Connected SBIG Camera Connected (set point -15°C) 00:19:32   Telescope Connected Telescope Connected (TheSky6) 00:19:46 Session Equilibration Session ready for dome & camera equilibration 00:19:49   Equilibration Started Dome Equilibration started 00:19:51   Equilibration Skipped Dome Equilibration skipped by user 00:19:53 Session Running Session running 00:19:55   Queue Started Observing Queue started (28 targets selected) 00:19:57     Target Missed (5/32) Target's time slot was missed (5/32, HIP 87937) 00:19:59     Target Missed (6/32) Target's time slot was missed (6/32, GCVS CY Lyr) 00:20:01     Target Started (7/32) Target started (7/32, 61 Cyg) 00:25:07     Target Completed Target partially completed (7/32, 61 Cyg) 00:25:09     Target Started (8/32) Target started (8/32, GCVS AM Her) 00:34:01     Target Completed Target completed (8/32, GCVS AM Her) 00:34:03     Target Started (9/32) Target started (9/32, GCVS Z And) 00:43:22     Target Completed Target completed (9/32, GCVS Z And) 00:43:24     Target Started (10/32) Target started (10/32, GCVS DY Per) 00:51:45   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 00:53:04     Target Completed Target completed (10/32, GCVS DY Per) 00:53:06   Queue Paused Queue paused for SoftSuspend 00:53:08 Session Suspended Session suspended 00:58:06     Target Cancelled Target cancelled (11/32, GCVS Z And) due to cloud 01:06:31     Target Cancelled Target cancelled (12/32, GCVS HH And) due to cloud 01:16:14     Target Cancelled Target cancelled (13/32, GCVS DY Per) due to cloud 01:25:10     Target Cancelled Target cancelled (14/32, GCVS GK Per) due to cloud 01:40:22     Target Cancelled Target cancelled (15/32, GCVS SS Cyg) due to cloud 01:49:11     Target Cancelled Target cancelled (16/32, GCVS Z And) due to cloud 01:59:04     Target Cancelled Target cancelled (17/32, GCVS DY Per) due to cloud 02:07:44     Target Cancelled Target cancelled (18/32, GCVS HH And) due to cloud 02:15:59     Target Cancelled Target cancelled (19/32, WDS GRB 34) due to cloud 02:25:42     Target Cancelled Target cancelled (20/32, GCVS DY Per) due to cloud 02:34:38     Target Cancelled Target cancelled (21/32, GCVS GK Per) due to cloud 02:46:04     Target Cancelled Target cancelled (22/32, GCVS TT Ari) due to cloud 02:59:25     Target Cancelled Target cancelled (23/32, GCVS U And) due to cloud 03:13:31     Target Cancelled Target cancelled (24/32, GCVS U And) due to cloud 03:23:11     Target Cancelled Target cancelled (25/32, GCVS RX And) due to cloud 03:33:20   Camera1 SetPoint SBIG Camera Set Point Temperature Updated (new set point -10°C) 03:36:51     Target Cancelled Target cancelled (26/32, 68P/Klemola) due to cloud 03:47:02     Target Cancelled Target cancelled (27/32, GCVS SU Tau) due to cloud 04:06:32     Target Cancelled Target cancelled (28/32, GCVS CM Tau) due to cloud 04:17:51     Target Cancelled Target cancelled (29/32, GCVS RR Tau) due to cloud 04:26:46     Target Cancelled Target cancelled (30/32, GCVS FU Ori) due to cloud 04:28:45 Session Resuming Session resuming 04:30:15   Dome Opening Dome opening hasn't finished after 90s 04:30:26 Session Suspended Session suspended 04:36:14     Target Cancelled Target cancelled (31/32, GCVS ER UMa) due to cloud 04:40:26   Camera1 SetPoint SBIG Camera Set Point Temperature Updated (new set point -15°C) 04:49:47 Session Resuming Session resuming 04:51:17   Dome Opening Dome opening hasn't finished after 90s 04:51:28 Session Suspended Session suspended 04:57:46     Target Cancelled Target cancelled (32/32, GSC 1224:29) due to cloud 04:57:51   Queue Completed Job Queue completed 04:57:58 Session Closing Session closing 05:35:40   Services Stopped Observatory Services stopped 05:43:50 Program Closed Program closed by User 05:43:52 Program Closed Program closed by User 05:46:21 Session Resumed Live Session Resumed (2020-09-04 S00805, ImageSaveNum: 805009) 05:46:54   Obs.Manager Started Obs.Manager started 05:46:56   Obs.Overseer Started Obs.Overseer started 05:47:00   Services Started Observatory Services started 05:47:23   Telescope Switched Off Telescope Power has been switched off via UPB Powerbox. 05:48:12   UPB.Switch Restart UPB.Switch has been restarted. 05:48:45 User Intervention User request to close and finish session 05:48:47 Session Closing Session closing 05:48:49   Dome Closed Dome already closed (closure time is unknown) 05:48:51   Dome Parked Dome already parked (parking time is unknown) 05:50:05   Services Stopped Night Services stopped 05:50:07 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files) 05:50:09 Session Finished Session Finished

Session Alerts

Time     Alert Detail 23:30:27 UPB.Switch UPB.Switch has been successfully restarted. 05:48:12 UPB.Switch UPB.Switch has been successfully restarted.

Back to Top

Operational Issues (2020-09-04, S805)

• Critical Issues (0)
• Major Issues (2)
• Minor Issues (6)
• Continuous Improvement (1)

[ Prev | Next ]

Critical Issues

• None

Major Issues

• CCDSoft Images not getting saved to file (23:37 onwards). BVR Images for CI Cyg at 23:37 to 23:40 took had image numbers "000" and couldn't be found on disk following the session.   Just before this Foc1 focusing failed when Take 1s Image (for Select Focus Star) gave the error message "Object reference not set to an instance of an object".    Later images 23:40 to 00:06 also had 000 image numbers and didn't save to disk.   This caused Centering Operations to fail as Locate Frames couldn't be Plate Solved (a process that requires images to be saved to disk).  It is unclear what caused the issue to begin between successfully locate frames/centering operations at 23:34 and start of Focusing &  Main Image Frames at 23:37  (changes to CCDSoft Extractor Settings to overcome the low AllSky Star Count issue where being made on a different computer and could not have been responsible). Issue resolved at the time by eventually restarting the entire software suite at 00:15.  Obs. Manager should give an alert on the first occasion that the problem occurs.   Even though the problem was resolved following restarting entire Software Suite and images were saved ok between 00:20 and 00:26 the same problem occurred again at 00:27 when again attempting to take 1s Focus Frame after which main images had 000 numbers and didn't save to disk.   The problem didn't happen in the following session (S806) and therefore doesn't seem to be a firm repeatable issue but is more of a 'Session Gremlin'.
  Originally stated to be 'Yellow/Green - Resolved (Session)',  but changed to 'Red - To be fixed (AstroMain)' following a fresh repeat of problem in S815 (2020-09-29). Ruling out a standard explanation it has to be assumed that problem was due to a CCDSoft.exe or CCDSoft Com Object issue and a fix put in place such that if a Locate Frames or Main Image Frame returns Image.Path ="" then CCDSoft will be restarted and the image repeated.   Issue fixed 2020-09-30 and only proven as fixed during session S831 (2020-11-21).  Due to small oversight subsequent FITS images began saving to an incorrect location but this was fixed in a later sprogram update (AstroMain 3.32.8)
  Fixed 2020-09-30 (AstroMain 3.29.5)
   
• Issues with Dome Connection / Closing Dome (00:53). Sky conditions deteriorated at 00:51 (AllSky StarCount dropped to 17, which was below the threshold of 20 stars being used), and Obs.Manager called a Soft Suspend. Following completion of current target (DY Per) the session suspended at 00:53 and 'Closing Dome' was reported.    However there was no corresponding message line or event to report that dome closure had been completed.
  AstroGuard observed that Dome was closing at 00:53 but noted that shutter had taken > 68s to close.   Later on when there where attempts to resume session and open the dome (shutter not open after 90s) and whilst checking/closing shutter during shutdown.  ObsPic analysis shows the problems began at 00:26 or 00:27 when dome stopped moving in response to movements in the telescope.  An investigation was carried out a few day later -  2020-09-07 (see Investigation - Dome appears non-responsive or stuck (S805) ) . This concluded that parts of the DeviceHub had seemingly become frozen at 00:27 though parts of the program remained working and passed on client requests to Open/Close the Shutter. Although these got through to the dome the changes in Dome Position / Status weren't being read and returned back to the Client causing the appearance of a non-responsive dome. Issue was eventually fixed on 2020-10-15 by the installation of Device Hub 6.5.0.1 which has better exception trapping.   Issue deemed closed on 2020-11-06 after there were no similar incidents during the first 8 sessions using the new DH version. 
  Closed 2020-11-06 (Device Hub 6.5.0.1).

Minor Issues

• Session AutoStart unexpectedly delayed even though sky was clear. At the earliest scheduled AutoStart time AstroMain reported "Unable to begin AutoStart at 21:45 due to cloud".  This happened even though the sky was clear and stars where appearing in the sky.  AllSky StarCount was lower than expected. Star Count was only 12-15, whilst 30-35 stars could be expected based on clear, albeit moonlit, sky.  AutoStart was using a threshold of 20 stars to begin the session.  It was eventually realised that the problem was a teething issue associated with running AstroAllSky/CCDSoft running on the new AllSky/Weather Computer which was fired up earlier in the day.  The threshold was temporarily removed to allow the session to autostart. Meanwhile the  Source Extraction Settings in CCDSoft on the new AllSky/Weather Computer were edited. After setting parameter values to those used on the old AllSKy computer, star count jumped up to 30 stars, and the threshold (20 stars) reenabled. AutoStart eventually started at 23:04 (1 hour 20 min of potential observing time were lost).
  Fixed during session 2020-09-04 (AllSky Computer/CCDSoft Extractor Settings)
   
• AstroMain program froze (23:25).  This might have been associated with user clicking on Program window to bring it to the front of other windows. This was the first time this issue had been seen for several sessions.  Issue resolved by doing a partial restart of Observatory Software.  (telescope left operating).  Issue closed on 2021-01-04 following a review that showed that the AstroMain was essentially stable (following a series of improvements and fixes), with an absence of any unexplained program hangs during the 35 sessions between  2020-09-18 & 2021-01-03.
  Closed 2021-01-04 (AstroMain)
   
• Target Centering Operations failing as Locate Frames were unable to be Plate Solved due to error  "Image must be saved before linking" (23:41).  Show Inventory was also failing "Exception caught in AnalyseStarsInImage.  Please save the image before using this command."  These problem were associated with a wider issue that CCDSoft Images weren't being saved to disk from 23:37 onwards.   Issue resolved at the time by eventually restarting the entire software suite at 00:15.  Obs. Manager should give an alert on the first occasion that the problem occurs. 
  Resolved (Session)
   
• AstroMain program crashed due to unhandled InvalidCastException (00:06).  Source Files were out of date on Observatory Computer and it is probable that the location reported by Visual Studio (GetDecimalDec() ) was not correct.  Issue closed on 2021-01-04 following a review that showed that the AstroMain was essentially stable (following a series of improvements and fixes), with an absence of any unexplained program hangs or crashes during the 35 sessions between  2020-09-18 & 2021-01-03.
  Closed 2021-01-04 (AstroMain)
   
• Observatory Shutdown 'stalled' during Close Dome task (04:57). Due to an issue with DeviceHub, which was reporting Dome Shutter was closing when it was in fact closed, Observatory Shutdown operations (ran as part of Session State=Closing) were unable to progress beyond the Close Dome stage. Obs.Manager/Close Dome code needs to be modified so that it can not become stuck and can instead allow Alerts to be generated and allow remaining shutdown tasks (Park Scope/TurnOff Scope/Shutdown Camera/HouseKeeping) to be conducted. 
  Code modified : CloseShutter() now calls objDome.CloseShutter() even if ShutterStatus is ShutterState.Closing.   Session S805 shows that DeviceHub can be showing Dome' shutter status is Closing even if it is physically open, yet will pass on a command to Close Shutter (even if Shutter Status doesn't change to reflect a Closed state). Progress can no longer be stuck in ClosingStage1, the shutter will either close or the stage will be exited after 3 x 120s attempts to close the dome.
  Fixed (AstroMain 3.28.4)
   
• "DeviceHub caught a fatal exception during startup" message (05:36).  Message noted at 05:37 and attempted debugging in VisualStudio (without source code) indicated that exception was a System.InvalidOperationException (cannot set visibility to Visible or Show, ShowDialog, Close or WindowInteropHelper.EnsureHandle whilst a Window is closing'.  It seems to have been associated with a possible restart of Device Hub at 05:36 Resolved by restarting DeviceHub.  Issue fixed by new DeviceHub version (6.4.1.11 and 6.5.1)
  Fixed 2020 (DeviceHub)
   
• Session.(log) file contains errors : Exception whilst saving Session.TargetInfo1.png.   Log file contains 11 occurances of exception 'The given path's 's format is not supported, whilst attempting to save Session.TargetInfo1.png in Save Target Chart routine. (1st at 04:36 during Target 31/32 (GCVS ER UMa), last at 04:57 during Target 32/32 (GSC 1224:29).  There were no occurances in preceeding session (S804), but in the following session (S806) there were 24 occurances of the exception  (1st at 00:46 during Target 13/36, last at 00:54 during Target 14/36).  Code modified in AstroMain 3.28.4 to output the full names of files being saved/copied when exception occurs, which can help identify if there is a file name issue.  Other possibilities is that file output file becomes 'locked' whilst it is being uploaded to website, and can't be overwritten.   Review of log file from S812 show 25 occurances of the exception each of them associated with the ChartFileName1   ..\2020-09-19 S00812\_TargetInfo\TargetInfo.20200919.chart31.GSC_1224:29.png. Reviewng the _TargetInfo folder shows that the ..chart31.. file is missing from the folder, 33 other .png files are present including ..chart30.. and ..chart 32..   ChartFileName2: is blank in the log file.   It would seem that issue is caused by trying to use the ":" that is in Target Name as part of the file name.   Fixed by extending GetFileBasedName() routine to also remove ":" in addition to other characters not allowed in file names.
  Fixed 2020-09-28  (AstroMain 3.29.3) 

Continuous Improvement

• Obs.Manger to generate an alert on first new occasion that Image Plate Solving fails due to "Image must be saved before linking" error and/or  Image has [0000] image number and isn't getting saved to disk. An attempted fix has now been put in place such that if a Locate Frame, Main Image Frame or Mapping Frame returns and Image with Image.Path ="" then CCDSoft will be restarted and the image repeated. after posting an alert.
  Fix Attempted 2020-09-30 (AstroMain 3.29.5)

[ Prev | Next ]

Back to Top

Replacement AllSky/Weather Computer - Goes Live (2020-09-04)

Back to Top