David's Astronomy Pages Notes - Session 1159 (2023-10-31)

Notes (S1158)

Notes Main

Home Page

Notes (S1160)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (1),  Minor Issues (8),  Small Defects (5),  Continuous Improvement (9)
	Images from 2023-10-31 >>
	2023-11-02
	Investigation - ObsManager freezing just after entering 'Session Running'

Session Aims & Highlights (2023-10-31)

Main aims

1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
2. AstroMain. Check new AstroMain version (3.63.4) for recent fixes and improvements, including fix to flag upload of new Detail' JPG pictures (TargetDetail)
3. Dome Slewing.  Check Dome Slewing after application of Anti-Slip Tape to inner roof flange and bush-fix to Drive Wheel Connector
4. Detail JPG pictures.  Check upload of 'Detail' JPG Pictures and display in 'latest_images' web page (Session.TargetDetailxx.jpg)

 Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• CCDApp2, TheSky6, CCDSoft5, DeviceHub 6.6.1.1,  MeadeGeneric 1.3.5.414, UPB 1.5, ASCOM 6.6
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.11dev4, Siril 1.0.6, deCONZ 1.18.2, ASCOM 6.6 SP1
   
• AstroMain 3.63.4, AstroPlan 1.38.8   AstroShCap 1.3.15, AstroGuard 1.10.2
  AstroVOE 1.13  AstroSN 1.10,  AstroProtect 1.7,  AstroMove 1.12.7
  (AstroWeather 1.22.1, AstroAllSky 3.19.6,  AstroObsCam 1.11.4, AstroLaunch 1.11.1

Highlights

• Targets. 5/28 planned targets observations were completed with 4 further targets partially completed.  Additionally one ToO target (M31 w/Nova M31 AT2023veq was completed. Other targets were missed, or failed or cancelled due to cloud/rain showers and session suspension.   Images from one target (NGC 7625 w/SN2023vyl ) were later rejected due to cloud. Images from other targets were partially affected by cloud and/or suffered bad guiding.
• AstroMain. No significant new issues with AstroMain version (3.63.4). Issue with ObsManager/TempMonitor freezing is an existing occassional issue.
•  Dome Slews. No dome slews encountered any hold-ups or delays. 
   (Hold Up Stats:0/12 slews affected,0%, 0.0 minutes total hold-up time).  Dome parked ok at end of session (despite alert that it wasn't).
• Detail JPG pictures.   'Detail' JPG Pictures created ok by AstroMain and correctly passed to Live folder and successfully uploaded to website.

Notes:

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Actual Weather vs Pre-Session Weather Forecast

Session Event Log

Time     Event Detail 21:41:36 Session Monitoring AutoStart monitoring for Live Session opportunity between 21:41 & 05:12 21:41:40 Session AutoStarting Session autostarting (21:41) 21:41:50   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.) 21:42:15   Camera1 Connected SBIG Camera connected (set point -25°C) 21:42:17 Session Created Session Created (Live, 2023-10-31 S01159, ImageSaveNum: 1159001) 21:42:22   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 21:44:08   Services Started Observatory Services started 21:44:14 Observatory (Auto) Observatory placed in Fully-Automated Mode 21:44:17 Session Pending Session pending (2023-10-31) 21:44:19 Session Initiating Session initiating (2023-10-31) 21:44:21   Plan Requested Observing Plan requested from AstroPlan (1.38.8) 21:45:52   Plan Loaded Observing Plan loaded to queue (Plan ID: 933) 21:45:58 Foc1FieldsUpdater Frozen Foc1FieldsUpdater appears to be frozen at around 21:45 in section 'Update Foc1Fields' 21:52:50   CCDSoft Restarting CCDSoft being restarted (user action) 21:52:57   SBIG Camera Reset SBIG Camera being power-cycled 21:53:28   Telescope Connected Telescope connected (TheSky6) 21:53:39   Camera1 Connected SBIG Camera connected (set point -20°C) 21:53:43 Foc1FieldsUpdater Frozen Foc1FieldsUpdater appears to be frozen at around 21:53 in section 'Update Foc1Fields' 21:53:52 Session Equilibration Session ready to Open Dome 21:54:37 Obs.Manager Obs.Manager is frozen in 'Equilibrating' (last cycle was at 21:54:37) 21:54:37   Dome Opened Dome opened (opening time 45s) 21:54:39 Session Running Session running 21:55:06 TempMonitor Frozen TempMonitor appears to be frozen at around 21:55 in section 'Get CcdTemp Data' 21:56:41   Make Observatory Safe Obs.Overseer intervened to Make Observatory Safe 22:18:24   CCDSoft Restarting CCDSoft being restarted (user action) 22:18:27   Dome Closed Dome closed (closing time 1,304s) 22:18:30 Obs.Manager Obs.Manager has resumed again after 23.9 min 22:18:33   SBIG Camera Reset SBIG Camera being power-cycled 22:19:13   Camera1 Connected SBIG Camera connected (set point -20°C) 22:21:27   Camera1 Disconnected SBIG Camera closed down & disconnected 22:21:47   Services Stopped Observatory Services stopped 22:21:51 Session Closed Session closed by User 22:21:54 Program Closed Program closed by User 22:23:37 Program Opened Program Opened (AstroMain 3.63.4) 22:23:39 Session Resumed Session Resumed (Live, 2023-10-31 S01159, ImageSaveNum: 1159001) 22:23:50   Obs.Overseer Started Obs.Overseer started 22:23:59   Obs.Manager Started Obs.Manager started 22:24:08   Services Started Observatory Services started 22:24:57 Session Monitoring AutoStart monitoring for Live Session opportunity between 22:24 & 05:12 22:24:59 Observatory (AutoStart) Observatory placed in Auto-Start Mode 22:25:02 Session AutoStarting Session autostarting (22:25) 22:25:47   Services Started Observatory Services started 22:25:54 Observatory (Auto) Observatory placed in Fully-Automated Mode 22:25:57 Session Pending Session pending (2023-10-31) 22:25:59 Session Initiating Session initiating (2023-10-31) 22:26:03   Plan Loaded Observing Plan loaded to queue (Plan ID: 933) 22:26:16   Camera1 Connected SBIG Camera connected (set point -20°C) 22:26:51   Telescope Connected Telescope connected (TheSky6) 22:27:15 Session Equilibration Session ready to Open Dome 22:28:01   Dome Opened Dome opened (opening time 45s) 22:28:03 Session Running Session running 22:28:05   Queue Started Observing Queue started (26 targets selected) 22:28:07     Target Started (NrZen) Target started (Focus Field 1, HIP 4911) 22:28:21       Dome Unparked Dome unparked 22:33:16       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 22:35:50       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide) 22:37:48       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 22:37:52       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 22:39:54       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 22:41:35       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 22:41:52     Target Completed (NrZen) Target completed (Focus Field 1, HIP 4911) 22:41:56     Target Started (2/27) Target started (2/27, NGC 7625 w/SN2023vyl) 22:46:00       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 22:49:18       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 22:49:30     Target Completed Target partially completed (2/27, NGC 7625 w/SN2023vyl) 22:49:32   Queue Paused Queue paused for Session Suspension 22:49:33 Session Suspending Session suspending 22:50:23   Dome Closed Dome closed (closing time 50s) 22:50:25 Session Suspended Session suspended 22:52:18     Target Cancelled Target cancelled (3/27, 61 Cyg) due to cloud 23:11:40     Target Cancelled Target cancelled (4/27, M33 w/AT2023nou) due to rain 23:23:58     Target Cancelled Target cancelled (5/27, GCVS BL Lac) due to rain 23:40:34 Session Resuming Session resuming 23:40:49   Critical Rain Alert Critical Rain Alert (Obs.Manager will close the Dome) 23:41:09   Dome Closed Dome closed (closing time 20s) 23:41:35     Target Cancelled Target cancelled (6/27, Nova Cas 2021) due to rain 23:42:14 Session Suspended Session suspended 23:44:49 Session Resuming Session resuming 23:45:14   Critical Rain Alert Critical Rain Alert (Obs.Manager will close the Dome) 23:45:44   Dome Closed Dome closed (closing time 30s) 23:46:30 Session Suspended Session suspended 23:49:20 Session Resuming Session resuming 23:49:40   Critical Rain Alert Critical Rain Alert (Obs.Manager will close the Dome) 23:50:05   Dome Closed Dome closed (closing time 25s) 23:51:00 Session Suspended Session suspended 00:01:01     Target Cancelled Target cancelled (7/27, 29P/Schwassmann-Wachmann) due to rain 00:01:41 Session Resuming Session resuming 00:02:26   Dome Opened Dome opened (opening time 45s) 00:02:29 Session Running Session running 00:02:31   Queue Resumed Observing Queue resumed 00:02:33     Target Started (8/27) Target started (8/27, AT2023tdi) 00:07:59       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 00:11:30     Target Aborted (8/27) Target aborted (8/27, AT2023tdi) due to Priority 1 ToO Target 00:11:34     Target Started (9/28) Target started (9/28, M31 w/Nova M31 AT2023veq, ToO) 00:14:52       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:16:49       Focusing Completed Foc1 AutoFocus Completed (Profile No 3) 00:32:54     Target Completed Target completed (9/28, M31 w/Nova M31 AT2023veq, ToO) 00:32:59     Target Started (10/28) Target started (10/28, 12P/Pons-Brooks) 00:41:01       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 00:41:26       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 00:46:46     Target Completed Target partially completed (10/28, 12P/Pons-Brooks) 00:46:50     Target Started (11/28) Target started (11/28, AT2023txn) 00:56:37     Target Completed Target completed (11/28, AT2023txn) 00:56:41     Target Started (12/28) Target started (12/28, NGC 3893 w/AT2023uhx) 00:59:58       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 01:00:43       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:01:59       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:02:04       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 01:03:22       Focusing Completed Foc1 AutoFocus Completed (Profile No 4) 01:05:19       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:08:40       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 01:08:45       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:12:20       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 01:12:35       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:15:05       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 01:15:15       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:16:00       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 01:16:46       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:17:11       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 01:17:16       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:17:31       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity) 01:18:11       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 01:19:14     Target Completed Target completed (12/28, NGC 3893 w/AT2023uhx) 01:19:18     Target Started (13/28) Target started (13/28, C/2023 H2 (Lemmon)) 01:22:30       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:23:01       Focusing Failed Foc1 focusing failed - star lost 01:35:58     Target Completed Target completed (13/28, C/2023 H2 (Lemmon)) 01:36:02     Target Started (14/28) Target started (14/28, M101 w/SN2023ixf) 01:39:42       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 01:53:40       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 01:55:46     Target Completed Target completed (14/28, M101 w/SN2023ixf) 01:55:48   Queue Paused Queue paused for Session Suspension 01:55:51 Session Suspending Session suspending 01:56:40   Dome Closed Dome closed (closing time 50s) 01:56:43 Session Suspended Session suspended 02:17:38     Target Cancelled Target cancelled (15/28, M31 w/Nova M31 AT2023uri) due to cloud 02:29:48     Target Cancelled Target cancelled (16/28, GCVS RX And) due to cloud 02:41:18     Target Cancelled Target cancelled (17/28, AT2022sru (CV)) due to cloud 03:01:38     Target Cancelled Target cancelled (18/28, M31 w/Nova M31 2023-10a?) due to cloud 03:12:29 Session Resuming Session resuming 03:13:15   Dome Opened Dome opened (opening time 45s) 03:13:17 Session Running Session running 03:13:19   Queue Resumed Observing Queue resumed 03:13:22     Target Started (19/28) Target started (19/28, SN 2023sfl) 03:20:10       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 03:20:50       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 03:29:39     Target Completed Target partially completed (19/28, SN 2023sfl) 03:29:43     Target Started (20/28) Target started (20/28, AT2022obq (CV)) 03:32:33       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 03:35:06       Focusing Completed Foc1 AutoFocus Completed (Profile No 6) 03:35:12       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 03:40:55     Target Completed Target completed (20/28, AT2022obq (CV)) 03:40:57   Queue Paused Queue paused for Session Suspension 03:41:00 Session Suspending Session suspending 03:41:48   Dome Closed Dome closed (closing time 50s) 03:41:50 Session Suspended Session suspended 03:49:58     Target Cancelled Target cancelled (21/28, AT2023wbv (CV)) due to cloud 04:04:42     Target Cancelled Target cancelled (22/28, C/2017 K2 (PANSTARRS)) due to rain 04:15:53     Target Cancelled Target cancelled (23/28, GCVS SU Tau) due to cloud 04:27:23     Target Cancelled Target cancelled (24/28, AT2023cub (CV)) due to cloud 04:39:23     Target Cancelled Target cancelled (25/28, AT2023ttx (CV)) due to cloud 04:59:23     Target Cancelled Target cancelled (26/28, AT2023twu) due to cloud 05:21:43     Target Cancelled Target cancelled (27/28, UGC 4308 w/SN2023usf) due to cloud 05:50:27     Target Cancelled Target cancelled (28/28, 103P/Hartley) due to cloud 05:50:32   Queue Completed Job Queue completed 05:50:38 Session Closing Session closing 05:50:43   Dome Closed Dome already closed (closed at 03:41, closing time 50s) 05:52:14   Dome Parked Dome parked (parking time 82s), Az: 90.1 deg 05:53:35   Telescope Parked Telescope parked (parking time 75s) 05:53:50   Telescope State Handbox reads 'Scope parked Turn scope off.' 05:54:12   Telescope Switched Off Telescope Power has been switched off via UPB Switch. 05:55:36   Services Stopped Night Services stopped 05:55:39 Session Housekeeping Session Housekeeping started (Create Fits Summary, Transfer Files) 05:55:49 Session Finished Session Finished

Session Alerts & Alarms

Time     Type       Name Detail 21:47:40 Red Alert Foc1Fields Updater Foc1FieldsUpdater appears to be frozen (last cycle at 21:45:58) in section 'Update Foc1Fields' 21:52:55 Green Clear Foc1Fields Updater Foc1FieldsUpdater has resumed again (21:52) after 6.9 mins 21:55:25 Red Alert Foc1Fields Updater Foc1FieldsUpdater appears to be frozen (last cycle at 21:53:43) in section 'Update Foc1Fields' 21:56:40 Red Alert Obs.Manager Obs.Manager is frozen in 'Equilibrating' (last cycle at 21:54:37) 21:57:41 Red Alert TempMonitor TempMonitor appears to be frozen (last cycle at 21:55:06) in section 'Get CcdTemp Data' 22:18:25 Red Alarm Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closed) 22:18:28 Green Clear Obs.Manager Obs.Manager has resumed again after 23.9 min 22:18:30 Green Clear Foc1Fields Updater Foc1FieldsUpdater has resumed again (22:18) after 24.7 mins 22:18:32 Green Clear TempMonitor TempMonitor has resumed again (22:18) after 23.3 mins 00:57:31 Yellow Alert Anomalous Dome Slew Dome slew resumed at Az 38.3° (T+45.5s) 05:46:10 Red Alert Dome Relay Dome Relay (Safe) doesn't reflect CloudSensor Relay for 2 minutes or more 05:54:44 Yellow Alert Dome Charging Dome hasn't started re-charging after 2.5 min. Dome may not have parked correctly 05:55:49 Red Alert Dome Charging Dome hasn't started re-charging after 3.6 min. Dome may not have parked correctly

Back to Top

Operational Issues (2023-10-31, S1159)

• Critical Issues (0)
• Major Issues (1)
• Minor Issues (8)
• Small Defects (5)
• Continuous Improvement (9)
  

[ Prev | Next ]

Critical Issues

• None

Major Issues

• AstroMain's ObsManager became frozen just after entering 'Session Running' state (21:54). Just after moving to 'Session Running'  ObsOverseer raised alerts with the messages "Foc1FieldsUpdater is frozen (21:55) in section 'Update Foc1Fields' ",  "Obs.Manager is frozen in 'Equilibrating' (last cycle at 21:54:37)" & "TempMonitor appears to be frozen (last cycle at 21:55:06) in section 'Get CcdTemp Data'", and then proceeded to call the MakeObservatorySafe routine. RestartCCDSoft was manually initiated at 22:18 which unlocked ObsManager and TempMonitor. AstroMain restarted.  This is similar to an issue in a session 5 weeks ago (2023-09-25, S1149).  Issue was subject of a post-session investigation (see Investigation - ObsManager freezing just after entering 'Session Running (2023-11-02) ) which determined that Obs.Manager was freezing in the SetState_Running() routine at the code line:   ReportComment(LPad10("Ok") + Camera.AutoSavePath), when  the CCDSoft.Camera was in a frozen/non-responsive state.  Fixed by changes made to AstroMain 3.63.5 so that SetState_Running no longer directly accesses CCDSoft.Camera, so that the Obs.Manager is no longer prone to freezing , but instead accesses locally stored recent values (obtained by CCDMonitor ). These are accessed using the properties of new CCDCamera object with names matching those of the CCDSoft.Camera (e.g.CCDCamera.AutoSavePath) and using new 'isolated' versions of specific routines, ReportCCDTemperature_Isolated() &  CheckCCDTemperature_Isolated(),  that also access the new CCDCamera properties. The results of using the new CCDCamera object should be reviewed after trialling new software to see if data efficiacy is adequate and whether any modifications are needed.
  Fixed 2023-11-01 (Session, AstroMain 3.63.5) 

Minor Issues

• A 'Dome Closing' Alert was raised with message "Dome closure hasn't finished after 90s" (22:18).  Red alert issued at 22:18:25 with the message "Dome closure hasn't finished after 90s (ShutterState=Closed)". Call to Close Shutter was made at 21:56:41 as part of a MakeObservatorySafe' operation. Zigbee Event Monitor reported the message "Shutter has closed" at 21:57:25. This is 44s later and therefore the close shutter operation was normal.  The alert message relates to a separate issue that ObsManager process became frozen whilst shutter was closing (related to CCDSoft.Camera not returning a Method/Property call), and once unlocked the ShutterClosing Monitor was then late in evaluating the dome shutter closure time, and eventually posted a Dome Closed event with the description  "Dome closed (closing time 1,304s)".
  Issue understood 2023-11-01 (Session, AstroMain ) 
   
• After end of the session 'Dome Charging' alerts were raised with message "Dome hasn't started re-charging ... Dome may not have parked correctly" (05:55). A new feature added to AstroMain 3.63.4 was to issue a green or yellow alert at end of  'Session Stopping' to flag whether the Dome Charging has or hasn't started after parking the dome , and in event that charging hasn't be detected to then issue a green or red alert at end of 'Session Housekeeping'.  However it doesn't seem to working properly in its first live session trial. At the end of Session S1159 (after finishing Dome Parking at 05:52:14), a yellow alert was issued at 05:54:44 with the message "Dome hasn't started re-charging after 2.5 min. Dome may not have parked correctly" followed later by a red alert at 05:55:49 with the message "Dome hasn't started re-charging after 3.6 min. Dome may not have parked correctly" .  The ObsPic at 05:33:27 however records "Dome Charging (0.2min ago)" with a Green Flag. Following ObsPics continue to show a green coloured 'Dome Charging' flag.  With case like this the suspicion is on the logic used by  if..else statements.  In this case the following code line is used
     If DomeRecord.LastChargeTime < LastDomeParkTime
  If true a Green Alert is given, whilst if false a Yellow or Red Alert is given.  The logic comparison '<' is incorrect. It should be '>'  so that the statement check if LastChargeTime is more recent (ie a greater date value) than LastDomeParkTime.
  Fixed 2023-11-01 (AstroMain 3.63.5) 
   
• AllSky/Weather Computer went offline and is not reachable via Network (2023-11-01 13:50).   CloudSensor, Weather and Scorecard Services in AstroMain running on Observatory Computer are not recieving records from the AllSky/Weather Computer (Laptop7).. Files that are uploaded from AllSky/Weather Computer haven't refresh since around 13:50. It look like computer has switched off or fatally crashed. Observatory visited in early evening and hold that the computer was off (a follow -up examination of computer log will be made to see it can shed any light on why the computer stopped).  Computer was successfully restarted and ALlSky/Weather and ObsCam programs were restarted.
  Resolved  2023-11-01 (AllSky/Weather Computer)
   
• Report messages were interlaced after manually initiating 'RestartCCDSoft' to unlock a frozen UpdateFoc1Fields process.  After noticing that session progress was seemingly held-up and there was an alert that UpdateFoc1Fields was frozen, the backdoor button routine 'Restart CCDSoft' was used. This unlocked the held up process(es) as expected but the report messages from Starting Session became interlaced with report messages still coming from RestartCCDSoft.  This is similar to another recent issue involving Restart CCDSoft and the accompanying interlacing of report lines. Review that earlier fix and extend it to the current case. Fixed in AstroMain 3.63.5.  Condition Checks have been added or extended in each Starting Stage to prevent operational calls / progress to next Starting Stage, unless bRestartingCCDSoft = False.   [ A call to be Subroutine called HoldForRestartOps() that waits of bRestarting to turn false or timesout after 120s was considered but is deemed inappropriate as would block the ObsManager from critical checks (like closing dome in event of critical conditions).
  Fixed 2023-11-01 (AstroMain 3.63.5 ) 
   
• Power cut to observatory (2023-11-03 17:36).  Due to a fuse going on the house's downstair's ring main the power to the observatory was cut at 17:36 (2023-11-03).  Fuse was reset restoring power to the Observatory. Checking the observatory computers it was noticed that there was a fatal exception in AstroMain program running on the Observatory Computer, that was caught by the Just-In-Time debugger.  The seems related a date/string conversion in the new Dome Relay Protection monitor and was fixed by adding a Try-Catch-End Try block around the 'offending code' in AstroMain 3.63.5.  On the AllSky/Weather Computer it was noted that communication had been lost with the Cloud Sensor which was resolved by restarting the CloudSensor III software. A reset of the USB Port to CloudSensor was also performed as a precaution.   It was noticed that AstroAllSky seems to have stopped updating graphs & images to the website, which was resolved by restarting the AstroAllSky program.
  Resolved and Fixed 2023-11-03 pm (Observatory,  Observatory Computer, AstroMain 3.63.5 Update, AllSky/Weather Computer,  CloudSensor, AstroAllSky) 
   
• Near end of session a red alert was issued with message "Dome Relay (Safe) doesn't reflect CloudSensor Relay for 2 minutes or more" (05:46). Dome Relay Protection is the Observatory's 3rd level protection against damage to observatory or equipment from rain or snow. If the Dome Control Unit detects the Relay being unsafe it will command the closure of the Shutter after 30s (if it is isn't already closing or is closed). AstroMain and AstroGuard are both programmed to command a Shutter Closue in the event of rain, cloud or deteriorating conditions and this is normally happens before the Relay/Dome Control Unit's closure kicks in. If Dome Relay Protection stops for any reason and AstroMain and AstroGuard have both failed or cannot communicate with the Dome the observatory and it's equipment are left exposed. A new feature added to AstroMain 3.63.4 was to monitor Dome Relay Protection and issue 'Dome Relay' alerts if the protection is deemed to have stopped working correctly.   At 05:46:10 the monitor raised a red alert with message "Dome Relay (Safe) doesn't reflect CloudSensor Relay for 2 minutes or more".   The alert was cleared later by a green alert at 07:08:27 with message "Dome Relay (Unsafe) matches CloudSensor Relay".  [ This Green Alert seems quite later, but it can only happen after seeing 'Dome Relay (Safe)' (which was the dome's relay state when red alert was raised), change back to 'Dome Relay (Unsafe)' and be consistent with CloudSensor Relay. This change is very dependant on when conditions change ] .  As this is the first live session trial of this new monitor and earlier tests of the monitor has produced false alerts it raises the question  'Is this alert at 05:46 a valid alert or false alert ?'
   CloudSensor Relay was percieved to have changed to 1 (Unsafe) at 05:44:00 but the Dome Relay was percieved to have remained on 0 (Safe). Consequently at around 120s later (actually at 126s later when the ReplayProtectionMonitor was run) the Safe-Unsafe condition led to the determination that Dome Replay Protection had stopped and the Red Alert was therefore raised. Issue fixed in AstroMain 3.63.5 Update (2023-11-04 am),  ParseCloudSensorRecord() changed to determine changes in Relay based on the actual Cloud Sensor .RelayFlag rather than the calculated .Relay state.   CheckRelayProtection() routine changed to use CloudSensorData.RelayFlag .
  Fixed 2023-11-04 am (AstroMain 3.63.5 Update) 
   
• AstroMain's 'Starting Session' routine held up during 'Connect Camera' / Foc1FieldsUpdater frozen (21:46).  Session Starting held up after entering 'Connect Camera' ObsOverseer reported "Foc1FieldsUpdater is frozen (21:47) in section 'Update Foc1Fields' ". Noticing this via remote monitoring 'Restart CCDSoft' was manually initiated. This release the locked process(es), however the Report Messages that followed were interlaced (see issue below) and later ObsManager became frozen just after entering 'Session Running'. This is similar to an issue in another recent session (S11xx, 2023-09-xx). Fix attempted in AstroMain 3.63.5 Update (2023-11-03 pm) by isolating the RunFoc1FieldsUpdater from direct calls to CCDSoft.Camera and to instead use locally stored values accessed using propert calls to a new CCDCamera object.  New version tested 2023-11-04 am (Session '2023-11-03 Test') but it unexpectedly produced a ObsOverseer warning with the message "Foc1FieldsUpdater is frozen (00:49) in section 'Update Foc1Fields' " , (this was a point ~ 4 minutes after Connect Camera ), which was followed by a second ObsOverseer warning "TempMonitor is frozen (00:51) in section 'Get CcdTemp Data'" .   Examining the code shows that the RunFoc1FieldsUpdater () is still not totally isolated from CCDSoft.Camera as the routine calls UpdateFoc1Fields() which is still accesses CCDSoft.Camera.  Fixed in a further update of 3.63.5
  Fixed 2023-11-03 pm (AstroMain 3.63.5 Update) by continue to Monitor for further instances or potentially related problems.
   
• AllSky Camera Hemisphere contains large area of dew droplets from earlier condensation.  Dew affects quality of AllSky images, reduces star count (stars that present in the sky are hidden/not recognised) and impacts automated sky conditions monitoring.  Fixed (for time being) during session S1161.
  Fixed 2023-11-06 (AllSky Camera)

Small Defects

• AstroMain's Zigbee Event Monitor is logging very large 'time since prev changes' numbers for GreenHouse sensor.  Log file for 2023-11-03 (and almost certainly before) are showing messages like "Greenhouse Temperature 9.0°C (09:42:33, time since prev change 17,731,833.7 hour)"  after restarting AstroMain program.  This is probably due to a TimeSince calculation using a Null date value.  Fixed in AstroMain 3.63.5 update,  the necessary initialising of Greenhouse sensor. LastStateTimes to vcNullDate were added to ResetLastStateTimes() routine.  (vcNullDate is specificaly tested for before reporting a TimeSince number where as a regular Null Date isn't.)
  Fixed 2023-11-03 (AstroMain 3.63.5 Update) 
   
• Date/Time of Last Data from CloudSensor is incorrectly displayed in AstroWeather (2023-11-03).  Date/Time of last data from CloudSensor display on AstroWeather's main tab and on Data tab is incorrect.  Date was showing as "2023-18-19 18:19" instead of expected value "2023-11-03 18:19.  The format of the displayed date would appear to incorrectly using "yyyy-HH-mm HH:mm" instead of the "yyyy-MM-dd HH:mm".  This must be a long-standing bug and it is surprising that it has only just now be noticed.  This is probably because the colour flags and 'time since' values have been as expected. The erronous code lines concerned have a comment   "' changed 2022-09-29 (CI S1034)", which suggests that this was probably when the bug was introduced ?.  Display format fixed in AstroWeather 1.22.2.
  Fixed 2023-11-03 (AstroWeather 1.22.2) 
   
• Dome Relay Protection unexpectedly changed to Critical (magenta) (2023-11-03 19:27).  Relay Protection Monitor reported that "Dome Relay (Safe) doesn't reflect CloudSensor Relay (19:27)". However ObsPIcs at the time show that the CS Relay was Safe, so it isn't clear why Relay Protection Monitor considered them to have different states. They both seem to be 'Safe'.    Reviewing log file indicates that CS Relay changed from 0 (Safe) to 1 (Unsafe) at 19:23:36 and stayed in this state until 19:28:49 when it chnage back to 0 (Safe).  On the otherhand, The Dome Relay on the otherhand change from 0 (Safe) to 1 (Unsafe)  at 19:24:08, and then alternated back and forth until it stabilised on 0 (Safe) at 19:25:30, where it remained on 0 (Safe) until 19:42:11. There was thus a 120s+ interval between 19:25:30 and a relay protection check at 19:27:37 when Dome.Relay was 0 (Safe) whilst the CloudSensor.Relay was 1 (Unsafe) and this triggered the Red Alert and the raising of Relay Protection to ThreatLevel.Critical and magenta flag on ObsPic. The discrepency between logfile (showing CS Relay 1 (Unsafe) during the above mentioned 120s interval) and ObsPics (showing CS Relay 0 (Safe) during the same 120s interval) can not be explained.  Examining the sessions (cloudsensor) log file shows CS Relay changing 0 (Safe) to 1 (Unsafe) at 19:24:22 and then alternated back and forth until is settled on 0 (Safe) at 19:25:32 where it remained in this state until 21:45:35. This shows that the CS Relay was 0 (Safe) during the forementioned 120s interval, which is opposite to what the Dome Relay Protection Monitor carried. Checking shows that CS Relay is using CloudSensorData.Relay which is calculated relay state based on comparison of clarity, rain and light values with thresholds in AstroMain program (which may differ from the thresholds held in the CloudSensor Head).  rather than the actual Relay value (RelayFlag) which is sent in the CloudSensor record recieved from AstroWeather runnning on the AllSky/Weather Computer.   Issue fixed in AstroMain 3.63.5 Update (2023-11-04 am),  ParseCloudSensorRecord() changed to determine changes in Relay based on the actual Cloud Sensor .RelayFlag rather than the calculated .Relay state.   CheckRelayProtection() routine changed to use CloudSensorData.RelayFlag .
  Fixed 2023-11-04 am (AstroMain 3.63.5 Update) 
   
• Using Restart CCDSoft' produced a fatal exception in recent modified CCDMonitor.MainLoop routine (2023-11-04 01:10). Whilst testing AstroMain 3.63.5 Update (2023-11-04 am) and after warnings that Foc1FieldUpdater and TempMonitor were frozen, and seeing that CCDSoft UI was 'Not Responding' the 'Restart CCDSoft' button was used in an attempt to Restart CCDSoft again and release the locked worker threads.  When this happened a unhandled exception was caught by Just-In-Time debugger which showed there was a System.NullReferenceException: at code line  CameraState = Camera.State in CcdMonitor.MainLoop.   Fixed in AstroMain 3.65.5 New Update by putting a Try, Catch, End Block around the code section. Considered to be fixed / resolved by changes to CCDMonitor and introduction on new CCDCamera object & thread.
  Resolved 2023-11-05  (AstroMain 3.64) 
   
  
• Cam2 JPG Pictures for last 2 targets show a noteable offset from Cam1. Although Cam2 image from first of these two targets seemingly linked ok, visual comparison of starfield in Cam1 / Locate frame and Cam2 frame puts a questions on the accuracy of the Cam2 image link. The Cam2 image from 2nd of these targets didn't link at all and the Cam2 offset from the previous target (which is shown to be incorrect) was used in creating the JPG picture.
  To investigate 2023-11-01 (Session, AstroMain) 

Continuous Improvement

• Extend AstroMain to archive the 'CloudSensor' records that have been recieved from AstroWeather via IP/TCP.  Saving the records to a data file would allow it to be later examined when trying to analyse, understand issues related to AstroMain's new Dome Relay Protection Monitor or Shutter_CanOpen, Shutter_MustClose discrepencies. New global variables setup :  CloudSensorFile, CloudSensorFileName & CloudSensorFileInUse. Implemented 2023-11-01 in AstroMain 3.63.5.  The resultant log files had proven somewhat useful, but on the downside the file produced gets very big  (194 MB in 24 hour period).  File contains a lot of non-useful data and an improvement would be to reduce the information written to file to just the important stuff.
  Done 2023-11-01 (AstroMain 3.63.5) 
   
• Extend AstroMain Pulsar Dome Monitor to read each and parse each new Volatile message since the last file read. Currently the routine which runs every 20s only picks up the very last (most recent) Volatile record and consequentially might miss out seeing changes to 'Relay' or 'Charging' information. This needs to be done in away that other routines that access Dome.Record aren't misled (thinking dome is not slewing (based on volatile message 15s old) when currently dome is slewing (based on latest record).   But this is prehaps no different from those using a DomeRecord that might be 15-19s old.  Implemented in AstroMain 3.63.5 and tested in observatory over a couple of days.
  Done 2023-11-01 (AstroMain 3.63.5) 
   
• Add a CCDCamera object to AstroMain that allows access to CCDSoft.Camera properties without having to directly access CCDSoft.Camera. The new class object (classCCDCamera) and the single instance (CCDCamera) contains read-only properties that return values of 'local' CcdMonitor variables that are updated from CCDSoft.Camera every few seconds by the CcdMonitor's main loop.  Aim of the new object is to allow various operations to access the most frequently CCDSoft.Camera properties without risk of freezing if CCDSoft.Camera freezes. This has the limitation that data is not necessarily up to date. Once completely finished and bugs ironed out AstroMain will be released as AstroMain 3.64. For time being the interim modifications are made in AstroMain 3.63.5.
  Done 2023-11-03 (AstroMain 3.63.5) 
   
• Modify AstroMain's RunFoc1FieldsUpdater routine to isolate it from direct calls to CCDSoft.Camera that can sometimes cause it to freeze.  RunFoc1FieldsUpdater is executed as a looped worker thread which is prone to freezing associated with calls to CCDSoft.Camera when it is in a non-responsive state.  Routine changed to use calls to new CCDCamera object properties to access locally stored copies of the CCDSoft.Camera properties where possible.  Where CCDSoft.Camera needs to be accessed the code has been moved to CCDMonitor . Implemented in AstroMain 3.63.5.
  Done 2023-11-03 (AstroMain 3.63.5) 
   
• Modify AstroMain's TempMonitor to isolate it from CCDSoft.Camera.  This is to stop issues where TempMonitor freezes with Obs.Overseer reports messages like "TempMonitor is frozen (00:51) in section 'Get CcdTemp Data' " (from 2023-11-04 00:51).
  Done 2023-11-04  (AstroMain 3.63.6)
   
• Extend Obs.Overseer oversight to include checks on the CCDMonitor process.  TSurprisingly CCDMonitor is not one of the worker/monitor threads that is not currently checked, but given the CCDMonitor increased role and the primary place where CCDSoft.Camera properties are accessed it is important to add it to the  process hreads that the  Obs.Overseer checks and reports on. "  . Added 5 new global & Observatory Manager variables :  CcdMonitorCycleTimePoint, CcdMonitorDesiredCycleSecs, CcdMonitorFrozenSecs, gCcdMonitorSection, bCcdMonitorCycleIsPending, bCcdMonitorReportedAsFrozen  and embedded them into CCDMonitor and ObsOverseer.
  Done 2023-11-04 (AstroMain 3.63.6) 
   
• Review DesiredCycleTime for CcdMonitor.   The CcdMonitor currently has a desired cycle time of 10s.  Given the recently widened role of the CCDMonitor as being the primary access to CCDSoft.Camera properties it is appropriate to review whether the existing cycle time (10s) is frequent enough.  Looking at other monitors that formerly directly accessed the CCDSoft.Camera., the TempMonitor had cycle time of SU.MonitorInterval * 60 (ie 120s). RunFoc1FieldsUpdater has a variable cycle time of  a) when triggered (normally every 2.5s) or b) at least every 10s.  Changed from 10s to 5s in AstroMain 3.63.6
  Done 2023-11-04 (AstroMain 3.63.6) 
   
• Explore new possibilities for CCDCamera & CcdMonitor.   Added CCDCamera.RefreshNow() routine that directs CcdMonitor to refresh local copies of Camera properties using CcdMonitor.bRefreshNow flag to trigger the refresh (flag=True) and to tell when operation is complete (flag=false). Added  Set CCDCamera.AutoSavePath  and Set CCDCamera.Status mediated by CcdMonitor.bSetProperty flag. SetAutoSavePath() routine modified to use new set CCDCamera.AutoSavePath property.
  Done 2023-11-04 (AstroMain 3.63.6) 
   
• Complete backlog of image building and web page documentation from Oct 2023.  2 sessions still incomplete as of 2023-11-13 (S1154-55).  Session Images completed 2023-11-16.
   Done  2023-11-16 (Session, Website)
   
• Extend AstroWeather to record the relay information from CloudSensor III. AstroWeather currently reads relay information from the single line file '../Cloud Sensor III/clarity_2.dat', extracts relay & status information and passes it onto AstroMain as part of a record.  The information is not currently preserved.  Saving the information to a composite file would allow it to be later examined when trying to analyse, understand issues related to AstroMain's new Dome Relay Protection Monitor.
  To do 2023-11-01 (AstroWeather 1.22.2+) 
  

[ Prev | Next ]

Back to Top

2023-11-02

Investigation - ObsManager freezing just after entering 'Session Running'

Back to Top