David's Astronomy Pages Notes - Session 1203 (2024-03-30)

Notes (S1202)

Notes Main

Home Page

Notes (S1204)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (2),  Minor Issues (9),  Small Defects (6),  Continuous Improvement (10)
	Images from 2024-03-30 >>
	Investigation - Target failed due to a 'RPC server is unavailable' exception
	2024-04-02
	Investigation - CcdCamera Loop freezing

Session Aims & Highlights (2024-03-30)

Main aims

1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
    ( 12P/Pons-Brooks is now an early evening object only and its was not a feasiblea im for the session due to expected early cloud)
2. AstroMain. Test new 3.67.2 version  of AstroMain. Verify there are no significant side-effects or issues with the new versions
3. Acquire images of Comet 12P/Pons-Brooks which has been undergoing regular outbursts.
4. DST Change.  Examine how well AstroMain program deals with the start of daylight saving time at 2023-03-31 01:00:00 UT,  when the local time will jumps forward by 1 hour from 01:00:00 to 02:00:00.

 Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• TheSky6, CCDSoft5, DeviceHub 6.6.1.1,  MeadeGeneric 1.3.9, UPB 1.5, ASCOM 6.6,   Pulsar Dome Firmware 1.51 & Driver 6.3.10
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.13, Siril 1.0.6, deCONZ 1.18.2, ASCOM 6.6 SP1
• [ DomeLink 1.0 / 1.1 - not used ]
   
• AstroMain 3.67.2,  AstroPlan 1.39.15,  AstroShCap 1.3.15, AstroGuard 1.10.3   (any new items or versions in bold)
  AstroVOE 1.13.1,  AstroSN 1.10.2,  AstroProtect 1.7,  AstroMove 1.12.11
  AstroWeather 1.24, AstroAllSky 3.20.3,  AstroObsCam 1.11.4, AstroLaunch 1.11.3

Highlights

• Targets. 12/26 planned targets observations were completed, with 1 one target partially completed.  Other targets failed due to 'missed time slot' associated with a DST Issue.
•  Dome Parking.  Dome parked at correct position at the end of the session.
• 12P/Pons-Brooks. Images of Comet 12P/Pons-Brooks were rejected due to cloud at the position of the comet (low in NW sky)
• DST Change. On the good side the AstroMain / AstroPlan programs didn't crash and produced appropriate charts & graphs but on the bad side JobExecutor inappropriately restarted CCDSoft at 02:00 as a TakeImage operation appeared to still be unfinished after 60 minutes, and the ObsOverseer also intervened at 02:00 believing that the ObsManager/ObsMonitor were frozen for 60 minutes and made the Observatory Safe. Subsequent targets all failed with the error 'Missed TimeSlot'.

Notes:

• Clocks went forward at 01:00 UT at start of daylight savings.

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Actual Weather vs Pre-Session Weather Forecast

Session Event Log

Time     Event Detail 19:18:53 Session Monitoring AutoStart monitoring for Live Session opportunity between 19:42 & 03:30 19:18:57   AutoStart Waiting AutoStart waiting till earliest start time at 19:42 19:42:15   AutoStart Delayed Session delayed due to too few stars 19:58:52 Session AutoStarting Session autostarting (19:58) 19:59:02   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.) 19:59:28   Camera1 Connected SBIG Camera connected (Set point -20°C) 19:59:30 Session Created Session Created (Live, 2024-03-30 S01203, ImageSaveNum: 1203001) 19:59:35   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 20:00:15   CcdCamera Frozen CcdCamera appears to be frozen at around 20:00 in section 'GetCameraProperties' 20:01:22   Services Started Observatory Services started 20:01:26   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera) 20:01:29 Observatory (Auto) Observatory placed in Fully-Automated Mode 20:01:31   AutoStart Delayed Session delayed due to too few stars 20:01:32 Session Pending Session pending (2024-03-30) 20:01:35   SBIG Camera Reset SBIG Camera being power-cycled 20:01:37 Session Initiating Session initiating (2024-03-30) 20:02:25   Camera1 Connected SBIG Camera connected (Set point -15°C) 20:02:30   Plan Requested Observing Plan requested from AstroPlan (1.39.15) 20:03:24   Plan Loaded Observing Plan loaded to queue (Plan ID: 978) 20:04:07   Telescope Connected Telescope connected (TheSky6) 20:04:31 Session Equilibration Session ready to Open Dome 20:05:16   Dome Opened Dome opened (Opening time 45s, Zigbee 40s) 20:05:31 Session Running Session running 20:05:33   Queue Started Observing Queue started (26 targets selected) 20:05:37     Target Started (NrZen) Target started (Focus Field 8, HIP 39357) 20:05:50       Dome Unparked Dome unparked 20:09:45       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 20:12:14       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide) 20:14:11       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 20:14:14       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 20:16:22       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 20:18:04       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 20:18:21     Target Completed (NrZen) Target completed (Focus Field 8, HIP 39357) 20:18:25   CcdCamera Frozen CcdCamera appears to be frozen at around 20:18 in section 'GetCameraProperties' 20:18:58     Target Started (1/26) Target started (1/26, 12P/Pons-Brooks) 20:19:03   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera) 20:19:10   SBIG Camera Reset SBIG Camera being power-cycled 20:20:02   Camera1 Connected SBIG Camera connected (Set point -15°C) 20:20:05     Target Failed (1/26) Target failed due to exception (1/26, 12P/Pons-Brooks) 20:34:48     Target Started (2/26) Target started (2/26, 13P/Olbers) 20:40:23       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S) 20:53:56     Target Completed Target completed (2/26, 13P/Olbers) 20:54:00     Target Started (3/27) Target started (3/27, 12P/Pons-Brooks, ToO) 21:02:01     Target Aborted (3/27) Target aborted (3/27, 12P/Pons-Brooks, ToO) due to User Intervention 21:02:05     Target Started (4/27) Target started (4/27, UGC 2964 w/AT2024epz) 21:17:29     Target Completed Target partially completed (4/27, UGC 2964 w/AT2024epz) 21:17:33     Target Started (5/27) Target started (5/27, 207P/NEAT) 21:20:44       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:22:53       Focusing Completed Foc1 AutoFocus Completed (Profile No 3) 21:39:45     Target Completed Target completed (5/27, 207P/NEAT) 21:39:49     Target Started (6/27) Target started (6/27, 29P/Schwassmann-Wachmann) 21:42:40       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:44:40       Focusing Completed Foc1 AutoFocus Completed (Profile No 4) 22:00:33     Target Completed Target completed (6/27, 29P/Schwassmann-Wachmann) 22:00:37     Target Started (7/27) Target started (7/27, NGC 3443 w/SN2024ehs) 22:03:37       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 22:19:48     Target Completed Target completed (7/27, NGC 3443 w/SN2024ehs) 22:21:32     Target Started (8/27) Target started (8/27, NGC 4192A w/AT2024exw) 22:24:29       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 22:40:27     Target Completed Target completed (8/27, NGC 4192A w/AT2024exw) 22:43:52     Target Started (9/27) Target started (9/27, 119P/Parker-Hartley) 22:46:44       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 22:49:05       Focusing Completed Foc1 AutoFocus Completed (Profile No 5) 23:01:05     Target Completed Target completed (9/27, 119P/Parker-Hartley) 23:01:10     Target Started (10/27) Target started (10/27, NGC 5036 w/SN2024dop) 23:04:15       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:06:33       Focusing Completed Foc1 AutoFocus Completed (Profile No 6) 23:23:15     Target Completed Target completed (10/27, NGC 5036 w/SN2024dop) 23:23:19     Target Started (11/27) Target started (11/27, IC 2190 w/SN2024dhp) 23:26:53       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:29:05       Focusing Completed Foc1 AutoFocus Completed (Profile No 7) 23:45:15     Target Completed Target completed (11/27, IC 2190 w/SN2024dhp) 23:45:20     Target Started (12/27) Target started (12/27, NGC 3206 w/SN2024bch) 23:48:31       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:50:49       Focusing Completed Foc1 AutoFocus Completed (Profile No 8) 00:06:49     Target Completed Target completed (12/27, NGC 3206 w/SN2024bch) 00:07:17     Target Started (13/27) Target started (13/27, NGC 4216 w/SN2024gy) 00:10:40       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 00:26:43     Target Completed Target completed (13/27, NGC 4216 w/SN2024gy) 00:29:36     Target Started (14/27) Target started (14/27, UGC 8630 w/SN2024drv) 00:32:18       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:34:56       Focusing Completed Foc1 AutoFocus Completed (Profile No 9) 00:50:56     Target Completed Target completed (14/27, UGC 8630 w/SN2024drv) 00:51:36     Target Started (15/27) Target started (15/27, AT2024crd) 00:54:30       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:56:45       Focusing Completed Foc1 AutoFocus Completed (Profile No 10) 00:59:55 Obs.Manager Obs.Manager is frozen in 'Sleep Period' (last cycle was at 00:59:55) 00:59:58 Obs.Monitor Frozen Obs.Monitor appears to be frozen at around 00:59 in section 'Sleep for Remaining Cycle' 02:00:00   SBIG Camera Take Image failed (Command timed-out) 02:00:02   CCDSoft Restarting CCDSoft being restarted (attempt to fix locked thread) 02:00:04   Make Observatory Safe Obs.Overseer intervened to Make Observatory Safe 02:00:05 Obs.Manager Obs.Manager has resumed again after 1.0 hour 02:00:07   SBIG Camera Reset SBIG Camera being power-cycled 02:00:50   Dome Closed Dome closed (Closing time 50s, Zigbee 44s) 02:00:59   Camera1 Connected SBIG Camera connected (Set point -25°C) 02:01:27   Dome Parked Dome moved to Az 90° (Park Position) 02:02:00   SBIG Camera Take Image command accepted (bSbigDriverException flag is reset) 02:05:18     Target Completed Target completed (15/27, AT2024crd) 02:23:12     Target Missed (16/27) Target's time slot was missed (16/27, NGC 3690 w/SN2023wrk) 02:39:31     Target Missed (17/27) Target's time slot was missed (17/27, GCVS U Her) 02:59:37     Target Missed (18/27) Target's time slot was missed (18/27, NGC 6745 w/SN2023ucy) 03:13:36     Target Missed (19/27) Target's time slot was missed (19/27, AT2024epa (CV)) 03:23:54     Target Missed (20/27) Target's time slot was missed (20/27, GCVS BL Lac) 03:44:14     Target Missed (21/27) Target's time slot was missed (21/27, NGC 6433 w/SN2024efn) 03:58:14     Target Missed (22/27) Target's time slot was missed (22/27, AT2023abtv (CV)) 04:10:34     Target Missed (23/27) Target's time slot was missed (23/27, AT2024dsj (CV)) 04:19:56     Target Missed (24/27) Target's time slot was missed (24/27, WDS GRB 34) 04:31:21     Target Missed (25/27) Target's time slot was missed (25/27, AT2023txn) 04:51:41     Target Missed (26/27) Target's time slot was missed (26/27, M101 w/SN2023ixf) 05:14:07     Target Missed (27/27) Target's time slot was missed (27/27, GCVS R CrB) 05:14:12   Queue Completed Job Queue completed 05:14:18 Session Closing Session closing 05:14:23   Dome Closed Dome already closed (closed at 02:00, closing time 50s, zigbee 44s) 05:14:24   Dome Parked Dome already parked 05:15:36   Telescope Parked Telescope parked (parking time 70s total) 05:15:51   Telescope State Handbox reads 'Scope parked Turn scope off.' 05:16:13   Telescope Switched Off Telescope Power has been switched off via UPB Switch. 05:17:38   Dome Charging Dome is parked and re-charging Ok (35mA) 05:17:40   Services Stopped Night Services stopped 05:17:43 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files) 05:18:05     Dome (Find Park) Find Park started (Search Az 87.0 to 93.0°, Step 0.2°, Narrow) 05:19:37     Dome Find Park found (Best Park Az 90.3°) 05:19:43     Dome Dome has been synced at park position (Az 90.0°, Adjustment: -0.3°) 05:19:45   Dome Parked Dome parked 05:22:49 Session Finished Session Finished

Session Alerts & Alarms

Time     Type       Name Detail 20:01:26 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 20:00:15) in section 'GetCameraProperties' 20:01:31 Green Clear CcdCamera CcdCamera has resumed again (20:01) after 1.3 mins 20:01:34 Yellow Alert Disk Space Free Space on C:\ Drive (39.45 GB) is less than 50 GB 20:19:03 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 20:18:25) in section 'GetCameraProperties' 20:19:08 Green Clear CcdCamera CcdCamera has resumed again (20:19) after 42s 02:00:00 Red Alert SBIG Camera Alert Take Image failed (Command timed-out) 02:00:02 Red Alert Obs.Manager Obs.Manager is frozen in 'Sleep Period' (last cycle at 00:59:55) 02:00:04 Red Alert Obs.Monitor Obs.Monitor appears to be frozen in section 'Sleep for Remaining Cycle' (last cycle at 00:59:58) 02:00:06 Yellow Alert Dome Dome has been stationary for more than 15 mins 02:00:08 Red Alert Dome Alert DomeRecords have stopped updating 02:00:10 Red Alert Dome Alert Shutter Data isn't being refreshed 02:00:12 Green Clear Obs.Manager Obs.Manager has resumed again after 1.0 hour 02:00:14 Green Clear Obs.Monitor Obs.Monitor has resumed again (02:00) after 60.0 mins 02:00:16 Yellow Alert Dome Shutter Pulsar Shutter appears to be asleep (last fresh data was at 00:57:50) 02:00:18 Green Clear Dome Shutter Pulsar Shutter has woken up (fresh data at 02:00:03, outage 62.3 min) 02:00:20 Green Clear Dome Alert DomeRecords have resumed updating 02:00:22 Green Clear Dome Alert Shutter Data has resumed updating 02:02:00 Green Clear SBIG Camera Take Image command accepted (bSbigDriverException flag is reset) 05:17:58 Yellow Alert Disk Space Free Space on C:\ Drive (37.86 GB) is less than 50 GB

Back to Top

Operational Issues (2024-03-30 S1203)

• Critical Issues (0)
• Major Issues (2)
• Minor Issues (9)
• Small Defects (6)
• Continuous Improvement (10)
  

[ Prev | Next ]

Critical Issues

• None

Major Issues

• After clock changed each Target in turn was judged to have missed its respective time-slot and was reported as failed (02:05 - 04:51).  Beginning with Target 16/27 (NGC 3690 ) and continuing Target 27/27 (R CrB) each target was failed with the reason 'Planned start time was missed' .   For example execution turned to Target NGC 3690 at 02:05, whose Planned Start and End Time were 02:08 and 02:31 respectively.  Since the time (02:05) was within the 5 minute window d before the scheduled start time, the target should have been immediately started, but it wasn't. Instead execution waited until the latest start time (02:23) and then proceeded to fail the target with the message 'Planed start time [02:08] was missed'.   Issue is  marked as a Major Issue as it prevented the observation/imaging for 12 targets which was a waste given the good sky conditions.   Initially this was thought to be due to an issue comparing times (Now(), StartTime, LatestStartTime etc) however investigation of the code eventually showed that the issue was occuring because the Shutter was closed (due to an inappropriate MakeObservatorySafe operation at 02:00) and there was no automated process to reverse the state and open the shutter again.  In this case a fix to the inappropriate MakeObservatorySafe operation (which was due to a DST issue) would have prevented the 'target failed due missed start'  issue from occuring on these 12 targets.  Ticket is closed after review 2024-04-01.
  Closed after review 2024-04-01 (AstroMain)   
   
• ObsOverseer miscalculated the time since the last ObsManager cycle directly after clocks changed at 01:00 UT and believing that ObsManager was frozen it ordered a 'MakeObservatorySafe' operation (02:00).  The operation closed the Shutter and had the consequence of causing remaining 12 targets to all fail due to missed starts.   TimeSince() routine isn't correctly calculating the time since last cycle time.  Originally raised as a Minor Issue it was re-ranked as a Major Issue because it led to the loss of several hours of good observing conditions.   ObsManagerCycleTimePoint is set to Now() at beginning of each cycle through ObsManager's Main Loop.
  ObsOverseer judges whether ObsManager is frozen or not by considering the magnitude of the SecsSinceLastObsManagerCycle. Existing function using following code doesn't work across changes in DayLight Savings.
      Return Now().Subtract(TimePoint).TotalSeconds
  Fixed in AstroMain 3.67.3, SecondsSince(TimePoint), and other similar functions (MinutesSince, HoursSince, DaysSince) modfied to correctly calculate TimeSpan in UTC time-space, using code like.
      Return DateTime.UtcNow.Subtract(TimePoint.ToUniversalTime).TotalSeconds
  Fixed 2024-04-01 (AstroMain 3.67.3)  

Minor Issues

• CcdCamera thread became frozen in section 'GetCameraProperties (20:00). ObsOverseer raised a CcdCamera alert at 21:06:26 with message "CcdCamera appears to be frozen (last cycle at 20:00:15) in section 'GetCameraProperties' " and restarted CCDSoft & SBIG Camera in response, which allowed CcdCamera thread to resume.   This occurred after Night Services had been started and the Restart overlapped with Session Pending/Session Initiating.   A wait loop is placed in AutoStartSession() routine in AstroMain 3.67.3 that waits for RestartCCDSoft operation to finish before calling SetMode_FullyAutomated(), so that report messages are no longer interlaced.  CCDCamera was re-factored and extended in AstroMain 3.67.3 in an attempt to reduce or prevent instances where the CCDCamera/CCDSoft Camera freezes.  Should the problem persist the he name of the property that was being read or set when CCDCamera froze is also now reported by AstroMain 3.67.3.
  Issue mitigated 2024-03-31 (AstroMain 3.67.3)
   
• CcdCamera thread became frozen in section 'GetCameraProperties (20:18). Some 18 minutes after first occurance of issue ObsOverseer raised a further CCDCamera alert at 20:19:03 with message "CcdCamera appears to be frozen (last cycle at 20:18:25) in section 'GetCameraProperties' "  and restarted CCDSoft & SBIG Camera in response and again this allowed the CcdCamera thread resumed.   This second incident occurred during the early (pre-slew) part of Target 1/26 (12P/Pons-Brooks), and likely led to an exception that caused execution to pass back to ExecuteQueue to be handled by an outer catch.   A potential reason for CcdCamera to lock-up around this time has been eliminated in AstroMain 3.67.3, however the root cause has still to be fully defined and fixed. CCDCamera was re-factored and extended in AstroMain 3.67.3 in an attempt to reduce or prevent instances where the CCDCamera/CCDSoft Camera freezes.  Should the problem persist the he name of the property that was being read or set when CCDCamera froze is also now reported by AstroMain 3.67.3.
  Partially fixed 2024-03-31 (AstroMain 3.67.3)
   
• Target 1/26 (12P/Pons-Brooks) failed due to a 'RPC server is unavailable' exception (20:19). As Target was going through pre-slew tasks the ObsOverseer raised a CcdCamera alert at 20:19:03 with message "CcdCamera appears to be frozen (last cycle at 20:18:25)" and restarted CCDSoft & SBIG Camera in response and again this allowed the CCDCamera thread resumed, when the Kill CCDSoft operation was performed at 20:19:04.26. This also seemed to have caused an exception to be thrown in Job Executor at 20:19:04.33  which wasn't handled within the TakeTarget () but lead to execution being thrown back to ExecuteQueue() routine where it was caught with message  "The RPC server is unavailable."   The issue was investigated (see Investigation - Target failed due to a 'RPC server is unavailable' exception) and concluded that the probably was caused by or highlighted by two unnecessary CCDSoft.Camera Binning assignments. Fixed in AstroMain 3.67.3,  the Camera.BinX/Y assignements are removed from Target pre- slew tasks, and a wait for RestartCCDSoft () wait loop is inserted.
  Fixed 2024-03-30 (AstroMain 3.67.3)
   
• Job Executor's Take Image operation had overrun by 60 minutes around the time for change in clock and ordered an inappropriate RestartCCDSoft operation (02:00).  TimeSince() routine isn't correctly calculating the actual time difference.  ObsOverseer also reported/logged apparents issue with other threads "Obs.Worker is Frozen (3602s)",.  "Obs.Monitor is Frozen (3602s)" at around the time that clocks went forward by 1 hour.    Fixed in AstroMain 3.67.3, SecondsSince(TimePoint), and other similar functions (MinutesSince(), HoursSince(), DaysSince() ), have been modfied to correctly calculate TimeSpan in UTC time-space.
  Fixed 2024-04-01 (AstroMain 3.67.3) 
   
• Several other Threads and Time Checks were reported to be 60 minutes overdue around the time for change in clock (02:00).  ObsOverseer also reported/logged apparents issue with other threads "Obs.Worker is Frozen (3602s)",.  "Obs.Monitor is Frozen (3602s)" at around the time that clocks went forward by 1 hour.   ObsManager reported and raised 'Dome Alert' with the message "DomeRecords have stopped updating".   Fixed in AstroMain 3.67.3, SecondsSince(TimePoint), and other similar functions (MinutesSince(), HoursSince(), DaysSince() ), have been modfied to correctly calculate TimeSpan in UTC time-space.
  Fixed 2024-04-01 (AstroMain 3.67.3) 
   
• Outdoor Temperature & Humidity reading were not available during course of S1202 & S1203 session (2024-03-30).  Outdoor Temperature & Humidity reading stopped refreshing in the afternoon of 2024-03-28 and didn't refresh throughout the course of S1202 session (2024-03-29) and S1203 session (2024-03-30). Investigating showed that the battery in Oregon Scientific Outdoor T/H  Sensor was flat, and showing no values on teh Sensors LCD screen.  Batteries replaced 2024-03-31 and Sensor reconnected to the Base Station and to AllSky/Weather Computer.  
  Fixed 2024-03-31 (Weather Station) 
   
• Observing Plan Chart is missing 3 targets that had planned start time between 02:00 and 03:00 LT. First 3 targets after clock went forward by one hour at 01:00 UT (NGC 3690 w/SN2023wrk, GCVS U Her & NGC 6745 w/SN2023ucy) all with start times between 02:00 and 03:00 LT, aren't being drawn on the Observing Plan Chart. They are drawn ok on the Altitude Plan however.  Bug identified in PlanChart.DrawChart, a Planned Target was being ignored in Chart when Night.DstChange = +1 and .StartTime.Hour = 1.  This should only apply when Night.DstChange = -1. Fixed in AstroPlan 1.40.
  Fixed 2024-04-01 (AstroPlan 1.40)  
   
• Camera.AutoSavePath wasn't set correctly for new Session or otherwise became reset. After taking some test Flat frames in General Session 2024-04-04 using AstroMain 3.67.3 it was found that had been saved to .\New/2024-03-30 folder instead of the expected ..\New\2024-04-04 folder.   Examining the report file suggests AutoSave Folder was set to .\New\2024-04-04 as part of RestartCCDSoft to set AutoSave No. operation at 09:00.  During a remedial RestartCCDSoft operation at 09:03 the AutoSave Folder was reported as being '.\New\2024-04-04'.  CCDSoft Camera's /Autosave tab after taking the Flat Frames shows AutoSave folder as being '.\New\2024-03-30' which explains why frames went to 'wrong' folder but not why the Autosave Path was wrong. Log file shows Get AutoSavePath timing out after 2s at 09:00:22 & at 09:00:28, with both failures occuring before the Camera was confirmed to be connected at 09:00:40.56. (Camera.Connect took 8.7s).  A  Set AutoSavePath operation at 09:00:42 appears to have been successful taking 0.050s.
  Fixed 2024-04-04 (AstroMain 3.67.4)
   
• CcdCamera thread became frozen in section 'GetCameraProperties/ Cooler Power (2024-04-04). Whilst using new AstroMain 3.67.3 version (with refactored and extended CCDCamera Handling), ObsOverseer raised a CcdCamera alert at 09:03:47 with the message "CcdCamera appears to be frozen (last cycle at 09:03:07) in section 'GetCameraProperties / CoolerPower'".  This occured about 3 minutes after a new General Session was created at 09:00 which include a RestartCCDSoft()  operation in order to 'set AutoSave No.', not including a SBIG Restart after which the Camera was reported as reconnected at 09:00:44.  ObsPics show fresh Camera data coming through and confirm a working CCDCamera loop to at least 09:03:00.  In response to the frozen CCDCamera loop, ObsOverseer commanded a RestartCCDSoft operation at 09:03:49 in 'an attempt to fix frozen CcdCamera', which included a power recycle of the SBIG Camera. CCDCamera was reported to have resumed again at 09:03:52.  It is unclear whether the reporting of CCDCamera being frozen in the Get  Property / CoolorPower section (presumably at the code line   _CoolerPower = Camea.ThermalElectricCoolerPower ) is significant or not.   To be monitored to understand frequency and location of continued Camera freezes.  Issue closed 2024-04-15 as no further cases have been enountered in the following 5 sessions (S1204-S1208), suggesting that the refactoring and extension of CCDCamera object and its exclusive use in accessing the CCDSoft.Camera has been a success.
  Closed 2024-04-15 (AstroMain 3.68)
   
• AstroObsCam 1.11.5 is crashing shortly after Connect Camera is clicked (2024-04-01).  AstroObsCam in 1.11.5 version upgraded to Net Framework 4.8 and includes fixes to SecondsSince(), MinutesSince() to calculate timespan in UTC Time-Space, however when installed and tried on AllSky/Weather Computer the program would crash shortly after clicking on Connect Camera. Issue was repeatable and would also occur even when SecondsSince(), MinutesSince routines were put back to their former code.
  Reinstalling and running 1.11.4 version works ok (so it is not a intrinsic problem with the WebCam).  Eventually found that a 'System.InvalidOperationException' with the message "Bitmap region is already locked" is happening in HandleFrameSaved() routine at the following code line.
    g.DrawImage(capImage, 0, 0, newWidth, newHeight)
  The same exception happens on both the AllSky//Weather Computer and the Development Computer. The program uses a .DLL 'DirectShowLib-2005.dll' which isn't touched by the project upgrade to net framework 4.8.  The might be some incompatability or problem.  It is proposed to recreate the 1.11.5 version by going back to 1.11.4 version and reopening the project in Visual Studio.  This was done but the same exception occurs.  Whilst problem is exposed when using the Net Framework 4.8 it is problem due to the same underlying issue seen in earlier program versions where capture image can repeatedly lock up for 6-7s at a time.  Until issue is resolved the AllSky/Weather Computer will be run with the original 1.11.4 version.  Using    'capImage = Handle_capImage.Clone'  instead of simply 'capImage = Handle_capImage'  didn't solve the issue, nor did adding  .Clone to 'Handle_capImage = capImage' in cam_FrameSaved. Whilst the issue might potentially be resolved by installing the unsupported Net Framework 4 onto the computer this would resolve the underlying defect in the program.  A long outstanding improvement has to be 'map' out exactly how the program collects an image from the webcam and to handle it / processes it to arrive at the final image. The task is probably a necessary step towards solving the current issue as well as the issue with corrupted images/ unside down text seen most recently in session S1200 (2024-03-21, S1200).
  To fix 2024-04-01 (AstroObsCam 1.11.4/ 1.11.5) 

Small Defects

• AstroMain partially mis-reported the night that DST change was due (05:22). At end of the S1203 session (night 2024-03-30 / 2024-03-31) and as the session changed from Live to General at 2024-03-31 05:22, there were a "em>Clock Changes Tonight!" report line with the message "Warning   Clock goes forward 1 hour at 01:00".  By this time the Clock Change had already occurred !.  The report line should have been given at the end of the previous night,  session S1202 (night 2024-03-29 / 2024-03-30).  The warning was however given correctly during 'Running Autostart' (logged in 2024-03-30 General Report File) at 2024-03-30 19:58,
      Clock Changes Tonight!        Warning   Clock goes forward 1 hour at 01:00  
    Force Suspension Start/End    Info      Option is diabled (previous code forced session to suspend between 00:50 and 01:10 UT )
  The message should also be given at the start of the Live Session where it is more readily seen.
  It is noted tht a "> DST Change     Info      Clock is changing shortly" report line was given 15 mins before clocks were due to change (00:45)
  Fixed 2024-04-01 (AstroMain 3.67.3)  
   
• Target Name on ObsPic not changing to new target when target is finished. & Target 15/27 (AT2024crd) finished at 02:05,  and even though job execution moved on to 12 subsequent targets (which all failed due to missed start) the target name displayed on ObsPics remained set to AT2024crd. Whilst scope is tracking and now slewed the scope will still be pointing at AT2024crd, but certainly once tracking is turned off the scope is ceratinly not longer pointing to AT2024crd. A similar situation has been seen in other sessions whilst session is suspended and job queue considers each target in turn as the clock ticks over. Fixed in AstroMain 3.67.3.  label changed from TP.Scope.TargetName to szTargetName. This 'fix' would make the label tick over to next name of the current Target, but may have unintended side consequences / misinformation.
  Fixed attempted 2024-03-31+ (AstroMain 3.67.3+)  
   
• Logs contain a multitude of  'Get CCDCamera property timed out after 2s' messages  after CCDCamera frozen (2024-04-04 09:03). After AstroMain CCDCamera loop because frozen at around 09:03:07, log file contains a multitude of  '(trace') log messages  from 09:03:09 onwards reporting various Get CCDCamera  properties as having timed out after 2s (eg, "Get CoolerPower timed out after 2s", "Get Status timed out after 2s"). Similar messages continued until 09:03:48, which is around the time that the ObsOverseer recognised that the CCDCamera loop was frozen and a RestartCCDSoft operation was conducted which eventually unlocked the CCDCamera loop.  These messages would be expected to happen, but the recognition of the potentially frozen CCD loop (raised bCcdCameraCycleIsPending flag) and should be used by the Camera1_IsAccessible to stop unnecessary extra Get Property requests at this time and thus prevent an overly long sequence of these log messages.  Implemented in AstroMain 3.67.4, bCcdCameraCycleIsPending & bCcdCameraReportedAsFrozen added as conditions within Camera1_IsAccessible() routine. Thresholds used by ObsOverseer simplifed also; pending flag now raised 10s after CcdCameraCycleTimePoint, frozen flag now raised 30s after LastCcdCameraCycleTimePoint.   CcdCameraDesiredCycleSecs is removed as no longer needed as CCD Camera now operates on a continuous loop.
  Fixed 2024-04-04 (AstroMain 3.67.4)
   
• Logs contain a number of log messages reporting exception in CCDCamera Get Property, during RestartCCDsoft operation (2024-04-04 09:03). After CCDCamera loop froze at 09:03:07, ObsOverseer commanded a RestartCCDSoft operation to begin at  09:04:39 which allow the CCDCamera Loop to resume after killing CCDSoft, which led to a  "The remote procedure call failed" exception from CCDCamera.GetProperty CoolerPower , which was the point that the CCDCamera loop was frozen.  This is exception, but the following 6 "Object variable or With block variable not set" in following CCDCamera.GetProperty for various property came from code that shouldn't have been executed as Camera was yet ready,  these are due to 'bGet_property flags still being raised. As part of RestartCCDSoft operation, all existing bSet, bCall and bGet flags should be reset.  Implemented in AstroMain 3.67.4 using a new CCDCamera.ResetAllRequestFlags()  routine, which when starting CCDCamera loop and when running RestartCCDSoft.
  Fixed 2024-04-04 (AstroMain 3.67.4)
   
• Logs using szTargetName "Dark" hours after 'Take Dark' operation was conducted the previous day (2024-04-03). During a final 'Take Darks' operation as part of a number of Camera tests were conducted 2024-04-04 using AstroMain 3.67.3 (to check on CcdCamera's new refactoring & extension), szTargetName was set to 'DARK'.  The variable remained unchanged since then and all log records since then have DARK' written to the S1 field, including log records written to the log file of the next day's General Session. (2024-04-04).  It would be expected that szTargetName be reset to 'Manual' either at the end of 'Take Darks' sequence or at least at the start of the next day's general session. Fixed in AstroMain 3.67.4.
  Fixed 2024-04-04 (AstroMain 3.67.4)
   
• Calls to Get CCDCamera.Bin are regularly timing out after the standard 2s (2024-04-04).  Whilst testing new code in AstoMain 3.67.3 & 3.67.4 Get operations for CCDCamera.Bin, CCDCamera.BinX and sometimes CCDCamera.FilterIndexZeroBased are noticed to be returning timed out after 2s messages.   Problem was CCDCamera.Bin was identified to be bug, which was fixed in AstroMain 3.67.4 update. No defect found to explain CCDCamera.BinX & CCDCamera.FilterIndexZeroBased cases, it is presumed there timeouts were due to holdup associated with the CCDCamera.Bin issue ?
  Believed Fixed 2024-04-04 (AstroMain 3.67.4)

Continuous Improvement

• Make a number of small tweaks to AstroMain's Session Logging.  Make various small tweaks,  minor corrections & small improvement  to lines & messages written to Log file.  Implemented in AstroMain 3.67.3.
  Done  2024-03-31 (AstroMain 3.67.3)
   
• Place AstroMain's btnPauseJob under VM control. Button text and backcolor are set by VM.JobPauseState = PauseState(integer), with Enum 'PauseState' (with values NotPaused, Pausing & Paused) .  Two new VM routines  VM.HidePauseJobButton and VM.ShowPauseJobButton are now used to set btnPauseJob's visibility. Implemented in AstroMain 3.67.3.
  Done  2024-04-01 (AstroMain 3.67.3)
   
• Create new 'WaitForEndPause()' routine that is called when Job Queue or Task encounters a raised jobPause flag.  New WaitForEndPause() routine created that is normally called via the code
    If jobPause Then
    WaitForEndPause(ThisRoutine())
  End If
  The routine changes jobPaused to true, updates the Pause Job button and enters a wait loop, that waits for jobPause flag to be lowered. The wait loop calls ExtendIamAliveTime() every 0.5s to extend IamAliveData and thereby prevent ObsOverseer from believing that a job is frozen.   Implemented in AstroMain 3.67.3.  Despite the addition of the ExtendIamAliveTime the routine nevetheless produced unwelcome side-efffect in S1204 session (2024-04-04) that gave "task ... is taking a long time" warnings from ObsOverseer on 4 occasions (1 whilst waiting for CCD cooler stabilisation and 3 whilst suspending the session).
  Done  2024-04-01 (AstroMain 3.67.3)
   
• Add new check for jobPause/WaitForEndPause code sections to key points in Job Queue and other Tasks.  This improvement is intended to i) provide the capability for user to pause the Job Queue in order to conduct a manually initiated task or make a particular settings change, and ii) provide a means for ObsManager/ObsOverseer to pause an executing Job Queue, in order to conduct a particular task or operation without risk of interlaced report messages. Implemented in AstroMain 3.67.3.
  Done  2024-04-01 (AstroMain 3.67.3)
   
• Apply fix to SecondsSince() & similar functions in AstroPlan, AstroWeather, AstroAllSky programs.  Following issues with the SecondsSince() and MinutesSince() functions in AstroMain at the point that Clocks went forward at 01:00 UT on 2024-03-31 the functions were modified to provide the correct timespan time differences. These fixes need to be applied across the entire AstroSuite.  Implemented in AstroPlan 1.40, AstroGuard 1.10.4, AstroVOE 1.13.2, AstroWeather 1.24.1, AstroSN 1.10.3, AstroAllSky 3.20.4, AstroMove 1.12.12 (2024-04-01). (The same modifications were also made to AstroObsCam 1.11.5, but could it couldn't be used as the project upgrade to use Net Framework 4.8 had a showstopping side-effect whereby an exception began occurring in the HandleFrameSaved() routine.
  Done  2024-04-01 (AstroSuite Programs)
   
• Refactor & extend the properties & methods handled by CCDCamera.  Refactor CCDCamera object to get & set Camera properties & make Camera Method calls on used a slightly revised method and extend the number of properties & methods handled. Being implemented in AstroMain 3.67.3.   Need to check Connect() can be made when Camera is not connected.   Need to check that certain calls (e.g Abort) can be made when Camera1_IsAccessible=True.  Note that properties associated with focusing and autoguiding are not presently included in the CCDCamera, as these CCDSoft/SBIg Camera properties are essentially not current used in AstroMain, though are still present in some legacy code within AstroMain.
  Done    2024-04-03 (AstroMain 3.67.3 / AstroMain 3.68)
   
• Check CCDCamera Implementation.  Made changes in AstroMain i)  CCDCamera.Connect  calls Camera.Connect directly if gCameraConnected=False otherwise uses indirect call. ii) Similarly CCDCamera.Disconnect calls Camera.Disonnect directly if gCameraConnected=False iii) KillCCDSoft automatically set gCameraConnected = False , although in practice it should already have been set to False within RestartCCDSoft. iv) Added gCcdCameraSubSection to record the code point reached by CCDCamera loop, iv) Added reporting of gCcdCameraSubSection by ObsOverseer.
  [ Note that we depend on TakeImage being used with Camera.Asynchronous = 1 otherwise will we get false Frozen Camera events  ]
  Done   2024-04-03 (AstroMain 3.67.3 / AstroMain 3.68)
   
• Add a new Log Line / Log Section when the Job Queue is finished. Whilst AstroMain's Report File for a live session shows a new Report Section that is titled 'Finish Queue' when a Job Queue finishes, the Log file doesn't reflect this but simply puts a short line break and continues to log messages with the szTargetName of the final target.  Implemented in AstroMain 3.67.4.
  Done 2024-04-04 (AstroMain 3.67.4)
   
• Make further tweaks to AstroMain's Session Logging.  Make various small tweaks,  minor corrections & small improvement  to lines & messages written to Log file.  Implemented in AstroMain 3.68.
  Done 2024-04-04 (AstroMain 3.67.4)
   
• Check for the potential that TakeImage operation could commence before all Set Camera Properties have taken effect.  In the case of the following sequence of operations
    CCDCamera.ExposureTime = 10.0 
  CCDCamera.Bin = 3
  CCDCamera.TakeImage
  is there a risk depending on where the CCDCamera Loop is that the following would happen
   Camera.BinX not updated (as code passed before the call to CCDCameraBin=3 was made)
   Camera.BinY not updated (as code passed before the call to CCDCameraBin=3 was made)
   Camera.Exposure = 10.0
   Camera.TakeImage()
   Camera.BinX = 3  (finally executed on next cycle through CCDCamera loop)
   Camera.BinY = 3  (finally xecuted on next cycle through CCDCamera loop)
  Normally the calls to set a CCDCamera property (like CCDCamera.Bin)  wait until the operation has been achieved or times out after 2s, so in theory the risk shouldn't arise in practice, however if the loop is slow for some reason, the operation (BinX, BinY assignments) might not been performed before CCDCamera.Bin returned after time out and the the CCDCamera.TakeImage() was called, allowing it to potentially execute for BinX/BinY assignements are made.
  Should the calling Thread wait an extra 1s or so before committing the CCDCamera.TakeImage.  Is there a way of checking that all prior settings had been all been completed before calling CCDCamera.TakeImage() ?    Implemented a WaitForFreshLoop() routine in AstroMain 3.67.4
  Done 2024-04-04 (AstroMain 3.67.4)

[ Prev | Next ]

Back to Top

Investigation - Target failed due to a 'RPC server is unavailable' exception

Back to Top

2024-04-02

Investigation - CcdCamera thread become frozen

Back to Top