David's Astronomy Pages
Notes - Session 1203 (2024-03-30)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (2),  Minor Issues (9),  Small Defects (6),  Continuous Improvement (10)
Bullet Images from 2024-03-30 >>         
   
Bullet Investigation - Target failed due to a 'RPC server is unavailable' exception
   
2024-04-02
Bullet Investigation - CcdCamera Loop freezing
   
   

Session Aims & Highlights (2024-03-30)

Main aims

  1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
     ( 12P/Pons-Brooks is now an early evening object only and its was not a feasiblea im for the session due to expected early cloud)
  2. AstroMain. Test new 3.67.2 version  of AstroMain. Verify there are no significant side-effects or issues with the new versions
  3. Acquire images of Comet 12P/Pons-Brooks which has been undergoing regular outbursts.
  4. DST Change.  Examine how well AstroMain program deals with the start of daylight saving time at 2023-03-31 01:00:00 UT,  when the local time will jumps forward by 1 hour from 01:00:00 to 02:00:00.

 Equipment & Software

Highlights

Notes:

 

Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Image
 
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Image
  
Sky Conditions (Locate Frames)
Image 
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image
  
Session Event Log
Time     Event Detail
19:18:53 Session Monitoring AutoStart monitoring for Live Session opportunity between 19:42 & 03:30
19:18:57   AutoStart Waiting AutoStart waiting till earliest start time at 19:42
19:42:15   AutoStart Delayed Session delayed due to too few stars
19:58:52 Session AutoStarting Session autostarting (19:58)
19:59:02   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.)
19:59:28   Camera1 Connected SBIG Camera connected (Set point -20°C)
19:59:30 Session Created Session Created (Live, 2024-03-30 S01203, ImageSaveNum: 1203001)
19:59:35   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
20:00:15   CcdCamera Frozen CcdCamera appears to be frozen at around 20:00 in section 'GetCameraProperties'
20:01:22   Services Started Observatory Services started
20:01:26   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera)
20:01:29 Observatory (Auto) Observatory placed in Fully-Automated Mode
20:01:31   AutoStart Delayed Session delayed due to too few stars
20:01:32 Session Pending Session pending (2024-03-30)
20:01:35   SBIG Camera Reset SBIG Camera being power-cycled
20:01:37 Session Initiating Session initiating (2024-03-30)
20:02:25   Camera1 Connected SBIG Camera connected (Set point -15°C)
20:02:30   Plan Requested Observing Plan requested from AstroPlan (1.39.15)
20:03:24   Plan Loaded Observing Plan loaded to queue (Plan ID: 978)
20:04:07   Telescope Connected Telescope connected (TheSky6)
20:04:31 Session Equilibration Session ready to Open Dome
20:05:16   Dome Opened Dome opened (Opening time 45s, Zigbee 40s)
20:05:31 Session Running Session running
20:05:33   Queue Started Observing Queue started (26 targets selected)
20:05:37     Target Started (NrZen) Target started (Focus Field 8, HIP 39357)
20:05:50       Dome Unparked Dome unparked
20:09:45       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:12:14       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
20:14:11       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
20:14:14       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
20:16:22       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
20:18:04       Focusing Completed Foc2 AutoFocus Completed (Profile No 2)
20:18:21     Target Completed (NrZen) Target completed (Focus Field 8, HIP 39357)
20:18:25   CcdCamera Frozen CcdCamera appears to be frozen at around 20:18 in section 'GetCameraProperties'
20:18:58     Target Started (1/26) Target started (1/26, 12P/Pons-Brooks)
20:19:03   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera)
20:19:10   SBIG Camera Reset SBIG Camera being power-cycled
20:20:02   Camera1 Connected SBIG Camera connected (Set point -15°C)
20:20:05     Target Failed (1/26) Target failed due to exception (1/26, 12P/Pons-Brooks)
20:34:48     Target Started (2/26) Target started (2/26, 13P/Olbers)
20:40:23       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
20:53:56     Target Completed Target completed (2/26, 13P/Olbers)
20:54:00     Target Started (3/27) Target started (3/27, 12P/Pons-Brooks, ToO)
21:02:01     Target Aborted (3/27) Target aborted (3/27, 12P/Pons-Brooks, ToO) due to User Intervention
21:02:05     Target Started (4/27) Target started (4/27, UGC 2964 w/AT2024epz)
21:17:29     Target Completed Target partially completed (4/27, UGC 2964 w/AT2024epz)
21:17:33     Target Started (5/27) Target started (5/27, 207P/NEAT)
21:20:44       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:22:53       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
21:39:45     Target Completed Target completed (5/27, 207P/NEAT)
21:39:49     Target Started (6/27) Target started (6/27, 29P/Schwassmann-Wachmann)
21:42:40       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:44:40       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
22:00:33     Target Completed Target completed (6/27, 29P/Schwassmann-Wachmann)
22:00:37     Target Started (7/27) Target started (7/27, NGC 3443 w/SN2024ehs)
22:03:37       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
22:19:48     Target Completed Target completed (7/27, NGC 3443 w/SN2024ehs)
22:21:32     Target Started (8/27) Target started (8/27, NGC 4192A w/AT2024exw)
22:24:29       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S)
22:40:27     Target Completed Target completed (8/27, NGC 4192A w/AT2024exw)
22:43:52     Target Started (9/27) Target started (9/27, 119P/Parker-Hartley)
22:46:44       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
22:49:05       Focusing Completed Foc1 AutoFocus Completed (Profile No 5)
23:01:05     Target Completed Target completed (9/27, 119P/Parker-Hartley)
23:01:10     Target Started (10/27) Target started (10/27, NGC 5036 w/SN2024dop)
23:04:15       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:06:33       Focusing Completed Foc1 AutoFocus Completed (Profile No 6)
23:23:15     Target Completed Target completed (10/27, NGC 5036 w/SN2024dop)
23:23:19     Target Started (11/27) Target started (11/27, IC 2190 w/SN2024dhp)
23:26:53       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:29:05       Focusing Completed Foc1 AutoFocus Completed (Profile No 7)
23:45:15     Target Completed Target completed (11/27, IC 2190 w/SN2024dhp)
23:45:20     Target Started (12/27) Target started (12/27, NGC 3206 w/SN2024bch)
23:48:31       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:50:49       Focusing Completed Foc1 AutoFocus Completed (Profile No 8)
00:06:49     Target Completed Target completed (12/27, NGC 3206 w/SN2024bch)
00:07:17     Target Started (13/27) Target started (13/27, NGC 4216 w/SN2024gy)
00:10:40       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
00:26:43     Target Completed Target completed (13/27, NGC 4216 w/SN2024gy)
00:29:36     Target Started (14/27) Target started (14/27, UGC 8630 w/SN2024drv)
00:32:18       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:34:56       Focusing Completed Foc1 AutoFocus Completed (Profile No 9)
00:50:56     Target Completed Target completed (14/27, UGC 8630 w/SN2024drv)
00:51:36     Target Started (15/27) Target started (15/27, AT2024crd)
00:54:30       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:56:45       Focusing Completed Foc1 AutoFocus Completed (Profile No 10)
00:59:55 Obs.Manager Obs.Manager is frozen in 'Sleep Period' (last cycle was at 00:59:55)
00:59:58 Obs.Monitor Frozen Obs.Monitor appears to be frozen at around 00:59 in section 'Sleep for Remaining Cycle'
02:00:00   SBIG Camera Take Image failed (Command timed-out)
02:00:02   CCDSoft Restarting CCDSoft being restarted (attempt to fix locked thread)
02:00:04   Make Observatory Safe Obs.Overseer intervened to Make Observatory Safe
02:00:05 Obs.Manager Obs.Manager has resumed again after 1.0 hour
02:00:07   SBIG Camera Reset SBIG Camera being power-cycled
02:00:50   Dome Closed Dome closed (Closing time 50s, Zigbee 44s)
02:00:59   Camera1 Connected SBIG Camera connected (Set point -25°C)
02:01:27   Dome Parked Dome moved to Az 90° (Park Position)
02:02:00   SBIG Camera Take Image command accepted (bSbigDriverException flag is reset)
02:05:18     Target Completed Target completed (15/27, AT2024crd)
02:23:12     Target Missed (16/27) Target's time slot was missed (16/27, NGC 3690 w/SN2023wrk)
02:39:31     Target Missed (17/27) Target's time slot was missed (17/27, GCVS U Her)
02:59:37     Target Missed (18/27) Target's time slot was missed (18/27, NGC 6745 w/SN2023ucy)
03:13:36     Target Missed (19/27) Target's time slot was missed (19/27, AT2024epa (CV))
03:23:54     Target Missed (20/27) Target's time slot was missed (20/27, GCVS BL Lac)
03:44:14     Target Missed (21/27) Target's time slot was missed (21/27, NGC 6433 w/SN2024efn)
03:58:14     Target Missed (22/27) Target's time slot was missed (22/27, AT2023abtv (CV))
04:10:34     Target Missed (23/27) Target's time slot was missed (23/27, AT2024dsj (CV))
04:19:56     Target Missed (24/27) Target's time slot was missed (24/27, WDS GRB 34)
04:31:21     Target Missed (25/27) Target's time slot was missed (25/27, AT2023txn)
04:51:41     Target Missed (26/27) Target's time slot was missed (26/27, M101 w/SN2023ixf)
05:14:07     Target Missed (27/27) Target's time slot was missed (27/27, GCVS R CrB)
05:14:12   Queue Completed Job Queue completed
05:14:18 Session Closing Session closing
05:14:23   Dome Closed Dome already closed (closed at 02:00, closing time 50s, zigbee 44s)
05:14:24   Dome Parked Dome already parked
05:15:36   Telescope Parked Telescope parked (parking time 70s total)
05:15:51   Telescope State Handbox reads 'Scope parked Turn scope off.'
05:16:13   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
05:17:38   Dome Charging Dome is parked and re-charging Ok (35mA)
05:17:40   Services Stopped Night Services stopped
05:17:43 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files)
05:18:05     Dome (Find Park) Find Park started (Search Az 87.0 to 93.0°, Step 0.2°, Narrow)
05:19:37     Dome Find Park found (Best Park Az 90.3°)
05:19:43     Dome Dome has been synced at park position (Az 90.0°, Adjustment: -0.3°)
05:19:45   Dome Parked Dome parked
05:22:49 Session Finished Session Finished

 
Session Alerts & Alarms
Time     Type       Name Detail
20:01:26 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 20:00:15) in section 'GetCameraProperties'
20:01:31 Green Clear CcdCamera CcdCamera has resumed again (20:01) after 1.3 mins
20:01:34 Yellow Alert Disk Space Free Space on C:\ Drive (39.45 GB) is less than 50 GB
20:19:03 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 20:18:25) in section 'GetCameraProperties'
20:19:08 Green Clear CcdCamera CcdCamera has resumed again (20:19) after 42s
02:00:00 Red Alert SBIG Camera Alert Take Image failed (Command timed-out)
02:00:02 Red Alert Obs.Manager Obs.Manager is frozen in 'Sleep Period' (last cycle at 00:59:55)
02:00:04 Red Alert Obs.Monitor Obs.Monitor appears to be frozen in section 'Sleep for Remaining Cycle' (last cycle at 00:59:58)
02:00:06 Yellow Alert Dome Dome has been stationary for more than 15 mins
02:00:08 Red Alert Dome Alert DomeRecords have stopped updating
02:00:10 Red Alert Dome Alert Shutter Data isn't being refreshed
02:00:12 Green Clear Obs.Manager Obs.Manager has resumed again after 1.0 hour
02:00:14 Green Clear Obs.Monitor Obs.Monitor has resumed again (02:00) after 60.0 mins
02:00:16 Yellow Alert Dome Shutter Pulsar Shutter appears to be asleep (last fresh data was at 00:57:50)
02:00:18 Green Clear Dome Shutter Pulsar Shutter has woken up (fresh data at 02:00:03, outage 62.3 min)
02:00:20 Green Clear Dome Alert DomeRecords have resumed updating
02:00:22 Green Clear Dome Alert Shutter Data has resumed updating
02:02:00 Green Clear SBIG Camera Take Image command accepted (bSbigDriverException flag is reset)
05:17:58 Yellow Alert Disk Space Free Space on C:\ Drive (37.86 GB) is less than 50 GB
 

Back to Top


Operational Issues (2024-03-30 S1203)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Small Defects

Continuous Improvement

[ Prev | Next ]

Back to Top


Investigation - Target failed due to a 'RPC server is unavailable' exception

Issue

1) Target 1/26 (12P/Pons-Brooks) failed due to a 'RPC server is unavailable' exception (20:19)
 
2) Ideally an exception when starting a Target shouldn't necessarily lead to the automatic failure of the target

Description

As Target 1/26 (12P/Pons-Brooks) was going through pre-slew tasks the ObsOverseer raised a CcdCamera alert at 20:19:03 with message "CcdCamera appears to be frozen (last cycle at 20:18:25)" and restarted CCDSoft & SBIG Camera in response and again this allowed the CCDCamera thread resumed, when the Kill CCDSoft operation was performed at 20:19:04.26. This also seemed to have caused an exception to be thrown in Job Executor at 20:19:04.33  which wasn't handled within the TakeTarget () but lead to execution being thrown back to ExecuteQueue() routine where it was caught with message  "The RPC server is unavailable."  
The target was  reported as 'Target Failed  - Target failed due to exception" .

Analysis

Conclusion

Actions:

Back to Top


2024-04-02


Investigation - CcdCamera thread become frozen

Issue

1) CcdCamera thread became frozen in section 'GetCameraProperties (20:00)
 
2) CcdCamera thread became frozen in section 'GetCameraProperties (20:18).

Description

Following original issues with CCDSoft Camera / SBIG Camera freezing causing various AstroMain threads to correspondingly freeze such as ObsManager, ObsMonitor, JobQueue, a new 'CcdCamera' thread was added through which as many CCDSoft Camera communications could be made, to miminise chance of conflicts  and to protect the other threads.   This is partly but not fully successful.  When the Camera freezes Job Executor and ObsOverseer automatically intervene to call a RestartCCDSoft routine that kills CCDSoft and restarts the SBIG Camera, before reconnecting again. This is restart operation is generally successful but delays the operation and is clearly not the ideal situation which would be where a freezing of CCDSoft Camera either never or only very rarely occurs.

A more detailed analysis of the two incidents  from the S1203 session and indeed other previous cases is made below to see if the underlying cause of the problem can be identified and a fix applied to prevent it happening. 

1) ObsOverseer raised a CcdCamera alert at 21:06:26 with message "CcdCamera appears to be frozen (last cycle at 20:00:15)" and restarted CCDSoft & SBIG Camera in response, which allowed CcdCamera thread to resume.   This occurred after Night Services had been started and the Restart overlapped with Session Pending/Session Initiating.   As a follow up a wait loop was placed in AutoStartSession() routine in AstroMain 3.67.3 that waits for RestartCCDSoft operation to finish before calling SetMode_FullyAutomated(), so that report messages are no longer interlaced.  This doesn't resolve the underlying issue.
 
2) Some 18 minutes after the first occurance of issue ObsOverseer raised a further CCDCamera alert at 20:19:03 with message "CcdCamera appears to be frozen (last cycle at 20:18:25)" and restarted CCDSoft & SBIG Camera in response and again this allowed the CcdCamera thread resumed.   This second incident occurred during the early (pre-slew) part of Target 1/26 (12P/Pons-Brooks), and likely led to an exception that caused execution to pass back to ExecuteQueue to be handled by an outer catch.   A potential reason for CcdCamera to lock-up around this time has been eliminated in AstroMain 3.67.3, however the root cause has still to be fully defined and fixed.

Previous sessions which have encountered the "CcdCamera appears to be frozen in section 'GetCameraProperties' " issue  are as follows
 S1156A (2023-10-01), S1183 (2024-01-16), S1185 (2024-02-13), S1196 (2024-03-04), S1197 (2024-03-13), S1198 (2024-03-15), S1200 (2024-03-21), S1201 (2024-03-27) showing the problem has been present for a fairly long time, but doesn't occur in every session and doesn't really occur in any particularly circumstance or pattern, except that there seems to have been an uptick in frequency since the beginning of March (i.e. in sessions S1196+)

Analysis (1)

19:17:41  AstroMain restarted in order to install version 3.67.2
19:17:46  CcdCamera and CcdMonitor services started
19:19:46  Monitor log shows CCDSoft Camera is connected (connected at somepoint between 19:17:46 & 19:19:46)
19:18:31  ObsPic shows CCDSoft Camera is connected (connected at somepoint between 19:15:48 & 19:18:31)
19:18:53  Autostart Requested
19:57:46  Monitor log records CCD Temperature 13.5, Power 0%
19:58:52  Autostart commenced when conditions ok
19:59:02  Autostart created Live Session (S1203)
19:59:02  CCDSoft Restarted in order to set Autosave No.
19:59:28  CCDSoft.Camera connected, TemperatureSetPoint set to -20°C
19:59:28  Setting CCDTempChart Graph StartTime
19:59:46  Monitor log records CCD Temperature  8.4, Power 100.0%
20:00:16  Last Successful cycle of CcdCamera
20:00:21  CcdCamera frozen (froze at some point between 20:00:16 & 20:00:21)
20:00:31  CcdCamera is Pending (15s) '  in section 'GetCameraProperties'
20:01:26  CcdCamera is Frozen (70s) in section 'GetCameraProperties'
20:01:26  CCDSoft being restarted in attempt to fix frozen CcdCamera
20:01:46  Monitor log records blank CCD Temperature & Power values
20:02:25  CCDSoft.Camera connected, TemperatureSetPoint set to -15°C
20:03:46  Monitor log records CCD Temperature -11.9 , Power 100.0%

Conclusion

Actions:

Update 2024-04-04
After refactoring CCDCamera, extending the properties handled, and making all calls to SBIG Camera go through CCDCamera, and installing AstroMain 3.67.3 on the Observatory Computer it was hoped that issues from freezing up of the CCDSoft Camera and CCDCamera would stop as all calls to the CCDSoft Camera & SBIG Camera would all be made sequentially with no risk of having two or more messages interferring.  

However a further incident occurred on 2024-04-04 at 09:03 shortly after the session had progressed to general session '2024-04-04'.  CCDCamera was seemingly frozen in GetProperty / CoolerPower.

Back to Top