David's Astronomy - Notes 2022-12-17 (S1072)

Session Aims & Highlights

- Observing Result
- Night Summary Plot
- Session Event Log

Operational Issues

- Critical Issues (0), Major Issues (4), Minor Issues (3), Small Defects (4), Continuous Improvement (9)

Images from 2022-12-17 >>

2022-12-18

Investigation - Job Queue and several monitoring services froze during Plate Solution (04:33)

2022-12-18

Issue. Job Queue and several monitoring services seemed to freeze at around 04:33, during Plate Solution.

Description

During Centering operations for target 37/45 (UGC 8046) several services seemed to freeze at around 04:33. ObsMonitor appeared to be frozen in section ''Refresh Dome Picture/Label Dome Azimuth" (last cycle at 04:33:22). ScopeMonitor appears to be frozen (last cycle at 04:33:20) and at 04:35 there was an alert message "TheSky6 isn't running or isn't working correctly".

Eventually the ObsManager aborted the Target and at 04:37 suspended the session (including closing the dome). The Obs.Overseer reported that the Job Queue appears frozen (04:43) and activated a Restart CCDSoft operation in an attempt to 'fix locked thread'. The Restart successfully occurred and Camera was reconnected (it is unclear whether CCDSoft had become frozen or not). Meanwhile the ObsManager made two further AbortTarget requests. CCDMonitor first changed CCD temperature to -25degC before lowering it to -30degC.

Obs.Overseer reported that Job Queue was still frozen at 04:45 and again at 05:00. At 05:00 there was another call to RestartCCDSoft (again an 'attempt to fix locked thread') which again seems to have been successful in restarting CCDSoft but unsuccessful in triggering a release of Job Queue. At 06:50 the 'End of Schedule Plan' was passed and the Observatory Manager (which was evidently still running ok) moved the session into 'Closing' state.

During Session Closing, progress stalled immediately after beginning Park Telescope Operations at 06:51 (this is a separate issue but as it turns out one that is related to the issue under investigation). User intervention had to be made at 08:45 to finally end the session.

7 targets from the end of the Observing PLan were not executed as a result of the problem.

Analysis
Logs clearly show that Locate Frame 3 (#1072308) had been successfully acquired and there was no issue with the CCDSoft.Camera in acquiring the image or with CCDSoft.Image with regard to opening and using the image acquired.

Logs show that Stars objects in the image inventory were counted (8) & validated, and the process of searching for a Plate Solution using TheSky6 ad begun. Several 'InsertWCS' operations were tried. The final insert, InsertWCS (0\1), at 04:33:20, seems to have provided the most successful Link (Stars 7 | Angle 179.66 | Scale 1.400). Shortly after this (2.5s later) the log would normally then show the Best Match, give details about the ImageCentre and give the results of a check on the WcsCenter, but there no such logging occurs in this case indicating that the Job Queue has frozen between 04:33:20 and 04:33:23. At this same time the ScopeMonitor and ObsMonitor both freeze.

One of DeviceHub's ASCOM log files which is believed to be the one reporting its communications with TheSky6 (via TeleLink/TeleApi) show queries of the Telescope RA/Dec coordinates (which normally occur every 0.5s) stopping at 04:33:21.50. Thus it seems that TheSky6 hung at this same point.

Automation had determined that the JobQueue was frozen due to a CCDSoft Issue, but this seems to be incorrect and the job queue was frozen because of some other reason.

Probably the only way of keeping the session's activities going at this point would have been for the ObsManager or ObsOverseer to Kill and Restart TheSky6 and restart TheSky objects & scope connection etc. Killing TheSky6 would probably have released the frozen threads, which may or may not have exited gracefully but they would have at lleast had the opportunity to do so. The act of killing and restarting TheSky6 requires coordination since AstroPlan, AstroVOE have connections to TheSky6, will will need to be informed that TheSKy6 is being terminated. and allow them to adjust their operations and to restart their connections after a specific amount of time.

Park Telescope operation at 06:51 failed as the routine to send the telescope to a Pre-Park position using TheSky6 Telescope slewing became stalled due to TheSky6 not responding (having been in this state since 04:33).

Findings & Conclusions
1) ObsMonitor should be made more robust to occasions when TheSky6 stalls. ScopeMonitor should be the sacrifical 'Canary' . All monitoring interactions with TheSky6 should be made through the ScopeMonitor alone in order to protect the ObsMonitor/ObsViewer.

2) Automation had determined that the JobQueue was frozen due to a CCDSoft Issue, but this seems to be incorrect and the job queue was frozen because of some other reason. Automated diagnosis should use a wider set of information in making a judgement about a stalled job queue in order to select the appropriate course of action to resolve the issue.

3) Automated diagnosis/problem resolution needs access to a new routine that can kill and restart TheSky6 in order to deal with issues where Job Queue is frozen due to a TheSky6 not responding.

Actions
Isolate ObsMonitor/ObsViewer from TheSky6 Telescope. Make all monitoring interactions with TheSky6 to be through the ScopeMonitor alone

Modify Job Queue self-diagnosis or oversight-diagnosis algorithums. Automated diagnosis should use a wider set of information in making a judgement about a stalled job queue in order to select the appropriate course of action to resolve the issue.

Create a Kill and Restart TheSky6 routine. Create a routine that can be called to resolve issues due to a non-responding TheSky6 instance.

Create an automated process(es) that can refresh TheSky6 objects in AstroPlan/AstroVOE when TheSky6 program is killed/restarted. AstroPlan/AstroVOE require access to TheSky6 and functionality needs to be restored if/when TheSky6 is killed and restarted.

This Web Page:

Notes - Session 1072 (2022-12-17)

Last Updated :

2024-09-25

Site Owner :

David Richards

Home Page :

David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 1072 (2022-12-17)

Session Aims & Highlights (2022-12-17)

Operational Issues (2022-12-17, S1072)

2022-12-18

Investigation - Job Queue and several monitoring services froze during Plate Solution (04:33)

David's Astronomy Pages Notes - Session 1072 (2022-12-17)

Session Aims & Highlights (2022-12-17)

Operational Issues (2022-12-17, S1072)

2022-12-18

Investigation - Job Queue and several monitoring services froze during Plate Solution (04:33)

David's Astronomy Pages
Notes - Session 1072 (2022-12-17)