David's Astronomy - Notes 2022-10-28 (S1050)

Session Aims & Highlights

- Observing Result
- Night Summary Plot
- Session Event Log

Operational Issues

- Critical Issues (0), Major Issues (1), Minor Issues (4), Small Defects (2), Continuous Improvement (4)

Images from 2022-10-28 >> [ Local Files >> ]

2022-10-29

Investigation - CCDSoft, ObsMonitor & ObsManager freezing

2022-10-29

Issues

1) Obs.Monitor and Obs.Manager freezing around end of Session Starting (2022-10-28 19:17, Major Issue S1050)

2) Obs.Monitor frozen in section 'Update Foc1 Fields' (2022-10-26 21:30, Major Issue S1049)

3) Problems with CCDSoft/SBIG - describe & investigate (2022-10-22, Minor Issue S1048)

Description

1) 2022-10-28 S1050: As part of session creation CCDSoft was restarted at 19:11 to allow the CCDSoft's AutoSave No. to be set (which included a power-cycling / reset of the SBIG Camera), the SBIG camera was connected during starting operations at 19:15. This seemed to be successful based on Camera CCD properties reported. Directly after opening the dome the ObsOverseer reported at 19:17 that Obs.Monitor is frozen in section Update Foc1 Fields. and shortly after it reported that Obs.Manager is frozen (last cycle at 19:17:26). At 19:19 ObsOver called the MakeObservatorySafe which turned off the Telescope tracking, check dome status and commanded the Dome to close (the monitoring of the dome closure was held up by the frzoen Obs.Manager . A manual initiated call to Kill & Restart CCDSoft was made at 19:30, but after putting a subtitle, it didn't seem to proceed any further. A manual iniated call to Switch Off SBIG Camera was made at 19:31. This released ObsManager and ObsMonitor. Dome closure monitoring resumed and mistakenly gave the alert that Shutter hadn't closed after 90s and eventually reported 'Dome is closed (closing time 735s)' . A manual initiated call to Switch On Sbig Camera was made at 19:31. 'Restart Sbig Camera was then executed (unclear if this was called by user or by ObsManager. Two attempts to connect camera at 19:32 failed with message "SBIG driver: Device not found". Report logs are unclear whether more than one RestartCCDSoft was running at this point. Camera seemed to connnect ok at 19:32. At 19:33 ObsOverseer noted that ObsMonitor frooze momentarily frooze at 19:33.
Around this time a "Discount Camera 1" operation failed with the error message "The RPC server is unavailable.". And attempt to connect Camera at 19:35 failed with the message "Object reference not set to an instance of an object" . A ' Kill&Restart CCDSoft ' call at 19:35 was successful, camera connected again. Session was set to Manual (it was in Running) and the 'Fully-Automated' button clicked and session then proceeded to run ok.

2) 2022-10-26 S1049: After creating a new live session a Restart CCDSoft operation was run (21:25) in order to allow session's initial AutoSave Number. to be set via the registery. This is a normal operation at this point. This operation includes an opportunistic power-recycling/reset of the SBIG Camera. AstroMain appeared to reconnect to the Camera ok (no errors/issue when querying or setting CCD camera properties. Querying of Camera properties was ok during Session at 21:29. Shortly after reporting the' table of expected timings', the Obs.Overseer reported at 21:30 that the Obs.Monitor is frozen in section 'Update Foc1 Fields'. This is normally due to an issue where execution of the monitor is waiting to return from a query to CCDSoft Camera (temperature, filter position etc). After seeing that Obs.Monitor was frozen and more than 2.5 minutes, user interventation made to use the button option to Restart CCDSoft (which calls Kill&RestartCCDSoft routine including a power reset of the SBIG Camera). Killing CCDSoft immediately released the hold on ObsMonitor execution. At the same time that ObsMonitor was frozen. TheSkyScopeMonitor intermittently froze or ran very slowly taking 4.9-9.9s to go through one loop cycle. The USB and power cable connections to/from the SBIG Camera were checked the following day and found to be ok.

3) 2022-10-22 S1048: issue description still to be written

Analysis

The problems in S1049 and S1050 (which are better documented) were all associated with the initial part of the session. After the problem was resolved with one or more user interventation to 'Restart CCDSoft' (including SBIG Camera Reset) or direct Turn off SBIG Camera / Turn on SBIG Camera, the rest of the session ran ok.

Although Restart CCDSoft (including SBIG Camera Reset) is being successfully used to resolve the problem in-session with needing to restart AstroMain or visit the observatory, at the same time it seems that the Restart CCDSoft operation made at the beginning of the session (to the allow the CCDSoft's initial AutoSave No. for the session to be set in the Registry) is also giving rise to the problem in the first place, i.e. the issue may be self-inflicted (at least in part).

Examining the 'Restart CCDSoft' code showed a somewhat complex arrangement of routines associated with stopping and restarting CCDSoft (an arrangement that has now been simplified, improved and standardised, AstroMain 3.56.6), but the a key observation is that a Camera disconnection (if connected) is not being performed at the beginning of the RestartCCDSoft() routine. The code lines disconnecting the CCDSoft camera are commented out due to past observations that querying or setting Camera.Connnected could lead to a freeze in RestartCCDSoft() execution if there is already a freeze in CCDSoft & AstroMain's ObsMonitor.

To get around this the 'RestartCCDSoft' code has been modified in AstroMain 3.56.6 so that the code dealing with disconnection of the camera is moved to a DisconnectCamera() routine which is then run as a threaded operation and can't cause the RestartCCDSoft' to freeze.

Issues

1) Job Queue continued to execute targets even though each attempt to take any image with SBIG Camera was failing with exception. (2022-10-13, Minor Issue S1046)

2) Focus Profile showed data points for last 5 focus steps (10/14 to 14/14) even though a Focus Frame image wasn't acquired. (2022-10-13, Minor Issue S1046)

Description

1) 2022-10-13 S1046: Whilst taking Focus Frame image for step 10/14 for Target 10/32 (GCVS S Cyg), and all subseuqent Take Image attempts (focus frames, locate frames and main frames) encountered exceptions messages 'Command in progress, Error code = 219 (0xdb)'. Job Queue executor continued processing targets and marked them as completed and produced 'invalid' JPG pictures based off last image (Focus Frame from step 9/14) even though no images were acquired. TakeImageASync() routine has been modified in AstroMain 3.55.4 to set a flag when this exception occurs and increment a counter. Job Executor should now be modified to use these flags and stop the execution of new targets until the flag becomes reset. Partial fix made in AstroMain 3.55.4 (2022-10-14) which ignores the call to create JPG pictures if bSbigDriverException is raised and sets FailureState.SbigCamera (29).

2) 2022-10-13 S1046: Due to a glitch with SBIG camera during Focus Profile for Target 10/32 (GCVS S Cyg) focus frame images weren't acquired for the final five focus steps. Despite this FWHM, HFD and MaxValue values were recorded with values taken from last valid focus frame (Step 9/14) and used in an attempt to get an inappropriate Best Focus position. Fortuanetly due to an ambiguous result the QC Checks failed and the Focus Offset wasn't modified. When the focus frame failed FWHM, HFD and Max Value should have been set to vcNull and the marked as non-Valid.

Analysis

This Web Page:	Notes - Session 1050 (2022-10-28)
Last Updated :	2024-03-22
Site Owner :	David Richards
Home Page :	David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 1050 (2022-10-28)

Session Aims & Highlights (2022-10-28)

Operational Issues (2022-10-28, S1050)

2022-10-29

Investigation - CCDSoft, ObsMonitor & ObsManager freezing

Investigation - CCDSoft TakeImage failing

David's Astronomy Pages Notes - Session 1050 (2022-10-28)

Session Aims & Highlights (2022-10-28)

Operational Issues (2022-10-28, S1050)

2022-10-29

Investigation - CCDSoft, ObsMonitor & ObsManager freezing

Investigation - CCDSoft TakeImage failing

David's Astronomy Pages
Notes - Session 1050 (2022-10-28)