David's Astronomy - Notes 2022-07-15 (S1022B)

Session Aims & Highlights

- Observing Result
- Night Summary Plot
- Session Event Log

Operational Issues

- Critical Issues (0), Major Issues (4), Minor Issues (8), Small Defects (5), Continuous Improvement (13)

No Images >> [ Local Files >> ]

2022-07-16

Investigation - CCDSoft and ObsMonitor (Section: Update Foc1 Fields) Hanging

2022-07-23

DeviceHub 6.6.0.16

2022-07-16

Issue
1) Operations held up due to ObsMonitor becoming frozen stalled in section 'Update Foc1 Fields whilst Starting Observatory Services (22:25)
2) ObsManager and ObsMonitor became frozen (23:25:01).
3) Job Queue froze whilst measuring first Foc1 focus profile (00:13 & 00:57)

These were all symptons of a showstopping issue that stopped mutliple attempts to get the job queue underway.
Problem is suspected to be due to some sort of conflict or circular dependancy between CCDSoft Camera and Foc1 Focuser.

Stalled ObsMonitor routine, particularly in the 'Update Foc1 Fields' section is on ongoing occasional program, but it seems to have suddenly become much worse during the S1022B session attempt and the following daytime testing.

Description
1) Operations held up due to ObsMonitor becoming frozen stalled in section 'Update Foc1 Fields whilst Starting Observatory Services (22:25). Starting from 22:25:43 the Obs.OverSeer reported that Obs.Monitor was frozen in 'section : Update Foc1 Fields'. This is an ongoing occasional issue. Problem occurred whilst ObsServices was waiting for Telescope to finish waiting 60s following Telescope Power On. Services were stopped and AstroMain closed. Problem didn't reoccur when AstroMain/Observatory Services were restarted. Suggest looking careful at the precise points where similar issues have occured in the past. Is there a relationship to starting Observatory Services ?

2) ObsManager and ObsMonitor became frozen (23:25:01). ObsOverseer reported that ObsMonitor became pending/frozen at 23:25:01 in section 'Update Foc1 Fields' with ObsManager reported as frozen at 23:25:16. This is an ongoing occasional issue, again the issue is reported to be due to a freeze in 'Update Foc1 Fields' section.
Observatory Services. ObsOverseer intervened by calling MakeObservatorySafe() routine at 23:27:06 which closed and parked the Dome. The problem occured at the end of Equilibrating State.

3) Job Queue froze whilst measuring first Foc1 focus profile (00:13 & 00:57). Job Queue froze whilst 'Taking Focus Frame 0.3s (C) 5/23' . ObsOverseer reports "Camera.TakeImage task in TakeFocusImage/TakeImage is taking a long time?". ObsOverseer also detected ObsMonitor pending/freezing at the same time in section 'Update Foc1 Fields'. After restarting AstroMain and all Observatory Software, the issue occured whilst 'Taking Focus Frame 0.3s (C) 3/23' durig the first Focus Profile. After killing CCDSoft the Job Queue and ObsMonitor both resumed. Job Queue aborted Foc1 focusing and went directly to Foc2 focusing. Problem is due to clash between different process trying to access Focuser1 and/or CCD Camera.

Analysis
What is known or suspected regarding the issue ?

- suspect some sort of conflict or circular dependancy between CCDSoft Camera 
and Foc1 Focuser 
 
- ObsMonitor freeze occurs almost exclusively in 
'Update Foc1 Fields' section - this seems to be very significant
 (This 
freeze originally occurred within a Focuser1 section, before it was split into 
separate "Monitor Focuser1" and "Update Foc1 Fields" sections on 2022-06-10 (CI, 
S1018) to help trace the problem)
 
- typically the issue isn't 
expressed when AstroMain is simply monitoring the Observatory Status

- 
issue can be expressed shortly after starting services
 
- issue 
typically occurs when CCD Camera is commanded to take an image whilst ObsMonitor 
and TCF-S Focuser services are running.
 
- issue can sometime lead 
to ObsManager freezing.   
 
- it is possible that Camera 
being operated in Synchronous mode doesn't respond well when property calls are 
made to CCDSoft.Camera whilst Camera is busy taking an image.
 
- 
problem wasn't seen during the S1021 session (2022-07-12), 
  [since 
then a Windows Update has occurred (2022-07-13) which created a critical issue 
for TheSky6/Telescope API during S1022A session attempt (2022-07-14)]


- it is possible that the issue could be resolved by operating the camera in 
Asynchronous mode, but this isn't certain.

- the potential fixes for the 
problem (see later in this investigation) were implemented (2022-07-17) but 
didn't resolve the issue

Update Foc1 Fields Section

' Update Foc1 Fields (provided CCDSoft.Camera is accessible) ' ------------------ gObsMonitorSection = "Update Foc1 Fields" ' added 2022-06-10 (CI, S1018) TimePoint = Now() If Camera1_IsAccessible() Then UpdateFoc1Fields() End If ' Debug nMilliSecs = SecondsSince(TimePoint) * 1000 If nMilliSecs > 500 Then MonitorNote = MonitorNote + "Foc1 Fields: " + CStr(nMilliSecs) + ", " End If nMilliSecs_Foc1Fields = nMilliSecs

Of this code there are only two lines of potential concern

a)   If Camera1_IsAccessible()
b)       UpdateFoc1Fields()

Line b) is suspect because it indirectly makes several 'get property' requests from the CCDSoft.Camera object.
Line a) calls a function called 'Camera1_IsAccessible()' that appears to be designed to avoid calling UpdateFoc1Fields (which makes property requests from CCDSoft.Camera) during periods where the Camera is busy taking an image (which could include changing filter, taking image and downloading it).

The Camera1_IsAccessible() function may not fulfill its intended role in the following situation
- where the bCamera1TakingImage flag isn't set for a given Take Image operation made in AstroMain
- where an image is being taken using CCDSoft Camera's Take Image Tab
- where there is a timing issue or there is a timing issue that could be related to
Now() > Camera1ExpectedStartDownloadTime.AddSeconds(-VM.Cam1TimeAllowance))
Camera1ExpectedStartDownloadTime is typically set to   Now().AddSeconds(Camera.Delay + Camera.ExposureTime)
potentially it doesn't take into account time for FilterWheel to be changed.
VM.Cam1TimeAllowance typically has the value 5.

The Camera1_IsAccessible() function includes the condition Now() > Camera1ExpectedStartDownloadTime.AddSeconds(-VM.Cam1TimeAllowance)). This was introduced in AstroMain 3.38.3 (2021-03-12) as a fix to an issue 'CCD and Foc1 Fields sections of Obs.Monitor experience occasional hangups of 1 - 3s'   (see ) Minor Issues ).

Notes from S868 Minor Issues (2021-03-11) : timings showed that CCD object becomes locked/unaccessible when it is downloading an image. This time is predictable based on when Camera.TakeImage is called and the delay and exposure time used. These hang-ups could potentially be fixed by using asynchronous camera ops but this requires very careful implementation, therefore the issue will be fixed in the shorterm by using a flag and time point (bCamera1TakingImage and Camera1ExpectedStartDownloadTime) to indicate when the Camera is / isn't available. Added a txtCam1TimeAllowance field (VM.Cam1TimeAllowance) with default value of 5.0s so that Camera isn't accessed during last 5s of taking an image when CCDSoft employs very fast refreshing of its Camera Status field which seemingly blocks other processes trying to access the CCDSoft.Camera object.

Function Camera1_IsAccessible() As Boolean '============================= If bCamera1TakingImage And Now() > Camera1ExpectedStartDownloadTime.AddSeconds(-VM.Cam1TimeAllowance) Then Return False Else Return True End If End FunctionUpdateFoc1Fields Routine Public Sub UpdateFoc1Fields() '=========================== Dim TempCompPosition As Integer Dim TempCompDifference As Integer ' Temperature ' -------------- VM.Foc1AvTemperature = MonitorFoc1AvTemperature VM.Foc1Temperature = MonitorFoc1Temperature ' Position ' -------- VM.Foc1Position = MonitorFoc1Position ' Temperature Compensated Position ' -------------------------------- If MonitorFoc1AvTemperature = vcNull Then VM.Foc1TempCompPosition = vcNull VM.Foc1TempCompDifference = vcNull Else TempCompPosition = Focuser1.GetTempCompPosition() VM.Foc1TempCompPosition = TempCompPosition ' Temperature Compensated Difference ' ---------------------------------- TempCompDifference = TempCompPosition - MonitorFoc1Position Try VM.Foc1TempCompDifference = TempCompDifference Catch ex As Exception End Try ' TempCompPosition (including Filter Offset) ' ------------------------------------------ TempCompPosition = Focuser1.GetTempCompPositionForFilter() VM.Foc1TempCompPositionForFilter = TempCompPosition TempCompDifference = TempCompPosition - MonitorFoc1Position VM.Foc1TempCompDifferenceForFilter = TempCompDifference ' Filter Offset ' ------------- Try If aFilter(Camera.FilterIndexZeroBased + 1) = "S" And VM.UseSpectraFocusing Then VM.Foc1FilterOffset = VM.SpectraFocusOffset Else VM.Foc1FilterOffset = SU.FocFilterOffset(Camera.FilterIndexZeroBased + 1) End If Catch ex As Exception VM.Foc1FilterOffset = vcNull End Try End If ' Focuser Status ' -------------- VM.Foc1Status = MonitorFoc1Status End Sub

Of this code there are 3 lines of potential concern:

c) TempCompPosition = Focuser1.GetTempCompPosition()
d) If aFilter(Camera.FilterIndexZeroBased + 1) = "S" and ...
e) VM.Foc1FilterOffset = SU.FocFilterOffset(Camera.FilterIndexZeroBased + 1)

All three lines are suspect because they directly or indirectly make 'get property' calls to the CCDSoft.Camera object

Focuser1.GetTempCompPosition()

Function GetTempCompPositionForFilter() As Integer ' Returns Temperature Compensated Focus Position (for Current Camera Filter) '========================= Dim TempCompPosition As Integer Try ' Calculate Temperature Corrected Focus Position (C filter) ' ---------------------------------------------- TempCompPosition = GetTempCompPosition() ' Add Offset for Filter ' --------------------- Try If aFilter(Camera.FilterIndexZeroBased + 1) = "S" Then If VM.UseSpectraFocusing Then TempCompPosition = TempCompPosition + VM.SpectraFocusOffset ' include S[1] Focus Offset Else TempCompPosition = TempCompPosition + SU.FocFilterOffset(Camera.FilterIndexZeroBased + 1) ' include S[0] Focus Offset End If Else TempCompPosition = TempCompPosition + SU.FocFilterOffset(Camera.FilterIndexZeroBased + 1) ' include Filter Focus Offset End If Catch ex As Exception End Try Return TempCompPosition Catch ex As Exception Return vcNull End Try End Function

Conclusion
A weakness in the Camera1_IsAccessible() function (called by Update Foc1 Fields section of the ObsMonitor) allows read requests to the Camera.FilterIndexZeroBased property to be made in the UpdateFoc1Fields() routine whilst the CCDSoft.Camera is busy taking an image (synchronously). This was conjectured leads to ObsMonitor and CCDSoft to freeze and not respond.

The condition check ' ... And Now() > Camera1ExpectedStartDownloadTime.AddSeconds(-VM.Cam1TimeAllowance) ' can allow the Camera1_IsAccessible() function to return True when it should arguably return False.

It is unclear why session S1022B and subsequent daytime tests using the ObsMonitorTest routine were so badly affected. Why was the issue not seen during S1021 session (2022-07-12). What's changed since 2022-07-12. WIndowsUpdate (2022-07-13)

It was subsequently concluded that issue was caused by a glitch with either the SBIG Camera or its driver or USB controller, which was resolved by power-cycling the SBIG Camera. See 'Update 2022-07-19'.

Fix
The Camera1_IsAccessible() function should be modified to make it entirely controlled by the bCamera1TakingImage flag. Viz.

Function Camera1_IsAccessible() As Boolean '============================= If bCamera1TakingImage Then Return False Else Return True End If End Function

Whilst this resolves the immediate issue that was causing multi-showstopping events during S1022B session, there is still a theoretical risk due to unpredictable interlacing of thread execution:

With the Take Image routines on one thread using the code

bCamera1TakingImage = True Camera.TakeImage() bCamera1TakingImage = False

and ObsMonitor routine on a separate thread using the code

If Camera1_IsAccessible() Then UpdateFoc1Fields() End If

There is the risk that lines might sometimes be executed in the following order

Thread 1 : bCamera1TakingImage = False

Thread 2 : If Camera1_IsAccessible() Then Thread 2: Function Camera1_IsAccessible() As Boolean Thread 2: If bCamera1TakingImage Then Thread 2 : Else Thread 2 Return True Thread 1 : bCamera1TakingImage = True Thread 1 : Camera.TakeImage() Thread 2: UpdateFoc1Fields() Thread 2: .... Thread 2: If aFilter(Camera.FilterIndexZeroBased + 1) = "S" Then >> ObsMonitor and CCDSoft.Camera both hang upon requesting Camera.FilterIndexZeroBased whilst Camera is busy taking image. Thread1 : bCamera1TakingImage = False

In the above example a call to UpdateFoc1Fields still manages to slip through the net

Use of .Net's SyncLock might be a solution but may produce issues of its own.

Proposal is therefore to add extra checks of Camera1_IsAccessible() within the UpdateFoc1Fields() routine so that if value of bCamera1TakingImage does change to True it can be recognised and used to skip code that tries to read the Camera.FilterIndexZeroBased property and thus prevent ObsMonitor/CCDSoft.Camera from locking up.

Actions

AstroMain/ObsMonitor. Modify Camera1_IsAccessible() function and UpdateFoc1Fields() routine to incorporate the above fixes.
Done 2022-07-17.

AstroMain/TakeImage. Modify TakeImage() routines to implement TakeImage() as a asynchronous operation.
Done 2022-07-20

Update 2022-07-19

Further testing indicates that problem still persists. It seem that CCDSoft stops responding (or runs very slowly) & ObsMonitorTest hangs when AstroMain GUI is clicked on. This suggests that problem might be associated with how CCDSoft.Camera object is created.
Trial carried out by reinstating the call to CreateCCDSoftObjects() from CreateObjects() (called from Main windows Form1_Load () routine.
however the problem still persists.
Trial carried out by manipulating AstroMain GUI whilst running ObsMonitorTest() however doesn't cause the routine to hang and therefore this is not the cause.
Manipulating the CCDSoft interface whilst ObsMonitorTest is running and calling CCDSoft.Camera.TakeImage does however seem to cause CCDSoft to hang.
Trial carried out using new TakeImage_ASync() routine.The routine works ok (performs Asynchronously), but the again issue reoccured toward end of Frame 4/14
Trial carried out by power cycling the SBIG Camera. This seemed to stop the issue.
Stress Testing ObsMonitorTest (including with Foc1 Moves & with manipulation of AstroMain and CCDSoft GUI's) showed that the Take Image process is operating robustly.
TakeImage() routines changed to use Camera.TakeImage as an asynchoronous operation
Camera1_IsAccessible() function modified to for time being always return True, allowing ObsMonitor and CCDMonitor to access Camera properties concurrent with imaging.

2022-07-23

DeviceHub was updated on the Observatory Computer and Development Conputer today (2022-07-23) from version 6.6.0.13 to new 6.6.0.16 version.

The new version fixes an issue where the Dome Slaved iproperty wasn't being reset to False when a Dome.AbortSlew is commanded. This issue had meant that a dome slew could resume moments after commanding dome to abort slew (and this directly interfere with the reason for the dome abort which might be a safety/mechanical problem or a need to urgently close the dome's shutter.

The fix make the DH behaviour consistent with the ASCOM API documentation of the Dome.AbortSlew method which says :
- '(the method) immediately stops any and all movement'
- 'calling this method will immediately disable hardware slewing (Slaved will become false). '
(https://ascom-standards.org/Help/Developer/html/M_ASCOM_DriverAccess_Dome_AbortSlew.htm)

6.6.0.14 Release Notes
- Added support for Application Startup Logging.

6.6.0.15 Release Notes
- Clean up shutdown due to fatal error during startup
- Remove dependency on AstroUtils

6.6.0.16 Release Notes
- Reset dome slaving when a slaving slew is aborted.

>> Device Hub Releases : https://github.com/ASCOMInitiative/ASCOMDeviceHub/releases

Activity Log
After installation of the new version the Activity Log Capacity value (which was reset to its standard default value (125000) during installation ) was again increased ten-fold (1250000) by editing the DeviceHub configuration file (see notes DeviceHub 6.6.0.10, 2022-06-02 )

Results
Daytime testing of DeviceHub 6.6.0.16 on 2022-07-23 showed that dome movements are stopped and Dome.Slaving is set to False when the Red (Abort) Rectangle is clicked on the DH's Dome Control tab. Daytime testing on 2022-07-24 showed that dome movements are also stopped and Dome.Slaving is set to False when AstroMain client calls Dome.AbortSlew().

Update 2022-07-26
DeviceHub 6.6.016 was used successfully during live session S1022 on 2022-07-26, however two issues occured in the AstroMain observatory control program as a consequence of the change (reset dome slaving when a slaving slew is aborted) that was made in 6.6.0.16.

1) Dome didn't slew to Focus Field 20 (01:16)
2) Dome slew to 61 Cyg initially went to the 'wrong' azimuth (01:29).

The two problems were due to the presence of code added on 2022-06-04 (MI, S1016) to AstroMain's Scope.SlewTheSkyScope() routine which calls Dome.AbortSlew when there is a dome slew already in progress, before slewing scope/dome to target. Because Devicehub 6.6.0.16 resets Dome.Slaved flag to False when Dome.AbortSlew is called, it prevented the a slaved dome slew to Focus Field 20 (at 01:16) . Because Dome.Slaved was then turned back on for 61 Cyg, it immediately caused DH slaving calculations to begin slewing the dome to the azimuth associated with the pre-slew scope coordinates, instead of slewing the dome directly to azimuth associated with 61 Cyg's target coordinates.

The two issue were resolved in AstroMain 3.53.1 by undoing the 'AbortSlew' code feature that was added 2022-06-04 and instead make the call to slew scope wait until any pre-existing dome slew is finished. A separate but related continuous improvement was made (and also implemented in AstroMain 3.53.1) to modify the Dome.SetSlavingOn() routine to wait up to 4s for any slaved dome slewing to commence, thus allowing dome slewing to be detected and a 'wait on dome slew to finish' to be made before commanding the scope slew to the next target.

This Web Page:

Notes - Session 1022B (2022-07-15)

Last Updated :

2024-09-25

Site Owner :

David Richards

Home Page :

David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 1022 Attempt 2 (2022-07-15)

Session Aims & Highlights (2022-07-15)

Operational Issues (2022-07-15, S1022B)

2022-07-16

Investigation - CCDSoft and ObsMonitor (Section: Update Foc1 Fields) Hanging

2022-07-23

DeviceHub 6.6.0.16

David's Astronomy Pages Notes - Session 1022 Attempt 2 (2022-07-15)

Session Aims & Highlights (2022-07-15)

Operational Issues (2022-07-15, S1022B)

2022-07-16

Investigation - CCDSoft and ObsMonitor (Section: Update Foc1 Fields) Hanging

2022-07-23

DeviceHub 6.6.0.16

David's Astronomy Pages
Notes - Session 1022 Attempt 2 (2022-07-15)