David's Astronomy - Notes 2020-10-09 (S0818)

Issue
Job Queue frooze around end of centering / start of autoguiding of Target 1/26 (61 Cyg) leading to Obs.Overseer making the observatory safe whilst autoguiding/monitoring continued with multiple 'star lost' messages reported.

Description
Last IamAlive message was from ExecuteTarget/CenterTarget/ at 22:25. Abort Target, Abort Job wouldn't work. Manually aborted PHD2 guiding and moved AstroMain to Manual and manually used the 'Kill Job' backdoor button to kill the Job Queue task. Restarted Automated mode without closing the program (a calculated risk / experiment). Target 2/26 (SS Cyg) and Target 3/26 (NGC 6978) successfully acquired.

The last few log messages , would seem to confirm that Centering had finished and operations had moved onto to AutoGuiding in preparation for focusing/imaging.

The same issue happened with Target 6/26 (NGC 514), where job queue again stopped at the same point. Again the same manual method was used to resume session.

The PHD2 Graph suggests that problem happened before Dec Drift Direction / Required Dec Guide Direction was determined and set.

The same issue happened again when Target 6/26 (NGC 514) was restarted., and then again later on with Target 11/26 (HH And) These times the job queue was aborted using 'Abort Job' button before the Obs.Overseer intervened to make the observatory safe (allowing a faster restart).

Analysis
Hypothesis is that Job Executor calls 'Guider.StartGuider, but execution doesn't return for some reason and/or some finished flag isn't set or not read..   Multiple 'Star ...    Fail      Star lost - low HFD" messages more than 10 mins after Job Queue is believed to have frozen indicate that the 'Guider.StartPHD2Guiding' is still running. Report records shows that the scope hasn't exit the routine and started the Focusing stage. Within StartPHD2Guiding routine the execution enters a Do While bGuideMethodFinished = false' loop where it appears that bGuideMethodFinished is never set to True and the execution remains locked in the loop giveing the outside appearance that the Job Queue has frozen. The only conditions in which bGuideMethodFinished is set to True are if a SettleDone event is received or an error Response is received.

Examining the PHD2 Message Log messages associated with 61 Cyg the Guide request was made at 22:25:10
    {"method": "guide", "params": [{"pixels": 2.0, "time": 6, "timeout": 10}, false], "id": 36}

for which there was a 'SettleDoneEvent' 27s later at 22:25:37.48
   {"Event":"SettleDone","Timestamp":1602278737.469,"Host":"LAPTOP9_CHILL","Inst":1,"Status":0,"TotalFrames":4,"DroppedFrames":0}

however it was superseded by a following "GuideStep" event just 30ms later at 22:25:37.51

It is supposed that due to unfortunate timing issue (Guide Steps are typically 2600ms apart) the LastEvent variable wih SettleDone event was overwritten before if had been read, after which there was no way for the loop to be exited.

Fix
A fix has been attempted in AstroMain 3.30.5 by making PHD2 Listening Loop send events to an Event queue (qEvent) instead of an overwrittable LastEvent variable and editing PHD2.GetEvent read from this queue, ensuring that 'Do While bGuideMethodFinished = false' loop processes each Event received without missing any. As a further backup a timeout of 2 x TimeOut of the Guide Method is used as additional condition to exit out of the StartPHD2Guiding routine.

After monitoring later sessions it was concluded that the fix had been successful.

Fixed 2020-10-10 (AstroMain 3.30.5).


	Session Aims & Highlights
	- Observing Result - Night Summary Plot - Session Event Log

	Operational Issues
	- Critical Issues (0), Major Issues (1), Minor Issues (6), Continuous Improvement (10)

	Images from 2020-10-09 >>

	Investigation - Frozen job queue following start of autoguide operations

This Web Page:	Notes - Session 818 (2020-10-09)
Last Updated :	2024-09-30
Site Owner :	David Richards
Home Page :	David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 818 (2020-10-09)

Session Aims & Highlights (2020-10-09)

Operational Issues (2020-10-09 S818)

Investigation - Frozen job queue following start of autoguide operations

David's Astronomy Pages Notes - Session 818 (2020-10-09)

Session Aims & Highlights (2020-10-09)

Operational Issues (2020-10-09 S818)

Investigation - Frozen job queue following start of autoguide operations

David's Astronomy Pages
Notes - Session 818 (2020-10-09)