David's Astronomy Pages
Notes - Session 1022 (2022-07-26)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (0),  Minor Issues (4),  Small Defects (2),  Continuous Improvement (10)
 
Bullet Images from 2022-07-26 >>        [ Local Files >> ]    
2022-07-27
Bullet Missing Observing Opportunity (2022-07-27)
2022-07-28
Bullet Investigation - PHD2 Client/Server Comms Issues
   

Session Aims & Highlights (2022-07-26)

Main aims

  1. AstroMain. Check out fixes and new features in 3.53 version of AstroMain. Ensure that new version hasn't introduced any new issues. 
  2. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by time & twilight conditions.
  3. C/2017 K2 (PANSTARRS). Acquire images of comet C/2017 K2 (PANSTARRS).
  4. DeviceHub.  Test DeviceHub 6.6.016 in live observing session.

 Equipment & Software

Highlights

Notes:

  Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Image
Image
  
Sky Conditions (Locate Frames)
Image
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image   
  
Session Event Log
Time     Event Detail
22:49:13 Session Monitoring AutoStart monitoring for Live Session opportunity between 22:49 & 02:34
22:49:16   AutoStart Delayed Session delayed due to cloud
01:10:24 Session AutoStarting Session autostarting (01:10)
01:11:04   Camera1 Connected SBIG Camera connected (set point -5°C)
01:11:06 Session Created Session Created (Live, 2022-07-26 S01022, ImageSaveNum: 1022001)
01:11:08   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
01:12:50   Services Started Observatory Services started
01:12:57 Observatory (Auto) Observatory placed in Fully-Automated Mode
01:12:59 Session Pending Session pending (2022-07-26)
01:13:01 Session Initiating Session initiating (2022-07-26)
01:13:03   Plan Requested Observing Plan requested from AstroPlan (1.30.10)
01:13:50   Plan Loaded Observing Plan loaded to queue (Plan ID: 784)
01:14:05   Camera1 Connected SBIG Camera connected (set point -10°C)
01:14:11   Telescope Connected Telescope connected (TheSky6)
01:14:35 Session Equilibration Session ready to Open Dome
01:15:20   Dome Opened Dome opened (opening time 45s)
01:16:05 Session Running Session running
01:16:07   Queue Started Observing Queue started (7 targets selected)
01:16:11     Target Started (NrZen) Target started (Focus Field 20, HIP 98535)
01:16:12   Dome Unparked Dome unparked
01:18:09       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:21:29       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
01:23:43       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
01:23:46       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
01:25:56       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
01:27:42       Focusing Completed Foc2 AutoFocus Completed (Profile No 2)
01:28:00     Target Completed Target completed (Focus Field 20, HIP 98535)
01:29:31     Target Started (1/7) Target started (1/7, 61 Cyg)
01:32:46       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:34:46       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
01:36:20     Target Completed Target completed (1/7, 61 Cyg)
01:36:24     Target Started (2/7) Target started (2/7, GCVS SS Cyg)
01:38:56       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:41:02       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
01:45:26     Target Completed Target completed (2/7, GCVS SS Cyg)
01:48:54   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome)
01:48:56   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
01:48:58   Queue Paused Observing Queue pausing for Session Suspension
01:48:59 Session Suspended Session suspended
01:49:49   Dome Closed Dome closed (closing time 50s)
02:01:40     Target Cancelled Target cancelled (3/7, GCVS BL Lac) due to cloud
02:11:08     Target Cancelled Target cancelled (4/7, GCVS EV Lac) due to cloud
02:18:03 Session Resuming Session resuming
02:18:48   Dome Opened Dome opened (opening time 45s)
02:18:50 Session Running Session running
02:18:52   Queue Resumed Observing Queue resumed
02:18:55     Target Started (5/7) Target started (5/7, NGC 157 w/SN2022jli)
02:20:23   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity)
02:21:08   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome)
02:21:10   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
02:21:12     Target Aborted (5/7) Target aborted (5/7, NGC 157 w/SN2022jli) due to cloud
02:21:14   Queue Paused Queue paused for Session Suspension
02:21:16 Session Suspended Session suspended
02:21:58   Dome Closed Dome closed (closing time 45s)
02:30:18     Target Cancelled Target cancelled (5/7, NGC 157 w/SN2022jli) due to cloud
02:52:18     Target Cancelled Target cancelled (6/7, M31 w/AT2022ors) due to cloud
03:14:04     Target Cancelled Target cancelled (7/7, GCVS RX And) due to cloud
03:14:09   Queue Completed Job Queue completed
03:14:15 Session Closing Session closing
03:14:20   Dome Closed Dome already closed (closed at 02:21, closing time 45s)
03:14:52   Dome Parked Dome parked (parking time 24s), Az: 90.0 deg
03:16:08   Telescope Parked Telescope parked (parking time 71s)
03:16:23   Telescope State Scope parked Turn scope off. (Handbox)
03:16:45   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
03:17:37   Services Stopped Night Services stopped
03:17:39 Session Housekeeping Session Housekeeping started (Create Fits Summary, Transfer Files)
09:54:02 Observatory (Manual) Observatory placed in Manual Mode
 
Session Alerts
Time     Alert Detail
01:30:23 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (61 Cyg)
 

Back to Top


Operational Issues (2022-07-26, S1022)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Small Defects

Continuous Improvement

[ Prev | Next ]

Back to Top


2022-07-27


Missed Observing Opportunity (2022-07-27)

An observing opportunity was missed on night 2022-07-27.  Conditons around dusktime looked hopeful but whilst Weather Forecasts (both DarkSky.net and xcweather.co.uk) indicated dry conditions (no rain) the cloud cover was expected to be poor (90-100% cover), and with AllSky Images indicated moving cloud with only minor gaps at 23:00 (sun alt -8.5°)  it was decided not to put the observatory into autostart mode. In the event skies cleared at around 00:00 (~ sun alt-12°) and stayed clear until at least 02:30 (~ sun alt -12°).

Missed Observing Opportunity 2022-07-27
 
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image   
  

Back to Top


2022-07-28


Investigation - PHD2 Client/Server Communication Issues

Issues

1) Since session In order to prevent issues with client/server data communications with PHD2 (where multiple incomplete PHD2 message reads occurs) a workaround has been used in which  the AstroMain program is always freshly restarted between Observing Sessions.  Is there preventing issues without having to perform this pre-session restart ?

This is not normally an inconvenience since AstroMain is typically restarted anyway between sessions in order to install a new version.   However it would be nice to close PHD2 client/server comms robustly in a way so that AstroMain doesn't necessarily have to be restarted prior to a new session.

2) Incomplete PHD2 data stream read. can indicate a problem with client/server data communications. Can there presence be used to automatically restart PHD2 program and restart client connection ?

Fixes where made to AstroMain so that incomplete PHD2 data stream reads were recognised and result/event messages built up by appending data from following data stream reads, and the presence of these incomplete data stream reads counted, and then flagged up to warn/alert the user, as they can be indicative of a problem with client/server data communications that can compromise the generation of guiding charts and the production of guiding stats.  Whilst problem of incomplete data stream reads seems to be associated in the past with occassions when AstroMain wasn't freshly restarted ahead of a new sessions, the problem could potentially appear at any time.   It would be handy if AstroMain could automatically resolve the problem itself.   Whilst server response and event messages are recoverable it situations where there is an incomplete data stream read, communications seem to be slowed down considerably and this affects quality of guiding charts/guiding stats.

A continuous improvement ticket was raised at time of S961 session to investigate ways to mitigate/correct INCOMPLETE Phd2 Data Stream reads, but has remained to now as an outstanding/non-completed task.


3) In session S942 (2021-12-16) the PHD2 exposure looping/guiding routines hung half-way through the night due to an unhandled & unknown program exception in PHD2 and led to all the remaining targets being unguided. Could AstroMain have intervened and attempted to fix problem by killing/restarting PHD2 ?

The incident during S942 was investigated at the time but was unable to find a specific reason behind the exception in PHD2 ( See Investigation - PHD2 application failed during guiding (unhandled and unknown exception) ).  A continuous improvement ticket was raised to add monitoring of PHD2 to AstroMain to provide the potential opportuntity to automatically kill/restart PHD2 in the event that the PHD2 application appears to have stalled, but has remained to now as an outstanding/non-completed task.
 
Analysis

Problems have occurred with PHD2 communications in sessions S951, S953, S958, S961, S965 & S972.  Whilst code has been previously changed to deal with 'Incomplete PHD2 data stream reads', in session S972 it was reported that this is still an occasional issue, which is resolved by closing and restarting AstroMain.  It is uncertain if this is still a neccessary/recommended action prior to a new session or not.

Reviewing the code associated with PHD2 Service / PHD2 Client, the principal routines are:

PHD2.StartClient
Client created and connected using
  Client = New TcpClient
  Client.Connect(ServerAddress, PortNumber)

  ' once connected

  DataStream = Client.GetStream()
 

PHD2.StopService
At end of an observing session the PHD2.StopService() is called, this
 - ensures that PHD2 guiding is stopped , by using the 'stop_capture' method
-  disconnects equipment attached to the current PHD2 profile by using the 'set_connected' (params: [false] method
-  stops the client listening loop, by setting the bStopListening false to True
-  waits 500ms to allow the Listening Loop to exit (but no feedback)
- disconnects the PHD2 Client by calling PHD2.StopClient
  

PHD2.StopClient
The StopClient() routine does the following
- closes the datastream (NetworkStream) with DataStream.Close()
- closes the client (TCPClient) with Client.Close()
- sets datastream to Nothing
- sets client to Nothing

DataStream.Close() closes the current stream and releases any resources (such as sockets and file handles) associated with the current stream. Instead of calling this method, ensure that the stream is properly disposed (using Datastream.Dispose() )

Client.Close() disposes theTcpClient instance and requests that the underlying TCP connection be closed.    Calling this method will eventually result in the close of the associated Socket and will also close the associated NetworkStream that is used to send and receive data if one was created.

Requirement
- A test routine that can replicate the issue with PHD2 client/server data communications.
- A revised method of closing PHD2 Client/Server that fixes the issue with client/server data communications when starting a new session without restarting AstroMain
- Rerunning of test routine to demonstrate that issue has been fixed.

Test Routine
Using On-The-Fly Observing without Slew&Centre, but with 'AutoGuide' should provide an appropriate test of PHD2 client/server comms.

 Select Telescope Service
 Select PHD2 Service
 Start Services

 Run 'test routine'   on-the-fly imaging of M67 with guiding

 Stop Night Services

 Select Telescope Service
 Select PHD2 Service
 Start Services

 Run 'test routine'   on-the-fly imaging of M67 with guiding

Stop Night Services (including PHD2.StopService / PHD2.StopClient )

Change Session

Select Telescope Service
 Select PHD2 Service
 Start Services

 Run 'test routine'   on-the-fly imaging of M67 with guiding

Unable to replicate issue with PHD2 client/server data communications where there are incomplete PHD2 reads.

Actions

1) Code Changes to Start/Stop PHD2 Client

Code changes made routine for starting/stopping PHD2 Client on 2022-07-28 (AstroMain 3.53.1).

StartClient() routine
   DataStream = Client.GetStream()  moved ahead of  StartListening(), and delay added to ensure that Listener thread has fully initialised :

 
  DataStream = Client.GetStream()
   StartListening()
   Thread.Sleep(500)

StopClient() routine
The following lines removed, based on .Net description of TcpClient.Close() method, which closes the datastream if there is one and disposes of the client.

 ' DataStream.Close() 
 ' DataStream = Nothing
 ' Client = Nothing

2) Automatical restart of PHD2
Implemented 2022-07-30 (AstroMain 3.53.1) by looking for nIncompletePHD2reads > 5 and invoking RestartPhd2ProgramAndService() routine from OB.ExecuteTarget() before initiating guiding.  

Previously it was was found that restarting PHD2 without restarting AstroMain in session S961 failed to resolve the problem of Incomplete PHD2 data stream reads, but it is hoped that new 'RestartPhd2ProgramAndService()' routine works better than past manual invoked methods.

3) Monitoring for Stalled PHD2 Program
Eventually implemented 2022-07-29 (AstroMain 3.53.1) by looking for the absence of any "LoopingExposures" events after calling PHD2' s "guide" method and then invoking RestartPhd2ProgramAndService() routine from OB.ExecuteTarget() routine before making a second attempt to initiate guiding.

4) Merging Incomplete PHD2 reads.
 Investigate 3s delay and optimise. (from S961)

Back to Top