Issue: CCDSoft Images not getting saved to file (23:57 onwards).

Description: VR Images for Target 18/43 (TT Ari) at 23:57 had image numbers reported as "0000" and couldn't be found on disk following the session.   Problem seemed tto originate moments before when Foc1 focusing failed when Take 1s Image (for Select Focus Star) gave the error message "Object reference not set to an instance of an object".   

Images for the remainder of the night similarly had "0000" image numbers and didn't save to disk.   This caused Centering Operations to fail as Locate Frames couldn't be Plate Solved (a process that requires images to be saved to disk).   This is the same issue as occurred in S805 in 2020-09-04, but actions to send alert/alarms and provide fault tolerance to prevent problem ruining the remainder of the session had not been put in place - the problem was percieved as ''.. a  Session Gremlin'.  Now the the problem has repeated itself (and with more consquence this time - loosing some 5 hours of observing time) effort must be put into developing an effective workaround to restart image collection.

Analysis
In exploring firstly for possible self-inflicted (bug related) reasons for images not to save to disk it was evident that problem could happen if Camera.AutoSaveOn = 0.

The TakeImage() routine sets AutoSaveOn to 0 if UseSubFrame is True.  Of the routines with code lines 'UseSubFrame = True'
several can be discounted (these are FocusUsingFWHM() - manually initiated, TakeAutoFlatSet(), TakeAutoSkyFlats(), TakeTrialFlat(), DeleteMeasureTracking(), MeasureDrift() ), leaving just one routine, TakeSnapImage_Detail() that might possibly be involved.  This would have had to be accessed from RunTakeSnapImage() routine / TakeSnapImage() but would have had to have had user involvement to initiate, and the routine in any case resets UseSubFrame = False after taking the snap image.  Camera.AutoSaveOn = 0 is used in several place, but TakeImage will reset Camera.AutoSaveOn to 1 (True) if UseSubFrame = False'.
(testing shows that Camera.AutoSaveOn does mind if 1 or True (or 0 or False) are used)

Camera.AutoSaveOn = 0 could also be set if NumShortSubs > 1 but NumShortSubs would have been reported and since it wasn't  this explanation can be ruled out.

Conclusion
Ruling out a self-inflcited explanation it has to be assumed that problem was due to a CCDSoft.exe or CCDSoft Com Object issue

Action
An attempted fix has now been put in place such that if a Locate Frame, Main Image Frame or Mapping Frame returns and Image with Image.Path ="" then CCDSoft will be restarted and the image repeated.

Fix Attempted 2020-09-30 (AstroMain 3.29.5)

Issue:
Dome Slew time to Target 6/44 (11P/Tempel-Swift-LINEAR starfield, sidereal) reported as being just 1s for 124 deg slew (21:16).  

Description:
The Dome Slew time to Target 6/44 (11P/Tempel-Swift-LINEAR starfield, sidereal) was reported as being just 1s for a 124 deg slew.  The dome slew time shown in the SlewTable file is impossibly short. 
A similar issue (1s dome slews) occured for TT Ari (23:52) and DY Per (01:25).  This might be due to DeviceHub.Dome stalling ? 

Analysis:
Logs show slew starting at 21:16:01, with overall slew ending at  21:16:53 (scope slewing time 52s, overall slewing time 52s).   Obs Pics show a normal slaved dome slew to DY Per of 30s+ duration and whilst DeviceHub.Dome did stall at 21:17:58, this was after the dome data stopped refreshing and doesn't explain the impossiblly short dome slew time.  Pulsar Trace logs show SlewToAzimuth 116.03 commanded at 21:16:01.268 with slew then occuring between 21:16:02 and  21:17:05 (showing a dome slew time of 63s)

It is difficult to see an explaination for why SlewTheSkyScope() reckoned that dome slew time was only 1.2s  !

Dome.LastSlewSeconds is set to vcNull before the slew and  bDomeSlewStarted is set to True once Dome.Slewing becomes True. 

After DomeSlew has started  'Dome.LastSlewSeconds' variable is set to SecondSince(StartTime) once Dome.Slewing becomes False. Due to a bug (just noticed & fixed in AstroMain 3.30) it was possible for Dome.LastSlewSeconds to be set to value that is as long as the overall slew time ( which can sometimes be the time taken by the telescope), but given that dome slew is known to have taken some 63s it is difficult to see how Dome.LastSlewSeconds could be set to a shorter time and not one as short as 1.2s for a 124 deg slew.

The hypothesis to explain the issue is that DeviceHub sets its Slewing property to True when it is sends the SlewToAzmiuth request to the Dome (which causes bDomeSlewStarted to be set to True), but then receives a Slewing = False from the Dome Driver before the dome actually starts slewing (this causes DeviceHub.Dome slewing property to become False, leading to the  SlewTheSkyScope() routine concluding that dome slewing has completed. 

Code could be modified to not make any conclusion about dome slew completon within the first 2-3s or to take slew duration based on the last time that dome slew was seen to be occuring,  but how long does the routine need to keep watching the Dome.Slewing property ?  Decision to set Dome.LastSlewSeconds based on when slewing becomes False, but watch for a minimum of 5s and only set bDomeCompleted = True after this time if Dome.Slewing is still False. 

Action:
Code fixed.   A new boolean variable bDomeSlewStopped is used as part of the new code

Issue: 11P/Tempel-Swift-LINEAR starfield (sidereal) sent to AstroMain as Priority 2 (Do Next) ToO during Target 3/43 (UGC 6930) but was commenced as Target 6/44 rather than the Target 4/44 expected.  

Description: Recognising that the upcoming Target 4/23 (11P/Tempel-Swift-LINEAR starfield with an explicit coords / non-sidereal combination) was going to fail, the Target was edited to be sidereal target in AstroPlan as then submitted as a Priority 2 ToO (Do Next) during the execution of Target 3/43 (UGC 6930).   The ToO target 11P/Tempel-Swift-LINEAR starfield (sidereal) was expected to be acquired as Target 4/44 (displacing the existing 11P/Tempel-Swift-LINEAR starfield target to Target 5/44).   Instead the ToO target was acquired as Target 6/44.   In the intervening time the original 11P/Tempel-Swift-LINEAR starfield (non sidereal) target was executed as 4/44 but quickly failed as expected because of unknown iobject name ( 'Explicit Coords' ) error.   Target 5/44 (CY Lyr) was then begun but entered a long waiting state as it was well ahead of its scheduled time slot. User Intervention was made to Abort CY Lyr target to finally allow the ToO target (11P/Tempel-Swift-LINEAR starfield (sidereal) ) to begin without further delay.

Analysis
Whilst centering Target 3/43 (UGC 6930) AstroMain's Report and Log files shows it receiving the ToO request and importing 11P/Tempel-Swift-LINEAR (priority 2) at 20:53:22.92, .   An attempt to insert the ToO into the Job Queue was made a second later at 20:53:23.86 but the task failed with the message 'Target (11P/Tempel-Swift-LINEAR) is deferred due to a 'Must Include' target'
> Received ToO Request        Ok        1 observation requested
> Loading ToO                 Ok        11P/Tempel-Swift-LINEAR (priority 2) imported at 20:53:22
> Inserting ToO Target...     Fail      Target (11P/Tempel-Swift-LINEAR) is deferred due to a 'Must Include' target

Job Executor made a successful second attempt to insert the ToO target queue into the job sometime between 21:08 and 21:16 (most-likely at 21:08) whilst waiting for Target 5/43 (CY Lyr) to start
> Inserting ToO Target...     Ok        Target (11P/Tempel-Swift-LINEAR, priority  2) inserted in job queue

According to the logs no attempt was made to insert the ToO target in the job queue during Target 4/43 (original 11P/Tempel-Swift-LINEAR target) which had preceeded beyond the Target Checks stage.

Conclusion
1) Code operated as per design by not inserting the ToO target into queue ahead of the original 11P/Tempel-Swift-LINEAR target as it was flagged as a Must-Include target for the session.

2) Design/code could be modified so that the Name or Target ID of the ToO target is compared with that of the 'Must-Include' target and if they match then allow the ToO to override the 'Must-Include' target,  working on basis that target will still be included. 

3) Code should have made an attempt to insert the ToO target before starting Target 5/43 (CY Lyr) and not wait till CY Lyr had started.   Code to insert ToO was presumed to have been skipped because the original  11P/Tempel-Swift-LINEAR target had failed so soon into execution and there wasn't a chance for ToO insert routine to be run.

The routine that inserts a pending ToO ( 'HandleToO' ) is called by the Obs.Manager main loop which operates on a basic 5s period, though depending on which part of the loop is running the HandleToO routine may be called as frequently as every 1s.   Target 4/43 (original 11P/Tempel-Swift-LINEAR target) was however executed in 0.71s or less. This meant there was a high likelihood that HandleToO routine wouldn't be called during course of the target's execution.   One solution would be to force a delay of up to 5.5s to ensure that at least one cycle of the Obs.Manager main loop is run before proceeding to the next target.

Action

• Modify InsertToO routine to allow a ToO Target to displace a Must-Include if Target Name or Target ID match.
  Done 2020-10-04 (AstroMain 3.30)
   
• Modify Job Executor code to allow a pending ToO to be inserted into job queue before moving onto next target.  This is achieved by forcing a delay at the end of any Target that have taken less than 6s to execute (as short as 0.5s for a target check failure, for example) and there is a Pending ToO. This is designed to allow sufficient time for the Obs.Manager main loop to call the HandleToO () routine before job execution moves onto the next target.
  Done 2020-10-04 (AstroMain 3.30)

Issue (S793): A ToO target was inserted into queue for immediate execution during execution of Target 3/12 but was unexpectedly reported as 'deferred' due to a current ToO. 

Issue (S794):  When aborting Must Include target M85, to make way for Barnards Star report message 'Target aborting due to GRB Alert' was inappropriate.

Issue (S804):  Priority 1 ToO (for Immediate Action) was unexpectedly deferred.  
Issue (S804):  Report message 'Target aborting due to GRB Alert'  is incorrect.

 Description (S793): M85 (a must-include target) was running but due to cloud along western horizon a new target (C/2017 T2 PanStarrs) with better chance of clear sky was inserted into the job queue as a ToO for immediate execution, however it was reported as deferred due to current ToO even though M85 was not a ToO target (though it was a 'must-include' target). To get M85 to stop and to get the comet target to execute immediately in its place a user intervention was made involving using the 'Abort Target' button.   This workaround worked, the M85 target was stopped and the job manager began executing the T2 comet target. However this intervention shouldn't have been necessary - its unclear why the comet ToO was 'deferred' rather executed immediately.    When current target was aborted manually used 'Abort Target' the reason for aborting the target in the message 'Target aborting due to GRB Alert' was unexpected and incorrect.

Description (S804): Whilst job quene was running UGC 6930 was set as a Priority 1 ToO target for immediate action in AstroPlan. The target (UGC 6930) was imported by AstroMain at 22:45:50 whilst the job executor was waiting to start Target 11/41 (WDS GRB 34).  The target (UGC 6930) was inserted into the job queue but 'deferred to due to a current ToO'.  This is a subtle bug,   WDS GRB 34 is not a ToO,  and should have been aborted to make way for UGC 6930.  Maybe the code treats WDS GRB 34 (Groombridge 34) as a ToO because it contains 'GRB' in its name even though it not a GRB (gamma ray burster) ?   The issue was resolved at the time by user intervention to abort the WDS GRB 34 target which allowed UGC 6930 to begin.  When it was aborted a report message is output saying 'Target aborting due to GRB Alert' . 

Target 11/41 (WDS GRB 34) was manually aborted by user to make way for a Priority 1 ToO (UGC 6930), which wasn't automatically scheduled for immediate effect but was deferred (see issue above). When GRB 34  was aborted a report message is output saying 'Target aborting due to GRB Alert' , it should say 'Target aborting due to (Urgent) ToO'.

Analysis (S793)
Whilst waiting for guider to settle for Target 3/12 (M35) AstroMain's Report and Log files shows it receiving the ToO request and importing C/2017 T2 PanStarrs (priority 1) at 20:29:08. There was then two messages, one saying the that ToO target had been inserted into the Job Queue and one saying that it is deferred due to a current ToO.
> Received ToO Request        Ok        1 observation requested
> Loading ToO                 Ok        C/2017 T2 (PanStarrs) (priority 1) imported at 00:29:08
> Inserting ToO Target...     Ok        Target (C/2017 T2 (PanStarrs), priority  1) inserted in job queue
> Inserting ToO Target...     Fail      Target (C/2017 T2 (PanStarrs)) is deferred due to a current ToO

> Target Aborting ...         Ok        Target aborting due to GRB Alert

These show that the ToO target is inserted in the job queue directly ahead of the next planned target 'R CrB,  but is then deferred 'due to a current ToO'. 
Code shows a bug in a line that checks whether the currently active item is a ToO.   The code was not aborting the current M35 target and giving the message about deferral because the condition test was 'if TargetOfOpportunity = True'  rather than  'if TargetOfOpportunity = False'

Although the original description (S793 Notes) stated that user has intervened by manually clicking the 'Abort Target' button the session logs don't seem to indicate this but suggested that Target was aborted by the ToO Handler.

However the Enum AbortReason declaration has two rows with the same value 
GRB = 2
User = 2
This would seem to be an error, and presumably causes  GetAbortReason() to return 'GRB Alert' when the global variable has the value AbortReason.User 

Conclusion
1) A bug in 'HandleToO routine'  is responsible for the unexpected message 'Target ... is deferred due to a current ToO'

2) A bug in 'Enum AbortReason' declaration appears to be responsible for the unexpected message 'Target aborting due to GRB Alert'

3) Problems in session S794 and S804 are due to the same two bugs

Action

• Fix bug (if TargetOpportunity = True) in 'HandleToO routine that is causing unexpected messages 'Target ... is deferred due to a current ToO' when trying to insert a Prioroity 1 (Do Now) Target of Opportunity.
  Done 2020-10-05 (AstroMain 3.30)
   
• Fix bug in 'enum AbortReason' (two members with same integer value) and extend 'GetAbortReason' routine  to return "User Intervention" when abort reason is AbortReason.User.  This is intended to fix the problem that that the message  'Target aborting due to GRB Alert' is given when user uses 'Aborts Target' facility.
  Done 2020-10-05 (AstroMain 3.30)
   

Proposal:  As part of a renewed initiative for getting improved secondary scope imaging capability it is proposed to to guide the LX200 mount using SBIG camera and PHD2 autoguiding software under the control of AstroMain's Job Executor (via PHD2 Server) with the necessary actions of automating the switching of the PHD2 profile between 'ZWO ASI178MC / TS80' profile and ' SBIG ST-10XME / 12" LX200' profile, and automating the disconnection/reconnection of SBIG camera from CCDSoft/AstroMain.

Existing methods used to guide the LX200 using Secondary Scope with ZWO ASI178MC camera and PHD2 autoguiding software under the control of AstroMain's Job Executor via PHD2 Server, with necessary actions for automated the switching ZWO ASI178MC camera operation between PHD2 (for guiding) and SharpCap (for Focusing / Imaging) provide the neccesary experience for delivering the new capability.

[ Details on Connection, Event Notification Messages and Server Method Invocation is given
  here:  PHD2 Event Monitoring (https://github.com/OpenPHDGuiding/phd2/wiki/EventMonitoring) ]

Alternatives :  Alternatives to the above proposal have been tried but they have their issues.

• Autoguiding using CCDSoft.    Autoguiding using CCDSoft has the limitation that Dec (Y Axis) guiding can't be done in just one directional, Dec Guiding has to be either turned off or performed bidirectionally.
   
• SharpCap's Livestacking with auto-align :  SharpCap's LiveStacking with auto-alignment of  frames works on starfields with lots of stars, but starfields with fewer than 10 stars or have stars that are too dim for routine measurement

Issues / Potential ShowStoppers

• Disconnecting SBIG camera from CCDSoft.  A reliable method of disconnecting the SBIG camera from CCDSoft / AstroMain and connecting again later on with destabilising AstroMain needs to be proven.  Obs.Monitor can have the behaviour of repeatedly reconnecting to camera even if camera is disconnected on CCDSoft Camera Dialog.
   
• Testing on Development Computer.  Development computer doesn't have SBIG hardware, so testing is difficult
   
• Automating connection of  SBIG camera to PHD2.  Initial trials show that when Connection is made to SBIG Camera, PHD2 puts up a 'CCD Choice' modal dialog and the user is required to click 'Yes' to use the Tracking (Guide) CCD, or 'No' to use the Main CCD.   It would be ok if this could be done once (and the answer remembered by PHD2) but the question has to be manually answered ever time the connection is made.  This could become a showstopper to automating the connection/disconnection of equipment / swapping of equipment profiles several times within an automated session.   It is not known if the choice can be made via PHD2 Server or whether KeyStrokes can be 'sent to PHD2 reliably enough to select 'No' every time. Larger Main CCD is better for autoguiding : more sensitive and larger FOV / greater choice of guide star.
  
  

CCD Choice Dialog Box

• Set Profile Error : Set Profile produces an error is Connect Equipment dialog is open.
   
• PHD2 Profile(Cam2) :   Fail   Exception occurred whilst trying to set PHD2 Profile on Laptop9 (Obs.Computer)
  
• Camera Change Warning : PHD2 popped up a dialog window asking for confirmation to connect to this camera anyway.
  
   
• Cannot open SBIG camera , code 29:  Message pops up 9 times (once for each pass through loop attempting to connect to Camera & Mount )
  
   
• Unable to change Telescope Properties at this time.  Seemed to be a DeviceHub message.
  
  
• PHD2 Profile doesn't match that which was set errors :
   
• SBIG ST-10 3 CCD Camera   : camera name once connected
   
• N/S Errors not in arc secs.   At the moment guidestep N/S errors and guide movements are reported and plotted in arc secs, by taking guidestep.dx and guidestep.dy and assuming an Image Scale is 1.00"/pixel. This is reasonable for guide frames taken previously through TS80 Scope using ZWO ASI178MC camera with 1x1 binning, as image scale (1.03"/pixel) is very close to 1.00.      Image Scales using 12" LX200 and SBIG ST-10XME (at 1x1 or 2x2 binning) are significantly different from 1.00, and therefore need to be accomodated in the reporting/plotting of Guide Errors and Guide Movements.   
  
  It is proposed to get ImageScale using the PHD2.GetImageScale() function and converting the resulting String to Double, and then multiple  dx,  dy,  RADistanceGuide and DecDistanceGuide by ImageScale within the ParseGuideRecord function
  
• DX/DY Errors not translated to correct North Orientation space.    At the moment Dx/Dy errors are assumed to be related to a North Up Camera Orientation , which the TS80/ZWO Guide Frames have, however the  LX200/SBIG Guide Frames have a North Down orientation and this again needs to be accomodated in reporting/plotting.
  At the moment it is uncertain what manipulations need to be.
  
  It is proposed to get North Orientation from the Guide System setting (ProfileMode).   such that
     if         ProfileMode = "Cam1" then NorthOrientation = Down
     elseif  ProfileMode = "Cam2" then NorthOrientation = Up
  and use a  checkbox option to either manipulate the Dx, DY values or not
  
  
  
  

Start Services

  PHD2 Client                 Ok        Started            (Profile:ZWO ASI178MC)
  PHD2 Camera                 Ok (3)    Started            (ZWO ASI178MC)
  PHD2 Mount                  Ok (3)    Started            (DeviceHub -> LX200GPS/R (ASCOM))

Prepare SBIG Guiding

  Swap PHD2 Profile
  Stop  PHD2 Equipment        Ok        Stopped            (ZWO ASI178MC Profile)
  Start PHD2 Profile          Ok        Started            (SBIG Camera Profile)
  Start PHD2 Camera           Ok (2)    Started            (SBIG/LX200GPS/R)
  Start PHD2 Mount            OK (2)    Started            (DeviceHub -> LX200GPS/R (ASCOM))
  Check PHD2                  Ok        Ready   

Logfile

22:30:40.98 | GCVS RX And          Disconnect PHD2 Equipment  Ok    | Profile : ZWO ASI178MC

22:30:40.98 | GCVS RX And          Connect PHD2 Profile       Ok    | Profile : SBIG Camera
22:30:40.98 | GCVS RX And          Connect PHD2 Equipment     Ok    | Camera  : SBIG
22:30:40.98 | GCVS RX And          Connect PHD2 Equipment     Ok    | Mount   : DeviceHub -> LX200GPS/R (ASCOM)

Route Map

• Variable to store PHD2 Profile State
• New Routine (ConnectCam1Profile & ConnectCam2Profile) to seaminglessly Disconnect Equipment, Change PHD2 Profile, Connect New Equipment
• Connection to SBIG Camera in AstroMain after disconnection of SBIG Camera from PHD2.
• Method to Workaround PHD2's CCD Choice Dialog Disconnect Equipment
• Modify GuideCharts to acknowledge/display the equipment profile
• Get SBIG CCD Temperature  (get_sensor_temperature ) - Not available/not implemented for SBIG camera.