---

Fig 1.   Report File covering CCDCamera freeze at 18:51 / Connect Camera Failure at 18:52.

  Taking 1s box (C)  1/23     **                          (Continued below) > Obs.Overseer                Warning   CcdCamera is frozen (18:51) in section 'GetCameraProperties / FilterIndexZeroBased'

 **Taking 1s box (C)  1/23        
  Restart CCDSoft                       2024-10-24 18:51  (Local, BST)   Restart CCDSoft             Info      CCDSoft being restarted (attempt to fix frozen CcdCamera)

  Stop CCDSoft                          2024-10-24 18:51  (Local, BST)   
  Stopping CCDSoft            **                          (Continued below) > CcdCamera                   Info      CcdCamera has resumed again (18:51) after 35s
 
**Stopping CCDSoft            Ok           

  Stop SBIG Camera                      2024-10-24 18:51  (Local, BST)   Switch Off SBIG Camera      Ok        Switching off SBIG Camera > SBIG Power                  Info      SBIG Power has switched off   (18:51:45, on time > 1.7 h)

  Restart SBIG Camera                   2024-10-24 18:51  (Local, BST)   Switch On SBIG Camera       Ok        Switching on SBIG Camera > SBIG Power                  Info      SBIG Power has switched on    (18:51:57, off time 12 sec)   Wait on SBIG Camera...      Ok        Waited 20s for camera to initialise   

  Restart CCDSoft                          
  Start CCDSoft5              Ok        Started (18:52)   (CCDSoft.exe)   
  Creating Camera object      Ok        CCDsoft.Camera   
  Creating Autoguider object  Ok        CCDsoft.Camera   

  Connect Camera                        2024-10-24 18:52  (Local, BST)   
  Connecting camera...        **                          (Continued below) > Obs.Overseer                Warning   Camera.TakeImageTask in TakeFocusImage/TakeImage is taking a long time? (18:52) **Connecting camera...        Fail      SBIG driver: Device not found.   
  Reconnecting camera (attempt 2)...

  Connect Camera                        2024-10-24 18:52  (Local, BST)   Connecting camera...        Fail      SBIG driver: Device not found.  
 
  Taking 1s box (C)  2/23     Fail      Image failed  
 

Fig 2  Scope Slew Path & / Dome Slew Path tables for slew at 18:57

18:57:31.53 | Set     ScopeMode(3) Info | Scope Mode : Slewing
18:57:31.53 | (trace) Telescope.Slew (trace) | [ SlewScopeNearZenith() Slewing ]
18:57:31.53 | Call    Scope.SlewToRaDec Info | Calling SlewTheSkyScope(RaDec)
18:57:31.53 | Start   Scope.SlewTSkyScope Ok | Starting SlewTheSkyScope()
18:57:31.53 | Set     Scope.SlewModeRaDec Ok | Setting slew mode to RaDec
18:57:31.53 | Check   Dome.Slaved Info | Slaving is On
18:57:32.41 | Get     PreSlew RaDec(Scope)Info | Ra= 18 02 37.000 Dec=+48 01 48.00 | Epoch Current
18:57:32.41 | Get     PreSlew AzAlt(Scope)Info | Az= 256.0 Alt=69.6
18:57:32.41 | Get     PreSlew AzAlt(Calc) Info | Az= 256.0 Alt=69.6
18:57:32.41 | Get     PreSlew Az(Dome) Info | Az: 252.9
18:57:32.41 | Set     Telescope.Async Ok | Setting Telescope.Asynchronous = 1
18:57:33.25 | Call    Scope.SlewToRaDec Ok | Calling Telescope.SlewToRaDec
18:57:33.26 | Report  Slew Scope(Coords) Info | Ra= 22 04 23.209 Dec=+48 48 29.90 | Epoch Current

18:57:33.53 | ScopePath Info | Time:  1.1, Slewing: 0, Slew: 0.000,  Az: 255.98, Alt: 69.63, Angle: 0.000
18:57:35.14 | ScopePath Info | Time:  2.7, Slewing: 0, Slew: 0.003,  Az: 255.99, Alt: 69.63, Angle: 0.003
18:57:37.41 | ScopePath Info | Time:  5.0, Slewing: 0, Slew: 0.006,  Az: 256.01, Alt: 69.62, Angle: 0.003
18:57:39.90 | ScopePath Info | Time:  7.5, Slewing: 0, Slew: 0.011,  Az: 256.02, Alt: 69.61, Angle: 0.003
18:57:42.38 | ScopePath Info | Time: 10.0, Slewing: 0, Slew: 0.014,  Az: 256.03, Alt: 69.61, Angle: 0.003
18:57:46.55 | ScopePath Info | Time: 14.1, Slewing: 1, Slew: 0.893,  Az: 248.03, Alt: 73.12, Angle: 0.879
18:57:49.85 | ScopePath Info | Time: 17.4, Slewing: 1, Slew: 11.047, Az: 215.03, Alt: 80.76, Angle: 10.154
18:57:53.25 | ScopePath Info | Time: 20.8, Slewing: 1, Slew: 21.013, Az: 138.78, Alt: 78.87, Angle: 9.966
18:57:56.32 | (Overseer)  Obs.Monitor is Pending (7s) in section 'Refresh Dome Picture/Draw Shutter Sensor'
18:57:56.84 | ScopePath Info | Time: 24.4, Slewing: 1, Slew: 30.940, Az: 108.93, Alt: 70.65, Angle: 9.927
18:57:57.18 | Focus Field 22 (Monitor) Check ObsMonitor Info | Obs.Monitor has resumed (cycle = 8.1s)
18:57:57.18 | Focus Field 22 (Monitor) Check ObsMonitor Info | Breakdown (ms) :
18:58:00.52 | ScopePath Info | Time: 28.1, Slewing: 1, Slew: 39.943, Az: 103.43, Alt: 69.92, Angle: 9.004
18:58:01.33 | (Overseer)  Obs.Monitor is Pending (4s) in section 'Refresh Dome Picture/Label Scope Position'
18:58:04.84 | (Monitor)  Check ObsMonitor Info | Obs.Monitor has resumed (cycle = 7.7s)
18:58:04.84 | (Monitor)  Check ObsMonitor Info | Breakdown (ms) :
18:58:05.22 | ScopePath Info | Time: 32.8, Slewing: 1, Slew: 40.043, Az: 103.72, Alt: 70.05, Angle: 0.100
18:58:08.50 | ScopePath Info | Time: 36.1, Slewing: 1, Slew: 40.090, Az: 103.85, Alt: 70.12, Angle: 0.047
18:58:10.34 | ScopePath Info | Time: 37.9, Slewing: 1, Slew: 40.123, Az: 103.88, Alt: 70.12, Angle: 0.033
18:58:14.08 | ScopePath Info | Time: 41.7, Slewing: 1, Slew: 40.129, Az: 103.89, Alt: 70.12, Angle: 0.006
18:58:15.04 | Note           | Time: 42.6, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:15.78 | ScopePath Info | Time: 43.4, Slewing: 0, Slew: 40.129, Az: 103.90, Alt: 70.13, Angle: 0.003
18:58:17.07 | Note           | Time: 44.7, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:18.07 | Note           | Time: 45.7, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:20.37 | Note           | Time: 48.0, Slewing: 0, Angle: 0.002 ignored (telescope slew already completed
18:58:20.66 | Note           | Time: 48.2, Slewing: 0, Angle: 0.002 ignored (telescope slew already completed
18:58:24.64 | Note           | Time: 52.2, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:25.68 | Note           | Time: 53.3, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:27.76 | Note           | Time: 55.3, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:28.66 | Note           | Time: 56.2, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:29.57 | Note           | Time: 57.2, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:31.73 | Note           | Time: 59.3, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:34.08 | Note           | Time: 61.7, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:34.42 | Note           | Time: 62.0, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:36.56 | Note           | Time: 64.1, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:36.82 | Note           | Time: 64.4, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:40.64 | Note           | Time: 68.2, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:41.71 | Note           | Time: 69.3, Slewing: 0, Angle: 0.003 ignored (telescope slew already completed
18:58:45.68 | Focus Field 22 Set Telescope.Asynch Ok | Setting Telescope.Asynchronous = 0
18:58:45.68 | Focus Field 22 Report SlewAngle Info | Total: 40.129, Direct: 60.645
18:58:45.68 | Focus Field 22 Report SlewTime(Scope) Info | Slew Time (Scope) = 40.9s
18:58:46.38 | Focus Field 22 (Overseer) Check ObsMonitor Info | Obs.Monitor is Pending (5s) in section 'Refresh Dome Picture/Draw Camera State'
18:58:46.52 | Focus Field 22 (Monitor) Check ObsMonitor Info | Obs.Monitor has resumed (cycle = 4.8s)
18:58:46.52 | Focus Field 22 (Monitor) Check ObsMonitor Info | Breakdown (ms) :
18:58:47.19 | Focus Field 22 Report SlewTime(Dome) Info | Slew Time (Dome) = 73.3s, Slew Angle (Dome) = 146.3°
18:58:47.19 | Focus Field 22 Report SlewTime(Overall) Info | Slew Time (Overall) = 73.3s
18:58:47.21 | DomePath Info | Time: 1.6, Slewing: 0, Slew: 360.0, Az: 252.9, Speed: 221.78, Time: 18:57:34.04
18:58:47.21 | DomePath Info | Time: 10.3, Slewing: 1, Slew: 356.1, Az: 249.0, Speed: 34.64, Time: 18:57:42.70
18:58:47.21 | DomePath Info | Time: 11.8, Slewing: 1, Slew: 353.9, Az: 246.8, Speed: 30.10, Time: 18:57:44.17
18:58:47.21 | DomePath Info | Time: 14.4, Slewing: 1, Slew: 346.4, Az: 239.3, Speed: 24.04, Time: 18:57:46.83
18:58:47.21 | DomePath Info | Time: 17.8, Slewing: 1, Slew: 339.7, Az: 232.6, Speed: 19.13, Time: 18:57:50.17
18:58:47.21 | DomePath Info | Time: 21.5, Slewing: 1, Slew: 330.3, Az: 223.2, Speed: 15.37, Time: 18:57:53.90
18:58:47.21 | DomePath Info | Time: 24.8, Slewing: 1, Slew: 323.4, Az: 216.3, Speed: 13.04, Time: 18:57:57.22
18:58:47.21 | DomePath Info | Time: 28.8, Slewing: 1, Slew: 313.9, Az: 206.8, Speed: 10.88, Time: 18:58:01.26
18:58:47.21 | DomePath Info | Time: 33.2, Slewing: 1, Slew: 301.8, Az: 194.7, Speed: 9.10, Time: 18:58:05.57
18:58:47.21 | DomePath Info | Time: 36.6, Slewing: 1, Slew: 294.9, Az: 187.8, Speed: 8.06, Time: 18:58:09.00
18:58:47.21 | DomePath Info | Time: 38.2, Slewing: 1, Slew: 290.3, Az: 183.2, Speed: 7.61, Time: 18:58:10.57
18:58:47.21 | DomePath Info | Time: 40.2, Slewing: 1, Slew: 285.5, Az: 178.4, Speed: 7.10, Time: 18:58:12.60
18:58:47.21 | DomePath Info | Time: 42.2, Slewing: 1, Slew: 280.9, Az: 173.8, Speed: 6.65, Time: 18:58:14.65
18:58:47.21 | DomePath Info | Time: 43.4, Slewing: 1, Slew: 278.5, Az: 171.4, Speed: 6.42, Time: 18:58:15.78
18:58:47.21 | DomePath Info | Time: 44.7, Slewing: 1, Slew: 276.1, Az: 169.0, Speed: 6.18, Time: 18:58:17.07
18:58:47.21 | DomePath Info | Time: 45.7, Slewing: 1, Slew: 273.7, Az: 166.6, Speed: 6.00, Time: 18:58:18.07
18:58:47.21 | DomePath Info | Time: 46.5, Slewing: 1, Slew: 271.3, Az: 164.2, Speed: 5.84, Time: 18:58:18.90
18:58:47.21 | DomePath Info | Time: 48.0, Slewing: 1, Slew: 268.5, Az: 161.4, Speed: 5.60, Time: 18:58:20.37
18:58:47.21 | DomePath Info | Time: 49.7, Slewing: 1, Slew: 263.9, Az: 156.8, Speed: 5.31, Time: 18:58:22.08
18:58:47.21 | DomePath Info | Time: 50.4, Slewing: 1, Slew: 261.5, Az: 154.4, Speed: 5.19, Time: 18:58:22.77
18:58:47.21 | DomePath Info | Time: 51.6, Slewing: 1, Slew: 259.1, Az: 152.0, Speed: 5.02, Time: 18:58:24.02
18:58:47.21 | DomePath Info | Time: 53.3, Slewing: 1, Slew: 254.4, Az: 147.3, Speed: 4.78, Time: 18:58:25.68
18:58:47.21 | DomePath Info | Time: 54.6, Slewing: 1, Slew: 252.1, Az: 145.0, Speed: 4.62, Time: 18:58:26.99
18:58:47.21 | DomePath Info | Time: 55.3, Slewing: 1, Slew: 249.7, Az: 142.6, Speed: 4.51, Time: 18:58:27.76
18:58:47.21 | DomePath Info | Time: 56.2, Slewing: 1, Slew: 247.3, Az: 140.2, Speed: 4.40, Time: 18:58:28.66
18:58:47.21 | DomePath Info | Time: 57.9, Slewing: 1, Slew: 244.9, Az: 137.8, Speed: 4.23, Time: 18:58:30.34
18:58:47.21 | DomePath Info | Time: 59.3, Slewing: 1, Slew: 240.0, Az: 132.9, Speed: 4.05, Time: 18:58:31.73
18:58:47.21 | DomePath Info | Time: 61.7, Slewing: 1, Slew: 234.8, Az: 127.7, Speed: 3.81, Time: 18:58:34.08
18:58:47.21 | DomePath Info | Time: 62.7, Slewing: 1, Slew: 232.5, Az: 125.4, Speed: 3.71, Time: 18:58:35.10
18:58:47.21 | DomePath Info | Time: 64.1, Slewing: 1, Slew: 230.2, Az: 123.1, Speed: 3.59, Time: 18:58:36.56
18:58:47.21 | DomePath Info | Time: 65.7, Slewing: 1, Slew: 225.6, Az: 118.5, Speed: 3.43, Time: 18:58:38.15
18:58:47.21 | DomePath Info | Time: 66.5, Slewing: 1, Slew: 223.2, Az: 116.1, Speed: 3.36, Time: 18:58:38.93
18:58:47.21 | DomePath Info | Time: 67.5, Slewing: 1, Slew: 220.8, Az: 113.7, Speed: 3.27, Time: 18:58:39.92
18:58:47.21 | DomePath Info | Time: 69.3, Slewing: 1, Slew: 218.5, Az: 111.4, Speed: 3.15, Time: 18:58:41.71
18:58:47.21 | DomePath Info | Time: 70.0, Slewing: 1, Slew: 216.1, Az: 109.0, Speed: 3.09, Time: 18:58:42.43
18:58:47.21 | DomePath Info | Time: 71.1, Slewing: 1, Slew: 214.2, Az: 107.1, Speed: 3.01, Time: 18:58:43.49
18:58:47.21 | DomePath Info | Time: 71.8, Slewing: 1, Slew: 213.7, Az: 106.6, Speed: 2.97, Time: 18:58:44.26
18:58:47.21 | DomeHoldups   | 1 holdups > 5s. Dome slew held up at Az. 252.1° (Delay 9s)
18:58:47.28 | (Monitor Detect Dome.Movement Ok  | Dome has moved 146.3° (Az 252.9 to 106.6°) following telescope slew
18:58:47.52 | Do PostSlewSettle            Info | Continuing (Telescope has already settled for >= 5 seconds)
18:58:49.93 | Report PostSlew.ScopeRaDec   Info | Ra= 22 03 51.000 Dec=+48 22 16.00 | Current Epoch
18:58:49.93 | Report PostSlew.ScopeAzAlt   Info | Az= 104.1 Alt=70.2
18:58:49.93 | Report PostSlew.AzAlt(Calc)       | Az= 103.9 Alt=70.1
18:58:49.93 | Report PostSlew.DomeAz       Info | Az= 106.6
18:58:49.95 | (trace) Continue           (trace)| [ SlewScopeNearZenith() Continue ]
18:58:49.95 | Report SlewResults           Info | Reporting Slew Results
18:58:49.95 | Report SlewSpeed(Scope)      Info | Slew Speed (Scope) = 1.5°/s | 60.6° slew in 41s
18:58:49.95 | Report SlewSpeed(Dome)       Info | Slew Speed (Dome) = 2.0°/s | 146.3° slew in 73s


Fig 3. Image Link Freeze at 19:00

19:00:15.53 | Focus Field 22 (trace) ImageLinking   (trace)| Calling DoImageLink (0 \ 0)
19:00:15.54 | Focus Field 22 (trace) ImageLinking   (trace)| Calling Image.InsertWCS(1)
19:00:19.16 | Focus Field 22 (trace) ImageLinking   (trace)| Back from DoImageLink (0 \ 0)
19:00:19.16 | Focus Field 22 (trace) ImageLinking    trace)| Finished x/y loops
19:00:19.16 | Focus Field 22 (trace) ImageLinking   (trace)| BestActiveStars= 0, Match=False
19:00:19.16 | Focus Field 22  Start  Search Distance1 Data | Searching for potential Plate Solutions (Radius 1)
19:00:19.18 | Focus Field 22 (trace) ImageLinking   (trace)| Calling DoImageLink (-1 \ -1)
19:00:19.20 | Focus Field 22 (trace) ImageLinking   (trace)| Calling Image.InsertWCS(1)
19:01:26.68 | (Overseer)      Check  ObsMonitor      Info  | Obs.Monitor is Pending (5s) in section
                                                             'Refresh Dome Picture/Draw Telescope Tracking & Slewing'
19:01:26.88 | (Monitor)       Check  ObsMonitor      Info  | Obs.Monitor has resumed (cycle = 4.7s)
19:01:26.88 | (Monitor)       Check  ObsMonitor      Info  | Breakdown (ms) :
19:01:30.44 | (Monitor)       Detect Dome.Movement   Ok    | Dome has moved 1.0° (Az 106.6 to 107.6°), static time 2.7 mins
19:02:51.83 | (Overseer)      Check  JobQueue        Info  | Task in ImageAndLink/LinkImage is taking a long time? ()
19:03:52.91 | (CCDCamera)     Run    GetProperty     Fail  | Exception in CCDCamera.GetProperty State
19:03:52.91 | (CCDCamera)     Run    GetProperty     Fail  | Error: System call failed. (Exception from
                                                               HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))
19:03:52.95 | (CCDCamera)     Run    GetProperty     Fail  | Exception in CCDCamera.GetProperty Status
19:03:52.95 | CCDCamera)      Run    GetProperty     Fail  | Error: System call failed. (Exception from
                                                               HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))
19:03:54.21 | (CCDCamera)     Run    GetProperty     Fail  | Exception in CCDCamera.GetProperty FilterIndexZeroBased
19:03:54.21 | (CCDCamera)     Run    GetProperty     Fail  | Error: System call failed. (Exception from
                                                               HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))
19:03:54.24 | (CCDCamera)     Run    GetProperty     Fail  | Exception in CCDCamera.GetProperty FilterIndexZeroBased

[ Prev | Next ]

Issue

1)  DeviceHub 7.0.1 Related Issues. Running automated session with AstroMain 3.71.9 & new DeviceHub 7.0.1 isn't practical due to issues/sideffects (18:50 to 20:55).

 
2) Delayed Slews.  Slew to Near Zenith Focus Field 22 (18:57) and indeed all slew seemed anomalous (18:47 - 20:47)

3) CCDSoft/TheSky6. CCDsoft & TheSky6 froze whilst performing Image Link for 2nd Locate Frame for Near Zenith Focus Field (19:00).
 (problem occurred on multiple other Image Links as well (19:00 to 20:31)

4) AstroMain.  AstroMain crashed due a System.NullReferenceException trying to access TheSky6.Utils.ComputeAnglularSeparation (20:17)

Description

1. DeviceHub 7.0.1 Related Issues. Several issues associated with get timely Slewing property values, slewing timeouts, ImageLinking with CCDSoft and TheSky6 both locking up necessiating several user interventions and restart. This caused a frustrating couple of house at the beginning of the session and prevented the acquisition of two important targets ( T CrB and C/2023 A3  (Tsuchinshan-ATLAS)  ) before they descended in western sky and got  too low to image.  These issues are listed individually as minor images, but collectively the rank as a Major Issue.  The issue was worked around in the session by dropping DeviceHub back to the former 6.6.1.1 version, and this saw the problems go away.   Pete Simpson has been contacted regarding the issues and a further update will be given after investigation and analysis of the night's various log files.
    
2. Anomalous Slews. During second Autostart run, the Telescope/Dome slewed (from Focus Field 18) to Focus Field 22 at 18:57.  This was possibly anomalous (60.6° scope slew in 41s, 146.3° dome slew in 73s) and Dome Slew was reported to have held up for 9s at Az. 252.1°.  Watching the Observatory Tab during the slew suggests that the Telescope doesn't start moving for 10s after the Slew command is given (or at least isn't seeing a Slewing = True state for 10s. (See Fig 2).    Examining all the slews made shows that the time of the first Slewing=True (1)  flag doesn't come until T=13.7s, 14.1s, 12.1s,  14.9s, (>241s), (>96.6s), (>93.9s), 28.1s,  15.5s after the Telescope slew is commanded.  After reverting back to DeviceHub 6.6.1.1 at 20:56 all subsequent slew had times till first Slewing=True (1) flag coming after only 3.1s, 4.1s, 3.3s, 2.2s, 3.3s, 2.3s after the slew is commanded, which is consistent to previous sessions using 6.6.1.1.
    
3. CCDSoft/TheSy Freezes during ImageLinking. After beginning Autostart again,  and successfully taking & linking 1st Locate Frame for Near Zenith Focus Field 22, the linking of 2nd Locate Frame froze,  ObsOverseer gave the warning "Task in ImageAndLink/LinkImage is taking a long time?", both CCDSoft and TheSky6 appeared to be frozen. There was no report that ScopeMonitor had frozen and the hang-up didn't release after the usual 5 - 5.3 minute period.   Investigating logfile shows that DoImageLink(0\0) was called, called Image.InsertWCS(1) & returned, after which DoImageLink (-1\-1) was called,  which called Image.InsertWCS(1) at 19:00:19. Things froze at this point. At 19:01:26 ObsOverseer logged message "Obs.Monitor is Pending (5s) in section "Refresh Dome Picture/Draw Telescope Tracking & Slewing", but this quickly resumed.  Starting at 19:03:52 a multitude of exception are encountered by  CCDCamera with messages "Exception in CCDCamera.GetProperty State -  System call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))". (see Fig 3) These errors continued until 19:08:51, when ObsOverseer intervened to 'Restart TheSky6'  in attempt to 'resolve unresponsive TheSky6'.   The Linking image operation was unfrozen and reported inappropriate that image linking was "Solved in 521.0s  (Stars 39, Scale 1.406, Angle 180.1°) ", but with Plate Coords of Ra/Dec2000 of  '00 00 00.000'  / ' +00 00 00.00'.  
    
4. AstroMain Crash (20:17) After attempted Slewing to GCVS T CrB at  20:16, during which the Scope & Dome might not have moved, a 'report line was written with the message "Checking Scope...   Fail   .GetRaDec failed - is the telescope connected ?", after which AstroMain crashed with "System.NullReferenceException (Object reference not set to an instance of an object)" whilst trying to access TheSky6.Utils.ComputeAnglularSeparation() method in the AngularSeparation2()  routine.  Problem is most likely a sideffect of a prior Restart TheSky6 operation which hadn't yet recreated TheSky6 objects including Utils after disposing them and killing/restarting TheSky6.  A Try..Catch.. End Try block should be surround the call to Utils.ComputeAnglularSeparation() method. Job Executor should enter a wait loop whilst a 'Restart TheSky' operation is taking place.

Analysis

1. DeviceHub 7.0.1 Related Issues.   The observation that issues went away after reverting back to DeviceHub 6.6.1.1 shows that the installation of DeviceHub 7.0.1 (not the specifically the ASCOM 7 platform) behaves in a manner that causing sideffects directly or indirectly by exposing some a limitation/issue with the underlying MeadeGeneric Telescope Driver.
    
2. Slewing Problems The log file detailing the slew to Focus Field 22 at 18:57 is shown in Fig 2.
   
   Timeline - Abnormal Slew using DeviceHub 7.0.1  (Slew to Focus Field 22 at 18:57)
   
   (Key:  C = AstroMain program log (my client),  
           MG = Meade Generic telescope driver log)
          DH = DeviceHub Activity log
          TS  = DeviceHub.Telescope log for the TheSky6.Telescope connection
   
   C  18:57:33.250 | Call  to Telescope.SlewToRaDec from client         (time of AstroMain command to TheSky6.Telescope)

C  18:57:33.530 | Slewing: 0, Slew: 0.000, Az: 255.98, Alt: 69.63   (calls to DeviceHub.Telescope)
C  18:57:35.140 | Slewing: 0, Slew: 0.003, Az: 255.99, Alt: 69.63   (client queries DeviceHub.Telescope every 0.2s during slew
                                                                      but only logs when Slewing changes or
                                                                      angular change > 0.1 arc min )
C  18:57:37.410 | Slewing: 0, Slew: 0.006, Az: 256.01, Alt: 69.62          (changes typically occur every 2.5s or so)

MG 18:57:38.918 | IsSlewingToTarget  IsSlewing = False : result =   ( Telescope driver is being asked if Telescope is slewing )
MG 18:57:38.918 | GetSlewing         Result = False                 ( not slewing as the command to slew hasn't been received yet)
MG 18:57:38.918 | Slewing            Result = False                 ( similar records around this time are not shown here)

C  18:57:39.900 | Slewing: 0, Slew: 0.011, Az: 256.02, Alt: 69.61

MG 18:57:40.058 | SlewToCoordinatesAsync Ra=22.0641, Dec=48.3739   (Command received by scope, this is 6.8s after command was given)

MG 18:57:41.482 | IsSlewingToTarget  IsSlewing = False : result =  ( Telescope driver is being asked if Telescope is slewing )
MG 18:57:41.482 | GetSlewing         Result = False                ( Initially not slewing, this is typical for MeadeGeneric)
MG 18:57:41.482 | Slewing            Result = False                ( and doesn't become True until 1s or so later)

MG 18:57:41.835 | SlewToCoordinatesAsync Completed Ra=22.0641, Dec=48.3739 (Telescope has acknowledged slew command)
                                                                           (= 1.78s after driver recieved command)
C  18:57:42.380 | Slewing: 0, Slew: 0.014, Az: 256.03, Alt: 69.61

MG 18:57:41.915 | IsSlewingToTarget  IsSlewing = True : result = [7F] (Telescope responds that it is slewing, =1.9s after
                                                                       it receives command, or 8.7s after slew command
MG 18:57:41.915 | GetSlewing         Result = True                     originally sent by client)

C  18:57:46.550 | Slewing: 1, Slew: 0.893, Az: 248.03, Alt: 73.12    (First time that client sees that the scope is Slewing )
                                                                      ( = 13.3s after slew command sent by client)

TS 18:57:48.393 | SlewToCoordinatesAsync: RightAscension = 22.06414, Declination = 48.37395  (TheSky6 sees this 15.1s after
                                                                                              slew command sent)
                               
MG 18:58:13.142 | IsSlewingToTarget  IsSlewing = False : result =    (Telescope indicates that it is no longer slewing)
MG 18:58:13.142 | GetSlewing         Result = False
MG 18:58:13.142 | Slewing            Result = False

C  18:58:15.040  |  Slewing: 0                                       (Client sees slewing has finished)
   
   
   The problem seen by 6 or more slews made via DeviceHub 7.0.1 whilst trying to goto / image targets during first 2 hours of session.
   
   I didn't have DeviceHub activity logging active during the slew at 18:57,  but I did for a later problem slew at 20:47.
   
   TS 20:47:12.199 | Get RightAscension: 16:00:24 (done)               (calls to DeviceHub.Telescope from TheSky6
TS 20:47:12.502 | Get Declination: 25[B0] 50' 50" (done)

C  20:47:12.530 | Call  to Telescope.SlewToRaDec from client        (time of client command to TheSky6.Telescope)

TS 20:47:13.046 | Get RightAscension: 16:00:24 (done)               (no direct indication that TheSky calls Slew at 20:47:12
                                                                    (but no indication that it has frozen and isn't
                                                                     getting RaDec values from scope)

C  20:47:13.060 | Slewing: 0, Slew: 0.000, Az: 283.38, Alt: 22.41   (calls from client to get slewing value from
                                                                      DeviceHub.Telescope)
TS 20:47:13.175 | Get Declination: 25[B0] 50' 49" (done)

TS 20:47:13.991 | Get RightAscension: 16:00:24 (done)               (other TS GetRa, GetDec calls not shown here  
TS 20:47:14.247 | Get Declination: 25[B0] 50' 49" (done)

DH 20:47:15.526 | Telescope Statuses: Get Slewing: False
DH 20:47:17.157 | Telescope Statuses: Get Slewing: False
C  20:47:17.410 | Slewing: 0, Slew: 0.004, Az: 283.40, Alt: 22.40   (client queries DeviceHub.Telescope every 0.2s during slew
C  20:47:18.800 | Slewing: 0, Slew: 0.008, Az: 283.40, Alt: 22.40     but only logs details when Slewing changes or
                                                                      angular change is > 0.1 arc min )
DH 20:47:18.901 | Telescope Statuses: Get Slewing: False
DH 20:47:19.733 | Telescope Statuses: Get Slewing: False
DH 20:47:19.846 | Telescope Statuses: Get Slewing: False
C  20:47:20.820 | Slewing: 0, Slew: 0.012, Az: 283.41, Alt: 22.40    
DH 20:47:21.301 | Telescope Statuses: Get Slewing: False
DH 20:47:22.213 | Telescope Statuses: Get Slewing: False
C  20:47:22.350 | Slewing: 0, Slew: 0.016, Az: 283.42, Alt: 22.39
DH 20:47:22.437 | Telescope Statuses: Get Slewing: False
DH 20:47:22.501 | Telescope Statuses: Get Slewing: False

MG 20:47:22.991 | Dome Commands: Started fast polling every 1000 ms.  (DH recognises that a slew is happened or about to start)

MG 20:47:22.993 | SlewToCoordinatesAsync Ra=17.2836, Dec=3.4858     (Command to slew received by Telescope
                                                                     = 10.5s after command sent by client)
DH 20:47:23.332 | Telescope Statuses: Get Slewing: False

MG 20:47:23.332 | IsSlewingToTarget IsSlewing = False : result =   ( Telescope driver is being asked if Telescope is slewing )
MG 20:47:23.332 | GetSlewing        Result = False                 ( Initially not slewing, this is typical for MeadeGeneric)
MG 20:47:23.332 | Slewing           Result = False                 ( and doesn't become True until 1s or so later)

DH 20:47:23.364 | Dome Commands: SetFastPolling called when already fast polling.

MG 20:47:24.085 | SlewToCoordinatesAsync Completed Ra=17.2836, Dec=3.4858  (Telescope has acknowledged slew command)
                                                                           (= 1.09s after driver recieved command)

DH 20:47:24.085 | Telescope Commands: SlewToCoordinatesAsync - RA = +17:17:01, Dec 003[B0] 29' 09": (slew started)

MG 20:47:24.868 | IsSlewingToTarget IsSlewing = True : result = [7F]  (Telescope responds that it is slewing
MG 20:47:24.868 | GetSlewing Result = True                          (= 1.88s after telescope driver receives slew command)
MG 20:47:24.868 | Slewing Result = True                             (= 12.3s after slew command originally sent by client)

DH 20:47:24.868 | Telescope Statuses: Get Slewing: True
DH 20:47:25.204 | Telescope Statuses: Get Slewing: True
C  20:47:27.010 | Slewing: 1, Slew: 0.459, Az: 283.43, Alt: 22.93   ( Client sees that the scope is Slewing )
                                                                    ( = 14.5s after slew command sent by client)

TS 20:47:29.046 | SlewToCoordinatesAsync: RightAscension = 17.28370, Declination = 3.48580  (TheSky6 sees this 16.5s after
                                                                                              slew command sent)
                 
C  20:47:46.80  | Slewing: 0                                        (Client sees that scope has stopped slewing) 

   
   DeviceHub reverted back to former 6.6.1.1 version at 20:56
   
   Timeline - Normal Slew using DeviceHub 6.6.1.1  (Slew to AT2024xpy at 21:03, this
   
   C  21:03:05.250 | Call  to Telescope.SlewToRaDec   (time of AstroMain command to TheSky6.Telescope)

C  21:03:05.280 | Slewing: 0, Slew: 0.000, Az: 257.37, Alt: 12.19


MG 21:03:05.297 | SlewToCoordinatesAsync Ra=18.7615, Dec=15.9159   (Command to slew received by Telescope Driver
                                                                    = 0.047s after command given by client)
 
MG 21:03:05.404 | IsSlewingToTarget IsSlewing = False : result =   ( Telescope asked if it is slewing )
MG 21:03:05.404 | GetSlewing        Result = False                 ( Initially not slewing, this is typical for MeadeGeneric)
MG 21:03:05.404 | Slewing           Result = False                 ( and doesn't become True until 1s or so later)

MG 21:03:05.596 | SlewToCoordinatesAsync Completed Ra=18.7615, Dec=15.915  (Telescope has acknowledged slew command)
                                                                           (= 0.30s after driver recieved command)

TS 21:03:06.667 | SlewToCoordinatesAsync: RightAscension = 18.7615, Declination = 15.91598

MG 21:03:06.378 | IsSlewingToTarget IsSlewing = True : result = [7F] (Telescope responds that it is slewing
MG 21:03:06.378 | GetSlewing        Result = True                    (= 1.08s after telescope driver receives slew command)
MG 21:03:06.378 | Slewing           Result = True                    (= 1.12s after slew command originally sent by client)

TS 21:03:06.667 | SlewToCoordinatesAsync: RightAscension = 18.7615, Declination = 15.91598   (TheSky6 sees this 1.4s after
                                                                                              slew command sent)

C  21:03:08.360 | Slewing: 1, Slew: 0.952, Az: 257.38, Alt: 12.19   ( Client sees that the scope is Slewing )
                                                                    ( = 3.1s after slew command sent by client)

MG 21:03:27.880 | IsSlewingToTarget IsSlewing = False : result =     (Telescope responds that it is no longer slewing)
MG 21:03:27.880 | GetSlewing        Result = False
MG 21:03:27.880 | Slewing           Result = False

C  21:03:28.22  | Slewing: 0                                         (Client sees that scope has stopped slewing) 
    
   Remaining 32 slews made during session were all normal (again all using DH 6.6.1.1).
   

Ideas

The intermediate client , TheSky6 in this case, is seemingly kept waiting 6.8s whilst it waits for response back from DeviceHub / MeadeGeneric from its call to SlewToRaDec, normally this takes only 0.05s.  This delay could cause certain sideeffects with TheSky unexpectedly freezing during the intervening period?,  Calls to TheSky6 by my client program during this time to get its view on Telescope Position (different from DeviceHub, because it goes through TPoint modification) are unexpectedly held up for 7s or so.   I see this as a Monitor 'Pending' status in a Client Thread which resumes once DeviceHub.Telescope responds back from its call to SlewToCoordinatesASync.

Problem seems to be that the call to telescope SlewToCoordinatesAsync is being held up somewhere either with TheSky6 or within DeviceHub.Telescope for 6.8s - 10.5s or so and DeviceHub likely dropping back to Normal Polling (every 5s) as it doesn't see the Scope is Slewing. These combine to delay the time that client first sees Telescope as Slewing by total of around 13-15s, and in some cases it never sees the scope is slewing. My client expects that scope will have reported that it is slewing with 5s of command being given, and if it doesn't it has to assume that slew has possibly been completed or not necessary if no coordinate change was required.

Conclusion

• The problems experienced in S1249 Session are very likely due in someway to the change in DeviceHub design where by Get Telescope Property requests sent by client programs are passed to Telescope Driver/Telescope, rather than being served by DeviceHubs own cache. 
   
  Initially it wasn't clear whether the problem lies with TheSky program, DeviceHub's SlewToCoordinatesAsync method or the MeadeGeneric Telescope Driver.   Suspicion is on TheSky6 application due to its role in commencing the Slew Process and its part in the PlateSolving (InsertWCS) issue involving CCDSoft & TheSky6.
   
• It was eventually concluded (2024-10-27) that the problem lay with TheSky6 which was carrying a telescope cross-hair update frequency setting of 500ms which was too low for the new DH 7.0.1 design that sends Ra/Dec property requests to the Telescope Driver/Telescope (taking around 530ms per Ra/Dec pair (range 200-1020ms), leading to TheSky6 being unable to properly serve its GUI and CCDSoft requests.
   
• It was eventually concluded (2024-10-27) that the issue can be resolved by relaxing TheSky6's telescope cross-hair update frequency setting to a new figure of 2000ms.

Actions:

•   More Tests. Carry out more tests to understand the problem further and attempt to find a solution so that DeviceHub 7.0.1+ can be used, meaning the Observatory can come off the former DH 6.6.1.1 version
  Done 2024-10-27
   
•   DH 7.0.2 RC3. Install DH 7.0.2 RC3 (beta), provided by Peter Simpson (ASCOM) which contains extra logging from DH's SlewToCoordinatesAsync method
  Done 2024-10-27

Update 2024-10-26  (Additional Analysis) 

1. DeviceHub serves several clients under normal Observation operation :
   
   - DH itself which polls the telesscope every 3-5s (normal polling) or every 1s (fast polling)  
   
   - TheSky6 application that indirectly polls DH every every 0.5s           (Frequency could be increased to 1s - 3s)
   
   -  AstroMain Application's Scope/General Monitor that poll DH every 1.5s or so.      (This is a relatively reasonable rate picking up whatever DH has as the latest value which might be which could be values every 5s under normal polling          (AstroMain) caches telescope properties internally and avoids polling DH twice within 0.1s for the same property)           (If the current value right now for a particular reason, the local cache value is passed over even it is within its retain period, and DH is polled directly           (If DH goes directly to the Telescope for all requests, the cache retention time could be increased to say 0.5s to reduce polling).
    
    - AstroMain Slew Monitor that polls DH every  0.2s during a slew. 
      (This was value was chosen in order to get coordinates as quickly as possible after DH has received & updated any cached/values
      AstroMain should therefore be a maximum  of 1.2s 'behind' the scope's / driver's latest value .     
     (AstroMain's fast polling rate of 0.2s could be changed  to every 1 - 1.2 s or so if DH is not sending a cached value )  
    
   - AstroGuard which polls DH every 2s or so.      (Observatory is mainly operated in Automated mode and the AstroGuard program is intended to stop Telescope Tracking (to stop risk of Cable Wrap) in a situation where it spots that my Main Program crashes/hangs for any reason, or the scope otherwise goes into a no-go zone.     (it might not need to poll scope position, and instead assume that the Main Client has things under control unless it not updating the  "I'm Alive"  Flag file.
    
   - PHD2 which polls scope when it requires (normally this is during guiding and not during conventional slewing)
    
   Basically there's 5 programs needing telescope coords etc every 1-3s  With DH 7.0.1 not supporting use of DH cached values to clients, a worst case might imply a roughly 4 -fold increase in amount of traffic & requests going to the Telescope/Driver.   This may not be critical all the time, but during slewing it would evidently seem that it is.  
    
   
   
2. Revisting data from the 3 earlier cases and adding data from a 4th case, the following information can be gleaned, which looked at load on Telescope Driver using two measures (no. of log rows &,  in brackets no. of Get RightAscension calls) , looking at 15s period before the request to slew and the 15s period after the request. 
   
   (Because of the delay in slew starting under DH 7.0.1 the 'after' stats were counted for both the 15s period after  the slew was requested by client and the 15s period after the slew request was logged by the MeadeGeneric Driver. For DH 6.6.1.1 cases the values are roughly the same as they correspond to very roughly the same time period, but they differ significantly for the DH 7.0.1 cases).  
   
   Case                                        No. of  MeadeGeneric Log Rows  (No. of Get RightAscension calls)
   
                                                    in 15s period before    in 15s period after     in 15s period after
                                                    Slew Requested           Slew Requested         Slew Request logged by Telescope Driver
   
   Case 1 18:57 (DH 7.0.1)               1438  (57)                     1209  (38)                  991  (29)
   
   Case 2 20:47 (DH 7.0.1)               1378  (56)                     1380  (50)                  967  (29)
   
   Case 3 21:03 (DH 6.6.1.1)              665  (6)                       1342  (15)                 1315  (15)
   
   Case 4 21:13 (DH 6.6.1.1)              477  (7)                        1295 (14)                 1296 (14)
   
3. Looking at the two DH 6.6.1.1 cases we can see an increase in the number of rows logged by Telescope Driver when we go into Fast Polling /  whilst making a slew. The frequency of Get RightAscension calls increases from 1 call every 2.3s  to 1 call every 1.0s.
   Loading on Telescope Driver/Telescope  seems reasonable with 6.6.1.1 and no problems are noted.
   
   Looking at the two DH 7.0.1 cases the data was somewhat surprising as there appears to be a decrease in number of rows logged by Telescope Driver during a slew (we don't seem to see effect of fast polling).  The number and frequency of Get RightAscension calls also  decreases during a slew from around 1 call every 0.26s to 1 call every 0.5s.  The loading on Telescope Driver/Telescope has clearly increased under 7.0.1 compared to 6.6.1.1
   
   It was thought that there might be a 4 -fold increase on loading on the Telescope Drive. Looking at no. of log rows in a 15s period before a slew there seems to be a 2.5 fold increase, but the mumber of log rows is just a proxy for loading level.  If the number of Get RightAscension calls in the 15s period before a slew is considered there is an 8.7 fold increase (higher than first throught).
   
   Possibly when Get RightAscension is requested in causes both a Get RightAscension and a Get Declination call,    and when a Get Declination is requested it causes both a Get Declination and a Get RightAscension call. i.e. there is a doubling up effect ?
   
   It was surprising that the number of rows logged by the telescope driver during a slew are actually less with 7.0.1 than with 6.6.1.1.
   
   It was also of surprise that the calls to Get RightAscension under DH 7.0.1  are higher before the slew  (56-57 in 15s) than during the slew (29 in 15s) when faster polling by DH and by Slew Monitor would be predicted to produce  more calls (not less)
   .   
   There is activity that AstroMain application does in the lead up to a slew like reporting pre-slew coordinates etc, but this can't really explain the higher rates to Get RightAscension prior to slew.  ( Perhaps DH 7.0.1 suppresses the property requests going directly to the Telescope Driver when it is using Fast Polling ? That's the only thing that can be thought of to explain this observation.)
    
4. It was surprising that the number of row logged by the telescope driver during a slew are actually less with 7.0.1 (967-991) than with 6.6.1.1 (1296-1315).  Perhaps this is an indication that Driver is struggling under the load in some way ? I'm not sure ?
   
   Perhaps Telescope Driver is already overwhelmed (and running 6-7s behind on requests) when the Slew is needed and doesn't manage to pick up request to slew for 6-7s from when it is sent ?. Various things could be done to reduce the problem in the Observatory system/environment, but it might be a very much trial and error approach to get the right balance. 
    
5. The Observatory could simply staying with DH 6.6.1.1 (as that is working without issue), but if after a certain point continuing with DH 6.6.1.1 become untenable (from a support point for view) and/or the other advantages in DH 7.0+ (like better logging capabilities) outweigh the pain from the issues experienced and there was no change to DH 7.0 to otherwise address the issue then I'd be forced to make changes to the Observatory system anyway.  
   
6. The work here has focused here on the Telescope Driver side of things, but similar issue could be present with the Dome Driver where its request load will presumably have also increased.