Incident

Session S688 had been due to finish its job queue at 04:47 and then shutdown the Observatory equipment. However upon checking the status at 05:30, it was noticed that the Observatory Shutter was open and directed approximately due West (i.e. 180 deg from its Park Position).

There was no noise was coming from the observatory (i.e. no noise to indicate that the dome motors turning furiously or anything)

Connecting remotely to the Observatory revealed that a Telescope Slew to the 1st Mosaic Point for Target 25/31 (NGC 1817) was held up by an ASCOM Meade LX200GPS/R message from TheSky program saying that 'The selected coordinates are below the horizon'.

Operations appeared to have stalled at this point waiting for user input that couldn't be provided. 

ASCOM Meade LX200GPS/R message 'The selected coordinates are below the horizon'

Response

A Screen capture was immediately taken to show the positions and status of various programs & windows and the precise ASCOM/Meade  message produced in TheSky program (05:33). 

Hit Ok button on the TheSky's '..below the horizon' dialog (05:33) where upon control again passed back to the Job Executor which then continued with another attempt to slew to Mosaic 1 target 7. TheSky again tried to make a slew and re-showed the same message dialog.
Hit the 'Abort Job' button on AstroMain. This brought the Session back to Manual. (no log/report records of this)

Tried to find out telescope coordinates via POTH but it was unresponsive, or at least couldn't be made to come to the front.
Closed the Telescope Connection in TheSky. This had the unexpected but immediate effect of causing the AdvMain program to crash (05:34)

The POTH.Hub window couldn't be brought to the front or otherwise shown. It seemed to be 'Not Responding'.
This status was confirmed later on (05:41) when looking at the list of applications in the Task Manager.

The shutter was closed (42s) using AstroGuard's 'Close Shutter' button (05:35)

Attempt the made to Park Dome using Astroguard (05:36), at which point the Astroguard window then became very slow to move around and appeared to freeze. Dome didn't appear to be rotating.

Visited the Observatory to check equipment status & position of the scope.  (05:40)
Found the Dome motors running but not turning the dome.   Found the Scope pointing c. 205-210 deg Az, c. 15 degs Alt
Twiddled with Motor tension knobs until they were eventually able to get a grip on turn the Dome.  Dome continued moving to its Park Position.

Upon reaching Park Position AstroGuard program then came back to life (05:46).  It reported that Scorecard had been not active for 10-12 mins.  The time that the AstroGuard was held up waiting for Dome Parking to finish.

Used AutoStar II handset to Park the Scope.   Turned Telescope Power Off via Pegasus UPB Software

Disconnected PHD2, POTH still unresponsive.  Killed CCDSof, POTH etc using AstroLaunch's special backdoor  'kill stranded processes button

Incident Timeline.

02:22 
Started preceeding target (24/31, NGC 2820). 
Checked location RA: 09 21 46.000, Dec: +64 15 30.99, (2000),  Azimuth 19.4° , Altitude, 35.0°
Warning  given during slew  'Telescope.SlewToRaDec task in SlewToRaDecQSO/Slewing is taking a long time?'
Slew took 4.95 mins.  (indicates a problem with dome rotation motor tension)
Centered target and took main frames normally. Guiding normal.
Finished NGC 2820 at 02:45

02:45
Started target (25/31, NGC 1817). This was to be a mosaic target (which is not a regular operation)
Checked location Con: Taurus,  Dec: -12.28°, Azimuth 157.0° 
Read in Mosaic Coords  (12 points of which 1st point was  Mosaic A1 18.866997 -6.497842)
Began slew to target
ObsPics showed Scope slewed to Az 157 / Alt 18 within 60s  or so (which corresponds to RA 01 49, Dec -12 16)

02:50
Finished Dome slew to target.
Checked scope position wrt to specified slew location (current epoch coords). No significant discrepency (<0..28 arc min only).
ObsPics showed Dome slewed slowly to Az  159.2° over 5 mins

02:51  Start slew to Mosaic Point 1  (This had coords from the file of 18.866997  -6.497842)
ASCOM Driver/Meade puts up the message 'The selected coordinates are below the horizon'.
Dome slews to Az 259°
Job Queue freezes (waiting for slew to finish)
Obs.Manager apparently also freezes (as evidenced by the fact that it doesn't output messages saying the slew is taken a long time/a very long time until user interventation in the morning when it comes back to life.

02:55 to 05:30
Telescope tracks to Az 200°, Alt 19°

05:33
User Interventation unlocks the frozen slew operation and the job queue continues.
Job executor makes a repeated attempt to slew to 1st Mosaic Point.
Again Meade ASCOM driver give s message that selected coordinates are below the horizon.
Job executor tries to Slew to 2nd Mosaic Point.
Similar Issue.  Eventually user initiates an action that leds to AdvMain program crashing (05:35)

Analysis

There details of the incident and the timeline raise a number of questions which have been investigated in detail:

Why did telescope not slew correctly to NGC 1817 ?

NGC 1817 coordinates from TheSky:
RA: 05h 13m 33s Dec: +16°42'17"(current)
RA: 05h 12m 27s Dec: +16°41'00" (2000)

NGC 1817 coordinates in AstroPlan :
RA: 05 12 27.001,  Dec +16 40 59.99 (2000)

Whilst log file shows the correct RaDec2000 coordinates, it shows highly invalid coordinates for RaDec (Current Epoch)  

02:45:43.66 | NGC 1817 Data Object Coords (2000) | RA= 05 12 27.001 Dec= +16 40 59.99 | Epoch 2000
02:45:43.66 | NGC 1817 Data Object Coords | RA= 01 49 23.459 Dec= -12 16 57.48 | Epoch Current
02:45:43.66 | NGC 1817
02:45:43.66 | NGC 1817 Data Target Offset | 0.00 arc min (W= 0.00', N= 0.00')
02:45:43.66 | NGC 1817 Data Target Coords (2000) | RA= 05 12 27.001 Dec= +16 40 59.99 | Epoch 2000
02:45:43.66 | NGC 1817 Data Target Coords | RA= 01 49 23.459 Dec= -12 16 57.48 | Epoch Current

Normally J2000 and current epoch coordinates will differ by a few arc minutes (as with the preceeding target, NGC 2820), however for NGC 2817 the discrepency is in the order of many degrees.

Whilst the J2000 coordinates are in Cetus,  and would have been positioned at Az 95 / Alt 23 in the sky at that time, the Current Epoch coords are in Taurus, and corresponded to a position Az 157 / Alt 18. 

The slew was performed by the 'SlewToRaDecQSO' routine using the command

  Telescope.SlewToRaDec(dRa, dDec, T.TargetName)

where

  dDec = T.Dec + JogN / 60
  dRa = T.Ra - (JogW / Math.Cos(T.Dec / DR)) / (15 * 60)

since JogN and JogW were chosen to  0.0 for NGC 1817, the Telescope was command to go to the current epoch coordinates T.Ra and T.Dec

Whilst Scope and Dome didn't slew to the actual location coordinates of NGC 1817,  they did both slew to the coordinates that they were sent to.

How did these invalid current epoch coordinates come to be generated ?

It's not understood why the epoch coordinates for NGC 1817 (T.Ra and T.Dec) where wrong. 

The call to 'SlewToRaDecQSO' routine  is made from the 'ImageTargetT_GotoTarget' routine for normal targets , whilst it is made from the  ImageMosaicT_QSO routine for mosaic targets

T.Ra and T.Dec are set from the 'LookupKeyTargetInfo' and 'LookupKeyTargetInfo2' routines . In the former routine Ra,Dec are set by looking up their values using the relevant ObjectInfo.Property, whilst it the second routine Ra,Dec are calculated using TheSky's Utils.ConvertRADecToAzAlt function based on T.Ra2000, T.Dec2000 coordinates)

For normal targets  'LookupKeyTargetInfo'  is called in ImageTargetT_QSO before calling 'ImageTargetT_GotoTarget' (and thus to 'SlewToRaDecQSO' )

'LookupKeyTargetInfo' is also called by UpdateTpnNextTargets()  .  Is it possible that T.Ra, T.Dec  values for NGC 1817 became overwritten by calls from UpdateTpnNextTarget ?.   Looking at the next 3 targets it is confirmed that none of them or indeed any of the remaining targets had these coordinates.

It is noted that the T.Ra/T.Dec coordinates slewed to match the coordinates of ARP 4 which happened to modified in database a few days beforehand (target ID 596).    NGC 1817 was the 25th Target of the plan whilst ARP 4 happened to be the 25th Target in the V1 Plan executed in the proceeding session 2019-08-31. It is suggested the T.Ra, T.Dec values hadn't been reset for the 25th element of the TIS array since ARP 4.  The V2 plan for 2019-08-13 only had 6 targets and wouldn't have overwritten the 25th array position.

Why did Poth stop responding ?

It's not entirely understood why POTH window stopped responding . It is assumed it may happened because the ASCOM driver was still busy waiting for response from the user or later on because AstroMain had crashed.

Why did Telescope attempt to slew to below the horizon ?

Coordinates for 1st Mosaic Point (Mosaic A1) were read in from file as  18.866997 -6.497842  (= RA 18 52 01.18, Dec -6 29 52 J2000)
and the telescope requested to slew to the Current Epoch Coords of  18 53 04.266 Dec= -06 28 16.63.

This Mosaic A1 location corresponded to Az 261, Alt -2 at the time, which is just below the horizon.
Whilst the Dome Slewed to Az 259 (ie the 'correct' location for the shutter), the ASCOM driver or TheSky itself recognised that the coordinates would be below the horizon and put up the message for the user.   WIth no one to acknowledge the dialog message, control remained with ASCOM driver and prevented further operations by the job executor whilst it waited in vain for slew to 'finish' or be 'cancelled'

These mosaic coordinates, located near M11 in Scutum are a long way from NGC 1817 target in Taurus.

Why did the mosaic coordinates file have incorrect coordinates ?

This is investigated separately (see Investigation - Incorrect Mosaic Coordinates).

It turns that the problem was due to insufficiennt care & checks when creating the mosaic framework in TheSky6.

 

Principal Issues and Actions

Mosaic Target 25/31 (NGC 1817) did not not slew to the Ra/Dec location , but slewed to that of a target from the last session
>> Fixed bug in 'ImageMosaicT_QSO'  so that routine now calls the LookupKeyTargetInfo' routine

Telescope slewed to Individual Mosaic Points that lay below the horizon
>> Add Altitude check to SlewToRaDec routine that is used by the 'TakeMosaicPoint' routine

Mosaic Coordinates for NGC 1817 were Incorrect
>> User must take more care when creating Mosaic Framework for a target.
  (user needs to use the 'Get From Virtual Sky' button to define the center coordinates over the intended target,
    and to check the framework / coords after the Mosaic has been 'applied'.)

- AstroPlan could check each mosaic coordinate point is within a set tolerance offset from the target object coordinates

- AstroMain must check each mosaic coordinates point before requesting telescope to slew to that point, and only proceed if it is safe to do so. (TakeMosaicPoint)   . Check needs to be in SlewToRaDec

- Update 'ImageMosaicT_QSO'  routine to call LookupKeyTargetInfo in order to correctly set T.Ra, T.Dec object coordinates for Mosaic Target

The coordinates of the Mosaic points for NGC 1817 were incorrect and were many degrees away from the target object itself.  
These mosaic coordinates, where located near M11 in Scutum rather than around the NGC 1817 target in Taurus.

This led to a serious operation problem ;  whilst error checking ensured that the target object was above the horizon, there were no checks to ensure that individual mosaic were also above the horizon, and this ultimately led to the Job Queue and ObS.Manager freezing up until unlocked by user intervention in the morning.

Why did the mosaic coordinates file have incorrect coordinates ?

The coordinates file 'NGC 1817 3x4.dat' was generated by script at 2019-08-17 12:55 after defining a 3 x 4 mosaic using the Mosaic tool in TheSky6.  It is assumed that either the mosaic was defined over a different target to NGC 1817 or the script generated incorrect coordinates in some way.

In an attempt to replicate or at least understand what might have caused the problem, a new  mosaic was built for NGC 1817.

NGC 1817 was centered in TheSky view, orientated N up and the Mosaic Tool opened.
Interestingly the Mosaic Coords showed up as RA  18.851666, Dec -6.266667 
(these are not dissimilar to those of the first mosaic point used during the live session1).

If Apply button is then clicked with first clicking the 'Get From Virtual Sky' button , the background instantly sets to the M11 with mosaic built over it.  The Object Information dialogue remains as NGC 1817, and if the user hasn't noticed this the 'wrong' mosaic cordinates gets picked up and written to the Mosaic File by the script .

It happens that a Mosaic was created in TheSky6  for M11 on 2018-05-21.   Whilst a single mosaic has in the created in the meantime  for NGC 7000 on 2018-10-20, the TheSky6 definiton file and the mosaic centre coords may not have been saved at that time.

When 'Get From Virtual Sky' button is clicked before using the Apply button,  then the correct mosaic framework is constructed and can be saved to file with usual script in which 1st mosaic point is      Mosaic A1 5.223383 16.451970  
This now compares well with NGC 1817's coords of RA: 05h 12m 27s Dec: +16°41'00" (J2000).

Action:
- User must take more care when creating Mosaic Framework for a target.
  (user needs to use the 'Get From Virtual Sky' button to define the center coordinates over the intended target,
    and to check the framework / coords after the Mosaic has been 'applied'.)

- AstroPlan could check each mosaic coordinate point is within a set tolerance offset from the target object coordinates, before target is saved  & before it is included in a live observing plan.

Replicating the Incorrect Mosaic Coordinates Issue All set-up ready to make 3 x 4 mosaic framework over NGC 1817 using TheSky6. (but forgot to click Get From Virtual Sky button)

Result of Applying the Mosaic without clicking the 'Get From Virtual Sky' button first is that the Mosaic Framework is built over the last recorded Mosaic Center (which happened to M11), rather than the intended target above.

Upon attempting to park the dome at 05:30 in the morning and after the Dome had been stationary for more than 2.1/2  hours, it was found the rotation motors wheels were unable to get sufficient leverage to physically rotate/drive the Dome Roof. The motor wheels would slip against the roof flange which was quite wet with dew by this time.   The tension wheels had to be adjusted in order for the wheels to get sufficient grip and move the dome.

The dome was noted to taken a long time (372s or 6.2 minutes)  for it to to park at the end of another recent session (2019-08-28, S686)

Examining the 'timings' data from the session shows that the slewing times from  first 2/3rds of session was generally fine, but problems developed in the last 1/3rd of the session  (particularly Target No's 18, 19, 22, 23 and 24 which were taking  4 - 10 mins to finish a slew

Graph showing Slewing Time per Target (2019-09-01, S688)

Daytime testing was conducted the following day to check that the tension knob adjustments were working and dome slewing was again at a normal pace with judder, slipping or delays.

Specific tests where made for the particular slews that had taken the longest times during the previous night's session
(Az 110 to 326°, Az 190 to 109° & Az 323 to 76°), and the slew that caused problems parking at end of session S686 (Azim 270 to 90°)

Issue was eventually found to be due to a loose / badly worn drive wheel connector (see Pulsar Dome - Loose Rotation Drive Wheel, 2023-10-27) and was fixed by installing a replacement drive wheel in Nov 2023 (see Pulsar Dome - Replacement Rotation Drive Wheel, 2023-11-17)

Stars on image frames C/2018 W2 (Africano) were noticed to have a rather slanted rain drop character. Is problem poor guiding/ poor telescope balance /atmospheric seeing near horizon ?  Issue first noted in last session, but seen with some other targets in the past. It was decided to investigate.

Details of investigation shown below.   However the summary is that the exact cause of the image quality issue is not found.  

C/2018 W2 (Africano) Video sequence shows motion of comet against background stars but also shows an image quaility issue with rain drop like stars (see Investigation : Image Quality Issue - C/2018 W2 (Africano)

Animated CCD Image (100% size, linear scale, cropped) 9 x 45s exposure, 3x3 binning, C Filter 2019-09-02 01:12 UT (#688185-93) 12" LX200R (at f/10.4) + ST-10XME Auto-guided using TS 80mm APO, ZWO ASI178MC & PHD2

Single CCD Image Frame (150% size, linear scale, cropped) 45s exposure, 3x3 binning, C Filter 2019-09-02 01:12 UT (#688190) 12" LX200R (at f/10.4) + ST-10XME Auto-guided using TS 80mm APO, ZWO ASI178MC & PHD2

- Frames were autoguided, and nothing abnormal was noticed.  

  Autoguiding                           
  Guidescope Guiding          Ok        Using PHD2 (ZWO ASI178MC), Guide Exposure 1.5s
  Send Guide Request
    Guiding ...               Ok        Guide Request Returned
    Lock Position             OK        X: 463.2, Y: 1222.1
    Star Selected             OK        X: 463.2, Y: 1222.1
    Lock Position             OK        X: 463.4, Y: 1221.6
    Guiding ...               Ok        Guiding Started

  Imaging                               
  Using C filter...
  Taking 45s image (C)...     Ok        [00688185] 01:08:13 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.4"      0.9" / 2.2", SNR: 431.9, SM: 594285, Guide: 100% (Lost: 0%), N: 17
  Taking 45s image (C)...     Ok        [00688186] 01:09:03 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.4"      1.4" / 2.0", SNR: 415.4, SM: 585819, Guide: 100% (Lost: 0%), N: 17
  Taking 45s image (C)...     Ok        [00688187] 01:09:52 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.6"      1.5" / 2.2", SNR: 414.4, SM: 585304, Guide: 100% (Lost: 0%), N: 18
  Taking 45s image (C)...     Ok        [00688188] 01:10:42 (UT)  Full 3x3 45s C
    Guide Stats  rms error    1.8"      0.9" / 1.5", SNR: 427.4, SM: 585007, Guide: 100% (Lost: 0%), N: 19
  Taking 45s image (C)...     Ok        [00688189] 01:11:31 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.5"      1.3" / 2.1", SNR: 415.8, SM: 582531, Guide: 100% (Lost: 0%), N: 16
  Taking 45s image (C)...     Ok        [00688190] 01:12:28 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.0"      1.2" / 1.7", SNR: 419.7, SM: 581529, Guide: 100% (Lost: 0%), N: 20
  Taking 45s image (C)...     Ok        [00688191] 01:13:20 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.2"      1.3" / 1.8", SNR: 428.8, SM: 596220, Guide: 100% (Lost: 0%), N: 16
  Taking 45s image (C)...     Ok        [00688192] 01:14:09 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.0"      1.3" / 1.5", SNR: 421.9, SM: 594852, Guide: 100% (Lost: 0%), N: 17
  Taking 45s image (C)...     Ok        [00688193] 01:14:58 (UT)  Full 3x3 45s C
    Guide Stats  rms error    2.1"      0.9" / 1.9", SNR: 397.1, SM: 577548, Guide: 100% (Lost: 0%), N: 16

  Guiding Results             7.6       mins (elapsed)
    Guide Stats  rms error    2.2"      1.2", 1.9", SNR:419.4, SM: 586802, Guide: 100% (Lost: 0%), N:157 (0)
  Stop Guidescope Guiding

- Autoguiding must have been on a star as the comet itself was too faint to have been picked up by the guidescope / ZWO ASI camera system.

- There was no image quality issues with targets taken directly before and directly after C/2018 W2 (Africano)

- Target Altitude (61°) was high and well away from any horizon / building / high airmass effects

- Sky Conditions were normal  , clear (10) before frames,  clear (8) after frames. Wind was very light.

- All Sky shows no anomalies and reasons for poor image quality.

AllSky Image at time of C/2018 W2 (Africano) frames