David's Astronomy Pages
Notes - Session 847 (2021-01-08)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
 
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (3),  Minor Issues (4),  Continuous Improvement (10)
Bullet Images from 2021-01-08 >>
Bullet Investigation:  Job Queue stopped for 1 hour after which targets executed taken for 3.1/2 hours with closed dome shutter
Bullet Investigation:  Shutter Opening failed after 10-27s with ShutterStatus=5 (Error)

Session Aims & Highlights (2021-01-08)

Main aims

  1. Targets. Acquire images of a selection of variable stars, nearby stars, comets and deep sky targets as allowed by conditions.

Equipment & Software

Highlights

Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Black Points correspond to Guide Runs 26 to 36 , where guding failed due to a closed observatory dome issue.
Apart from that the Guiding was relatively good.
Image
Image
  
Sky Conditions (Locate Frames)
Data for second part of night is erronous as Observatory shutter was unknowingly closed
Image
Chart unavailable for this session
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
  
Pre-Session Weather Forecast (API data from darksky.net)
Pre-Session Weather forecast showed initial snow showers followed by slight clearing (85-91% cloud cover),
followed by overcast skies,  with possible sleet/snow showers moving to later to rain showers
as temperatures rose from -2 degC at dusk to 4 degC at dawn
Wind calm/light in first part of night, but strengthening later reached 17-19 mph by dawn.

In the event skies were very clear from dusk to 22:30 before clouding over for the remainder of the night with
prolonged rain shower at rain shower at 02:30-03:15.
Minumum Temperature of -4 degC at 22:00 before warming later in night under cloud.
Winds relatively light in first half of night, but strengthening towards dawn.
Image
Image   
  
Session Event Log
Time     Event Detail
16:38:53 Session Monitoring AutoStart monitoring for Live Session opportunity starting between 16:54 & 06:35
16:54:20   AutoStart Waiting AutoStart waiting till earliest start time at 16:54
16:54:21   AutoStart Delayed Session delayed due to cloud
17:08:27 Session AutoStarting Session autostarting (17:08)
17:08:57 Session Created Live Session Created (2021-01-08 S00847, ImageSaveNum: 847001)
17:08:59   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
17:10:55   Services Started Observatory Services started
17:11:04 Observatory (Auto) Observatory placed in Fully-Automated Mode
17:11:07 Session Pending Session pending (2021-01-08)
17:11:09 Session Initiating Session initiating (2021-01-08)
17:11:11   Plan Requested Observing Plan requested from AstroPlan (1.18)
17:12:08   Plan Loaded Observing Plan loaded to queue (2021-01-08 V1 BEST PLAN, Plan ID: 582)
17:12:21   Camera1 Connected SBIG Camera Connected (set point -25°C)
17:12:25   Telescope Connected Telescope Connected (TheSky6)
17:12:36 Session Equilibration Session ready for dome & camera equilibration
17:13:52   Dome Opened Dome opened (opening time 45s)
17:14:08   Equilibration Started Dome Equilibration started
17:19:08   Equilibration Ended Dome Equilibration ended (5.0 mins total)
17:19:10 Session Running Session running
17:19:12   Queue Started Observing Queue started (42 targets selected)
17:20:30       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
17:20:32       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
17:22:47       Focusing Completed Foc2 AutoFocus Completed (Profile No 1, wide)
17:24:35       Focusing Completed Foc2 AutoFocus Completed (Profile No 1)
17:24:51       Focusing Checked Foc2 Focusing Check Succeeded
17:31:07     Target Started (1/42) Target started (1/42, 156P/Russell-LINEAR)
17:33:45       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
17:37:31       Focusing Completed Foc1 AutoFocus Completed (Profile No 2)
17:39:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 2)
17:48:13     Target Completed Target completed (1/42, 156P/Russell-LINEAR)
17:48:15     Target Started (2/42) Target started (2/42, 277P/LINEAR)
17:51:05       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
17:53:03       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
18:03:43     Target Completed Target completed (2/42, 277P/LINEAR)
18:03:45     Target Started (3/42) Target started (3/42, GSC 1224:29)
18:06:53       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
18:13:14     Target Completed Target completed (3/42, GSC 1224:29)
18:13:16     Target Started (4/42) Target started (4/42, 398P/Boattini)
18:16:03       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
18:18:00       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
18:26:06     Target Completed Target completed (4/42, 398P/Boattini)
18:26:49     Target Started (5/42) Target started (5/42, GCVS SS Cyg)
18:30:10       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
18:33:10       Focusing Completed Foc1 AutoFocus Completed (Profile No 5)
18:36:49     Target Completed Target completed (5/42, GCVS SS Cyg)
18:42:01     Target Started (6/42) Target started (6/42, Nova Per 2020 w/AT2020abap)
18:45:10       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
18:47:07       Focusing Completed Foc1 AutoFocus Completed (Profile No 6)
18:55:09     Target Completed Target completed (6/42, Nova Per 2020 w/AT2020abap)
18:56:53     Target Started (7/42) Target started (7/42, IC 2458)
19:00:21       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:02:17       Focusing Completed Foc1 AutoFocus Completed (Profile No 7)
19:17:51     Target Completed Target completed (7/42, IC 2458)
19:19:15     Target Started (8/42) Target started (8/42, WDS GRB 34)
19:22:46       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:24:37       Focusing Completed Foc1 AutoFocus Completed (Profile No 8)
19:26:23     Target Completed Target completed (8/42, WDS GRB 34)
19:27:30     Target Started (9/42) Target started (9/42, NGC 514 w/AT2020uxz)
19:30:50       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:32:47       Focusing Completed Foc1 AutoFocus Completed (Profile No 9)
19:44:01     Target Completed Target completed (9/42, NGC 514 w/AT2020uxz)
19:44:52     Target Started (10/42) Target started (10/42, PGC 3792)
19:47:36       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:49:34       Focusing Completed Foc1 AutoFocus Completed (Profile No 10)
20:25:44     Target Completed Target completed (10/42, PGC 3792)
20:27:18     Target Started (11/42) Target started (11/42, IC 161)
20:32:19       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S)
20:50:14     Target Completed Target completed (11/42, IC 161)
20:50:16     Target Started (12/42) Target started (12/42, GCVS DY Per)
20:52:16       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:54:14       Focusing Failed Foc1 focusing failed - failed QC check (Ambiguous)
20:58:35     Target Completed Target completed (12/42, GCVS DY Per)
20:59:48     Target Started (13/42) Target started (13/42, UGC 1161 w/SN2020acch)
21:02:22       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
21:19:54     Target Completed Target completed (13/42, UGC 1161 w/SN2020acch)
21:22:10     Target Started (14/42) Target started (14/42, GCVS RX And)
21:24:46       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:26:44       Focusing Completed Foc1 AutoFocus Completed (Profile No 12)
21:29:22     Target Completed Target completed (14/42, GCVS RX And)
21:30:30     Target Started (15/42) Target started (15/42, GCVS U And)
21:34:44       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:36:02   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
21:36:31       Focusing Completed Foc1 AutoFocus Completed (Profile No 13)
21:46:09     Target Completed Target completed (15/42, GCVS U And)
21:46:11   Queue Paused Queue paused for Session Suspension
21:46:13 Session Suspended Session suspended
21:47:03   Dome Closed Dome closed (closing time 50s)
22:03:28     Target Cancelled Target cancelled (16/42, UGC 2528 w/AT2020adnx) due to other condition
22:22:11 Session Resuming Session resuming
22:22:56   Dome Opened Dome opened (opening time 45s)
22:22:58 Session Running Session running
22:23:00     Target Started (17/42) Target started (17/42, UGCA 105)
22:23:02   Queue Resumed Observing Queue resumed
22:32:27     Target Completed Target partially completed (17/42, UGCA 105)
22:32:29     Target Started (18/42) Target started (18/42, NGC 4414 w/AT2021J)
22:52:42     Target Completed Target partially completed (18/42, NGC 4414 w/AT2021J)
22:52:45     Target Started (19/42) Target started (19/42, UGC 4148)
22:54:52       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
22:56:48       Focusing Completed Foc1 AutoFocus Completed (Profile No 14)
23:12:42     Target Completed Target completed (19/42, UGC 4148)
23:13:04     Target Started (20/42) Target started (20/42, PGC 25211)
23:15:27       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:17:34       Focusing Completed Foc1 AutoFocus Completed (Profile No 15)
23:33:38     Target Completed Target completed (20/42, PGC 25211)
23:38:38     Target Started (21/42) Target started (21/42, UGC 4043)
23:40:40       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:42:59       Focusing Completed Foc1 AutoFocus Completed (Profile No 16)
23:58:57     Target Completed Target completed (21/42, UGC 4043)
23:59:00     Target Started (22/42) Target started (22/42, GCVS RZ LMi)
00:01:10       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:02:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 17)
00:07:51     Target Completed Target completed (22/42, GCVS RZ LMi)
00:09:36     Target Started (23/42) Target started (23/42, UGCA 220)
00:12:36       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:14:48       Focusing Completed Foc1 AutoFocus Completed (Profile No 18)
00:50:57   Dome Running test to see Dome Driver is stalled
00:51:17   Dome Dome Driver/Dome Slaving found to be ok.
00:51:27     Target Completed Target completed (23/42, UGCA 220)
00:52:03     Target Started (24/42) Target started (24/42, UGC 4134)
00:56:10       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
01:13:46     Target Completed Target completed (24/42, UGC 4134)
01:14:24     Target Started (25/42) Target started (25/42, GCVS DX Cnc)
01:16:40       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
01:22:53     Target Completed Target completed (25/42, GCVS DX Cnc)
01:25:06     Target Started (26/42) Target started (26/42, KUG 0830+278 w/SN2020adow)
01:27:01       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:29:06       Focusing Completed Foc1 AutoFocus Completed (Profile No 19)
01:45:03     Target Completed Target completed (26/42, KUG 0830+278 w/SN2020adow)
01:46:59     Target Started (27/42) Target started (27/42, UGC 5245)
01:47:29 Job Queue Frozen Job Queue appears to have frozen at around 01:46 - 01:47
01:58:21   Dome Closed Dome closed (closing time 50s)
01:59:09   Dome Parked Dome moved to Az 90 deg (Park Position)
03:02:10   Dome Running test to see Dome Driver is stalled
03:02:52   Dome Dome Driver/Dome Slaving found to be ok.
03:08:59     Target Completed Target partially completed (27/42, UGC 5245)
03:09:01     Target Missed (28/42) Target's time slot was missed (28/42, UGC 5267)
03:09:03     Target Missed (29/42) Target's time slot was missed (29/42, UGC 5760 w/SN2020zoq)
03:09:05     Target Missed (30/42) Target's time slot was missed (30/42, GCVS AM CVn)
03:09:07     Target Started (31/42) Target started (31/42, UGC 5459)
03:30:55     Target Completed Target completed (31/42, UGC 5459)
03:30:57     Target Started (32/42) Target started (32/42, UGCA 297)
03:53:13     Target Completed Target completed (32/42, UGCA 297)
03:53:15     Target Started (33/42) Target started (33/42, UGCA 216)
04:15:07     Target Completed Target completed (33/42, UGCA 216)
04:15:09     Target Started (34/42) Target started (34/42, NGC 4454 w/SN2020abgq)
04:37:32     Target Completed Target completed (34/42, NGC 4454 w/SN2020abgq)
04:37:34     Target Started (35/42) Target started (35/42, IC 4074)
04:59:22     Target Completed Target completed (35/42, IC 4074)
04:59:24     Target Started (36/42) Target started (36/42, M85)
05:03:19       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
05:13:48     Target Completed Target completed (36/42, M85)
05:13:50     Target Started (37/42) Target started (37/42, NGC 4394)
05:17:37       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
05:35:31     Target Completed Target completed (37/42, NGC 4394)
05:35:33     Target Started (38/42) Target started (38/42, UGC 7802)
05:39:53       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S)
05:56:56     Target Completed Target completed (38/42, UGC 7802)
05:56:58     Target Started (39/42) Target started (39/42, UGCA 322)
06:00:53       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
06:18:30     Target Completed Target completed (39/42, UGCA 322)
06:18:32     Target Started (40/42) Target started (40/42, NGC 5331 w/AT2020abir)
06:22:46       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
06:29:39   Dome Closed Dome closed (closing time 15s)
06:29:44     Target Aborted (40/42) Target aborted (40/42, NGC 5331 w/AT2020abir) due to cloud
06:29:46   Queue Paused Queue paused for Session Suspension
06:29:49 Session Suspended Session suspended
06:30:04   Dome Closed Dome closed (closing time 15s)
06:31:54 Session Resuming Session resuming
06:32:04 User Intervention User request to close and finish session
06:32:06   Queue Aborted Job Queue aborted
06:32:09 Session Closing Session closing
06:32:29   Dome Closed Dome closed (closing time 15s)
06:33:20   Dome Parked Dome parked (parking time 43s), Az: 90.0 deg
06:33:57   Telescope Parked Telescope parked (parking time 29s)
06:34:27   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
06:35:16   Services Stopped Night Services stopped
06:35:18 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files)
06:36:29 Session Finished Session Finished
 
Session Alerts
Time     Alert Detail
20:51:16 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (GCVS DY Per)
22:23:55 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (UGCA 105)
23:39:41 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (UGC 4043)
00:22:12 Dome Dome has been unexpectedly stationary for more than 12 mins
00:27:38 Dome Dome Movements have begun again (outage 17.5 mins)
00:39:39 Dome Dome has been unexpectedly stationary for more than 12 mins
00:51:10 Dome Dome Movements have begun again (outage 23.5 mins)
00:51:17 Dome Dome Driver/Dome Slaving found to be ok following Slew Tests.
01:49:55 LAN Local Area Network has been inactive for more than 3 mins
01:49:57 AllSky/Weather Computer Link to AllSky/Weather Computer (Laptop7) has been inactive for more 3 mins
01:56:31 Dome Dome has been unexpectedly stationary for more than 12 mins
01:57:30 Job Queue Job Queue appears to be frozen (last IamAlive message at 01:46)
03:02:12 LAN Local Area Network is active again (outage 75.3 mins)
03:02:14 AllSky/Weather Computer Link to AllSky/Weather Computer (Laptop7) is active again (outage 75.3 mins)
03:02:22 Dome Dome Movements have begun again (outage 77.9 mins)
03:02:52 Dome Dome Driver/Dome Slaving found to be ok following Slew Tests.
06:29:24 Unexpected Dome Closure Dome is unexpectedly closing (Good Conditions)

Back to Top


Operational Issues (2021-01-08 S847)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Continuous Improvement

[ Prev | Next ]

Back to Top


Investigation:  Job Queue unexplainably frooze for 1 hour after which targets executed with closed dome shutter

Issue: Images taken over several hours with shutter closed (wasting useful observing time), and possible related issues (Obs.Monitor and Job Queue unexplainably freeze at 01:56 and unexplainably restarted at 03:02 (approx 1 hour later)), 

Description.
Observatory was taking series of Targets under relatively clear (but lowered transparency skies).   The job executor had completed Target 26/42 (KUG 0830+278 w/SN2020adow) at 01:45 and was performing target checks for Target 27/42 (UGC 5245) at 01:46.   These checks don't seem to have been completed.

Logs show that the  Obs.Monitor had become frozen at 01:47.   Then at 01:49 there were alerts reporting that the LAN and link to AllSky/Weather Computer had both been inactive for 3 minutes.    ( Last Successful Pings were at 01:46:52).

At 01:57 the Observatory Overseer determined the Job Queue had become frozen (last IamAlive message at 01:46) and proceeded to call the 'Make Observatory Safe' routine, which closed the Shutter (50s), turned off Telescope tracking and slewed the dome to Park position (Az 090 deg).

At 03:02 the LAN and Link to AllSky/Weather Computer became active again (outage of 75 minutes).   After testing for Stalled Dome condition (automatic response to Dome appearing stationary for 78 minutes, and large difference between dome and scope positions), the Job Queue resumed by completing target checks for Target 27/42 (UGC 5245) and slewing to the target at 03:03. After attempting to center the target, the main images were then all skipped as the target's End Time slot had been passed. 

The next 3 targets were skipped as their planned start times had been missed.    Next 9 targets were then fully executed (ie slew to target , attempted centering, attempted guiding/focusing, acquisition of main frames) with the observatory shutter closed.  Dome Position checks (including reporting the Shutter Status) were made but no action was taken  (to suspend/stop session or open the shutter)

Session was checked remotely at around 06:20. Seeing that the Shutter was closed, an attempt was made to Open the shutter using DeviceHub at 06:26:49.    AstroMain reported that Shutter State was Error  (See  Investigation:  Shutter Opening failed after 10-37s with ShutterStatus=5 (Error) ) and the Shutter was then commanded to close using DeviceHub at 06:29:20.   A second attempt to open observatory also failed and a User Interventation as therefore made to close the Session at 06:32.

Time     Event Detail
01:25:06     Target Started (26/42) Target started (26/42, KUG 0830+278 w/SN2020adow)
01:45:03     Target Completed Target completed (26/42, KUG 0830+278 w/SN2020adow)
01:46:59     Target Started (27/42) Target started (27/42, UGC 5245)
01:47:29 Job Queue Frozen Job Queue appears to have frozen at around 01:46 - 01:47
01:58:21   Dome Closed Dome closed (closing time 50s)
01:59:09   Dome Parked Dome moved to Az 90 deg (Park Position)
03:02:10   Dome Running test to see Dome Driver is stalled
03:02:52   Dome Dome Driver/Dome Slaving found to be ok.
03:08:59     Target Completed Target partially completed (27/42, UGC 5245)
03:09:01     Target Missed (28/42) Target's time slot was missed (28/42, UGC 5267)
03:09:03     Target Missed (29/42) Target's time slot was missed (29/42, UGC 5760 w/SN2020zoq)
03:09:05     Target Missed (30/42) Target's time slot was missed (30/42, GCVS AM CVn)
03:09:07     Target Started (31/42) Target started (31/42, UGC 5459)
03:30:55     Target Completed Target completed (31/42, UGC 5459)
       
06:18:32     Target Started (40/42) Target started (40/42, NGC 5331 w/AT2020abir)
06:29:39   Dome Closed Dome closed (closing time 15s)
06:29:44     Target Aborted (40/42) Target aborted (40/42, NGC 5331 w/AT2020abir) due to cloud
06:29:46   Queue Paused Queue paused for Session Suspension
06:29:49 Session Suspended Session suspended
06:30:04   Dome Closed Dome closed (closing time 15s)
06:31:54 Session Resuming Session resuming
06:32:04 User Intervention User request to close and finish session
06:32:06   Queue Aborted Job Queue aborted
06:32:09 Session Closing Session closing
06:32:29   Dome Closed Dome closed (closing time 15s)
06:33:20   Dome Parked Dome parked (parking time 43s), Az: 90.0 deg
       
06:36:29 Session Finished Session Finished

Time     Alert Detail
01:49:55 LAN Local Area Network has been inactive for more than 3 mins
01:49:57 AllSky/Weather Computer Link to AllSky/Weather Computer (Laptop7) has been inactive for more 3 mins
01:56:31 Dome Dome has been unexpectedly stationary for more than 12 mins
01:57:30 Job Queue Job Queue appears to be frozen (last IamAlive message at 01:46)
03:02:12 LAN Local Area Network is active again (outage 75.3 mins)
03:02:14 AllSky/Weather Computer Link to AllSky/Weather Computer (Laptop7) is active again (outage 75.3 mins)
03:02:22 Dome Dome Movements have begun again (outage 77.9 mins)
03:02:52 Dome Dome Driver/Dome Slaving found to be ok following Slew Tests.

No ObsPics were generated for the period 01:47 to 03:01.  These are produced by the DrawDome() routine which is called by the Observatory Monitoring Thread (RunObservatoryMonitoring() routine)

Obs.Manager seemed to operate ok throughout . Arguably not enough action taken by Obs.Manager to understand the actual status of the observatory operations and intervene.

Investigation

Q.  What was cause and what was effect ?   Did a) the LAN/AllSky Computer link became inactive, and this somehow caused the Obs.Monitor and Job Queue to freeze  or  b) the LAN/AllSky Computer were simply percieved to have become inactive due to a frozen Obs.Monitor ?.   How to test whether LAN was working or not ?

One way might be to see if ObsEnv data was getting passed from Obs. Computer through to AllSky/Weather computer between 01:56 and 03:02

On the Observatory Computer the OTA Temp became 'stuck' on -5.6 degC between 01:56 and 03:02,  but Pulsar Shutter health data continued to record real values. e.g. at around 01:59-02:00 there are distinct changes in all voltage, battery % and  milliamp values  as the Shutter is closed.   These changes are seen in the ObsEnv file held on the AllSky/Weather computer indicating that the LAN and AllSky Computer link were working throughout.

AstroPlan's log shows 'Live' session files being uploaded to Website throughout this period (with the exception of the ObsPic.gif file). This again indicates than the LAN was operational thoughout and the Alerts were a by-product of a stalled Observatory Monitoring thread.

Q. What caused Obs.Monitor and Job Queue to both freeze at 01:56 and then start working again at 03:02 ?.  

The last reported message from Job Queue was
   Checking Coordinates ...
with the results (Ok     J2000 Coords RA: 09 47 34.441, Dec: -02 01 58.44) only being reported some 10 minutes later.

This would point towards a stall point within  LookupKeyTargetInfo() routine (or alternatively in LookupKeyTargetInfo2() )  however it isn't obvious what would have caused it.  It is conceivable that the problem was caused by an issue with TheSky6 program or its COM objects.

It isn't possible to determine where the stall point lay within the RunObservatoryMonitoring's loop.

Issues / Actions

These Issues /actions have been added as Operational Issues or Continuous Improvement items for the session (see S847 Operational Issues)

Back to Top


Investigation:  Shutter Opening failed after 10-27s with ShutterStatus=5 (Error)

Issue: a) OpenShutter request at  06:26 failed after 27s with ShutterStatus=5 (Error). 
              b) OpenShutter request at  06:29 failed after 10s with ShutterStatus=5 (Error).

Description.

Open/Close Shutter activity recorded by DeviceHub:

06:26:49.040: Dome - Commands: OpenShutter: (done)
06:29:20.051: Dome - Commands: CloseShutter: (done)
06:29:39.863: Dome - Commands: OpenShutter: (done)
06:29:49.567: Dome - Commands: CloseShutter: (done)
06:31:54.499: Dome - Commands: OpenShutter: (done)
06:32:04.420: Dome - Commands: AbortSlew: (done)
06:32:14.402: Dome - Commands: CloseShutter: (done)

Pulsar Trace Log shows the following:

1) dome driver received (Manually Initiated) request to OpenShutter at 06:26:49 after which ShutterStatus changes to 2 (Opening) at 06:26:50.
Shutter and continues to show as 'Opening' until 06:27:16 (27s later), when it changes to 5 (Error).  

2) dome driver received (Manually Initiated) request  to CloseShutter at 06:29:20 after which ShutterStatus changes to 3 (Closing) at 06:29:20
Shutter and to 1 (Closed) at 06:29:37 (closing time 17s)

3) dome driver received (Manually Initiated) request to OpenShutter at 06:29:39 after which ShutterStatus changes to 2 (Opening) at 06:29:40.
Shutter and continues to show as 'Opening' until 06:29:49 (9s later). At 06:29:49 there is a (Obs.Manager/Session Suspension) request  to CloseShutter after which the ShutterStatus changes to 3 (Closing), before changing to 1 (Closed) at 06:30:00 (closing time 10s).

4) dome driver received a (Obs.Manager/Session Resuming) request to OpenShutter at 06:31:54 and ShutterStatus changes to 2 (Opening) at 06:31:55.
Shutter, changing to 5 (Error) at 06:32:05 (10s later).  At 06:32:14 there is a (Obs.Manager/Session Closing) request  to CloseShutter,  and the ShutterStatus changes to 3 (Closing), before changing to 1 (Closed) at 06:32:25 (closing time 10s).

Conditions at Time
  Frosty/Freezing.
  Outside Temperature -7 degC

Investigation

Dome Open/Closing was tested the following day in daylight conditions.  The shutter opened ok but the stutters as the shutter/chain connector jumps over the pulley wheels were again very noticeable (as they usually are).  Examining the higher guide wheel shown that it has a slight notch worn it to it where the screw head on the shutter/chain connector hits the wheel.  It is theorised that the resistance at the guide wheel points, cold temperature and ice combined to cause shutter opening to stall and produce the Shutterstatus=5 (Error).   It is assumed that the stall after 27 sec corresponded with the upper guide wheel, whilst the later stall after 10 sec corresponded with lowermost wheel.  The higher pulley wheel was rotated to offer a fresh surface for the chain and the Shutter/Chain Connector spacing was adjusted in an attempt to reduce the resistance of the connector coming over the wheels, but there didn't seem to be any improvement.       

Photo showing shutter chain passing
over upper guide wheel
(chain is too slack / i.e. not tight enough)
(photo taken 2021-01-09)
Photo showing chain/shutter connector passing
over upper guide wheel
(chain bolt spacing is 2 nuts + 1 washer)

(photo taken 2021-01-09)
Image    Image
  

Action:
Pulsar Observatories were contacted for advice, and after receiving advice,  a couple for adjustments were made to the Observatory's Shutter Chain. The shutter chain was tightened by removing one link and the spacing on Chain bolt was reduced by taking out one bolt and replacing with a couple of washers (see Shutter Chain Adjustements, 2021-01-15).

Photo showing shutter chain
after tightening

(photo taken 2021-01-15)
  Photo showing upper pulley wheel with
tightened chain and reduced chain bolt spacing
(chain bolt spacing is 1 nut + 3 washers)
(photo taken 2021-01-15) 
 Image    Image
  
  

Back to Top