David's Astronomy Pages Notes - Session 1130 (2023-08-16)

Notes (S1129)

Notes Main

Home Page

Notes (S1131)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (1),  Major Issues (1),  Minor Issues (2),  Small Defects (1),  Continuous Improvement (6)
	Variables Only >>
	2023-08-17
	Investigation - Dome Shutter failed to close when commanded

Session Aims & Highlights (2023-08-16)

Main aims

1. AstroMain, CCDSoft.  Check AstroMain, CCDSoft, SBIG Camera following issues in recent sessions (S1126, 2023-08-06 & S1127A, 2023-08-07)
2. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions

 Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• CCDApp2, TheSky6, CCDSoft5, DeviceHub 6.6.1.1,  MeadeGeneric 1.3.5.414, UPB 1.5, ASCOM 6.6
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.11dev4, Siril 1.0.6, deCONZ 1.18.2, ASCOM 6.6 SP1
   
• AstroMain 3.62.7, AstroPlan 1.38.1,   AstroShCap 1.3.15, AstroGuard 1.10.2
  AstroVOE 1.13  AstroSN 1.10,  AstroProtect 1.7,  AstroMove 1.12.4
  (AstroWeather 1.21.1, AstroAllSky 3.19.1,  AstroObsCam 1.11.4, AstroLaunch 1.10.9 (1.11.1 on Dev Computer)

Highlights

• Targets. 9/20 planned target were completed, with one other target partially completed.  Other targets failed were cancelled due to cloud/session suspension.
• Dome Slews.  2 dome slews suffered significant hold-ups during the session, with delays of 40s & 49s  (Hold Up Stats: 2/12 slews affected, 17%, 1.8 minutes total hold-up time). 

Notes:

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Actual Weather vs Pre-Session Weather Forecast

Session Event Log

Time     Event Detail 21:25:44 Session Monitoring AutoStart monitoring for Live Session opportunity between 21:53 & 03:22 21:25:48   AutoStart Waiting AutoStart waiting till earliest start time at 21:53 21:53:46   AutoStart Delayed Session delayed due to too few stars 22:22:48 Session AutoStarting Session autostarting (22:22) 22:23:17   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.) 22:23:30 Session Created Session Created (Live, 2023-08-16 S01130, ImageSaveNum: 1130001) 22:23:57   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 22:50:29   Services Started Observatory Services started 22:50:36 Observatory (Auto) Observatory placed in Fully-Automated Mode 22:50:38 Session Pending Session pending (2023-08-16) 22:50:40 Session Initiating Session initiating (2023-08-16) 22:50:42   Plan Requested Observing Plan requested from AstroPlan (1.38.1) 22:51:36   Plan Loaded Observing Plan loaded to queue (Plan ID: 904) 22:51:56   Camera1 Connected SBIG Camera connected (set point -10°C) 22:52:33   Telescope Connected Telescope connected (TheSky6) 22:52:57 Session Equilibration Session ready to Open Dome 22:53:42   Dome Opened Dome opened (opening time 45s) 22:53:45 Session Running Session running 22:53:47   Queue Started Observing Queue started (19 targets selected) 22:53:50     Target Started (NrZen) Target started (Focus Field 19, HIP 93376) 22:54:01       Dome Unparked Dome unparked 22:59:06       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:01:47       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide) 23:03:45       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 23:03:46       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 23:05:37       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 23:07:20       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 23:07:37     Target Completed (NrZen) Target completed (Focus Field 19, HIP 93376) 23:07:41     Target Started (1/19) Target started (1/19, M101 w/SN2023ixf) 23:12:02       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 23:28:53     Target Completed Target completed (1/19, M101 w/SN2023ixf) 23:29:08     Target Started (2/19) Target started (2/19, C/2019 T4 (ATLAS)) 23:33:11       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 23:41:36     Target Completed Target completed (2/19, C/2019 T4 (ATLAS)) 23:43:41     Target Started (3/19) Target started (3/19, HIP 87937) 23:45:00     Dome Slew Dome slew held up at Az. 246.1, 221.5° (Total delay 40s) 23:48:16       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:50:15       Focusing Completed Foc1 AutoFocus Completed (Profile No 3) 23:52:45     Target Completed Target completed (3/19, HIP 87937) 23:52:49     Target Started (4/19) Target started (4/19, AT2022zkp (CV)) 23:55:49       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:58:10       Focusing Completed Foc1 AutoFocus Completed (Profile No 4) 00:03:57     Target Completed Target completed (4/19, AT2022zkp (CV)) 00:04:22     Target Started (5/19) Target started (5/19, AT2022qnm) 00:07:26       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:09:40       Focusing Completed Foc1 AutoFocus Completed (Profile No 5) 00:26:14     Target Completed Target completed (5/19, AT2022qnm) 00:26:18     Target Started (6/20) Target started (6/20, C/2022 W3 (Leonard), ToO) 00:29:19       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:31:30       Focusing Completed Foc1 AutoFocus Completed (Profile No 6) 00:40:04     Target Completed Target completed (6/20, C/2022 W3 (Leonard), ToO) 00:40:07     Target Missed (7/20) Target's time slot was missed (7/20, 80P/Peters-Hartley) 00:41:33     Target Started (8/20) Target started (8/20, GCVS RX And) 00:45:13   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome) 00:45:15   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud) 00:45:17     Target Aborted (8/20) Target aborted (8/20, GCVS RX And) due to cloud 00:45:19   Queue Paused Queue paused for Session Suspension 00:45:21 Session Suspending Session suspending 00:46:08   Dome Closed Dome closed (closing time 50s) 00:46:09 Session Suspended Session suspended 00:48:14 Session Resuming Session resuming 00:48:59   Dome Opened Dome opened (opening time 45s) 00:49:00 Session Running Session running 00:49:02   Queue Resumed Observing Queue resumed 00:49:05     Target Started (8/20) Target started (8/20, GCVS RX And) 00:55:28     Target Completed Target completed (8/20, GCVS RX And) 00:55:32     Target Started (9/20) Target started (9/20, NGC 513 w/AT2023jpp) 01:14:00     Target Completed Target completed (9/20, NGC 513 w/AT2023jpp) 01:14:04     Target Started (10/20) Target started (10/20, GCVS S Cyg) 01:18:32       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:20:30       Focusing Completed Foc1 AutoFocus Completed (Profile No 7) 01:30:52     Target Completed Target completed (10/20, GCVS S Cyg) 01:30:57     Target Started (11/20) Target started (11/20, 61 Cyg) 01:32:30     Dome Slew Dome slew held up at Az. 267.4, 266.4, 246.2, 222.3, 221.4° (Total delay 49s) 01:34:54       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:36:50       Focusing Completed Foc1 AutoFocus Completed (Profile No 8) 01:38:24     Target Completed Target completed (11/20, 61 Cyg) 01:38:28     Target Started (12/20) Target started (12/20, C/2023 E1 (ATLAS)) 01:41:18       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 01:43:15       Focusing Completed Foc1 AutoFocus Completed (Profile No 9) 01:49:19   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome) 01:49:21   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud) 01:49:24     Target Aborted (12/20) Target aborted (12/20, C/2023 E1 (ATLAS)) due to cloud 01:49:26   Queue Paused Queue paused for Session Suspension 01:49:28 Session Suspending Session suspending 01:50:14   Dome Closed Dome closed (closing time 50s) 01:50:16 Session Suspended Session suspended 01:55:24     Target Cancelled Target cancelled (12/20, C/2023 E1 (ATLAS)) due to cloud 02:00:55 Session Resuming Session resuming 02:01:30   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome) 02:02:36 Session Suspended Session suspended 02:03:01   Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 02:05:01     Target Cancelled Target cancelled (13/20, GCVS EV Lac) due to cloud 02:13:35     Target Cancelled Target cancelled (14/20, WDS GRB 34) due to cloud 02:32:39     Target Cancelled Target cancelled (15/20, IC 1797 w/SN2023nnz) due to cloud 02:54:39     Target Cancelled Target cancelled (16/20, NGC 2326 w/AT2023pgb) due to cloud 03:16:59     Target Cancelled Target cancelled (17/20, LEDA 1488100 w/SN2023olz) due to cloud 03:28:21     Target Cancelled Target cancelled (18/20, GCVS HH And) due to cloud 03:47:26     Target Cancelled Target cancelled (19/20, M33 w/AT2023nou) due to cloud 04:11:28     Target Cancelled Target cancelled (20/20, GSC 1224:29) due to cloud 04:11:33   Queue Completed Job Queue completed 04:11:38 Session Closing Session closing 04:13:13   Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 04:15:19   Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 04:17:24   Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 04:19:14   Dome Parked Dome parked (parking time 73s), Az: 90.0 deg 04:20:33   Telescope Parked Telescope parked (parking time 73s) 04:20:48   Telescope State Handbox reads 'Scope parked Turn scope off.' 04:21:11   Telescope Switched Off Telescope Power has been switched off via UPB Switch. 04:22:36   Services Stopped Night Services stopped 04:22:39 Session Housekeeping Session Housekeeping started (Create Fits Summary, Transfer Files) 04:22:54 Session Finished Session Finished

Session Alerts & Alarms

Time     Type       Name Detail 23:45:00 Yellow Alert Dome Slew Dome slew held up at Az. 246.1, 221.5° (Total delay 40s) 01:32:30 Yellow Alert Dome Slew Dome slew held up at Az. 267.4, 266.4, 246.2, 222.3, 221.4° (Total delay 49s) 02:03:01 Red Alarm Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 02:03:41 Red Alert Dome Alert Dome hasn't reached 'Closed' state after 120s during SoftSuspend 04:13:13 Red Alarm Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 04:15:19 Red Alarm Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing) 04:17:24 Red Alarm Dome Closing Dome closure hasn't finished after 90s (ShutterState=Closing)

Back to Top

Operational Issues (2023-08-16, S1130)

• Critical Issues (1)
• Major Issues (1)
• Minor Issues (2)
• Small Defects (1)
• Continuous Improvement (6)
  

[ Prev | Next ]

Critical Issues

• Dome Shutter seemingly failed to close when requested by ObsManager (02:03) and possibly remained open until 04:18.  At 02:03:01 an error/alert was issued "Dome closure hasn't finished after 90s"  followed by a red alert at 02:03:41 "Dome hasn't reached 'Closed' state after 120s during SoftSuspend".  Shutter status was reported a 'Closing' by both DeviceHub and Pulsar ASCOM log.  ObsCam pictures show that the shutter as being wide open.   At 02:09 AstroGuard reported that conditions were 'critical' and checked the shutter status, but finding the the shutter was 'Closing' it did not make any independant action to close the shutter.  The Zigbee Shutter Sensor detected the shutter as being Open (ie not Closed) between 02:01 & 04:19.During Session Closing stage at the end of session (Session Closing) 3 attempts were made to close the dome (at 04:11, 04:13 & 04:15) but in case an alert was again raised to say that "Dome closure hadn't finished after 90s", and Shutter status was still reported as "Closing" by DeviceHub and Pulsar ASCOM log. After the 3rd attempt ObsManager/Closing Session seemingly gave up on trying to close the dome and at 04:17:54 it began a 'Park Dome' operation and issued a SlewDomeToAzimuth(90) at 04:18:03. Just prior to this at 04:17:59 an 'Abort DomeSlew' message is written in the logfile which is not normally posted at this point.  At 04:19:10 the Shutter's Zigbee Sensor reported that  Observatory Shutter had closed (open time 2.3 hour).  Issue is deemed 'Critical' as the observatory's shutter was left open for more than 2 hours with sky conditions in a critical state. Fortunately  the overcast sky didn't produce any rain during this time.  Issue was investigated the following day. (See "Investigation - Dome Shutter failed to close when commanded" ). It apppears that the Shutter Closure at 04:18:33 was in response to the Dome's Relay being in the 'Unsafe' state.  as advised by the observatory's CloudSensor. The ParkDome2() routine that is called during 'Session Closing' calls AbortSlew() if Dome.Slewing = True.  Whilst the cause of the original Closing Issue at 02:03 is most-likely a glitch of some kind, it would seem that AstroMain's ShutterMonitoring could usefully employ a Dome.`AbortSlew call if the dome's shutter status is seemingly 'stuck' on 'Closing' (3). A separate Continuous Improvement ticket has been raised to modify AstroMain's Shutter Monitoring routine to use AbortSlew if Dome's shutter status becomes seemingly stuck on 'Closing' or 'Opening'.
  Closed after review/investigation  2023-08-17 (Session, AstroMain, Pulsar Dome)

Major Issues

• Autostart held-up whilst starting ShCap service. Autostart operations became stalled whilst starting observatory services.  Problem appeared to be associated with a stall during start ShCap/SharpCap services.    Issue is classified as 'Major' as it prevents the automated session from preceeding.  Problem has been seen before but a solution hasn't be found.  Resolved on this occasion by user spotting that the website didn't show expected session progression, and making an intervention to use the backdoor facility to 'Restart ShCap Service' . This had the effect of releasing the stalled start observatory services routine, and allowed the session to proceed. A separate continuous improvement ticket has been raised that will allow ObsOverseer to monitor start Observatory Services and automatically Restart ShCap Services in the Start ShCap Services task seems to have stalled.
  Resolved in-session by user intervention 2023-08-16 (AstroMain, AstroShCap)

Minor Issues

• Unable to initiate Autostart due to a raised bDomeDataIssue flag (2023-08-17). Following a 'Shutter Closing' issue in the S1130 session (2023-08-16) beginning at 02:09 an AbortSlew command was sent to the Dome at 04:18 during Park Dome routine.  Status continued as Error (2) throughout the following day.  An attempt to initiate Autostart at around 21:35 failed with the message "AutoStart Mode can not be set (The bDomeDataIssue flag is raised !)" . The AstroMain program was restarted and the (Pulsar) Dome Service restarted.   This time Autostart was initated without problem
  Resolved 2023-08-18 (Session, AstroMain, Pulsar Dome)
   
• ObsPics showing 'Dome Relay Error (2)'  (2023-08-17). Following a 'Shutter Closing' issue in the S1130 session (2023-08-16) beginning at 02:09 an AbortSlew command was sent to the Dome at 04:18 during Park Dome routine.  Status continued as Error (2) throughout the following day.  The Pulsar Dome Service and the AstroMain program were both restarted at 21:39 restarted 21:39 (2023-08-17), however field 12 (Dome Relay) of Volatile records in the Pulsar ASCOM log file continued to show the value 2 and ObsPics continued to show 'Dome Relay Error (2)'. Issue was resolved by using the 'Restart Pulsar Dome' facility to perform a Power Reset on the Pulsar Dome Controller at 00:05 (2023-08-18), after which ObsPic show 'Dome Relay Unsafe'
  Resolved 2023-08-18 (Session, Pulsar Dome)

Small Defects

• Fresh Wind Speed reading not accessible (2023-08-19).   After changing batteries, moving the base station and resetting the Weather Station on 2023-08-16 (ahead of the S1130 Session) wind speed data rates have been very good until 2023-08-19 05:45 when Weather Station's wind data seemingly went offline.   Problem eventually resolved itself at 12:10.  It may be just coincidence but wind speed data was noted to have gone offline (flat-lined) during the time that heavy rain occurred.  See Fig 1.
  Resolved itself 2023-08-19 (Weather Station).

Continuous Improvement

• Weather graphs should show blank 'Outside' data lines when Outside Temperature/Humidity has flat-lined for more than 2 hours.  ValidateWeatherStationData() modified to set Outside Temperature, Outside Humidity, OutsideDewPointDepression, ObservatoryTemperatureDelta, ObservatoryHumidityDelta to vcNull when Temperature & Humidity have flat-lined. Graphs affected include vwsGraph26, vwsGraph29, wsGraph8 & wsGraph18.
  Done 2023-08-19 (AstroWeather 1.21.2)
   
• Add back door routine to allow user to reset bDomeDataIssue flag.  Button and reset facility was already added to More/Backdoor Tab on  2022-02-08 (CI, S966)
  Already done 2022-02-08  (AstroMain )
   
• Extend ObsOverseer to identifiy and resolve a stalled Start ShCap Service operation by calling RestartShCapService. Autostart operations became stalled in S1123 & S1130 Sessions whilst starting ShCap/SharpCap service (within start Observatory Services).  This is a Major Issue as it currently prevents an automated session from preceeding (unless the user spots the problem and intervenes to 'Restart ShCap Services').  ObsOverseer should be extended to monitor the Start Observatory Services operation and automatically attempt to 'Restart ShCap Services' if the process has become stalled whilst attempting to Start ShCap Service. Fixed 2023-08-20. The fix has been tested on Development Computer but its needs to be proven to work in a live observatory session.
  Done 2023-08-20 (AstroMain 3.62.8)
   
• Reporting of 'Reset MakeObservatorySafe flags'  by the user should show the time that flags were reset (2023-08-17).
  Done 2023-08-20 (AstroMain 3.62.8)
   
• Modify Shutter Monitoring routine to use AbortSlew if Dome's shutter status is seemingly stuck on 'Closing' or 'Opening'. Observatoty Dome's shutter status became stuck on 'Closing (3)'  during S1130 session and the shutter remained open for over 2 hours even though sky conditions (Clarity) were critical. Fortunately the overcast sky didn't produce any rain during this time.  During Park Dome operations at the end of the session, an AbortSlew command was used by ParkDome2() routine as the Dome reported that it was Slewing and this appeared to 'reset' the dome/shutter such that an Unsafe Relay condition triggered the Shutter into Closing which it successfully did. Building upon this observation AstroMain's shutter monitoing should employ an AbortSlew if the shutter status is seemingly stuck on Closing (3) or Opening (2).  Changes made to ObsManager gObsManagerSection = "Suspended" (SuspendedStageNo 4 to 8) & gObsManagerSection = "Closing"
  Done 2023-08-20 (AstroMain 3.62.8)
   
• Conduct experiment to see if Weather Station's USB Hub can be remotely reset and perform a reconnection to Weather Station's base station.   In the event that Weather Station's Base Station is reset (in order to reconnect to 'lost' sensors) the USB Hub has to be reset to pick up fresh data.   This is done by manually pressing a reset button on the back of the Weather Station's USB Hub and typically also simultaneously pressing Up & Down buttons on the base station.  (Recent observations show that Virtual Weather Station & AstroWeather programs don't have to be restarted). Carry out expriement to see if the USB Hub can be routed through the Pegasus USB Hub (if not already) and the USB Port turned off/turned on in order to reset the Weather Station's USB Hub.   If so it may then be possible to use the main Pegasus USB Hub (either directly from Pegasus USB Hub software or through an ASCOM connection (accessed as a backdoor option in AstroWeather) to perform the operation.  However in view of manual requirement to set base station into 'transmit a connect invite' (using Up + Down buttons), the benefits of remote reconnection to Weather  Station's USB Hub may be limited.
  To do 2023-08-19 (Observatory, AllSky/Weather Computer,  Oregon WMR80 Weather Station)

[ Prev | Next ]

Back to Top

2023-08-17

Investigation - Dome Shutter failed to close when commanded

Back to Top