David's Astronomy Pages Notes - Session 1211 (2024-04-25)

Notes (S1210)

Notes Main

Home Page

Notes (S1212)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (0),  Minor Issues (5),  Small Defects (3),  Continuous Improvement (0)
	Images from 2024-04-25 >>

Session Aims & Highlights (2024-04-25)

Main aims

1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
2. Dome.  Check out AstroMain's new recoverDome() routine introduced earlier in AstroMain 3.68.4  / 3.68.5.

 Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• TheSky6, CCDSoft5, DeviceHub 6.6.1.1,  MeadeGeneric 1.3.9, UPB 1.5, ASCOM 6.6,   Pulsar Dome Firmware 1.51 & Driver 6.3.10
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.13, Siril 1.0.6, deCONZ 1.18.2, ASCOM 6.6 SP1
• [ DomeLink 1.0 / 1.1 - not used ]
   
• AstroMain 3.68.6,  AstroPlan 1.40.1,  AstroShCap 1.3.15, AstroGuard 1.10.5   (any new items or versions in bold)
  AstroVOE 1.13.2,  AstroSN 1.10.3,  AstroProtect 1.7,  AstroMove 1.12.12
  AstroWeather 1.24.1, AstroAllSky 3.20.4,  AstroObsCam 1.11.4 (not 1.11.5), AstroLaunch 1.11.3

Highlights

• Targets. 3/16 planned targets observations were completed, before sky became cloudy.
• Dome.  There was an issue with Pulsar Dome not recognising when shutter became finished opening or closing, even though commands to Open/Close were occuring ok.  AstroMain's ObsManager didn't call RecoverDome() routine, which would have fixed the problems.

Notes:

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Actual Weather vs Pre-Session Weather Forecast

Session Event Log

Time     Event Detail 23:16:47 Session Monitoring AutoStart monitoring for Live Session opportunity between 23:16 & 03:02 23:16:49 Session AutoStarting Session autostarting (23:16) 23:17:10   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.) 23:17:37   Camera1 Connected SBIG Camera connected (Set point -20°C) 23:17:39 Session Created Session Created (Live, 2024-04-25 S01211, ImageSaveNum: 1211001) 23:17:44   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 23:19:31   Services Started Observatory Services started 23:19:38 Observatory (Auto) Observatory placed in Fully-Automated Mode 23:19:39 Session Pending Session pending (2024-04-25) 23:19:41 Session Initiating Session initiating (2024-04-25) 23:19:51   Camera1 Connected SBIG Camera connected (Set point -20°C) 23:19:57   Plan Requested Observing Plan requested from AstroPlan (1.40.1) 23:21:05   Plan Loaded Observing Plan loaded to queue (Plan ID: 986) 23:21:44   Telescope Connected Telescope connected (TheSky6) 23:22:08 Session Equilibration Session ready to Open Dome 23:22:53   Dome Opened Dome opened (Opening time 45s, Zigbee 41s) 23:22:58 Session Running Session running 23:23:01   Queue Started Observing Queue started (16 targets selected) 23:23:04     Target Started (NrZen) Target started (Focus Field 13, HIP 63666) 23:23:18       Dome Unparked Dome unparked 23:27:18       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:30:18       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide) 23:32:44       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 23:32:47       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 23:34:55       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 23:36:37       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 23:36:54     Target Completed (NrZen) Target completed (Focus Field 13, HIP 63666) 23:36:59     Target Started (1/16) Target started (1/16, NGC 4192A w/SN2024exw) 23:43:58       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 00:00:23     Target Completed Target completed (1/16, NGC 4192A w/SN2024exw) 00:00:27     Target Started (2/16) Target started (2/16, NGC 5235 w/AT2024grb) 00:03:22       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:06:02       Focusing Completed Foc1 AutoFocus Completed (Profile No 3) 00:22:09     Target Completed Target completed (2/16, NGC 5235 w/AT2024grb) 00:22:13     Target Started (3/16) Target started (3/16, MCG -1-33-30 w/SN2024byg) 00:25:50       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 00:39:30       SoftSuspend Called Soft Suspend called due to Deteriorating Conditions (too few stars) 00:42:35     Target Completed Target completed (3/16, MCG -1-33-30 w/SN2024byg) 00:42:37   Queue Paused Queue paused for Session Suspension 00:42:39 Session Suspending Session suspending 00:43:26   Dome Closed Dome closed (Closing time 50s, Zigbee 44s) 00:43:28 Session Suspended Session suspended 01:00:57     Target Cancelled Target cancelled (4/16, C/2023 A3 (Tsuchinshan-ATLAS)) due to cloud 01:13:17     Target Cancelled Target cancelled (5/16, AT2024crd) due to cloud 01:17:50 Obs.Monitor Frozen Obs.Monitor appears to be frozen at around 01:17 in section 'Monitor PowerBoxEnv' 01:33:37     Target Cancelled Target cancelled (6/16, NGC 4216 w/SN2024gy) due to cloud 01:53:09     Target Cancelled Target cancelled (7/16, 62P/Tsuchinshan) due to cloud 02:13:59     Target Cancelled Target cancelled (8/16, NGC 3690 w/SN2024gzk) due to too few stars 02:15:12 Session Resuming Session resuming 02:16:52 Session Suspended Session suspended 02:19:07   Dome Closed Dome closed (Closing time 2s, Zigbee 44s) 02:21:37 Session Resuming Session resuming 02:23:18 Session Suspended Session suspended 02:25:22     Target Cancelled Target cancelled (9/16, GCVS HH And) due to 02:25:33   Dome Closed Dome closed (Closing time 2s, Zigbee 44s) 02:28:03 Session Resuming Session resuming 02:29:43 Session Suspended Session suspended 02:31:58   Dome Closed Dome closed (Closing time 2s, Zigbee 44s) 02:38:59     Target Cancelled Target cancelled (10/16, C/2021 S3 (PANSTARRS)) due to cloud 02:49:17     Target Cancelled Target cancelled (11/16, GCVS BL Lac) due to cloud 02:58:39     Target Cancelled Target cancelled (12/16, WDS GRB 34) due to cloud 03:09:04     Target Cancelled Target cancelled (13/16, Nova Cas 2021 (Nova)) due to cloud 03:21:09     Target Cancelled Target cancelled (14/16, AT2024dsj (CV)) due to cloud 03:41:29     Target Cancelled Target cancelled (15/16, NGC 6433 w/SN2024efn) due to cloud 03:45:45 Obs.Monitor Frozen Obs.Monitor appears to be frozen at around 03:45 in section 'Refresh Dome Picture/Draw DewHeaters' 04:02:50     Target Cancelled Target cancelled (16/16, HIP 87937) due to too few stars 04:02:54   Queue Completed Job Queue completed 04:02:58 Session Closing Session closing 04:03:03   Dome Closed Dome already closed (closed at 02:31, closing time 2s, zigbee 44s) 04:06:12     Dome Slew Dome slew held up at Az. 200.1° (Delay 180s) 04:06:14   Dome Parked Dome parked (parking time 182s), Az: 90.0 deg 04:10:39   Telescope Parked Telescope parked (parking time 263s total) 04:10:49   Telescope State Handbox reads 'Scope parked Turn scope off.' 04:11:11   Telescope Switched Off Telescope Power has been switched off via UPB Switch. 04:12:36   Services Stopped Night Services stopped 04:12:39 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files) 04:12:46     Dome (Find Park) Find Park started (Search Az 88.0 to 92.0°, Step 0.2°, Narrow) 04:13:47     Dome Find Park aborted (Timeout (60s) waiting for dome to stop slewing) 04:13:50       Dome Unparked Dome unparked 04:13:52 Session Finished Session Finished

Session Alerts & Alarms

Time     Type       Name Detail 01:18:51 Red Alert Obs.Monitor Obs.Monitor appears to be frozen in section 'Monitor PowerBoxEnv' (last cycle at 01:17:50) 01:18:56 Green Clear Obs.Monitor Obs.Monitor has resumed again (01:18) after 1.0 mins 02:19:02 Red Alert Dome Alert Dome hasn't reached 'Closed' state after 120s during SoftSuspend 02:25:28 Red Alert Dome Alert Dome hasn't reached 'Closed' state after 120s during SoftSuspend 02:31:53 Red Alert Dome Alert Dome hasn't reached 'Closed' state after 120s during SoftSuspend 03:46:20 Red Alert Obs.Monitor Obs.Monitor appears to be frozen in section 'Refresh Dome Picture/Draw DewHeaters' (last cycle at 03:45:45) 03:46:37 Red Alert Dome Alert DomeRecords have stopped updating 03:46:42 Green Clear Obs.Monitor Obs.Monitor has resumed again (03:46) after 0.3 mins 03:46:47 Green Clear Dome Alert DomeRecords have resumed updating 04:06:12 Red Alert Dome Slew Dome slew held up at Az. 200.1° (Delay 180s) 04:06:14 Yellow Alert Dome Discrepency Hub reports dome at 90.0°, but Pulsar reports dome at 90.0° 04:10:40 Red Alert Unexpected Scope Movement Scope appears to be tracking even though Tracking=False

Back to Top

Operational Issues (2024-04-25 S1211)

• Critical Issues (0)
• Major Issues (0)
• Minor Issues (5)
• Small Defects (3)
• Continuous Improvement (0)
  

[ Prev | Next ]

Critical Issues

• None

Major Issues

• None

Minor Issues

• Obs.Monitor red alert was raised "Obs.Monitor appears to be frozen in section 'Monitor PowerBoxEnv' (01:18) .   ObsOverseer raised a red alert at 01:18:51 with the message "Obs.Monitor appears to be frozen in section 'Monitor PowerBoxEnv' (last cycle at 01:17:50)".   ObsOverseer also reported warnings "Task in ExecuteQueue/Wait is taking a long time? (01:18)" & "Obs.Manager is frozen  in 'Dome Monitoring' (last cycle at 01:17:54)" at around the same time which was with the Session Suspended and waiting on weather for Target 6/16 (NGC 4216).   Around this time the logfile recorded an  exception  in Dome.Connected and in Dome.Slaved at 01:18:51 with the message "Object synchronization method was called from an unsynchronized block of code".  The 'Read PulsarDomeRecord routine raised an alert at 01:19:01 with the message "Read PulsarDomeRecord Alert | Volatile Exception at 01:18:51 nExceptions=76" . Pulsar log file shows
  01:18:51.174 Volatile Exception System.ApplicationException: Object synchronization method was called from an unsynchronized block of code.at System.Threading.Mutex.ReleaseMutex()
   After which Volatile messages read
    Volatile 200.1 0 0.000000 200.1 0 1 960 14965 -2 7720 1527 2595 0 0
  from 01:17:09   till 01:21:39
  Unclear what was the root cause of the problem,  however the issue fortunately resolved itself after around 1 minute when the ObsOverseerer reported "Obs.Manager has resumed again at 01:18:52 after 1.0 min"  &  "Obs.Monitor has resumed again (01:18) after 1.0 mins". 
  Resolved itself in-session 2024-04-25 (Session, AstroMain)
   
• Dome Monitor raised a warning "Dome opening hasn't finished after 60s (02:16). An 'Open Dome" warning was reported at 02:16 with the message "Dome opening hasn't finished after 60s , followed by message "Resume Session   Fail   Shutter hasn't opened within 100s.  Issue eventually resolved by user at 05:13 by running the 'RecoverDome' utility .  Pulsar logs record OpenShutter at 02:15:12 and ShutterStatus changing to 2 (Opening) at 02:15:12 and then changing to 0 (Open) at 02:15:55, implying an opening time of 43s (vs 41s Zigbee) and show that the ShutterStatus values are normal (ie not frozen). Despite this the ObsManager records "Dome opening hasn't finished after 60s" at  02:16:12, around 17s after the the shutter actually closed. DeviceHub.Dome log indicates ShutterOpening at 02:15:13 and then remaining as ShutterOpening until 02:16:47 when it changes to ShutterClosing at 02:16:57. AstroMain is getting ShutterStatus value that are consistent with values in DeviceHub.Dome but are not consistent with those in PulsarDome. AstroMain ObsManager code needs extending so that if 'Shutter Not Open' it calls RecoverDome, before reattempting Dome.Open.  Fixed in AstroMain 3.68.8,   ResumingStageNo=1 calls RecoverDome() if Shutter is not opened within 100s. ResumingStageNo=2 then waits for RecoverDome operation to finish and then calls Dome.OpenShutter & recycles back to ResumingStageNo=1 towait for Shutter to finally open. 
  Fixed 2024-04-27 (AstroMain 3.68.8)
   
• A Red 'Dome Alert'  "Dome hasn't reached 'Closed' state after 120s" was issued 3 times in a 10 minute period (02:19 - 02:31). Red Alerts were raised at 02:19:02,  02:25:28 & 02:31:53  with the message "Dome hasn't reached 'Closed' state after 120s during SoftSuspend".  In each case this was whilst the session was suspended after briefly resuming (sky conditions were borderline during this period).  The problem with the dome seems to have begun during Open Dome attempt at 02:16,  but may even have begun earlier based on Read PulsarDoneRecord exceptions and on  Dome.Connected/Dome.Slaved exceptions at around 01:18.  DeviceHub.Dome log shows shutterClosing from 02:16:57 to 02:19:07 when status changes to ShutterClosed (this is seemingly in response to an Abort.Slew command at 02:19:02.  In each case ShutterStatus is stuck' on ShutterClosing, and an AbortSlew command in SuspendedStage4 is executed according to current design in an attempt to clear a ShutterStatus that is stuck on 'Closing'. This operations is partly successful, in that it causes ShutterStatus to stop returning ShutterClosing and begin to return ShutterClosed,  but it doesn't fix the underlying issue (which seems to be a problem with DeviceHub.Dome), and the symptons are again repeated on the next Open Shutter/ Close Shutter cycle.   The RecoverDome() routine, which would likely resolve the problem, isn't called as ShutterStatus = ShutterClosed going into SuspendedStageNo 6.  Fix attempted in AstroMain 3.68.7, the call to AbortSlew is removed, so that if Shutter Not Closed it execution goes direct to RecoverDome().
  Fix attempted 2024-04-26 (Session, AstroMain 3.68.7)
   
• Time-outs occurred trying to Slew Dome to Az 90° & later to slew dome to Az 88° (04:03 & 4:12) .  Commands to Slew Dome to Az 90 (Park) at 04:03 appears successful but the Slew operation timed-out as Park Dome had been waiting more than 3 minutes,  Later during FindDomeParl operation, Jog Dome to 88° (from 90°) timed out after 60s the message ' Timeout waiting for dome to stop slewing".  Resolved by user intervention at 05:13. A separate fix for 'dome not closed in 120s' issue will reduce number of cases involving SlewToAzimuth timeouts. However AstroMain ObsManager code needs extending so that if SlewToAzimuth still Times Out it calls RecoverDome.   Fixed in AstroMain 3.68.8,   ClosingStageNo=5 calls RecoverDome() if Park Dome takes longer than 3 minutes. ClosingStageNo=6 (new) then waits for RecoverDome operation to finish and then calls Dome.OpenShutter & recycles back to ClosingStageNo=4 to re-attempt the Park Dome operation. New routine tested on Development Computer 2024-04-27.
  Fixed 2024-04-27 (AstroMain 3.68.8)
   
• CloudSensor and WeatherStation Data rates to AstroMain seem to have slowed right down.  Problem noticed in last 2-3 sessions. Fix attempted by archiving off ObsEnv file which had grown to 124 MB and cropping the data in live file down to just data records since 2024-04-20, and re-importing ObsEnv to AstroWeather on AllSky/Weather Computer.  Although this seemed to resolve the problem, CloudSensor & WeatherStation data was again only slowly refreshing on 2024-04-27. This seemed to be resolved by restarting AstroWeather on AllSky/Weather Computer.  So there seems to be something in AstroWeather that progressively slow down the CloudSensor/WeatherStation looping.  Unable to find the cause of this issue. Ticket closed after review 2024-12-10.
  Closed after review 2024-12-10 (Session, AstroMain ObsEnv, AstroWeather)

Small Defects

• Certain LogFile Records involving 'C/2023 A3 (Tsuchinshan-ATLAS)' are not shown correct columns (00:57).  Certain log file records involving S1 fields such as " C/2023 A3 (Tsuchinshan-ATLAS) (Overseer)" are too long and push S2, S3, S4 fields outside their intended column positions. Fixed in AstroMain 3.68.8, the S1 field is cropped to width 39 (27+12).
  Fixed 2024-04-27 (AstroMain 3.68.8)
   
• Unexpected Dome Discrepency message whilst Parking Dome (04:06).  Yellow Dome Warning was raised at 04:06 with message "Hub reports dome at 90.0°, but Pulsar reports dome at 90.0° "  during Closing Stage 5 (wait for Dome Park). The two azimuth numbers given in the message are both reported as 90.0° so a simple comparison wouldn't be expected to raise a discrepency, yet a discrepency has been raised. To investigate.
  Fixed 2024-04-27 (AstroMain 3.68.8)
   
• An 'Unexpected Scope Movement' red alert raised after parking scope (04:10).  Immediately after parking scope and before collecting the post-park Handbox status a red alert was unexpectedly raised at 04:10:40 with the message "Unexpected Scope Movement - Scope appears to be tracking even though Tracking=False".  To investigate. Bug identified (ClosingStageNo's had changed in ObsManager , but hadn't been modified in ObsMonitor to reflect the changes. ParkingScope stages moved from stage no's 5 & 6   to new stage no's 7 & 8. Fixed in AstroMain 3.68.7. Changed later to new stage no's 8 & 9 following introduction of new RecoverDome stage (stage 6)
  Fixed 2024-04-26 (AstroMain 3.68.7)

Continuous Improvement

• None

[ Prev | Next ]

Back to Top