David's Astronomy Pages
Notes - Session 1196 (2024-03-04)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (0),  Minor Issues (4),  Small Defects (8),  Continuous Improvement (16)
 
Bullet Images from 2024-03-04 >>         
   
2024-03-05
Bullet AstroMain - Task Manager Assistant
   
2024-03-11
Bullet System Tests - Examining robustness of Observatory system to a mains power supply outage
   
   

Session Aims & Highlights (2024-03-04)

Main aims

  1. Telescope Resync.  Perform Telescope re-sync on a known star or position and make a short mapping job in order to reestabish telescope pointing and to tie in the TPoint Model, following undesirable switching off of the LX200 Telescope in an unparked state.
  2. MeadeGeneric.  Check that there are no issues using the new 1.3.9 version of MeadeGeneric telescope driver which has been installed since the last session.
  3. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
  4. 12P/Pons-Brooks. Acquire images of Comet 12P/Pons-Brooks which has been undergoing regular outbursts.
  5. AstroMain. Continue to test fixes to CCDSoft Camera/SBIG Camera operations to prevent stalled execution during starting, fix issues due to occasional repeated failures during Cam1 focusing and occassional unexpected CCD Temperature regulation.

 Equipment & Software

Highlights

 

Notes:

Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Image
 
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Image
  
Sky Conditions (Locate Frames)
Image 
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image
  
Session Event Log
Time     Event Detail
18:22:37   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.)
18:23:02   Camera1 Connected SBIG Camera connected (Set point -20°C)
18:23:04 Session Created Session Created (Live, 2024-03-04 S01196, ImageSaveNum: 1196001)
18:23:26   CcdCamera Frozen CcdCamera appears to be frozen at around 18:23 in section 'GetCameraProperties'
18:24:05   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera)
18:24:12   SBIG Camera Reset SBIG Camera being power-cycled
18:25:02   Camera1 Connected SBIG Camera connected (Set point -20°C)
18:25:38   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
18:27:24   Services Started Observatory Services started
18:28:36   Telescope Connected Telescope connected (TheSky6)
18:31:26       Dome Unparked Dome unparked
18:51:24   Dome Opened Dome opened (Opening time 48s, Zigbee 41s)
18:55:33 Telescope Running 'ResetScopePointing' under user supervision
18:57:19 Telescope Running 'ResetScopePointing' under user supervision
18:59:36   Telescope Telescope pointing successfully synced
18:59:42   Telescope Pointing Mapping run started
18:59:45   Automapping Started Automapping started (12 points planned)
19:11:19   Automapping Completed Automapping completed (12 points mapped out of 12 points)
19:11:46   Telescope Pointing Mapping run finished (Mapped 12 / 12)
19:13:57   Telescope Parked Telescope parked (parking time 69s total)
19:14:12   Telescope State Handbox reads 'Scope parked Turn scope off.'
19:14:43   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
19:17:00   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
19:18:51 Session Monitoring AutoStart monitoring for Live Session opportunity between 19:18 & 04:46
19:18:53 Observatory (AutoStart) Observatory placed in Auto-Start Mode
19:18:55 Session AutoStarting Session autostarting (19:18)
19:19:13   Services Started Observatory Services started
19:19:20 Observatory (Auto) Observatory placed in Fully-Automated Mode
19:19:23 Session Pending Session pending (2024-03-04)
19:19:25 Session Initiating Session initiating (2024-03-04)
19:19:27   Plan Requested Observing Plan requested from AstroPlan (1.39.7)
19:20:45   Plan Loaded Observing Plan loaded to queue (Plan ID: 970)
19:20:59   Camera1 Connected SBIG Camera connected (Set point -20°C)
19:21:05   Telescope Connected Telescope connected (TheSky6)
19:21:43 Session Running Session running
19:21:45   Queue Started Observing Queue started (29 targets selected)
19:21:49     Target Started (NrZen) Target started (Focus Field 6, HIP 28634)
19:25:29       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:28:00   SBIG Camera Take Image failed (Command timed-out)
19:28:10   CcdCamera Frozen CcdCamera appears to be frozen at around 19:28 in section 'GetCameraProperties'
19:28:48   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera)
19:28:55   SBIG Camera Reset SBIG Camera being power-cycled
19:29:45   Camera1 Connected SBIG Camera connected (Set point -20°C)
19:29:48   SBIG Camera Take Image command accepted (bSbigDriverException flag is reset)
19:30:27       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
19:31:44   SBIG Camera Take Image failed (Command timed-out)
19:32:04   SBIG Camera Take Image failed (Command in progress. Error code = 219 (0xdb).)
19:32:09   CcdCamera Frozen CcdCamera appears to be frozen at around 19:32 in section 'GetCameraProperties'
19:32:48   CCDSoft Restarting CCDSoft being restarted (attempt to fix frozen CcdCamera)
19:32:50       Focusing Failed Foc1 focusing failed - star lost
19:32:55   SBIG Camera Reset SBIG Camera being power-cycled
19:33:18       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
19:33:52   Camera1 Connected SBIG Camera connected (Set point -20°C)
19:35:31       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
19:36:25       Focusing Failed Foc2 focusing failed - star lost
19:36:29     Target Completed (NrZen) Target completed (Focus Field 6, HIP 28634)
19:36:34     Target Started (1/29) Target started (1/29, NGC 1122 w/SN2024dru)
19:37:39   SBIG Camera Take Image command accepted (bSbigDriverException flag is reset)
19:38:48   PHD2 Communication Restarting PHD2 service (14 incomplete data stream reads)
19:39:16   PHD2 Restarted PHD2 service restarted
19:41:21       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:44:20       Focusing Completed Foc1 AutoFocus Completed (Profile No 3, wide)
19:46:17       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
19:59:29     Target Completed Target partially completed (1/29, NGC 1122 w/SN2024dru)
19:59:33     Target Started (2/29) Target started (2/29, UGC 831 w/SN2023wdd)
20:01:28   PHD2 Communication Restarting PHD2 service (195 incomplete data stream reads)
20:01:49   PHD2 Restarted PHD2 service restarted
20:03:07       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:05:33       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
20:20:49       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
20:21:21     Target Completed Target completed (2/29, UGC 831 w/SN2023wdd)
20:21:23   Queue Paused Queue paused for Session Suspension
20:21:27 Session Suspending Session suspending
20:22:14   Dome Closed Dome closed (Closing time 50s, Zigbee 43s)
20:22:17 Session Suspended Session suspended
20:25:15 Session Resuming Session resuming
20:26:00   Dome Opened Dome opened (Opening time 45s, Zigbee 41s)
20:26:02 Session Running Session running
20:26:04   Queue Resumed Observing Queue resumed
20:26:06     Target Started (NrZen) Target started (Focus Field 5, HIP 23569)
20:28:38       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
20:30:46       Focusing Completed Foc2 AutoFocus Completed (Profile No 5, wide)
20:31:35       Focusing Failed Foc2 focusing failed - star lost
20:31:39     Target Completed (NrZen) Target completed (Focus Field 5, HIP 23569)
20:31:44     Target Started (3/29) Target started (3/29, M101 w/SN2023ixf)
20:34:41       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
20:36:06     Target Failed (3/29) Target failed due to cloud (3/29, M101 w/SN2023ixf)
20:36:08   Queue Paused Queue paused for Session Suspension
20:36:10 Session Suspending Session suspending
20:36:56   Dome Closed Dome closed (Closing time 50s, Zigbee 43s)
20:36:58 Session Suspended Session suspended
20:38:51     Target Cancelled Target cancelled (3/29, M101 w/SN2023ixf) due to cloud
20:58:23     Target Cancelled Target cancelled (4/29, 12P/Pons-Brooks) due to too few stars
21:06:55 Session Resuming Session resuming
21:07:40   Dome Opened Dome opened (Opening time 45s, Zigbee 41s)
21:07:42 Session Running Session running
21:07:44   Queue Resumed Observing Queue resumed
21:07:46     Target Started (NrZen) Target started (Focus Field 6, HIP 28634)
21:11:46       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:12:05       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
21:13:43       Focusing Completed Foc1 AutoFocus Completed (Profile No 6)
21:13:46       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
21:14:00     Target Completed (NrZen) Target completed (Focus Field 6, HIP 28634)
21:14:04     Target Started (5/29) Target started (5/29, 13P/Olbers)
21:18:04     Target Failed (5/29) Target failed due to cloud (5/29, 13P/Olbers)
21:18:06   Queue Paused Queue paused for Session Suspension
21:18:08 Session Suspending Session suspending
21:18:56   Dome Closed Dome closed (Closing time 50s, Zigbee 43s)
21:18:58 Session Suspended Session suspended
21:32:48 Session Resuming Session resuming
21:33:33   Dome Opened Dome opened (Opening time 45s, Zigbee 40s)
21:33:35 Session Running Session running
21:33:37   Queue Resumed Observing Queue resumed
21:33:39     Target Started (NrZen) Target started (Focus Field 6, HIP 28634)
21:36:41       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
21:36:57     Target Completed (NrZen) Target completed (Focus Field 6, HIP 28634)
21:36:59     Target Missed (6/29) Target's time slot was missed (6/29, UGC 3174 w/SN2023mut)
21:37:18   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome)
21:37:20   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
21:37:22   Queue Paused Observing Queue pausing for Session Suspension
21:37:24 Session Suspending Session suspending
21:38:13   Dome Closed Dome closed (Closing time 50s, Zigbee 43s)
21:38:16 Session Suspended Session suspended
21:44:04 Session Resuming Session resuming
21:44:49   Dome Opened Dome opened (Opening time 45s, Zigbee 40s)
21:44:51 Session Running Session running
21:44:53   Queue Resumed Observing Queue resumed
21:44:55     Target Started (7/29) Target started (7/29, AT2024ll (CV))
21:48:15       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
21:52:30       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions
21:54:42     Target Completed Target completed (7/29, AT2024ll (CV))
21:54:46     Target Started (8/29) Target started (8/29, 207P/NEAT)
21:59:12       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
22:18:57     Target Completed Target completed (8/29, 207P/NEAT)
22:19:01     Target Started (9/29) Target started (9/29, IC 2190 w/AT2024dhp)
22:21:44       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
22:24:14       Focusing Completed Foc1 AutoFocus Completed (Profile No 7)
22:40:02     Target Completed Target completed (9/29, IC 2190 w/AT2024dhp)
22:40:06     Target Started (10/29) Target started (10/29, AT2022wlz)
22:43:04       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S)
22:56:23       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
22:57:33       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions
23:00:00     Target Completed Target completed (10/29, AT2022wlz)
23:00:04     Target Started (11/29) Target started (11/29, 103P/Hartley)
23:02:41       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:05:04       Focusing Completed Foc1 AutoFocus Completed (Profile No 8)
23:10:24       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
23:13:15       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions
23:15:31     Target Completed Target completed (11/29, 103P/Hartley)
23:16:11     Target Started (12/29) Target started (12/29, GCVS YZ Cnc)
23:19:45       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
23:20:31       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
23:20:56       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions
23:27:19     Target Completed Target completed (12/29, GCVS YZ Cnc)
23:29:11     Target Started (13/29) Target started (13/29, C/2022 E2 (ATLAS))
23:32:02       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
23:38:53       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
23:41:11     Target Completed Target completed (13/29, C/2022 E2 (ATLAS))
23:41:13   Queue Paused Queue paused for Session Suspension
23:41:15 Session Suspending Session suspending
23:42:03   Dome Closed Dome closed (Closing time 50s, Zigbee 43s)
23:42:06 Session Suspended Session suspended
23:54:39     Target Cancelled Target cancelled (14/29, AT2024nn (CV)) due to cloud
00:14:39     Target Cancelled Target cancelled (15/29, AT2024djo) due to cloud
00:35:35     Target Cancelled Target cancelled (16/29, 29P/Schwassmann-Wachmann) due to cloud
00:55:35     Target Cancelled Target cancelled (17/29, NGC 2642 w/SN2023aaby) due to cloud
01:08:44     Target Cancelled Target cancelled (18/29, GCVS RZ LMi) due to cloud
01:29:33     Target Cancelled Target cancelled (19/29, UGC 11858 w/SN2023rjj) due to cloud
01:43:53     Target Cancelled Target cancelled (20/29, AT2024dsj (CV)) due to cloud
02:25:55     Target Cancelled Target cancelled (21/29, Segue 1) due to cloud
02:47:55     Target Cancelled Target cancelled (22/29, UGC 5886 w/SN2024hw) due to cloud
03:10:15     Target Cancelled Target cancelled (23/29, NGC 3690 w/SN2023wrk) due to cloud
03:25:47 Session Resuming Session resuming
03:26:37   Dome Opened Dome opened (Opening time 50s, Zigbee 41s)
03:26:39 Session Running Session running
03:26:41   Queue Resumed Observing Queue resumed
03:26:43     Target Started (NrZen) Target started (Focus Field 12, HIP 58514)
03:29:08       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:31:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 9)
03:31:44       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
03:33:51       Focusing Completed Foc2 AutoFocus Completed (Profile No 10, wide)
03:35:33       Focusing Completed Foc2 AutoFocus Completed (Profile No 10)
03:35:50     Target Completed (NrZen) Target completed (Focus Field 12, HIP 58514)
03:35:52     Target Missed (24/29) Target's time slot was missed (24/29, NGC 3206 w/SN2024bch)
03:35:54     Target Started (25/29) Target started (25/29, NGC 3467 w/AT2024dda)
03:39:36       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S)
03:54:15       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity)
03:55:35   Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome)
03:55:37   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
03:55:40     Target Aborted (25/29) Target aborted (25/29, NGC 3467 w/AT2024dda) due to cloud
03:55:42   Queue Paused Queue paused for Session Suspension
03:55:44 Session Suspending Session suspending
03:56:35   Dome Closed Dome closed (Closing time 55s, Zigbee 43s)
03:56:38 Session Suspended Session suspended
04:14:07     Target Cancelled Target cancelled (26/29, 62P/Tsuchinshan) due to too few stars
04:17:28 Session Resuming Session resuming
04:18:13   Dome Opened Dome opened (Opening time 45s, Zigbee 41s)
04:18:15 Session Running Session running
04:18:17   Queue Resumed Observing Queue resumed
04:18:20     Target Started (27/29) Target started (27/29, NGC 5036 w/SN2024dop)
04:23:06       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
04:28:09       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
04:28:39       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions
04:32:50       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
04:39:44     Target Completed Target completed (27/29, NGC 5036 w/SN2024dop)
04:39:46   Queue Paused Queue paused for Session Suspension
04:39:48 Session Suspending Session suspending
04:40:36   Dome Closed Dome closed (Closing time 50s, Zigbee 43s)
04:40:38 Session Suspended Session suspended
04:57:17     Target Cancelled Target cancelled (28/29, NGC 4995 w/SN2023gfo) due to cloud
05:29:17     Target Cancelled Target cancelled (29/29, UGC 9002 w/SN2023pls) due to cloud
05:29:23   Queue Completed Job Queue completed
05:29:27 Session Closing Session closing
05:29:32   Dome Closed Dome already closed (closed at 04:40, closing time 50s, zigbee 43s)
05:30:42   Dome Parked Dome parked (parking time 61s), Az: 90.0 deg
05:31:54   Telescope Parked Telescope parked (parking time 70s total)
05:32:09   Telescope State Handbox reads 'Scope parked Turn scope off.'
05:32:31   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
05:33:53   Dome Charging Dome is parked and re-charging Ok (15mA)
05:33:55   Services Stopped Night Services stopped
05:33:58 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files)
05:34:15     Dome (Find Park) Find Park started (Search Az 87.0 to 93.0°, Step 0.2°, Narrow)
05:35:39     Dome Find Park found (Best Park Az 90.4°)
05:35:45     Dome Dome has been synced at park position (Az 90.0°, Adjustment: -0.4°)
05:35:47   Dome Parked Dome parked
05:37:03   Dome Charging Dome is parked and re-charging Ok (20mA)
05:37:05 Session Finished Session Finished

 
Session Alerts & Alarms
Time     Type       Name Detail
18:24:05 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 18:23:26) in section 'GetCameraProperties'
18:24:10 Green Clear CcdCamera CcdCamera has resumed again (18:24) after 44s
19:21:47 Yellow Alert PHD2 Alert No. of incomplete PHD2 stream reads has exceeded 5
19:28:00 Red Alert SBIG Camera Alert Take Image failed (Command timed-out)
19:28:48 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 19:28:10) in section 'GetCameraProperties'
19:28:53 Green Clear CcdCamera CcdCamera has resumed again (19:28) after 43s
19:29:46 Green Clear SBIG Camera Take Image command accepted (bSbigDriverException flag is reset)
19:31:44 Red Alert SBIG Camera Alert Take Image failed (Command timed-out)
19:32:04 Red Alert SBIG Camera Alert Take Image failed (Command in progress. Error code = 219 (0xdb).)
19:32:48 Red Alert CcdCamera CcdCamera appears to be frozen (last cycle at 19:32:09) in section 'GetCameraProperties'
19:32:53 Green Clear CcdCamera CcdCamera has resumed again (19:32) after 44s
19:37:39 Green Clear SBIG Camera Take Image command accepted (bSbigDriverException flag is reset)
19:39:54 Yellow Alert PHD2 Alert No. of incomplete PHD2 stream reads has exceeded 5
19:41:04 Red Alert PHD2 Alert No. of incomplete PHD2 stream reads has exceeded 20
20:22:54 Green Clear PHD2 Alert PHD2 stream reading has stablised again
05:37:03 Green Clear Dome Charging Dome is parked and re-charging Ok (20mA)
 

Back to Top


Operational Issues (2024-03-04 S1196)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Small Defects

Continuous Improvement

[ Prev | Next ]

Fig 1.  Proposed new configuration for Observatory Computing Bay with new UPS System (2024-03-03)

Note: Only 4 of the 8 sockets on the Eaton 3C 850B Unit have battery backup protection. The other 4 have surge protection only.

 Image
 

Fig 2. Final configuration for Observatory Computing Bay with new UPS System (2024-03-07)

In the event it wasn't possible to fit the 4 planned plugs directly into the 4 sockets with battery backup on the UPS. This was due to the 3 of the plugs having a wider footprint than a conventional plug (one a square smart plug for SBIG Camera and two A/C Adpater Plugs for USB Hubs). To get around this a small power strip had to be employed.

 Image

 

Back to Top


2024-03-05


AstroMain - Task Manager Assistant

Recently it has become recognised that there is need to coordinate a series of Tasks so that they can appears in ordered and synchronous mannner and avoid the interlacing of report lines and log messages from different tasks. This includes preventing report lines & log message from tasks from interlacing badly with those from the Job Executor if running.

It is proposed that the function of coordinating such tasks as RestartCCDSoft, RestartTcfFocuser, Restart PHD2 Service etc, should be rolled into a set of routines and operations that are collectively referred to as the 'Task Manager Assistant'  that works alongside and in cooperation with the ObsManager.
.
Example Scenario

The ObsMonitor identifies that the UPB.Switch has appeared to have become frozen / stopped responding and the UPB.Switch Service needs to be restarted.
so it raises a request.  
     RequestTask(TaskCode.RestartSwitch, CallMode.Manual)

The RequestTask(TaskId, Mode) routine accepts the request and stores it in the TM array.   (TM stands for TaskManager)
    TM(TaskId).Requested = True
    TM(TaskId).TimeRequested = Now()

The CheckForTaskRequests() routine which is called from the ObsManager every 5s then looks through the list of possible tasks
  It sees that there is a new request TM(TaskCode.RestartSwitch).Requested = True and acknowledges it by M(TaskCode.RestartSwitch).Queued = True and resets
TM(TaskCode.RestartSwitch).Requested to False.    As there are more than 1 queued taks, it checks if the TaskExecutor is running or not, and if it isn't the 'ExecuteTasks' routine isn't running it submits 'ExecuteTasks' as a new thread.

The ExecuteTask routine goes though the list of potential tasks and finding ones which are queued  (e.g. ... if TM(i).Queued then ...
and calls relevant routines to perform the task.

The TaskManager looks through list of possible tasks and if a task has been queued, it runs it and then waits for it to finish, either as a) a synchonous operation when execution only returns to TaskManager when the task finishes or b) an asynchronous operation where it waits for 'the approriate finished' flag to indicate it can proceed with the next task.

Back to Top


2024-03-11


System Tests - Examining robustness of Observatory Operations to a mains power supply outage

Following a review looking at robustness of the observatory & sessions to a main power supply outage (see Review - Increasing observatory & session resistance to small power outages, 2024-03-28), review and purchase & installation of a UPS system (see  Review - Uninterruptible Power Supply (UPS) for the Observatory Telescope, 2024-03-03 Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply System, 2024-03-03) ) and subsequent actions to improve the AstroMain Observatory Control program on the Observatoy Computer and supporting programs (AstroWeather & AstrolAllSky) on the AllSky/Weather Computer,  a series of System Tests will be carried out in order to verify the .efficiency of actions taken and to identify any remaining gaps or issues.

Test Date   Description Result, Defects & Gaps
1 2024-03-11
16:10:43

Short Test
  Daytime / General Session
No Telescope
No CCD Cooling
Shutter Closed
AstroMain 3.66.2
 
10s outage
Completed.
- MI: AllSky/Weather checks are made too soon after power restored
- MI: Pulsar Dome checks (Shutter Status: Error) are made too soon after power restored
- MI: Zigbee Events not being reported afterwards for Obs.Door
- SD: Dew Heaters values (66%) are shown too soon after being capped at 30%
  (VM.AutoDewState is being reported before it has had chance to fresh by RunObsMonitor)
- SD: ObsPic shows cyan flag against Dome T. when its value goes to 0 during outage.
- SD: Small Reporting Defects (cloud sensor time ago in wrong units)
- CI: LAN & Internet needs to be checked after power restoration.
         
2 2024-03-11
16:28:56

Long Test
  Daytime / General Session
No Telescope
CCD Cooling
Shutter Closed
AstroMain 3.66.2
 
11 min outage

Completed.
- MaI: Internet not available after power restoration (manually power-cycling the Ethernet over Powerline didn't fix it)
- MaI: AstroPlan/AstroAllSky Web Uploaders stopped uploading at start of outage and didn't automatically restart
- MI: CCD Cooling wasn't restored after power restoration (no attempt was made !)
- SD: ObsPic show TCF-S On (green) whilst its power is off 16:37 to 16:40:
 -CI: CloudSensor Flags (Rain, Clarity, Light) on ObsPic need to be set to white during outage.
- CI: Dome Flags on ObsPic need to be set to white during outage.
- CI: Need to add a Smart Plug to reboot Switch Hub from AstroMain.

UPS Battery Capacity fell from 100% to 86%.  Load fell to just 13 W (est.)

         
3 2024-03-12
13:48:41

Short Test
(Test 1 Rpt)
  Daytime / General Session
No Telescope
No CCD Cooling
Shutter Closed
AstroMain 3.66.3
 
10s outage
Completed

Issues
- MI: Shutter Check shows Error during Post Outage Checks
- SD: AstroAllSky didn't restart WebUploader (bug)
- Checking Weather data, Last data ago might be from pre-outage ?
         
4 2024-03-12
14:58:36

Short Test
(Test 1 Rpt)
  Daytime / General Session
No Telescope
No CCD Cooling
Shutter Closed
AstroMain 3.66.3 (updated)
 
14s outage
Completed
- SD: AstroAllSky didn't restart WebUploader (bug)
- SD: ObsPic shows cyan flag against Dome T. when its value goes to 0 during outage.
- SD:  values (0%) after being capped at 30% ?
- CI: LAN & Internet needs to be checked after power restoration.
         
5 2024-03-12
15:01:52 

Long Test
  Daytime / General Session
No Telescope
No CCD Cooling
Shutter Closed
AstroMain 3.66.3 (updated)
Reset Ethernet Switch
Manually
 
11 min outage

UPS Battery Capacity
fell from 100% to 86%
Load fell to just 13 W (est.)

Completed
- MI: SBIG Camera not turned on after power restoration  ?
- MI:  Weather Station data stopped refreshing at 15:01 and didn't come back after outage
- SD: Dew Heaters values (16%) after being turned off.
- SD: Log file containing multiple "Exception in Telecomm.SetIPAddress" messages
 - CI Restart Virtual Weather Station after restart (manual fix worked).

         
6 2024-03-12
16:53:13  

Long Test
  Daytime / General Session
WithTelescope
With CCD Cooling
State: Running
Shutter Closed
AstroMain 3.66.3 (updated)
Reset Ethernet Switch
Manually
 
10.2 min outage

Completed. Scope couldn't be restarted.
- MaI: RestartCCDSoft unexpectedly kicked in at 16:55 & 16:56 due to 'frozen CcdCamera'
- MaI: During Park Scope not allowed to Turn Off Scope Power (Running / Suspend)
   (had to turn off manually from Pegasus PDB Software)
- MaI: Restart Scope failed as Power was off
- MI: RecoverAfterOutage Thread held up by bug in ConnectTheSkyTelesope
- MI: After stopping/restarting Services, Outage Management Stage 6 began again (from prior settings / need to reset them prior to starting again ?) and proceeded to Restart TCF-S and Restart SBIG Camera,  Check Dome, Check AllSKy/Weather data
- MI: Weather Station stopped refreshing at 15:01 and didn't come back after restore
- SD: Dew Heaters values (50%) after being capping at 30%
- SD: Dew Heaters values (22%) after turning them off.
- SD: UPS Remaining Battery reporting every 1 min when this was disabled til lbattery < 20 mins
- SD: SBIG Camera not turned on after power restoration  ?
- SD: Connect TheSky6 Telescope unexpectedly reporting Telescope Setup (TeleAPI)
- SD: Log file containing multiple "Exception in Telecomm.SetIPAddress" messages
- CI: HardSuspend timeout to be logged./reported
- CI:  Restart Virtual Weather Station after restart if fresh data doesn't resume (manual fix worked).
- CI:  Cam1 CCD :to show SBIG Off if not powered up.

         
7 2024-03-13
00:55:15
  Nighttime / General Session
Actual 11s power cut
Actual Power Cut, provided an unplanned 'test'
- MaI: Cloud Sensor data didn't come back after power cut.
- MI: Shutter status reported to be 'Error' due post-outage Dome checks
- MI: NightSummaryPlot (allsky trace) affected between 00:55 and the following outage recovery 03:40
- SD: Reported Dew Heater values not reflecting 30% cap
- CI: ObsPic should show white flag when CloudSensor data is offline
- CI: Obs Pic should show Dome Power as Off during the outage.
- CI: AstroWeather should recognise issue and repeat its 'Restart CloudSensor' routine
         
8 2024-03-13
03:40:08
  Nighttime / General Session
Actual 11s power cut
Actual Power Cut, provided an unplanned 'test'
- MaI: Cloud Sensor data still didn't come back after new power cut.
- MI: Reported Dew Heater values (72%) not reflecting 30% cap
-
         
9 2024-03-13
10:18:16
  Daytime / General Session
Actual 1.8 min power cut
Actual Power Cut, provided a further unplanned 'test'
- CI: Restart WebLoader might need to wait until an internet location can be successfully pinged
  (LAN access doesn't demonstrate internet access)
         
10 2024-03-13
14:38:10   

Long Test
  Daytime / General Session
WithTelescope
With CCD Cooling
State: Running
Shutter: Open
AstroMain 3.66.4
 
12 min outage

Completed. Scope couldn't be restarted.
- CI: Dome wouldn't close at expected 5m after Power Off (shut at 14:51:59 , which is around 13 min after)
- MaI: Restart Scope failed as Power was off (14:50)
- MI:  Weather Station data didn't come back after power restored
- SD: UPS Remaining Battery reporting every 1 min when this was disabled til lbattery < 20 mins
- SD: slight reporting / misalignments during Switch On SBIG Camera
- SD: Log file containing multiple "Exception in Telecomm.SetIPAddress" messages
- CI: HardSuspend timeout to be logged./reported
- CI:  Restart Virtual Weather Station after restart if fresh data doesn't resume (manual fix worked).
- CI:  Cam1 CCD :to show SBIG Off if not powered up.

         
         
         
         

xxx

Back to Top