David's Astronomy Pages Notes - Session 1194 (2024-02-27)

Notes (S1193)

Notes Main

Home Page

Notes (S1195)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot  - Session Event Log
	Operational Issues
	- Critical Issues (0),  Major Issues (0),  Minor Issues (0),  Small Defects (2),  Continuous Improvement (4)
	Images from 2024-02-27 >>
	2024-02-28
	Review - Increasing observatory & session resistance to power outages

Session Aims & Highlights (2024-02-27)

Main aims

1. Focusing.  Refocus telescope following the Foc1 Focusing failure on Near Zenith Focus Field & out of focus images during S1193 session the previous night
2. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
3. 12P/Pons-Brooks. Acquire images of Comet 12P/Pons-Brooks which has been undergoing regular outbursts.
4. AstroMain. Continue to test fixes to CCDSoft Camera/SBIG Camera operations to prevent stalled execution during starting, fix issues due to occasional repeated failures during Cam1 focusing and occassional unexpected CCD Temperature regulation.

 Equipment & Software

• 12" LX200 and ST-10XME
• 80mm APO and ZWO ASI 178MC
• TheSky6, CCDSoft5, DeviceHub 6.6.1.1,  MeadeGeneric 1.3.5.414, UPB 1.5, ASCOM 6.6,   Pulsar Dome Firmware 1.51 & Driver 6.3.10
• Pegasus UPB & Motor Focus, TCF-S
• SharpCap 3.2, PHD2  2.6.13, Siril 1.0.6, deCONZ 1.18.2, ASCOM 6.6 SP1
• [ DomeLink 1.0 / 1.1 - not used ]
   
• AstroMain 3.65.21,  AstroPlan 1.39.7,  AstroShCap 1.3.15, AstroGuard 1.10.2   (any new items or versions in bold)
  AstroVOE 1.13,  AstroSN 1.10.2,  AstroProtect 1.7,  AstroMove 1.12.11
  (AstroWeather 1.23.10, AstroAllSky 3.19.11,  AstroObsCam 1.11.4, AstroLaunch 1.11.1

Highlights

• Focusing. Main scope was focussed ok on Near Zenith Focus Field and focus was adjusted through the session.  Focus offset from Near Zenith Focusing (3.5°C) appears to have jumped by +874 focuser steps since the end of the S1192 session (4.3°C).  Compared to Near Zenith focusing at start of the S1192 session  (2.1°C) the jump in focus offset is less,  but is still +410 focuser steps.
• Targets. 23/35  planned targets observations were completed.. Other targets failed due to cloud / session suspension . Images from two targets (62P/Tsuchinshan  & NGC 3467 w/AT2024dda) were later rejected due to cloud.
• 12P/Pons-Brooks. Images of Comet 12P/Pons-Brooks were successfully acquired
• AstroMain. No issues with CCDSoft Camera/SBIG Camera.
•  Dome Parking.  Dome parked ok at the correct position at end of the session.

Notes:

• ...

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Actual Weather vs Pre-Session Weather Forecast

Session Event Log

Time     Event Detail 18:06:31 Session Monitoring AutoStart monitoring for Live Session opportunity between 18:27 & 05:02 18:06:34   AutoStart Waiting AutoStart waiting till earliest start time at 18:27 18:27:47   AutoStart Delayed Session delayed due to too few stars 18:36:33 Session AutoStarting Session autostarting (18:36) 18:36:43   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.) 18:36:57 Session Created Session Created (Live, 2024-02-27 S01194, ImageSaveNum: 1194001) 18:37:03   Scope Switched On Telescope Power has been switched on via UPB Powerbox. 18:38:49   Services Started Observatory Services started 18:38:56 Observatory (Auto) Observatory placed in Fully-Automated Mode 18:38:58 Session Pending Session pending (2024-02-27) 18:39:00 Session Initiating Session initiating (2024-02-27) 18:39:02   Plan Requested Observing Plan requested from AstroPlan (1.39.7) 18:40:17   Plan Loaded Observing Plan loaded to queue (Plan ID: 968) 18:40:38   Camera1 Connected SBIG Camera connected (Set point -20°C) 18:41:15   Telescope Connected Telescope connected (TheSky6) 18:41:40 Session Equilibration Session ready to Open Dome 18:42:25   Dome Opened Dome opened (Opening time 45s, Zigbee 41s) 18:42:27 Session Running Session running 18:42:29   Queue Started Observing Queue started (35 targets selected) 18:42:30     Target Started (NrZen) Target started (Focus Field 5, HIP 23569) 18:42:45       Dome Unparked Dome unparked 18:47:20       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 18:49:24       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide) 18:51:34       Focusing Completed Foc1 AutoFocus Completed (Profile No 1) 18:51:37       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap) 18:53:27       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide) 18:55:09       Focusing Completed Foc2 AutoFocus Completed (Profile No 2) 18:55:26     Target Completed (NrZen) Target completed (Focus Field 5, HIP 23569) 18:55:30     Target Started (1/35) Target started (1/35, 12P/Pons-Brooks) 18:59:31       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 19:01:27       Focusing Completed Foc1 AutoFocus Completed (Profile No 3) 19:13:47     Target Completed Target completed (1/35, 12P/Pons-Brooks) 19:13:52     Target Started (2/35) Target started (2/35, UGC 831 w/AT2023wdd) 19:17:06       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 19:19:33       Focusing Completed Foc1 AutoFocus Completed (Profile No 4) 19:35:33     Target Completed Target completed (2/35, UGC 831 w/AT2023wdd) 19:35:52     Target Started (3/35) Target started (3/35, 13P/Olbers) 19:39:27       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 19:50:47     Target Completed Target completed (3/35, 13P/Olbers) 19:53:42     Target Started (4/35) Target started (4/35, UGC 12639 w/SN2023jri) 19:57:08       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 20:13:18     Target Completed Target completed (4/35, UGC 12639 w/SN2023jri) 20:15:42     Target Started (5/35) Target started (5/35, Jupiter) 20:22:45     Target Completed Target completed (5/35, Jupiter) 20:31:39     Target Started (6/35) Target started (6/35, 103P/Hartley) 20:35:12       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 20:37:23       Focusing Completed Foc1 AutoFocus Completed (Profile No 5) 20:47:49     Target Completed Target completed (6/35, 103P/Hartley) 20:48:23     Target Started (7/35) Target started (7/35, AT2023cwd (CV)) 20:52:02       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 20:58:11     Target Completed Target completed (7/35, AT2023cwd (CV)) 21:00:23     Target Started (8/35) Target started (8/35, UGC 3174 w/SN2023mut) 21:03:48       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:05:59       Focusing Completed Foc1 AutoFocus Completed (Profile No 6) 21:21:58     Target Completed Target completed (8/35, UGC 3174 w/SN2023mut) 21:22:23     Target Started (9/35) Target started (9/35, AT2023bml) 21:25:16       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:27:28       Focusing Completed Foc1 AutoFocus Completed (Profile No 7) 21:43:27     Target Completed Target completed (9/35, AT2023bml) 21:44:23     Target Started (10/35) Target started (10/35, AT2023rey (CV)) 21:47:06       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 21:49:19       Focusing Completed Foc1 AutoFocus Completed (Profile No 8) 22:05:07     Target Completed Target completed (10/35, AT2023rey (CV)) 22:06:23     Target Started (11/35) Target started (11/35, AT2023cub (CV)) 22:09:26       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 22:11:35       Focusing Completed Foc1 AutoFocus Completed (Profile No 9) 22:17:23     Target Completed Target completed (11/35, AT2023cub (CV)) 22:18:23     Target Started (12/35) Target started (12/35, AT2023cuz) 22:21:06       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 22:23:31       Focusing Completed Foc1 AutoFocus Completed (Profile No 10) 22:39:44     Target Completed Target completed (12/35, AT2023cuz) 22:40:23     Target Started (13/35) Target started (13/35, GCVS U Gem) 22:43:12       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 22:45:38       Focusing Completed Foc1 AutoFocus Completed (Profile No 11) 22:48:46     Target Completed Target completed (13/35, GCVS U Gem) 22:49:56     Target Started (14/35) Target started (14/35, SN 2022ukr) 22:53:22       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 23:09:42     Target Completed Target completed (14/35, SN 2022ukr) 23:12:16     Target Started (15/35) Target started (15/35, CGCG 119-104 w/SN2024cdi) 23:15:49       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 23:31:54     Target Completed Target completed (15/35, CGCG 119-104 w/SN2024cdi) 23:34:36     Target Started (16/35) Target started (16/35, NGC 3690 w/SN2023wrk) 23:38:17       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 23:40:29       Focusing Completed Foc1 AutoFocus Completed (Profile No 12) 23:56:33     Target Completed Target completed (16/35, NGC 3690 w/SN2023wrk) 23:56:56     Target Started (17/35) Target started (17/35, GCVS RZ LMi) 00:00:20       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:02:22       Focusing Completed Foc1 AutoFocus Completed (Profile No 13) 00:07:30     Target Completed Target completed (17/35, GCVS RZ LMi) 00:08:34     Target Started (18/35) Target started (18/35, GCVS AM CVn) 00:11:34       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:13:45       Focusing Completed Foc1 AutoFocus Completed (Profile No 14) 00:16:51     Target Completed Target completed (18/35, GCVS AM CVn) 00:18:07     Target Started (19/35) Target started (19/35, NGC 3893 w/AT2023uhx) 00:21:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 00:23:43       Focusing Completed Foc1 AutoFocus Completed (Profile No 15) 00:39:46     Target Completed Target completed (19/35, NGC 3893 w/AT2023uhx) 00:40:27     Target Started (20/35) Target started (20/35, 62P/Tsuchinshan) 00:44:57       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 00:56:55     Target Completed Target completed (20/35, 62P/Tsuchinshan) 00:58:29     Target Started (21/35) Target started (21/35, NGC 3467 w/AT2024dda) 00:59:57       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 01:01:53       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S) 01:18:24     Target Completed Target completed (21/35, NGC 3467 w/AT2024dda) 01:18:26   Queue Paused Queue paused for Session Suspension 01:18:28 Session Suspending Session suspending 01:19:15   Dome Closed Dome closed (Closing time 50s, Zigbee 44s) 01:19:17 Session Suspended Session suspended 01:39:25     Target Cancelled Target cancelled (22/35, CGCG 217-021 w/AT2023clc) due to cloud 02:01:45     Target Cancelled Target cancelled (23/35, SN 2022mox) due to cloud 02:24:06     Target Cancelled Target cancelled (24/35, NGC 3206 w/SN2024bch) due to cloud 02:46:05     Target Cancelled Target cancelled (25/35, M105 w/AT2024bpy) due to cloud 03:08:25     Target Cancelled Target cancelled (26/35, UGC 8317 w/AT2022qvu) due to cloud 03:30:45     Target Cancelled Target cancelled (27/35, IC 4688 w/SN2023ftg) due to cloud 03:45:05     Target Cancelled Target cancelled (28/35, AT2023txn) due to too few stars 03:46:49 Session Resuming Session resuming 03:47:34   Dome Opened Dome opened (Opening time 45s, Zigbee 41s) 03:47:36 Session Running Session running 03:47:38   Queue Resumed Observing Queue resumed 03:47:41     Target Started (29/35) Target started (29/35, AT2023clo (CV)) 03:51:04       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S) 03:53:00       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 03:53:02       Focusing Completed Foc1 AutoFocus Completed (Profile No 16) 03:53:10       SoftSuspend Cancelled Soft Suspend is cancelled due to improved conditions 03:59:20     Target Completed Target completed (29/35, AT2023clo (CV)) 03:59:24     Target Started (30/35) Target started (30/35, UGC 11858 w/SN2023rjj) 04:02:35       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S) 04:07:01       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars) 04:18:29     Target Completed Target completed (30/35, UGC 11858 w/SN2023rjj) 04:18:31   Queue Paused Queue paused for Session Suspension 04:18:33 Session Suspending Session suspending 04:19:23   Dome Closed Dome closed (Closing time 50s, Zigbee 44s) 04:19:26 Session Suspended Session suspended 04:29:43     Target Cancelled Target cancelled (31/35, GCVS BL Lac) due to cloud 04:45:12     Target Cancelled Target cancelled (32/35, GCVS S Cyg) due to cloud 04:58:12     Target Cancelled Target cancelled (33/35, AT2023abtv (CV)) due to cloud 05:18:32     Target Cancelled Target cancelled (34/35, AT2023tow) due to cloud 05:50:32     Target Cancelled Target cancelled (35/35, AT2023tki) due to cloud 05:50:37   Queue Completed Job Queue completed 05:50:40 Session Closing Session closing 05:50:45   Dome Closed Dome already closed (closed at 04:19, closing time 50s, zigbee 44s) 05:51:13   Dome Parked Dome parked (parking time 20s), Az: 90.0 deg 05:52:39   Telescope Parked Telescope parked (parking time 83s) 05:52:54   Telescope State Handbox reads 'Scope parked Turn scope off.' 05:53:16   Telescope Switched Off Telescope Power has been switched off via UPB Switch. 05:54:40   Dome Charging Dome is parked and re-charging Ok (4mA) 05:54:42   Services Stopped Night Services stopped 05:54:45 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files) 05:55:22     Dome (Find Park) Find Park started (Search Az 87.0 to 93.0°, Step 0.2°, Narrow) 05:56:50     Dome Find Park found (Best Park Az 90.6°) 05:56:57     Dome Dome has been synced at park position (Az 90.0°, Adjustment: -0.6°) 05:56:59   Dome Parked Dome parked 05:58:35   Dome Charging Dome is parked and re-charging Ok (14mA) 05:58:37 Session Finished Session Finished

Session Alerts & Alarms

Time     Type       Name Detail 05:58:35 Green Clear Dome Charging Dome is parked and re-charging Ok (14mA)

Back to Top

Operational Issues (2024-02-27 S1194)

• Critical Issues (0)
• Major Issues (0)
• Minor Issues (0)
• Small Defects (2)
• Continuous Improvement (4)
  

[ Prev | Next ]

Critical Issues

• None

Major Issues

• None

Minor Issues

• None

Small Defects

• TCF-S Focuser Service can't be restarted if Focuser 1 power is turned off (post-session testing). Attempting to Restart the TCF-S Focuser service fails with the message "Unable to restart (Focuser1 Power is off)",  yet the Restart Operation involves turning off the Focuser 1 power and then turning it on again, so it could proceed with the Restart Service regardless of whether the power is initially on or not.  Fixed in AstroMain 3.65.22.
  Done  2024-02-28 (AstroMain 3.65.22)
   
• AllSky Heater and Observatory Dehumidifier unexpectedly turned on briefly after starting AstroMain / Continue General Session (2024-02-29 13:03).  After continuing General Session 2024-02-28 the follows lines were reported in the middle of 'Start Services' (incl ObsManager) execution breaking up the flow of report lines
  
    > Dehumidifier    Info    Dehumidifier has switched on (13:03:19, off time > 16 sec)
  > AllSky Heater   Info    AllSky Heater has switched on (13:03:19, off time > 16 sec)
  Very shortly after Start Services had completed the following lines were reported:
  
    > Dehumidifier    Info    Dehumidifier has switched off (13:03:49, on time 29 sec)
  > AllSky Heater   Info    AllSky Heater has switched off (13:03:49, on time 29 sec)
  Humidity levels were low at the time and there was no good reason for either item to have been switched on. This would appear to be a bug.  Zigbee log shows that commands were issued at 13:03:19 to turn on the   Dehumidifier (lights/5) and a second command to turn on the AllSky Heater (lights/7).  It is possible that they didn't have fresh (real) Humidity values had had only null values ?  Log file show lines at 13:03:19 with messages
  
     (Manager)   Exec   TurnOnDehumidifier      Info | Dehumidifier will be turned on (Obs Humidity = 0)
   (Manager)   Exec   TurnOnAllSkyHeater      Info | AllSky Dew Heater will be turned on (Humidity = 0)
  This shows that we have 'invalid'  humidity values (0). Code show that that  The two items are managed via ManageDehumidifier() and ManageAllSkyHeater() routines respectivily,  when they are on 'Auto'.  Fixed in AstroMain 3.66, code modified to include an extra reason why device can't be turned on
  
      ElseIf Env.DataIsValid(EProp.WeatherStation) = False Or
           Env.ObservatoryHumidity = vcNull Or
           Env.ObservatoryHumidity = 0 Then
             S = "No Humidity data"
  Auto-monitoring for both items also modified to exit the routines if Starting Service is running to reduce the likelihood that report lines are interlaced. 
  Fixed  2024-02-29 (AstroMain 3.66)

Continuous Improvement

• Make a number of cosmetic tweaks to Shutter Battery Charts. Make a number of improvement to Shutter Battery Charts (GraphID 28 & 40), position of Fully Chrged and Parked flag, position and thickness & width of Legend. flag lines. Implemented in AstroWeather 1.23.12. 
  Done  2024-02-28 (AstroWeather 1.23.12)
   
• Query. If  the LX200 Telescope is powered and in operation is there is a risk of accidentally turning off its power if UPB Powerbox Software is restarted. ? The LX200 is powered via the Output 1 of Pegasus UPB Powerbox attached to Telescope. (Output 1 is not set to Power On at startup). If the Telescope is turned off without parking the scope, first the telescope will not have correct position information when it is next started up causing a significant hold-up until the telescope's pointing has been resynced on a know star and a short mapping run is conducted (it's possible that the Telescope Park position may need to be re-defined ?).   If the user closes the UPB Powerbox software and then restarts it or the AstroMain kills and restarts the Powerbox Software.is there a risk that the power to scope via output 1 will be switched off when the UPB Software is started.  A test was conducted (2024-02-28) to answer this question.   Output 3 (which is similarly not set to Power On at startup) was turned ON, and the Powerbox Software closed.  Upon restarting the UPB Software the state of Output 3 was still ON.  This shows that a powered up LX200 Scope on Output 1 can't be turned off by a closure of the UPB Software.   (The LX200 Scope will almost certaining turned off if the UPB Powerbox Controller is rebooted or there is a power cut and the Powerbox reboots on power restoration, however the scope would already be powered off by the powercut itself )
  Question answered  2024-02-28 (Pegasus UPB Powerbox, LX200)
   
• Transferring files onto Observatory Computers seems a bit slow than normal (2024-02-28 late pm). Speed tests run at 00:15 showed the Average Speed of 9.21 Mbps, 1.15 MB/s  for connection to Observatory Computer (LAPTOP9) and 8.51 Mbps, 1.06 MB/s for connection to AllSky/Weather Computer (LAPTOP7) (see Fig 1)  A second test a few days later (2024-03-01) showed much faster performance and confirm that LAN link to observtory was indeed somewhat slow on the night of 2024-02-28. It showed  Average Speed of 12.14 Mbps, 1.52 MB/s  for connection to Observatory Computer  and 13.34 Mbps, 1.67 MB/s  for connection to AllSky/Weather Computer (LAPTOP7).
    To Monitor  2024-03-01 (LAN)
   
• Extend Job Executor to insert throw-away frames to remove the ghost of Jupiter before taking images for the next deep sky target. Imaging Jupiter in variably leads to an electron signature of Jupiter being temporarily embedded in the CCD Chip which can be released as a residual image signal / ghost on subsequent target images. After finishing Jupiter or other bright object throw-away (snap) frames should be acquired to eliminate or at least minimise the strength of bright objects ghost in image frames belonging to the next target.’’  
    (Paper : Residual images in charged-coupled device detectors, Rest et al, 2002,   https://web.pdx.edu/~d4eb/ccd/RevScInstr_73_2002_2028.pdf ).
  Implemented in AstroMain 3.67.1,  9 x 5s Dark Frames are taken using a call to new subroutine ClearGhostImage() in T.TargetName = Jupiter, Mars, Vega,Moon. The new features was utilised for first time in session S1204 after images of Jupiter where acquired (S1204,  2024-04-04).
  Done 2024-03-29 (AstroMain 3.67.1)

[ Prev | Next ]

Back to Top

2024-02-28

Review - Increasing observatory & session resistance to power outages

Issue

• Small power cuts can knock-out a number of elements of the Observatory system which can cause loss of monitoring and stop equipment operation/connection, which then remain not working/not available, even when power is restored,  until the particular element(s) are manually restored.
   
• A power cut during Telescope operations leaves the Telescope Powered down in an 'Unparked' State.  This when telescope starts up it no longer has any knowledge about it physical point direction and has to be resynced & re-tied to the mount's TPoint Model.
   
•  In addition small power cuts could disrupt the zigbee event monitoring and result in the Greenhouse Heater coming back on again after the power cut and then remain uncontrolled, consuming electricity unnecessarily and costing money.

Note:  The issue under review concerns short outages in power supply only, ones that come back on after seconds or a few minutes, and not about longer outages in which battery's on observatories computers are drained leaving no automated recovery after the outage ends.

Review

The purpose of this review is two fold  :

1) consolidate the full list of actions to harden the Observatory System & Equipment to the effects of short interruptions in power supply.  What is the specific programming work and/or other work neccesary to achieve the automated recovery for each element of the observatory following a power cut.

 2) examine whether a UPS System (Uninterruptible Power Supply) can be added to the Observatory in a way that provide power to Observatory Equipment in the event of a short outage in incoming power supply and in a way that doesn't effect existing Ethernet Over Powerline / LAN usage. 

Status

Problems caused small power outages have been previously recorded and investigated in various session notes, the most recent of which was the outage at 06:14 on 2024-04-26 which occurred shortly after the finish of the S1192 Session  (see Operational Issues, S1192). There have been several small outages in the last fortnight for reasons unknown.

Equipment/System Recovery  - TBA
The effects of power outage on each element of the observatory system, and whether this based on a known incident or is hypothesized is shown in the table below :

Element	Effect during Power Outage	Effect of Power Resumption	After Effect	Impact		Existing Recovery	Work Required (with updated status at 2024-03-11)
Power Supply	- Power supply goes down - Ethernet over Poweline stops	- Power Supply restored - Ethernet over Powerline resumes	- Various Observatory elements remain down until restarted manually.	HIGH		Various	Modify AstroMain to record outages better and more timely - Done   Modify AstroMain to recognise outages shorter than 5s (currently might only recognise outages > 5s) - Done Consider UPS system for critical Observatory elements - Done
LX200 Telescope	- LX200 powers down (due to loss of power to UPB Powerbox)	- Scope likely stays powered down (UPB Powerbox has the LX200 Off at start-up)	- LX200 not practically available until telescope is resynced on known star.	V.HIGH		Significant manual effort required to correct Telescope Pointing	Modify AstroMain to suspend Live Session when a power cut occurs - Done    Consider UPS system for UPB Powerbox in order to protect LX200 from an uncontrolled shut down -Done
CloudSensor	- Cloud Sensor data goes off	- Cloud Sensor data stays off	- No fresh CloudSensor data to AstroWeather, AstroAllSky, AstroMain	HIGH		Manual restart of Cloud Sensor Program	Modify AstroWeather to Auto-Restart the Cloud Sensor Program - Done
TCF-S Focuser	- TCF-S powers down - TCF-S service goes down	- TCF-S powers up - TCF-S service stays down	TCF-S service stays down	HIGH		Manual restart of TCF-S Focuser service	Modify AstroMain to Auto-Restart the TCF-S Focuser service - To Do
Zigbee Sensors	- Zigbee plugs and connected sensors stop - AstroMain's WebSocket connection to deCONZ stops - AstroMain stops receiving events about zigbee sensors	deCONZ WebSocket remains unavailable	deCONZ WebSocket remains down AstroMain doesn't recognise when there are changes to the zigbee sensors AstroMain unable to control plugs like the GreenHouse Heater, AllSky Heater, Dehumidifier.	MED		Manual Restart deCONZ program plus a manual restart of the Zigbee Service Might only have to Restart the Zigbee Service once power is restored.	Add button to AstroMain to Restart Zigbee Service  Modify AstroMain to recognise outage in WebSocket Service and automatically restart zigbee service.
UPB Powerbox	- Powerbox powers down - UPB Switch stops			MED		OverOverseer restarts PowerBox Program /Switch	Modify AstroMain to only restart UPB Switch if Power is On - Done
AllSky Camera	- CCD stops getting fresh images from Camera - AstroAllSky repeatedly processes the same image, or gets error code=206.	None	CCDSoft continues to not get fresh images from the camera AstroAllSky continues to repeatedly process the same image or gets error code=206	MED		Restart CCDSoft Restart AstroAllSky Later shown to just need to disconnect and then reconnect the CCDSoft.Camera	Fix problem of repeated processing of same image & Error Code=206 in AstroAllSky - Done
Pulsar Dome	- Dome powers down - Volatile records show '0'  for each data field - Shutter shuts if/when outage exceeds 5 mins	- Dome Powers Up - Volatile records with fresh data again	None	LOW		n/a	Modify AstroMain to ignore Shutter battery level of 0% - Done

Note : Even if a UPS System is added to the Observatory, actions to provide automated recovery for the particular observatory elements identified is still warrented since they fill the gap before any UPS system is installed or would be useful anyway as a element might fail for reasons other than a short power outage and therefore an automated recovery of that element would still be appropriate & useful.

Power Monitoring (Current)

AstroMain's Power Monitoring is currently performed in two places.

1. ObsMonitor has a "Monitor Power" sectiion that normally executes every 2.5s  (but can be longer if the ObsMonitor is held up by any slow tasks).
    
   If the computer is on A/C Power the monitor sets iLaptopPowerOk = 2, whilst If the computer is on Battery it sets iLaptopPower to either 1 or 0 depending on Battery Level.  
    
   Since the Observatory Laptop is permanently attached to a A/C Adapter plugged into the Observatory Power Supply, the variable can be used to monitor the Power Supply status (i.e. whether Power Supply is on (the normal state) or it is off (as in a power outage).  Turning off the A/C Adapter by the user or a failure of the Adapter itself would obviously give a false indication of the Observatory Power Supply status
    
   'LaptopPowerOutageStartTime' is set to the time when the computer is recorded as switching to battery (or the time that monitor starts running if is already battery).  'NumLaptopPowerOutages' is incremented each time an new outage is detected.
    
2. ObsManager has "Power Outages"  & "Power Outage Period" sections that normally execute every 5s.
   
   If 'NumLaptopPowerOutages' is greater than the previous number ('LastNumLaptopPowerOutages') an Alert is raised and reported with the current time.
   This will indicate that at least one power outage has occurred within the past 5s or so, but not the time that outage begun, or whether there's been more than one outage within the period. 
   
   Alerts are raised and reported when a Power Outage lasts more than 30s.  
    
   A further alert is raised and reported when a Power Outage lasts more than 20 minutes.

Critic on current monitoring:

• The time that a power outage starts (LaptopPowerOutageStartTime) is approximate. The outage may have actually begun up to 2s or so before this time, or longer if ObsMonitor looping is slowed down by other tasks.
   
• Power outages shorter than 2s or so are not necessarily detected  e.g the case where the Computer switches to Battery and  back to A/C Power between monitoring reads.
   
• The time that an outage was recorded to start (LaptopPowerOutageStartTime) is not reported, instead the time that is reported by the ObsMananger is the current time which could be up to 5s later than the time that the outage was recorded.  The timestamp written with event/alert  to database  and the position of reported outage in log file are potentially shown after events/alerts than are a consequence of the outage.
   
• Only one outage can be reported in a 5s period.
   
• No all-clear 'alert' is raised and reported with a power outage ends. The duration of an outage is not reported.

Proposal:

• Power Supply Monitoring is moved and placed in a new separate thread.
   
• The new monitor runs on a more frequent basis (say 200 ms) and conducts all the necessary raising of events and reporting which can be more accurate and more timely.
   
  Reporting of alerts/events no longer depends on the ObsManager to be running.
   
  (reporting of alerts & events associated with power  being off for 30s, 20 min are retained by the ObsManager)

Update:

• New Monitor ('PowerMonitor') implemented in AstroMain 3.66

UPS System

The potential using one or two UPS Systems with offline battery backup are identifed as options for further consideration.   However due to the finding that it is the Observatory LX200 Telescope that can be most affected by an unscheduled power outage, the focus of the review turned to the selection of a UPS System with Battery Backup.  See Review - Off Line Uninterruptible Power Supply (UPS) for the Observatory Telescope.

Actions

•   AstroMain. Modify AstroMain to record outages better and more timely.  
  Done, 2024-02-29 (AstroMain 3.66)
   
• Power Plugs.  Carry out audit of Power Plugs & Connected Devices that are located in the Observatory's two bays ahead on installation of a UPS System to use as input to decision over what equipment will be assigned to the 4 UPS sockets that will have back-up and input to a decision on whether to purchase and install a second, separate UPS System for devices in the AllSky/Weather bay .
  Done  2024-03-04  (Observatory)

Back to Top