David's Astronomy Pages
Notes - Session 1195 (2024-02-29)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (3),  Minor Issues (4),  Small Defects (10),  Continuous Improvement (14)
 
Bullet Images from 2024-02-29 >>         
   
Bullet Investigation - Telescope Failed to Park at end of session and was turned off in an 'unparked' state
Bullet Investigation - Shutter Opening/Closing Issues (comms between Shutter & Controller frozen)
   
2024-03-02
Bullet MeadeGeneric 1.3.9
Bullet Recovery from Telescope Shut-Down in Unparked State
   
2024-03-03
Bullet Review - Uninterruptible Power Supply (UPS) for the Observatory Telescope
Bullet Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply System
   

Session Aims & Highlights (2024-02-29)

Main aims

  1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
  2. 12P/Pons-Brooks. Acquire images of Comet 12P/Pons-Brooks which has been undergoing regular outbursts.
  3. AstroMain. Continue to test fixes to CCDSoft Camera/SBIG Camera operations to prevent stalled execution during starting, fix issues due to occasional repeated failures during Cam1 focusing and occassional unexpected CCD Temperature regulation.

 Equipment & Software

Highlights

Notes:

Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Image
 
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Image
  
Sky Conditions (Locate Frames)
Image 
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image
  
Session Event Log
Time     Event Detail
17:58:52 Session Monitoring AutoStart monitoring for Live Session opportunity between 18:32 & 04:57
17:58:57   AutoStart Waiting AutoStart waiting till earliest start time at 18:32
18:32:06   AutoStart Delayed Session delayed due to too few stars
18:53:49 Session AutoStarting Session autostarting (18:53)
18:53:59   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.)
18:54:24   Camera1 Connected SBIG Camera connected (Set point -20°C)
18:54:26 Session Created Session Created (Live, 2024-02-29 S01195, ImageSaveNum: 1195001)
18:54:31   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
18:56:17   Services Started Observatory Services started
18:56:23 Observatory (Auto) Observatory placed in Fully-Automated Mode
18:56:26 Session Pending Session pending (2024-02-29)
18:56:28 Session Initiating Session initiating (2024-02-29)
18:56:30   Plan Requested Observing Plan requested from AstroPlan (1.39.7)
18:57:26   Plan Loaded Observing Plan loaded to queue (Plan ID: 969)
18:57:41   Camera1 Connected SBIG Camera connected (Set point -20°C)
18:58:16   Telescope Connected Telescope connected (TheSky6)
18:58:40 Session Equilibration Session ready to Open Dome
18:59:30   Wait On Weather Waiting for acceptable conditions
18:59:33 Session Suspending Session suspending
18:59:35   Dome Closed Dome already closed
18:59:37 Session Suspended Session suspended
19:00:02   Queue Started Observing Queue started (33 targets selected)
19:21:13 Session Resuming Session resuming
19:21:58   Dome Opened Dome opened (Opening time 45s, Zigbee --)
19:22:00 Session Running Session running
19:22:02   Queue Resumed Observing Queue resumed
19:22:04     Target Started (NrZen) Target started (Focus Field 7, HIP 34070)
19:22:15       Dome Unparked Dome unparked
19:25:12       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:27:23       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
19:29:20       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
19:29:23       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
19:31:29       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
19:33:11       Focusing Completed Foc2 AutoFocus Completed (Profile No 2)
19:33:29     Target Completed (NrZen) Target completed (Focus Field 7, HIP 34070)
19:33:31     Target Missed (1/33) Target's time slot was missed (1/33, UGC 12639 w/SN2023jri)
19:34:56     Target Started (2/33) Target started (2/33, UGC 831 w/AT2023wdd)
19:39:01       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
19:48:11       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
19:56:20     Target Completed Target completed (2/33, UGC 831 w/AT2023wdd)
19:56:22   Queue Paused Queue paused for Session Suspension
19:56:24 Session Suspending Session suspending
19:57:12   Dome Closed Dome closed (Closing time 50s, Zigbee 0s)
19:57:15 Session Suspended Session suspended
20:14:02     Target Cancelled Target cancelled (3/33, 13P/Olbers) due to cloud
20:34:22     Target Cancelled Target cancelled (4/33, UGC 11858 w/SN2023rjj) due to cloud
20:53:54     Target Cancelled Target cancelled (5/33, 12P/Pons-Brooks) due to cloud
21:04:06 Session Resuming Session resuming
21:05:46 Session Suspended Session suspended
21:10:21   Dome Closing Dome closure hasn't finished after 140s (ShutterState=Closing)
21:14:24     Target Cancelled Target cancelled (6/33, UGC 3174 w/SN2023mut) due to hardware
21:28:24     Target Cancelled Target cancelled (7/33, AT2023cub (CV)) due to hardware
21:49:56     Target Cancelled Target cancelled (8/33, AT2023tjz) due to hardware
22:06:09 User Intervention User Intervention to Restart Pulsar Dome
22:06:09   Pulsar Dome Restarting Dome Controller
22:06:38   Pulsar Dome Dome Controller has been restarted
22:06:42   Pulsar Dome Dome Service has been restarted
22:07:10     Target Cancelled Target cancelled (9/33, 103P/Hartley) due to hardware
22:10:40 User Intervention User Intervention to Restart AstroMain due Dome Object/Connection issue
22:10:49   Services Stopped Observatory Services stopped
22:10:59   Queue Aborted Job Queue aborted
22:11:08 Session Closed Session closed by User
22:11:12 Program Closed Program closed by User
22:11:52 Program Opened Program Opened (AstroMain 3.66)
22:11:54 Session Resumed Session Resumed (Live, 2024-02-29 S01195, ImageSaveNum: 1195011)
22:11:57   Obs.Overseer Started Obs.Overseer started
22:12:23   Obs.Manager Started Obs.Manager started
22:12:32   Services Started Observatory Services started
22:12:46 Session Monitoring AutoStart monitoring for Live Session opportunity between 22:12 & 04:57
22:12:48 Observatory (AutoStart) Observatory placed in Auto-Start Mode
22:13:39 Session AutoStarting Session autostarting (22:13)
22:14:26   Services Started Observatory Services started
22:14:33 Observatory (Auto) Observatory placed in Fully-Automated Mode
22:14:36 Session Pending Session pending (2024-02-29)
22:14:38 Session Initiating Session initiating (2024-02-29)
22:14:43   Plan Loaded Observing Plan loaded to queue (Plan ID: 969)
22:14:55   Camera1 Connected SBIG Camera connected (Set point -20°C)
22:15:30   Telescope Connected Telescope connected (TheSky6)
22:15:55 Session Equilibration Session ready to Open Dome
22:16:40   Dome Opened Dome opened (Opening time 45s, Zigbee --)
22:16:42 Session Running Session running
22:16:44   Queue Started Observing Queue started (23 targets selected)
22:19:52     Target Started (11/33) Target started (11/33, UGC 9002 w/SN2023pls)
22:25:16       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
22:41:51     Target Completed Target completed (11/33, UGC 9002 w/SN2023pls)
22:41:55     Target Started (12/33) Target started (12/33, AT2024crd)
22:47:39       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
23:04:53     Target Completed Target completed (12/33, AT2024crd)
23:04:57     Target Started (13/33) Target started (13/33, NGC 3115)
23:11:15       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
23:27:29     Target Completed Target completed (13/33, NGC 3115)
23:27:34     Target Started (14/33) Target started (14/33, UGC 5886 w/SN2024hw)
23:32:16       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
23:48:40     Target Completed Target completed (14/33, UGC 5886 w/SN2024hw)
23:49:12     Target Started (15/33) Target started (15/33, NGC 3467 w/AT2024dda)
23:53:02       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
00:09:55     Target Completed Target completed (15/33, NGC 3467 w/AT2024dda)
00:11:12     Target Started (16/33) Target started (16/33, UGC 6049 w/SN2023dff)
00:15:31       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
00:31:44     Target Completed Target completed (16/33, UGC 6049 w/SN2023dff)
00:33:12     Target Started (17/33) Target started (17/33, IC 564 w/SN2023abqf)
00:38:10       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
00:48:04       SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
00:48:29     Target Aborted (17/33) Target aborted (17/33, IC 564 w/SN2023abqf) due to User Intervention
00:48:31   Queue Paused Queue paused for Session Suspension
00:48:34 Session Suspending Session suspending
00:49:24   Dome Closed Dome closed (Closing time 50s, Zigbee 0s)
00:49:26 Session Suspended Session suspended
00:52:08     Target Cancelled Target cancelled (17/33, IC 564 w/SN2023abqf) due to too few stars
00:55:55 User Intervention User request to close and finish session
00:55:58   Queue Aborted Job Queue aborted
00:56:00 Session Closing Session closing
00:56:05   Dome Closed Dome already closed (closed at 00:49, closing time 50s, zigbee 0s)
00:57:10   Dome Parked Dome parked (parking time 57s), Az: 90.0 deg
00:59:20   Telescope Parked Telescope may have failed to park correctly (attempt 123s)
00:59:30   Telescope State Handbox reads 'Select Item: Object'
00:59:32   Telescope Park Telescope might not be parked. Check scope at start of next session !
00:59:51   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
01:01:15   Dome Charging Dome is parked and re-charging Ok (15mA)
01:01:18   Services Stopped Night Services stopped
01:01:20 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files)
01:01:37     Dome (Find Park) Find Park started (Search Az 87.0 to 93.0°, Step 0.2°, Narrow)
01:03:12     Dome Find Park found (Best Park Az 90.2°)
01:03:19     Dome Dome has been synced at park position (Az 90.0°, Adjustment: -0.2°)
01:03:21   Dome Parked Dome parked
01:05:16   Dome Charging Dome is parked and re-charging Ok (14mA)
01:05:18 Session Finished Session Finished

 
Session Alerts & Alarms
Time     Type       Name Detail
21:07:56 Red Alert Dome Alert Dome hasn't reached 'Closed' state after 120s during SoftSuspend
21:10:01 Red Alert Dome Alert Dome hasn't reached 'Closed' state after 120s during SoftSuspend
21:10:21 Red Alarm Dome Closing Dome closure hasn't finished after 140s (ShutterState=Closing)
21:24:33 Red Alert Dome Alert Shutter Data isn't being refreshed
21:24:44 Yellow Alert Dome Shutter Pulsar Shutter appears to be asleep (last fresh data was at 21:04:31)
22:06:42 Green Clear Dome Shutter Pulsar Shutter has woken up (fresh data at 22:06:42, outage 62.2 min)
22:06:45 Green Clear Dome Alert Shutter Data has resumed updating
00:59:20 Red Alert Scope Failed To Park Correctly Telescope may have failed to park correctly
00:59:30 Red Alert Scope Failed To Park Correctly Telescope may have failed to park correctly
01:05:16 Green Clear Dome Charging Dome is parked and re-charging Ok (14mA)
 

Back to Top


Operational Issues (2024-02-29 S1195)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Small Defects

Continuous Improvement

[ Prev | Next ]


Fig 1.   NIght Summary Plot from 2024-03-01 showing effect of power cut on AllSky recordings.

Image



Fig 2  Session Events and Session Alerts recorded by AstroMain for the Power Cut at 2024-03-02 12:46
Session: 2024-03-02 (General)
Date Time     Event Detail
2024-03-02 12:46:38   Power Supply Power outage at 12:46:38
2024-03-02 12:46:50   Power Supply Power returned at 12:46:50 (outage time 11s)
 
Session: 2024-03-02 (General)
Date Time     Type       Name Detail
2024-03-02 12:46:38 Red Alert Power Supply Power outage at 12:46:38
2024-03-02 12:46:44 Yellow Alert UPB.Switch disconnected UPB.Switch has unexpectedly disconnected. Restart will be attempted
2024-03-02 12:46:50 Green Clear Power Supply Power returned at 12:46:50 (outage time 11s)
2024-03-02 12:47:05 Yellow Alert UPB.Switch Restarting UPB Powerbox Software
2024-03-02 12:47:13 Green Clear UPB.Switch UPB.Switch service has been restored
 

Back to Top


Investigation - Telescope Failed to Park at end of session and was turned off in an 'unparked' state

Issue
At end of the S1195 Session (night of 2024-02-29 / 2024-03-01) the Observatory's LX200 Telescope failed  to park.

Also the telescope was turned off in an Unparked State, requiring a Telescope Resync & Short Mapping Run at/before start of next live session.

Description
During Closing operations at end of S1195 session,  the Observatory's LX200 telescope (12" LX200 GPS/R) was moved to 'Near Park' position at Az 180°/Alt 5° via TheSky6.Telescope, and then parked using objScope (Ascom).  A Red Alert were raised with the message "Telescope may have failed to park correctly", a message "Telescope may have failed to park correctly (attempt 123s)",  a warning was given that the handbox shows "Select Item:  Object", rather than the expected "Scope parked  Turn scope off." , with the further warning "Telescope might not be parked (warning for next session)" . Scope was then turned off.  

Analysis
Examining ObsCam pictures and logs indicates that the telescope didn't move to the Near Park Positon and didn't move to its Park Position, but was still at position associated with the  last active target (Target 17/33, IC 564, 00:33 to 00:48) and after continuing to track (until 00:55), the telescope was turned off 'Unparked' at a most likely postion at Az 209.5° / Alt 33.7° (Dec 4.3°) based on final ObsCam pictures at 00:57:17 and 01:00:11 (confirmed by visual inspection the following morning), having been stationary at this position since 00:55 or so.

Investigating further showed that there was an exception at 00:57:18, associated with the Slew to Near Park position, with the message "CheckDotNetExceptions ASCOM.DeviceHub.Telescope SlewToAltAzAsync System.Exception: Unable to start the direct slew!!!"..

No exceptions were logged from the call to objScope.Park.   There may have been an exceptions but it wasn't captured & logged or just wasn't raised. 

There is more than a possibility that the issue was somehow related to the UTCDate Errors that appear in MeadeGeneric driver log directly after midnight & the passage from a leap day (2024-02-29) to a standard day (2024-03-01). Indeed the log shows the SlewToAzAltSync (Az 180°, Alt 5°) request at 00:57:17 and the SlewToAltAz Error with the message  "Error: Year, Month, and Day parameters describe an un-representable DateTime".   Similarly the log shows Park request at 00:57:20.404  and a Park Error at 00:57:20.909 with the same error message "Error: Year, Month, and Day parameters describe an un-representable DateTime". 

  00:57:17.456 SlewToAltAzAsync   Az=180 Alt=5 polar=True
  00:57:17.456 UTCDate            Get started
  00:57:17.456 AtPark             Get - False
  00:57:17.549 UTCDate Get        Error: Year, Month, and Day parameters describe an un-representable DateTime.
  00:57:17.549 SlewToAltAzAsync   Error: Year, Month, and Day parameters describe an un-representable DateTime.
    and
  00:57:20.404 Park Parking telescope
  00:57:20.909 UTCDate Get        Error: Year, Month, and Day parameters describe an un-representable DateTime.
  00:57:20.909 Park               Error: Year, Month, and Day parameters describe an un-representable DateTime.

Conclusion
So the issue is explained as being related specifically to the Leap Day situation and isn't a general issue or matter of wider concern.  It is of note that all commands to SlewToCoordinatesAsync worked ok including those made after midnight.  I guess that SlewToCoordinatesAsync doesn't require going to RaDec from AzAlt and therefore doesn't need date whilst the other 2 operations (SlewToAzAltSync & Park) do. 

Colin Dawson has been contacted regading the case. 
(need to recontact him again regarding the apparent absence of an exception from the call to Telescope.Park)

In the meantime this issue is deemed understood and can be closed.

Recovery at / before next session
At the start of next session position of scope to be marked,  telescope to be turned on,  it will then move clockwise for a while (to find its ''worm reference") and wait for commands.   Scope should be moved back to its start-up position using directional controls, and then synced in TheSky6 on the location noted above.   Scope to then to move to suitable position and image taken and plaet solved, scope to be resynced on this position.  Same point to then be mapped 6 times.  Followed by Short Mapping Run with 12 points.  Telescope to be parked and restarted (should Park Position be redefined before this as a precaution ?).   A continuous improvement ticket has be raised to add a button facility in AstroMain to position TheSky6's Virtual Sky on specified Az/Alt Coordinates with a telescope view width.

Actions:

Back to Top


Investigation - Shutter Opening/Closing Issue (comms between Shutter & Controller frozen)

Issue

Description
Whilst resuming session at 21:04 an attempt to open the dome led to a warning message at 21:05  "Dome opening hasn't finished after 60s"  whilst attempting to resume session. During this time sky conditions had deteriorated and there was a call to Close Dome (with session still in Suspended Mode, having never made it to Running Again (which would direct to 'Suspending Mode first).

After taking an ObsCam picture, the following lines were reported

  >>SoftSuspend... Alert    Dome hasn't reached 'Closed' state 120s after being requested to close (setting bDomeDataIssue)
    Abort           Dome    Slew Info Calling Dome.AbortSlew to 'release' ShutterStatus
    Close Dome..    Info    Reattempting Dome Closure...
    Close Dome..    Info    Closing (Attempt 2)
  >>SoftSuspend...  Alert   Dome hasn't reached 'Closed' state 120s after being requested to close (setting bDomeDataIssue)
    Dome Safety     Error   Dome closure hasn't finished after 140s
    Dome Safety     Info    There may be a mechannical or electrical issue.
    Check Shutter   Ok      Hub reports dome is Closing (3), PulsarLog reports Closing (3)

Associated with these closing operations  Alerts "Dome hasn't reached 'Closed' state after 120s during SoftSuspend" were raised at 21:07:56 & 21:10:01, followed by an Alert/Alarm at 21:10:21 with the message "Dome closure hasn't finished after 140s (ShutterState=Closing)" and later a warning at 21:24 "Shutter Data isn't being refreshed" (last data at 21:04) . 

'Dome Closing Diagnosis & Recovery' was automatically begun at 21:10 but results aren't reported.  A flag was raised that then prevented execution of the following 3 targets with them failing due to 'Other - Hardware'.

Problem noticed during remote monitoring and the user intervened to Restart Pulsar Dome at 22:06 (including a Power Reset). AstroMain restarted at 22:10 after failing to reestablish proper link/connection to the dome after its Restart. operation (likely an issue with client object in AstroMain ?). Pulsar Log indicates following information about changes in Shutter State, in which there is a definate anomaly from 21:05 until dome was started at 22:06:48 .

  21:04:04   1, shutter closed
  21:04:05   2  shutter opening
  21:05:45   2  shutter still opening  (opening time> 90s)
  21:05:46   3  shutter is closing
  22:06:08   3  shutter still closing after 60 mins   (driver stuck ?
  22:06:38  (0) records with zero values across all data fields (* Dome is restarting by user commands)
  22:06:48   1  shutter closed   (closing time  61 mins)
  22:15:55   2  shutter opening
  22:16:38   0  shutter open     (opening time 43s)
  00:48:34   3  shutter closing
  00:49:21   1  shutter closed  (closing time was 47s)

Analysis

Because the zigbee open/closed sensor at the bottom end of shutter had stopped working it isn't possible to fully describe the dome timings but the indication seems to that the Shutter had stopped sending fresh data to the Dome Contoller Unit (or alternatively the Dome Control Unit stopped hearing communications from the Shutter Unit.

Logs show battery properties in Volatile records becoming 'stuck'  on values  "14405 -396 13693 5013 19782"  from 21:04:31 until Pulsar Dome was restarted by user at 22:06

Conclusion

Actions:

Back to Top


2024-03-02


MeadeGeneric 1.3.9

A new version of the MeadeGeneric driver (1.3.9) was installed on the observatory computer today (2024-03-02). 

There were no issues with installation and previous settings carried through to the new version ok.

This new version (1.3.9.482) contains several changes since the previously installed development version (MeadeGeneric 1.3.5.414, installed 2022-05-30). New features are listed below. Only one new feature ("tracelogger to the calls that send the actual serial commands") is directly relevant to me for use during incident/anomaly investigation, but it is appropriate to align my setup with the current released version of MeadeGeneric every so often and ensure that none of the recent changes have an adverse/retrograde effect for my telescope/observatory system.

Actions
Image Test out the new Meade Generic driver during daytime tests - Done, 2024-03-02
Image Test out the new Meade Generic driver during a live session - Done, 2024-03-04 (Session S1196)

Results
Daytime tests were successfully carried out after installation of the new driver. These checked that 
- LX200GPS/R telescope could be connected to
- Day/Time prompts would be passed ok
   (this wasn't achieved on first attempt as the connection was made before the Telescope had completed its initialisation, but worked on a subsequent start-up)
- Scope would slew to appropriate coordinates when commanded via TheSky6
- Scope would park ok.

MeadeGeneric 1.3.9 was successfully used for the S1196 live session (2024-03-04). No problems or issues arose.

Observations on new driver:
None at this time.

New Driver Features since previously installed version (1.3.5 (.414) )

1.3.9 Release Notes, 10-Jun-2023
- Added extra values to GW translation. Also added extra log message that includes the hour angle when calculating the destination side of pier.

1.3.8 Release Notes, 02-Jan-2023
- Upgraded focuser maxstep to 30000 from 7000
- Old Autostar firmware is inconsistent in it's support for setting Target RA/DEC.
   A failsafe has been added that if one version fails, the driver will try the other mode before giving up.
- Added support for LX200GPS with 4G0m firmware loaded.

1.3.7 Release Note, 27-Sep-2022
-  Some audiostar firmwares return the product Audiostar. Adding support for this as a synonym of Autostar.

1.3.6 Release Notes 25-Jul-2022
- Upgraded requirements to need Ascom 6.6.0 and .net framework 4.8 or higher.  (see below)
- Added support for allowing pulse guiding for AltAz telescope using older guiding method.
- Added long enter and long goto commands.
- Added support for RCX400 using newer pulse guiding commands.
- Refactored the connect set to be a little less cluttered.
- Added support for setting the guide rate on the RCX400
- Added code to allow pulseguide commands to be sent to telescopes with StarPatch installed.
- Added the tracelogger to the calls that send the actual serial commands.

1.3.5 Release Notes, 17-May-2022
- Modified the SideOfPier code so that it reports properly. Confirmed with an LX75.
>> See MeadeGeneric Releases : https://bitbucket.org/cjdskunkworks/meadeautostar497/wiki/Release%20History

Ascom & .Net Framework Requirements

It should be noted that Version 1.3.6+ requires Ascom 6.6.0 and .Net framework 4.8 or higher.

The Observatory Computer is running ASCOM 6.6.1, and has .Net Framework 4.8 as shown by this screenshot from RegEdit, so it passes this requirement ok.

Image

Back to Top


Recovery from Telescope Shut-Down in Unparked State

At end of the S1195 Session the Observatory Telescope (12" LX200 GPS/R) was shudown without it being first parked.  This was due to a "Leap Day" related bug in LX200 (or less likely the MeadeGeneric driver) that cause the final SlewToAzAlt Async command and Park command to fail.  Scope power was turned off 20s later, leaving the Telescope shutdown in an Unparked State.

This means that the scope won't have correct coordinates when it starts up at the next session (it will assume coords of RA at the Meridian Crossing & Dec 0.0°).  Therefore the telescope will need Syncing on a known star (or other known position) followed by a short mapping run to tie the scope into the TPoint model in TheSky6.

Recovery
As it was known that the telescope was last pointing at Az 209.5° / Alt 33.7° (Dec 4.3°) when it was switched off, this fact was used to perform an initial resync & map as an interim fix, ahead of a final resyc & map at the start of the next session, which will hopefull speed up the recovery process.   This also provided the opportunity to perform a daytime test of MeadeGeneric Driver 1.3.9 which had just been installed on the Observatory Computer.

The steps taken in achieving this are as follows.

This shows that AstroMain's Connect Telescope needs to be modified to allow a longer initialisation period when starting up the telescope from an unparked state.
say 180s instead of 60s.  

Actions:
Image ParkScope routines no longer turn off the telescope if it appears to be unparked (AstroMain 3.66.1).
Image A checkbox option "Wait 180s after turning on scope" (with default value of unchecked) has been added to AstroMain's More/Obs. tab (AstroMain 3.66.1)

Back to Top


2024-03-03


Review - Uninterruptible Power Supply (UPS) for the Observatory Telescope

Review
As part of a review looking at increasing the Observatory and Live Session resistance to small power outages the potential of using one or two UPS Systems with offline battery backup was identifed as options for further consideration (see Review - Increasing observatory & session resistance to small power outage, 2024-02-28)

However given that the review has identified that the Observatory's LX200 Telescope is the one observatory element that is most affected by an unscheduled power outage there is requirement to prioritise the selection, purchase and installation of a UPS System for the Observatory Computing Bay. 

The telescope is most affected because a uncontrolled shutdown leaves the telescope in an unparked state meaning that the scope loses all knowledge about its physical pointing direction upon power restoration & telescope restart,  requiring  a resync of telescope position & a re-tie into the T-Point Base Model which is a task that can't be automatically performed without user involvement & oversight.  In comparsion other observatory elements recover by themselves upon power restoration or there recovery can be performed by an automated manager.

This leaves consideration of a UPS System for the AllSky/Weather Computing bay to a later date (and one that can learn from experienes operating the UPS System that is in the Observatory Computer bay for supporting safe Telescope parking & shutdown.

Due to an incident during a power outage test during live session S1197 on 2024-03-13, when Pulsar Dome settings became reset (see Investigation) a review to look at a UPS system to provide Battery Backup for the Pulsar Dome & Cloud Sensor has been kicked off (see Review).

A previous review of UPS needs was made in Dec 2018.  No detailed notes were completed at the time other it being recorded that at some stage 1 or 2 UPS systems will be ordered for the Observatory, but that the decision was put on hold (see UPS Review,  2018-12-22).

Analysis
A UPS Backup to the Pegasus UPB Powerbox that distributes 12V power to the LX200 Telescope is being considered as the telescope is the one item in the Observatory where even a very short power cut can create a very significant problem causing a significant user effort to resolve & delay the next live session. This is because the telescope will turn off / power-down without being parked, requiring the telescope to be re-synced on a known star (or known position) and a short mapping run completed. Whilst program routines can aide the process of resync & short mapping it has to be done under strict user supervision & control.  

For all other observatory elements their recovery following a power outage can either be automated or achieved relatively easily over a remote connection, once power is restored.

The Pegasus UPB Powerbox also distribues power to Focuser1 (TCF-S) , Focuser2 and the Dew Heaters.  

Are there any other equipment/devices in the Observatory Computing bay that would also benefit from having UPS battery backup ?   

Ensuring that both the Powerbox and the USB Hubs remain powered during a short power supply outage may prevent or reduce the number of instances where the power box and UPB.Switch driver stops working and produce the error pop-ups with messages like "Lost Connection with UPB".

For all other observatory elements their recovery following a power outage can either be automated or achieved relatively easily over a remote connection.

Requirements
The requirements of UPS System for the Observatory Computing bay are listed below

Options & Selection

Based on a bit of research and comparison, the main options and the reasons for rejection or selection of each option are given below:

Chosen Option

The chosen option is the Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply 

See Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply System for details about specs and other things to do with the 3S 850B Unit.

Risks
The 3C 850B Model has an advertised workingTemperature range of 0 to 40 deg C and a humidity range 0-85% (non condensing).

Given that the Unit will be in the Observatory where temperature can be sub-zero on winter nights (say down to -6 deg C) and humidites can on occasions be potentially in range up to 100% .

One mitigation is the fact the Unit will be located in the Observatory Computing Bay where equipment is always running and helps to keep temperature there a fraction higher than ambient and humidities a fraction lower than ambient.  Despite that, what are the risks ?

Decision
Decision made (2024-03-03) is to purchase one Eaton 3S 850B  UPS System for the time being, to provide short-term back-up power for the Observatory's  LX200 Telescope and prevent the telescope having an unplanned shutdown leaving the scope in an unparked state, which then requires a significant effort to restore from. This is because the re-establishment of the Telescope's Pointing and the tie-in to the TPoint base model can't be done fully automatically but requires the user in attendance or to provide remote direction & oversight. 

Attached Devices
The telescope runs on 12V which is supplied via a Pegasus UPB Powerbox  which is mounted on one arm of the telescope's fork mount.  Beside the LX200 Telescope the Powerbox also distributes power to an Optec TCF-S Focuser, a Secondary Scope Focuser,  two Dew Heaters,  as well as operating as a USB Hub, a controller for the Secondary focuser, and for measuring Environment/Temperature.

One of the 4 UPS sockets with battery back-up will obviously to be used for the Pegasus UPB Powerbox, but leaves a decision over what the other 3 sockets will be used for.

An audit of the power plugs & equipment/devices was made and the results will help with the decision (The audit will also be used as input to a decision on whether to purchase and install a second, separate UPS System for devices in the AllSky/Weather bay and what its requirements would be ).

At present time the other 3 UPS Sockets will be used for
- the SBIG Camera (with Smart Plug)
- USB Hubs (2 of them powered from one UPS Plug, using adapter)
- Weather Station's Base Station.

The Observatory Computer (a laptop) has its own (longer lasting) battery and therefore doesn't need battery backup support from the UPS system, indeed having the computer not attached to a UPS Socket, means that the Observatory Control Program (AstroMain) can poll the state of its own power supply and use it as a proxy for the state of the Observatory's Power Supply, not withstanding the possiility that AstroMain may be able to determine the source of the power being output by UPS (mains power or UPS battery power).  Once installed in the Observatory the Observatory Computer will have a USB connection to the UPS System and information about power supply & UPS Battery can be viewed in the Eaton UPS Companion software, however it as yet unclear if that information can be passed on to the AstroMain program

(Note: the Eaton's Technical Department replied to a query I had submitted on this point, and they indicted that a 3rd party software program couldn't access the information coming from the UPS system, however a quick dive into the UPS Companion's program folder reveals that alarm data from the UPS system is written to a JSON formatted euc.log file that can be read and used by the AstroMain program)

Configuration
The current configuration of the Observatory Computing Bay and the proposed new configuration are shown in the two diagrams below:

Image
 
 
Image
 
During installtion of UPS in the Observatory it was discovered that it wasn't possible to fit the 4 planned plugs directly into the 4 sockets with battery backup on the UPS, due to the 3 of the plugs (a square smart plug for SBIG Camera & A/C Adpater Plugs for 2 USB Hubs)  having a wider footprint than a conventional plug. To get around this problem a small power strip had to be employed. The Final Configuration is shown below:

Image
 
Following the failure of the 7-Port 'Other' USB Hub during a thunderstorm on 2024-08-12, the transfer of USB cables from UPB Powerbox & External Hard Drive to empty slots on the Lindy USB Hub, and the successful observing session on 2023-08-14 (S1227), the 'Other' USB Hub and its A/C Adapter were removed from the Observatory on 2024-08-14. A revised picture showing the Modified Configuration as at 2024-08-14 is shown below.

Image

Proposed Equipment & UPS Battery Management during a Power Supply Outage

The amount of time that the 3C 850B unit will be able to deliver is dependant on a number of factors, the main ones at the level of the battery going into the utility outage and the load demanded by the attached equipment & devices.

Battery Level would normally be 95%+ when outage occurs, but it won't if there was an earlier outage from which the battery hasn't fully recovered by re-charging in the meantime, it could be alot less.  Load will depend on the sum of the load from each attached device or equipment, which are themselves dependant on the activity levels & demands of the respective equipment.

The Telescope will have a higher load when it is slewing than when it is stationary or just slowly tracking.  The TCF-S focuser and Secndary Scope will have higher loads when adjusting focus than when they are idle.  The SBIG Camera will have higher load if it is using CCD Cooling  it will also have a higher load if it is taking & downloading an image than if it is idle.  The dew heaters will have a higher load when they are operating at 80-100% power level than if they are operating at 0-20% level.  The size of the load can be lowered by Observatory Control programming whilst still keeping the equipment/devices powered up.

According to Eaton' Run Time charts the 3C 850 models (when fully charged) can provide
 - 1 min of backup at 500 W     (2.0 A at 240V)
 - 10 min backup at 200 W       (0.8 A at 240V)
 - 20 min backup at 127 W       (0.5 A at 240V)
-  25 min backback at 100 W   (0.4 A at 240V)

(a 9 amp-hour battery at 12 V = 108 W Hours or 6480 W Minutes, so a 200 W Load would theoretically last 32 min till total discharge, but in practice the time in mintues would inevitably be lower than this)

AstroMain's observatory control will need to recognise an outage, consider the effects of any other recent backups, tie into any information it can get about the UPS System itself and progressively reduce power consumption of the equipment & devices that are operating on backup power during a utility power outage, eventually park the telescope.   A balance needs to be struck between having the equipment a) carry on in a reduced activity mode so that they are immediately ready for action when the Utility Power Supply is restored and b) minimising the battery usage outage (so it available for second or third outage) by putting closing down equipment as soon as possible (eg. parking the telescope & switching it off immediately after it is put onto UPS Battery Power, immediately stop CCD Camera cooling etc)

The preliminary assessment of the management requirements are as follows:

Actions 

Back to Top


Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply Unit

A Eaton 3S 850B UPS unit has been chosen to provide Off Line Uninterruptible Power Supply (UPS) for the Observatory Telescope, but will also provide power backups for the SBIG CCD Camera and USB Hubs in the Observatory Computing Bay.

It should be noted that only 4 of the 8 output sockets on the Eaton 3C 850B Unit have battery backup protection (the other 4 have surge protection only).

Key Specs
It's key specs are tabulated below

Manufactorer   Eaton
Voltage:   230V     (220/230/240V)
Wattage:   510 W
Battery Type:   Sealed, lead-acid , 12 V / 9 Ah   (User Replaceable)
Battery Management : Automatic battery test, deep-discharge protection, cold-start capable
VA Rating   850 VA
Outputs with battery back-up   4 UK (Battey backup & surge)
Other Outputs   4 UK (Surge only)
2 USB Charge
Surge Protection:   Yes
Communication :   USB port (HID compliant)
Construction Type:   Free standing mode
Auto Shutdown Mode:   Yes
Size:   335 x 170 x 86 mm
Weight 4.3 kg
Input Cord Length   1.5m
Relative Humidity   0 - 85%
Software Compatibility   Eaton UPS Companion (enables safe system shutdown,
energy usage metering and configuration of UPS settings)
Warrenty:   2 years
     
Full Specifications :   https://www.eaton.com/gb/en-gb/skuPage.3S850B.pdf#Specifications
Product Page:   https://www.amazon.co.uk/Eaton-850B-UPS-Uninterruptible-Type-Black-white/dp/B0B3RQN8QT/
Run Time Chart    https://eg.eaton.com/ups-battery-runtime/en-gb/3S850D  
     
New Battery  Cost   CSB batteries - these batteries are also used by Eaton
£28.99 + £4.95 delivery from Hardware Express
https://www.hardwarexpress.co.uk/products/eaton-3s-850b-battery-replacement

Actions:

Update 2024-03-05 
The UPS Unit arrived today (2024-03-05) and was unpacked, and set up indoors next to the Development Computer and tested for a while.
UPS Companion was downloaded and installed (the program is name mc2.exe).

Image

 Initial Battery Capacity level was 90% when first connected.

Image

Tests conducted with and without Utility Power Supply connected to simulate the effect of a power supply outage.

In terms of be able to tie into the data from the UPS Unit  I've found that the Companion Software writes alarms to the following file
  C:\Program Files (x86)\Eaton\UPSCompanion\configs\euc.log
with information recorded like the following

[
{"date": "March 5, 2024-6:47:54 pm", "level": 1, "message": "The system is powered by the utility"},
{"date": "March 5, 2024-6:47:45 pm", "level": 2, "message": "Shutdown in 44 min 50 s"},
{"date": "March 5, 2024-6:47:43 pm", "level": 2, "message": "The system is powered by the UPS battery"},
{"date": "March 5, 2024-6:33:40 pm", "level": 1, "message": "The system is powered by the utility"},
{"date": "March 5, 2024-6:33:19 pm", "level": 2, "message": "Shutdown in 42 min 10 s"},
{"date": "March 5, 2024-6:32:18 pm", "level": 2, "message": "Shutdown in 54 min 00 s"},
{"date": "March 5, 2024-6:31:17 pm", "level": 2, "message": "Shutdown in 55 min 00 s"},
{"date": "March 5, 2024-6:30:16 pm", "level": 2, "message": "Shutdown in 56 min 00 s"},
{"date": "March 5, 2024-6:30:16 pm", "level": 2, "message": "The system is powered by the UPS battery"}
]

where the most recent alarm message is listed first.  This provides some idea of the remaining battery capacity based on the current load.  

One iinitial concern was that the UPS Companion settings imply that at some stage the UPS Companion will shutdown the Computer. Now this is not what I want to happen, the Observatory Computer (a laptop) has its own battery (which can last longer that the UPS Unit), and intentionally the computer won't be connected to the UPS Unit allowing it to independantly monitor the state of the Utility Power Supply.  Computer Shutdown Criteria on the 'Settings' tab were changed to 'Ignore' where ever possible. The  'Shutter computer when battery remaining is below " setting can't be set to 'Ignore' however."   It is hoped that the 'Do Nothing' setting for 'Shutdown Type / When first shutdown criteria is reached' is effective in preventing a Computer Shutdown.
 (the other options are 'Hibernate' and 'Shutdown', so ' Do Nothing' seems the appropriate option to choose).

Image

Update 2024-03-07 
Following completion of indoor testing the UPS was installed in the Observatory on 2024-03--07.  

It wasn't possible to fit the 4 planned plugs directly into the 4 sockets with battery backup on the UPS, due to the 3 of the plugs having a wider footprint than a conventional plug (one a square smart plug for SBIG Camera and two A/C Adpater Plugs for USB Hubs). To get around this a small power strip had to be employed. (see diagram below).

Image

Following the installation of UPS (including USB cable to computer), UPS Companion Software was installed, settings checked and relevant tests successfully carried out, switching off / switching on mains power to the UPS Unit.

Web Companion software indicated that the base load from the attached equipment / devices (Telescope, Camera Cooling & Dew Heater all off ) was 34 W (with potential run-time of 52 mins) .   (Later with the UPS System Battery charged to 100%, the reported Run Time with 34 W base load was noted to be 56 mins.

With the Camera turned off altogether the base load falls to an estimated value of 13 W (the UPS system can't seem to measure loads as low as this / the UPS Companion displays load as being 0.0 W.)

A series of tests were carried out to understand the load contribution from each discretionary activity. 
The resolution on the UPS's load measurement seems to 7 W (load increases in 7 W steps)

Activity    Additional Load 
Minimal Base Load (UPB Powerbox & USB Hubs only)    13 W 
Normal Base Load (with TCF-S Focuser & CCD Camera, w/o Cooling)    34 W 
     
Camera Cooling On at 100% power (during CCD cool down)    14 W 
Camera Cooling Off at 62% (stabilised CCD temperature)    7 W 
Taking 60s image (possible 5 W during image download ?)   0 W  
+Turning Off SBIG Camera   - 21 W
     
+Dew Heaters at 25% load    14 W 
+Dew Heaters at 50% load    20 W 
+Dew Heaters at 75% load    27 W 
+Dew Heaters at 100% load    41 W 
     
Turning Off TCF-S Focuser   - 7W
     
Telescope On & Tracking Off   7 W 
Tracking On   Neglible
+Slewing   7 - 14 W 
     
Normal Total Load (Cooling 70%, Telescope,  Dew Heaters 75%)   75 W 
Maximum Total Load (Cooling 100%, Telescope Slewing,  Dew Heaters 100%)   103 W 
     

Expected run times based on range of loads is shown in the attached table

Case   Load (W)   RunTime (mins) 
Minimal Base Load (UPB Powerbox & USB Hubs only)    13 W    60 min +
Normal Base Load (with TCF-S Focuser & CCD Camera, w/o Cooling)    34 W    52 min
Normal Total Load (CCD Cooling 70%, Telescope,  Dew Heaters 75%)   75 W    34 min
Maximum Total Load (CCD Cooling 100%, Telescope Slewing,  Dew Heaters 100%)   103 W    23 min
         

Actions 

Image Add 'UPS Companion' (mc2.exe) to Observatory's AstroLaunch Menu. This was tricky to do, as there can be more one mc2.exe process running, one that is owned by the System that runs as a service, and one than is owned by the user.  The ProcessRunning() function has to have a custom modification to look for a mc2.exe process that is not owned by the System, before returning True. To launch the Web Companion User Application the argument "-open" has to be used.  The RunShell() routine was extended to pass an argument to the routine.  This appears to be working ok. Note the service needs to be running in order for the user application to work.
- Done 2024-03-05 (AstroLaunch 1.11.2)

Image Add a  'euc.log' reader to AstroMain, display the UPS Status on ObsViewer/ObsPic picture and raise and report key events/alerts. Being implemented in AstroMain 3.662.2, a euc.log reader was added and ObsViewer was extended to display the UPS Status (2023-03-08).  To enable this the height of the ObsViewer/ObsPic picture was increased by 20 px..
- Done 2024-03-08 (AstroMain 3.66.2)

Back to Top