Fig 1.   NIght Summary Plot from 2024-03-01 showing effect of power cut on AllSky recordings.

Fig 2  Session Events and Session Alerts recorded by AstroMain for the Power Cut at 2024-03-02 12:46

Session: 2024-03-02 (General) Date Time     Event Detail 2024-03-02 12:46:38   Power Supply Power outage at 12:46:38 2024-03-02 12:46:50   Power Supply Power returned at 12:46:50 (outage time 11s)

Session: 2024-03-02 (General) Date Time     Type       Name Detail 2024-03-02 12:46:38 Red Alert Power Supply Power outage at 12:46:38 2024-03-02 12:46:44 Yellow Alert UPB.Switch disconnected UPB.Switch has unexpectedly disconnected. Restart will be attempted 2024-03-02 12:46:50 Green Clear Power Supply Power returned at 12:46:50 (outage time 11s) 2024-03-02 12:47:05 Yellow Alert UPB.Switch Restarting UPB Powerbox Software 2024-03-02 12:47:13 Green Clear UPB.Switch UPB.Switch service has been restored

Issue
At end of the S1195 Session (night of 2024-02-29 / 2024-03-01) the Observatory's LX200 Telescope failed  to park.

Also the telescope was turned off in an Unparked State, requiring a Telescope Resync & Short Mapping Run at/before start of next live session.

Description
During Closing operations at end of S1195 session,  the Observatory's LX200 telescope (12" LX200 GPS/R) was moved to 'Near Park' position at Az 180°/Alt 5° via TheSky6.Telescope, and then parked using objScope (Ascom).  A Red Alert were raised with the message "Telescope may have failed to park correctly", a message "Telescope may have failed to park correctly (attempt 123s)",  a warning was given that the handbox shows "Select Item:  Object", rather than the expected "Scope parked  Turn scope off." , with the further warning "Telescope might not be parked (warning for next session)" . Scope was then turned off.  

Analysis
Examining ObsCam pictures and logs indicates that the telescope didn't move to the Near Park Positon and didn't move to its Park Position, but was still at position associated with the  last active target (Target 17/33, IC 564, 00:33 to 00:48) and after continuing to track (until 00:55), the telescope was turned off 'Unparked' at a most likely postion at Az 209.5° / Alt 33.7° (Dec 4.3°) based on final ObsCam pictures at 00:57:17 and 01:00:11 (confirmed by visual inspection the following morning), having been stationary at this position since 00:55 or so.

Investigating further showed that there was an exception at 00:57:18, associated with the Slew to Near Park position, with the message "CheckDotNetExceptions ASCOM.DeviceHub.Telescope SlewToAltAzAsync System.Exception: Unable to start the direct slew!!!"..

No exceptions were logged from the call to objScope.Park.   There may have been an exceptions but it wasn't captured & logged or just wasn't raised. 

There is more than a possibility that the issue was somehow related to the UTCDate Errors that appear in MeadeGeneric driver log directly after midnight & the passage from a leap day (2024-02-29) to a standard day (2024-03-01). Indeed the log shows the SlewToAzAltSync (Az 180°, Alt 5°) request at 00:57:17 and the SlewToAltAz Error with the message  "Error: Year, Month, and Day parameters describe an un-representable DateTime".   Similarly the log shows Park request at 00:57:20.404  and a Park Error at 00:57:20.909 with the same error message "Error: Year, Month, and Day parameters describe an un-representable DateTime". 

  00:57:17.456 SlewToAltAzAsync   Az=180 Alt=5 polar=True
  00:57:17.456 UTCDate            Get started
  00:57:17.456 AtPark             Get - False
  00:57:17.549 UTCDate Get        Error: Year, Month, and Day parameters describe an un-representable DateTime.
  00:57:17.549 SlewToAltAzAsync   Error: Year, Month, and Day parameters describe an un-representable DateTime.
    and
  00:57:20.404 Park Parking telescope
  00:57:20.909 UTCDate Get        Error: Year, Month, and Day parameters describe an un-representable DateTime.
  00:57:20.909 Park               Error: Year, Month, and Day parameters describe an un-representable DateTime.

Conclusion
So the issue is explained as being related specifically to the Leap Day situation and isn't a general issue or matter of wider concern.  It is of note that all commands to SlewToCoordinatesAsync worked ok including those made after midnight.  I guess that SlewToCoordinatesAsync doesn't require going to RaDec from AzAlt and therefore doesn't need date whilst the other 2 operations (SlewToAzAltSync & Park) do. 

Colin Dawson has been contacted regading the case. 
(need to recontact him again regarding the apparent absence of an exception from the call to Telescope.Park)

 In the meantime this issue is deemed understood and can be closed.


Recovery at / before next session
At the start of next session position of scope to be marked,  telescope to be turned on,  it will then move clockwise for a while (to find its ''worm reference") and wait for commands.   Scope should be moved back to its start-up position using directional controls, and then synced in TheSky6 on the location noted above.   Scope to then to move to suitable position and image taken and plaet solved, scope to be resynced on this position.  Same point to then be mapped 6 times.  Followed by Short Mapping Run with 12 points.  Telescope to be parked and restarted (should Park Position be redefined before this as a precaution ?).   A continuous improvement ticket has be raised to add a button facility in AstroMain to position TheSky6's Virtual Sky on specified Az/Alt Coordinates with a telescope view width.

Actions:

• Park Scope routines.  Routine to no longer turn off the telescope if it appears to be unparked, but it instead turnoff Telescope Tracking and leave the Telescope switched on, pending User Attendance.
  Done   (AstroMain 3.66.1).
   
• Start Scope Routines.  A checkbox option "Wait 180s after turning on scope" (with default value of unchecked) to be added to AstroMain, and used during telescope start-up. If checked a 180s wait period is used after turning on the telescope power, before attempt to connect to the telescope is made. This is to allow for the LX200's longer initialisation process when starting up from an 'UnParked' state.  
  Done   (AstroMain 3.66.1).
   
• MeadeGeneric Driver. Author of MeadeGeneric driver to be contacted regarding leap-day bug found at transition 2024-02-29/2024-03-01, UTCDate error  "Year, Month, and Day parameters describe an un-representable DateTime"  and the associated errors using SlewToAzAltAsync & Park.
  Done 2024-03-01 (MeadeGeneric Driver)
   
• MeadeGeneric Update.  MeadeGeneric Telescope Driver on the Observatory Computer should be updated to the latest version, in order to benefit from the logging of serial communications between driver and telescope, which would have helped in this or similar future situation.
  Done  2024-03-02  (Meade Generic 1.3.9 ).

Issue

•  Dome Opening/Closing Issues
   - Dome opening hasn't finished after 60s"  whilst attempting to resume session. (21:05) 
   
• The following 3 targets all failed with the reason 'due to Hardware'

Description
Whilst resuming session at 21:04 an attempt to open the dome led to a warning message at 21:05  "Dome opening hasn't finished after 60s"  whilst attempting to resume session. During this time sky conditions had deteriorated and there was a call to Close Dome (with session still in Suspended Mode, having never made it to Running Again (which would direct to 'Suspending Mode first).

After taking an ObsCam picture, the following lines were reported
  >>SoftSuspend... Alert    Dome hasn't reached 'Closed' state 120s after being requested to close (setting bDomeDataIssue)
    Abort           Dome    Slew Info Calling Dome.AbortSlew to 'release' ShutterStatus
    Close Dome..    Info    Reattempting Dome Closure...
    Close Dome..    Info    Closing (Attempt 2)
  >>SoftSuspend...  Alert   Dome hasn't reached 'Closed' state 120s after being requested to close (setting bDomeDataIssue)
    Dome Safety     Error   Dome closure hasn't finished after 140s
    Dome Safety     Info    There may be a mechannical or electrical issue.
    Check Shutter   Ok      Hub reports dome is Closing (3), PulsarLog reports Closing (3)
 
Associated with these closing operations  Alerts "Dome hasn't reached 'Closed' state after 120s during SoftSuspend" were raised at 21:07:56 & 21:10:01, followed by an Alert/Alarm at 21:10:21 with the message "Dome closure hasn't finished after 140s (ShutterState=Closing)" and later a warning at 21:24 "Shutter Data isn't being refreshed" (last data at 21:04) . 

'Dome Closing Diagnosis & Recovery' was automatically begun at 21:10 but results aren't reported.  A flag was raised that then prevented execution of the following 3 targets with them failing due to 'Other - Hardware'.

Problem noticed during remote monitoring and the user intervened to Restart Pulsar Dome at 22:06 (including a Power Reset). AstroMain restarted at 22:10 after failing to reestablish proper link/connection to the dome after its Restart. operation (likely an issue with client object in AstroMain ?). Pulsar Log indicates following information about changes in Shutter State, in which there is a definate anomaly from 21:05 until dome was started at 22:06:48 .
  21:04:04   1, shutter closed
  21:04:05   2  shutter opening
  21:05:45   2  shutter still opening  (opening time> 90s)
  21:05:46   3  shutter is closing
  22:06:08   3  shutter still closing after 60 mins   (driver stuck ?
  22:06:38  (0) records with zero values across all data fields (* Dome is restarting by user commands)
  22:06:48   1  shutter closed   (closing time  61 mins)
  22:15:55   2  shutter opening
  22:16:38   0  shutter open     (opening time 43s)
  00:48:34   3  shutter closing
  00:49:21   1  shutter closed  (closing time was 47s)

Analysis

Because the zigbee open/closed sensor at the bottom end of shutter had stopped working it isn't possible to fully describe the dome timings but the indication seems to that the Shutter had stopped sending fresh data to the Dome Contoller Unit (or alternatively the Dome Control Unit stopped hearing communications from the Shutter Unit.

Logs show battery properties in Volatile records becoming 'stuck'  on values  "14405 -396 13693 5013 19782"  from 21:04:31 until Pulsar Dome was restarted by user at 22:06

Conclusion

•  Issue is understood to be a communication breakdown between Shutter Unit and the Dome Controller.
   
• The solution in this case was a restart of the Pulsar Dome (including a power reset).
  
  A separate continous improvement ticket should be raised to modify Management Oversight to detect such issues where Shutter data is evidentally become frozen and automatically restart the Pulsar Dome.

Actions:

•   Raise continous improvement ticket to modify ObsOverseer/ObsManager to automatically restart the Pulsar Dome when Shutter Data appears to become frozen.
  Pending ?   Need to check status.

A new version of the MeadeGeneric driver (1.3.9) was installed on the observatory computer today (2024-03-02). 

There were no issues with installation and previous settings carried through to the new version ok.

This new version (1.3.9.482) contains several changes since the previously installed development version (MeadeGeneric 1.3.5.414, installed 2022-05-30). New features are listed below. Only one new feature ("tracelogger to the calls that send the actual serial commands") is directly relevant to me for use during incident/anomaly investigation, but it is appropriate to align my setup with the current released version of MeadeGeneric every so often and ensure that none of the recent changes have an adverse/retrograde effect for my telescope/observatory system.

Actions
Test out the new Meade Generic driver during daytime tests - Done, 2024-03-02
Test out the new Meade Generic driver during a live session - Done, 2024-03-04 (Session S1196)

Results
Daytime tests were successfully carried out after installation of the new driver. These checked that 
- LX200GPS/R telescope could be connected to
- Day/Time prompts would be passed ok
   (this wasn't achieved on first attempt as the connection was made before the Telescope had completed its initialisation, but worked on a subsequent start-up)
- Scope would slew to appropriate coordinates when commanded via TheSky6
- Scope would park ok.

MeadeGeneric 1.3.9 was successfully used for the S1196 live session (2024-03-04). No problems or issues arose.

Observations on new driver:
None at this time.

New Driver Features since previously installed version (1.3.5 (.414) )

1.3.9 Release Notes, 10-Jun-2023
- Added extra values to GW translation. Also added extra log message that includes the hour angle when calculating the destination side of pier.

1.3.8 Release Notes, 02-Jan-2023
- Upgraded focuser maxstep to 30000 from 7000
- Old Autostar firmware is inconsistent in it's support for setting Target RA/DEC.
   A failsafe has been added that if one version fails, the driver will try the other mode before giving up.
- Added support for LX200GPS with 4G0m firmware loaded.

1.3.7 Release Note, 27-Sep-2022
-  Some audiostar firmwares return the product Audiostar. Adding support for this as a synonym of Autostar.

1.3.6 Release Notes 25-Jul-2022
- Upgraded requirements to need Ascom 6.6.0 and .net framework 4.8 or higher.  (see below)
- Added support for allowing pulse guiding for AltAz telescope using older guiding method.
- Added long enter and long goto commands.
- Added support for RCX400 using newer pulse guiding commands.
- Refactored the connect set to be a little less cluttered.
- Added support for setting the guide rate on the RCX400
- Added code to allow pulseguide commands to be sent to telescopes with StarPatch installed.
- Added the tracelogger to the calls that send the actual serial commands.

1.3.5 Release Notes, 17-May-2022
- Modified the SideOfPier code so that it reports properly. Confirmed with an LX75.
>> See MeadeGeneric Releases : https://bitbucket.org/cjdskunkworks/meadeautostar497/wiki/Release%20History

Ascom & .Net Framework Requirements

It should be noted that Version 1.3.6+ requires Ascom 6.6.0 and .Net framework 4.8 or higher.

The Observatory Computer is running ASCOM 6.6.1, and has .Net Framework 4.8 as shown by this screenshot from RegEdit, so it passes this requirement ok.

At end of the S1195 Session the Observatory Telescope (12" LX200 GPS/R) was shudown without it being first parked.  This was due to a "Leap Day" related bug in LX200 (or less likely the MeadeGeneric driver) that cause the final SlewToAzAlt Async command and Park command to fail.  Scope power was turned off 20s later, leaving the Telescope shutdown in an Unparked State.

This means that the scope won't have correct coordinates when it starts up at the next session (it will assume coords of RA at the Meridian Crossing & Dec 0.0°).  Therefore the telescope will need Syncing on a known star (or other known position) followed by a short mapping run to tie the scope into the TPoint model in TheSky6.

Recovery
As it was known that the telescope was last pointing at Az 209.5° / Alt 33.7° (Dec 4.3°) when it was switched off, this fact was used to perform an initial resync & map as an interim fix, ahead of a final resyc & map at the start of the next session, which will hopefull speed up the recovery process.   This also provided the opportunity to perform a daytime test of MeadeGeneric Driver 1.3.9 which had just been installed on the Observatory Computer.

The steps taken in achieving this are as follows.

• Meade Generic 1.3.9 installed on Observatory Computer. Properties checked to ensure previous settings had been retained.
   
• Marked the current physical position of the Telescope prior to start-up  (most importantly its position on the RA circle)
   
• Telescope Startup via AstroMain, which turned on power to telescope, waited normal 60s for initialisation and made connection via ASCOM / MeadeGeneric
  
  Telescope proceded to slow rotate in RA at start-up which is normal when starting up from an unparked state (telescope is looking for its worm drive reference point) and takes time in doing so.
  
  Scope connected but instead of the expected message at this point "Date/Time prompts have been by-passed" AstroMain instead reported a failure with the message "Date prompts are still showing !"  ,  "AutoStar II Handbox shows  "Daylight Savings> No" and raised an alert with message "(Time/Date by-pass has failed)", with the warning  "Telescope and Computer Times differ by 35,159 seconds"
  
  This is believed to have happened because the Telescope was still going through the extra initialisation procedure of slowly rotating the scope in RA to find the worm drive reference point when the connection was made. It seems the set wait time of 60s after telescope power-up which is ok for starting up the telescope from a Parked state, is too short when starting up the telescope from an Unparked state.
   
• Date/Time details entered manually, using Virtual Keyboard (as the Hand Controller can't be readily used as screen text is not showing, due to recent defect in the screen/hand controller)
   
• Telescope connected in TheSky6
   
• Moved telescope back anitclockwise to the previously marked position on the RA Circle
   
• Moved TheSky's Virtual Sky to be centered on Az 209.5° / Alt 33.7°, an attempted to Sync Telescope on this position, but it failed with a dialog box error "1401"
  
• Stopped the connection to Telescope from AstroMain and TheSky6 and waited to ensure MeadeGeneric driver had closed.
  
• Parked Scope from AstroMain.
  Confirmed the expected message "Scope parked  Turn scope off." was given, and that the telescope was at expected position
   
• Waited 20s and turned off power to telescope.
   
• Turned on power to Telescope
   
• Reconnected to Telescope from AstroMain.
  This time the message "Date/Time prompts have been by-passed" was given.  (as date/time had been manually entered earlier)
  
• Connected to Telescope in TheSky6
   
• Repeating earlier step :
  - Moved telescope back anitclockwise to the previously marked position on the RA Circle
  - Moved TheSky's Virtual Sky to be centered on Az 209.5° / Alt 33.7°
  
• Synced Telescope on this position in TheSky6 (taking the option 'Sync Telescope and begin short mapping run')
  
  (SyncTelescope initially failed as TheSky6 gave the error  something like "Telescope must be tracking in order to Sync"
  So Turned Tracking back on and then successfully synced the scope)
  
• Using the same position, mapped 6-7 points in TheSky6 (effectively performing a short mapping job)
  
• Parked Scope and confirmed that scope moved to its expected position around Az 180°, Alt 1°.  Disconnected to scope in AstroMain & TheSky6
   
• Turned Off Scope Power
  
• Turned On Scope Power and Connected to Scope from AstroMain. Confirmed that the "Date/Time prompts have been by-passed" message was present, showing that the MeadeGeneric Driver had performed the necessary startup steps.
   
• Connected to Telescope in TheSky6
  
• Slewed Telescope to object using TheSky6 and confirmed telescope moved to the expected position
  
• Parked Scope from AstroMain. Confirmed the expected message "Scope parked  Turn scope off." was given, and that telescope was at expected position
   
• Waited 20s and turned off power to telescope.

This shows that AstroMain's Connect Telescope needs to be modified to allow a longer initialisation period when starting up the telescope from an unparked state.
say 180s instead of 60s.  

Actions:
ParkScope routines no longer turn off the telescope if it appears to be unparked (AstroMain 3.66.1).
A checkbox option "Wait 180s after turning on scope" (with default value of unchecked) has been added to AstroMain's More/Obs. tab (AstroMain 3.66.1)

Review
As part of a review looking at increasing the Observatory and Live Session resistance to small power outages the potential of using one or two UPS Systems with offline battery backup was identifed as options for further consideration (see Review - Increasing observatory & session resistance to small power outage, 2024-02-28)

However given that the review has identified that the Observatory's LX200 Telescope is the one observatory element that is most affected by an unscheduled power outage there is requirement to prioritise the selection, purchase and installation of a UPS System for the Observatory Computing Bay. 

The telescope is most affected because a uncontrolled shutdown leaves the telescope in an unparked state meaning that the scope loses all knowledge about its physical pointing direction upon power restoration & telescope restart,  requiring  a resync of telescope position & a re-tie into the T-Point Base Model which is a task that can't be automatically performed without user involvement & oversight.  In comparsion other observatory elements recover by themselves upon power restoration or there recovery can be performed by an automated manager.

This leaves consideration of a UPS System for the AllSky/Weather Computing bay to a later date (and one that can learn from experienes operating the UPS System that is in the Observatory Computer bay for supporting safe Telescope parking & shutdown.

Due to an incident during a power outage test during live session S1197 on 2024-03-13, when Pulsar Dome settings became reset (see Investigation) a review to look at a UPS system to provide Battery Backup for the Pulsar Dome & Cloud Sensor has been kicked off (see Review).

A previous review of UPS needs was made in Dec 2018.  No detailed notes were completed at the time other it being recorded that at some stage 1 or 2 UPS systems will be ordered for the Observatory, but that the decision was put on hold (see UPS Review,  2018-12-22).

Analysis
A UPS Backup to the Pegasus UPB Powerbox that distributes 12V power to the LX200 Telescope is being considered as the telescope is the one item in the Observatory where even a very short power cut can create a very significant problem causing a significant user effort to resolve & delay the next live session. This is because the telescope will turn off / power-down without being parked, requiring the telescope to be re-synced on a known star (or known position) and a short mapping run completed. Whilst program routines can aide the process of resync & short mapping it has to be done under strict user supervision & control.  

For all other observatory elements their recovery following a power outage can either be automated or achieved relatively easily over a remote connection, once power is restored.

The Pegasus UPB Powerbox also distribues power to Focuser1 (TCF-S) , Focuser2 and the Dew Heaters.  

Are there any other equipment/devices in the Observatory Computing bay that would also benefit from having UPS battery backup ?   

• USB Hubs.   It also very important to keep both of the USB Hubs that are attached to the Observatory Computer powered, since without a USB connection to the Pegasus UPB Powerbox the LX200 can't be commanded to Park, and with connection to Conbee II the Observatory's various Smart Plugs can be operated to carry out certain tasks like restarting the SBIG Camera if needed.
   
• SBIG Camera.  The SBIG CCD Camera is prone to having camera dropouts and although these are power cupply related, the camera would probably benefit from being kept powered  during at least short power outages, and would allow the session to resume more and without needing to restore the camera connection.
   
  
• Weather Station. The base station would benefit from having UPS Battery backup if possible. This is own AA batteries (not normally checked) are flat
  

Ensuring that both the Powerbox and the USB Hubs remain powered during a short power supply outage may prevent or reduce the number of instances where the power box and UPB.Switch driver stops working and produce the error pop-ups with messages like "Lost Connection with UPB".

For all other observatory elements their recovery following a power outage can either be automated or achieved relatively easily over a remote connection.

Requirements
The requirements of UPS System for the Observatory Computing bay are listed below

• 240V
   
•  Provide UPS back-up for at least the telescope & SBIG Camera for 15-20 minutes.
   
• A system providing 700 MV or 850MV backup.     
   
  (UPS battery backup are given a power rating in volt-amperes (VA) that range from 300 VA to 5,000 kVA.  
  This rating represents the maximum load that a UPS can support )
   
• Minimum of 2 sockets, though 4-5 would be better
   
• Sockets taking UK Style Power Plugs as this is what the current equipment & a/c adapters are configured with
   
• Some means of communication with the Observatory Computer, and ideally with a means for AstroMain, the Observatory Control program, to access information about the state of the UPS System.
  

Options & Selection

Based on a bit of research and comparison, the main options and the reasons for rejection or selection of each option are given below:

• UPS System of the type typically used for Computer & Servers, having 4 or 8 computer style power plugs (which seem the most common UPS System),
  
   Rejected : because of the requirement / very strong preference for using a system with UK Style Power Sockets
  
• An Online UPS System (as opposed to a UP Offline System)
  
  Rejected:  although Online System provides more stable voltage output as the power is always going via the battery, the cost of Online systems is higher than offline systems.
   
• UPS system from the range offered by CyberPower (700, 1000 & 1200 VA models) or by  APC Schneider Electric  (850 VA model)
   
  Rejected : style & backup support offered by Eaton range was considered to be better / more to my liking  (specs are similar). 6 sockets but only 3 with UPS battery backup.
   
• UPS system from the 3S range offered by Eaton :   3S 550B  (550 VA),  3S 700B (700 VA),  3S 850B (850 VA)
  https://www.eaton.com/gb/en-gb/catalog/backup-power-ups-surge-it-power-distribution/eaton-3s-gen2-ups.html
   
    Chosen : chosen as the upper 2 models (700B & 850B) meets requirements
   
•  Eaton  3S 850B (850 VA)
   
  Chosen : chosen as the VA of 850B model is 150 VA higher than the 700B model (850 VA vs 700 VA)
  through it's larger battery/higher amp-hours (9AH vs 7AH)  though at higher cost   (£126 vs £101 )
  
  It was noted that Eaton UPS systems (though not specifically my model) and other power kit and control software has been installed at several professional observatories around the world including the Mt Washington Observatory, USA & Siding Springs, Australia
   https://www.eaton.com/us/en-us/markets/success-stories/Maintaining-uptime-even-in-the-worlds-worst-weather.html
   https://www.eaton.com/au/en-gb/company/news-insights/news-release-2018-2020/eaton-powers-up-telescopes-at-siding-spring-observatory.html
  

Chosen Option

The chosen option is the Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply 

See Eaton 3S 850B UPS - Off Line Uninterruptible Power Supply System for details about specs and other things to do with the 3S 850B Unit.

Risks
The 3C 850B Model has an advertised workingTemperature range of 0 to 40 deg C and a humidity range 0-85% (non condensing).

Given that the Unit will be in the Observatory where temperature can be sub-zero on winter nights (say down to -6 deg C) and humidites can on occasions be potentially in range up to 100% .

One mitigation is the fact the Unit will be located in the Observatory Computing Bay where equipment is always running and helps to keep temperature there a fraction higher than ambient and humidities a fraction lower than ambient.  Despite that, what are the risks ?

•  Damage to unit and to its battery due to sub-zero ambient temperatures.  Provided the UPS unit's lead-acid battery is fully charged (which will be it's normal opertaing state then it won't freeze at any temperature that it could possibily be exposed to.  However there is risk of electrolyte freezing if the battery is significantly disharged,which can effect the system's remaining run time and possibily reduce the battery's life.  So if ambient tempeartures are sub-zero then it would seem important to park amd turn-off the scope earlier than might be the case if the temperature is higher.
   
• Damage to unit due to high humidity.  Most of the equipment in the observatory is not rated for operation below 0 degC yet it all functions ok in the conditions that we have here in NE Scotland and  < 1km from the sea and prone to high hunidity and North Sea haar.  The unit will be located in the equipment cabinat in the Observing Computer bay which tends to be slightly drier / lower humidity and there is no risk from liquid water damage.

Decision
Decision made (2024-03-03) is to purchase one Eaton 3S 850B  UPS System for the time being, to provide short-term back-up power for the Observatory's  LX200 Telescope and prevent the telescope having an unplanned shutdown leaving the scope in an unparked state, which then requires a significant effort to restore from. This is because the re-establishment of the Telescope's Pointing and the tie-in to the TPoint base model can't be done fully automatically but requires the user in attendance or to provide remote direction & oversight. 

Attached Devices
The telescope runs on 12V which is supplied via a Pegasus UPB Powerbox  which is mounted on one arm of the telescope's fork mount.  Beside the LX200 Telescope the Powerbox also distributes power to an Optec TCF-S Focuser, a Secondary Scope Focuser,  two Dew Heaters,  as well as operating as a USB Hub, a controller for the Secondary focuser, and for measuring Environment/Temperature.

One of the 4 UPS sockets with battery back-up will obviously to be used for the Pegasus UPB Powerbox, but leaves a decision over what the other 3 sockets will be used for.

An audit of the power plugs & equipment/devices was made and the results will help with the decision (The audit will also be used as input to a decision on whether to purchase and install a second, separate UPS System for devices in the AllSky/Weather bay and what its requirements would be ).

At present time the other 3 UPS Sockets will be used for
- the SBIG Camera (with Smart Plug)
- USB Hubs (2 of them powered from one UPS Plug, using adapter)
- Weather Station's Base Station.

The Observatory Computer (a laptop) has its own (longer lasting) battery and therefore doesn't need battery backup support from the UPS system, indeed having the computer not attached to a UPS Socket, means that the Observatory Control Program (AstroMain) can poll the state of its own power supply and use it as a proxy for the state of the Observatory's Power Supply, not withstanding the possiility that AstroMain may be able to determine the source of the power being output by UPS (mains power or UPS battery power).  Once installed in the Observatory the Observatory Computer will have a USB connection to the UPS System and information about power supply & UPS Battery can be viewed in the Eaton UPS Companion software, however it as yet unclear if that information can be passed on to the AstroMain program

(Note: the Eaton's Technical Department replied to a query I had submitted on this point, and they indicted that a 3rd party software program couldn't access the information coming from the UPS system, however a quick dive into the UPS Companion's program folder reveals that alarm data from the UPS system is written to a JSON formatted euc.log file that can be read and used by the AstroMain program)

Configuration
The current configuration of the Observatory Computing Bay and the proposed new configuration are shown in the two diagrams below:


 
 

 
During installtion of UPS in the Observatory it was discovered that it wasn't possible to fit the 4 planned plugs directly into the 4 sockets with battery backup on the UPS, due to the 3 of the plugs (a square smart plug for SBIG Camera & A/C Adpater Plugs for 2 USB Hubs)  having a wider footprint than a conventional plug. To get around this problem a small power strip had to be employed. The FInal Configuration is shown below:


 

Proposed Equipment & UPS Battery Management during a Power Supply Outage

The amount of time that the 3C 850B unit will be able to deliver is dependant on a number of factors, the main ones at the level of the battery going into the utility outage and the load demanded by the attached equipment & devices.

Battery Level would normally be 95%+ when outage occurs, but it won't if there was an earlier outage from which the battery hasn't fully recovered by re-charging in the meantime, it could be alot less.  Load will depend on the sum of the load from each attached device or equipment, which are themselves dependant on the activity levels & demands of the respective equipment.

The Telescope will have a higher load when it is slewing than when it is stationary or just slowly tracking.  The TCF-S focuser and Secndary Scope will have higher loads when adjusting focus than when they are idle.  The SBIG Camera will have higher load if it is using CCD Cooling  it will also have a higher load if it is taking & downloading an image than if it is idle.  The dew heaters will have a higher load when they are operating at 80-100% power level than if they are operating at 0-20% level.  The size of the load can be lowered by Observatory Control programming whilst still keeping the equipment/devices powered up.

According to Eaton' Run Time charts the 3C 850 models (when fully charged) can provide
 - 1 min of backup at 500 W     (2.0 A at 240V)
 - 10 min backup at 200 W       (0.8 A at 240V)
 - 20 min backup at 127 W       (0.5 A at 240V)
-  25 min backback at 100 W   (0.4 A at 240V)

(a 9 amp-hour battery at 12 V = 108 W Hours or 6480 W Minutes, so a 200 W Load would theoretically last 32 min till total discharge, but in practice the time in mintues would inevitably be lower than this)

AstroMain's observatory control will need to recognise an outage, consider the effects of any other recent backups, tie into any information it can get about the UPS System itself and progressively reduce power consumption of the equipment & devices that are operating on backup power during a utility power outage, eventually park the telescope.   A balance needs to be struck between having the equipment a) carry on in a reduced activity mode so that they are immediately ready for action when the Utility Power Supply is restored and b) minimising the battery usage outage (so it available for second or third outage) by putting closing down equipment as soon as possible (eg. parking the telescope & switching it off immediately after it is put onto UPS Battery Power, immediately stop CCD Camera cooling etc)

The preliminary assessment of the management requirements are as follows:

• Dew Heater should  be turned off immediately;   if the outage is short the effect of heaters being off is neglible; if the outage is longer the use of heaters would reduce the amount of backup time for parking the telescope. 
   
• USBs Hubs will be kept on as they are need for communications to work.  Are there any attached USB devices with other than trival power usage ?
   
• SBIG Camera should be kept on and connected, but imaging should stop after the frame in progress. After 1 minute of outage CCD Cooling should be taken back to -5°C, and then after 3 minutes of outage CCD Cooling should be turned off.
   
• Focusers should not be used during the Utility Outage.
   
• Telescope Tracking should be switched off after 1 min of outage. No slewing should be performing, other than to keep the telescope safe or in association with Parking the Telescope.  Telescope should be parked after 5 min of outage after which the Telescope should be is switched off.

Actions 

• UPS Purchase  Order the selected Eaton 3C 850B UPS system
  - Done 2024-03-03 (Amazon)
   
• Extra Product Info.  Contact Eaton regarding particular questions about the 3C 850B model:  Transfer Time, Run Time Chart,  Gen2 vs Gen1 difference & 3rd party program access to UPS System Information.
  - Done 2024-03-05 (Eaton)
   
• UPS Testing. Install the UPS system next to the Development Computer in a temporary setup for testing and for developing a means for accessing UPS / UPS Companion information from AstroMain.
  - To do 2024-03-08 (Observatory)
   
• UPS Software.  Download and install UPS Companion onto the Development Computer and test/examine functionality. Understand whether AstroMain can somehow tap into the data/ information & events from the UPS System.
  - Done 2024-03-05   (Development Computer)
   
• UPS Installation. Install the UPS system in the Observatory Computing Bay.    Move equipment plugs into the proposed new configuration. Connect USB cable between Computer/USB Hub and UPS System.   Install UPS Companion Software onto the Observatory Computer and modify settings.
  - Done 2024-03-07 (Observatory)
   
• Device Loads. Using AstroMain & UPS Companion carry out  to understand the base load of attached equipment/devices (Telescope, Camera/Camera Cooling and Dew Heaters off, and then measure and understand the additional load for each piece of discretionary activity.
  - Done 2024-03-07 (Observatory, UPS Companion)
   
• Equipment & Battery Management.  Extend AstroMain's observatory control to progressively reduce power consumption of the equipment & devices that are operating on backup power during a power supply outage and safely park theTelescope. 
  - Done 2024-03-08 (AstroMain 3.66.5)

A Eaton 3S 850B UPS unit has been chosen to provide Off Line Uninterruptible Power Supply (UPS) for the Observatory Telescope, but will also provide power backups for the SBIG CCD Camera and USB Hubs in the Observatory Computing Bay.

It should be noted that only 4 of the 8 output sockets on the Eaton 3C 850B Unit have battery backup protection (the other 4 have surge protection only).

Key Specs
It's key specs are tabulated below

Manufactorer		Eaton
Voltage:		230V     (220/230/240V)
Wattage:		510 W
Battery Type:		Sealed, lead-acid , 12 V / 9 Ah   (User Replaceable)
Battery Management :		Automatic battery test, deep-discharge protection, cold-start capable
VA Rating		850 VA
Outputs with battery back-up		4 UK (Battey backup & surge)
Other Outputs		4 UK (Surge only) 2 USB Charge
Surge Protection:		Yes
Communication :		USB port (HID compliant)
Construction Type:		Free standing mode
Auto Shutdown Mode:		Yes
Size:		335 x 170 x 86 mm
Weight		4.3 kg
Input Cord Length		1.5m
Relative Humidity		0 - 85%
Software Compatibility		Eaton UPS Companion (enables safe system shutdown, energy usage metering and configuration of UPS settings)
Warrenty:		2 years
Full Specifications :		https://www.eaton.com/gb/en-gb/skuPage.3S850B.pdf#Specifications
Product Page:		https://www.amazon.co.uk/Eaton-850B-UPS-Uninterruptible-Type-Black-white/dp/B0B3RQN8QT/
Run Time Chart		https://eg.eaton.com/ups-battery-runtime/en-gb/3S850D
New Battery  Cost		CSB batteries - these batteries are also used by Eaton £28.99 + £4.95 delivery from Hardware Express https://www.hardwarexpress.co.uk/products/eaton-3s-850b-battery-replacement

Actions:

• UPS Companion Software.  Once the UPS Unit has arrived, proceed to register it and to download and install UPS Companion onto the Development Computer and test/examine functionality. Understand whether AstroMain program / ObsManager can somehow tie into the data from the UPS System or not.
  - Done 2024-03-06  (Development Computer, UPS Companion)

Update 2024-03-05 
The UPS Unit arrived today (2024-03-05) and was unpacked, and set up indoors next to the Development Computer and tested for a while.
UPS Companion was downloaded and installed (the program is name mc2.exe).

 Initial Battery Capacity level was 90% when first connected.



Tests conducted with and without Utility Power Supply connected to simulate the effect of a power supply outage.

In terms of be able to tie into the data from the UPS Unit  I've found that the Companion Software writes alarms to the following file
  C:\Program Files (x86)\Eaton\UPSCompanion\configs\euc.log
with information recorded like the following

[
{"date": "March 5, 2024-6:47:54 pm", "level": 1, "message": "The system is powered by the utility"},
{"date": "March 5, 2024-6:47:45 pm", "level": 2, "message": "Shutdown in 44 min 50 s"},
{"date": "March 5, 2024-6:47:43 pm", "level": 2, "message": "The system is powered by the UPS battery"},
{"date": "March 5, 2024-6:33:40 pm", "level": 1, "message": "The system is powered by the utility"},
{"date": "March 5, 2024-6:33:19 pm", "level": 2, "message": "Shutdown in 42 min 10 s"},
{"date": "March 5, 2024-6:32:18 pm", "level": 2, "message": "Shutdown in 54 min 00 s"},
{"date": "March 5, 2024-6:31:17 pm", "level": 2, "message": "Shutdown in 55 min 00 s"},
{"date": "March 5, 2024-6:30:16 pm", "level": 2, "message": "Shutdown in 56 min 00 s"},
{"date": "March 5, 2024-6:30:16 pm", "level": 2, "message": "The system is powered by the UPS battery"}
]

where the most recent alarm message is listed first.  This provides some idea of the remaining battery capacity based on the current load.  

One iinitial concern was that the UPS Companion settings imply that at some stage the UPS Companion will shutdown the Computer. Now this is not what I want to happen, the Observatory Computer (a laptop) has its own battery (which can last longer that the UPS Unit), and intentionally the computer won't be connected to the UPS Unit allowing it to independantly monitor the state of the Utility Power Supply.  Computer Shutdown Criteria on the 'Settings' tab were changed to 'Ignore' where ever possible. The  'Shutter computer when battery remaining is below " setting can't be set to 'Ignore' however."   It is hoped that the 'Do Nothing' setting for 'Shutdown Type / When first shutdown criteria is reached' is effective in preventing a Computer Shutdown.
 (the other options are 'Hibernate' and 'Shutdown', so ' Do Nothing' seems the appropriate option to choose).

Update 2024-03-07 
Following completion of indoor testing the UPS was installed in the Observatory on 2024-03--07.  

It wasn't possible to fit the 4 planned plugs directly into the 4 sockets with battery backup on the UPS, due to the 3 of the plugs having a wider footprint than a conventional plug (one a square smart plug for SBIG Camera and two A/C Adpater Plugs for USB Hubs). To get around this a small power strip had to be employed. (see diagram below).

Following the installation of UPS (including USB cable to computer), UPS Companion Software was installed, settings checked and relevant tests successfully carried out, switching off / switching on mains power to the UPS Unit.

Web Companion software indicated that the base load from the attached equipment / devices (Telescope, Camera Cooling & Dew Heater all off ) was 34 W (with potential run-time of 52 mins) .   (Later with the UPS System Battery charged to 100%, the reported Run Time with 34 W base load was noted to be 56 mins.

With the Camera turned off altogether the base load falls to an estimated value of 13 W (the UPS system can't seem to measure loads as low as this / the UPS Companion displays load as being 0.0 W.)

A series of tests were carried out to understand the load contribution from each discretionary activity. 
The resolution on the UPS's load measurement seems to 7 W (load increases in 7 W steps)

Activity		Additional Load
Minimal Base Load (UPB Powerbox & USB Hubs only)		13 W
Normal Base Load (with TCF-S Focuser & CCD Camera, w/o Cooling)		34 W
Camera Cooling On at 100% power (during CCD cool down)		14 W
Camera Cooling Off at 62% (stabilised CCD temperature)		7 W
Taking 60s image (possible 5 W during image download ?)		0 W
+Turning Off SBIG Camera		- 21 W
+Dew Heaters at 25% load		14 W
+Dew Heaters at 50% load		20 W
+Dew Heaters at 75% load		27 W
+Dew Heaters at 100% load		41 W
Turning Off TCF-S Focuser		- 7W
Telescope On & Tracking Off		7 W
Tracking On		Neglible
+Slewing		7 - 14 W
Normal Total Load (Cooling 70%, Telescope,  Dew Heaters 75%)		75 W
Maximum Total Load (Cooling 100%, Telescope Slewing,  Dew Heaters 100%)		103 W

Expected run times based on range of loads is shown in the attached table

Case		Load (W)		RunTime (mins)
Minimal Base Load (UPB Powerbox & USB Hubs only)		13 W		60 min +
Normal Base Load (with TCF-S Focuser & CCD Camera, w/o Cooling)		34 W		52 min
Normal Total Load (CCD Cooling 70%, Telescope,  Dew Heaters 75%)		75 W		34 min
Maximum Total Load (CCD Cooling 100%, Telescope Slewing,  Dew Heaters 100%)		103 W		23 min

Actions 

Add 'UPS Companion' (mc2.exe) to Observatory's AstroLaunch Menu. This was tricky to do, as there can be more one mc2.exe process running, one that is owned by the System that runs as a service, and one than is owned by the user.  The ProcessRunning() function has to have a custom modification to look for a mc2.exe process that is not owned by the System, before returning True. To launch the Web Companion User Application the argument "-open" has to be used.  The RunShell() routine was extended to pass an argument to the routine.  This appears to be working ok. Note the service needs to be running in order for the user application to work.
- Done 2024-03-05 (AstroLaunch 1.11.2)

Add a  'euc.log' reader to AstroMain, display the UPS Status on ObsViewer/ObsPic picture and raise and report key events/alerts. Being implemented in AstroMain 3.662.2, a euc.log reader was added and ObsViewer was extended to display the UPS Status (2023-03-08).  To enable this the height of the ObsViewer/ObsPic picture was increased by 20 px..
- Done 2024-03-08 (AstroMain 3.66.2)