David's Astronomy - Notes 2024-03-13 (S1197)

David's Astronomy Pages
Notes - Session 1197 (2024-03-13)

Notes
(S1196)

Notes
Main

Home
Page

Notes
(S1198)


	Session Aims & Highlights
	- Observing Result - Night Summary Plot - Session Event Log

	Operational Issues
	- Critical Issues (0), Major Issues (2), Minor Issues (4), Small Defects (6), Continuous Improvement (9)

	Images from 2024-03-13 >>

	Investigation - Pulsar Dome Settings reset themselves after a power outage
	Investigation - Zigbee Shutter sensors not showing correct states after a power outage

Session Aims & Highlights (2024-03-13)

Main aims

12P/Pons-Brooks. Acquire images of Comet 12P/Pons-Brooks which has been undergoing regular outbursts.
Power Cut Tests. Perform a live session test of the Observatory response to a brief power cut (10s) and to a longer power cut (10 min). This follows installation of a UPS System to provide battery back up for telescope and addition of Outage Management procedures to AstroMain's Obs.Manager and other improvements and new features to AstroMain, AstroWeather and AstroAllSky.
Targets. Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
AstroMain. Test new 3.66.6 version of AstroMain in a live session, following mutliple changes in association with a new Power Monitor, a UPS Monitor and significant changes to the Obs.Manager. Purpose is to verify there are no significant sideeffects or issues with the new version / new code.

Equipment & Software

12" LX200 and ST-10XME
80mm APO and ZWO ASI 178MC
TheSky6, CCDSoft5, DeviceHub 6.6.1.1, MeadeGeneric 1.3.9, UPB 1.5, ASCOM 6.6, Pulsar Dome Firmware 1.51 & Driver 6.3.10
Pegasus UPB & Motor Focus, TCF-S
SharpCap 3.2, PHD2 2.6.13, Siril 1.0.6, deCONZ 1.18.2, ASCOM 6.6 SP1
[ DomeLink 1.0 / 1.1 - not used ]
AstroMain 3.66.4, AstroPlan 1.39.10, AstroShCap 1.3.15, AstroGuard 1.10.2 (any new items or versions in bold)
AstroVOE 1.13, AstroSN 1.10.2, AstroProtect 1.7, AstroMove 1.12.11
AstroWeather 1.23.14, AstroAllSky 3.20.1, AstroObsCam 1.11.4, AstroLaunch 1.11.1

Highlights

12P/Pons-Brooks. Images of Comet 12P/Pons-Brooks were successfully acquired. The target was acquired as two ToO target observations with 30s and 60s exposures rather than the 90s exposures originlly planned due to the increased brightness of the comet as it approaches.
Power Cut Tests. Tests were successfully conducted. Although the short test went well, there were issues with the longer test and required user intervention to resolve. Issue appeared to be related to the Dome and Zigbee The identfication of gaps & remaining defects was a particular aim for the testing, so in that regard the test was a success.
Targets. 8/27 planned targets observations were completed . Other targets failed due to cloud /session suspension or were cancelled when session was ended early due to cloud. One the planned targets (12P/Pons-Brooks) was acquired as two ToO target observations instead of the original planned target observation. Image from Target 12/27 (Segue 1 ) was later rejected due to Occluded View (following issue with Pulsar Dome Settings after a power outage test). Images from UGC 831 with SN 2023wdd rejected later due to cloud/poor sky transparency.
Zigbee Sensor. ZIgbee open/shut sensors were giving incorrect values after the 10 min power cut test. Shutter had to be opened in order for sensors to be correct.
Dome Parking. Dome failed to parked at correct position at end of the session. This is due to the Pulsar Dome resetting its Azimuth & Settings following a power outage test.
AstroMain / CCDCamera. There were a number of issues with CCDCamera during the early part of the session. The CCDCamera froze at xx (frozen/stuck CcdCamera) xx & xx and there were 'Command timed-out' errors trying to take images at xx. In connection with these CCDSoft was restarted by program control (including SBIG Camera reboot) at xx. xxx, and eventually resolved the issue.

Notes:

Mains Outage Test at 20:37:23 (outage 32s)
Mains Outage Test at 22:35:18 (outage 10.7 min)

Summary Plots & Logs

Observing Plan

Observing Result

Dome & Scope Slewing Performance

Slew/Centering Performance

Guiding Performance

Sky Conditions (Locate Frames)

Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
The black traces across the plot at 20:37 & 22:35 are due to absence of
AllSky imaging at these times due to two Power Outage Tests.

Actual Weather vs Pre-Session Weather Forecast

Session Event Log

Time	Event	Detail
18:22:39	Session Monitoring	AutoStart monitoring for Live Session opportunity between 19:01 & 04:21
18:22:41	AutoStart Waiting	AutoStart waiting till earliest start time at 19:01
19:01:11	AutoStart Delayed	Session delayed due to too few stars
19:18:14	Session AutoStarting	Session autostarting (19:18)
19:18:24	CCDSoft Restarting	CCDSoft being restarted (to set AutoSave No.)
19:18:49	Camera1 Connected	SBIG Camera connected (Set point -10°C)
19:18:51	Session Created	Session Created (Live, 2024-03-13 S01197, ImageSaveNum: 1197001)
19:18:56	Scope Switched On	Telescope Power has been switched on via UPB Powerbox.
19:20:38	Services Started	Observatory Services started
19:20:45	Observatory (Auto)	Observatory placed in Fully-Automated Mode
19:20:49	Session Pending	Session pending (2024-03-13)
19:20:51	Session Initiating	Session initiating (2024-03-13)
19:20:53	Plan Requested	Observing Plan requested from AstroPlan (1.39.10)
19:22:04	Plan Loaded	Observing Plan loaded to queue (Plan ID: 971)
19:22:20	Camera1 Connected	SBIG Camera connected (Set point -15°C)
19:22:56	Telescope Connected	Telescope connected (TheSky6)
19:23:20	Session Equilibration	Session ready to Open Dome
19:24:05	Dome Opened	Dome opened (Opening time 45s, Zigbee 40s)
19:24:07	Session Running	Session running
19:24:09	Queue Started	Observing Queue started (27 targets selected)
19:24:11	Target Started (NrZen)	Target started (Focus Field 8, HIP 39357)
19:24:25	Dome Unparked	Dome unparked
19:28:25	Focusing Started-Foc1	Foc1 Focusing Started (TCF-S)
19:30:44	Focusing Completed	Foc1 AutoFocus Completed (Profile No 1, wide)
19:32:42	Focusing Completed	Foc1 AutoFocus Completed (Profile No 1)
19:32:44	Focusing Started-Foc2	Foc2 Focusing Started (Secondary Scope, using ShCap)
19:34:52	Focusing Completed	Foc2 AutoFocus Completed (Profile No 2, wide)
19:36:36	Focusing Completed	Foc2 AutoFocus Completed (Profile No 2)
19:36:53	Target Completed (NrZen)	Target completed (Focus Field 8, HIP 39357)
19:37:21	Target Started (1/27)	Target started (1/27, AT2024dof (CV))
19:42:26	Focusing Started-Foc1	Foc1 Focusing Started (TCF-S)
19:44:49	Focusing Completed	Foc1 AutoFocus Completed (Profile No 3)
19:50:51	Target Completed	Target completed (1/27, AT2024dof (CV))
19:50:55	Target Started (2/28)	Target started (2/28, 12P/Pons-Brooks, ToO)
19:54:56	Focusing Skipped	Foc1 focusing skipped - star is too dim (TCF-S)
20:12:42	Target Completed	Target completed (2/28, 12P/Pons-Brooks, ToO)
20:12:46	Target Started (3/29)	Target started (3/29, 12P/Pons-Brooks, ToO)
20:16:12	Focusing Started-Foc1	Foc1 Focusing Started (TCF-S)
20:18:37	Focusing Completed	Foc1 AutoFocus Completed (Profile No 4)
20:29:08	Target Completed	Target completed (3/29, 12P/Pons-Brooks, ToO)
20:29:10	Target Missed (4/29)	Target's time slot was missed (4/29, UGC 12639 w/SN2023jri)
20:29:12	Target Missed (5/29)	Target's time slot was missed (5/29, 12P/Pons-Brooks)
20:29:23	Target Started (6/29)	Target started (6/29, AT2024dsj (CV))
20:32:23	Focusing Started-Foc1	Foc1 Focusing Started (TCF-S)
20:34:36	Focusing Completed	Foc1 AutoFocus Completed (Profile No 5)
20:37:23	Power Supply	Mains Power Outage at 20:37:23
20:37:25	Power Supply	Beginning Mains Power Outage Management
20:37:27	UPS System	Mains Power Outage at 20:37:25
20:37:56	Power Supply	Mains Power restored at 20:37:56 (outage time 32s)
20:37:57	UPS System	Mains Power restored at 20:37:56 (outage time 31.0s)
20:38:34	SoftSuspend Cancelled	Soft Suspend is cancelled due to improved conditions
20:40:25	Target Completed	Target completed (6/29, AT2024dsj (CV))
20:41:43	Target Started (7/29)	Target started (7/29, UGC 831 w/SN2023wdd)
20:45:56	Focusing Skipped	Foc1 focusing skipped - unable to find a star (TCF-S)
21:02:30	Target Completed	Target completed (7/29, UGC 831 w/SN2023wdd)
21:04:03	Target Started (8/29)	Target started (8/29, NGC 1122 w/SN2024dru)
21:07:10	Focusing Skipped	Foc1 focusing skipped - star is too dim (TCF-S)
21:23:18	Target Completed	Target completed (8/29, NGC 1122 w/SN2024dru)
21:26:23	Target Started (9/29)	Target started (9/29, NGC 3206 w/SN2024bch)
21:30:37	Focusing Skipped	Foc1 focusing skipped - star is too dim (TCF-S)
21:46:37	Target Completed	Target completed (9/29, NGC 3206 w/SN2024bch)
21:48:43	Target Started (10/29)	Target started (10/29, C/2022 E2 (ATLAS))
21:52:02	Focusing Started-Foc1	Foc1 Focusing Started (TCF-S)
21:54:24	Focusing Completed	Foc1 AutoFocus Completed (Profile No 6)
22:02:35	Target Completed	Target completed (10/29, C/2022 E2 (ATLAS))
22:03:15	Target Started (11/29)	Target started (11/29, 207P/NEAT)
22:28:55	Target Completed	Target completed (11/29, 207P/NEAT)
22:28:59	Target Started (12/29)	Target started (12/29, AT2024djo)
22:32:20	Focusing Started-Foc1	Foc1 Focusing Started (TCF-S)
22:34:30	Focusing Completed	Foc1 AutoFocus Completed (Profile No 7)
22:35:18	Power Supply	Mains Power Outage at 22:35:18
22:35:23	Power Supply	Beginning Mains Power Outage Management
22:35:26	UPS System	Mains Power Outage at 22:35:22
22:40:19	HardSuspend Called	Hard Suspend called due to Power Outage
22:40:20	SBIG Camera	Take Image failed (Command in progress. Error code = 219 (0xdb).)
22:40:22	SBIG Camera	Take Image command accepted (bSbigDriverException flag is reset)
22:40:26	Target Completed	Target completed (12/29, AT2024djo)
22:40:29	Queue Paused	Queue paused for Session Suspension
22:40:31	Session Suspending	Session suspending
22:40:33	Session Suspended	Session suspended
22:42:41	Telescope Parked	Telescope parked (parking time 76s total)
22:42:56	Telescope State	Handbox reads 'Scope parked Turn scope off.'
22:43:36	Telescope Switched Off	Telescope Power has been switched off via UPB Switch.
22:43:38	TCF-S Switched Off	TCF-S Power has been switched off via UPB Powerbox.
22:46:01	Power Supply	Mains Power restored at 22:46:01 (outage time 10.7 mins)
22:46:03	UPS System	Mains Power restored at 22:46:02 (outage time 10.7 min)
22:46:27	Scope Switched On	Telescope Power has been switched on via UPB Powerbox.
22:48:01	Telescope Connected	Telescope connected (TheSky6)
22:48:45	Dome Closing	Dome closure hasn't finished after 140s (ShutterState=Closing)
22:48:52	Camera1 Connected	SBIG Camera connected (Set point -20°C)
22:48:54	TCF-S Switched On	TCF-S Power has been switched on via UPB Powerbox.
22:50:03	CcdCamera Frozen	CcdCamera appears to be frozen at around 22:50 in section 'GetCameraProperties'
22:50:39	CCDSoft Restarting	CCDSoft being restarted (attempt to fix frozen CcdCamera)
22:50:46	SBIG Camera Reset	SBIG Camera being power-cycled
22:51:36	Camera1 Connected	SBIG Camera connected (Set point -20°C)
22:52:54	Pulsar Dome	Restarting Dome Controller
22:53:26	Pulsar Dome	Dome Controller has been restarted
22:53:30	Pulsar Dome	Dome Service has been restarted
22:57:01	Session Resuming	Session resuming
22:58:41	Session Suspended	Session suspended
23:04:15	Target Cancelled	Target cancelled (13/29, 103P/Hartley) due to hardware
23:09:10	User Intervention	User Intervention to abort job queue & switch back to Manual Mode
23:09:13	Queue Aborted	Job Queue aborted
23:09:15	Observatory (Manual)	Observatory placed in Manual Mode
23:13:08	Dome Closed	Dome closed (Closing time 51s, Zigbee 44s)
23:14:13	Services Stopped	Observatory Services stopped
23:14:31	Session Closed	Session closed by User
23:14:34	Program Closed	Program closed by User
23:16:06	Program Opened	Program Opened (AstroMain 3.66.4)
23:16:08	CCDSoft Restarting	CCDSoft being restarted (to set AutoSave No.)
23:16:19	Session Resumed	Session Resumed (Live, 2024-03-13 S01197, ImageSaveNum: 1197108)
23:16:33	Obs.Overseer Started	Obs.Overseer started
23:16:45	Obs.Manager Started	Obs.Manager started
23:16:54	Services Started	Observatory Services started
23:17:55	Dome Opened	Dome opened (Opening time 45s, Zigbee 41s)
23:18:21	Session Monitoring	AutoStart monitoring for Live Session opportunity between 23:18 & 04:21
23:18:23	Observatory (AutoStart)	Observatory placed in Auto-Start Mode
23:18:25	Session AutoStarting	Session autostarting (23:18)
23:19:13	Services Started	Observatory Services started
23:19:20	Observatory (Auto)	Observatory placed in Fully-Automated Mode
23:19:22	Session Pending	Session pending (2024-03-13)
23:19:24	Session Initiating	Session initiating (2024-03-13)
23:19:29	Plan Loaded	Observing Plan loaded to queue (Plan ID: 971)
23:19:42	Camera1 Connected	SBIG Camera connected (Set point -20°C)
23:19:47	Telescope Connected	Telescope connected (TheSky6)
23:20:11	Session Running	Session running
23:20:14	Queue Started	Observing Queue started (16 targets selected)
23:20:20	Target Started (12/27)	Target started (12/27, Segue 1)
23:20:31	Dome Unparked	Dome unparked
23:21:51	Dome Closed	Dome closed (Closing time 45s, Zigbee 44s)
23:21:52	Target Aborted (12/27)	Target aborted (12/27, Segue 1) due to
23:21:54	Queue Paused	Queue paused for Session Suspension
23:21:56	Session Suspending	Session suspending
23:21:58	Dome Closed	Dome already closed (closed at 23:21, Closing time 45s, Zigbee 44s
23:22:00	Session Suspended	Session suspended
23:24:01	Session Resuming	Session resuming
23:24:46	Dome Opened	Dome opened (Opening time 45s, Zigbee 41s)
23:24:48	Session Running	Session running
23:24:50	Queue Resumed	Observing Queue resumed
23:24:53	Target Started (12/27)	Target started (12/27, Segue 1)
23:35:27	User Intervention	User request to close and finish session
23:35:30	Target Completed	Target partially completed (12/27, Segue 1)
23:35:33	Queue Aborted	Job Queue aborted
23:35:35	Session Closing	Session closing
23:36:27	Dome Closed	Dome closed (Closing time 50s, Zigbee 44s)
23:38:05	Dome Parked	Dome parked (parking time 90s), Az: 0.1 deg
23:39:19	Telescope Parked	Telescope parked (parking time 69s total)
23:39:34	Telescope State	Handbox reads 'Scope parked Turn scope off.'
23:39:56	Telescope Switched Off	Telescope Power has been switched off via UPB Switch.
23:41:20	Services Stopped	Night Services stopped
23:41:23	Session Finishing	Session Finishing started (Create Fits Summary, Transfer Files)
23:41:40	Dome (Find Park)	Find Park started (Search Az 80.0 to 100.0°, Step 1.0°, Wide)
23:43:38	Dome	Find Park failed (Unable to detect Park Sensor in the range searched)
23:43:43	Session Finished	Session Finished

Session Alerts & Alarms

Time	Type		Name	Detail
20:37:23	Red	Alert	Power Supply	Mains Power Outage at 20:37:23
20:37:27	Red	Alert	UPS System	Running on UPS Battery 20:37:25
20:37:39	Red	Alert	Dome Alert	DomeRecords have stopped updating
20:37:54	Yellow	Alert	Power Alert	A/C Power has been off for 30 seconds
20:37:56	Green	Clear	Power Supply	Mains Power restored at 20:37:56 (outage time 32s)
20:37:57	Green	Clear	UPS System	Mains Power restored at 20:37:56 (outage time 31.0s)
20:38:19	Green	Clear	Dome Alert	DomeRecords have resumed updating
22:35:18	Red	Alert	Power Supply	Mains Power Outage at 22:35:18
22:35:24	Red	Alert	UPS System	Running on UPS Battery 22:35:22
22:35:38	Red	Alert	Dome Alert	DomeRecords have stopped updating
22:35:48	Yellow	Alert	Power Alert	A/C Power has been off for 30 seconds
22:38:18	Red	Alert	LAN	LAN Ping Checks have stopped for more than 3 mins
22:39:49	Red	Alert	AllSky/Synopsis	AllSky Star & Synopsis data has been inactive for more than 5 mins
22:39:59	Red	Alert	Cloud Sensor	Cloud Sensor data has been inactive for more than 5 min
22:40:20	Red	Alert	SBIG Camera Alert	Take Image failed (Command in progress. Error code = 219 (0xdb).)
22:40:22	Green	Clear	SBIG Camera	Take Image command accepted (bSbigDriverException flag is reset)
22:46:01	Green	Clear	Power Supply	Mains Power restored at 22:46:01 (outage time 10.7 mins)
22:46:03	Green	Clear	UPS System	Mains Power restored at 22:46:02 (outage time 10.7 min)
22:46:10	Green	Clear	AllSky/Synopsis	AllSky Star & Synopsis data is active again (outage 10.9 mins)
22:46:12	Green	Clear	LAN	Local Area Network is active again (outage 10.9 mins)
22:46:20	Green	Clear	Dome Alert	DomeRecords have resumed updating
22:46:25	Red	Alert	Unexpected Dome Closure	Dome is unexpectedly closing (Poor Conditions)
22:46:35	Green	Clear	Cloud Sensor	Cloud Sensor data is active again (outage 11.5 min)
22:48:45	Red	Alarm	Dome Closing	Dome closure hasn't finished after 140s (ShutterState=Closing)
22:50:39	Red	Alert	CcdCamera	CcdCamera appears to be frozen (last cycle at 22:50:03) in section 'GetCameraProperties'
22:50:44	Green	Clear	CcdCamera	CcdCamera has resumed again (22:50) after 41s
22:59:16	Red	Alert	Dome Alert	Dome hasn't reached 'Closed' state after 120s during SoftSuspend
22:59:18	Red	Alert	Dome Alert	Dome hasn't reached 'Closed' state after 120s during SoftSuspend
22:59:21	Red	Alarm	Dome Opening	Dome failed to finish opening after 140s (ShutterState=vcNull)
23:21:06	Red	Alert	Unexpected Dome Closure	Dome is unexpectedly closing (Good Conditions)
23:38:05	Red	Alert	Dome Position Issue	Reported park angle (0.1 degs) shows a discrepency of 89.9°
23:38:07	Yellow	Alert	Dome Discrepency	Hub reports Dome is Parked, but Dome reports Az is 0.1°

[ Prev | Next ]

Critical Issues

None

Major Issues

During Power Outage Testing the Pulsar Dome somehow became reset , causing dome to lose its azimuth pointing and its settings. Before the outage the dome was at Az. 278° but after the outage (with its resultant dome restart) the dome was at Az 0° dome. Relay Settings also became reset which seemingly caused the shutter to close when conditions were ok and would have prevented the shutter was closing if conditions were bad (a potential critical issue). Dome Park positioned was seemingly also reset to Az 0° (equivalent to physical Az 278°), and explains where search for Park Position in range 75-105° (equivalent to physical Az 345 to 015°) failed to work as dome was working 90° away from the correct position (phyiscal Az 90°) and why the Dome was found pointing in a northely direction the following morning. Dome was moved to its correct park Az to enable shutter battery recharging and then later in the day the dome setting were reentered and a reclaibration perform, with Home at 180° and Park at Az 90°. This is the first time this has happened and it is unclear why it might have done on this particular occassion. It is put down as a glitch but it shows that the dome system isn't infalliable and could benefit from being powered from a UPS System with battery backup instead of directly from the main. The ticket is closed.
Resolved 2024-03-14 (Pulsar Dome, Power Supply)
Zigbee sensors located at Top & Bottom Shutter did not showing correct states after a power outage test during the session (22:35-22:46). Issue investigated (see Investigation - Zigbee sensors located at Top & Bottom Shutter not showing correct states after a power outage) and conclude that a change in Shutter State during Power Outage was undetectable as an Event as Smart Plugs that the sensors connect to network through were powered off. and this didn't automatically update when the Power returned. Situation may has resolved by itself after a period of time or by the user opening/closing the Shutter to trigger new events from the Shutter Sensors The ticket is closed. Tickets is closed. A separate continuous improvement ticket is raised to examine this incident in more detail.
Closed 2024-03-15 (Pulsar Dome, AstroMain, Power Supply)

Minor Issues

AstroMain's "Waiting on weather , Wait for conditions to improve" report line is not appropriate where the reason for suspended state is a power outage. (22:40). During the 10.7 min power outage test that started at 22:35:18 the job executor proceeded to Target 13/29 (1-3P/Hartley). After reporting that the session status was "Session Suspended (Due to power supply outage)", it was inappropriate to then report "Waiting on weather... Hold Wait for conditions to improve (22:40)" . Fixed in AstroMain 3.66.5.
Fixed 2024-03-14 (AstroMain 3.66.5)
AstroAllSky's WebUploader stopped uploading to website (2024-03-15 06:43). After successfully uploading AllSky/Weather/Greenhouse images to website at around 06:31, further images weren't uloaded until the issue was discovered and AstroAllSky's 'Restart WebUploader' utility was used. The issue didn't appears to be related to a power outage or particular LAN outage. Examining logs shows that attempted rounds of file uploads at 06:43 (allsky videos & associated images), 06:46:33 (weather /greenhouse graphs) and subsequently round failed due to exceptions in the UploadFileUsingSecureTransfer() routine. First file fails with the exception "Timeout waiting for WinSCP to respond", whilst further files fail with the exception "Session is not opened". At the moment these exceptions indicating an issue with WinSCP session don't trigger the 'RestartWebUploader() routine. Fix attempted in AstroAllSky 3.20.3, the UploadFileUsingSecureTransfer() routine calls RestartWebUploader upon encountering the 'Session is not opened'. Provided the RestartWebUploader successfully restarts a WinSCP upload session the 2 files that initially failed to upload should be both uploaded on the next attempt (the RunFtpQueue makes up to 3 attempt to load each file in the queue)
Fixed attempted 2024-03-15 (AstroAllSky 3.20.3 , WinSCP)
The required Pulsar Dome setting to access the correct state of Cloud Sensor Relay seems to have changed from NC to NO (2024-04-14). Pulsar Dome settings reset themselves at the time of a 10.7min power outage test during S1197 session at 2024-03-13 22:35:18. Checking the operation of the dome the following day. it was found that Relay Setting need to be set to NO ('Normally Open') in order for Pulsar Relay state to match that indicated by the Cloud Sensor. Pulsar Dome was restarted to check that the state of the relay (safe or unsafe ) is dependant on the relay state when Pulsar Dome is started (no conclusions on this). Pulsar Seting changed to 'NO'. Checking a screenshot of settings used at 2023-11-15 when the most recent version of Pulsar Dome Software was installed confirmed that 'NO' is the setting in use at that time. A shutter opening test needs to be conducted to verify that the relay is having the expected behaviour (i.e. the shutter closes when conditions are unsafe, and stays open when conditions are safe). Required settings have been been reentered via the Pulsar Dome Software and the appropriate Relay Safe/Unsafe is being interpreted correctly by the dome. Tests to check Shutter responds correctly to the change in Relay state from Safe to Unsafe have been checked and verified (with a further check made on 2024-03-16)
Resolved 2024-03-14 (Pulsar Dome)
AstroAllSky's automated 'Restart WebUploader' after a power outage isn't reliable. Unclear what the issue is ? does the delay period after power is restored need to be lengthened ? or that a second attempt is locked up the WinSCP and it can't then work. Problem happened and the two occasions that a power outage test wa conducted at 20:37:23 & 22:35:18. A manual workaround had to be used which was to close and reopen AstroAllSky. A later failure of WinSCP uploading on 2024-03-15 06:43 was investigated and a fix attempted. This which may also help as a backup to the issue of Restarting WebUploader after a power cut. Improvements made 2024-03-20 in AstroAllSky 3.20.3. Ticket closed. Any new issues will be raised as new tickets if/when they occur.
Closed 2024-03-20 (AstroAllSky, Power Supply, WinSCP)

Small Defects

The Pulsar Dome's Ascom Log indicates that AstroMain's RestartPulsarDome() routine doesn't allow sufficient time before the dome is reconnected to (15:05). During 'Restart Pulsar Dome' operations 12s are currently allowed for the dome's initialisation process (between turning the Dome On and making an ASCOM Connection). Examining the ASCOM.Pulsar_Observatories_Dome log indicates that 'Volatile' records only begin to contain valid Shutter infornation at a point around 5s after the Connection is made. This can mean that AstroMain may be seeing inappropriate / incorrect values for the first 5s after reconnecting dome, which could result in unexpected/unwanted actions. It is proposed to increase the wait period from 12s to 20s. Fixed in AstroMain 3.66.5.
Fixed 2024-03-14 (Pulsar Dome, AstroMain 3.66.5)
Dome Relay flag &state are being shown on ObsPics even though Dome is not connected . ObsViewer to onlt Dome Relay flag & state when Obs.bDomeConnected is True otherwise it should be hidden. Fixed in AstroMain 3.66.5.
Fixed 2024-03-14 (AstroMain 3.66.5)
ObsPics show DomeT. jumping to a Cyan flag with text 0.0°C (22:53). This occured whilst restarting the Pulsar Dome and is due to Volatile message having zero values as the connection is made too soon after restarting the dome controller (see separate issue). FIx in AstroMain 3.66.5, Dome T. is shown as a white flag if DomeRecord.BatteryLevel = 0.
Fixed 2024-03-14 (AstroMain 3.66.5)
"Dome hasn't reached 'Closed' state after 120s during SoftSuspend" alert & report message is issued twice within 2s (22:59) Alert & report message is given in both SuspendedStageNo = 4 and SuspendedStageNo = 5. Implemented in AstroMain 3.66.5 using flag bDomeNotClosedAfter120sReported .
Fixed 2024-03-14 (AstroMain 3.66.5)
Single anomalously formatted report line (20:38) . Report line at 20:38 "p; Alert DomeRecords have resumed updating (20:38)" between start outage recovery and resume dew heaters. Examining code indicates that the lines should have read
> Observatory Manager Alert DomeRecords have resumed updating
Unclear why this has happened - the code line for reporting this is correct. This has either not been seen before or is very rare. Issue is closed for now, but continue to monitor future sessions for similar cases in future sessions.
Closed 2024-03-14 (AstroMain 3.66.4)
AstroAllSky showed Red Flag against CloudSensor data even though AstroWeather shows CloudSensor data being current and ticking over.
To check 2024-03-14 (AstroAllSky )

Continuous Improvement

Add 'Outage Management, Stage No' to AstroMain Console title whilst outage management is active. .
Done 2024-03-14 (AstroMain 3.66.5)
Consider reporting time since outage started and time since outage ended. During Outage Management consider give more information regarding timings. Implemented a solution in AstroMain 3.66.5, but need to see whether it is cosmetically Ok.
Done, for review 2024-03-14 (AstroMain 3.66.5)
Add 'Close TextPad Windows' button & routine to AstroLaunch's Admin Tab. When running on the Observatory Computer it would be useful to have a utility to close all Texpad windows in one go.
Done 2024-03-14 (AstroLaunch 1.11.3)
ObsPic to show 'Dome Restarting' when Pulsar Dome is being restarted.
Done 2024-03-14 (AstroMain 3.66.5)
Query. Attempt to Open Shutter via button giving dialog with message "Can't open Dome due to Hardware". Provide user with option to clear bDomeDataIssue. There's already a button 'btnResetDomeDataIssueFlag' but can't find it on the ObjConsole. The button 'Reset bDomeDataIssue' is on the More/Backdoor tab in the 3rd column of buttons.
Done 2024-03-15 (AstroMain, User Awareness)
Query. Dome Alert at 22:46:25 with message "Dome is unexpectedly closing" (22:46). Whilst waiting for mains power to be restored (Outage Stage 5) the message "Dome is unexpectedly closing (Poor Conditions)" was reported at 22:46:25 (this is 11.1 min after Outage Started. This was around the time that Mains Power was restored at 22:46:01, and the Dome would have started to initialise, taking 16s or so. (ie to around 22:46:17). (whilst Dome was off the Pulsar Dome Driver logged "System.TimeoutException: The operation has timed out", every 5s or so. At 22:44:18.920 the dome recieves a SlewToAzimuth (278.9) request but it seemingly fails with the same System.TimeoutException. Volatile messages begin again at 22:46:04 (3s after controller was powered up) but contained largely zero values until 22:46:22 (21s since mains power restoration). ShutterStatus began at value 6, becoming 3 at 22:46:21 and staying at 3 until 22:50:50 when the value changes to 1 (Closed). Problem is likely related in some way to either the ending of power outage and the appearance of Relay = Unsafe from initially invalid CloudSensor/Rain values, or the reset of Dome Settings by the Dome Controller (including Relay settings). Ticket closed after review. 2024-03-17.
Closed after review 2024-03-17 (Session, Pulsar Dome)
Extend Obs.Manager logic to detect an unexpected resetting of Pulsar Dome's settings. This could involve spotting an unexpected jump in reported Dome Azimuth from the Az prior to an outage to an Az value of 0.0 post outage, and to raise an appropriate Alert in response. Provided that dome isn't moved the Dome Az could in theory be resynced to the Azimuth of the dome immediately prior to the power outage & settings reset. This wouldn't address the loss of correct dome settings, but would al least allow the dome to be parked correctly. A new 'Dome.DomeOffAz' variable has been added to record the Dome's Azimuth at time that Power Outage is detected or before performing a Restart Dome operation. Implemented in AstroMain 3.66.6, with changes to Dome.RestartDome() and ObsManager.RecoverFromOutage().
Done 2024-03-18 (AstroMain 3.66.6+)
Review the case 'Zigbee sensors located at Top & Bottom Shutter did not showing correct states after a power outage test during the session' (22:35-22:46). Review the Major Issue that arose in the session where the Top/Bot Zigbee Sensors didn't hold the correct values and the events associated with opening the dome weren't recognised / weren't reported. This is most-likely due to the parent zigbee router (a smart plug) was powered down during the outage and had reestablished it place in the network when the shutter closed, causing the change in Top/Bot Shutter Sensors state to be 'lost', and it was only after Top/Bot Shutter sensors recorded a new change in state did websockets events then fire off. This is always likely to be an issue. Prehaps more post outage checks / tests need to be conducted. Closed
Done 2024-03-18 (AstroMain, Zigbee Network, Power Supply, User Awareness).
Dome Info should be reported after performing a Restart Dome Operation. A dome check routine should be run after restarting dome & reconnecting. Dome Az before the restart should be compared with Dome Az after the restart. Similarly after an outage recovery the dome check should make and report the dome az comparison.
Done 2024-03-18 (AstroMain 3.66.6)

[ Prev | Next ]

Fig 1. Report messages associated with Azimuth and Dome Closing issues after the second power outage test.

Mains Outage Test at 22:35:18 (outage 10.7 min)

Slewing 2024-03-13 22:29 (Local, GMT) Dome Position Check Dome Position Info Dome Az. 277.9° (vs. Scope Az. 286.8°, Delta Az. 8.9°) Dome Position Info Dome Az. 277.9° (vs. Slave Az. 277.9°, Delta Az. 0.0°) Last Dome Movement Ok 0.2 mins ago Dome Slaving Ok Slaving is On Shutter Status Ok Shutter is Open [ Dome is at Az 278 prior to the outage ] Power Monitor Alert Mains Power Outage (22:35:18, Outage No. 3) Outage Management (5a) 2024-03-13 22:43 (Local, GMT) Dome Closure: Alert Dome is unexpectedly closing (Poor Conditions), 2024-03-13 22:46 (T+11 min) Dome Safety Error Dome closure hasn't finished after 140s (22:48) Dome Safety Info There may be a mechannical or electrical issue. Check Shutter Ok Hub reports dome is Closing (3), PulsarLog reports Closing (3) Dome Closing Diagnosis & Recovery 2024-03-13 22:48 (Local, GMT) Dome Connection: Ok Dome is connected Shutter Status: Info Closing (Slewing = True) Check Dome 2024-03-13 22:49 (Local, GMT) Checking Dome Ok Dome is connected (Az. 0.00°) Checking Shutter Ok Closing (vs Zigbee: Open) Outage Management (6b) 2024-03-13 22:50 (Local, GMT) Outage Recovery Info Outage Recovery has been completed Continue operations... Info Returning to normal operations Restart Dome (incl Power Reset) 2024-03-13 22:52 (Local, GMT) Restart Dome Service 2024-03-13 22:53 (Local, GMT) Open Dome 2024-03-13 22:57 (Local, GMT) Dome Connection Error Dome Connection has been lost. Attempting reconnection Open Dome Warning Dome opening hasn't finished after 60s Resume Session Fail Shutter hasn't opened within 100s Open Dome Fail Dome opening hasn't finished after 140s Open Dome Fail There may be a mechannical, electrical or software issue. Start Hub Services 2024-03-13 23:00 (Local, GMT) DeviceHub: Ok Started already (ASCOM.DeviceHub, 6.6.1.1) Dome: (COM3) Ok Started (23:00) (ASCOM.Pulsar_Observatories_Dome.Dome, 6.4.0.0) Dome Connection Ok Dome Connection has been recovered Ok Close Dome 2024-03-13 23:12 (Local, GMT) Observatory Shutter Info Shutter is closing(23:12:18, open time > 5.5 min) Observatory Shutter Info Shutter has closed(23:13:02, closing time 44s) Close Dome... Ok Dome is closed (Closing time 51s, Zigbee 44s) Close Session 2024-03-13 23:14 (Local, GMT) Close Session Ok Session closed by User Reopening Session 2024-03-13 23:16 Open Dome 2024-03-13 23:17 (Local, GMT) Open Dome... Ok Dome is opening (Clarity: 32.7, Stars: 38, Rain: -1.1, Wind: 2.5) > > Observatory Shutter Info Shutter is opening(23:17:11, close time > 38s) > Observatory Shutter Info Shutter is open (23:17:52, opening time 41s) Open Dome... Ok Dome is open (Opening time 45s, Zigbee 41s) > Dome Closure: Alert Dome is unexpectedly closing (Good Conditions), 2024-03-13 23:21 Close Dome... Ok Dome is closing (Clarity: 32.9, Stars: 33, Rain: -1.0, Wind: 3.4) > Observatory Shutter Info Shutter is closing(23:21:03, open time 3.2 min) > Observatory Shutter Info Shutter has closed(23:21:46, closing time 44s) Close Dome... Ok Dome is closed (Closing time 45s, Zigbee 44s) Session Suspending 2024-03-13 23:21 (Local, GMT) Session: 2024-03-13 Info Close Dome 2024-03-13 23:21 (Local, GMT) Checking Dome... Ok Dome is Closed (Closed at 23:21, Closing time 45s, Zigbee 44s) Session Resuming 2024-03-13 23:24 (Local, GMT) Open Dome 2024-03-13 23:24 (Local, GMT) Open Dome... Ok Dome is opening (Clarity: 32.7, Stars: 31, Rain: -1.1, Wind: 3.0) > Observatory Shutter Info Shutter is opening(23:24:02, close time 2.3 min) > Observatory Shutter Info Shutter is open (23:24:43, opening time 41s) Open Dome... Ok Dome is open (Opening time 45s, Zigbee 41s) Dome Position Check Dome Position Info Dome Az. 192.7° (vs. Scope Az. 193.5°, Delta Az. 0.8°) Dome Position Info Dome Az. 192.7° (vs. Slave Az. 192.6°, Delta Az. 0.1°)

AstroGuard logs record

- stop/restart services at 22:41 - 22:48 - dome closing at 22:50 (closing time 31s) - exception in Main Loop at 22:53. Property read ASCOM.DeviceHub.Dome ShutterStatus is not implemented in this driver. - dome opening at 23:11 (opening time 42s) - dome closing at 23:12 (closing time 47s) - dome opening at 23:17 (opening time 47s) - dome closing at 23:21 (closing time 47s) - dome opening at 23:24 (opening time 42s) - dome closing at 23:35 (closing time 47s)

Investigation - Pulsar Dome Settings reset themselves after a power outage

Issue
During Power Outage Testing the Pulsar Dome settings somehow all became reset , causing dome to lose its azimuth pointing and its settings.
After restarting the dome it believed that it's Azimuth was some 90° different from its actual physical azimuth.

This resulted in

slaved dome not moving to the correct physical Azimuth to correspond with the Telescope position
(leading to images with occluded view of the sky)
shutter unexpectedly closing when sky conditions were good
(due to Relay settings having been reset)
dome not parking at correct physical Azimuth at end of session
(leading to the dome being unable to recharge the Shutter battery following the session)

Description
Following the installation of a UPS System to provide battery backup to the observatory's LX200 Telescope & SBIG Camera (2024-03-03), live tests was carried out during the S1197 session, involving firstly a 30s outage and later on a 11 min outage. The short outage proceeded fine, but the longer outage test produced very anomalous dome

Before the outage the dome was at Az. 278° but after the outage (with its resultant dome restart) the dome carried the value Az 0° . Besides loosing its knowledge of its last pre-outage Azimuth, Relay Settings also became reset which seemingly caused the shutter to close when conditions were ok and would have prevented the shutter was closing if conditions were bad (a potential critical issue). Dome Park positioned was seemingly also reset to Az 0° (equivalent to physical Az 278°), and explains where the search for Park Position in Az range 75-105° (equivalent to physical Az 345 to 015°) failed to work as dome was working 90° away from the correct position (physical Az 90°) and why the Dome was found pointing in a northerly direction the following morning.

Dome was moved to its correct park Az to enable shutter battery recharging and then later in the day the dome setting were reentered and a reclaibration perform, with Home at 180° and Park at Az 90°. This is the first time this has happened and it is unclear why it might have done on this particular occassion. It is put down as a glitch but it shows that the dome system isn't infalliable and could benefit from being powered from a UPS System with battery backup instead of directly from the main. The ticket is closed.

Analysis
For some reason all the settings in the Pulsar Dome Controller reset themselves to default values and didn't carry the settings previously entered for the Dome.
Current Azimuth and Park Azimuth were both reset to the value 0.0° . Relay settings were set to 'NC' and no longer carried the required value 'NO' . Times were also reset.

It is likely that the Dome reset in response to a particular current/voltage state during the power-off or during the subsequent power-on, since this issue hasn't previously been seen during other power outages or during operations to restart the Pulsar Dome using a smart plug connected to the Observatory's Zigbee Network.

Conclusion

The Pulsar Dome Settings likely become reset on this occasion due to a random glitch.
This is thought to be a relatively rare event but if /when it occurs it is clearly highly disruptive to observatory operations until the matter is addressed by the user.
The Pulsar Dome would benefit from the support of UPS battery backup, but it is unclear as yet if the cost/benefit justifies it.
(this is the subject of a subsequent review, see Review - Uninterruptible Power Supply (UPS) for the Pulsar Dome & Cloud Sensor, 2023-03-15)

Recovery (2024-03-14)

Following the session, the required dome settings were re-entered to the Dome Controller using the Pulsar Dome Software and then checked in the Controller's LCD Screen Menu/Settings. In particular the times that Shutter would stay open if it looses bluetooth connection with the Controller is set to 5 min, relay mode is set to NO and delay on Relay is set to 30s. Home Az was set to 180° and Park Az was set to 90°.
The state of the Relay was checked to ensure that CloudSensor's view on Safe/Unsafe State matched the Pulsar Dome's view. Shutter was checked to ensure that it closed when conditions changed from Safe to Unsafe. This was tested during daylight conditions by opening the Shutter with the 'Light disabled in the CloudSensor Head., and then reenabling the 'Light' component in the CloudSensor Head and checking that the Shutter began to close 30s afterwards.

(Note: 30s is used to provide time for Observatory Control Program and back-up (AstroGuard) to have first bite at closing the Observatory in a manner that interlaces well with overall suspension of the session & the executing job queue)
Since the dome's rotational calibration is very likely to have been also been reset it was deemed prudent & necessary to perform a new Calibration of the Dome. This was done in the normal manner, which involved inserting the dome's home magnet into its slot in the dome roof flange (taking acre to ensure that its north/south oriientation is the correct one) and then using the Calibrate option in the Pulsar.

(Note: once completed the magnet is removed for safe keeping since the magnet is found to corrode away if left in place)
Following the calibration the dome was parked at it correct Park Poistion (Az 90.0°) and Shutter Battery allowed to recharge.

Actions:

Recovery. Complete tasks necessary to recover the dome so that it is ready for the next session.
Done 2024-03-14 (Observatory, Pulsar Dome).
UPS Review. Commence a review to look at the option of adding a UPS unit with backup battery to reduce potential issues assoicated with power outage. Review has commenced, but a decision on adding UPS is still outstanding.
Done (Review).
ObsManager. Add appropriate new logic to AstroMain's ObsManager to detect unexpected resetting of Pulsar Dome settings. This could involve spotting an unexpected jump in the reported Dome Azimuth from the Az value prior to an outage to an Az value of 0.0 after the outage, with the raising of appropriate Alerts.
Done 2024-03-18 (AstroMain 3.66.6)

Investigation - Zigbee Shutter sensors not showing correct states after a power outage

Issue
Zigbee sensors located at Top & Bottom Shutter did not showing correct states after a power outage test during the session (22:35-22:46)

Description
A 10.7 minute power outage test was conducted starting at 22:35:18 to help understand the robustness of the Observatory systems to a power outage during a live session. Following the test he Zigbee sensors located at Top & Bottom Shutter did not showing correct states.

Analysis

It is conjectured that during the power outage test when the 'parent' Smart Power Plug (that the sensors are normally 'attached to' in the zigbee network) were powered off, the observatory shutter closed. It is entirely possible (and indeed expected) that this was because the shutter hadn't heard from the Dome Controller for more than 5 mins due to the power outage (and by design automatically closes the shutter)

Alternatively the closure was commanded by the dome controller after the power returned and the Dome Controller had rebooted. But as conditions were good it would be unclear clear why the dome controller would do this. It is possible the Dome Relay may have been adversely affected when the CloudSensor was powered off or because the Pulsar Relay settings went from NO to NC when dome settings were somehow all reset upon the power up at 22:46:01, the closure could have been triggered by the relay (or rather a different interpretation of the relay state)
After power came back there was a report at 22:46:25 that the dome was unexpectedly closing. (or unexpectedly still closing if Shutter Controller had tried to hut it earlier at only 5-6 mins into the 10.7 min power cut. This is a separate issue, and one that can be potentially sorted out by looking at states of the Zigbee Top & Base Shutter sensors. However as the Smart Plug was either unpowered and hadn't yet attached itself to sensors, it meant that the deCONZ system which was still running on the Observatory Computer didn't receive news of the change in the 'physical' state of the two Shutter sensors.
The obseratory was approached and it was found that the shutter was apparently closed and pointing at physical Az 90°, but with hindsight it could have pointing at a physical Az 278° and have been either open or closed, as the shutter aperature would have been on the other (hidden) side of the observatory.
A zigbee network report at 22:46 indicated that the Shutter was open (top sensor closed and bottom sensor open). However the sensors were last seen 51-53 minutes previously and were thus not showing the current situation. A further zigbee network report at 23:05 & 23:06 again indicated that the Shutter was open (with Last Seen time of 1.2 hours ago)
At 23:12 the Shutter was closed via AstroMain's 'Close Shutter' utility and the dome closed which was then correctly indicated by a new Zigbee Network Report at 23:13 which confirm Shutter as Closed (with Top/Bot Sensors with Last Seen time of just 2.2 and 0.2 mins respectively). Prior to this the dome which had been closed was opened at 23:11 as confirmed by the Pulsar ASCOM Log.

It hadn't been possible to open the shutter from AstroMain at that time (due to 'Hardware' 'lock' ) and so the dome was opened from DeviceHub and it took a normal time of 43s. (23:11:02 to 23:11:45).
Zigbee Service including WebSocket were restarted at 22:46:25, which might have been too soon after power was restored (22:46:01). It was again restared at 23:06:47 , but no zigbee events were reported for the top/bottom shutter sensors at 23:11 (when the shutter was opened from DeviceHub) but events were reported at 23:12 (when the Shutter was closed from AstroMain). The zigbee log shows the Bot Sensor (22) becoming 'Open' at 23:11:03 indicating the Shutter has begun opening at this time, and shows the Top Sensor (12) becoming 'Closed' at 23:11:44 indicating the Shutter had become fully open.
The failure to report that changes might be due to 'Shutter' appearing to be be already open, so the the Zigbee events when unnoticed/ unreported. Opening of the Shutter at 23:17 was reported correctly. (Zigbee sensors reported as expected, and was the (unexpected) closing of the Shutter at 23:21, the opening of the Shutter at 23:24. A separate continuous improvement ticket has been raised to review this case and potential actions in more detail. The ticket is closed, as the issue has resolved.