David's Astronomy Pages Notes - Session 620  (2018-08-21)

Notes (S619)

Notes Main

Home Page

Notes (S621)

	Session Aims & Highlights
	- Observing Result  - Night Summary Plot
	Operational Issues
	- Critical Issues (0),  Major Issues (4),  Minor Issues (2),  Small Defects (0),  Continuous Improvement (0)
	Fully Automated Operation
	Images from 2018-08-21  >>
	2018-08-22
	Observatory Scorecard / Weather Synopsis
	Shutter Drive Unit - Reinstallation

Session Aims & Highlights (2018-08-21)

Main Aims

1. AstroAllSky. Test recent bug fix and fine-tuning in my AllSky Software (running on AllSky/Weather computer) to correctly display telescope status, location and next targets (running on Main Observatory Computer).
   (Conditions prevented a live test on this occasion. A near-live test was performed the following night (2018-08-22) using a test schedule and telescope simulator as POTH.Scope to send information across to the live AllSky Computer running live images)
2. Autoguiding. Continue test of autoguiding using PHD2 and 80mm APO Guidescope with autoguiding start/stop controlled by an executing QSO Job Queue with observatory manager operating in fully automated mode, including a newly added feature where autoguiding is only used when the longest exposure for target is > 20s or target duration is > 15 mins.
   (Conditions prevented a live test on this occasion. Test to be carried over to the next session)
3. Automated Operations. Perform further live test of Observatory System operating in Fully Automated Mode and test correct response to advancing weather front with developing cloud, increased wind and potential rain.
   (Weather conditions expected to deteriorate with 2 hours of opening observatory or less)
4. Targets. Acquire some images of selected targets (subject to weather conditions)
5. System Stability. Continue checking stability of the Observatory system components (POTH.Hub (Telescope + Dome), Observatory Control Software, Computers), noting in particular any further re-occurrences of the '203 error' or 'Blue Screen' crashes.
   

Equipment & Software

• 12" LX200 and ST-10XME
•  80mm APO and ZWO ASI 178MC
   
•  CCDApp2, TheSky6, CCDSoft 5, PHD2,  POTH Hub,

Highlights

• Automated Operations. Automated Observatory Operations worked as expected
      (shutter closed when the clouds roll-in and wind picked up.  Critical Wind Threshold (15 mph) could be set higher (say 20 mph) )
      Observatory Manager issued appropriate alerts and appropriate actions to Critical Cloud, Critical Wind, and Critical Rain (in this case potential imminent rain based on radar data)
• System Stability. The Observatory System was stable apart from the issues regarding the correct Shutter Status following hardware fault. Importantly there were no instances of "Error code=213(0xd5)"  
• SBIG Camera. There were problems starting the ST-10 camera at the start of the session, after which the drivers for SBIG Camera were updated successfully.
• Shutter. A hardware fault (shutter chain coming off the shutter drive sprocket) revealed a weaknesses in the shutter drive setup, shutter drive firmware and control software identification of a problem & response. 
   (this was a unexpected event whose lessons will be useful for later full-time automated operations)
• Targets. Conditions prevented the acquisition of any useful images

Summary Plots & Logs

Observing Result (2018-08-21a>, S620) Observatory Operations started at 21:50, Fully-Automated Mode from 22:11 Job Queue started at 22:17,  Operations interrupted at 22:22, Shutter closed at 22:28 due to Cloud Shutter Issue at 22:30

(Observation Status :  Green=Completed, Yellow= Partially Completed, Red= Failed)

Night Sky Summary Plot - 2018-08-21 Top axis: Sky Brightness at Zenith (in ADU/s) Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Back to Top

Operational Issues (2018-08-21, S620)

• Critical Issues (0)
• Major Issues (4)
• Minor Issues (2)
• Small Defects (0)
• Continuous Improvement (0)

[ Prev | Next ]

Critical Issues

• None

Major Issues

• Unable to connect to SBIG ST-10 Camera from CCDSoft5 / Driver Error. Unable to connect to ST-10 camera at start of session. CCDSoft not responding and/or producing the following error message   "SBIG driver: operating system error  (Error Code=1032 (0x408)"  (Fig 1). The problem could not be resolved by either restarting CCDSoft5, turning off/turning on Power to Camera or restarting the computer. It would seem that an existing driver had become corrupted or is incompatible with newly installed window or other software / driver (not sure exactly). Problem fixed by updating SBIG 64 bit drivers using SBIGDriverCheck64 (Fig 2).
  Fixed (SBIG Camera)
   
• Observatory Control Program locked up/hung due 'SystemContextDeadlock' . The CCDApp2 program become unresponsive whilst Job Queue was running, and various windows screens were being manipulated /moved around. I not sure if I was stopped execution with a debugging breakpoint or not.  The program was being run with Visual Studio debugger at the time. Either way the principle observatory software suite (TheSky6, CCDSoft5 and CCDApp2) had to be restarted. POTH.Hub was left running on this occasion and it appears that the restarted software where able to reattach to POTH.Dome and POTH.Scope ok.  Possible root cause: When execution was stopped at a break point, it just stopped the managed code. However, un-managed code is still running and in this case, it might have stopped at a time out. At this stage, the un-managed code execution has completely stopped is not being able to understand it has stopped by the managed debugger. Now the managed code sees this as a deadlock situation (Fig 3).  [reference]. It is possible for this to be falsely activated when all of the following conditions are met:
  * An application creates COM components from STA threads either directly or indirectly through libraries.
  * The application was stopped in the debugger and the user either continued the application or performed a step operation.
  * Unmanaged debugging is not enabled.
  One workaround is to disable this specific MDA [reference] . Issue resolved by ensuring that the Program is now run outside of Visual Studio interface/ debugger during Live Sessions.
  Resolved  (Observatory Computer, CCDApp2/AstroMain)
   
• Observatory Manager putting continuous stream of weather condition alert and messages that it will close the shutter. Beginning at 22:30 the observatory manage began issuing weather condition alerts every 5 secs  with messages saying that "Observatory Manager will close the Dome Shutter" . The alerts were initially for  cloud, then wind, then rain (or the threat of rain). Alerts & Messages continued until 23:13 when the observatory manager was stopped and the Observatory closed down under manual mode (Parking Scope, Park Dome etc).   It was noticed at this point that the POTH.Hub was reporting the Dome Shutter as 'Opening'. Close Shutter commands would initially cause the shutter status to show as 'Closing' but be quickly replaced by 'Opening'.  Visiting the Observatory revealed that the Dome was around 20 deg from its Park Position, and whilst the Shutter was closed and not moving, the Shutter Drive motors were busy operating.   Going inside the observatory revealed that Shutter Drive Chain had come off the Drive Sprocket. The Shutter was tight down on the relay, but the shutter Drive Motor was busy running (it is presumed that it had been running for the previous 45 minutes).  The Shutter Drive motors was stopped by pushing the red button on front of drive unit. The Dome was parked from using the Dome Control Unit' keypad & screen.  From 23:00 to 23:10 Shutter Status was being transmitted and recorded by the AllSky Computer on 30s interval scorecards. These show the shutter was principally in an 'Opening' state through this time with occasional moments where Shutter state was shown as 'Closing' .
  Interpretation of Events:    The chain is believed to have come off when an Open Shutter Command (issued when Shutter was closed and when cloud conditions had momentarily improved) was almost immediately followed by a Close Shutter Command (issued when conditions had deteriorated again).  The Shutter Drive unit and Dome.Driver believed the Shutter to be Opening.   Every 5 seconds the Observatory Manager issued weather alerts because shutter was no closed and weather conditions were at 'critical' levels. Commands to Close the Observatory either went ignored by the Dome Driver/Dome Controller/Shutter Drive system  The command to Park Dome at end of session was either ignored by Dome Driver/Dome Controller or not sent because the prior problem dealing with Close Shutter command. 
  The most likely underlying root cause is that the Shutter Drive Unit is sub-optimally placed with its sprocket not being perfectly lined up with the Drive Chain attached to the Dome Roof & Shutter (see notes Shutter Chain jumped off Drive Sprocket).  The software was unable to recognise the exceptional event to adjust the reporting output appropriate, to raise a red flag and carry out mitigating actions.
  Fixed (Pulsar Dome)
   
• Shutter Chain jumped off drive Sprocket.  Shutter Chain jumped off drive Sprocket, meaning that the Dome Shutter could be neither opened or closed.  This occurred when there was a Open Shutter Command (issued when Shutter was closed and when cloud conditions had momentarily improved) was almost immediately followed by a Close Shutter Command (issued when conditions had deteriorated again). Reason:  The most likely underlying root cause is that the Shutter Drive Unit is sub-optimally placed with its sprocket not being perfectly lined up with the Drive Chain attached to the Dome Roof & Shutter.  A gradual loosening of the retaining bolts over last few session may have contributed. The immediate close command following an open command at a position where the chain is at its slackest might have been an unlucky final straw.  Fixes Required. 
  - Chain needs to be put back onto the shutter drive sprocket and motor retaining bolts/tension checked.
  - The shutter motor really needs to be reattached in a position that allows perfect alignment of sprocket and chain. This task has been previously recognized, but delayed as it requires drilling extra holes in the fiberglass dome close to existing holes, with potential risk of damage & place for water to ingress.  Shutter Drive repositioned to correctly align Shutter Chain with the Drive Spocket to reduce risk of chain jumping again.  (Chain has remained on its sprocket ok in 2019-2023)
  Fixed (Pulsar Dome)

Minor Issues

• Frost on ST-10 Camera CCD. Although only a single image was collected during the session it showed the tell-tale beginnings of a frost problem in the camera, which will get worse in following sessions if not dealt with.   Humidity is able to get into the SBIG camera somehow and the desiccator become saturated.  This is an issue that comes up every 6 months or so.  Fixed (Desiccator plug was retrieved from the camera the following day, and regenerated in an oven at 130 deg C. Then replaced in camera.  (The Dummy Plug that is usually used to temporarily screw into the camera whilst the main plug is away in the oven has been temporarily mislaid, a ball of cling film was pushed into the opening instead).
  Resolved (SBG Camera)
   
• Slow File Operations over Local Network. Operation to transfer 3 small files between Main Observatory Computer (Laptop9) and AllSky Computer (Laptop7) very slow (10 mins to move 10 KB !). Other  windows operations over local network also very sluggish.   Only affected WIndows operations.   VNC connections, FTP Upload and AllSky Imaging/Weather program were all working fine. The problem was still present the following day.  Unclear what the problem was and what if anything was done to resolve issue. But it was resolved, and network continues to function ok.
  Resolved (Network. Computer)

Small Defects

• None

Continuous Improvement

• None

[ Prev | Next ]
 

• Fig 1 . CCDSoft / SBIG Driver  Error Message 1032
  
   
  
• Fig 2. List of SBIG 64bit Drivers following update
  It was noticed that the 'sbigudrv.dll' driver (Version 4.99 Build 3, dated 2018-06-14) is new.
  
   
• Fig 3. Visual Studio Message
  

Back to Top

Fully- Automated Operation (2018-08-21)

Back to Top

2018-08-22

Observatory Scorecard / Weather Synopsis

Back to Top

Shutter Drive Unit - Reinstallation (2018-08-22)

After using the Shutter Drive for a while it became obvious that its position wasn't sufficiently correct and that there was a misalignment between the drive sprocket and the line of the plastic chain that pulls the shutter open and closed (a risk of a self-installing a dome for the first time). Slack chain would develop close to the drive unit when the shutter began to open until the shutter reached the first xxxx.  leaving one with some lack of confidence in the system.  It was apparent that there would be long term wear on the side of the plastic chain that pushed against the metal drive sprocket the most. There might also be some extra wear on the connection of the sprocket to the drive axis or on the drive axis itself.    Relocation the shutter drive unit so that the drive sprocket would align perfecting with the chain was put done as a 'task to do' - some time before the winter season.  

The need for this remediation came to head following an incident during the observing session on 2018-08-21 when the chain jumped  off the drive sprocket when the motor tried to open the shutter from a closed position.    With the shutter not moving and unable to close the relay at top of dome, the motor continued running for 45 minutes in a vain attempt to open the shutter before the observatory was visited. Problem wasn't picked up via remote monitoring as the observatory control program was putting out continuous messages that it was closing the shutter due to cloud, wind and threat of rain (from local rain radar).  The incident highlighted several things, but among them was the need to correct the current installation deficiencies, and relocate the shutter drive in order to prevent or strongly reduce the risk of another chain problem in the future.

Pictures below taken whilst re-locating the shutter drive unit the following day (2018-08-22). It was noticed that the drive sprocket was somewhat loose and the opportunity was taken to correctly position the position it on the axle and tighten up the retaining grub screw with a suitable Allen key. It was necessary to move the bracket that comes in contact the relay attached to the shutter drive unit. This was by a small amount only as the bracket still had to reliably hit the relay at the top of shutter travel. The chain is now perfectly aligned with the drive sprocket and works more smoothly, and my confidence in the system is restored.

It is noted that a slack in the chain on the 'downstream' of the drive sprocket is still created in the process of opening the shutter.  Hopefully with the perfectly aligned chain/drive sprocket the chain won't try to crawl its way off the sprocket.
(The slackness create when opening the dome highlights the inherent weakness in Pulsar shutter system due to the inherent changes in the total direct path distance around the various pulley wheels and sprocket as the shutter open/closes. Ideally the system would incorporate a 'tension wheel' that would maintain a constant tension in the chain whatever the position of the shutter.)

Shutter Drive Unit removed (a piece of timber is used to hold the shutter open)		New Holes drilled (10mm to right of original holes)
Re-Installing Shutter Drive Unit (Old holes covered on the inside with tape and then filled from outside with white sealant. They end up largely covered by the washers on the new holes)		Shutter Drive Unit installed in the new position
Chain reattached to Drive Unit (Chain run is now perfectly straight)		Chain aligned with drive sprocket
View of Chain with shutter closed (chain is nicely tight)		View of Chain with shutter partly opened (chain goes slack on downstream side of the drive sprocket when the total direct path length reduces - this highlights the continuing inherent weakness in shutter system)

Back to Top