Issue  Observatory Shutter failed to fully close (hung up on bottom aperture lip / rubber strip.

Description
Due to deteriorating conditions a SoftSupend was called by the Obs.Manager at 22:43 during automated operations. After finishing the active target the session was suspended at 22:58 and the shutter commanded to close. Checking the observatory status  at 23:20 it was noticed that the shutter state in AstroMain and DeviceHub was still 'Closing' , which was a state that was confirmed by examining the Pulsar Ascom log file.

A trip out to the Observatory showed that the lower edge of the Shutter had hung up on the rubber strip that sits on the lower edge of the aperture leaving the shutter almost closed but still around 2cm from its fully shut position.  

The Shutter drive unit motor had turned itself off, or at least there were no growling noises coming from it (this is good and answers a long standing question about what happens if the shutter hangs up).

Whilst the dome was reasonably weather tight the shutter hadn't had been able to reach and depress the lower limit switch.  This left the shutter still communicating a closing state.    Automated operations were left in limbo with the shutter in this protracted 'closing' state.

The shutter was partially opened using the red button on Shutter Drive Unit and then pulling the lower aperture inwards by hand the shutter was finally closed (again using the red button).

It was decided that it would be prudent to end the observing session given the uncertainty in continued operation of the shutter and then review things the following day  (4 hours of good observing time was lost later in the night, but ending the session was the safest option).

Failure Analysis
Investigating it is clear that the primary cause of the failure was that the roof panel below the aperture, to which the shutter drive is attached, bows out somewhat, there is 1 to 1.5cm of play in the panel and it supposed that the shutter just happened to hang up on this occasion.  The roof has been this way since its original construction and erection of the observatory in 2018 , but it hasn't presented a problem before now .   Dome roof was constructed on a flat surface and at the time it didn't seem that the 4 four quadrants would fit together without the bulge appearing and keep the surfaces of the four quadrants smooth where they abutted.

Day time pictures that try to illustrate the failure the previous night are shown below.

Photo showing shutter and roof paneling below lower aperture Dome is at 158° Az position		Photo showing bulge in roof paneling below lower aperture This is with shutter in its fully closed position
Photo showing shutter and lower aperture lip This is with the shutter in the hung up position and reconstructs the failure the previous night		Photo showing rubber strip on lower aperture lip This has now been removed to reduce the  probability of a future failure
Earlier Photo showing Assembly of the Dome Roof (2018)		Earlier Photo showing Dome Roof Completed (2018)

 
A rubber strip is attached to the bottom lip of the aperture.  This strip does make the lip wider than it would be without it,  but even with this rubber strip the shutter has previously always managed to ride over the bottom aperture lip before now and fully close.

Re-reading the Installation manual I see that it does say  "Note that domes with shutter drives do not have the rubber strip"  however I had sufficient length of rubber strip supplied with the observatory that I used a piece of it on the lower aperture lip without any problem up to now. I just had to cut out an appropriate region from the strip at the point where the limit switch is positioned.   The manual shows pictures in the Dome Drive section both with and without this rubber strip being present.

With hindsight this was probably a problem just waiting to happen, but there may be two contributing factors to trigger it on this occasion

•  The dome and its shutter got a significant buffeting from 50-70mph gale force winds last week which may or may not altered the roof or shutter in some subtle way.
   
• The dome was orientated to Az 158° which may or may not be a position where the bottom aperture lip pushes out a bit further than other dome positions.

Attempted Fix/Workaround
In looking for a fix/resolution to the problem I looked at a) ways to resolve the underlying cause and b) ways to minimise the risk of a future failure.

I lfirst looked at potential ways of pulling in the panel but I couldn't think of anything that would work:

• pushing the panel in works temporarily, but take away the force and the panel bows out again
  (I'm reluctant to try any heat treatment to 'mould' the panel into a better shape)
   
• a piece of cable tied to say the pier would pull the panel in but it would obviously interfere with the rotation of the dome
  (a piece of cable attached on the dome roof itself wouldn't be able to exert enough force to hold the panel in)

I then considered ways in which the shutter might be able to sit further from the bottom aperture lip, but again I couldn't think of anything that would easily work (the force of the chain tied to bolt attachment on bottom end of shutter automatically draws the bottom of the shutter inwards towards the aperture lip each time the shutter is closed)

•  replacing the two lower 'wheels' on the shutter with larger diameter wheels would theoretically work, but could interfere with the smooth opening of the dome if too large, they would have to be sourced from some unknown supplier and they couldn't be fitted without deconstructing the roof.
  
• adding a 'ramp to the the lower part of shutter groove to cause the lower wheels on shutter to 'ride up' and outwards when the shutter closes would be problematic. It would be very difficult/impossible to secure any DIY ramp and forming a ramp in overmoulded fibreglass would again require a deconstruction of the roof and it again it could interfere with the smooth operation of the shutter when opening.

Unable to see a means to stop the panel from bowing outwards (i.e. no clear way to fix the underlying problem) it was decided that the best way of resolving the issue (or at least minimising the risk of it happening again) is to remove the rubber strip from the aperture, reducing the effective width of the aperture lip. This was done today (2021-11-10). 

Daytime tests (2021-11-11) indicate that the fix works and observatory operations can continue. The potential of hang up is still there however since the root cause hasn't been dealt with. Dome closing operations will consequently be monitored closely in the next two or three sessions to allow confidence in the shutter system to be regained.

Update 2021-12-05

An attempt was made to see if the panel could be 're-trained' to stay in its correct 'non-bulged' state,  by employing a bungee cord to pull the panel inwards whilst heat was applied to the panel with a hair-dryer and then left to cool down.  The tie was left in place for about one hour. It may or may not have made a slight improvement.   The cord was removed after an hour since an observing session will probably proceed this evening, and the tie cannot be left in place whilst the dome rotates in azimuth.   Although it is doubtful that the 'training' will work, further and longer attempts to 'train' the panel will be made over the next few weeks.

Photo showing bulge in roof paneling below lower aperture This is with shutter in its fully closed position		Photo showing roof paneling with it pulled in by bungee cord (bulge is pulled in but returns again when elastic cord is removed)
Photo showing bungee cord being used to pull-in the bulge in the roof panel (this is in an attempt to retrain the panel to remove the buldge that caused the shutter to hang up)		Photo showing bungee cord connecting roof panel and telescope pier/wedge (obviously the tie can't be left in place whilst the Observatory is operational since the dome needs to rotate)

Issue
A couple of targets showed a signicant Dec jump  just before imaging started.

Description
Large Dec Jumps were noted at start of Dec guiding for two targets (NGC 157 and TT Ari) both of which lay in the SSE sky. In each case a single Dec guide pulse caused the scope position to significantly overshoot to the opposite side of lock position. For example from a position 2' north of lock position a single Dec guide (85% agression, ie a 1.7' guide) lead to the scope moving 5' south of lock position.    The issue comes onto top of an ongoing and annoying general issue with Dec Guiding (especially west of Meridan), which still hasn't been resolved even after replacing the Dec Motor Unit Assembly.

Although imaging wasn't unduly affected on these two occasions, the issue is worthy of documentation, investigation and further monitoring.

1) NGC 157
Sky conditions good/stable. Target lay east of Meridian and a North Dec Guide Mode was autoselected as slight polar misalignment causes stars on this side of Meridian to drift northwards (leading to -ve Dec errors).   Guide star would normally begin at position with a zero Dec error. On this occasion the guide star (actually the average of multiple guide stars) seemed to start at an anomolous position some 5" above the lock position (ie +ve Dec Error) were uni-directional (North) Dec guiding couldn't correct for the position error. The star gradually drifted back towards zero, but only reached around 2-3" by end of autofocusing.  At this point the Guide manager recognised the issue and with the message 'Star needs to be moved back to lock position'  it intervened to change the Dec Guide Mode to South.  With a single output the Dec position jumped from +3" to -4".   The guide star was then on the 'wrong side' for Uni-directional guiding to correct.   After 4 minutes and with the target approaching Meridian (Frame 1), the guide star drifted back towards the lock position again afterwhich with reduced star drift and occasional small Dec South nudges, the target was successfully guided for the remaining 4 frames .

2) TT Ari
Sky conditions moderate / unstable. Target lay east of Meridian and again a North Dec Guide Mode was autoselected as the slight polar misalignment causes stars on this side of Meridian to drift northwards (leading to -ve Dec errors).  Again the guide star (actually the average of multiple guide stars) seemed to start at an anomolous position some 4" above the lock position (ie +ve Dec Error) were uni-directional (North) Dec guiding couldn't correct for the position error.  Autofocusing aborted itself as the selected focus star was deemed to be too dim for reliable focusing (focusing will be pretty reasonable as scope had been focused at start of previous target, NGC 157). The Guide manager recognised that the guide star position couldn't be corrected with the selected North Guide Mode and with the message 'Star needs to be moved back to lock position'  it intervened at 21:19:33 to change the Dec Guide Mode to South.  After just a single guide pulse the Dec position jumped from +2.5" to -5".  After a 30s delay following Dec Guide Mode change the, 1st frame was started at 21:20:04.  The guide star was then on the 'wrong side' for South Mode guiding to correct.   After a couple of minutes the Guide Manager intervened again at 21:21:16 and with a further message 'Star needs to be moved back to lock position' and the Dec Guide mode was put back to 'North', the guide star was then slowly brought back toward the lock position but didn't reach zero line until after a further two minutes of North guiding. Frames belonging to second image set began at 21:21:47

Analysis
This behaviour hasn't really been noticed before.  It has the hallmarks of being due to an overactive backlash compensation.
(large Dec backlash has historically been an issue for the telescope and is why Uni-Directional Dec Guiding is used   Backlash hasn't been remeasured since installing new Dec Motor Unit assembly and it might have changed).   Backlash compensation is turned off in PHD2, but in the Scope itself the Dec/Alt backlash is set to 130% (permitted values 0% to 200%) .

Other Issues & Observations
a) Vertical lines marking start of image set are being plotted at positions which are before their actual start times.
- In case of start of Set 1, imaging started at 21:20:04, but graph seems to show line being plotted at 21:19:56 (8s difference)
- In case of start of Set 2, imaging started at 21:22:44, but graph seems to show line being plotted at 21:21:34 (10s difference)
Is this due to mis-positioning of the line or a mispositioning of the ticks/labels or is this simply the limit of the resolution of the graph ?

b) Vertical Line at 21:21:39 doesn't correspond to either a start or an end of image set and shouldn't have been drawn.  
It occurs 33s (~30s) after Dec Guide Mode change at 21:21:16 which would normally be associated with start of a frame and lead to a vertical line, but in this case it is occuring within a frame.

Code for putting in cross-hatching when camera isn't actively imaging is confused by the Dec Guide Mode change and draws cross-hatching going back to start rather than just back to end of last active image (ie end of frame 1)

Actions
  Dec/Alt Backlash in scope changed reduced from 130% to 80%.   (Done 2021-11-10)
  Monitor Dec Guide behaviour in following sessions (Done 2021-11-17)
  Run PHD2's guiding assistant in order to re-measure scope's backlash value (Done 2021-11-21)
       These should be measured at a number of points but including points where low backlash levels
       are evident from recent data   (S934, Guide Run 23 at Az 117.8, Alt 40.6 ,  S934, Guide Run 31 at Az 144.6, Alt 40.7)
  Modify Guide Manager code that deals with insert of frame boundary starts/end lines and insert of cross-hatching when taking ImageSets (VRVR mode) instead of just individual frames (VVRR mode)

Update 2021-11-17

The same issue occurred in Session S934 (2021-11-16) on two specific guide runs : Guide Run 23 (Scholz's Star) and Guide Run 31 (UGC 5994).  Again these two cases lay in the SSE sky as seen previously. Examining the session (PHD2) log file that documents the communications to/from PHD2 for the last target of the session (UGC 5994, Guide Profile 31) shows that prior to the first Dec Mode Change shown on the guiding chart just prior to Frame1 image when Dec position jumped from +4 to -4 with a single guide pulse (South), there was an earlier single guide pulse (North) just after guiding starte, but before the start of the guiding chart plot, when from a dy position of -0.72 a Dec Guide pulse of -0.553 saw the scope jump to +4.894.   This explains how the guiding charts can start with the guidestar several arc secs away from the lock position.  The two unexpected jumps in Dec position with just a single pulse suggests that the Dec Anti-Backlash setting in the scope is still too excessive even after being reduced from +130% to +80%.   It seems that the backlash characteristics of the scope with the new Dec Motor Unit are significantly different to those with the old motor unit.

Data from this initial jump suggests that the current 80% anti-backlash setting is moving scope in this particular case by around 5" (4.894 - (-0.72)  - 0.533)   At the used guide speed of x0.90 this corresponds to spinning the Dec Motor equivalent to a guide period of 0.37s (calculated from  5 / (15.04 * 0.90)  ).

Update 2021-11-21

Four Dec backlash measurements were made during a test session on 2021-11-21. These included two measurements at sky positions corresponding to the earlier NGC 157 and TT ARi guiding anomalies which seemed to show excessive anti-backlash behaviour by the scope.     These showed no change in Declination Backlash from that measured in 2019 and doesn't indicate why the scope anti-backlash setting would be excessive now when the same anti-backlash setting evidentally wasn't prior to the installation of the new Dec motor on 2021-10-15.