Fig 1.   Dome Relay Settings in Pulsar Dome Settings  (from screen capture 2023-10-28)

Note that 'Polarity' is set to the 'NO' option for it to get the replay state intended by the observatory's CloudSensor.

'Replay Dome Slews' is a new capability of the Observatory's AstroMain Program, which replays the Dome Slews conducted during the last or other previous session and compares dome slewing performance with the performance seen during the actual live session.

Background
An ongoing issue is that some sessions experience poor dome slewing performance whereby up to 30-35% of slews experience a significant hold-up during the course of the slew amount to 20-120s ( and occasionally up to 150-180s). During the hold-up the rotation drive wheels spin on the roof flange and don't generate rotation of the dome roof until some point where they gain sufficient friction to begin rotating the dome again.   The amount of total time lost can sometimes reach 15-18 minutes in a session.
 
In other sessions (on other nights) the dome slewing performance is good with no significant hold-ups at all.

The are several factors at play here.
- Observatory walls form a non-perfect circle which varies the effort needed to move the dome at differing points.
- Deformations in the roof flange (caused) by the force of wheels acting on the certain section of the roof flange (made worse by the wall's non-perfect circle shape)
- Amount of force that dome rotation wheels (2 wheels on one side of the flange, 1 wheel on the other) can apply to the dome's roof fange, controlled by tension adjusters.
- Amount of condensation/dew formed on the roof flange (both internal & externally facing) and the amount of dampness collected on the dome's rotation drive wheels.

Increasingly  'condensation/dew' is the factor that is believed to be the most significant in terms of whether a session will or will not experience significant hold-up.

New Capability - Replay Dome Slews
In order to assess the impact of condensation/dew in particular a new button and thread routine have been written and added to AstroMain 3.16.2 on 2023-10-21 .

After the user selects a session to 'replay', the 'Replay Dome Slews' will read key information about actual dome slew made during the previous session from that session's '(slewtable).dat' file  and directs the dome to make the precise same series of movements that the dome made during the live session, and compares dome slewing times, speeds and hold-ups (if any) with the equivalent numbers from the live session.

Testing
After first testing using dome simulators on Development Computer the new version of AstroMain (3.63.2) was installed on the Observatory after which the dome slews from the previous session (2023-10-16 S1157) were successfully replayed on the morning of 2023-10-22.

Replay was conducted after wiping / drying the roof flange and indirectly also drying the dome's rotation drive wheels.  An initial attempt to move the dome failedas the rotation wheels where spinning of the same spot for 2+ minutes.  This prompted the action to wipe / dry the rotation system.

Report messages generated during the course of the playback are shown below:  For each slew the details of the slew from live session are given.  These are the Start and End Points for the slew (Az1 and Az2) , the Slew time, the Angle traversed, Speed and Hold-Up details (if any),.  The slew from Az1 to Az2 is replayed and Slew time, Speed and Hold-Up details (if any) are then presented, for the user to compare

Replay Dome Slews                       2023-10-22 11:18  (Local, BST)
  Source Session              Ok        2023-10-16 S01157
  Source Slews Table          Ok        2023-10-16 S01157 Observatory (slewtable).dat


  Replay slew to Focus Field 5          2023-10-22 11:18  (Local, BST)
  |Target                          Time     Az1     Az2    Slew   Angle   Speed  Holdup  Dome Holdup Az
  |                               hh:mm     deg     deg    time     deg   deg/s    time    degs
  |Focus Field 5                  04:31    90.1   179.8     89s    89.7    1.00     10s  
  Preparing for slew 1        Ok        Dome at Az1 90.1°
  Making slew                 Ok        Dome at Az2 179.8°  48s    89.7     1.9          

  Replay slew to Nova Cas 2021          2023-10-22 11:19  (Local, BST)
  |Nova Cas 2021                  04:42   187.1   310.5    130s   123.4    0.95 D*  34s  
  Preparing for slew 2        Ok        Dome at Az1 187.1°
  Making slew                 Ok        Dome at Az2 310.5°  60s   123.4     2.1          

  Replay slew to M31                    2023-10-22 11:21  (Local, BST)
  |M31                            05:02   312.7   280.3     32s    32.4    1.01      5s  
  Preparing for slew 3        Ok        Dome at Az1 312.7°
  Making slew                 Ok        Dome at Az2 280.3°  30s    32.4     1.1      7s  308.5

  Replay slew to 103P/Hartley           2023-10-22 11:21  (Local, BST)
  |103P/Hartley                   05:25   283.7   143.8    176s   139.9    0.79 D*  60s  264.3, 247.4, 223.0, 222.4, 189.4, 144.2
  Preparing for slew 4        Ok        Dome at Az1 283.7°
  Making slew                 Ok        Dome at Az2 143.8°  70s   139.9     2.0          

  Replay slew to NGC 2708               2023-10-22 11:23  (Local, BST)
  |NGC 2708                       05:41   148.3   137.5     10s    10.8    1.14          
  Preparing for slew 5        Ok        Dome at Az1 148.3°
  Making slew                 Ok        Dome at Az2 137.5°   8s    10.8     1.3          

  Replay slew to AT2023cub              2023-10-22 11:23  (Local, BST)
  |AT2023cub                      06:01   142.1   174.1     33s    32.0    0.98 D*   6s  
  Preparing for slew 6        Ok        Dome at Az1 142.1°
  Making slew                 Ok        Dome at Az2 174.1°  20s    32.0     1.6          

  Replay slew to GCVS SY Cnc            2023-10-22 11:23  (Local, BST)
  |GCVS SY Cnc                    06:12   176.3   135.4     80s    40.9    0.51 D*  43s  
  Preparing for slew 7        Ok        Dome at Az1 176.3°
  Making slew                 Ok        Dome at Az2 135.4°  26s    40.9     1.6          

  Replay slew to Pre-Park               2023-10-22 11:24  (Local, BST)
  |Pre-Park                       06:22   136.6    90.1     57s    46.5    0.81 D*  13s  
  Preparing for slew 8        Ok        Dome at Az1 136.6°
  Making slew                 Ok        Dome at Az2 90.1°   28s    46.5     1.7    


During the actual S1157 live session
- 7 slews were recorded to have a hold-up of which 3 were significant with holds-ups of 34, 43 and 60s total duration.  
- Total hold-up time across all slews was 2.9 minutes.

During the replay (conducted in daytime hours after drying the roof flange and rotation drive wheels)
- Only 1 slew recorded any hold-up (and it was of only 7s duration). 
- Total hold-up time across all slews was 0.1 minute.

Slewing performance was clearly a lot better during the Replay than during the live session. The difference is considered to be due to the difference in the friction between drive motor wheels and the roof flange (with the friction being significantly higher during the Play followin the drying off of the roof flange & motor drive wheels).

For full disclosure it should be noted that actual dome slews were made with an open Dome Shutter, whilst the replay slews were made with a closed Dome Shutter. The impact of different roof weight distributions (open vs close shutter) on slew hold-ups is unknown at this stage.

Actions

• Create a Dome Slewing Comparison Chart.  A new 'Replay Dome Slews' facility has just been added to AstroMain 3.63.2, which successfully runs and write results to Report File. It would be most helpful if the results could be visually shown on a DomeReplay Chart that shows the new replayed slew performance on one side and the original slew performance on the other. 
  Done  2023-10-26 (AstroMain 3.63.3)

 

Issue:  Certain sessions (such as S1158 session, 2023-10-22) experience significant hold-ups during Dome Slewing. This lengthens the overall slew time, delaying the start of target imaging and potentially delaying the following targets.  In the S1158 session 62% of slews had significant hold-ups, and the total time lost to hold-ups was 45 minutes.

Background
Poor slewing performance from the observatory's Pulsar Dome had been a regular issue with dome slews to new azimuth positions being prone to hold-ups during slewing  :

• the dome roof will hold-up at one or spots with the rotation drive wheels spinning away without moving the dome, until they eventually grip and the rotation continues.
• hold-ups are typically of 5 to 60s duration, but either individually or as a collection the hold-ups can sometimes be as long as 180-200s. 
• hold-ups typically affect 10-35% of slews but can reach as high as 85-100% of slews (e.g S1122, 2023-07-15). 
• the issue wastes times and causes the imaging of targets to have to wait until the dome slew has finished.
• the issue can sometimes led to a Dome Parking failure where the park dome operation is aborted by the AstroMain's ObsManager after the slew to the Park Position  (Az 90 deg) takes more than 180s,
• the issue can sometimes led to the dome loosing some of its azimuth accuracy such that the dome azimith can be out by 2- 5 degrees when Parked, this results in the two halves of the Induction Charger to not be aligned and the Shutter Battery won't recharge following the session, unless an intervention is made to correct park the dome and resync the dome azimuth to the Park Azimuth (090 deg).  Sometimes loss in azimuth accuracy is more sudden / more evident causing sky view to be occluded by the observatory roof after moving to a new target and the final position when Parked can be as much as 45 degrees (eg S1158, 2023-10-22).
  
• the issue can sometimes appears to have been resolved either through some intervention (an attempted fix or mitigation) or just on its own, and there can then be a few observing sessions which are either completely free from or largely free from any significant hold-ups.  But later on there is typically another series of sessions that suffer from significant hold-ups.

The issue was first described and briefly investigated in Sep 2019 (see Investigation - Dome Drive Slippage, Slow Slewing & Park Dome Failure, 2019-09-01) . The issue has been logged as a "Major Issue" in 5 out of 20 sessions conducted since beginning of Sep 2023  (S1158, S1156, S1154, S1145, S1143).

The impact of lost time on the scheduled acquisition of targets in a session due to hold-ups is presently mitigated by an increase in the timing tolerance allowed for each target in the observing plan.  Provided that centering operations proceed well (good sky / plenty of stars to link), the time lost to dome hold-ups can be accomodated within each scheduled target's time slot.   If the dome slewing issue can be resolved the timing tolerance can be reduced which would allow more targets to be acquired and imaged in a night. For the S1158 session where 45 minutes were lost in total due to hold-ups this time could have been used to acquire 2 extra targets acquiring 5 x 180s main frames or 3 or 4 extra targets acquiring 3 x 60s main frames.

The following two charts show examples of good slewing performance in Feb-Apr 2023, with session S1101 (2023-02-22) where none of its 32 dome slews suffered from any significant hold-up. Similary in the S1110 session (2023-04-02) there were no significant hold-ups. (Green arcs in Charts are good (normal), Orange arcs are less good, but generally still tolerable). , Red arcs are bad (poor slew perfomance), with red spoke lines (if any, indicating azimuths with hold-ups)

 

The issue was noted to have became progressively worse during sessions in late Apr - Jul 2023, such that in session S1122 (2023-07-15) nearly every single slew suffered a significant hold-up and a total of 22.7 minutes were lost during the session because of this problem (this is particularly significant given the very short observing period associated with a short summer night). 

The two charts below show the bad dome slewing performance from sessions S1121 (2023-06-07) & S1122 (2023-07-15) in which the majority of slews suffered from significant hold-ups.   (Green arcs in the Charts indicate good (normal) performance, Orange arcs indicate less good performance, but generally still tolerable, Red arcs indicate bad performance. The red spoke lines indicate the azimuth positions where a hold-up occured, with its line-length indicating the duration of the hold-up. )

  

Following S1122 session an intervention was made to tighten the tension of the dome rotation drive system (see Pulsar Dome - Rotation Drive Tension Adjustment, 2023-07-19). Initially the adjustment appeared to have resolved the issue, the chart below shows that the following session (S1123, 2023-07-22) had very good slewing performance, with none of the 13 dome slews being affected by any hold-ups, however the next session (S1124, 2023-07-30) again had 2 slews that were affected by significant hold-ups, costing a total of 4.1 minutes.

  

Following further degradation in dome slewing performance in Sep-Oct a new tool was developed following the S1157 session (2023-10-16) that could replay the same dome slews that were made during a session, but under potentially difference test scenarios.  The following chart shows the slewing performance during S1157 session (2023-10-16) itself.  This can be compared with performance during a playback of the S1157 session slews (see below rightt) made on 2023-10-22 (daytime). This was after the dome's roof flange and its 3 rotation drive wheels had been dried of condensation and moisture and results show that the dome was clearly performing better during the replay (dry state) than during the original live session (presumed damp/wet state)

  

A second replay of the S1157 slew was attempted on 2023-10-26 (~ 23:30) (see below right). This was conducted without any prior drying of the roof flange & drive wheels. This time 4 out of 8 slews failed due to the dome monitor not receiving a Slewing = True state from the dome within 8s of the 'SlewToAzimuth' instruction (this is presumed to be due to the dome being held-up at the starting point for the slew). Out of the slews that proceeded, 3 of the 4 slews encountered significant hold-ups. (overall we can say 7 / 8 slews had significant holdups). The dome was performing worse than during the original live session.

A third play was made on 2023-10-27.  This was conducted during daytime with the dome and its roof flange is a dry state. Slew performance was good. These 3 playbacks reveal how dome slewing performance is heavily dependant on the state (dry or damp/wet) of the roof flange & drive wheels.

      

Following the initial S1157 replay during which the dome slewing performance was good, a new live session (S1158, 2023-10-22) was conducted. The initial slewing (i.e the first 3 slews, corresponding to 3 green arcs in the picture below) was good with no hold-ups but during the remainder of the session the majority of the dome slews were affected by significant hold-ups (62% of slews overall), with individual slews being held up for total of 34-226s (in fact there were 9 slews held up for more than 2 minutes). A total of 45 minutes were lost to hold-ups across the whole session.

A particular feature highlighted in this session is that hold-ups points indicated by red spokes in the picture below is distributed pretty much the whole way around the dome. (Note the azimuth between

 
 
The Observatory Dome is suspected to have a tendancy to hold-up as the dome wall is not perfectly circular but is very slightly elliptical (this is due to a vital check being missed during erection of the observatory which is difficult to rectify now as the dome is securely bolted to its concrete base). This slight ellipticity causes the dome roof flange to be distorted as it passes over the wheels in the roof wall and can cause some 'semi-permanent distortion of the certain parts of the roof flange because of the long periods of time that the dome sits at its park position. The distortion leads to varying amount of tension between the drive wheels and the roof flange, and when the tension/friction is not great enough the wheels can spin on the same spot for up to 180-240s.  Hold-ups (if any) are likely to be shorter and/or fewer when the dome flange/drive wheels are dry. There is sometimes a correspondance between the problem nights and nights with higher dew levels,  and tricks of wiping the roof flange from traces of grease, water & dirt can be sometimes be used to improve dome slewing performance. Taking out some of the distortion in the roof flange can be relieved by prior manual manipulation of the flange.   Attempting to tighten the drive tension knob generally fails to improve the situation, as the knob is pretty much as tight as it can go.

Pictures showing the roof flange are shown below.  The left hand picture shows a normal section of the flange whilst the right hand section shows distortion over a particular wheel when the dome is parked.

Normal Section The more shiney strip is where the innermost drive wheel is in contact with the flange as it passes by.	Abnormal Section When dome is parked, the flange is near the door is distorted  over horizontal guide wheel on the wall

History
There have been incidents of slow dome slewing or failure to slew/park going back over 4 years. For example this note from Session 688 (2019-09-01) in Sept 2019

Upon attempting to park the dome at 05:30 in the morning and after the Dome had been stationary for more than 2.1/2  hours, it was found the rotation motors wheels were unable to get sufficient leverage to physically rotate/drive the Dome Roof. The motor wheels would slip against the roof flange which was quite wet with dew by this time.   The tension wheels had to be adjusted in order for the wheels to get sufficient grip and move the dome.

The dome was noted to taken a long time (372s or 6.2 minutes)  for it to to park at the end of another recent session (2019-08-28, S686)

The Rotation Drive Tension was last adjusted on 2023-07-19, when included getting acces to the nut attaching the drive tension bolt to the metal plate that holds the single drive motor that turns the drive wheel against the inside of the roof flange, there was no mention or recollection that the drive wheel was loose or that there were any aluminum filings/shavings. It is presumed that it was simply a case of them being unoticed since it is hard to imagine that the wheel was securely attached to the drive at this time and that there were no metal shavings. 

The following photo shows the rotation drive system before its installation in the observatory in 2018. The two rear wheels are for driving the back of the dome's roof flange whilst the single front wheel is for driving the front of the roof flange.

Rotation Drive Wheels (blue) & Encoder Wheel (grey) The two rear wheels are for driving the back of the dome's roof flange whilst the single front wheel is for driving the front of the roof flange.		Side view looking at the Rotation Drive System  Single Blue Wheel drives the front (innerside) side of the dome's roof flange. Two other wheels, which are hidden, drive the rear (outside) side of the roof flange

Analysis
It is clear that at the times of a dome hold-up that

• the drive motors wheels are still turning
• the drive motor wheels are spinning against the inner and outer surfaces of the roof flange. 
• the dome can be made to rotate again by pushing the dome slightly in direction of travel until it begins moving again on its own
• eventually the dome generally resumes moving again on its own accord

At the time when the dome is held-up the force being applied to the roof flange is insufficient to overcoming the roof's weight and the frictional forces that otherwise keep the dome roof stationary.  Dome hold-ups may be due to one or more of the following fundamental changes.

a) an increase in the frictional forces that tend to make the dome stationary
b) a reduction in force that drive wheels are able to intrinsically apply
c) a decrease in the the friction between drive wheels and roof flange (and thus a reduction in the force that drive wheels actually impart on the roof flange)

There is no evidence suggesting that the force that the drive wheels can intrinsically exert is changing during an individual slew, or changing between slews or between sessions. (Whilst there was no evidence at the time of this analysis, later on it was discovered that the inner drive wheel wheel, norminally hidden, was loose and wobbled on its motor shaft (see Pulsar Dome - Loose Rotation Drive Wheel, 2023-10-27). This must be limiting the force that this motor can apply to the roof flange)

Potential responsible factors
The potential factors that could be at least partially responsible for the hold-ups are as follows :-

• A weakening of the tension in the rotation drive system.
  The intrinsic force applied by rotation drive wheels could theoretically decrease due to some factor or change. It is possible to increase the tension (and thus increase the force applied by the drive wheels) by adjusting the relatively positions of the inner and outer motors . But increasing the tension too much could theorerically itself cause the dome to stick and hold-up.   Following session S1122 (2023-07-15) the dome tension was adjusted. Whilst slewing performance seemed to be much better in session S1123, 2023-07-22 but it generally deteriorated again thereafter or proceeding with a mixture of sessions having poor performance and sessions having good/tolerable performance. 
  
  (Later on it was discovered that the inner drive wheel wheel, norminally hidden, was loose and wobbled on its motor shaft (see Pulsar Dome - Loose Rotation Drive Wheel, 2023-10-27). This must be limiting the force that this motor can apply to the roof flange. The connector was found to be worn and it seems that whilst the two outermost drive wheels are spinning on the same spot the innermost drive wheel is semi-stationary whilst the shaft of its drive motor was spinning within the wheel to motor aluminium connector.  With the connector loose and with the wheel flopping around on the shaft it is noted to be difficult to get a tension that works (ie one that is not too slack and too tight).   
   
• A non -circular dome wall
  In plan view the Observatory is believed to be not perfectly circular but is very slightly elliptical (this is due to a vital check being missed during erection of the observatory which is difficult to rectify now as the dome is securely bolted to its concrete base). This slight ellipticity causes the dome roof flange to be distorted as it passes over the wheels in the roof wall and can cause some 'semi-permanent distortion of certain parts of the roof flange because of the long periods of time that the dome sits at its park position. The distortion might lead to varying amounts of tension between the drive wheels and the roof flange and might cause changes to the intrinsic forces that would otherwise tend to make the dome stationary.  The fact that some sessions experience no hold-ups during slewing suggest this is not the dominant factor.
   
  Watching the dome rotation later with the inner drive wheel visible (induction charger removed) it is noted that in certain sections the roof flange would 'pull away' from the drive wheel, meaning that drive to roof flange would only have being coming from 2 wheels.
  
  The following highly exaggerated illustration shows that ensuring diagonals (lines A1 and A2) were of equal length (as was done during erection of the Observatory) does not automatically mean that the top to bottom and side to side lengths (lines B1 and B2) are equal.  I
  
  
• An open shutter
  The force exerted by the combined weight of the dome roof and its shutter will be distributed somewhat differently depending on whether the shutter is open or closed, however it is unknown whether an open shutter increases or actually decreases the potential for slew hold-ups.  Apart from Dome Parking, dome slews are almost always made with the shutter open.  The fact that some sessions can experience no slewing hold-ups indicates that an open shutter is not a significant factor in the amount of and intensity of hold-ups.   Even if dome slewing happened to be better with the shutter closed, it is not really feasible to close and then open the shutter afterwards just to accomodate a dome slew.   The operations to close and open the shutter again afterwards would by themselves add around 100s to the slew time, and of course some slews have no hold-up and when there is a hold-up it is typically shorter than 90s.
   
• A decrease in the grip between rotation drive wheels and the roof flange due to condensation/dew.
  The presence of a lubricant between drive wheels and the roof flange,  likely to be water from condensation and dew, would clearly lead to a reduction in the force than the drive wheels can apply to the roof flange.  Because of this, either by itself or in combination with changes in the force needed to move the dome because of its geometry, the wheels slip on the roof flange to a hypothesied point that enought moisture/water has been thrown off the wheels/roof flange and rotation begins again.

After consideration it is judged that changes such as a) increasing the tension in the rotation drive system, b) retrospectively improving the geometry of the dome or c) closing the dome during slewing have either been tried already (a), are difficult or not feasible (b) or are unlikely to provide the amount of improvement needed (c).

To resolve the dome slewing issue, or to at least significantally moderate it, the most important action is to increase the grip between rotation drive wheels and the roof flange (by either removing water/condensation lubricant or nullifying its impact)

Potential Solutions
Potential solutions to removing condensation/dew from the roof flange are:

• Attach sponge foam or other material at one or more points to wipe water from the roof flange as it passes by.
  
  This was tried out in the S1158 session, a foam sponge was attached to one of the dome's roof clamps that was pressed against the roof flange and wiped it as the dome rotated. There is no indication that it produced any significant benefit, as the S1158 had particularly bad guide performance with significant hold-ups affecting 62% of slews. One risk already considered for this idea was that the sponge would wipe water along the flange or reapply it and it may not 'soak' up or otherwise remove the water by forcing it to be squeezed out and drop to the floor.  Alternatively if sponge stayed wet it would spread water to a dry flange at next session.
   
• Use one or two 5v/12v computer fans to blow air onto the rotation drive wheels and adjacent roof flange in attempt to dry or blow off water drops from their surfaces , thereby reducing or controlling the amount of lubricant present.   This air movement wouldn't effect other sections of the roof flange would could still be accumulating condensation/dew, and upon a slew move towards the drive wheels at 2 deg/s.  The risk is that the fan's air flow is simply not enough to dry the approaching roof flange in time for it to give a significant enough improvement.

Potential solutions to nullify the impact of condensation/dew

• Replace drive wheel on the inner drive motor with a cog and apply a corresponding cogged strip to the roof flange and retrospectively convert the rotation drive system to a direct drive one.  Assuming it was possible to secure a suitable sized cog/cogged strip system there are quite a few uncertainties/risks.  There may not be room for cogged strip to pass by the dome induction charger support or pass the dome's 4 roof clamps.  There is also the risks that the strip can be permanently afixed to the roof flange, and that the cog/cogged strip seizes up as roof tries to move pass certains point/sections due to dome geometry issues mentioned above.  This idea is considered to be a non-starter based on current considerations
   
• Apply anti-slip tape to the roof flange to increase grip and nullify the impact of any water presence. In practice this would be applied to the inner surface of the roof flange due to the difficulty to apply tape to the outer side due to limited space between roof flange and the observatory wall.   In theory this increases the grip between the inner drive wheel and the roof however much dewing there is.  The uncertainty is how well the tape adheres to the roof flange and whether it separate in future or it stays in place.

Most Promising Solution
The most promising of these potential solutions is judged to be the application of anti-slip tape to the dome's roof flange.

Conclusion

• Presence of condensation/dew acting as a lubricant is the most significant factor involved in the dome slewing issue. 
   
• To resolve the issue, or to at least significantally moderate it, the most important general requirement is to
    increase the grip between rotation drive wheels and the roof flange
   
• Of the potential options identified to achieve this the application of anti-slip tape to the roof flange is considered to be the most-promising one, as it should nullify the impact of water as a lubricant whatever the dewing conditions.  For the 2.2m diameter dome the length of tape needed is estimated to be 6.9m  (from 2 * Pi  (3.45575) * 1.1)
  
  (The 5v/12v computer fan(s) option was being considered  but will be placed on-hold for the time being as a backup. The use of a form of sponge/material to dynamically wipe the roof flange and the retrofitting of a form of direct drive system are both dropped as options).
  

Actions

• Identify and source a reel Anti-Slip Tape. Acquire ant-slip tape that will be suitable for exterior use, at least 7m long and will attach ok.
  Done 2023-10-26   (see secton below Anti-Slip Tape for Improving Observatory Dome Rotation, 2023-10-26)
   
• Apply anti-slip tape to the Dome's roof flange to improve slewing performance. Apply anti-slip tape to the Dome's roof flange to increase grip and nullify the impact of water presence under dewing conditions that is most-likely causing holds-ups and poor slewing performance. In practice the tape would be applied to the inner surface of the roof flange due to the difficulty in applying the  tape to the outer side due to limited space between roof flange and the observatory wall.   In theory this tape could increases the grip between the inner drive wheel whatever the dewing conditions. There is uncertainty over how well the tape adheres to the roof flange over the longer term and and whether it separates in future or it stays in place.
  Done  2023-10-27 (Observatory, Pulsar Dome)
   
• Make series of measurements that record the relative shape differences between the dome wall and the dome's roof.  Idealy one would measure various diagonal lengths across the observatory at wall and roof level but this is not possible due to the telecope/pier in the centre of observatory. Propose to instead to determine the relative shape differences between the dome wall and the dome's roof . With dome parked measure the roof flange to observatory wall distance at 16 positions around the dome (essentially at N, NNE, NE,ENE, E etc). Then move the dome to say Az 1355 deg and repeat measurements, Move to Az 180 and repeat measurements etc.
  To do  2023-10-29+ (Observatory, Pulsar Dome)

Update 2023-10-27
The reel of Anti-Slip Tape has now been collected from Screwfix (2023-10-26) and successfully applied to the dome's roof flange today (2023-10-27). It was relatively easy to attach and it seems pretty secure (see secton below Anti-Slip Tape for Improving Observatory Dome Rotation, 2023-10-27). The length of tape required to go completely around the dome flange was 6.18 m, which was shorter than the possible 6.9m length estimated based on the Dome's 2.2m diameter.

Afterwards, in the course of making checks to see if an adjustment to the rotation drive tension was needed, which involves removal of the wall side half of the induction charger that normally hides the inner motor drive,  it was noticed that the drive wheel was loose and there where indications of aluminum filings in the vicinity of the motor drive (See Pulsar Dome - Loose Rotation Drive Wheel, 2023-10-27). Investigation showed that the inner drive wheel, and the associated aluminium cylinder that it screwed to it,  was only loosely attached to the motor's drive shaft and had a tendancy to wobble when the motor turned.  Wear to the hole in the bottom of the aluminium cylinder and aluminium filings point to a probabe extended period of damage and wear due to the loose fit.   This is clearly another (previously unknown) factor in explaining why the drive system's grip on the roof flange is at times insufficient to rotate the dome.

The observatory has had an ongoing issue (2019 to 2023) whereby during some sessions the dome will experience poor slewing performance with some slews being held up whilst the rotation drive wheels spin on the same spot on the dome's roof flange without moving the dome. Typically the drive wheels gradually nudge forward to they eventually grip and the rotation continues but they can in some cases be a hold-up of up to 120-180s. The issue has been recently investigated (see Investigation - Hold-ups during Dome Slewing, 2023-10-25). The study & analysis concluded that Condensation/Dew, acting as a lubricant, reduces the force that the rotation drive motors can apply to the roof flange and is the number factor responsible for the significant hold-ups but is in part due also the the domes' 'not-perfectly circular' geometry The investigation went on to propose the application of Anti-Slip Tape to the inner surface of the roof flange to increase the grip from the innermost drive motor wheel. 

Whilst the root cause to slewing performance issues was eventually found and fixed, and the Anti-Slip Tape was eventually removed due to later issues the following description regarding the tape is left as a record.

Tape
A source of Anti-Slip Tape (18m x 50 mm) was found at Screwfix and ordered (2023-10-25).
https://www.screwfix.com/p/anti-slip-tape-black-18m-x-50mm/57217

       
The tape is meant to be
- anti-slip
- easy to apply
- for Internal & External Use
- self-adhesive
- durable

- at 18m long it is more than long enough to go around the roof flange's 6.9m circumference.
- at 50 mm wide tape is wider than necessary (the drive wheels are only 20mm wide or so), but it will be difficult to cut it down to a narrower width and in any case the wider width might allow the tape to stick to the roof flange more securely.

Collection
The tape was collected today (2023-10-26). 
 
Checking (2023-10-26 pm) confirms that the roof flange can just about accomodate the 50mm width tape ok

Installation Plan
The tape will be applied principally to the inner surface of the roof flange

Fitting to the outer (hidden) surface of roof flange around the whole circumference of the dome would be be very difficult due to limited space between roof flange and the observatory wall with the tape being difficult to control as it will be prone to sticking to the first surface it touches.  Despite this an attempt will be made to apply a short section of tape on at least the outside of the flange along the section that lies in the vicinity of the rotation drive when the dome is at it's Park azimuth (90 deg). This woud help allow the dome to moved off it's Park position when commencing a new session, and not hold up any time passing the point when the dome spends most of its time whilst parked and the forces to rotate are notionally at their highest.

Longer term there is uncertainty over how well the tape adheres to the roof flange and whether it separates in future or it stays in place.

Installation
The anti-slip tape has been successfully applied to the dome's roof flange today (2023-10-27). It was relatively easy to attach and it seems pretty secure. The length of tape required to go completely around the dome flange was 6.18 m, which was shorter than the possible 6.9m length estimated based on the Dome's 2.2m diameter.

In the course of a make an adjustment to the Dome's Rotation Drive tension, which involves removal of the wall side half of the induction charger that normally hides the inner motor drive, it was noticed that the inner drive wheel was loose and there where indications of aluminum filings in the vicinity of the motor drive (see Pulsar Dome - Loose Rotation Drive Wheel, 2023-10-27).  Investigation,  showed that the inner drive wheel, and the associated aluminium cylinder that it bolts to,  was only loosely attached to the motor's drive shaft and had a tendancy to wobble when the motor turned.  Wear to the hole in the bottom of the aluminium cyclinder and aluminium filings point to a probabe extended period of damage and wear due to the loose fit.   This is clearly another (previously unknown) factor in explaining why the drive system's grip on the roof flange is at times insufficient to rotate the dome.

Pictures showing the roof flange before and after applying the anti-slip tape are shown below.

Roof Flange (Before)	Roof Flange (After) Showing flange after application of Anti-Slip Tape

Consideration will be given to attempt to apply tape to the outermost (hidden) side of the roof flange. Whilst the tape was easy to apply to the inner (accessible) side of the flange, appying the tape to the other side will be a lot more difficult.  However the outermost side is naturally smoother and prone to be more slippery that innermost surface was.   The Dome slewing performance after fixing the loose rotation drive wheel will also need to be factored in.

Update 2023-11-17
Although the primary cause of hold-ups during dome slewing was ultimately found to be due to a loose drive wheel connector which was replaced today (2023-11-17), the anti-slip tape will be kept in place and it is thought to be an extra aid to ensuring smooth rotation of the dome.

It was however noticed that the anti-slip tape is coming away from the roof flange in a number of places. These places were mainly at and around the joins between the 4 roof quadrants where the either side of the roof flange flexes as it passes across the 8 horizontal wheels at the top of the observatory wall that keep the dome roof centralised as it rotates.  Loose areas of tape where stuck down with glue and held in place for 3 hours with clamps.

The Dome's Rotation Drive tension was checked following the installation of Anti-Slip Tape in case it needed to be adjusted to allow for the thickness of tape. This involved the removal of the half of the induction charger that sits about the Dome's rotation drive unit and normally hides the inner motor drive. When this was down it was immediately noticed that the drive wheel was loose and there were indications of aluminum filings in the vicinity of the motor drive.

Photos and description follow :

The Dome's Rotation Drive tension was checked today and adjust to allow for the earlier attachment of Ant-Slew Tap to the roof flange. After removal of the half of the induction charger that is attached about the rotation unit , that normally hides the inner motor drive, I came across the problem.

• it was noticed that the drive wheel was loose and there where indications of aluminium filings in the vicinity of the motor drive.
   
•  removing the wheel from the shaft, a pile of aluminium filings were revealed along with a loose 'pin' of some kind. The filings point to a probable extended period of damage and wear due to drive wheel connector/
   
    
   
• After cleaning away the metal filings, the motor shaft was examined and it looks ok with no significant sign of wear or damage (I think).
  I assume that the hole in the shaft is normal, but I don't know for sure as I don't know where the loose 'pin' has come from
  
  I've checked and the motor itself is operating/turning ok.
  
  
   
• examining the rubber wheel, it was found to be securely attached to the aluminium cylinder and the grub screw was present in the cylinder
  
  
  
  
• examining the hole in the bottom of the aluminium cylinder (where it attaches to the motor shaft) it was noticed to be very worn and somewhat elliptical. This wear seems to be due to shaft rotation within the hole. There is also wear around the outer edges of the cylinder base that seem to be due to rubbing of the cylinder against the 3 small bolts that attach the motor to its supporting arm.
   
  
   
• looking into the hole these seem to be small hole on the inside that appears to correspond with the line of the grub screw but it seems smaller diameter than the grub screw, and has a rough edge that might potentially be the site of attachment of the 'pin' that was found when the wheel was removed from the motor
   
  
   
• removing the grub screw shows it to be intact and seemingly normal. Once removed and looking in the tiny hole there is no clear line of sight to the centre of the cylinder, there seems to be something looking like another grub screw or something beyond ?
   
  
  
  It is unclear to me what the original mechanism was for holding the wheel+cylinder to the motor shaft. ?
  As far I can see the grub screw doesn't seem able to have locked against the motor shaft (by pure friction) or projected into the void where the flat side of the shaft is (forcing a connection), the grub screw it too short to reach, unless screwed a very long way in but its path to do this is blocked.
  
  It's not clear if the problem relates to an original flaw in the part or its just wear and tear. Either way I need a new aluminium cylinder. Its assumed that the rubber wheel (which seems ok), can be un-screwed from my failed cylinder and put on a new one, but I haven't tried to remove it.

Loosely fitting the wheel/cylinder back on the drive motor again and doing some try slews shows the system is partially working but is still very prone to hold-ups where the dome stops turning for 5-30s before resuming again. It seems the 2 outermost motors are driving the roof and rotation is transferred to the single wheel on the inner side and can cause the shaft to spin away within the 'failed' connector. With the wheel flopping around on the shaft its difficult to get a tension that works (ie one that is not too slack and too tight).   

This is clearly another (previously unknown) factor in explaining why the drive system's grip on the roof flange is at times insufficient to rotate the dome.

Previous History

This drive wheel was previously noted to be loose in Jan 2019 (see notes Pulsar Dome - Loose Drive Motor Head, 2019-01-18)

 "whilst the replacing the encoder arm spring on the rotation drive unit it was noticed that head of one of the rotation driver motors was somewhat loose and its grub screw was partially unscrewed/loose. Opportunity was taken to tighten the grub screw and ensure that the motor head was secure.  It was fortunate that the problem was caught before it caused a complication during observatory operation".   2019-01-18.
 

Close Up of Drive Motor Head Photo : 2019-01-18 showing the grub screw that was loose (picture taken after grub screw was tightened)

Note that there is no evidence of any metal filings at this time.  

Problem of slow dome slewing/slippage noted & investigated in Sep 2019 (see Investigation - Dome Drive Slippage, Slow Slewing & Park Dome Failures, 2019-09-01)

Given that it has been shown during the current (Oct 2023) instance of the problem that the grub screw cannot possibly lock onto the motor's drive shaft, it can be assumed that the observatory has been running with this drive wheel being loose for over 4 years.  This extended period can certainly explain the significant wear on the drive wheel connector and the amount of aluminium filings  since 2019.

Actions:

• Contact Pulsar Dome Observatories in regard to sourcing a replacement for the failed/worn aluminium cylinder.
  Pulsar Dome Observatories have been contacted by email today (2023-10-27 pm)
  Done 2023-10-27 (Request Help). 
   
• Adopt a short-term solution for the loose Rotation Drive Wheel until a replacement connector arrives and is fitted .  Options :
  - Add tape around motor shaft to make the wheel fit more securely
  - Drill a new hole through connector in place of the grub screw and insert and secure a long 'pin' that goes through to the hole in flat side of the motor shaft
  - Combination of the two.
  Done 2023-10-31 (Observatory, Pulsar Dome).
  
  A completely new hole was drilled through the connector (preserving the original hole with its grub screw guide). Pin inserted into new hole and into the hole on the flat side of drive shaft. Trying to add tape around the top of the motor drive shaft was unsatisfactory and removed. Drive Motor tensioned adjusted and slewing tested.   The drive wheel still wobbles a lot but can no longer spin on the drive shaft and should no longer wear.
  
  To complete the test Dome Slews from Session S1157 was replayed. This was satifactory (see below) and the Pulsar Dome's Rotation Drive system is considered sufficiently ok to conduct new nightime sessions.
  
  
   
• Fit replacement Drive Wheel and Connector when then arrive from Pulsar Observatories.
  Done 2023-11-17 (Observatory, Pulsar Dome).  See Pulsar Dome - Replacement Rotation Drive Wheel, 2023-11-17 )
  

Update 2023-11-08

Whilst the bush-fix to drive wheel connector has held up well during sessions S1159 - S1162 (e.g. see below left) , the dome slewing issues suddenly returned during the second part of the S1163 Session (2023-11-08, see below right).

    

Update 2023-11-09
A check the following morning (2023-11-09) showed that the metal 'pin' that has been used in the bush fix had pushed itself out through the holding tape allowing the drive wheel to again spin on the motor's drive shaft. The pin was re-positioned, re-taped and the drive tension re-tightened.  Tests show that drive motors are gripping and turning the dome ok.

The repaired bush-fix was operating during the S1164 Session (2023-11-09) and the Dome Slewing Performance appears to be good.

Update 2023-11-08
Whilst the bush-fix to drive wheel connector has held up well during session S1164 slewing performance again suddenly degraded part way through session S1165 (2023-11-10) for Target 7/36 to Target 9/36 taking 217-240s and in some cases not reaching the target.  Issue was noticed during remote monitoring and after halting the dome slews and halting the session a visit was made to the observatory in order to re-fix the loose drive wheel, which is now known to be principal cause of poor slewing performance.  Exposing the drive wheel showed that the earlier bush fix on 2023-11-09 had failed. After re-fixing the issue the session was re-continued. 

The repaired bush-fix was operating ok during the remainder of the S1165 session and the during the S1166 session  (2023-11-12).  Full resolution of the issue requires installation of a replacement drive wheel connector but this is still waiting on a delivery from Pulsar Observatories. 

Update 2023-11-15
Repaired bush-fix failed again part way through session S1167 and once noticed with the dome stuck on Az 222° the session was prematurely ended as it was too late and too cold to go out to redo the bush-fix.

Update 2023-11-17
Replacement Drive Wheel and Connector eventually arrived from Pulsar Observatories this morning and was successfully installed in the Observatory.
(See Pulsar Dome - Replacement Rotation Drive Wheel, 2023-11-17 )

Issue:  Dome Relay Protection stopped working (2023-22-07 14:52).  

The observatory's Dome Relay feed's directly to the Dome's Controller Unit. The relay is meant to provide 'fail-safe' protection for the observatory that will directly command the dome's shutter to close 30s after the observatory's CloudSensor (rain, clarity & light) changes from 'Safe' to 'Unsafe' state.   This is a third-level protection that is is intended to kick-in in the event that AstroMain and AstroGuard have both failed to close the Shutter for any reason such as a program crash or communication failure.

The issue arose because the Dome Replay began indicating a 'Safe' state over a protracted period, whilst Cloud Sensor was indicating an 'Unsafe' state, meaning that the Dome Relay would no longer have closed the Shutter in the event that it began raining or snowing casuing damage to the observatory and equipment.

Description
Whilst demonstrating the observatory prior to session S1187, 2023-10, the dome' shutter was opened using DeviceHub (AstroMain wouldn't allow opening due to 'critical light conditions')  at 17:57:20 with the expectation that the CloudSensor to Dome relay would kick-in and the Dome's Drive unit would command the Shutter to close 30s later, but it didn't automatically close and was closed instead by a manual command from DeviceHub at 18:01:34.  

The ObsPic shows that the Dome Relay setting (obtained from the active Pulsar Log file) was indicating 'Safe Conditions' even though the Cloud Sensor was reporting Light value of 27 which is critical and would have had the relay in an 'Unsafe' state.  Investigating ObsPics for earlier in the day shows that Pulsar Dome was correcting seeing an 'Unsafe' relay state up until 14:52 when it started to see an apparently 'Safe' relay state.   This corresponded to a time when Observatory was being visited and the Observatory door was briefly opened for 8s. It isn't known (from memory or log records) what activity was conducted at this time. 

ObsPic pictures the following morning show that Pulsar Dome was still showing a 'Safe' Relay indicating that the S1158 session was running without a working Dome Replay Protection (3rd level protection) and is why this issue is ranked as a 'Major Issue' . The Pulsar Dome continued reporting a Safe Relay condition until 2023-10-24 19:18.   Records to not show an activity on the Observatory Computer that might explain this change at 19:18, so is the something that happens on the AllSky/Weather Computer ?

Pulsar Dome was similarly reporting a 'Safe Relay' on the morning of 2023-10-27 even though light conditions were critical (daytime) and this lasted until 11:26 when it changed to reporting an 'Unsafe' Relay state.  Records show that AstroMain program was closed at 19:51 and reopened at 10:53 including Starting Dome Service and the opening of new Pulsar Log File (ASCOM.Pulsar_Observatories_Dome.1054.118990.txt) but Pulsar Dome continued reporting Safe Replay until 11:12 (At 10:54 Dome Service was started . At 11:04 was opened and a Test Session was commenced at 11:12 in order to Replay Dome Slews from the earlier S1157 session.    At this same time (11:12) and until 11:20 the ObsPics still show the Pulsar Dome reporting a 'Safe' relay despite critical daylight and cloud conditions.  At 11:21 clarity falls below 20 and the relay changes to an 'Unsafe' state, before changing back to 'Safe' for a few momenets and then back to Unsafe again at 11:39. The Relay seems to be responding to 'Clarity'  though there is a delay between Pulsar data and Cloud Sensor reporting (or vice versa ) but not to 'Light' ? Examining CloudSensor's Sensor Head show that is enabled to set relay to 'Unsafe' when Clarity falls below 20, Rain rises above 2 or Light rises above 20. Examining CloudSensor data record for 11:15 (10:15 UT) shows Clarity 17.7 and Light 32.2 that should, on both counts, have the relay in an 'Unsafe'. However the ObsPic file and the Pulsar Log file both show the Pulsar Dome interpretating the Replay as 'Safe'   Almost like the Dome is interpreting the opposite state to that intended by CloudSensor III ?.  

Dome.Relay value is read from the 12th field in the 'volatile' records shown in Pulsar Ascom Logs. 0 indicates 'Safe' or a momentarily glitch, 1 indicates 'Unsafe'.

It is currently showing 1 (Unsafe) at 2023-28-15:0 (daylight + rain + cloud).

On 2023-10-22 the Dome Relay was showing 1 at 14:50 (correct) , but 0 at 14:54 (incorrect as it should be 1 according to CloudSensor intention) with the change happening at 14:52:44). It was still showing 0 ('Safe') at 18:01 (incorrect) when the shutter was opened and didn't automatically close as expected based on the Cloud Sensor's Light value (27). 

The Dome Relay settings (see screen capture below (captured 2023-10-28) ) were checked in the Pulsar Dome Program on 2023-10-28 and are confirmed to be correct.   Note that 'Polarity' is set to the 'NO' option for it to get the replay state intended by the observatory's CloudSensor (This is to get around the observation that CloudSensor and Pulsar Dome use opposite terminologies for open & closed).

Analysis
Questions to be answered are:

• Why did Dome Relay begin reporting a 'Safe' state when Cloud Sensor was reporting an 'Unsafe' state ?
   
• Why did Dome Relay begin to correctly report 'Unsafe' state again ?
   
• What can be done to raise alerts when the Dome Relay is reporting an incorrect state and that relay protection of the Observatory has stopped working
   
• What can be done to ensure that Dome Relay Protection is always working correctly (i.e. that it never breaks or fails)

Conclusion

• Reason for Dome Relay Protection to stop working and to later resume are unknown.
  
  It is hypothesised that it happened due to some glitch in the Pulsar Dome system or drive.
   
• A Dome Relay Protection monitor should be added to AstroMain to raise an alert if Relay Protection is deemed to have stopped working to advise observatory user.
   
• In event of Dome Relay Protection Alert,  the Dome should be restarted using 'to determine if this is a reliable means to resolve the issue.   This might give confidence to go ahead and allow AstroMain ObsManager to automatically conduct a Dome Restart (including power cycling).
  

Actions

• Add a dome relay monitor to AstroMain that will raise an alert if the Dome Relay state appears to be incorrect. This monitor should raise a Red Alert if the Dome Relay shows 'Safe' whilst Cloud Sensor is shows 'Unsafe' , a Yellow Alert if Dome Relay shows 'Unsafe' whilst the Cloud Sensor show's 'Safe'. A Green alert should be raised following a an earlier red or yellow alert if the Dome Relay state again matches the Cloud Sensor state.  A continuous improvement ticket has been raised. In due course consideration should to be given to whether the Dome shouldn't be opened if a Red or Yellow alert is raised & still active, but only after extensive trialling of the Dome Relay Monitor.   Dome Relay Monitor added to AstroMain 3.63.4. Several attempts were needed before it worked satisfactorily without producing false alerts.
  Done 2023-10-31 (AstroMain 3.62.4)