Calibration (Profile TS80/178MC)

PHD2 Calibration performed for Profile 'TS80/178MC' on a starfield close to Dec 0 using LX200 Mount with guide rate x0.90.

(This was required as previous Calibration had been lost when the Operation System and Software (including PHD2) on the Observatory Computer had to be reinstalled following an otherwise unresolveable boot access failue on 2022-06-16, details)

Notes

• Calibration chosen to assume axes are orthogonal
   
  (RA-Dec axis are noted to be less orthogonal than expected.  Why ?)
   
  
• Ra/Dec rates were accepted as measured
   
  RA Rate (14.574 a-s/sec) is > than the expected RA rate (13.5 a-s/sec)
  Dec Rate (12.938 a-s/sec) is < than the expected Dec rate (13.5 a-s/sec)
   
  (Ra & Dec rates differ from expected Rates. Why ?)

Results

Guiding results from session using new PHD2 Calibration in terms of overall guiding performance and individual guide runs were acceptable

Actions

• Create a Bad -Pixel Map
  
• Export Profile to a disk file as a backup in the event of any future re-installation of PHD2
  
• Perform Calibration for LX200/ST10 profile, create bad-pixel map and export profile as a backup.
  

DeviceHub was updated on Observatory Computer today (2022-06-29) from version 6.6.0.12 to new 6.6.0.13 version. 

It is hoped that this version with get around the issue that Pulsar Dome doesn't set Slewing=True until some 0.7-0.8s after receipt of a SlewToAzimuth command.  Current 6.6.0.12 version goes to normal polling (every 5000ms) after seeing Slewing=False from the first Get Slewing poll after beginning SlewToAzimuth.  The new 6.6.0.13 version will hopefully stay on fast polling (every 1000ms) for 3s and after eventually polling Slewing=True command will then stay on Slewing=True until the end of the slew.

6.6.0.13 Release Notes
- For all devices, extend fast polling for 3 seconds after completion of a slew..

Activity Log
After installation of the new version the Activity Log Capacity value (which was reset to its standard default value (125000) during installation ) was again increased ten-fold (1250000) by editing the DeviceHub configuration file (see notes DeviceHub 6.6.0.10, 2022-06-02 )

Results
Daytime testing of DeviceHub 6.6.0.13 showed that whilst fast polling is extended for 3s after completion of a slew the change hasn't resolved the issue whereby short dome slews (<5 deg Az change) and first part of longer slews (first 5s) are not being seen by the Dome Client.  

It looks likely that I will need to contact Pulsar Observatories to ask for an ASCOM driver update for the Pulsar Dome that honours the expectations of Asynchronous operations described in  https://ascom-standards.org/COMDeveloper/Async.htm. In particular it needs to correctly set Slewing Completion Property (Dome.Slewing) to True when theDome.SlewToAzimuth method is called (or otherwise raise an exception).  Currently Dome.Slewing is still False for up to 1s after calling Dome.SlewToAzimuth which can be interpreted by the client during this time as the dome has reached its new position and it can continue with the next task (such as Imaging).  This happens because the dome driver waits for information in its next poll of the hardware to see that dome is moving before it sets Dome.Slewing=True.

>> Device Hub Releases :  https://github.com/ASCOMInitiative/ASCOMDeviceHub/releases

Update 2022-07-30 Senior member of ASCOM Support advises against trying to work round the issue in Device Hub because this is really a bug in the dome driver. They recommend to point the issue out to Pulsar Observatories and get them to fix it.

Update 2022-07-02.  Pulsar Observatories have been contacted with regard to the Pulsar ASCOM Driver, the issues using it and the potential to get it updated to bring into line with current ASCOM programming standards for domes.

Outside Temperature/Humidity Sensor readings were noticed to have flat-lined on weather/environment graphs, and data still hadn't come back online after 36 hours. 

Low battery level was suspected and was confirmed today (2022-07-02) by the absence of any digits on the small LED screen on the Oregon Scientific sensor.  Batteries (2 x AAA) were replaced. 

It was hoped that Base Station might automatically pick up the new data from the sensor but it didn't.  The issue was resolved by resetting the Weather Station's  base station,  resetting the USB Comms Hub / Link between AllSky/Weather Computer and Base Station and by restarting VWS and AstroWeather.

The Observatory's Dome (a 2.2m Pulsar Observatories Dome) is normally operated via an ASCOM Driver that was supplied with the dome in 2018. It's not known whether the driver has been updated since then or not, so the comments below are entirely with reference to the originally supplied driver (“Pulsar Observatories ASCOM Setup.msi”) which has a last modified date of 2017-12-15.

Whilst the driver has been successfully used the driver since 2018 it’s not been without the employment of various workarounds in my observatory control program when it comes to monitoring dome behaviour during slewing and shutter operation. In the current system DeviceHub (an application supplied with the ASCOM installation) provides automated slaved operation of the Pulsar Dome based on the telescope's position / target). This mostly works but proper ASCOM based control of the dome can’t be fully achieved without a fix to the Pulsar Driver that would bring it into line with current ASCOM programming standards for domes.

Dome.SlewToAzmiuth
The Pulsar ASCOM Driver tries to follow ASCOM best practice of operating Asynchronously such that on receiving a call to say Dome.SlewToAzimuth, the driver initiates the slew process (unless there was any reason to throw an exception) and then returns immediately (in practice this is a time of around 5-40ms) which leaves the client program to do other things in the meantime (this is as opposed to Synchronous operations, an older & less favoured style of programming, where a Dome.SlewToAzimuth call only returns when the dome has reached the new azimuth position).

When the operation is asynchronous and the client program needs to know that the dome has reached its new position, it checks the ‘completion’ property, which in this case is the Dome.Slewing property. If it is False then the client program can proceed with its next task - typically imaging (of course this will be once the telescope has also reached its new position). If Slewing is still True, then the client program needs to continue waiting until Slewing eventually becomes False (unless there is some issue during the slew and an exception if thrown). That’s how it is meant to work...

... but to work correctly the client program expects to see Dome.Slewing=True immediately after a call to Dome.SlewToAzimuth.

[ see the ASCOM Standard on Asynchronous Programming (https://ascom-standards.org/COMDeveloper/Async.htm) regarding this expectation ]

The present Pulsar driver violates this fundamental rule and doesn’t return Slewing=True immediately after a call to Dome.SlewToAzimuth. Instead it returns Slewing=False to the client program for up to 1s after a call to SlewToAzimuth(). The driver only begins to return True after it receives a fresh set of states from the dome hardware - this is taken from dataset that is written to driver's ASCOM Log file as a ‘Volatile’ message line.   Apparently this is a common mistake for driver developers in that it exposes its internal state rather the necessary external facing state.

Whilst the slew monitoring routines in the observatory's control program (AstroMain) can recognise and workaround the idiosyncrasies in the Pulsar driver behaviour and make additional checks to ensure that imaging can’t begin immediately that a Dome.Slewing=False is seen from the Dome, the monitoring of a dome slew (used to automatically spot any performance issues with the dome such as any hang-ups /slipping points etc) becomes compromised as the observatory system is tied to using the DeviceHub (for it’s critical role in providing coordinated slaved operation of the Pulsar Dome based on telescope position / target) and how it goes about polling the Pulsar Dome.

The DeviceHub application follows ASCOM standards/expectations; after calling Dome.SlewToAzimuth() if it sees Slewing=False from the Pulsar Dome it interprets this as the Slewing has finished (i.e. the dome is at it’s new target) and changes its polling rate from fast rate (every 1s) to normal rate (very 5s). If a slew is less than say 5 deg, then even though the driver’s internal state becomes true at the next reading of the dome hardware (typically at around T+0.7s or so), the entire slew will complete within 3-4s, so that by the time of the next poll by DeviceHub, the dome has already arrived at it's new position without the observatory control program ever seeing the dome go through a Slewing=True state.  The following log example show the ASCOM log messages from a 5 deg slew from Az 95.0 to Az 90.0, with commentary in blue.

11:49:20.806 Slewing Get True (Slewing=False returned to client but written true in log file due to a reporting bug)
11:49:21.433 Volatile 90.0 0 0.000000 90.0 0 1 970 15004 -2 -25625 18710 2363 1 (dome stationary)
11:49:22.441 Volatile 90.0 0 0.000000 90.0 0 1 970 15004 -2 -25625 18710 2363 1 (dome stationary)

11:49:22.739 SlewToAzimuth Start 95 (Command received to slew dome to Az 95.0 (from 90.0) )
11:49:22.776 SlewToAzimuth Completed (Call returned to client)

11:49:22.777 Azimuth Get Compleate
11:49:22.777 Slewing Get True (Slewing=False returned to client)

11:49:22.778 Azimuth Get Compleate
11:49:22.778 Slewing Get True (Slewing=False returned to client, Device Hub concludes Slew is completed & reverts to Normal Polling, every 5s)

11:49:23.448 Volatile 90.9 1 0.000000 95.0 1 1 970 15004 -2 -25562 18710 2363 1 (dome slewing)
11:49:24.456 Volatile 93.2 1 0.000000 95.0 1 1 970 15004 -2 -25404 18710 2363 1 (dome slewing)
11:49:25.464 Volatile 94.6 1 0.000000 95.0 1 1 970 15004 -2 -25302 18710 2363 1 (dome slewing)
11:49:26.664 Volatile 95.0 0 0.000000 95.0 0 1 970 15004 -2 -25275 18710 2363 1 (dome stationary)
11:49:27.671 Volatile 95.0 0 0.000000 95.0 0 1 970 15004 -2 -25274 18710 2363 1 (dome stationary)

11:49:27.784 Azimuth Get Compleate (Azimuth 95.0 returned to client at this point)
11:49:27.784 Slewing Get True (Slewing=False returned to client. Slew has completed without the client ever seeing Slewing=True during process)

[ Note: There is bug in the present driver logging, whereby incorrect Dome.Slewing values are written to the driver's ASCOM log file. Whilst the driver sends the correct Slewing value through to the client (apart from the issue during the first 0 to 1s after a SlewToAzimuth) but the value that is written to the log file is opposite to its actual value, such that the log file incorrectly reports “Slewing Get True” when the dome is stationary iIt should be written ‘False’), and incorrectly reports “Slewing Get False” when the dome is moving (it should be written ‘True’) ].

[[ Note:  There is a small typographic bug in the present driver logging whereby the word  “Compleate” appears against some Property 'Gets'. It is presumed that it means to say either “Complete” or “Completed”, examples include   “AtPark Get Compleate” &   “Azimuth Get Compleate”.  There is also an inconsistency here since other property messages ‘Connected Get   True’ and ShutterStatus (Get)  3  do write out the property value to the log file. ‘Slewing Get’ also writes out the property value (though incorrectly), whilst the properties showing 'Compleate' don't.  It would be handy if the driver logged the value that the client is being sent in association with all Get requests,  like :  “AtPark Get  False” or Azimuth Get  315.0”.

For longer slews DeviceHub will eventually (after 5-6s) see the dome is slewing (Slewing=True) and go back to a fast polling rate, but the information about the initial part of the slew is lost.

[ It is thought that Dome.FindHome would also need to return Slewing=True immediately after the method Call but it is not known if the Pulsar driver does this or not as the only time .FindHome is used this is when Conform uses it; it's not used during normal operations (Dome.Park is used instead ) ]

There is a reluctance to modify the DeviceHub application for what is seen as a ‘non-compliant’ driver.  The non-compliance (that is : not returning Slewing=True immediately) isn’t picked up by the checks made by the current (6.6.8048.17861) version of the ASCOM Conform tool, but this is due to present limitations of the Conform tool. For methods like SlewToAzimuth that are “async ambiguous” Conform accepts both sync and async behaviours. It differentiates between the two based on whether Slewing is set True on return from the initiating method. If Slewing is True it assumes async behaviour, while if it is False it assumes sync behaviour and that the slew has completed (Althought the Conform checks all seem to pass, the observation that the ASync/Sync modes reported by SlewToAzmiuth Checks alternates between 'Synchrononous slew ' and 'Asynchronous slew' is indicative of an anomaly).  As of Jun 2022 the API documentation of ASCOM Dome Class is not as tight as it should be. This is apparently due to be updated as some point, in the meantime ASCOM documents like https://ascom-standards.org/COMDeveloper/Async.htm  provide important clarification for developers.

Dome.OpenShutter & Dome.CloseShutter

The text in https://ascom-standards.org/COMDeveloper/Async.htm  clearly indicates that following a call to Dome.OpenShutter, the client expects to immediately see Dome.Slewing = True, and Dome.ShutterStatus = shutterOpening (2). It implies that following a Dome.CloseShutter call the client would expect to immediately see Dome.Slewing = True, and
Dome.ShutterStatus = shutterClosing (3)

The API description for the ASCOM Dome.Slewing property (https://ascom-standards.org/Help/Developer/html/P_ASCOM_DriverAccess_Dome_Slewing.htm) is a follows
 - True if any part of the dome is currently moving,
 - False if all dome components are stationary.

This means that Dome.Slewing should be True whenever the Dome's shutter is opening or closing.    Review of log file written by the present dome driver shows “Slewing Get True” during intervals when the shutter is opening/closing, but with the known reporting bug this means that it is actually sending Slewing=False to the connected client which means that is not honouring the ASCOM standard.

Review of log file written by the present dome driver shows the client does immediately see Dome.ShutterStatus = 2 (shutterOpening) after a Dome.OpenShutter command, and does immediately see Dome.ShutterStatus = 3 (shutterClosing) following a Dome.CloseShutter call. However the current process can occasionally be caught out by a timing gap issue and lead to an unexpected sequence of shutterStatus values being sent to the client.

In the following CloseShutter example from a Pulsar ASCOM log file (where commentary is in blue), it can see that after the CloseShutter command at 00:41:39.498 (T0), the shutter status returned to the client immediately changes to ShutterStatus 3 (Closing), through Slewing remains as False (written incorrectly as True in the Pulsar log file). But at 00:41:39.901 (T+0.5s) a dataset is read from the hardware which shows the dome as still being open, and the ShutterStatus supplied to the client changes back to 0 (Open). Because Slewing is False and not changed to True during the closing operation, DeviceHub changes to Normal Polling, which means the downstream client sees ShutterStatus as 0 (Open) for a further 5s, until the Shutter Status is again read by DeviceHub at 00:41:45.500 (T+6.0s) when ShutterStatus is again seen as being 3 (Closing). This continues until the ShutterStatus eventually returns 1 (Closed) at the end of the operation at 00:42:24 (T+44.5s).

00:41:38.698 Volatile 262.3 0 0.000000 262.3 0 0 970 14989 -2 -13524 13231 2352 0
 
00:41:38.734 ShutterStatus 0  (ShutterStatus 0 (Open) is returned to client)
00:41:38.734 Slewing Get True (Slewing = False returned to client, but written incorrectly as True in the log file)
 
00:41:39.498 CloseShutter Completed (Command received to CloseShutter)
 
00:41:39.498 ShutterStatus 3  (ShutterStatus 3 (Closing) is returned to client)
00:41:39.498 Slewing Get True (Slewing = False returned to client, it should return True)

00:41:39.499 ShutterStatus 3  (ShutterStatus 3 (Closing) is returned to client)
00:41:39.499 Slewing Get True (Slewing = False returned to client, it should return True)

00:41:39.901 Volatile 262.3 0 0.000000 262.3 0 0 970 14989 -2 -13524 13231 2352 0

00:41:40.507 ShutterStatus 0  (ShutterStatus 0 (Open) is returned to client)
00:41:40.507 Slewing Get True (Slewing = False returned to client, it should return True)

00:41:40.508 ShutterStatus 0  (ShutterStatus 0 (Open) is returned to client)
00:41:40.508 Slewing Get True (Slewing = False returned to client, it should return True)

00:41:40.922 Volatile 262.3 0 0.000000 262.3 0 3 970 14989 -2 -13524 13231 2352 0
00:41:41.897 Volatile 262.3 0 0.000000 262.3 0 3 970 14989 -2 -13524 13231 2352 0
00:41:42.873 Volatile 262.3 0 0.000000 262.3 0 3 970 14989 -2 -13524 13231 2352 0
00:41:43.849 Volatile 262.3 0 0.000000 262.3 0 3 970 14989 -2 -13524 13231 2352 0
00:41:44.840 Volatile 262.3 0 0.000000 262.3 0 3 970 14989 -2 -13524 13231 2352 0

00:41:45.500 ShutterStatus 3  (ShutterStatus 3 (Closing) is returned to client)
00:41:45.502 Slewing Get True (Sl(Slewing = False returned to client, it should return True)

                              (and so on until)

00:42:23.145 Volatile 262.3 0 0.000000 262.3 0 3 970 14989 -2 -13523 13231 2352 0
00:42:23.547 ShutterStatus 3  (ShutterStatus 3 (Closing) is returned to client)
00:42:23.547 Slewing Get True  

00:42:24.163 Volatile 262.3 0 0.000000 262.3 0 1 890 14727 -273 -13523 14227 2334 0

00:42:24.565 ShutterStatus 1  (Sh(ShutterStatus 1 (Closed) is returned to client)
00:42:24.565 Slewing Get True ((S(Slewing = False returned to client, but written incorrectly as True in the log file)

Summary

Following review  the present ASCOM driver has the following issues:

• Dome.Slewing isn't set to True immediately after a call to Dome.SlewToAzimuth
• Dome.Slewing isn't set to True immediately (or indeed at all) after a call to Dome.OpenShutter or Dome.CloseShutter
• Dome.ShutterStatus sometimes exposes an inappropriate value (due to the driver passing on a value that doesn't yet represent the changed state of the shutter)
• Dome.Dome.Slewing value isn't written correctly in the ASCOM log file (it is written opposite with value to one given to the client)

This issues have been sent to Pulsar Observatories today (2022-07-02) with a request to provide an updated driver, that resolves these issues and brings the driver into line with current ASCOM programming standards for Domes.

Benefits

The benefits of an updated ASCOM driver would be as follows:

• it will be fully compliant with ASCOM Standards
• it can be fully checked with the ASCOM Conform Tool (by exposing the driver's asynchronous behaviour correctly)
• it allows short slews and the first part of longer slews to be 'seen' by the client program
• it allows dome performance monitoring to check slews for any holdups during the initial part of the slew which can't be currently checked,
• it removes/reduces some of the present workarounds to monitoring initiation and completion of SlewToAzimuth, OpenShutter & CloseShutter commands due to current non-compliant behaviour & unexpected values from the dome.

Potential issues using updated driver

There is a possibility that during a HardSuspend operation to close the dome's shutter (in event of heavy cloud or rain) the Dome.CloseShutter call may not be heard or may not happen if the dome is in the middle of a previous SlewToAzimuth operation.  This may not neccessary be an issue associated with the new updated driver as there may already be a potential for the issue in the current system.  

DeviceHub's DomeControl tab (in current 6.6.0.13 version) doesn't allow the shutter to be operated whilst the dome is rotating.  The 'Close Shutter' becomes greyed-out whilst the dome is changing azimuth position (this is throught to be possibly by design since for some dome designs there could be a mechanical/physical issue in moving the shutter if the dome is the rotating, and vice versa) .

Tests with DeviceHub and ASCOM.Simulator.Dome show that Dome.CloseShutter can nevertheless still be successfully called via DeviceHub.Dome from the client program whilst the dome is rotating.  It is unclear whether this can occur with the Pulsar Dome using even the present ASCOM driver or a future updated driver.    It may be safest if the AstroMain's CloseShutter operation, turns-off dome slaving (Dome.Slewing=False) and aborts any slew in progress (Dome.AbortSlew and then waiting until Dome.Slewing=False), before it calls objDome.CloseShutter.   When the shutter closure is complete the operation could potentially re-instate the prior Dome.Slaving value if necessary.

Because there is risk that in event of a HardSuspend  the dome may already be closing (due to Relay initiated closure or due to AstroGuard initiated closure) - even though timing thresholds shouldn't allow this - there is a need to check that AbortSlew at this point wouldn't put the Shutter into an 'Error' state where it couldn't then be closed without user intervention. Existing code may not call Dome.CloseShutter if the ShutterStatus already has state 3 (ShutterClosing).   Daytime test using Pulsar Dome with present driver shows that AbortSlew with shutter opening/closing stops the shutter movement and puts the ShutterStatus into state shutterError (4).  There is no problem in then calling OpenShutter or CloseShutter at this point.

[ Note: The API documentation for Dome.AbortSlew method suggests that the Dome.AbortSlew should set Dome.Slaving to False, however tests using DeviceHub, with Simulator Scope and Simulator Dome indicate that this doesn't happen.  If a slaved dome slew is in progress and Dome.AbortSlew is used, the slew stops temporarily but Slaving remains True and then after a few seconds the dome resumes slew towards a slaved position based on telescope position ]

Registering for 64 bit
In case Pulsar Driver needs to be registered as 64-bit driver, it would be done like this

for 64-bit registration - "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\RegAsm.exe" /register /codebase "yourdriverpath.dll"

for 32-bit registration - "C:\Windows\Microsoft.NET\Framework\v4.0.30319\RegAsm.exe" /register /codebase "yourdriverpath.dll"