• The software in Observatory Shutter Unit was successfully updated  from 1.24 to 1.25 today (2021-03-31)
  
  - Moved Dome to Az 270 deg (close to computer)
  - Connected USB cable from computer to shutter unit
  - Ran “ES Updater.exe”
  - Selected COM port of shutter unit   (COM 15 in my case)
  - Clicked “Select File” and seleced t “Shutter_1_25.hex”
  - Clicked “Update”
  - Once updated removed fuse, waited a minute then replaced
  - Checked Shutter Information on Rotation Unit LCD Panel and confirmed that shutter version had changed to 1.25 ~C
  - Checked that Battery Information data were being received and confirmed that values were as expected.
  - Returned Dome to its Park position (Az 090 deg)
  
  ( Unable to check that shutter opened / closed , due to prevailing wet weather . This was checked later on at start of Observing Session  S878)
  
  The update follows an earlier update to v1.24 on 2021-03-18 (details here)  which had failed to resolve an annoying issue whereby the shutter randomly falls asleep and stops sending fresh battery information and doesn't respond to an ASCOM  request to open/close shutter (requiring a 2nd or even 3rd request to be sent in order to achieve the desired result)
   
• Following plot shows battery related information covering the period from installation of 1.25 version of Shutter Software till several hours after the end of session S878.
  
  - no indication of episodes of shutter falling asleep
  - slightly higher battery drain (amperage –3 mA instead of –1 mA with earlier versions)
  - recharging afterobserving session as expected
  - top-up charging is keeping voltage at 15.0v.
     (frequency of the top up charges is higher than earlier version & is presumably related to the higher background battery drain ?)
  
  
  
  
• It appears that version 1.25 is on course for resolving the issues reported with earlier versions of Shutter software (1.17, 1.22, 1.24)
  
  Shutter Health Information and observing sessions will be monitored over the next fortnight to see how 1.25 behaves over a longer time frame.

Update 2021-04-08

• Battery drain with 1.25 version of the Pulsar Shutter Software is higher than former versions.  The frequency of trickle charges is higher than before but with the dome at its nornal Park position the induction charger easily manages to keep the shutter at a constant 15.0 V. The drain rate is acceptable.
  
  Graph set below in from a 5 day period in early April showing the (acceptable) rate of drain on the battery during a 18 hour period when the dome had been deliberately rotated away from its Parked (Charging) position to a position that would prevent snow being blown under the upper lip of shutter by strong wind/snow conditions at that time. Battery level was recorded to drop from 97% (15.0V) to 91%, (14.8V) over 18 hrs.
   
  
   

Update 2021-05-03

• Pulsar Shutter Software Version 1.25 has been in use for over 4 weeks now.  There have been no occurrences of the former ‘sleep’ issue. This was the issue whereby the shutter sometimes wouldn’t open or wouldn’t shut upon a first (sometimes even a second) ASCOM command because it was in a sleep state.
  
  During 16 observing sessions in April, the shutter responded to each and every command to open or to close. Battery data was continuously available throughout the month with no flat spots that occurred with earlier software versions.
  
  Graph set shows last 5 days of data from the Shutter battery (2021-04-29 to 2021-05-03) demonstrating its behaviour in its new 'continuously awake' state.
   
  
   
• Latest data is shown on the Observatory Status page . ( See Shutter Battery Health - last 5 days )

Issue:   Show-stopping bug in new AstroMain 3.38.11 version.  Retrieve Targets Thread,  Obs.Monitor Thread & TheSky6 freezing when importing night's observing plan as part of AutoStart Start-Up

Description:  Import of Observing Plan stalled after writing report message "Retrieving Targets ..." and after writing log message 'Retrieving 19 targets'  After 3 minutes the  Obs.Manger came back with the Alert that "Loading appears to be frozen. (23:33)", and the Session dropped back to Manual Mode.

Session Initiating                      2021-03-31 23:45  (Local)
  NightDate:                  Ok        2021-03-31        (NightDate)
  Import Observing Plan...
  Selecting Plan...           Ok        2021-03-31   V1   BEST PLAN   (Plan ID: 616)
  Retrieving Targets ...      **        
> Obs.Manager                 Alert     Plan Loading appears to be frozen. (23:48)
> Obs.Manager                 Alert     Returning Observatory to Manual Mode

Observatory Mode                        2021-03-31 23:48  (Local)
  Setting Mode ...            Ok        Manual

The log file showed messages that Obs.Monitor is frozen and TheSky6 was found to be non-responsive.

After parking scope & restarting observatory software (including killing and restarting AstroMain 3.38.11) autostart was again used to kick off session but the same issue occurred again at exactly the same point.  In order to allow the S878 observing session and establish if there was a wider issue or not AstroMain installation was rolled back to previous 3.38.10 version, and this time the observing plan imported ok and session continued normally.

Log File from first instance

23:30:51.67 | Observatory Manager    Plan                  Info  | Loading Latest Observing Plan
23:30:51.67 | Database               Get ListOfPlans       Ok    | Getting List of Plans from Database
23:30:51.67 | Database               Get ListOfPlans       Ok    | 1 plans are listed
23:30:51.67 | Observatory Manager    Select Plan           Info  | Latest Plan: 2021-03-31 V1 BEST PLAN
23:30:51.67 | Plan Loader            Start LoadPlanFromDB  Ok    | Starting Thread to LoadPlanFromDBShell
23:30:52.12 | Latest Plan Loader     Load PlanFromDB       Ok    | Calling ObservingDB.RetrievePlan(PlanID=616)
23:30:52.12 | Database               Retrieve Plan         Ok    | Retrieving Plan
23:30:52.12 | Database               Retrieve Targets      Ok    | Retrieving 19 targets
23:31:01.54 | Observatory Overseer   Check.ObsMonitor      Info  | Obs.Monitor is Pending (7s)
23:31:06.55 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (12s)
23:31:11.56 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (17s)
23:31:16.56 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (22s)
23:31:21.57 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (27s)
23:31:26.58 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (32s)

Log File from second instance (showing that issue occurs at exactly the same point)

23:45:02.89 | Observatory Manager    ObservingPlan         Info  | Importing Observing Plan
23:45:02.90 | Database               Get ListOfPlans       Ok    | Getting List of Plans from Database
23:45:02.90 | Database               Get ListOfPlans       Ok    | 1 plans are listed
23:45:02.90 | Observatory Manager    Select Plan           Info  | Latest Plan: 2021-03-31 V1 BEST PLAN
23:45:02.90 | Plan Loader            Start LoadPlanFromDB  Ok    | Starting Thread to LoadPlanFromDBShell
23:45:03.56 | Latest Plan Loader     Load PlanFromDB       Ok    | Calling ObservingDB.RetrievePlan(PlanID=616)
23:45:03.58 | Database               Retrieve Plan         Ok    | Retrieving Plan
23:45:03.58 | Database               Retrieve Targets      Ok    | Retrieving 19 targets
23:45:12.72 | Observatory Overseer   Check.ObsMonitor      Info  | Obs.Monitor is Pending (9s)
23:45:17.73 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (14s)
23:45:22.74 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (19s)
23:45:27.74 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (24s)
23:45:32.75 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (29s)
23:45:37.76 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (34s)
23:45:42.77 | Observatory Overseer   Check.ObsMonitor      Alert | Obs.Monitor is Frozen (39s)

Analysis
Examination of code shows the point in the ObservingDB.RetrievePlan() routine where  "Retrieving 19 targets" is written to log file.  This is last point that can be traced. After this point the code enters a loop ( For i = 1 To TIS.nItems ) in which  RetrieveTarget () routine is called once for each target.   This routine retrieves data from database for a given target and performs some manipulation and expansion of the target dataset.  At the end of routine the name of the retrieved target is written to the logfile.

The expected set of messages are as follows

00:10:31.06 | Database Retrieve       Target               Ok    | Retrieved Nova Per 2020 (CV)
00:10:31.11 | Database Retrieve       Target               Ok    | Retrieved AT2021gyc (CV)
(For brevity 16 other target loading messages are not shown here)
00:10:38.32 | Database Retrieve       Target               Ok    | Retrieved HIP 87937
00:10:38.32 | Database Retrieve       Targets              Ok    | Total of 19 targets have been loaded
00:10:38.38 | Latest Plan Loader      Load PlanFromDB      Ok    | Plan has been loaded

(Note: these are the messages that were written to logfile later on when the V1 Best Plan was successfully loaded later on using AstroMain 3.38.10)

It is clear from the log file that execution of the RetrieveTarget() routine stalled before reaching the routines end for first target (Nova Per 2020) since logfile contain neither aneither a message confirming retrieval of target nor a message reporting an exception (there was no message ' Exception in Database.RetrieveTarget' written to logfile)

RetrieveTarget() routine in 3.38.11 is identical to that in 3.38.10, as are GetObjectSize()  &  LookupTargetRaDec() routines which are called from the RetrieveTarget()

Possibilities  ?
1)   stall happened at one of three or four calls to the database using
         objReader = objCommand.ExecuteReader() 
            ' where objCommand  is of type SQLiteCommand  , and  objReader is of type SQLiteDataReader
           (ie a database lock issue)

2)  stall in the GetObjectSize() routine that is called to get Object Size, and in which there are calls to TheSky6's Chart object
        ObjectInfo = Chart.Find(MOT(Sky6Name))
         with subsequent calls to ObjectInfo.Property()

3) stall in the LookupTargetRaDec() routine which is called to get Targets's Ra Dec,  and in which there are calls to TheSky6's Chart object
        ObjectInfo = Chart.Find(MOT(TargetName))
        with subsequent calls to ObjectInfo.Property()

4) Some issue with the 3.38.11 build/executable used on Observatory Computer

5) Some environmental glitch.  But hard to see why the glitch happened twice in a row after observatory software had been completely restarted.


RetrieveTargets Testing
Executing 'Session Initiating' routine (3.38.11) in on Development Computer shows no problem,  the code executes normally and targets from a new test plan are all retrieved ok.

Executing 'Session Initiating' routine in a test version (based on 3.38.11) but manipulated to load the same plan (plan id 616) as used during the S878 session also executed normally.

Changes in 3.38.11 compared to 3.38.10
 - New Feature: AstroMain checks and automatically opens AstroGuard at start of a new session
 - New Feature: Session Alerts added when Pulsar Shutter Data suffers an outage of >20 mins (Yellow Alert) or resumes again (Green Alert)
 - Fixed Issue:  Guiding Results are now reset/incorrectly reported for Target when SoftSuspend occurs or other conditions interfere with normal operations.

Prehaps looking for issue in RetrieveTarget is a red herring and the issue lies with Obs.Monitor which frooze at the same time.

Interaction with AstroPlan / TheSky6

1st instance stalled at ~ 23:30:52.12

AstroPlan finished making plan at 23:30:49.28
AstroPlan uploading Session.TargetFrame16.jpg & Session.Images.js at 23:30:52
AstroPlan, FTP Queue Loop resumed at 23:34:02
AstroPlan, RunFtpQueue has finishing uploading a batch of 141 files at 23:34:46.35

2nd instance stalled at ~23:45:03.58

AstroPlan finished getting Sun data at 23:43:08.46
AstroPlan uploading Session.TargetFrame18.jpg & Session.Images.js at 23:45:03
AstroPlan had Exception in 'UploadFileFTP2' whilst getting RequestStream at 23:46:28.07