David's Astronomy Pages
Notes - Session 700 (2019-09-26)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
 
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (0),  Minor Issues (7),  Continuous Improvement (7)
 
 
Bullet ObsOverseer, ObsWorker and other new Processes (AstroMain)
Bullet Investigation - Invalid Sun Altitude at session start
 
Bullet Images from 2019-09-26  >>
 
 

Session Aims & Highlights (2019-09-26)

Main aims

  1. Centering Chart: Test out fixes to 'Centering Charts' (AstroMain 3.11.6)
  2. Mixed Target - Image a selection of targets, but mainly Open Clusters, Variable Stars and Nearby Star Targets due to bright moonlit sky conditions
  3. Software & Systems  - Check stability of software and presence of any side effects following ongoing enhancements to AstroMain and associated support programs.

 
Equipment & Software

Highlights

Lowlights

Summary Plots & Logs

Observing Plan
Image
 
Observing Result
Image
   
Slew/Centering Performance
Image
  
Dome Slewing Performance
Image
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
   
Pre-Session Weather Forecast (AstroWeather 1.7.8)
Cloudy for most of night with skies potentially clearing at 03:00,  low probability of any significant rain
(As it turned out the evolving forecast through the late evening extended the cloud further and further through the night,
and indeed it was cloudy for the the whole night, only clearing at around 06:30 after the sun has risen.
A few spots of rain occurred at around 00:45 and 02:40)
Image   
  
Session Event Log
Time     Event Detail
22:17:35 Session Created Live Session Created (2019-09-26 S00700, ImageSaveNum: 700001)
22:18:57   Obs.Manager Started Obs.Manager started
22:18:59   Obs.Overseer Started Obs.Overseer started
22:19:05   Services Started Observatory Services started
22:20:31 Observatory (Auto) Observatory placed in Fully-Automated Mode
22:20:33 Session Pending Session pending (2019-09-26)
22:20:35 Session Initiating Session initiating (2019-09-26)
22:20:37   Plan Requested Observing Plan requested from AstroPlan (1.11.2)
22:21:04   Plan Loaded Observing Plan loaded to queue (2019-09-26 V1 BEST PLAN, Plan ID: 374)
22:21:20   Camera1 Connected SBIG Camera Connected (set point -10°C)
22:21:23   Telescope Connected Telescope Connected (TheSky6)
22:21:33 Session Equilibration Session ready for dome & camera equilibration
22:21:38 Session Closing Session closing
22:21:40 Critical Light Alert Critical Light Alert (Obs.Manager will close the Shutter)
22:21:48   Shutter Closed Shutter closed (closing time 5s)
22:22:21   Telescope Parked Telescope parked (parking time 28s)
22:23:11 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files)
22:23:13 Session Finished Session Finished
22:30:20 Session Resumed Live Session Resumed (2019-09-26 S00700, ImageSaveNum: 700001)
22:30:57   Obs.Manager Started Obs.Manager started
22:30:59   Obs.Overseer Started Obs.Overseer started
22:31:04   Services Started Observatory Services started
22:32:50   Services Started Observatory Services started
22:33:46   Camera1 Connected SBIG Camera Connected (set point -10°C)
22:33:57   Telescope Connected Telescope Connected (TheSky6)
22:35:06 Observatory (Auto) Observatory placed in Fully-Automated Mode
22:35:09 Session Pending Session pending (2019-09-26)
22:35:11 Session Initiating Session initiating (2019-09-26)
22:35:13   Plan Loaded Observing Plan loaded to queue (2019-09-26 V1 BEST PLAN, Plan ID: 374)
22:35:15   Camera1 Connected SBIG Camera Connected (set point -10°C)
22:35:17   Telescope Connected Telescope Connected (TheSky6)
22:35:24 Session Equilibration Session ready for dome & camera equilibration
22:35:26 Wait on Weather Waiting for acceptable weather conditions
22:37:34   Shutter Opened Shutter opened (opening time 45s)
22:37:37   Equilibration Started Dome Equilibration started
22:37:54   Equilibration Skipped Dome Equilibration skipped by user
22:37:56 Session Running Session running
22:37:58   Queue Started Observing Queue started (22 targets selected)
22:38:00     Target Started (1/22) Target started (1/22, SHK 38)
22:47:43     Target Completed Target partially completed (1/22, SHK 38)
22:47:45     Target Started (2/22) Target started (2/22, MCG5-5-29)
22:50:14       Focusing Failed Foc1 Focusing Failed - star is too dim (TCF-S)
23:06:13     Target Completed Target completed (2/22, MCG5-5-29)
23:07:17     Target Started (3/22) Target started (3/22, NGC 7428)
23:10:12       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:12:23       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
23:28:25     Target Completed Target completed (3/22, NGC 7428)
23:29:09     Target Started (4/22) Target started (4/22, NGC 7779)
23:31:38       Focusing Failed Foc1 Focusing Failed - unable to find a star (TCF-S)
23:47:49     Target Completed Target completed (4/22, NGC 7779)
23:51:01     Target Started (5/22) Target started (5/22, IC 89)
23:53:51       Focusing Failed Foc1 Focusing Failed - star is too dim (TCF-S)
00:10:11     Target Completed Target completed (5/22, IC 89)
00:10:48 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter)
00:10:50   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
00:10:52   Queue Paused Queue pausing for HardSuspend
00:12:49 Session Suspended Session suspended
00:13:38   Shutter Closed Shutter closed (closing time 49s)
00:13:41   Queue Resumed Queue resumed after HardSuspend (170s)
00:31:45     Target Cancelled Target cancelled (6/22, NGC 125) due to cloud
00:41:01 Session Resuming Session resuming
00:41:46   Shutter Opened Shutter opened (opening time 45s)
00:41:47 Session Running Session running
00:41:49   Queue Resumed Observing Queue resumed
00:41:51     Target Started (7/22) Target started (7/22, NGC 38)
01:01:05     Target Completed Target completed (7/22, NGC 38)
01:01:07     Target Started (8/22) Target started (8/22, IC 176)
01:01:42 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter)
01:01:44   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
01:01:46     Target Aborted (8/22) Target aborted (8/22, IC 176) due to cloud
01:01:48   Queue Paused Queue pausing for HardSuspend
01:03:44 Session Suspended Session suspended
01:04:33   Shutter Closed Shutter closed (closing time 49s)
01:04:35     Queue Resumed Queue resumed after HardSuspend (169s)
01:31:25     Target Cancelled Target cancelled (9/22, Whiting 1) due to cloud
01:51:17     Target Cancelled Target cancelled (10/22, SHK 317) due to cloud
02:13:09     Target Cancelled Target cancelled (11/22, NGC 783) due to cloud
02:35:01     Target Cancelled Target cancelled (12/22, UGC 1641) due to cloud
02:56:53     Target Cancelled Target cancelled (13/22, NGC 777) due to cloud
03:18:45     Target Cancelled Target cancelled (14/22, NGC 809) due to cloud
03:40:37     Target Cancelled Target cancelled (15/22, NGC 985) due to cloud
03:48:48 Session Resuming Session resuming
03:49:33   Shutter Opened Shutter opened (opening time 45s)
03:49:36 Session Running Session running
03:49:38   Queue Resumed Observing Queue resumed
03:49:40     Target Started (16/22) Target started (16/22, Hickson 22)
04:09:25     Target Completed Target completed (16/22, Hickson 22)
04:09:27     Target Started (17/22) Target started (17/22, C/2019 Q4 (Borisov))
04:13:03       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
04:14:57       Focusing Failed Foc1 Focusing Failed - failed QC check
04:34:38     Target Completed Target completed (17/22, C/2019 Q4 (Borisov))
04:34:40     Target Started (18/22) Target started (18/22, GCVS U Gem)
04:41:44     Target Completed Target completed (18/22, GCVS U Gem)
04:41:46     Target Started (19/22) Target started (19/22, IC 2098)
04:45:26       Focusing Failed Foc1 Focusing Failed - unable to find a star (TCF-S)
04:51:58 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Shutter)
04:52:00   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
04:52:02     Target Aborted (19/22) Target aborted (19/22, IC 2098) due to cloud
04:52:04   Queue Paused Queue pausing for HardSuspend
04:54:01 Session Suspended Session suspended
04:54:49   Shutter Closed Shutter closed (closing time 48s)
04:54:51     Queue Resumed Queue resumed after HardSuspend (167s)
05:07:05 Session Resuming Session resuming
05:07:50   Shutter Opened Shutter opened (opening time 45s)
05:07:52 Session Running Session running
05:07:54   Queue Resumed Observing Queue resumed
05:07:56     Target Started (20/22) Target started (20/22, Hickson 32)
05:26:53     Target Completed Target completed (20/22, Hickson 32)
05:26:56     Target Started (21/22) Target started (21/22, GCVS NV Ori)
05:30:25       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
05:32:02       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
05:35:34     Target Completed Target completed (21/22, GCVS NV Ori)
05:35:36     Target Started (22/22) Target started (22/22, GCVS RR Tau)
05:37:29       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
05:39:20       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
05:45:41     Target Completed Target completed (22/22, GCVS RR Tau)
05:45:44   Queue Completed Job Queue completed
05:45:48 Session Closing Session closing
05:46:38   Shutter Closed Shutter closed (closing time 45s)
05:47:15   Dome Parked Dome parked (parking time 36s)
05:47:58   Telescope Parked Telescope parked (parking time 39s)
05:48:48 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files)
05:48:55 Session Finished Session Finished
   

Back to Top


Operational Issues (2019-09-26, S700)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Continuous Improvement

[ Prev | Next ]

Back to Top


Obs.Overseer, Obs.Worker and other new Processes (AstroMain)

Session 700 used a new version of AstroMain which contains a number of new processes/routine, and it useful to document the changes.

The overall aim is increase the independance of different parts of the program and reduce chnaces that a problem in one part of the program causes another part to freeze.

Obs. Overseer is a new Threaded Process loop under the umbrella of the Manager Object that is initiated and stopped at the same time as the Obs.Manager.  It's purpose is to provide high level  monitoring of Obs.Manager and Job Queue. and specifically to recognise and log when the Observatory Manager freezes. The process also used to action specific tasks (some formerly actioned within the Observatory Manager loop, others are new) either calling routines directly to perform the tasks or via the setting of flags to trigger tasks managed by a further threaded process loop called Obs.Worker.

At the moment it actions the following
 - WriteLiveHeaderJs.   This outputs of 'System.Header.js' file that describes the Session to Javascipt running in the Current Session Webpage
 - bCaptureObsManagerTab & .    This flag triggers the collection of screen-captures of the Obs.Manager tab and the
 - bCaptureSkyConditionsTab. This flag trggers the collection of a screen capture of the Sky Condition tabs
 - bUpdateSynopsisFields. This flag triggers the writing of various Observatory,Weather and Envirnonment data
 - WriteLatestLiveLogs.  This generates Session.latest (run).log & Session.latest (report).htm containing the more recent log/report messages
SendObsStatusToFile() . This generate a small file to pass key information to AstroGuard
CreateObsManagerChart. Placeholder for code to draw a suitably designed chart with Obs.Manager and other data,

 - Checks status of Job Queue by looking a IamALive Queue / messages for potential overruns during Job Queue execution.
 - Checks status of Observatory Manager
 - Check status of UI

Obs.Worker is a new threaded Process loop under the umbrella of the Manager Object and like the Obs.Overseer thread is initiated and stopped at the same time as the Obs.Manager. It's purpose is to perform listen to flag settings ad perofrm specific tasks (some formerly actioned within the Observatory Manager loop, others are new) either calling routines directly to perform the tasks or via the setting of flags to trigger tasks managed by a further threaded process loop called Obs.Worker.

 - bCaptureObsManagerTab & .    Calls  objConsole.RunCaptureObsMangerTab() &  objConsole.RunCaptureServicesPanel()
to capture specific tabs/panels in AstroMain for uploading to website
 - bCaptureSkyConditionsTab.  Calls objConsole.RunCaptureSkyConditionsTab to capture views of the Sky Condition tabs
 - bUpdateSynopsisFields. Calls  objConsole.UpdateSynopsisFields 

ReportQueueManager is a new thread Process loop that continuously looks for new items  in the qReport queue and write them to the txtLogBox console.
Items are added to the queue in ReportComment, AddComment routines etc by the command  qReport.Enqueue('string to be written')
and then in the queue manager if  qReport.Count > 0 then  AppendTextBox(qReport.Dequeue)


Observations
a) The Obs.Manager freezes during the following operations
 - Connecting Scope / Telescope (5s)
 - Set Tracking Off (5s)
 - Parking Dome (36s +)
 - Parking Telescope (39s +)
 - Closing Down Cameta (30s +)
b) The UI stops responding for around 5s -10s at a time on 3 occasions during 'Start Services')

Back to Top


Investigation - Invalid Sun Altitude at session start

After creating a new session,  setting the Observatory to Fully Automated Mode, and seeing an Observing Plan created and loaded the Observatory Manager proceeded to close down the observatory and finish the session, 5 secs after starting to open the Shutter.

After entering the Equilibrating State at 22:21 and commanding the shutter to open,  the Observatory Manague issued a "Critical Light " and  'Critical Sun Alt Alerts. SunAlt was reported to be of -0.8. and this would have automatically triggered the Critcial Light Alert regardlesss of the light meter reading since a sun alt value of -0.8 might indicate approaching Dawn).  The actual sun altitude at the time was -26 deg. After restarting the Software and unparking the telescope (by power-cycling the scope using the UPB PowerBox Program),  the session was successfully re-started. 
 
During the original session creation , the plan generation happened pretty fast (just 27s, due to code/target improvements) .

The Observatory Manager was started at 20:18.  During initialisation of the main Observatory Loop the Sun Alt is obtained from TheSky6 (This should have been the correct number of -25.7 deg  if TheSky6 was using the Computer Clock which it nornally does when it starts up)  and  LastSunAlt is set to vcNull.  Within the Main Loop, a new Sun Alt value is fetched every 30s.  For first minute or so whilst the Observatory was in Manual mode, the Sun Alt is obtained from Synopsis.SunAlt provided that the published Synopsis time is < 2 minutes old. After moving to Fully-Automated Mode, at 20:20  Sun Alt is obtained from TheSky6  (unless Observatory State is 'Running') .    LastSunAlt is set to SunAlt at the end of code section, ready for the next cycle of the loop.   It would seem SunAlt was read as ~ -25.7 at  some stage (and copied to LastSunAlt) before taking on the value of -0.8 before the next cycle, since at 22:21 the Obs Manager produced the message. "Sun has reached SunAlt of -2 degs" which is produced by the condition 'If LastSunAlt < -2 And SunAlt >= -2 Then'.

So why did SunAlt take on the value -0.8 ?    The most probable explanation is that Sun Alt was read from TheSky during a moment that AstroPlan Schedule Builder had set TheSky6 time to either near sunset or near sunrise in order to collect Sun/Moon data for the night.    Because the Schedule Building was very quick there was no time for Obs. Manager to get a 'new' correct Sun Lat before the Shutter was opened at which point the SunAlt Alert was triggered lead to closin the shutter and shutting down the Observatory.

When the AstroMain program was restarted the original V1 plan was loaded directly without the steps manipulating  the time/date in  TheSky6, and hence the issue wasn't seen the second time around.

Issue was fixed 2019-09-27 (AstroMain 3.12.3) by modifying code to get a fresh SunAlt value from TheSky6 after loading the Observing Plan and before moving to 'Starting' State.

Back to Top