David's Astronomy Pages
 Notes - Session 617  (2018-08-07)

 Previous Notes
Notes
(S616)		   Notes
Notes
Main		   Home
Home
Page		   Next Notes
Notes
(S618)		  
   
Bullet Session Aims & Highlights
  - Observing Result
 - Night Summary Plot
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (0),  Minor Issues (0),  Small Defects (0),  Continuous Improvement (0)
   
Bullet Autofocussing
Bullet Fully Automated Operation Test 
 
Bullet Images from 2018-08-07  >>
 
2018-08-09 to 2018-08-11+
Bullet AllSky Computer Problems
Bullet AllSky Computer Outages & Events
 

2018-08-07 (Automated Operation Testing)

Session Aims & Highlights (2018-08-07)

Main aims of this nighttime session was to continue the ramp-up phase for the Clair3 Observatory including

1) Test a newly consolidated autofocusing routine in manual mode.
  (This will be routine that will be later incorporated into the Fully Automated Observatory Mode).

2) Perform further live test of Observatory System operating in Fully Automated Mode,
    in particular testing newly corrected code to perform precise positioning of targets (including N/W Offset).
    and newly modified code for performing session closing and session housekeeping

3) Acquire some images of selected deep sky and variable star targets

4) Continue checking stability of POTH.Hub (Telescope + Dome) & Observatory Control Software

Equipment & Software
- 12" LX200 and ST-10XME
- CCDApp2, TheSky6, CCDSoft 5, POTH Hub,

Highlights :

- The newly consolidated autofocusing routine worked more or less as designed, but due to a bug it generated an invalid Focus Offset as the end result.  This will need investigation and correction.

- Specific successful achivements of the automated system  : 
    - a new session (session ID, session name, session folders) was created for the night
    - an optimised Observing Plan/Schedule was created for the night from a database of targets and loaded ready for observing
    - Dome, Telescope and CCD Camera were connected to
    - Temperature Set Point on CCD Camera was set (based on ambient temperture)
    - Dome, Telescope and CCD Camera were operated 
    - Weather and Cloud Sensor data was used to keep the Dome Shutter open for the duration of the session
    - Execution of Observing Plan / Target Queue proceeded according to scheduled start times.
    - Close down of the Observatory (Telescope & Dome Parking, CCD Disconnection)
    - Post Session Housekeeping (File copying etc)

   (Telescope was already unparked and operating at the start of automated run. 
   Wait to Next Night and Commence Next Session were not tested on this occassion.   )
  
- No specific failures of the automated system, however issues arising
   - creation of Observing Plan was very slow (due to very slow creation of Gen1 population)
   - Observing Plan dominated by deep sky 'gallery' targets (with longer duration) at expense
     of Variable Star & Nearby targets (shorter duration), this prevent testing of newly corrected code to
     perform precise positioning of targets (including N/W Offset)

- Images of several targets were obtained  (images)

- There were with no major issues with the Observatory System itself during the course of the session.

Observing Result (2018-08-07, S617)
(Observation Status :  Green=Completed, Yellow= Partially Completed)
 
Image
  
Night Sky Summary Plot - 2018-08-07
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image

Back to Top

Operational Issues (2018-08-07, S617)

Issues arising during the session:

Function ComputeAngularSeparationNew(Ra1 As Double, Dec1 As Double, Ra2 As Double, Dec2 As Double) As Double
'===============================
Dim A As Double

  'Cos(A) = sin(Decl.1)sin(Decl.2) + cos(Decl.1)cos(Decl.2)cos(RA.1 - RA.2)

  ' Convert RA hours to RA degrees
  Ra1 = Ra1 * 15
  Ra2 = Ra2 * 15

  ' Calculate Angular Separation (in Radians)
  A = Math.Acos(Math.Sin(degToRad(Dec1)) * Math.Sin(degToRad(Dec2)) +
  Math.Cos(degToRad(Dec1)) * Math.Cos(degToRad(Dec2)) * Math.Cos(degToRad(Ra1 - Ra2)))

 ' Return Angle Separation (in Degrees)
  Return radToDeg(A)

End Function

' Julian Dates
' ------------

  ' Initial Julian Date 
    ProfileStartJD = GetJulianDate(ProfileStartTime)
  ' Note: GetJulianDate is a local subroutine that calls Utils.ConvertCalenderToJulianDate)

  ' Other Julian Dates
    JD = ProfileStartJD + TStart.Subtract(ProfileStartTime).TotalDays

 

Issues arising outwith of the session itself:

 

Back to Top

Autofocussing (2018-08-07)

A newly built autofocusing routine was tested, comprising the consolidation of existing seperate elements.   Routine was tested in 'manual mode' but is designed to eventually be available to Fully Automated operations.  The routine worked more or less as design, but produced and saved an invalid Focus Offset (-1711) rather than the correct figure of -7.  This critical bug requires investigation and correction.

The correction value of -7 indicates that the temperature correction coefficients calculated from Focussing data from Session S609 to S616 are working pretty well.

Focus Telescope                         2018-08-08 01:13 (Local)

  Select Focus Star                     
  Taking 1s image (C)...      Ok        [0000] 00:13:55 (UT)  Full Locate C
  Selecting Star ...          Ok        Star located at 366.2, 251.0
  Checking Position           Info      18 pixels from image centre
  Taking 1s box (C) ...       Ok        | 3285 |  2.2 | 11.5 | 15236 | -13.20 | 434 |  1.13 | 
  Examining Star...           Ok        Changing Focus Exposure to 0.3s

  Prepare Focus Profile                 
  Focus Range                 Ok        2665 - 3905
  Points                      Ok        20
  Filter                      Ok        C
  Box                         Ok        150x150
  Binning                     Ok        1x1
  Exposure                    Ok        0.3s

  Measure Focus Profile                 
| Image                     | Result    | DRO  | FWHM | Temp | Max   | Mag    | SN  | Elong |
  Taking 0.3s box (C)  1/20   Ok        | 2665 |  8.3 | 11.5 |   766 |  -9.85 |  79 |  1.13 | 
  Taking 0.3s box (C)  2/20   Ok        | 2785 |  5.8 | 11.5 |  1145 | -10.46 | 113 |  1.14 | 
  Taking 0.3s box (C)  3/20   Ok        | 2905 |  4.4 | 11.5 |  1715 | -11.06 | 153 |  1.32 | 
  Taking 0.3s box (C)  3/20   Ok        | 2905 |  5.2 | 11.6 |  1752 | -10.77 | 132 |  1.10 | 
  Taking 0.3s box (C)  4/20   Ok        | 3025 |  3.0 | 11.5 |  2878 | -11.51 | 192 |  1.13 | 
  Taking 0.3s box (C)  5/20   Ok        | 3065 |  2.6 | 11.5 |  3595 | -11.73 | 215 |  1.08 | 
  Taking 0.3s box (C)  6/20   Ok        | 3105 |  2.8 | 11.6 |  3122 | -11.57 | 199 |  1.03 | 
  Taking 0.3s box (C)  7/20   Ok        | 3145 |  2.5 | 11.5 |  3649 | -11.72 | 214 |  1.04 | 
  Taking 0.3s box (C)  8/20   Ok        | 3185 |  2.4 | 11.5 |  4437 | -11.85 | 228 |  1.22 | 
  Taking 0.3s box (C)  8/20   Ok        | 3185 |  2.5 | 11.6 |  3411 | -11.73 | 215 |  1.06 | 
  Taking 0.3s box (C)  9/20   Ok        | 3225 |  2.2 | 11.6 |  4538 | -11.89 | 232 |  1.10 | 
  Taking 0.3s box (C) 10/20   Ok        | 3265 |  2.5 | 11.6 |  3534 | -11.72 | 213 |  1.04 | 
  Taking 0.3s box (C) 11/20   Ok        | 3305 |  2.2 | 11.6 |  4626 | -11.82 | 224 |  1.08 | 
  Taking 0.3s box (C) 12/20   Ok        | 3345 |  2.5 | 11.5 |  3180 | -11.62 | 204 |  1.10 | 
  Taking 0.3s box (C) 13/20   Ok        | 3385 |  2.5 | 11.5 |  3611 | -11.56 | 197 |  1.14 | 
  Taking 0.3s box (C) 14/20   Ok        | 3425 |  2.9 | 11.4 |  2960 | -11.49 | 190 |  1.02 | 
  Taking 0.3s box (C) 15/20   Ok        | 3465 |  2.9 | 11.5 |  3201 | -11.46 | 188 |  1.10 | 
  Taking 0.3s box (C) 16/20   Ok        | 3505 |  2.7 | 11.5 |  3311 | -11.56 | 197 |  1.11 | 
  Taking 0.3s box (C) 17/20   Ok        | 3545 |  4.7 | 11.5 |  2081 | -10.94 | 144 |  1.35 | 
  Taking 0.3s box (C) 17/20   Ok        | 3545 |  3.4 | 11.5 |  3285 | -11.33 | 176 |  1.14 | 
  Taking 0.3s box (C) 18/20   Ok        | 3665 |  4.5 | 11.5 |  1846 | -10.85 | 138 |  1.19 | 
  Taking 0.3s box (C) 19/20   Ok        | 3785 |  6.7 | 11.5 |   936 | -10.13 |  93 |  1.27 | 
  Taking 0.3s box (C) 19/20   Ok        | 3785 |  5.8 | 11.4 |  1069 | -10.30 | 103 |  1.08 | 
  Taking 0.3s box (C) 20/20   Ok        | 3905 |  8.5 | 11.5 |   870 |  -9.57 |  68 |  1.01 | 

  Review Focus Profile                  
    Profile No.               Ok        1
    Altitude                  Ok        41.63 deg
    Start/End Time            Ok        0014.23 - 0017.58
    Best Focus                Ok        3278 steps (Av Temp: 0.00 °C ) (<< AvTemp value is invalid)
    FWHM                      Ok        2.23 arc secs
    Airmass                   Ok        1.50
    Zenith FWHM (est.)        Ok        2.01

  Adjust Focus                          
  Moving to Best Focus ...    Ok        3278  (adjustment -7)
  Updating Focus Offset...    Ok        -1711 (<<<<  This value is invalid )

Operation...                  Completed
Focus Profile & Temperature Coefficients (2018-08-07)
Image
  
Image

Back to Top

Fully- Automated Operation Test (2018-08-07)

After skies had finally cleared, and with key hardware powered up (Observatory Laptop, Dome, Telescope and CCD Camera),  key software running (TheSky6, CCDSoft5 and CCDApp2), key observatory services started (Observatory Manager, Weather/Cloud Sensor, ASCOM Dome, ASCOM Scope) and a newly developed automated focussing routine tested, the observatory manager control was placed into 'Fully-Automated' Mode. This initiated the Fully-Automated cycle of operation, comprising the following session states : Pending, Initiating, Starting, Equilibrating, Running (or Suspended), Closing, Housekeeping, Finished, Back to Pending. 

Observatory Mode                        2018-08-08 01:18 (Local)
  Setting Mode...             Ok        Fully-Automated
 

The Session began in the 'Pending' State, but as the Sun had already set (Sun Altitude < 0) the Session immediately moved to 'Initiating' where the routine built an observing plan for the night with a scheduled start time equivalent to a sun altitude of -10 deg.
The process was very slow to create initial Gen1 population and around 25 mins of observating time was wasted. Eventually the Plan finished building and it was successfully loaded into the Target/Job Queue ready for execution.

Session Pending                         2018-08-08 01:18 (Local)
  NightDate:                  Ok        2018-08-07       (NightDate)
Session Initiating                      2018-08-08 01:18 (Local)
  NightDate:                  Ok        2018-08-07       (NightDate)
  Import Observing Plan...
Create Observing Plan                   2018-08-08 01:18 (Local)
  Request Observing Plan...   Ok
 

The Session then moved to 'Starting'  where CCD Camera and Telescope were successfully connected and an appropriate CCD Temperature Set Point was reconfirmed.  Session setup was successfully performed (new Session ID, Session Name and Session Folders).  

Session Starting                        2018-08-08 01:43 (Local)
  Session:                    Ok        2018-08-07       (NightDate)
Connect Camera                          2018-08-08 01:43 (Local)
  Connecting camera...        Ok
  Temperature Regulation...   Ok
  Temperature Set Point...    Ok        -15 °C
  TempMonitor:                Ok        Already active
Connect Telescope                       2018-08-08 01:43 (Local)
  Establishing link...        Ok
  Telescope Coords...         Ok        Az: 177.9, Alt: 42.1
  Turn Off Tracking..         Ok        Telescope Tracking is OFF
Session                                 2018-08-08 01:43 (Local)
  Session Name:               Ok        2018-08-07 S00617 (Session)
  Creating session folder ... Ok        .\New\2018-08-07 S00617
  Creating analysis folder ...Ok        .\...\2018-08-07 S00617
  Creating subfolders ...     Ok        .\...\2018-08-07 S00617\_RAW, \_REDUCED etc
  Stopping CCDSoft            Ok        
  Setting Autosave Number     Ok        617001
  Restarting CCDSoft
  Creating Camera object...   Ok        CCDsoft.Camera
  Creating Autoguider object..Ok        CCDsoft.Camera
  Creating Folder object..    Ok        CCDSoft.Folder
  Reconnecting camera ...
Connect Camera                          2018-08-08 01:43 (Local)
  Connecting camera...        Ok
  Temperature Regulation...   Ok
  Temperature Set Point...    Ok        -15 °C
  TempMonitor:                Ok        Already active
  Setting AutoSave Folder ... Ok        .\New\2018-08-07 S00617
  

The Session then moved to 'Equilibrating', and perfomed 4 minutes of equilibration with the Shutter Open, during which CCD Temperature reached its Regulated Set Point (-15°C), but Temperature Difference between Observatory/Scope and Outside air remained firmly at +1.5°C.

Session Equilibrating                   2018-08-08 01:43 (Local)
  Dome Equilibration:         Started   2018-08-08 01:43 (Local)
  Turn off Dome Slaving...    Ok        Dome Slaving is OFF
    Rotating Dome into wind             Wind Direction 225°
    Temperatures at 01:43     Open      Amb.  9.7°C, Obs. 11.2°C (+1.5°C), Scope 11.1°C, CCD -10.2°C (-10.2°C)
    Temperatures at 01:44     Open      Amb.  9.7°C, Obs. 11.2°C (+1.5°C), Scope 11.1°C, CCD -14.8°C (+ 0.2°C)
    Temperatures at 01:45     Open      Amb.  9.6°C, Obs. 11.1°C (+1.5°C), Scope 11.1°C, CCD -15.2°C ( -0.2°C)
    Temperatures at 01:46     Open      Amb.  9.6°C, Obs. 11.1°C (+1.5°C), Scope 11.1°C, CCD -15.2°C ( -0.2°C)
  Dome Equilibration:         Stopped   4.0 mins (4.0 mins total)
  

After the allotted time the session moved to 'Running' and begun executing the Job Queue.  

Session Running                         2018-08-08 01:47 (Local)
  NightDate:                  Ok        2018-08-07       (NightDate)
  Turn on Dome Slaving...     Ok        Dome Slaving is ON
  Turn On Tracking..          Ok        Telescope Tracking is ON
Take Frame Jobs (Live MODE)             2018-08-08 01:47 (Local)
  Checking Plan...            Ok        2018-08-07   V1   BEST PLAN            (Plan ID= 176)
  Checking Targets...         Ok        14 targets selected
  AutoSave Folder...          Ok        C:\Data\CCD Imaging\New\2018-08-07 S00617
  AutoSave Start Num...       Ok        617001
  Date/Time (TheSky6)...      Ok        2018-08-08 01:47:44 (Local)
                                        2018-08-08 00:47:44 (UT)
  Lat/Long (TheSky6)...       Ok        57 19 09.00 N / 02 00 24.99 W
  CCD Regulation ...          Ok        Regulation is ON (Setpoint: -15.0 °C)
  Checking CCD Temp...        Ok        -14.8 °C (81%)
  Airmass Cutoff...           Ok        Airmass 3.3
  Checking Targets ...        Ok        14 targets selected
  Setting Up RealTime VMA...  Ok        (1.2 Method)
  Checking VMA Folders ...    Ok        
  Reading Reference File ...  Ok        ..\Reference\ProjectStars.txt
 


After skipping 10 initial targets which has planned schedule start times that had already been missed the Job Queue was executed observations on 4 targets.  This were completed successfully apart from the final frame of the 1st target which was skipped due to time.

Hickson 60  (1/14)                      2018-08-08 01:47 (Local)
  Checking Start Time ...     Fail      Planned Start Time (23:09) has been missed. Target will be skipped
  
 [ lines removed for brevity ]

GCVS V Sge  (10/14)                     2018-08-08 01:47 (Local)
  Checking Start Time ...     Fail      Planned Start Time (01:42) has been missed. Target will be skipped
NGC 7008  (11/14)                       2018-08-08 01:47 (Local)
  Target:                     Ok        NGC 7008 (ID: 379)
  Object Type:                Ok        Pl. Neb
  Constellation:              Ok        CYG
  Comment:                    Missing   
  Observation Type:           Ok        Normal
  Project:                    Ok        Summer Targets (ID: 6)
  Checking Start Time ...     Ok        
  Waiting to Start...         Wait      Wait till 01:51 to commence target (4.1 mins)

  [ lines removed for brevity ]

  Taking 60s image (V)...     Ok        [00617015] 01:13:39 (UT)  1/4 2x2 60s V
  Taking 60s image (V)...     Ok        [00617016] 01:14:51 (UT)  1/4 2x2 60s V
  Taking 60s image (V)...     Failed    Image cancelled to prevent observation from overrunning its time slot
PLN 93-2.1  (12/14)                     2018-08-08 02:15 (Local)
  Target:                     Ok        PLN 93-2.1 (ID: 402)
  Object Type:                Ok        Pl. Neb
  Constellation:              Ok        CYG
  Comment:                    Ok        Ast. Now, Aug 2018, p70
  Observation Type:           Ok        Normal
  Project:                    Ok        Deep Sky Challenge (ID: 9)
  Checking Start Time ...     Ok        
  Waiting to Start...         Wait      Wait till 02:17 to commence target (1.2 mins)

  Continue..                  Ok        2018-08-08 02:16 (Local)

  [ lines removed for brevity ]

  Taking 30s image (C)...     Ok        [00617024] 01:22:41 (UT)  Full 3x3 30s C
  Taking 30s image (C)...     Ok        [00617025] 01:23:20 (UT)  Full 3x3 30s C
LBN 534  (13/14)                        2018-08-08 02:25 (Local)
  Target:                     Ok        LBN 534 (ID: 401)
  Object Type:                Ok        Gas Cloud
  Constellation:              Ok        AND
  Comment:                    Ok        Imag p130 Ast Now Jan 2018
  Observation Type:           Ok        Normal
  Project:                    Ok        Winter Targets (ID: 7)

  [ lines removed for brevity ]

  Taking 180s image (C)...    Ok        [00617042] 01:52:46 (UT)  Full 3x3 180s C 
  Taking 180s image (C)...    Ok        [00617043] 01:55:56 (UT)  Full 3x3 180s C
Hickson 8  (14/14)                      2018-08-08 02:59 (Local)
  Target:                     Ok        Hickson 8 (ID: 437)
  Object Type:                Ok        Galaxy Group
  Constellation:              Ok        AND
  Comment:                    Missing   
  Observation Type:           Ok        Normal
  Project:                    Ok        Hickson Groups (ID: 13)
  Checking Start Time ...     Ok        
  Waiting to Start...         Wait      Wait till 03:00 to commence target (1.7 mins)

 [ lines removed for brevity ]

  Taking 180s image (C)...    Ok        [00617049] 02:12:38 (UT)  Full 3x3 180s C
  Taking 180s image (C)...    Ok        [00617050] 02:15:48 (UT)  Full 3x3 180s C

Operation...                  Completed

Upon completion of the Job Queue, the Session moved to 'Closing' and proceeded to close the dome, park the dome, park the telescope and disconnect the CCD camera.

Session Closing                         2018-08-08 03:19 (Local)

  Begin Shutdown              Ok        2018-08-07 S00617(Session)

  Close Shutter                         
  Close Shutter...            Closing   
  Close Shutter...            Ok        Shutter is Closed (45s)

  Park Dome                             
  Park Dome...                Ok        Dome is Parked

  Park Telescope                        
  Park Telescope ...
  Park Telescope ..           Ok        Telescope is Parked
  Disconnect Telescope ...    Ok        Telescope is Disconnected (TheSky6)

  Close Camera                          
  Turnoff CCD Cooling ...     Ok        CCD Cooling is OFF
  Turnoff CCD Fan ...         Ok        CCD Fan is OFF
  Wait for CCD to re-warm...            CCD Temperature : -14.8°C
  Wait for CCD to re-warm...  Ok        CCD Temperature : -5.2°C
  Disconnect Focuser ...      Ok        Focusser is Disconnected
  Disconnect Camera ...       Ok        Camera is disconnected

  Shutter Stats                         
  - Opening                   Info      Open Shutter Requests:  0 (Completed: 0)
  - Closing                   Info      Close Shutter Requests: 1 (Completed: 1)
 

The Session then moved to 'Housekeeping' and proceed to clean up raw files and move them to 'analysis/raw' folder, and to move log files to 'analysis/log' folder, before moving to 'Finished'.

Session Housekeeping                    2018-08-08 03:21 (Local)

  Begin Housekeeping          Ok        2018-08-07 S00617(Session)

  Cleanup FITS files
  Preparing clean-up ...      Ok        
  Cleaning file names ...     Ok        50 fits files have been tidied up
  Removing .SRC files ...     Ok        50 src files have been deleted

  Create FITS summary file
  Preparing Summary file ..   Ok        Fits Summary.dat
  Writing Summary file ..     Ok        50 fits files have been summarised

  Transfer FITS Files
  Preparing transfer...       Ok        
  Transferring Files...       Ok        Copying .fits files
                              Ok        Copying Fits Summary.dat
                              Ok        50 fits files have been copied to the ./_RAW folder  (27.0 MB)

  Archive Session Files                 
  Archiving Files ...         Ok        (.log, .htm, .dat etc)

Session Finished                        2018-08-08 03:21 (Local)
  Waiting till next night...

 

Back to Top

2018-08-09  to  2018-08-11+

AllSky Computer Problems

At 2018-08-09 00:56 and with AllSky & Weather programs appearing to be operating perfectly ok beforehand,  communication with Oculus AllSky Camera and with Cloud Sensor appear to have both been suddenly lost at precisely the same time.   Cycle time (normally 30s or so) dropped down to around 13 minutes for the remainder of the night with the log file appearing to shown CCDApp3_AllSky waiting long periods for the Camera leading to time overruns, typically around 716s, for each cycle before it seemingly timed-out.  AllSky processing appeared to operate on what is seemingly a blank / empty image.

Then at 2018-08-09 10:20 and with programs appearing to be operating perfectly ok beforehand (apart from missing AllSky Camera and missing Cloud Sensor data) the AllSky/Weather laptop stopped running for no apparent reason.

Restarting the laptop services on 2018-08-11 was difficult with similar sorts of problems to those noted on other occasions). This time the principal problems was due to the inability to reconnect to the AllSky camera using the normally used port on the USB Hub. It could only be reconnected by inserting the camera's USB lead into a USB port on side of laptop insetad of the hub. Other problems were missing G Drive connection to USB External Hard Drive.

(AllSky Computer Operations were eventually restarted at 2018-08-11 20:00 (approx))

Night Summary Log (NightDate 2018-08-08)
showing problem with AllSky & Cloud Sensor commencing at 00:56
Image

 

Further AllSky Computer Outages & Events

2018-08-13 
AllSky Operations went offline at 2018-08-13 09:10 when the AllSky computer stopped working for no identifiable reason.
The number of computer failures is becoming both intolerable and frustrating to diagnose.   The laptop is a second hand one, so maybe a new laptop is the only answer ?)

(AllSky Operations were restarted at 2018-08-14 08:53.   (Outage 24 hours)

2018-08-14
AllSky Computer connections to Network & Internet were lost on around 5 occasions on 2018-08-14, but the events were successfully caught and the Ethernet Adapter successfully reset via a .bat script to restore connectivity.
(Whilst the underlying cause of the loss of connection is still unknown, the workaround is at least working successfully).

2018-08-15
AllSky Operations went offline on 2018-08-15 (at ~17:35). Checking the computer showed that it had stopped in someway, had restarted and was waiting for user login. A Windows Update (2018-08 Cumulative Update for Windows 10 Version 1803 for x64-based Systems (KB4343909)) had taken place at 2018-08-15 17:49 and the computer may have restarted in response to this or alternatively it may have had a Blue Screen crash and then automatically restarted. To provide protection against operating system restarts Windows Update was configure to temporarily pause updates from installing in the next 35 days
AllSky Operations were restarted at 2018-08-15 22:10 (Outage  5.5 hours)

2018-08-20
AllSky Scorecard went offline on 2018-08-20 (at ~5:35). Checking showed that the CCDApp3_AllSky program had stopped in executing due to an exception caused by a bug in some newly added code. Bug was fixed and the program restarted at 08:10.  (Outage 2.5 hours)

AllSky Operations went offline on 2018-08-20 (at ~08:37) when the AllSky computer stopped working for no identifiable reason (it was powered up with just a black screen)
(AllSky Operations were restarted at 2018-08-15 09:28 )

2018-08-28
AllSky Operations went offline at 2018-08-28 08:51 when the AllSky computer stopped working for no identifiable reason. The only Windows acknowledge was an event message under 'Reliability Review' saying that Windows hadn't been properly shutdown at 2018-08-28 08:30.
Before this failure AllSky Operations had been working continuously for 8 days.
AllSky Operations were restarted at 2018-08-28 10:45.   (Outage 2 hours)

2018-09-02
AllSky Operations went offline at 2018-09-02 08:46 when the AllSky computer stopped working for no identifiable reason. The only Windows acknowledge was an event message under 'Reliability Review' saying that Windows hadn't been properly shutdown at 2018-09-02 08:30.
Before this failure AllSky Operations had been working continuously for 5 days.
AllSky Operations were restarted at 2018-09-02 10:30.   (Outage 1.75 hours)

A pattern is beginning to be noticed about the times at which the AllSky computer stops working . It s always around the same time time of day;  which is the morning between 08:30 and 09:10 (09:10. 08:37, 08:51, 08:46). This seems to be more than coincidence and would suggest that the failure are associated with some regular occurring computer operation, a scheduled task for example, however the number of days between failure events varies.  Another possibility is that it is driven by a particular temperature or humidity change associated with the observatory warming up in the morning, but the observatory environment is generally hotter later in the day than at 08:45.  A third possibility is that I interact with the AllSky Laptop via Windows or UltraVNC at this time of the day, however whilst this hasn't been specifically tracked, remote interactions with the laptop occur at other times of the day without instigating a failure.

2018-09-05
AllSky Computer connections to LAN & Internet went down at 2018-09-05 22:23.   The event wasn't rectified by 'Reset Ethernet Adapter' routine in Weather Server Program. Either the lost connection happened a way that hid the fact from the watching routine (with checks connection by ping the router's IP address every 5s) or the problem couldn't be resolved by resetting the adapter. )

Windows 10 Network Connection Troubleshooter was run to identify/resolve the connection problem the next morning, but was unable to fix things.   Ethernet Switch/Hub and 'Ethernet over Power''  were power cycled and after a short delay the internet connection came back  at 2018-09-06 08:30  (Outage 10 hours)

2018-09-06
Cloud Sensor data went offline at 2018-09-06 08:30.  This was at the time a Network Connection outage was being investigated and resolved.  
The Cloud Sensor III program was successfully restarted  at 2018-09-06 12:30.
Data flow to Weather Server, Weather Graphs  and AllSky program resumed at 2018-09-06 12:31.  (Data outage 4 hours)

2018-09-07
AllSky Computer connections to LAN & Internet went down at 2018-09-07 21:40.   The event wasn't rectified by 'Reset Ethernet Adapter' routine in Weather Server Program. Either the lost connection happened a way that hid the fact from the watching routine (with checks connection by ping the router's IP address every 5s) or the problem couldn't be resolved by resetting the adapter)

Problem was noticed but not dealt with until the following morning. It appears to be the same problem as the one that occurred two days before 2018-09-05 22:23.  Windows 10 Network Connection Troubleshooter unable to identify/resolve the connection problem  'Ethernet over Power'' adapter was power cycled and the Ethernet Cable taken out and reinserted into the the laptop, and after a short delay the internet connection came back at 2018-09-08 10:10   (Outage 12.5 hours)

2018-09-09
AllSky Operations went offline at 2018-09-09 19:14 when the AllSky computer stopped working for no identifiable reason. The only Windows acknowledge was an event message under 'Reliability Review' (dated 2018-09-09 19:26) saying that Windows hadn't been properly shutdown at 2018-09-09 18:38.
Before this failure AllSky Operations had been working continuously for 7 days (but with some outages due to loss of network connection issue)
AllSky Operations were restarted at 2018-09-02 19:39.   (Outage 0.5 hours)

2018-09-14
The AllSky Computer stopped out sending Weather Synopsis / Scorecard data at 2018-09-14 22:51. This was quickly picked up as Observatory Computer was being remotely monitoring as a live observing session was in progress.   Checking the AllSky computer showed that the "Take Image Series" routine had exited due to a critical exception and 5 attempts to automatically restart the routine had also failed.

Unable to diagnose the cause of the program at the time and needing a quick fix the Take Image Series routine was freshly re-started at 2018-09-14 23:05.   (Outage 0.25 hours)

2018-09-15
AllSky operations continued working as normal for a further 5 hours until 2018-09-15 03:41 when the "Take Image Series" routine again crashed out (after 5 auto-restart attempts) immediately stopping the generation of the Weather Synopsis/Scorecard and AllSky products such as the AllSky Images and Night Summary Plots.     

Problem was diagnosed the following morning (see details), and a short term fix made to allow the Take Image Series to commence working again. The probable underlying cause of the problem has been identified but the longer term fix still needs to be made.  Take Image Series routine was freshly re-started at 2018-09-15 10:21.   (Outage 10.6 hours)

2018-09-21
AllSky operations appeared to go offline around  2018-09-21 17:35 (no updates to Website after this time).   Examination of files showed that the last AllSky files were generated at 17:41.
When computer was physically examined at a later data (2018--10-01) the computer was non-responsive.

AllSky Computer and Operations  re-started at 2018-10-01 14:00   (Outage 7 days)

The outage was later investigated.   Analysis showed that the computer had a automatic update to windows 10 v1803 at  2018-09-21 17:44 and it would seem that the computer was automatically rebooted which killed the AllSKy operations. There is evidence of windows program updates on 09-23 and 09-25.     The AllSky suite is not designed to automatically load / start upon computer start-up and therefore there was no AllSky /Weather operation after the apparent automatically machine reboot on 09-21. 

Earlier 'regular user' attempts to turn off Automatic Updates has failed.  This time Local Group Policy (Windows Professional), was used either a) Disable the "Configure Automatic Updates policy   (under \Computer Configuration\Administrative Templates\Windows Components\Windows Update.  or b) Enabling the same policy with configuration no. 2,  "Notify before downloading and installing any updates"  ( from  https://www.windowscentral.com/how-stop-updates-installing-automatically-windows-10 )
The occurance of further automatic window updates (or not) will be monitored  over the next fortnight.

2018-10-01
Unable to connect to AllSky Laptop over Network Windows. Ethernet connection type was set to Public and there was no option given in Windows 10 settings to set it to Private (there was o Private/Public toggle as there should have been).  Fix
1) Use Regedit and navigate to :  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList\Profiles
2).Expand the Profiles key and navigate through each GUID key below that, noting the corresponding ProfileName on the right
3).When you find the ProfileName of the network you want to change to Private, double-click on the Category DWORD on the right-handside:  Change 6  ‘Value data’ from 0 (Public) to 1 (Private).0
4) Close Regedit and Roboot
( https://superuser.com/questions/1160421/how-do-i-force-windows-10-to-see-a-network-as-private )

2018-10-04
AllSky operations appeared to go offline around  2018-10-04  12:35, and couldn't be restarted remotely or in person.  Reliability report indicates that the Computer unexpectedly shutdown at this time.

The Computer was restarted and AllSky operation resumed at 2018-10-17 11:30  (Outage 13 days)

2018-10-26
AllSky/Weather System stopped updating graphs and stopped supplying AllSky Program with weather data at 2018-09-26 08:00. Investigating showed that the Weather Server program had crashed. Attempts to restart the Weather Server program failed, crashing a few moments after starting Services.

Analysing the problem showed that the crash was associated with the Graph Service and specifically the import or use of AllSky Stars data. Reducing the size of the data file by removing older rows appeared to fix the problem, though the code wouldn't suggest data rows being an issue. To release the lock on the file, the AllSky routine had to be stopped, which might have fixed the problem without the need to remove rows.    It seems too coincidental that the problem occurred immediate after the AllSky program switched from nightime to daytime operations.

Full operations were resumed at  2018-09-26 14:00 (Outage 6 hours).

It's noticed that whilst WIndows 10 updates are now successfully turned off, Windows continues to do Game App  updates &installs like CandyCrush and MarchOfEmpires etc. An attempt to stop these updates from happening has been made by using  Group Policy Editor, navigating to Computer Configuration > Administrative Templates > Windows Components > Cloud Content and enable policy to "Turn off Microsoft consumer experiences".   Situation will be monitored.

2018-10-27
AllSky operations appeared to go offline around  2018-10-27 14:09  (no updates to Website after this time).  Investigating showed the Computer had stopped working for no identifiable reason at 2018-10-27 14:12.   There were no associated event messages under Maintainance /  'Reliability Review'.  Reliability report indicates that the Computer unexpectedly shutdown at around 13:52 .

Computer was restarted and Full operations were resumed at  2018-10-28 00:12 (Outage 10 hours).  The previous uptime lasted 24 hours. (1.0 day).
It was noticed that Micrsoft App Updates (microsoft.stickly notes, ZoneVideo, BubbleWitch32Saga, CandyCrushSodaSaga) are still happening despite the modification to Policy Editor.     App Updates were finally turned off from the Microsoft Store. This done by clicking on the '3 dots' on right hand side of Store window, selecting 'Settings' , then setting 'Update apps automatically' to OFF.

2018-11-04
AllSky operations running smoothly.  WIndos App Updates have finally stoppedCurrent uptime is c. 7 days.

2018-11-xx
Pressure Data from Weather Station system noticed to be showing anomlously low values and finally went offline along with other Weather Station data (Wind, Temperature, Humidity). Checking the base station revealed that its screen had become very dim before dying altogether.

Problem was due to exhausted batteries in the base station. Pressure sensor is housed in the base station and the anomalous falling values can be used as a red flag indicator.   Eventually bought new batteries  (4 x AA) and installed them in the base station. Ggot the station rebooted and the connection to the USB receiver at the AllSky/Weather Laptop.    Weather data flow resumed at 2018-11-xx  (Weather Data outage of 24 hrs)

2018-11-14
Data Communications to/from observatory went down for 30 minutes period at 2018-11-14  17:35 - 18:25.  No remote connections to the Observatory computers at this time.  Weather Data/Cloud Sensor data not displayed on AllSky products from this time. Data not uploaded to website.

Communications and dependant services resumed by themself (LAN outage 30 minutes)

Apart from short data comms and weather station outages, the AllSky/Weather operations have been running continuosly now for over 2 weeks

Part of Night Summary Log from 2018-11-14
showing outage in Rain,  DPP (Temperature & Humidity), Clairty and Wind data
Image

2018-11-24
Live Cloud Sensor, Weather Station and Forecast Weather data feeds dropped out at 2018-11-24 15:02 .  Weather Data/Cloud Sensor data not displayed on AllSky products from this time. Production of graphs by the Weather Server Program also stopped at this time.  Investigating showed that the Weather Server program has apparently crashed

Problem was asscociated with the introduction of new ObsEnc data source & graphs where data is read from file to a maximum of 12,000 data records (the size limit of of array) but due to a bug a mismanagement of array position pointer led to graph routines attempted to read a record beyond the linits of the array. The problem was nither prevented or trapped. On spotting the problem the  data file was temporarily reduced by removing older data, pending a corrective fix and the program restarted at 17:22  (Data Outage 2h 20m)

[ The routine will be modified so that for data between 1.5 and 5 days old  (stored in a file at 30s timesteps) will be only be incorporated into data array at a frequency of only 1 point every minute (instead of 2 points per minute). In addition the code will be wrapped within a Try .. Catch ... End Try code structure and management of the array position pointer corrected
- holding 5 days worth of data at 1 record every 30s requires a 14400 array size.  
- holding 1.5 days woth of data at 1 record every 30s plus 3.5 days at 1 record every 60s would require a 9360 array size only ]

Apart from short data comms ,  weather station and this weather server outages, the AllSky/Weather operations have been running continuosly now for nearly 4 weeks.
(Unfortunately its been mainly cloudy with frequent rain through this time, leading to limited numbers of stars on AllSky Images, and few Observing opportunities)

Back to Top

This Web Page: Notes - Session 617 (2018-08-07)
Last Updated : 2023-11-27
Site Owner : David Richards
Home Page : David's Astronomy Web Site