David's Astronomy - Notes 2021-10-13 (S924)

Session Aims & Highlights

- Observing Result
- Night Summary Plot
- Session Event Log

Operational Issues

- Critical Issues (0), Major Issues (0), Minor Issues (2), Small Defects (3), Continuous Improvement (14)

Images from 2021-10-13 >> [ Local Files >> ]

2021-10-14

Investigation - AstroPlan stopped uploading files to website

  Forecast                              2021-10-13 20:19  (Local)
  Minimum Temperature         Ok        9.4°C             (Min. Temperature between now and sunrise)
  Temperature Change (High)   Ok        0.0°              (Temperature change between now and sunrise)
  Temperature Change (Low)    Ok        -2.6°
  Maximum Wind Speed          Ok        11.1 mph          (Max. Wind speed between now and sunrise)
  Maximum Wind Gust           Ok        17.8 mph

2021-10-14

Issue
AstroPlan stopped uploading files from Observatory Computer to website (01:31)

Description
AstroPlan stopped uploading files from Observatory Computer to website at around 01:31. Uploading of ObsCam and Alll Sky pictures from AllSky/Weather Computer continued however and it was the morning ObsCam picture which was useful in showing that the Dome was shut and parked, that the tlescope was successfully parked in normal position and the night's session had returned to General /Manual mode (again consistent with normal operations).

This is ongoing occassional issue. Need to look at log and report files to see if it sheds a new light on the problem.

Analysis
Report files shows that the last FTP upload was at 01:31:13. Log file show first exception at this time.

2021-10-14 01:31:15.33 | Exception in UploadFileFTP (outer catch) for Session.LatestReport.htm 2021-10-14 01:31:15.33 | Error: Unable to Upload File : C:\Data\CCD Imaging\Live\Session.LatestReport.htm 2021-10-14 01:31:15.33 | Error: The process cannot access the file 'C:\Data\CCD Imaging\Live\Session.LatestReport.htm' because it is being used by another process.

This was then followed by an ongoing series of exceptions like the following until AstroPlan was eventually manually restarted in the morning at 09:02.

2021-10-14 01:31:45.35 | Exception in 'UploadFileFTP2' whilst getting RequestStream for Session.LatestLog.log 2021-10-14 01:32:15.36 | Exception in 'UploadFileFTP2' whilst getting RequestStream for Session.ObservatoryReportFlag.dat 2021-10-14 01:32:15.36 | Error: The operation has timed out.

There were no messages from the FTP Overseer which is meant to be monitoring the performance of the FTP Uploader with checks to see if loader has stalled at all. The FtpOverseer thread would have started at Program startup and should have still been operating at the time.

It would seem that the FTP Queue was operating ok, but was failing to achieve success uploads. The FtpOverseer currently only spots problems where the FTP Queue actually activity stops for more than 60s. It can not currently spot issue where queue is active, but upload attempts are all failing) . The Session.LatestReport.htm (28 KB in size) was last uploaded at 01:31:00 in 2.65s. It would have probably have loaded again at around 01:31:23 or so.

With the ongoing failure to fix the root cause it is proposed to add a new operational check in AstroPlan to not only recognise that uploading has stalled, but to operate a procedure/process that closes and then automatically restart AstroPlan (a manual restart of AstroPlan is so far the only known workaround to the issue). This is raised as a continuous improvement ticket (see below).
Fix attempted but monitor for future occurences (AstroPlan 1.25.1)

Conclusion
This is still an ongoing occasional issue. Further action required

Actions
With the ongoing failure to fix the root cause it is proposed to add a new operational check in AstroPlan to not only recognise that uploading has stalled, but to operate a procedure/process that closes and then automatically restart AstroPlan (a manual restart of AstroPlan is so far the only known workaround to the issue). This is raised as a continuous improvement ticket (see Operational Issues section above for details). Following actions identified and raised as ticket items (see Operational Issues section above for details)

Add monitoring loop in AstroPlan to restart AstroPlan automatically in the event that Uploading has stalled.

Update (2022-05-16)
Issue is significantly less frequent now (2022-05-16), and is considered to be manageable and can be closed.

This Web Page:

Notes - Session 924 (2021-10-13)

Last Updated :

2024-09-30

Site Owner :

David Richards

Home Page :

David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 924 (2021-10-13)

Session Aims & Highlights (2021-10-13)

Operational Issues (2021-10-13, S924)

2021-10-14

Investigation - AstroPlan stopped uploading files to website

David's Astronomy Pages Notes - Session 924 (2021-10-13)

Session Aims & Highlights (2021-10-13)

Operational Issues (2021-10-13, S924)

2021-10-14

Investigation - AstroPlan stopped uploading files to website

David's Astronomy Pages
Notes - Session 924 (2021-10-13)