David's Astronomy Pages
Notes - Session 877 (2021-03-29)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
 
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (0),  Minor Issues (3),  Continuous Improvement (2)
Bullet Images from 2021-03-29 >>     [ Local Files >> ]
Bullet Investigation - AstroPlan's FTP Uploading Stalls

Session Aims & Highlights (2021-03-29)

Main aims

  1. Targets. Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by conditions
  2. New Cable Routing.  Continue nighttime testing of new cable routing for SBIG power cable and ZWO USB 3 cable.

Equipment & Software

Highlights

Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Dec Notes :
Run 3 (black) impacted by cloud/lost star on final frame
Run 4 (black) failed due to cloud/lost star
Run 14 (black) impacted by anomalous Dec Jumps
Run 5 (red) impacted by periods of ineffective Dec guiding
Run 7 (red) impacted by cloud/lost star
Image
Image
  
Sky Conditions (Locate Frames)
Image
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Image   
  
Pre-Session Weather Forecast (API data from darksky.net)
Image
Image   
  
Session Event Log
Time     Event Detail
21:07:59 Session Monitoring AutoStart monitoring for Live Session opportunity starting between 21:07 & 04:44
21:08:01 Session AutoStarting Session autostarting (21:08)
21:08:33 Session Created Live Session Created (2021-03-29 S00877, ImageSaveNum: 877001)
21:08:36   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
21:10:35   Services Started Observatory Services started
21:10:43 Observatory (Auto) Observatory placed in Fully-Automated Mode
21:10:46 Session Pending Session pending (2021-03-29)
21:10:48 Session Initiating Session initiating (2021-03-29)
21:10:50   Plan Requested Observing Plan requested from AstroPlan (1.21.4)
21:11:38   Plan Loaded Observing Plan loaded to queue (, Plan ID: 615)
21:11:54   Camera1 Connected SBIG Camera Connected (set point -10°C)
21:12:00   Telescope Connected Telescope Connected (TheSky6)
21:12:11 Session Equilibration Session ready for dome & camera equilibration
21:12:57   Dome Opened Dome opened (opening time 46s)
21:13:47 Session Running Session running
21:13:50   Queue Started Observing Queue started (29 targets selected)
21:15:05       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:18:38       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
21:20:42       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
21:21:10       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
21:23:33       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
21:25:23       Focusing Completed Foc2 AutoFocus Completed (Profile No 2)
21:25:39       Focusing Checked Foc2 Focusing Check Succeeded
21:25:46     Target Started (1/29) Target started (1/29, AT2021dbp (CV))
21:29:26       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:31:20       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
21:37:06     Target Completed Target completed (1/29, AT2021dbp (CV))
21:37:38     Target Started (2/29) Target started (2/29, AT2021ckv)
21:41:15       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:43:19       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
21:49:03     Target Completed Target completed (2/29, AT2021ckv)
21:50:00     Target Started (3/29) Target started (3/29, IC 357 w/AT2021fyd)
21:53:44       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
22:11:27     Target Completed Target completed (3/29, IC 357 w/AT2021fyd)
22:11:53     Target Started (4/29) Target started (4/29, AT2021gyc (CV))
22:17:18       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
22:24:00     Target Completed Target completed (4/29, AT2021gyc (CV))
22:24:02     Target Started (5/29) Target started (5/29, UGC 3502 w/SN2021cwc)
22:27:09       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
22:43:56     Target Completed Target completed (5/29, UGC 3502 w/SN2021cwc)
22:46:06     Target Started (6/29) Target started (6/29, GCVS ER UMa)
22:49:27       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-S)
22:54:43     Target Completed Target completed (6/29, GCVS ER UMa)
22:56:24     Target Started (7/29) Target started (7/29, NGC 3310 w/SN2021gmj)
22:59:14       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
23:17:03     Target Completed Target completed (7/29, NGC 3310 w/SN2021gmj)
23:18:16     Target Started (8/29) Target started (8/29, UGC 5829 w/LBV2021blu)
23:20:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:23:23       Focusing Completed Foc1 AutoFocus Completed (Profile No 5)
23:39:30     Target Completed Target completed (8/29, UGC 5829 w/LBV2021blu)
23:40:08     Target Started (9/29) Target started (9/29, NGC 4227 w/SN2021biz)
23:42:51       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:45:01       Focusing Completed Foc1 AutoFocus Completed (Profile No 6)
00:00:54     Target Completed Target completed (9/29, NGC 4227 w/SN2021biz)
00:02:01     Target Started (10/29) Target started (10/29, UGC 5760 w/SN2020zoq)
00:04:43       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:06:49       Focusing Completed Foc1 AutoFocus Completed (Profile No 7)
00:22:50     Target Completed Target completed (10/29, UGC 5760 w/SN2020zoq)
00:24:22     Target Started (11/29) Target started (11/29, UGC 6930 w/SN2020rcq)
00:26:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:28:53       Focusing Completed Foc1 AutoFocus Completed (Profile No 8)
00:39:46     Target Completed Target completed (11/29, UGC 6930 w/SN2020rcq)
00:41:44     Target Started (12/29) Target started (12/29, AT2021bgi)
00:44:30       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:46:44       Focusing Completed Foc1 AutoFocus Completed (Profile No 9)
01:02:50     Target Completed Target completed (12/29, AT2021bgi)
01:03:36     Target Started (13/29) Target started (13/29, Nova Per 2020 (CV))
01:07:11       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:09:03       Focusing Completed Foc1 AutoFocus Completed (Profile No 10)
01:14:03   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
01:14:23   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions
01:17:39     Target Completed Target completed (13/29, Nova Per 2020 (CV))
01:18:28     Target Started (14/29) Target started (14/29, NGC 3515 w/AT2021fmd)
01:21:29       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:23:55       Focusing Completed Foc1 AutoFocus Completed (Profile No 11)
01:39:42     Target Completed Target completed (14/29, NGC 3515 w/AT2021fmd)
01:44:43     Target Started (15/29) Target started (15/29, MCG +6-30-84 w/SN2021K)
01:47:11   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
01:47:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:49:31       Focusing Completed Foc1 AutoFocus Completed (Profile No 12)
01:58:07   SoftSuspend Cancelled Soft Suspend is cancelled due to Improved Conditions
02:03:38   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (too few stars)
02:05:07     Target Completed Target completed (15/29, MCG +6-30-84 w/SN2021K)
02:05:09   Queue Paused Queue paused For Session Suspension
02:05:11 Session Suspended Session suspended
02:05:53   Dome Closed Dome closed (closing time 45s)
02:21:34     Target Cancelled Target cancelled (16/29, LEDA 1634284 w/SN2021bkw) due to other condition
02:43:26     Target Cancelled Target cancelled (17/29, NGC 4165 w/SN2021gno) due to cloud
02:55:14     Target Cancelled Target cancelled (18/29, GCVS AM CVn) due to cloud
02:57:38   UPB.Switch Restart UPB.Switch has been restarted.
03:04:35     Target Cancelled Target cancelled (19/29, GCVS SS Her) due to cloud
03:14:13     Target Cancelled Target cancelled (20/29, GCVS CY Lyr) due to other condition
03:14:54 Session Resuming Session resuming
03:15:39   Dome Opened Dome opened (opening time 45s)
03:15:41 Session Running Session running
03:15:43     Target Started (21/29) Target started (21/29, 61 Cyg)
03:15:45   Queue Resumed Observing Queue resumed
03:19:30       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:21:12       Focusing Completed Foc1 AutoFocus Completed (Profile No 13)
03:22:33     Target Completed Target completed (21/29, 61 Cyg)
03:22:35     Target Started (22/29) Target started (22/29, GCVS BL Lac)
03:25:40       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:27:43       Focusing Completed Foc1 AutoFocus Completed (Profile No 14)
03:31:31     Target Completed Target completed (22/29, GCVS BL Lac)
03:32:35     Target Started (23/29) Target started (23/29, GCVS U Her)
03:36:02       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:37:54       Focusing Completed Foc1 AutoFocus Completed (Profile No 15)
03:45:14     Target Completed Target completed (23/29, GCVS U Her)
03:45:16     Target Started (24/29) Target started (24/29, NGC 4631 w/AT2021biy)
03:49:37       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-S)
04:07:11     Target Completed Target completed (24/29, NGC 4631 w/AT2021biy)
04:07:13     Target Started (25/29) Target started (25/29, C/2020 T2 (Palomar))
04:10:21       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
04:25:12     Target Completed Target completed (25/29, C/2020 T2 (Palomar))
04:25:14     Target Started (26/29) Target started (26/29, GCVS R CrB)
04:28:13       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
04:29:52       Focusing Completed Foc1 AutoFocus Completed (Profile No 16)
04:32:48     Target Completed Target completed (26/29, GCVS R CrB)
04:33:31     Target Started (27/29) Target started (27/29, 7P/Pons-Winnecke)
04:36:28       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
04:38:42       Focusing Completed Foc1 AutoFocus Completed (Profile No 17)
04:52:46     Target Completed Target completed (27/29, 7P/Pons-Winnecke)
04:52:49     Target Started (28/29) Target started (28/29, AT2021hdk (CV))
04:55:26       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
04:57:29       Focusing Completed Foc1 AutoFocus Completed (Profile No 18)
05:03:02     Target Completed Target completed (28/29, AT2021hdk (CV))
05:04:20     Target Started (29/29) Target started (29/29, HIP 87937)
05:06:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
05:08:15       Focusing Completed Foc1 AutoFocus Completed (Profile No 19)
05:10:31     Target Completed Target completed (29/29, HIP 87937)
05:10:34   Queue Completed Job Queue completed
05:10:37 Session Closing Session closing
05:11:30   Dome Closed Dome closed (closing time 50s)
05:12:09   Dome Parked Dome parked (parking time 30s), Az: 90.0 deg
05:12:41   Telescope Parked Telescope parked (parking time 28s)
05:13:12   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
05:14:02   Services Stopped Night Services stopped
05:14:04 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files)
05:14:26 Session Finished Session Finished
 
Session Alerts
Time     Alert Detail
22:34:09 Dome Dome has been stationary for more than 10 mins
22:46:10 Dome Dome Movements have begun again (outage 22.1 mins)
22:47:52 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (GCVS ER UMa)
22:57:44 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (NGC 3310)
02:57:37 UPB.Switch disconnected UPB.Switch has unexpectedly disconnected. Restart will be attempted
02:57:40 UPB.Switch UPB.Switch has been successfully restarted.

Back to Top


Operational Issues (2021-03-29 S877)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Continuous Improvement

[ Prev | Next ]

Back to Top


Investigation - AstroPlan's FTP Uploading Stalls

Issue:   Uploading of session images and files to website by AstroPlan's FTP File Uploader sometimes stalls.

Description.  AstroPlan application has a side role of uploading various session images and files to the website so that the status of a live session can be conveniently established from a single online web page ( Current/Latest Session ).  There has been an ongoing issue whereby every few sessions the uploading stalls, and the page now contains fresh information, and insstead of checking in with this one page, a VNC connection has to be opened to the Observatory Computer to check on the status of a session.

The stall is silent, in that AstroPlan just stops uploading, other AstroPlan facilities and functions work normally.    Extra tracing was added in AstroPlan 1.21.4 (2021-03-18).

In case of the occurrance of the issue during the latest session (S877, 2021-03-28), uploading of session files to website by AstroPlan FTP Uploader stalled at 03:09.

Analysis.    Extracts from AstroPlan log file from 2021-03-28 are shown below:

2021-03-29 10:14:08.60 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-29 10:13:07
2021-03-29 10:14:08.60 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-29 10:14:09.67 | Exception in 'UploadFileFTP' whilst building/writing byte array for Session.ObsManagerTab.png
2021-03-29 10:14:09.67 | Error: Unable to Upload File : C:\Data\CCD Imaging\Live\Session.ObsManagerTab.png
2021-03-29 10:14:09.67 | Error: The underlying connection was closed: An unexpected error occurred on a receive.
2021-03-29 10:14:12.62 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-29 10:14:12

2021-03-29 17:52:47.32 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-29 17:51:45
2021-03-29 17:52:47.32 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-29 17:52:47.74 | Exception in 'UploadFileFTP' whilst building/writing byte array for Session.ObsManagerTab.png
2021-03-29 17:52:47.74 | Error: Unable to Upload File : C:\Data\CCD Imaging\Live\Session.ObsManagerTab.png
2021-03-29 17:52:47.74 | Error: The underlying connection was closed: An unexpected error occurred on a receive.
2021-03-29 17:52:53.36 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-29 17:52:52

2021-03-29 21:09:31.73 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-29 21:08:30
2021-03-29 21:09:31.73 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-29 21:11:59.43 | Exception in 'UploadFileFTP' whilst building/writing byte array for Session.TargetFrame09.jpg
2021-03-29 21:11:59.43 | Error: Unable to Upload File : C:\Data\CCD Imaging\Live\Session.TargetFrame09.jpg
2021-03-29 21:11:59.43 | Error: The underlying connection was closed: The server committed a protocol violation.
2021-03-29 21:14:49.67 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-29 21:14:49

2021-03-29 21:15:50.00 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-29 21:14:49
2021-03-29 21:15:50.00 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-29 21:15:56.04 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-29 21:15:55

2021-03-29 21:58:34.99 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-29 21:57:33
2021-03-29 21:58:34.99 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-29 21:58:36.00 | Exception in 'UploadFileFTP' whilst building/writing byte array for Session.Cam2TempChart.png
2021-03-29 21:58:36.00 | Error: Unable to Upload File : C:\Data\CCD Imaging\Live\Session.Cam2TempChart.png
2021-03-29 21:58:36.00 | Error: The underlying connection was closed: An unexpected error occurred on a receive.
2021-03-29 21:59:07.16 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-29 21:59:06

2021-03-29 22:49:14.65 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-29 22:48:13
2021-03-29 22:49:14.65 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-29 22:49:18.68 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-29 22:49:17

2021-03-30 00:11:59.22 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.Header.js
2021-03-30 00:11:59.22 | Error: The underlying connection was closed: The server committed a protocol violation.

2021-03-30 00:19:07.20 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.Cam2TempChart.png
2021-03-30 00:19:07.20 | Error: Unable to connect to the remote server

2021-03-30 00:19:20.87 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 00:18:20
2021-03-30 00:19:20.87 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 00:19:24.90 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 00:19:24

2021-03-30 02:41:51.86 | Exception in UploadFileFTP (outer catch) for Session.LatestReport.htm
2021-03-30 02:41:51.86 | Error: Unable to Upload File : C:\Data\CCD Imaging\Live\Session.LatestReport.htm
2021-03-30 02:41:51.86 | Error: The process cannot access the file 'C:\Data\CCD Imaging\Live\Session.LatestReport.htm'
                         because it is being used by another process.
2021-03-30 02:42:50.92 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 02:41:50
2021-03-30 02:42:50.92 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 02:42:51.88 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.LatestLog.log
2021-03-30 02:42:51.88 | Error: The operation has timed out.
2021-03-30 02:43:51.90 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.ObservatoryReportFlag.dat
2021-03-30 02:43:51.90 | Error: The operation has timed out.
2021-03-30 02:44:51.91 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.ObsManagerTab.png
2021-03-30 02:44:51.91 | Error: The operation has timed out.
2021-03-30 02:45:51.89 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.ServicesTab.png
2021-03-30 02:45:51.89 | Error: The operation has timed out.
2021-03-30 02:46:51.91 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.SkyConditionsTab01.png
2021-03-30 02:46:51.91 | Error: The operation has timed out.
2021-03-30 02:46:52.38 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 02:46:51
2021-03-30 02:47:51.92 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.Header.js
2021-03-30 02:47:51.92 | Error: The operation has timed out.
2021-03-30 02:47:52.75 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 02:46:51
2021-03-30 02:47:52.75 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 02:48:51.93 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.ObsPic.gif
2021-03-30 02:48:51.93 | Error: The operation has timed out.
2021-03-30 02:49:51.95 | Exception in 'UploadFileFTP' whilst getting RequestStream for event.dat
2021-03-30 02:49:51.95 | Error: The operation has timed out.

..... Multiple timeout exceptions...  (not shown here)

2021-03-30 03:09:52.17 | Error: The operation has timed out.
2021-03-30 03:09:52.84 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 03:09:52
2021-03-30 03:10:52.18 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.Header.js
2021-03-30 03:10:52.18 | Error: The operation has timed out.
2021-03-30 03:10:53.21 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 03:09:52
2021-03-30 03:10:53.21 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 03:11:52.20 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.ObsPic.gif
2021-03-30 03:11:52.20 | Error: The operation has timed out.
 
... Multiple timeout exceptions...  (not shown here)

2021-03-30 03:29:52.35 | Exception in 'UploadFileFTP' whilst getting RequestStream for Session.CcdTempChart.png
2021-03-30 03:29:52.35 | Error: The operation has timed out.
2021-03-30 03:30:21.07 | Closing Databases...
2021-03-30 03:30:21.10 | Closing Log Files...
2021-03-30 03:30:21.10 | Closing Log File

As well as  the terminal stall at 03:09 (21m+, program was closed before the queue resumed),  there were a number of shorter stalls that occurred at 10:14 (1m 5s duration),  17:52 (1m 7s duration),    21:14 (1m 6s duration),  21:58 (1m 33s duration), 22 48 (1m 4s duration) &  00:19 (1m 4s duration), plus a number of somewhat long stalls that occurred at 21:09 (6m 19s duration), 02:41 (5m 1s) &  02:46 (23m 1s).

Reporting of stalls is consistently made from FTP Overseer and nowhere else.   The messages consistently report that it is the 'FTP Queue Loop' that is stalled and that the stall point is 'inside UploadFileFTP2'.

A large proportion of 'stalls' are in range between 1m 4s &  1m 7s in duration. This suggests something about the frequency at which stall checks are made and the natural range of ftp upload timings.

The longer stall at 21:09 (6m 19s duration) might be connected to the creation of the night's observing plan around this time.  The log file shows an unexplained delay between 21:08:29 (logging of Sun Data) and 21:10:50 (logging of Moon Data)  

The stall at 21:14 (1m 6s duration)  follows directly on from that 21:09 (6m 19s duration) - no files are uploaded in the 0s interval between the first resuming and the second one starting.
Similarly the stall at  02:46 (23m 1s) follows directly on from stall at 02:41 (5m 1s).   Similarly the stall at 03:09 02:46 (21m+) follows directly on from stall at 02:46 (23m 1s)

The exception at 00:19:07 (exception whilst getting RequestStream for Session.Cam2TempChart.png) may or may not share a relation with an exception in AstroMain shortly beforehand at 00:18:37

00:18:37.64 | UGC 5760 DrawLegend Cam2Temp Fail | Exception in Cam2TempChart.DrawLegend (Mode: 1)
00:18:37.64 | UGC 5760 DrawLegend Cam2Temp Fail | Object is currently in use elsewhere.

Of the 6 exceptions  "Exception in 'UploadFileFTP'"  prior to the terminal stall ( should really be in 'UploadFileFTP2'") they were split as follows
    whilst building/writing byte array for file ....   (x4)
    whilst getting RequestStream for file ....  (x2)

The terminal stall at 03:09 is followed by series of exceptions  'Exception in UploadFileFTP' whilst getting RequestStream for file xxx .  The operation has timed out'
Since this exception type began to be logged from 02:42:51, just after the Stall at 02:41:50  (but not before then), it is probably appropriate to say the terminal stall began from 02:41:50.

The stall at 02:41:50 (logged at 02:42:50) appears to be immediately preceeded by an Exception in UploadFileFTP (outer catch) at 02:41:51.86:
     'Unable to Upload File Session.LatestReport.htm because it is being used by another process'.  

It is unclear whether this triggered the terminal stall or not.  Timing uncertainties are such that it could have been immediately succeeded by the exception.   Other shorter stalls seemed to associated with 'Unable to Upload File' exceptions , but they didn't lead to a terminal stall.  However the errors given for these other cases was different: 
     'The underlying connection was closed: An unexpected error occurred on a receive'. or 
     'The underlying connection was closed: The server committed a protocol violation'.
and were posted after the stall was logged rather than directly beforehand.

Later Cases
After restarting AstroPlan there were further reported short stalls in the FTP Queue

2021-03-30 03:32:11.92 | Starting 'FtpQueueThread'... (i.e. RunFtpQueue)
2021-03-30 03:32:11.94 | FTP Overseer >> FTP Queue Loop stalled at 0001-01-01 00:00:00
2021-03-30 03:32:11.94 | FTP Overseer >> Stall point is outside UploadFileFTP2
2021-03-30 03:32:11.94 | Executing RunFtpQueue...
2021-03-30 03:32:14.03 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 03:32:13
2021-03-30 03:32:16.15 | RunFtpQueue has finishing uploading a batch of 1 files

2021-03-30 03:33:24.47 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 03:32:24
2021-03-30 03:33:24.47 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 03:39:10.38 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 03:39:10

2021-03-30 03:40:10.70 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 03:39:10
2021-03-30 03:40:10.70 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 03:40:40.88 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 03:40:39

The first stall at 03:32 is a bug relating to checking SecondsSince against a NullDate which has been reset by the first passage through queue.  2nd stall had duration 6m 46s, 3rd stall had duration 1m 29s.

In later session

2021-03-30 17:57:20.85 | Starting 'FtpQueueThread'... (i.e. RunFtpQueue)
2021-03-30 17:57:20.87 | Executing RunFtpQueue...
2021-03-30 17:57:20.87 | FTP Overseer >> File Upload Loop stalled at 0001-01-01 00:00:00
2021-03-30 17:57:22.90 | FTP Overseer >> File Upload Loop resumed at 2021-03-30 17:57:20
2021-03-30 17:58:23.27 | FTP Overseer >> FTP Queue Loop stalled at 2021-03-30 17:57:22
2021-03-30 17:58:23.27 | FTP Overseer >> Stall point is inside UploadFileFTP2
2021-03-30 18:04:41.57 | FTP Overseer >> FTP Queue Loop resumed at 2021-03-30 18:04:41
2021-03-30 18:04:49.01 | RunFtpQueue has finishing uploading a batch of 124 files
 
2021-03-30 18:12:00.84 | Exception in 'UploadFileFTP2' whilst getting RequestStream for Session.ObsManagerTab.png
2021-03-30 18:12:00.84 | Error: The remote server returned an error: 227 Entering Passive Mode (173,254,28,204,175,117).

Earlier Stalls
Checking back at logs from Session S876 (2021-03-26) when there was no major stall in FTP Upload, shows 16 occurances of Short Stalls

Errors were:
 'The underlying connection was closed: An unexpected error occurred on a receive.' (x8)
 'Unable to connect to the remote server' (x3)
 'System error.' (x3)
 'The operation has timed out (x1)

Further Analysis
FTP upload request has a request.TimeOut set at 60,000ms  (60s) and FtpOverseer has a stall trigger point set to be 60s.  If one file upload attempt failed and held up execution for 60s and the next upload attempt was successful this would nevertheless be reported as a stall.    Proposal is to reduce request.timeout to 30,000 ms (30s)

Still unclear why FTP Queue Upload stops working ,and requests continuously timeout.  (Issue with server connection is understandable but is unclear why restarting the program resolves the issue.

FtpOverseer updated to recognise and report when more than 5 timeouts have occurred in a row, and take action to stop FTP Queue and then restart it again.

Actions

Update

Back to Top