Issue.    Loss of Ethernet Connection to/from Observatory before, during and after session S1074 (2022-12-20 & 2022-12-21)

1) Unable to make VNC Connection from or open Windows Explorer Views from Development Computer to either Observatory Computer or the AllSky/Weather Computer (pre-session, 2022-12-20 16:00)

2) Observatory and AllSky/Weather Computers stopped updating website less than 2 hours into Session (20:15).

3) Observatory remains non-operational due to the Observatory's Ethernet Connection being down or unreliable (2022-12-21)

Background
Connection to the household LAN and to the Internet is normally provided by 'Ethernet over Powerline', and is generally very reliable in comparison with WIFI connectivity to the Observatory which is normally unreliable and very slow/problematic.   LAN in critical to Observatory Operations to allow i) TCP/ IP Communication between Observatory Computer and AllSky/Weather Computer ii) VNC Connections for Remote Monitoring of Observatory Operations,  iii) File transfer to/from Observatory (software updates, targets database handover, session results),  whilst Intenet access is critical for  iv) Uploading of files and data to Observatory's WebSite for Remote Monitoring, and v) Download of weather forecast data including minutely rain radar based precipitation forecast.

Description

1) Before the S1074 session at around 16:00 the VNC connection from the Development (House) Computer to the Observatory Computer went down and couldn't be readily restored.  Also unable to connect to AllSky/Weather Computer or successfully ping either computer. Tried power-resetting Ethernet over Power Plugs at both router end and observatory end, but unable to restore connection.  Observatory Computer able to see files on the AllSky/Weather Computer.

- Some question over whether the Development Computer was able to see the Internet ok. 
- Unclear if files from Observatory are being uploaded to website or not (they are borderline overdue)
   (last VWS picture 15:35, Last Obs Status 15:43, last 60 min rain,  15:44 (later 17:10 but through weak WIFI link). 
- Rebooting Development Computer, rebooting Observatory Computer and rebooting Router didn't resolve the issue.
- Power resetting Ethernet over power plugs (both ends) and power-resetting Observatory Ethernet Switch didn't resolve the issue. 
- By passing the Ethernet Switch didn't resolve issue. 
- Ethernet over Power confirmed to be working in the house.  
- Ethernet over power plug taken from Observatory confirmed to work when located inside the house. 
- Employing a different (working) Ethernet over Power plug in the observatory didn't resolve the issue. 
- WIFI connection to Router working ok (one user querying about the quality of WIFI connection ?).  
- The Observatory and AllSky Computers remained in communication to each other up until 04:30 which is presumed to be the point where the Ethernet Hub in the Observatory was reset in a vain attempt to restore connectivity with the house. 
- Ethernet connectivity restored by itself at around around 18:10 or so. Thinking that the problems were fully resolved a live Observatory Session was kicked off at 18:28 using AuoStart.

2) Observatory and AllSky/Weather Computers stopped updating website less than 2 hours into Session (20:15).  At the same time there was a return of the pre-session problem whereby both the Development Computer and IPad couldn't make a VNC connection to either of the computers in the observatory.   As weather was fine and forecast to be remain so until the follwoing day, the Observatory was allowed to operate under total automonous control. Observatory was checked in the morning and everything was closed and parked up as expected. Checking the Computer showed that the session had run successfully and data had collected the whole night.

Whilst the session was succcessful it is a nevertheless a major issue that the status of the session and any alerts weren't available for the majority of the session.  Connectivity problem prevented the download of data from the session until two days later (2022-12-22) when a temporary powerline was put out to the observatory and the Observatory's Ethernet over PowerLine plug attached to it.

Analysis and Remediation

2022-12-21
Attempts were made on 2022-12-21 to use WIFI connection to return some operational capability back or at least to help diagnostics, but the WIFI connection was not reliable enough or fast enough to permit any more than occasional use. This was consistent with past experience and it why Ethernet over Powerline was adopted for the Observatory Connectivity,

Since it had been pretty much confirmed that Router and Ethernet over Powerline were working ok within the house,  that Ethernet Over Powerline pugs and Ethernet Cables were working ok, and the Computer involved were all working ok it was beginning to look like that there was an issue with the Main Powerline between house and Observatory or with the electrics in the Observatory. 

2022-12-22 am
A temporary power cable was laid out between the house and the observatory the following day (2022-12-22).  The plug end was connected to a plug socket in house/garage that is directly adjacent to the socket used for the main powerline to the Observatory), whilst the EtherNet over PowerLine plug was connected to the socket end located in the Observatory.  Turning on power, and checking the Observatory Computer it was confirmed that Ethernet over Powerline is working.    With the temporary powerline in place, data and results from the session was finally downloaded from the Observatory at 2022-12-22 11:00.

2022-12-22 pm

The following observations were evident following the basic investigations and observations :

- Router was working ok. All computers were working ok.
- Power to the observatory is operating ok through the main powerline. All devices operating normally.  ~
- 'Ethernet over Powerline' is working ok within the house
- The 'Ethernet over PowerIine' plugs themselves are functioning ok
- Ethernet cables seem to be all functioning ok
- 'Ethernet over Powerline' to Observatory works ok through temporarily laid powerline
- 'Ethernet over Powerline' has previously operated ok through the main powerline, and worked again for the nearly two hour period on 2022-12-20 (18:30 to 20:15)

It would seem that there is some sort of interference issue with the power circuit to the observatory or within the observatory itself ? 

Began a series of random tests and eventually got the Ethernet working over Observatory's Powerline again for a short time (60 minutes) before it went down again.   Triggeing the resumption may or may not have been a result of temporarily disabling the Zigbee Network, removing the Conbee II USB stick and turning off the Hue Smart Plugs in the AllSky/Weather Cabinet (Pulsar Dome, Dehumidifier, AllSky Heater and Aux Light).   Ethernet was up for over an hour but then went down again at around 14:25.  

Got Ethernet running again at 15:50  (again this was after disabling Zigbee Network but again it was unclear if this was critical or it was the effect of turning off/turning on the Ethernet over Powerline Plug (probably not) or leaving it off for 5 minutes and only then turning it back on again (but later on showed that turning Ethernet OverPowerline at router end for 10 mins didn't resolve issue)

The temporary Powerline whilst useful for diagostics is not viable workaround even for a short period as it mean leaving Observatory Door and Garage Door open.

So what could be preventing reliable use of the Observatory Main Powerline

 - Is it due to issue in the powerline to Observatory, a small short of something. ? (But has been working and still works at least part time, so not a terminal failure )
 - Could it be an intermittant issue with Ethernet Hub in Observatory.  However it was running ok with the temporary powerline and works ok for at least some of the time when using ethernet over Observatory's main powerline.  (Could test by connecting Observatory Computer directly to the Ethenet PowerLine plug ?)
 - Could it be related to ground currents ? (powerline to observatory runs underground and could be affected)
 - Could it be related to power glitches in the power-supply ? (Powerline to Observatory carries lower voltage due to its distance and might be more exposed to a weakness or instability in the supply
-  Could it be related to Power Socket issue on one particular socket in the kitchen whose switch has become stuck in last few days (could it be weakening the ethernet over powerline signal just enough to cause the issues). 
- Could it be related to Zigbee Network  (But Zigbee Network has been running with problems for the Ethernet Connectivity in the past 8 weeks). Could tTest by disabling network, regain Ethernet connectivity)
- Could it be a Kaspersky Firewall issue. (Ethernet again went down at 16:08.  Connection to both computers is lost at the same time ?, so why would Kaspersky kick in at the same time ?)
- Issue is unrelated to AllSky Heater/Dehumidifier, as the power to both Smart Plugs was turned off at time of the 16:09 disconnection).

Disabled Zigbee Network at 16:40 (pulling out the Conbee II USB stick) but it didn't immediately resolve problem,

With the Zigbee still disabled, turned off Power Sockets using Zigbee Smart Plugs (Pulsar Dome, Dehumidifier, AllSky Heater and Aux Light) and reset the Ethernet Over Powerline  Adapter, and eventually got Ethernet Connectivity back again (unclear what the precise critical factor was).    Left the Zigbee Network and associated plugs disconnected, and monitored the situation over the next 12- 18 hours using a combibation of green Ping flags in AstroMove and up to date timestamps on  AllSkyWeather and Observatory charts on Website.

2022-12-23
a)  Ethernet Connectivity is working ok with the Zigbee Network disabled and with plugs to Pulsar Dome, Dehumidifier, AllSky Heater and Aux Light all turned off  (15 hour test)

b) Dome noted to be a couple of degrees from its standard park position (Az 90) meaning that Induction Charging was also certainly compromised.  Could this have set up current fluctuations that   caused problems to the 'Ethernet over PowerLine' signal . Dome moved to its correct Park position and resynced it to Az 90 using DeviceHub's Dome Tab.

c)  Ethernet Connectivity is working ok with the Zigbee Network Connected and with power connected to Aux Light's smart plug  and with power connected directly to Pulsar Dome (ie without Smart Plug)   (6 hour test)

c)  Ethernet Connectivity is working ok  with the Zigbee Network Connected and with power connected to the  Aux Light's, Dehumidifier's and AllSky Heater's smart plugs and with power connected directly to Pulsar Dome (ie without Smart Plug)   (4 hour test)

d) Ethernet Connectivity issue returned after 35 mins with the Zigbee Network Connected and with power connected to the Aux Light, Dehumidifier, AllSky Heater and Pulsar Dome via smart plugs  (started at 22:15, Ethernet went down at 22:50 as part of planned 9 hour test, but came on before midnight

2022-12-24.
a) Added a LAN Ping monitor to AstroMove 1.9.9 to record when computers are reachable with a ping and when they are not.
b) No outages in Observatory Ethernet

2022-12-25.
a) No outages in Observatory Ethernet

2022-12-26.
a) Added a LAN Speed Check to measure LAN performance and help in the identification of improvements to improve LAN performance & speed

b) Measurements identified that earlier relocation of Observatory Ethernet Over Powerline adapter has a deterimental affect on LAN performance
-  Av Speed with adapter in socket 3  :    5.78 Mbps, 0.72 MB/s (Laptop9),    4.58 Mbps, 0.57 MB/s (Laptop7)
-  Av Speed with adapter in socket 1  :  19.72 Mbps, 2.47 MB/s (Laptop9),  23.91 Mbps, 2.99 MB/s (Laptop7),   VNC to Laptop9 on
-  Av Speed with adapter in socket 1  :  23.67 Mbps, 2.96 MB/s (Laptop9),  23.84 Mbps, 2.98 MB/s (Laptop7) ,  VNC to Laptop9 off

c) No outages in Observatory Ethernet

Findings & Conclusions

The exact cause of the issue before, during and after session S1074 is uncertain.

 - It could have been due to a temporary issue with the Power Supply (spikes, lowered voltage ?)
 - It may have been due to the interference effect of a particular fluctuation in current supply due to power induction charging of the Dome Shutter when it was sitting at a particular offset (but this hasn't been previously seen)

An important finding was that even if the LAN connection to house computer and to internet does down the normal LAN communications between Observatory and AllSkyWeather Computers wasn't interrupted. This is significant and means that a session can be expected to and be permitted to continue as the session can still be conducted safely based on the Observatory Cloud Sensor and AllSky Camera. The chance of success session is probably similar.  The one area of slightly enhanced risk is that the usually available 60 min precipitation forecast from radar wouldn't be available unless it can still be gleaned over wifi.

Actions
  Make a number of small fixes and improvements to AstroMain Zigbee reporting & ObsViewer picture drawing.

  Keep Observatory's Ethernet over Powerline adapter on Socket 1 (one that is closed to observatory's power supply input).

Issue deemed to be resolved 2022-12-25.  No LAN or Ethernet connection issues during sessions S1075 & S1076 (2022-12-26 & 2022-12-27).

Update:
There was a further period of interruption to LAN/Router & Internet Connection in Sept 2024 during which there were 5 outages within a 24 hour period (see Investigation - Interruptions in LAN & Internet Connectivity, 2024-09-08 ). An investigation concluded that user attempts to fix/shortern the outages probably didn't work and that certain actions like resetting the Observatory's Ethernet Switch can actually lead to more problems. It is probably best to leave thing to resolve themselves and only intervene if connection is still down after 24 hours. An improvement was made in AstroMain 3.69.6 to provide better recording & alerting of LAN/Internet Outages.  A proposal was made to consider installing a dedicated Ethernet Cable between Observatory and House/Router.

Update 2024-10-19
A Cat 6 Ethernet Cable was finally installed between Router and Observatory's Ethernet Switch on 2024-10-18 / 2024-10-19. As well as the expected better reliability intial speed tests show that the Cat6 Wired Ethernet Connection to Observatory is 4x faster than the preceeding Ethernet Over PowerLine Connection. For more details see Observatory - Wired Ethernet Cable Connection, 2024-10-18/19 .

Back to Top