Last Updated on October 25, 2021 by tkp
On September 29, with Robert Hantzsche’s help, the second microwave radio was replaced (at Table Mt). From there, it appeared there was still an issue on the Studio side that was limiting. The most likely issue would be the cable, as that radio had been replaced, and when replacing it it had been noticed that there were unexpected behaviors regarding the power, etc. Additionally, the same cable from the studio at Hersey St was used at the new location, and I had previously discovered that the cable at Table Mt had not been wired correctly, casting suspicion on the cable being used at the new studio.
On October 3, Dan Sullivan went up on the roof and we replaced the entire cable to the microwave dish, then tested it to make sure it was good. This was expected to solve the problems, and it did solve the unexpected cable behaviors, but, unfortunately there were frequent disconnects over the microwave link for about 5 hours – during which time I received many complaints despite the fact that I had warned folks about what was happening.
That prompted more messages to the community forum for the company that manufactures the radio, and from that I discovered this is not that unusual – it is not unusual for the radios to take hours to stabilize after their power has cycled, which happens whenever they have their cables disconnected. In addition, there could be issues with the new version of the firmware that we have installed on both radios.
Bottom line: it could take some time to resolve the issues entirely, but they will be resolved. In the meantime, in order to prevent having to listen to complaints, it is likely the transmitter will be switched to the backup USB drive for extended periods.
On October 5, I checked to see how many errors the Barix had accumulated since its last restart – which was at 2 AM (it is currently auto restarted at 2 AM). The status information showed 9 lost frames and 10 duplicated frames. This is in contrast to what was previously seen with approximately 20000 lost frames over 14 hours.
Also, a small change was made to the data rate, but this resulted in a restart, which then interrupted our transmission for a time while the device stabilized. Because I didn’t have infinite time to wait for the device to stabilize and I knew the old data rate worked, I changed it back – still we had to wait for the stabilization to occur (approx 3 – 6 PM).
On October 6, it was found that the microwave link had still not stabilized, so the power was cycled on the studio end radio. Then, again, it was necessary to wait for the network to stabilize. It stabilized enough by 8:45 PM (7 hours later), that it was possible to get a reasonable transmission. This remained into October 7, and continues. However, there are still errors occurring at a rate that is close to what it was prior to October 5 (on the order of 2,000/hour). So, more work is yet to be done…….
On October 9, the Studio side STL radio was shut down again as it was necessary to put in an uninterruptible power supply (UPS) – this radio is powered slightly differently than the old radio so the UPS’s in the studio do not power it as its power supply is in the utility closet in the building rather than the studio itself. At the same time the UPS was installed, the firmware on the device was downgraded two versions, which was recommended by other users of the equipment. The device was given a hard reboot at 11:24 AM, and now we wait for the network to stabilize……. in the meantime, the transmitter was switched to the backup.
On October 10, it initially looked like the network was still not stable. However, the situation stabilized by 11 AM, and as of 1:30 PM it looks very good. Although, again, more work needs to be done, for now I am not going to touch anything and see how long this lasts.
On October 13, after being completely stable for approximately 3 days (72 hours), the transmitter side radio restarted itself for some reason at approximately 10:15 AM, and this put the system back into an “unstable” state. All that can be done is to wait it out to see what happens – this will result in the station periodically cutting out.
On October 14, it was noted that the situation stabilized at approximately 2 AM; 16 hours after it started. It isn’t quite as good as before the previous hiccup, but is fairly good. Further work is still needed to ensure instability does not return.
On October 18, the radio at Table Mt rebooted itself 3 times after midnight (once near midnight, then at 3 AM and at 4 AM); most likely this was weather related. This caused network instability that persisted for hours. However, at 3 PM I was able to downgrade the microwave radio firmware 2 versions – recommended in the situation we have – and will now wait to see how much difference it makes…..
On October 19, as of 11 AM, there have been no Barix restarts as of 3:45 PM on the 18th, with the exception of the scheduled restart at 2 AM. The latency and packet loss are also very good so far. The next question is what happens when the system restarts due to something beyond our control. Only time will tell….. this will likely be tested in the next 24 hours based on the current weather forecast.
As of October 20 at 2:40 PM, the situation is still very good – the only Barix restarts over the past 2 days have been those scheduled at 2 AM. The network latency and packet loss are also very good. The device did not restart in the past day, despite the high winds and rain we have had over past 24 hours. Looks like we are in for several days of wind and rain, so it is likely there will be at least one restart over that period, and it will be interesting to see what happens then….
As of October 21, at 4:45 PM, there have still been no Barix restarts (other than those scheduled at 2AM) since October 18 – over 3 days. In addition, there has been fewer than one lost Barix packet every 2 hours since 2 AM today (when the counter was reset). This is quite good – the best I have seen in fact…… It still remains to be seen what happens when the inevitable reboot of the Table Mt STL radio occurs, and I am still thinking that will happen in the next couple of days based on the weather forecast…..
As of October 25, 11 AM, there has been only one Barix restart that was not associated with the 2 AM restarts, and this is despite very high winds, rain, and snow that have occurred over the past 3 days (highway cameras show snow at the elevation of the transmitter).
The graph below shows the maximum packet latency since 4 PM on Monday October 18 – approximately one week ago. It is pretty constant despite the weather; the mean (of the max) is 17 msec and the max (of the max) is 38 msec. Previously the maximum packet latency was well over 1000 msec:
