Monday, 27 February 2017

Using sFlowTrend to analyse IEEE 802.1ah (PBB or MAC-in-MAC) traffic

The Provider Backbone Bridges (PBB or MAC-in-MAC) standard IEEE 802.1ah defines an architecture and protocol that allows service providers to build large, scalable ethernet bridged networks, interconnecting multiple Provider Bridge networks without losing each customer's individually defined VLANs. It operates using a MAC tunnelling scheme in which a customer packet, including MAC addresses, is encapsulated in a new ethernet frame with new MAC addresses (the backbone bridge MAC addresses). This eliminates the need for backbone core bridges to learn all MAC addresses of every customer and provides complete separation of provider and customer domains. However, visibility of both the backbone traffic and the encapsulated customer traffic is important for troubleshooting configuration problems and managing performance. sFlowTrend (version 6.5 onwards) understands the IEEE 802.1ah frame format, decoding the outer backbone header and the inner customer frame. Here is an example of using sFlowTrend-Pro to gain full visibility of traffic in a PBBN.

The diagram below illustrates a typical IEEE 802.1ah PBB frame and shows the key fields used by sFlowTrend-Pro to represent the header fields. The sFlowTrend-Pro help gives a full list of the MAC, VLAN, priority, and IEEE 802.1ah key fields.


One way to view the details of traffic in a PBBN, is to use the Network > Top N tab and create a custom top N chart.



In this example we have build a custom top N chart showing the backbone header fields and the MAC and IP addresses and VLAN in the customer frame. Selecting this custom top N chart from the Chart selection list, generates a chart showing the details of the PBB traffic.





Using sFlowTrend to analyse tunnelled and encapsulated traffic

Layer 3/4 tunnels (Geneve, GRE, NVGRE, VXLAN) are often used to virtualise network services so that communication between virtual machines can be provisioned and controlled without dependencies on the underlying network. Hiding the physical network topology is a useful abstraction which offers a significant benefit of operational flexibility, however lack of visibility into the physical and virtual network can result in poorly placed workloads, inefficient use of resources and as a consequence, performance problems. sFlowTrend-Pro v6.5 provides the comprehensive visibility into tunnelled traffic which is essential for effective management of these more complex environments. Here is an example of how you can use sFlowTrend-Pro to understand and analyse tunnelled traffic.

sFlowTrend-Pro recognises VXLAN tunnelled traffic using the well known port UDP 4789. It then decodes the encapsulated packet in the UDP payload and stores the encapsulated packet header fields using key fields such as sourceAddress.1, destinationAddress.1 etc. It also records the VXLAN Network Identifier (VNI). The sFlowTrend-Pro help includes a section on L3/4 encapsulations which lists the key fields available for tunnelled traffic. One way to view a VXLAN tunnel is to Network > Top N tab and select the Top source-destination flows chart and then add a filter isVXLAN:


If you click on the source and destination address in the legend, you can also add the tunnel end points to the filter:

To see the traffic inside the tunnel, you can build a custom top N chart (click on the edit button next to the Chart selection list):

In this example we have built a custom Top N chart with fields vni, sourceAddress.1, sourcePort.1, destinationAddress.1, destinationPort.1. Selecting this custom top  N chart from the Chart selection list, generates a chart showing the details of the traffic flows carried by the tunnel that we are filtering on:
You can use a similar technique to look at traffic flows carried by other tunnelling protocols (Geneve, GRE, NVGRE).

You can also create reports using the Reports tab and creating a query section using Advanced settings to select key fields for encapsulated packets.



Tuesday, 10 January 2017

Upgrading sFlowTrend

sFlowTrend will tell you when a new version is available, if it has internet access. You can also check using Configuration>Check for updates in the web client.

To upgrade sFlowTrend, you just need to install the new software on top of the old. There's no need to uninstall first, that is all taken care of by the installer. The one thing that is important to remember is to use the same version of installer as you did for the initial install:
  • If you used the 32-bit Windows installer, update using the 32-bit installer. Just run the new installer, and it will uninstall the old version and install the new one.
  • If you used the 64-bit Windows installer, update using the 64-bit installer. Again, just run the new installer.
  • If you used the interactive Linux installer, use this to upgrade, again by running the new installer.
  • If you used a Linux rpm package to install, then upgrade using the command
  • # rpm -Uvh sFlowTrend-linux-version.rpm
  • If you used a Linux deb package to install, then upgrade using the command
  • # dpkg -i sFlowTrend-linux-version.deb
If you want to change how sFlowTrend is installed (eg, from the Linux interactive installer to a package), then you must first uninstall sFlowTrend using the original method. This will not remove any configuration or data. Then install using the new method. If sFlowTrend was originally installed using the interactive installer, you can use the uninstall program located in the main installation directory to uninstall it.

The data and configuration should not be affected when upgrading, but you can of course back this up to be sure. The best way is to stop the sFlowTrend-Pro service, then just copy all of the contents of the sFlowTrend-Pro data directory (this is called the home directory in the user interface, you can see it using options dialog in the Java client or the system config dialog in the web client). Once you have copied it (or zipped or tared it up as appropriate), then restart the service.

Friday, 9 December 2016

Migrating to a new server

Sometimes you need to move your sFlowTrend installation to a new server. This is quite easy to do without losing any data, but you have to be careful to make sure it goes smoothly. All of the data is in the sFlowTrend home directory. For free sFlowTrend, this contains the configuration and user preferences (when using the web client). For sFlowTrend-Pro, it also includes all of your traffic data contained in the database.

First, find the location of the home directory on the old system. If you installed sFlowTrend using an interactive installer, then you would have selected the directory during installation. If you used a package installer on Linux, then you can't change the location. In any case, you can find the name of the directory using Configuration>System Configuration in the web client:


Next, stop the sFlowTrend service on the old system. It is essential that you do this step, otherwise data is likely to be corrupted.

On Windows, use Windows Administrative Tools>Services, find sFlowTrend-Pro server, and stop it.
On Linux, depending on which distribution you are using, one of the following commands will work:
# systemctl stop sflowtrend-server
# service sflowtrend-server stop
# /etc/init.d/sflowtrend-server stop

Once you have stopped the service on the old system, move to the new one. Go ahead and install sFlowTrend on this system, again taking note of the home directory. Then, stop the sFlowTrend service on the new system - again, it is critical that you do this step. After this, copy the entire home directory from the old system to the new system, ensuring that the ownership and file permissions stay the same. Start the service on the new system.

At this stage, sFlowTrend should be running on the new system. There may be a couple of other tasks that need to be done. First, if you are using sFlowTrend-Pro and the new system has a different hostname from the old, then you will need to update the license to have the new hostname. To do this, go to www.myinmon.com, log in, and click the Request New Key link to the license. The license will be updated shortly. Next, if the IP address of the new system is different, then you will need to update the sFlow configuration on your infrastructure to forward sFlow data to the new system, rather than the old. There's more information on configuring sFlow in this blog post, and in the on-line help.

Now you have sFlowTrend running on the new system, you can uninstall it on the old. To recover some space, you can also delete the home directory on the old system, as this isn't automatically deleted when uninstalling.

Wednesday, 9 November 2016

Diagnosing abnormal network traffic levels



A common question that network managers handle is "Why is the network slow?". In many cases, poor network performance can be caused by a localised overload. This may be caused by mistakes in the network configuration, equipment malfunctions, inadvertent mis-use, or because capacity is insufficient for the normal load. Since an sFlow enabled network and sFlowTrend provide complete visibility of network usage, it is easy to pinpoint overload conditions and take appropriate action. It is even possible to receive alerts so that proactive controls can be implemented to prevent poor performance occurring. Here is an example of how sFlowTrend helps you identify and diagnose network overload conditions.


Dashboard, Thresholds indicates an abnormally high level of unicast traffic. Click on the unicast indicator to view the Thresholds tab and find out why.




The Thresholds tab indicates the switch 10.1.4.253 is experiencing the abnormally high level of unicast traffic. Click on the unicast indicator to see which interfaces are affected.
The interface with ifIndex 23 is most affected by the unicast traffic. Click on the unicast indicator to bring up the menu and select Root cause to see who and what is contributing to the unicast traffic.

100% of unicast frames are sent from hosts in the External subnet to hosts in the External subnet. Looking at the fourth row, 58% of the unicast frames are sent from 208.65.153.251 TCP:80 to 64.151.76.36. From this we can conclude that the major factor causing the abnormally high unicast traffic is web traffic from server 208.65.153.251 (which is in the External subnet). To see for how long abnormal levels of unicast traffic have affected this interface, click on the Network > Counters tab (or choose View chart from the Root cause tab menu).


The Network > Top N tab displays the details of the top connections.
In this example we have created a custom top connections chart that allows us to focus on the server port and ignore the ephemeral client port.
The Network > Circles tab allows you to visualise the traffic flows between groups of addresses to help understand the communication patterns across the network.

Wednesday, 26 October 2016

Configuring switches to send sFlow to sFlowTrend

The sFlow agent running on a switch or router is responsible for sending sFlow data to sFlowTrend. The sFlow agent must be configured with three main parameters:
  1. Receiver address and port:  The IP address and UDP port of the system that sFlowTrend is running on and on which it will receive data. By default sFlowTrend-Pro listens on all the system IP addresses using UDP port 6343. The configured receiver address must be reachable from the switch running the sFlow agent.
  2. Sampling rate: The ratio of packets carried by an interface to the number of packets sampled. For example a sampling rate of 1024 specifies that, on average, 1 packet sample will be generated for every 1024 packets carried by an interface. The sampling rate configured should  generate enough samples to be statistically significant, but not too many samples such that the scalability of the system would be affected. Some suggested sampling rates for different traffic levels are given in the sFlowTrend help.
  3. Polling interval: Controls how frequently counter data (for example interface counters) are exported. Configuring a counter polling interval of 30 seconds is recommended. This will cause counter data to be exported every 30 seconds on average. Since sFlowTrend accumulates data with one minute granularity, setting a counter polling interval of less than 20 seconds generates more load for the switch and the network without improving the resulting measurements.
In addition, some sFlow agent implementations allow the sFlow agent address to be configured. The sFlow agent address is included in the exported sFlow data and must uniquely identify the switch. sFlowTrend uses this address to attribute data to the different switches. Some sFlow agent implementations have an inappropriate default sFlow agent address of 0.0.0.0 or 127.0.0.1. This must be changed to a unique address for the switch, preferably an address that will respond to SNMP requests, in order for sFlowTrend to work properly. If the IP addresses for a switch are changed, then you should check that the sFlow agent address is updated properly; this sometimes requires a restart of the sFlow agent.

There are two methods for configuring the sFlow agent: SNMP and Command Line Interface (CLI).

sFlow configuration using SNMP

If a switch implements the sFlow MIB, sFlowTrend can use SNMP to configure the sFlow agent. In this case, you must ensure sFlowTrend is configured with the IP address of the switch and the correct SNMP credentials. The switch must also be configured to allow SNMP read and write from the sFlowTrend system. sFlowTrend will then make sure that the switch is configured with the correct settings for receiver address and port, sampling rate and polling interval for all interfaces.

Follow the steps in the section in the help Adding a switch configured by SNMP. The help also outlines steps for verifying and troubleshooting the configuration.


sFlow configuration using SNMP is accomplished as follows:
  1. sFlowTrend uses SNMP to walk the ifTable to discover all interfaces and their ifSpeeds. This process also ensures that basic SNMP read access is working.
  2. sFlowTrend then tests for the existence of the sFlow MIB.
  3. On discovering the sFlow MIB, sFlowTrend then tries to claim a receiver entry by using SNMP SETs to write its configured receiver IP address and sFlow port, a unique string identifying this sFlowTrend instance, and a timeout value into the receiver entry. The receiver address and port used by sFlowTrend are configured in the System configuration, sFlow configuration dialog. A switch will support a fixed number of receiver entries, limiting the number of sFlow collectors that it can send data to. If all of the receiver entries have been claimed by other sFlow collectors, then sFlowTrend will fail to configure this switch and in the Configure agents dialog will report Already in use and list the other sFlow collectors. You can tell sFlowTrend to overwrite an existing receiver entry when there are no free entries by using the sflowtrend.useForce setting.
  4. After having successfully claimed a receiver entry, sFlowTrend will then set sFlow MIB entries for each interface to configure the polling interval and the appropriate sampling rate. sFlowTrend chooses the sampling rate based on the ifSpeed of the interface discovered by walking the ifTable.
  5. Periodically, sFlowTrend will refresh the timeout value in the receiver entry using an SNMP SET. This ensures that the switch will continue to send sFlow. However, if sFlowTrend is stopped, the timeout will decrease to zero and on reaching zero, the switch will clear the receiver entry and stop sending sFlow to sFlowTrend. This means that no resources are used in generating unwanted sFlow.

sFlow configuration using CLI

The sFlowTrend help gives some examples of CLI configuration of sFlow. Consult your switch documentation for further details. In this case, no configuration of sFlowTrend is required. Instead, as soon as sFlowTrend receives sFlow from a switch, the switch will be listed in the sFlowTrend System configuration, Configure agents dialog and the various switch selectors. If sFlowTrend does not list the switch, check the following:
  1. The switch is configured with the correct receiver address and UDP port. You can verify the UDP port that sFlowTrend is listening on in the System configuration, sFlow configuration dialog.
  2. There are no firewalls on the sFlowTrend system or the network that are blocking sFlow data.
  3. The sFlowTrend system is reachable from the switch.
  4. The sFlow agent on the switch is configured with a unique sFlow agent address.
 If you want sFlowTrend to display interfaces using ifName or ifAlias, or to display the sysName of the switch, you must configure sFlowTrend with SNMP read access to the switch.



Monday, 12 September 2016

Getting started with sFlowTrend and sFlowTrend-Pro

To use sFlowTrend or sFlowTrend-Pro, first download the installation package. The installation package for sFlowTrend is available from http://www.sflowtrend.com. If you have purchased a license, or requested an evaluation license for sFlowTrend-Pro, you can download the installation package from your account at http://www.myinmon.com under the Products > Download Software. Choose the correct installation package for the system that will run sFlowTrend or sFlowTrend-Pro:
  • For a Windows installation, use the Windows installation package unless you are running a 64-bit version of Windows with a 64-bit JRE installed, in which case use Windows 64-bit installation package. 
  • For a Linux installation,  select either the Linux installer download, which is a graphical and CLI installer, or install using an RPM or .deb file. If you choose the RPM or .deb option, and have a previous installation using the installer, uninstall the previous version before installing the RPM/.deb file. 
  • For an Apple Mac installation, use the .dmg installation package.
Make sure that you have Java 1.7 (or later) installed, then run the sFlowTrend or sFlowTrend-Pro installation package following the instructions.

Once you have installed sFlowTrend or sFlowTrend-Pro, open the GUI by pointing a web browser to http://[hostname]:8087/sflowtrend or https://[hostname]:8443/sflowtrend. You must now configure the license.

To use sFlowTrend-Pro, deselect "Use sFlowTrend (free)" and enter your license number from your account at http://www.myinmon.com.

Note that normally sFlowTrend-Pro will use the Internet to download the license key, once the license number has been entered. If a proxy configuration is required for the server to connect to the Internet, please make sure that the proxy is correctly configured. On initial installation, until you configure the license you cannot use the rest of the product; this means that the proxy also cannot be configured. To work around this, if you have to configure a proxy, first select the option to use the free sFlowTrend license, then configure the proxy, and finally go back to the license dialog and enter your actual license number. If the system has no Internet connectivity at all, then the license key can be entered manually. You can request a manual license key using a support request from your account at http://www.myinmon.com.

Now configure your switches to send sFlow to sFlowTrend-Pro. Another blog post will describe this in more detail. The Dashboard will show the incoming sampling rate and the status bar will indicate the number of switches and hosts being monitored. The troubleshooting section of the help includes some things to check if sFlow is not being received.