Cisco Blogs
Share

FOSDEM 2019: a new view from the NOC


February 5, 2019 - 2 Comments

The first weekend of February is for me by now traditionally the FOSDEM weekend. FOSDEM is arguably the biggest Open Source Developers conference in the world with 742 lectures regarding various topic spread over the ULB university campus. The entrance is free and all presentations are recorded and can be streamed live during the conference (useful when you were not able to enter a full room), or later at your leisure from video.fosdem.org.

This year marks the 10th time that Cisco sponsors the network infrastructure. As this is a ‘least effort, maximum results’ exercise we try to not change the architecture between the years, so there should be no nasty surprises. So again, just like the last few years, we had an IPv6-only with DNS64/NAT64 ‘main’ network and a dual stack ‘legacy’ network, both routed towards the internet via an ASR1006 which provided DHCP, SLAC and NAT64 services.

Every year we try to analyze the network and traffic to provide a glimpse of what ‘state of the art’ traffic looks like. The population of users at FOSDEM is far from a typical business environment, but this data does provide an idea of future trends.

Network-wise, the main takeaways this year are:

  • There were more clients on the IPv6 native network then on the IPv4 network, with on Sunday afternoon ~3330 IPv4 DHCP clients against ~4300 reachable IPv6-only clients and ~1300 IPv6 clients on the dual stack network. The grafana statistics show a similar picture:
  • This shows that native IPv6 plus DNS64/NAT64 is a valid alternative for the vast majority of users.
  • When we look into the legacy dual stack network, we notice that for the IPv4 traffic distribution we see outgoing ~214M TCP packets and ~6M ESP (VPN) packets while incoming was ~394M TCP packets with ~8M ESP packets. This means that at least about 2-3% of all traffic was on an IPSEC VPN. And this excludes the TCP VPN traffic on ports 443/TCP and 22/TCP. On the IPv6 network we do not see a similar amount of ESP traffic.
  • This strongly suggests that the people remaining on the dual stack network do so because their VPN solution does not work with an IPv6 only network.
  • Traffic wise we broke all records with 2,960,975,295,300 bytes input (up 16.7% compared to last year’s 2,537,252,681,481 bytes) and 1,755,298,210,796 bytes output towards the internet (an increase of 57% compared to last year’s 1,116,884,145,475 bytes).
  • Most traffic was IPv6, by far:


As we did in the previous years, we used NBAR to classify the data, so we can give the 10 top data consuming protocols:

asr1k#show ip nbar protocol-discovery interface gi2/2/1 stats byte-count top-n 10
...
                             Input                    Output
                             -----                    ------
Protocol                     Byte Count               Byte Count
---------------------------- ------------------------ ------------------------
ssl                          1024765056806            251043670421
rtmp                         17859365809              839473061541
google-services              175407914561             161785828996
binary-over-http             241852948792             4245784798
statistical-download         29057423539              153776145315
apple-services               128893914984             11742821363
youtube                      102626951521             5252216684
facebook                     93538045035              12754299118
amazon-web-services          89391297834              15666057435
unknown                      81551056347              19952504555
Total                        2958863503183            1756617044390

Or the top bandwidth protocols:

asr1k#show ip nbar protocol-discovery interface gi2/2/1 stats max-bit-rate top-n 10
...
                             Input                    Output
                             -----                    ------
Protocol                     30sec Max Bit Rate (bps) 30sec Max Bit Rate (bps)
---------------------------- ------------------------ ------------------------
ssl                          305058000                71069000
binary-over-http             282241000                18044000
statistical-download         66230000                 218672000
ssh                          70355000                 210668000
google-services              93300000                 81333000
amazon-web-services          119330000                48004000
dropbox                      89163000                 63882000
apple-services               124853000                28109000
icloud                       109747000                41144000
youtube                      74360000                 58941000
Total                        4359491000               1769693000

Again, we see that even open source developers use Google, Amazon, Apple, Facebook and watch YouTube. The large amount of ssh traffic would stand out in a more corporate environment however.

Last year we also noticed that identifying the clients from the HTTP traffic became a lot more difficult as we started seeing randomized MAC addresses: Saturday statstotal stats, statistics for ‘real’ MAC addresses. This makes it almost impossible to guess the operating system used by the clients.

This year these trends continued. We had 129,959 unique MAC addresses during the conference, this excludes the random MAC addresses uses to probe for wireless networks, only MAC addresses which send data across or toward the ASR1006 router are counted. Of these we can assign 11,781 to a known manufacturer. Last year we saw 128,463 unique MAC addresses with only 10,689 MAC addresses were attributable to a manufacturer. From this we can guess that the number of visitors went up by about 10%. The cause for all these random MAC addresses remains unclear.

The distribution of manufacturers shows that Apple is still king of the MACs:

We can the evolution of devices when comparing 2018/2019:

Looking forward we hope to upgrade the backbone of the network from 1Gbps links to 10Gbps links, as we are reaching ~70% of capacity when talking to the WLC which is hosted on the ULB network:

Adding another provider for the internet uplink would be a good thing, currently mostly for redundancy, but the video team’s appetite for bandwidth is large and growing, notice the jump in the internet bandwidth on Sunday afternoon to over 300Mbps, so more bandwidth might be needed in the future.

See you all next year!

 



In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

2 Comments

  1. One statistic I didn't see in this was how much IPv6 traffic was translated to IPv4 or vice versa. On the IPv6-only SSID, was the service IPv6 end to end, or did it simply "work OK" with a CNAT? What I would prefer to see is the service the party is going to actually use IPv6 and translation therefore not be used very much – which is what I hear from T-Mobile about its 464XLAT implementation. From the dual stack network's behavior, I would conclude that there is probably a fair bit of translation.

    • Hi,

      On the IPv6-only SSID there was IPv6 traffic end to end if possible. There was no NAT for the IPv6 or IPv4 ranges. Both IPv6-only and dual-stack networks used public IPv6 and IPv4 address ranges.

      If an IPv6-only client wanted to communicate with an IPv4 only server then the DNS server, using DNS64, would synthesise an IPv6 address with the 'well known' IPv6 prefix. When that IPv6 client would then send traffic towards this synthetic address the ASR1006 would NAT this traffic using NAT64 into an IPv4 (from the pool on the router) to IPv4 (extracted from the synthesised address).

      During FOSDEM we translated 6,711,290 connections composed out of 374,671,589 IPv6 -> IPv4 packets and 587,005,529 IPv4 -> IPv6 reply packets using NAT64. For reference there were 1,255,170,975 outgoing IPv6 packets in total. So about 30% of IPv6 traffic was towards IPv4-only servers and needed to get translated using NAT64.

      There was no setup to allow IPv4 devices to talk to IPv6 clients. All our servers are dual-stacked.