As for the last 11 years Cisco sponsored the FOSDEM 2020 conference in Brussels. This is one of the largest, if not the largest, open source conference in the world with this year 872 events spread over 2 days.
From a network point of view the important part is that the main SSID at the conference is IPv6-only, using NAT64 and DNS64 to provide access to IPv4-only resources on the internet. For users which require IPv4 connectivity a ‘legacy’ dual stack network is also available. The name of this SSID is changed yearly so that people consider moving to the IPv6-only network.
As in the last few years we provided and configured the switches and the ASR 1006 router, where we use the netflow and NBAR features to collect the HTTP user agents field the router sees. These user agents are sent by the clients to web servers in clear text and can tell us something about the application, client operating system or architecture. We then analyse this data to see trends.
Unique MAC addresses == number of clients
Last year (our report of 2019) we faced the problem that we saw a lot of artificial MAC addresses on the network, in total we saw 129,959 unique MAC addresses last year. This year we enforced the usage of DHCP on the IPv4 network, strengthened the security and fixed a few bugs and added more checks in or reporting tools. If this worked is unclear, however this year we ‘only’ saw 11,372 unique MAC addresses, of which 9,993 could be attributed to a known manufacturer and 1,379 seem randomized. Of these we could guess the operating system used for 1,007, all of which were running Android 10 on diverse platforms.
Last year we could attribute 11,781 unique addresses, so we are seeing a reduction in the number of clients. This seems to be confirmed by the statistics gathered by the wireless controller, which saw about 6,000 concurrent users in 2019 and about 5,500 in 2020. We assume that more people simply use roaming as for EU guests this is almost always included in their package.
More Bandwidth used by fewer clients
However, these fewer users did consume more bandwidth. Far more bandwidth, input from the internet went from last years 2,960,975,295,300 bytes to 23,658,552,410,903 bytes, a growth of 799%. Also output towards the internet went from 1,755,298,210,796 bytes to 12,715,921,100,135 bytes, a growth of only 724%. Our uplink was almost constantly loaded during the whole event.
We also checked the distribution between the IPv4 and IPv6 traffic and the IPv6 traffic largely trumps the IPv4 traffic, just as it did last year.
Of the 8,233 clients whch were on the IPv6-only network at one time, 7,854 or ~95% used the NAT64 feature. We had 7,462 connections which were not using ICMP, UDP or TCP, 109,553 ICMP, 722,511 UDP and 11,496,472 TCP connections over NAT64. Below are the top 10 services carried by NAT64:
As to the destinations it is more complicated. We only see the IP’s and when trying to identify them it’s difficult. In one case there is even a mystery: we see a multicast IP (!). The top 10 destinations handled by NAT64 were:
What can we tell about the clients?
Some devices send the architecture of the CPU, which we can use to collect statistics. Android and Apple Mobile devices in general are ARM based.
When we analyse the MAC addresses we can identify the manufacturer of the device, based on the ‘manuf’ list from the Wireshark sources. This gives the following distribution:
We can try to guess the operating system running on the device, based on the HTTP user agent string. This is not only difficult as some user agents are schizophrenic (what browser is “Mozilla/5.0 (Linux; Android 9; MI 8 Build/PKQ1.180729.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 XWEB/1163 MMWEBSDK/191102 Mobile Safari/537.36 MMWEBID/6033 MicroMessenger/126.96.36.1990(0x2700096B) Process/toolsmp NetType/WIFI Language/en ABI/arm64“, and doesn’t it leak way too much information?) but some devices emit only very generic useragents like “ConnMan/1.32+git97.3 wispr”.
So, we manually try to detect the operating systems and architectures used, which means that for some we can only detect it is a Linux running on x86_64, while for others we can only say it is some kind of Apple device. We get the following results:
On which networks do we find the clients?
The rise of IPv6 continues. Like last year we had ~3460 IPv4 DHCP on the legacy network, but we had 6235 clients which stayed on the IPv6 only network and did not go back to the legacy network. 1,998 clients were seen on both the legacy and the IPv6 only network and 2,561 clients never even tried the IPv6 only network. We can identify which clients were on the different networks:
Everybody’s favourite client: The Windows Phone, was on the IPv6 only network, counted as part of the ‘Windows’ family.
Eating addresses like candy?
Some devices rotate through the IPv6 addresses at a high rate, we found a few which used more then 400 IPv6 addresses over the course of the weekend. All of those clients are in fact iPhone 13.3 devices and not some more ‘hacker’ device. Most devices only use a few, and the majority only two: the link-local and the global IP.
What can we say about the traffic?
We continue to see the majority of VPN based ESP traffic on the dual stack legacy network. Again, we used NBAR2 to classify the traffic giving the following results:
asr1k#show ip nbar protocol-discovery interface gi2/0/4 stats byte-count top-n 20 ... Input Output ----- ------ Protocol Byte Count Byte Count ---------------------------- ------------------------ ------------------------ shoutcast 47029363893 2112934159987 ssl 969293704522 608105734792 statistical-p2p 52012656918 1110991906422 unknown 283397102118 529552327874 http 85483662261 713775820214 google-services 266502623472 147675163463 proxy-server 6012538536 333564857312 binary-over-http 116312457935 1770591381 amazon-web-services 74296965211 32039361466 twitter 68570673404 8287476941 facebook 67610858696 7142510259 akamai 60348002605 4396237959 netflix 56448209276 2820631545 apple-updates 57130900493 1120491337 youtube 55489033038 1850317608 github 52301021133 2322815054 openvpn 42285183215 8426470948 instagram 47000986713 3638936268 icloud 20648621225 29671510399 conference-server 4028241437 43888642924 Total 2432202806101 5703975964153
The high amount of traffic for shoutcast is probably the people in the corridor streaming the talk inside of the room they could not enter. It is strange that we have more netflix then github traffic, but as someone told me: “byte for byte github content takes a lot more time to properly enjoy”. We can also check which traffic caused the highest bit rate:
asr1k#show ip nbar protocol-discovery interface gi2/0/4 stats max-bit-rate ... Input Output ----- ------ Protocol 30sec Max Bit Rate (bps) 30sec Max Bit Rate (bps) ---------------------------- ------------------------ ------------------------ ssl 353356000 322821000 ms-services 443143000 10610000 ssh 42297000 297418000 google-services 151221000 91569000 statistical-p2p 63718000 162086000 amazon-web-services 101362000 93969000 http 67107000 114832000 unknown 90678000 85028000 shoutcast 10624000 156234000 binary-over-http 151637000 2102000 apple-updates 139678000 3757000 isakmp 61012000 69309000 ipsec 66291000 62469000 netflix 76857000 44271000 github 101191000 3399000 icloud 65720000 36435000 adobe-services 58527000 43031000 openvpn 58213000 34107000 youtube 89154000 1589000 statistical-conf-audio 45154000 44640000 crashplan 38509000 48593000 proxy-server 1423000 83464000 bittorrent 51471000 32060000 akamai 75391000 5746000 google-downloads 75773000 2096000 dropbox 61787000 14648000 http-alt 44900000 28934000 statistical-conf-video 11006000 61704000 ms-update 65564000 1084000 ms-office-365 31290000 33657000 Total 2694054000 1991662000
I think only at FOSDEM the third highest protocol in maximum bitrate can be ssh at 2,97,418,000 bps of outgoing traffic. That’s ~35.5 megabytes per second of outgoing traffic.
Looking towards the future:
Next year we will try:
- To get multiple inter-building links. We are reaching limits here and this would help for redundancy.
- To get a second uplink, again we are coming close to the limits of our 1Gbps uplink here.
- To disable 2.4GHz on the main SSID and add extra AP’s to fix coverage holes. This should help with high density situations like in Janson.