pfSense RRD graphs for NTP - system jitter vs clock jitter

Since installing pfSense and setting up the NTP server, I've been wondering for a while what the difference is between "System Jitter" (sjit) and "Clock Jitter" (cjit) in the RRD graphs.  For instance, in the following graph, we can see that the system jitter value has gone way up.

So what causes system jitter? Looking at the Status -> NTP page gives us a bit of a clue.

At least two of our upstream time servers have huge jitter numbers, 20-40ms, while all of the other upstream time clocks are reporting jitter of less than 1ms.  So something is happening to those two time servers, or a common route between us and those time servers.

That leads me to believe that the green line in the NTP graphs (sjit) indicates how much jitter there is between our NTP server and the upstream servers.  In general, it probably means there is congestion between us and that server and that packets are arriving out of order or late.

Using aliases pfSense to create rules for protocols with multiple port ranges

File this one under "things I wish I had known sooner".  When setting up pfSense firewall rules on an interface, you'll run into protocols which have multiple ports that are not in a contiguous range.  One example of this is the common web server (HTTP) ports of 80, 443 and 8080-8081.

This leaves you with two options.

1. Setup multiple rules.  This is the best option because you only specify the exact ports that you want, with no extras thrown in.  The downside is that for some protocols, you will end up with multiple rules that have to be maintained.
2. Specify a rule with a broad port range.  Which is sort of okay if you are only allowing a handful of extra ports, but it is not ideal.

Enter the concept of aliases (under the Firewall -> Aliases menu) in the pfSense web UI.  Here you can create an alias which lists out all of the ports associated with a particular protocol.

After creating the alias, you then create or edit a rule and use that alias in any fields with a red background.  Such as the destination port field.

After clicking the "Save" button, rules that are using port aliases will show up in the rule list looking like:

Needless to say, that can make your life much easier when maintaining large lists of ports as long as all of the ports in question are using the same protocols.

Mail client ports (IMAP/POP3/SMTP) are also good candidates for an alias rule.  One caution is to never allow 25/tcp to egress your network, only your mail server in the DMZ should be allowed to contact other mail servers via port 25.  Every internal client should be forced to either use tcp/465 (SMTP/SSL) or tcp/587 (SMTP Submission) or route their SMTP traffic through your mail server.

pfSense rate limiting, egress filtering, opendns filtering for wifi hotspot

One of the experiments that I'm running with the new network is running an open / unsecured WiFi hotspot for the neighbors.

Some of the protections that I'm using:

• Uses OpenDNS servers with some categories of websites blocked.  I'm using the "OpenDNS Home" service which lets me pick and choose which categories are blocked by default.  In addition, the OpenDNS server will display a "blocked content" page for regular HTTP traffic where users can request an unblock.  Unfortunately, this feature does not work well for HTTPS (SSL) sites, but it still blocks the site.
• Access to other DNS servers is blocked, clients can only access the two OpenDNS server IP addresses.
• All rules on the interface are rate-limited to 3Mbps down and 1Mbps up.  This limits bandwidth abuse and slows down file sharing.
• Heavy egress filtering.  All outbound traffic is blocked by default except for the whitelisted ports/protocols.

Now, this is not 100% foolproof.  But I at least want to limit the possible damage and take at least some steps against abuse.  I'll probably use this setup at a company that I'm consulting for where they want to offer open WiFi in their waiting area.

One thing I would like to do is setup a "Captive Portal" on the interface which forces the user to enter their cell phone number and receive a voucher code via SMS that is good for 3 or 7 days.  I have to figure out how to do that with pfSense and see how it works in practice.

The other thing I plan on doing is setting up a similar SSID/VLAN, but with higher bandwidth limits, more ports and no OpenDNS filtering for authenticated guests.  That would probably be a 20Mbps down / 5Mbps up setup protected by WPA2/PSK.  Think along the lines of "neighbors" or "friends" who you want to allow use of the internet pipe, but do not want to allow onto your interior network.  This would also be a good setup to use in an office environment for BYODs that only need internet access (such as clients).

pfSense on Core2Duo E8400 refurbished PC

A rough power estimate for my little SFF (small form factor) refurbished PC that I'm using for a pfSense firewall:

• Intel Core2 Duo E8400 @ 3GHz
• 4GB RAM
• 120GB SSD
• Dual-port Intel PCIe x4 NIC

At idle, it consumes about 38W when the CPU throttles back to 1.8GHz.  That's pretty good for a PC that is not designed to be a low-power / fanless unit.  Under load, that goes up to about 60-65W.

38W @ $0.15/kWh = $50/yr

65W @ $0.15/kWh = $85/yr

So even if I got something that stayed below 15W, I'd only save $50-$60/yr.  A lot of the low power units are $300-$500, which would be a very long time before they'd pay off.

Note: You do need to enable PowerD under System -> Advanced -> Miscellaneous -> Power Savings in order to get the CPU to throttle down when idle.  I recommend "Hiadaptive" for an office firewall, but you might want to experiment.

The biggest CPU hog on my current pfSense setup is "ntopng".  While doing bi-directional gigabit testing, that ate up 10-12% of my CPU power.

pfSense Firewall CPU load estimate

According to the pfSense dashboard, I have:

Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
2 CPUs: 1 package(s) x 2 core(s)

When running a quick speed test, "top" shows about 5% system load at 60Mbps.  That gives a rough upper-end of around 1200Mbps (1.2Gbps) for switching speed.  At a guess, that might be closer to only 1Gbps performance under heavy traffic.

1Gbps of capacity is plenty for the moment where I have:

- 50/50 Mbps service from Verizon FIOS (seems to peak at 60/60)
- 802.11 b/g/n (11-54Mbps)
- 802.11ac (tops out at around 1Gbps)

But it may not be enough for connecting together multiple gigabit LAN segments.  So I will need to keep all high bandwidth traffic on the same VLAN so that the traffic gets handled by the switches without touching the pfSense firewall.

Update #1 (Jul 10 2015): Suricata cuts the performance of the WAN interface (in terms of CPU load per Mbps) by a factor of 5x-10x.  While I could probably route 1.2-1.5Gbps with this firewall, a 30Mbps load on the WAN, which is monitored by Suricata, resulted in 20% CPU load.  That puts my upper-bound for WAN traffic at only 150Mbps.

Firewall build (part 2) hardware needed for VPN duties

One thing to think about when sizing the hardware for a firewall is how much CPU power will be needed for OpenVPN (or IPSec / L2TP).  OpenVPN comes with a built in "speed" command which will benchmark your system and give you an idea of maximum possible bandwidth.

Just run "openssl speed" at the command line and look for the AES-128 and/or Blowfish results.  I prefer to look at the 1024 byte or 8192 byte columns in the output to figure out the upper range.  While Blowfish is good at the smaller block sizes, AES-128 catchs up and surpasses it with the larger block sizes.

Values at or above 100000k should indicate that the firewall has enough performance to drive an OpenVPN connection at close to gigabit speeds.  Or handle multiple OpenVPN connections at the same time, without completely saturating the CPU.

AMD Opteron 2210 HE @ 1.8GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
blowfish cbc     68358.84k    74350.46k    75845.03k    76373.67k    76556.97k
aes-128 cbc      50477.29k    53816.28k    55093.08k   128709.63k   130465.79k

AMD Phenom II X4 810 @ 2.6GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
blowfish cbc     94477.65k   101825.28k   103154.35k   103857.83k   104060.25k
aes-128 cbc      76376.65k    81608.09k    83915.50k   213516.45k   216016.95k

AMD Opteron 4180 @ 2.6GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes

blowfish cbc     93781.75k   101154.41k   102983.68k   103730.52k   103923.71k

aes-128 cbc      76233.64k    81631.07k    83197.52k   213366.89k   215309.87

103720k = 98 MiB/s or ~980Mbps, which is pretty close to gigabit speeds

General guidelines/notes:

• I'm a firm believer in multi-core for servers and desktops.  So look for hardware that is at least dual-core when shopping.  An inexpensive quad-core would be even better and give a bit of headroom for monitoring tasks.
• For the AMD CPUs (Opteron / Athon64 / Phenom) made in 2007-2011, you'll want at least a 2.2GHz core.  For Intel Core2 CPUs or 1st/2nd generation i3/i5, try to get at least a 2.0GHz core.
• Intel Atom CPUs are underpowered, the 1.8GHz dual-core units are reported to top out at around 500Mbps for general routing and definitely can't handle gigabit speed OpenVPN.  But they are low power, so maybe that outweighs the performance issue.  A rule of thumb is that the Atom CPUs are about 1/3 to 1/2 as powerful as i3/i5 for the same clock speed.

Resource links:

• Firewall Hardware Sizing Guide [www.firewallhardware.it]
• Choose Your Hardware [www.pfsense.org]