Nebula is a distributed mesh VPN and is written by the folks behind Slack. Unlike traditional VPNs which typically rely on a hub and spoke topology Nebula is contextually aware of peer layout and routes packets between them intelligently.
For example, if two nodes are on the same LAN the mesh won't route packets via the "hub" but instead make a direct connection between these two hosts. This reduces unnecessary overhead and makes self-hosting a central server - or as Nebula calls it a lighthouse - on a bandwidth constrained VPS a viable proposition.
Furthermore, we can use Nebula to traverse NAT. As long as a node can connect to a publicly reachable lighthouse then the mesh can traverse NAT making firewall rules a thing of the past.
The Problem
That is until a clever feature built-in to OPNsense and pfSense decide to make things more secure by rewriting all outgoing traffic using UDP - details here. Due to this, you will find that nodes behind a *sense based firewall are not able to talk to one another.
Take the following mesh topology for example:
However, in this example with the default firewall configuration nodes 10.10.10.2 and 10.10.10.3 cannot communicate with each other. Pinging each node will most likely show some kind of activity in the logs on both nodes however traffic will not successfully flow.
The Solution
It's actually a really simple solution. I wish to thank @icebladerage on the Self-Hosted podcast Discord server for providing the answer.
You will need to create a firewall rule on each OPNsense firewall involved in the transaction to disable UDP port rewriting that looks like this:
Firewall -> NAT -> OutboundThe mechanics of the rule look like this:
Save and apply the rule and your traffic will start to flow almost immediately.