There is an unwritten rule in networking: “Don’t try to route Layer 2 multicast traffic (like IP camera discovery protocols) over a Layer 3 VPN. It will be unstable, just add the cameras manually by IP.” Well, we didn’t accept that answer.
Our goal: We wanted a geographically distributed network (across multiple sites) to act exactly as if all our Techson IP cameras and NVRs were plugged into one massive, shared physical switch. We needed automatic L2 device discovery and smooth multicast streaming, all running blazingly fast on industrial Teltonika routers (RUTX10, RUTX11, RUT241) and network devices (TSW202, TAP200) through a secure WireGuard tunnel.
After weeks of experimenting, analyzing thousands of dropped packets, and deep-diving into Linux networking, everything finally clicked. We solved the “impossible.” Here is how we built an Enterprise-grade SD-WAN using native RutOS/OpenWrt features.
Challenge 1: The “Matryoshka Doll” of VXLAN over WireGuard (The MTU Issue)
Our core concept was to encapsulate the Layer 2 traffic into a VXLAN tunnel, and then encrypt that inside a WireGuard (L3) tunnel. The network came up, the cameras saw each other, but high-res streams and large packets randomly dropped.
The Cause: Packet fragmentation (MTU asymmetry). The default MTU is 1500 bytes. WireGuard takes 80 bytes for encryption overhead (MTU: 1420). If you put a VXLAN frame inside that, the original 1500-byte camera payload simply gets “stuck in the pipe” and dropped by the router.
The Solution: We had to force asymmetric MTU on the interfaces at boot. We set the jz (WireGuard) interface to 1420, and the vxlan interfaces to 1370. The brilliance of Linux networking is that once the VXLAN interfaces were added to the local br-lan bridge, the bridge itself adapted to the lowest common denominator (1370). This inherently signaled the cameras/servers to generate smaller frames. Packet drops instantly went to zero.
Challenge 2: Multicast Blindness (IGMP Proxy & Switch Configuration)
Even with MTU fixed, sometimes the discovery packets (UDP 23456 to 234.55.55.x) wouldn’t traverse the tunnels reliably.
The Solution: 1. On the RUTX10 (VPN Server): We explicitly had to enable IGMP Proxy and IGMP Snooping under the network settings. Without the Proxy, the router kernel refused to forward the L2 multicast frames bridging the local LAN and the VXLAN. 2. On the TSW202 (Managed Switch): We bypassed IGMP unreliability by writing a startup script that statically forces the camera multicast groups toward the SFP1 port (which leads to the VPN router).
Challenge 3: Enterprise DHCP Snooping & Wi-Fi Noise
When you have multiple foreign devices or Wi-Fi clients on the network, you face the risk of DHCP Starvation attacks and broadcast storms. Furthermore, Wi-Fi access points notoriously hate multicast traffic.
The Solution: * On the routers, using nftables, we created a custom input chain that limits DHCP requests to 10/second and drops the rest, simulating hardware-level DHCP Snooping. (We added a “VIP Lifeline” for ARP, ICMP, SSH, and WebUI to prevent locking ourselves out).
- On the TAP200 (Wi-Fi AP), we used
ebtablesat Layer 2 to block rogue DHCP servers, kill Windows noise (NetBIOS, SSDP), and explicitly allow our camera discovery packets.
The “Techson Guard” Configurations
Here are the scripts we used. Feel free to use them to stabilize your own L2-over-L3 deployments!
1. RUTX10 (Central VPN Router) - Custom Firewall Rules
Put this in Network → Firewall → Custom Rules. Wait 60s for WG interfaces to spin up.
Bash
# Wait after boot to ensure WG/VXLAN interfaces are fully up
sleep 60
logger -t "FirewallGuard" "Starting Custom L2/L3 Rules..."
# --- 1. VPN & VXLAN STABILIZATION (Crucial MTU Fix) ---
ip link set dev jz mtu 1420 2>/dev/null
ip link set dev vxlan1 mtu 1370 2>/dev/null
# --- 2. NETDEV INGRESS FILTER (Drop noise at the tunnel entrance) ---
nft add table netdev filter_noise 2>/dev/null || true
nft add chain netdev filter_noise filter_vxlan1 { type filter hook ingress device vxlan1 priority 0 \; } 2>/dev/null || true
nft flush chain netdev filter_noise filter_vxlan1 2>/dev/null || true
# Allow IP Camera Discovery (Multicast UDP 23456)
nft add rule netdev filter_noise filter_vxlan1 udp dport 23456 accept
nft add rule netdev filter_noise filter_vxlan1 ip daddr { 234.55.55.55, 234.55.55.56 } accept
# Drop unnecessary IGMP noise
nft add rule netdev filter_noise filter_vxlan1 ip protocol igmp drop
# --- 3. BRIDGE TABLE (Deep Network Protection) ---
nft add table bridge security_guard 2>/dev/null || true
# --- A) INPUT CHAIN (DHCP Protection + Management Lifeline) ---
nft add chain bridge security_guard input { type filter hook input priority 0 \; } 2>/dev/null || true
nft flush chain bridge security_guard input 2>/dev/null || true
# VIP Lifeline: Prevent L2 management lockout (ARP, Ping, WebUI, SSH)
nft add rule bridge security_guard input ether type arp accept
nft add rule bridge security_guard input ip protocol icmp accept
nft add rule bridge security_guard input tcp dport { 22, 80, 443 } accept
# Allow Multicast Discovery to reach the router
nft add rule bridge security_guard input udp dport 23456 accept
nft add rule bridge security_guard input ip daddr { 234.55.55.55, 234.55.55.56 } accept
# DHCP Starvation Protection
nft add rule bridge security_guard input udp dport 67 limit rate 10/second burst 5 packets accept
nft add rule bridge security_guard input udp dport 67 counter drop
# --- B) FORWARD CHAIN (Filtering traversing traffic) ---
nft add chain bridge security_guard forward { type filter hook forward priority 0 \; } 2>/dev/null || true
nft flush chain bridge security_guard forward 2>/dev/null || true
# Allow Camera streams across the VPN
nft add rule bridge security_guard forward udp dport 23456 counter accept
nft add rule bridge security_guard forward ip daddr { 234.55.55.55, 234.55.55.56 } counter accept
# Block Windows NetBIOS/SSDP network "garbage" towards the VXLAN
nft add rule bridge security_guard forward oifname "vxlan*" udp dport { 137, 138 } drop
nft add rule bridge security_guard forward oifname "vxlan*" tcp dport { 139, 445 } drop
# --- C) OUTPUT & MULTICAST FORCE ---
nft add chain bridge security_guard output { type filter hook output priority 0 \; } 2>/dev/null || true
nft flush chain bridge security_guard output 2>/dev/null || true
# Force Multicast forwarding on the VXLAN interface
ip link set dev vxlan1 allmulticast on 2>/dev/null || true
logger -t "FirewallGuard" "Network stabilized. MTU Fix and DHCP Guard active."
2. TSW202 (Managed Switch) - Static Multicast Route
Put this in System → Custom Scripts (/etc/rc.local). This forces the camera Multicast groups directly to the SFP1 port (uplink to the router).
Bash
# Techson Multicast Fix - TSW202
# Force SFP1 port towards the VPN
# Wait 30s for switch ports to wake up
sleep 30
logger -t "TechsonFix" "Applying Multicast rules to SFP1..."
bridge mdb add dev br0 port sfp1 grp 234.55.55.55 permanent 2>/dev/null
bridge mdb add dev br0 port sfp1 grp 234.55.55.56 permanent 2>/dev/null
logger -t "TechsonFix" "Done."
exit 0
3. TAP200 (Wi-Fi AP) - Wireless Edge Protection
We run this ebtables script at boot to filter the wireless traffic right at Layer 2.
Bash
#!/bin/ash
# --- TAP200 L2 GUARD ---
ebtables -F
ebtables -X
# 1. Block Rogue DHCP Servers from the Wi-Fi side
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-sport 67 -i wlan+ -j DROP
ebtables -A INPUT -p IPv4 --ip-proto udp --ip-sport 67 -i wlan+ -j DROP
# 2. DHCP Flood Limit (Max 10/sec per client)
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 67 -i wlan+ --limit 10/sec --limit-burst 20 -j ACCEPT
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 67 -i wlan+ -j DROP
# 3. Drop Noise (NetBIOS, SSDP)
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 137:139 -i wlan+ -j DROP
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 1900 -i wlan+ -j DROP
Conclusion
If someone tells you “it can’t be done,” don’t believe them. With persistence, high-quality industrial hardware, and a deep understanding of Linux networking under the hood (nftables, VXLAN, ebtables), you can bring the most complex Enterprise features to the edge sector.
Knowledge is only power if you share it. Happy networking!
UPDATE (V2.0 Lean Architecture): After deploying the initial solution, deep network analysis revealed that standard firewalling techniques (like netdev ingress hooks and packet counters) caused CPU jitter on the routers, resulting in 200-500ms ping spikes. Furthermore, complex Layer 2 bridges can sometimes leak DHCP broadcasts across the VXLAN before standard forward rules can catch them.
To solve this, we completely rewrote the scripts focusing on a “clean slate”, hardware-efficient approach to avoid reusing problematic logic. The new V2.0 scripts below use atomic loading (EOF), strict TCP MSS Clamping (to eliminate MTU fragmentation overhead without lowering local LAN MTU), and prerouting drops to guarantee 100% DHCP isolation without burdening the CPU. These are the definitive, production-ready versions.
1. RUTX10 (Central VPN Router) - V2.0 Custom Firewall Rules
Put this in Network → Firewall → Custom Rules. This uses atomic loading to prevent WebUI lockouts.
Bash
# V2.0 RUTX10 FINAL - DHCP Isolation, Starvation & Ping Fix
sleep 2
logger -t "FirewallGuard" "Loading V2.0 Lean L2 protection..."
# MTU fixes
ip link set dev vxlan1 mtu 1370 2>/dev/null
ip link set dev vxlan2 mtu 1370 2>/dev/null
ip link set dev br-lan mtu 1500 2>/dev/null
nft -f - <<EOF
flush table bridge security_guard 2>/dev/null
delete table bridge security_guard 2>/dev/null
table bridge security_guard {
chain prerouting {
type filter hook prerouting priority -300; policy accept;
# 1. DHCP ISOLATION (Block foreign DHCP from VXLAN)
iifname "vxlan*" udp dport { 67, 68 } drop
iifname "vxlan*" udp sport { 67, 68 } drop
# 2. DHCP STARVATION PROTECTION (Limit local LAN requests)
iifname "br-lan" udp dport 67 limit rate 5/second burst 10 packets accept
iifname "br-lan" udp dport 67 drop
}
chain forward {
type filter hook forward priority 0; policy accept;
# 3. PING STABILIZATION (MSS Clamping)
tcp flags syn tcp option maxseg size set 1330
# Techson Camera Discovery
udp dport 23456 accept
ip daddr { 234.55.55.55, 234.55.55.56 } accept
# Noise Filtering (Block Windows/NetBIOS towards VXLAN)
oifname "vxlan*" udp dport { 137, 138 } drop
oifname "vxlan*" tcp dport { 139, 445 } drop
}
}
EOF
logger -t "FirewallGuard" "V2.0 rules active."
2. TSW202 (Managed Switch) - V2.0 Smart Multicast Route
Put this in System → Custom Scripts (/etc/rc.local). It intelligently waits for the SFP1 port instead of a blind timer.
Bash
# V2.0 TSW202 Techson Multicast Fix
(
# Wait until the SFP1 port is actually UP
while ! ip link show sfp1 | grep -q "LOWER_UP"; do sleep 2; done
logger -t "TechsonFix" "SFP1 active, adding hardware mdb entries..."
bridge mdb add dev br0 port sfp1 grp 234.55.55.55 permanent 2>/dev/null
bridge mdb add dev br0 port sfp1 grp 234.55.55.56 permanent 2>/dev/null
logger -t "TechsonFix" "Hardware mdb OK."
) &
exit 0
3. TAP200 (Wi-Fi AP) - V2.0 Wireless Edge Protection
Run this ebtables script at boot. We disabled Promiscuous mode to drastically lower Wi-Fi latency.
Bash
#!/bin/ash
# V2.0 TAP200 FINAL - No Promisc, Clean Multicast & Starvation Protection
logger -t "APGuard" "Starting V2.0 TAP200 protection..."
# 1. Hardware Offloading (Promisc OFF, Allmulti ON)
for iface in $(ls /sys/class/net/ | grep wlan); do
ip link set dev $iface promisc off 2>/dev/null
ip link set dev $iface allmulticast on 2>/dev/null
done
# 2. Bridge settings
ip link set dev br-lan allmulticast on
echo 0 > /sys/devices/virtual/net/br-lan/bridge/multicast_snooping 2>/dev/null
# 3. ebtables: Targeted L2 Filtering
ebtables -F
ebtables -t nat -F
# Allow Techson Discovery & IGMP
ebtables -A FORWARD -p IPv4 --ip-proto igmp -j ACCEPT
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 23456 -j ACCEPT
ebtables -A FORWARD -p IPv4 --ip-dst 234.55.55.55 -j ACCEPT
ebtables -A FORWARD -p IPv4 --ip-dst 234.55.55.56 -j ACCEPT
# DHCP Starvation Protection (From Wi-Fi)
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 67 -i wlan+ --limit 5/sec --limit-burst 10 -j ACCEPT
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 67 -i wlan+ -j DROP
# Noise Filtering
ebtables -A FORWARD -p IPv4 --ip-proto udp --ip-dport 137:138 -j DROP
ebtables -A FORWARD -p IPv4 --ip-proto tcp --ip-dport 139 -j DROP
ebtables -A FORWARD -p IPv4 --ip-proto tcp --ip-dport 445 -j DROP
logger -t "APGuard" "TAP200 V2.0 ACTIVE."
4. RUTX11 (Single VXLAN VPN Router) - V2.0 Custom Firewall
For deployments requiring only a single VXLAN tunnel. It uses the exact same atomic nftables logic and prerouting DHCP isolation as the RUTX10.
Bash
# V2.0 RUTX11 FINAL - DHCP Isolation, Starvation & Ping Fix
sleep 2
logger -t "FirewallGuard" "Loading V2.0 Lean L2 protection..."
ip link set dev vxlan1 mtu 1370 2>/dev/null
ip link set dev br-lan mtu 1500 2>/dev/null
nft -f - <<EOF
flush table bridge security_guard 2>/dev/null
delete table bridge security_guard 2>/dev/null
table bridge security_guard {
chain prerouting {
type filter hook prerouting priority -300; policy accept;
# 1. DHCP ISOLATION (Block foreign DHCP from VXLAN)
iifname "vxlan*" udp dport { 67, 68 } drop
iifname "vxlan*" udp sport { 67, 68 } drop
# 2. DHCP STARVATION PROTECTION (Limit local LAN requests)
iifname "br-lan" udp dport 67 limit rate 5/second burst 10 packets accept
iifname "br-lan" udp dport 67 drop
}
chain forward {
type filter hook forward priority 0; policy accept;
# 3. PING STABILIZATION (MSS Clamping)
tcp flags syn tcp option maxseg size set 1330
# Techson Camera Discovery
udp dport 23456 accept
ip daddr { 234.55.55.55, 234.55.55.56 } accept
# Noise Filtering
oifname "vxlan*" udp dport { 137, 138 } drop
oifname "vxlan*" tcp dport { 139, 445 } drop
}
}
EOF
logger -t "FirewallGuard" "RUTX11 V2.0 rules active."
5. RUT241 (Client VPN Router) - V2.0 Lean Architecture (IPTables)
The RUT241 has a more modest CPU compared to the RUTX series. To preserve CPU cycles and keep the ping stable, we bypassed nftables bridge filtering and used the iptables raw table. This drops rogue DHCP requests at the earliest possible stage in the kernel. MTU fragmentation is fixed via mangle table MSS Clamping.
Bash
# V2.0 RUT241 FINAL - Starvation & Isolation (Lean version)
sleep 5
logger -t "ClientGuard" "Starting V2.0 RUT241 protection..."
# MTU fixes
ip link set dev jz mtu 1420 2>/dev/null
ip link set dev vxlan1 mtu 1370 2>/dev/null
ip link set dev br-lan mtu 1500 2>/dev/null
# RAW table setup (Fastest filtering level for lower-end CPUs)
iptables -t raw -F PREROUTING 2>/dev/null
# 1. DHCP STARVATION PROTECTION (Local LAN)
iptables -t raw -A PREROUTING -i br-lan -p udp --dport 67 -m limit --limit 5/sec --limit-burst 10 -j ACCEPT
iptables -t raw -A PREROUTING -i br-lan -p udp --dport 67 -j DROP
# 2. DHCP ISOLATION (Block foreign DHCP from VXLAN)
iptables -t raw -A PREROUTING -i vxlan+ -p udp --dport 67:68 -j DROP
# 3. PING FIX & FORWARDING
iptables -t mangle -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1330
iptables -I FORWARD -p udp --dport 23456 -j ACCEPT
iptables -I FORWARD -d 234.55.55.55 -j ACCEPT
iptables -I FORWARD -d 234.55.55.56 -j ACCEPT
# Noise Filtering
iptables -A FORWARD -o vxlan+ -p udp --dport 137:139 -j DROP
iptables -A FORWARD -o vxlan+ -p tcp --dport 139 -j DROP
iptables -A FORWARD -o vxlan+ -p tcp --dport 445 -j DROP
logger -t "ClientGuard" "RUT241 V2.0 rules active."
FINAL MEGA-UPDATE (V2.3 / V2.5) – THE “DOUBLE-SIDED SEAL” PRODUCTION RELEASE