the box that runs your house, opened ▸ ┃┃ pull one cable, nothing drops
the box that runs your house, opened ▸ ┃┃ pull one cable, nothing dropsEvery home has one box doing five jobs — carrying the internet, keeping the bad traffic out, handing each device an address, turning names like google.com into numbers, and running the Wi-Fi.
Mine was a sealed plastic box I could not read a line of. Four of those jobs run on my own server now — each a file I can open, with a log line per packet.
Pull one of the two cables feeding it and the network stays up — that check is in the record below, with its output.
The five jobs, where they run now, and what that buys you
| the job | runs on now | what that buys you |
|---|---|---|
| carrying the internet | my server | a log line per packet |
| keeping the bad out | software on my server | the rules are a text file |
| handing out addresses | my server, 41 free | your pool, your leases |
| turning names into numbers | my server | you decide what answers |
| running the Wi-Fi | the old router | nothing re-pairs in the house |
The words here, in plain English
- a bond — two cables used as one, so pulling either changes nothing
- vmbr0 — my server’s own name for that pair
- OPNsense — the free firewall software that took the deciding job
- Proxmox — the free software that runs another computer inside my server
Three doors in — take the one for your evening
Understand it — the five jobs, where each runs now. Five minutes, no hardware.
Move the cables — two per side, so a pull changes nothing. The step I did first.
Hand the deciding over — the firewall takes the internet, the rules, the addresses and the names. The old router keeps the Wi-Fi.
Every address here is a made-up example — 192.0.2.x never belongs in your live network. Swap in your own before copying a line of this.
🧰 The kit — the parts, the files, the settings
Everything you need to copy any part of this: the parts list, the file that is live now, the two-cable settings, and the machine’s own settings.
The parts, as built
| System | Role / Notes |
|---|---|
| Hypervisor A | Proxmox VE host; Linux 7.0.14-19-pve; hostname observed as hypervisor-a . |
| Intel i350 | 4 × 1 GbE NIC. Proxmox names: nic1 through nic4 . |
| Onboard Intel | nic0 ; currently provides safety/SSH path at 192.0.2.58/24 . |
| Aruba S3500 | L3 switch. Hypervisor A LACP ports: GE-0/0/22 + GE-0/0/23 . |
| RealHD 8-port managed L2 | Receives TP-Link uplink; ports 2 + 3 form Dexter WAN LACP. Former direct 3-port LACP to Aruba is disconnected. |
| TP-Link BE4500 / BE400 | Current NAT/DHCP router; target role is AP-only Wi-Fi 7. |
| fiber ISP / ISP ONT | 8-Gig fiber service. Future goal: remove eero/router layer from routing path and connect own firewall behind ONT if ISP provisioning permits. |
The file that is live now
auto lo
iface lo inet loopback
iface nic1 inet manual
iface nic2 inet manual
auto bond1
iface bond1 inet manual
bond-slaves nic1 nic2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer2+3
iface nic3 inet manual
iface nic4 inet manual
auto bond0
iface bond0 inet manual
bond-slaves nic3 nic4
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer2+3
auto vmbr1
iface vmbr1 inet static
address 192.0.2.5/24
gateway 192.0.2.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
iface nic0 inet manual
source /etc/network/interfaces.d/*
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond1
bridge-stp off
bridge-fd 0
The two-cable settings — the same on each side
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer2+3
The machine, as built
| Setting | As built |
|---|---|
| VM | ID 100 ; name firewall-vm100 ; Start at boot ON |
| Platform | q35, OVMF UEFI, EFI disk local-zfs , pre-enroll keys OFF, TPM OFF |
| Storage | VirtIO SCSI single; scsi0 32 GiB on local-zfs , discard ON, IO thread ON, SSD emulation ON, backup ON |
| CPU / RAM | 1 socket, 4 host-type cores, 8192 MiB RAM, balloon OFF, KSM ON |
| WAN | VirtIO net0 on vmbr0 , MAC [WAN-vNIC-MAC] → OPNsense vtnet0 ; DHCP lease observed 198.51.100.191/24 |
| LAN | VirtIO net1 on vmbr1 , MAC [LAN-vNIC-MAC] → OPNsense vtnet1 ; 192.0.2.1/24 |
| Proxmox NIC firewall | OFF on both VM NICs; OPNsense provides firewall policy |
| Guest agent | Configured in VM options; guest-side installation/operational status still pending verification |
| OPNsense identity | Hostname firewall-vm ; domain lab.example ; FQDN firewall-vm.lab.example |
| Management GUI | https://192.0.2.1 ; self-signed HTTPS certificate |
| Timezone | America/New_York |
📚 The full build record — the deep end, every chapter
Every chapter, in the order it happened — the full engineering record, checks and outputs included. Skip it and the kit above still works.
What is in here: the parts and the physical topology · the hypervisor as it was before the change · the switch’s link aggregation · the address plan · the migration sequence · the PCI mapping and the verification command reference · the post-reboot proof · the current as-built state (13) · the firewall VM’s configuration (14) · the firmware upgrade and its results (15).
One Hack — extended technical build log (sanitized public edition)
Publication note: This is the full engineering record, not a short recap. All private hostnames, user names, domain names, interface MAC addresses, and internal IPs have been replaced or masked. The example ranges
192.0.2.0/24,198.51.100.0/24, and203.0.113.0/24are documentation-only TEST-NET ranges, not working RFC1918 LAN configurations. Do not paste them into your own router or hypervisor. Exact historic and current states are distinguished below.
Timeline: The earlier sections record the original state and migration plan, including steps that are no longer current. Later sections contain the verified as-built configuration. This is intentional: preserving the chronology explains why the migration decisions were made.
Current state — read this before the historical steps
As of the final documented verification: Proxmox VE 9.2.20 hosts a two-NIC OPNsense VM. The WAN-side switch reaches an IP-less hypervisor bridge through a two-port LACP bond. The firewall VM’s LAN reaches the Aruba through a separate two-port LACP bond and the management bridge. The old direct switch-to-switch bypass has been unplugged. The upstream router still supplies WAN DHCP; OPNsense supplies LAN DHCP/DNS. OPNsense was upgraded to 26.7.4_1 with zero upgrade candidates at the last check, and the workstation’s public-IP ping and firewall DNS tests passed.
Do not follow historical steps about disabling upstream DHCP or removing the WAN bridge. They refer to a different planned stage. The current as-built chapter near the end supersedes them.
Quick index
Original inventory and topology; pre-change interfaces; Aruba LACP; address plan; safe migration; WAN/LAN design; ISP handoff and throughput; verification command reference; PCI mapping; post-reboot proof; current topology; full VM settings; wizard decisions; staged firmware upgrade; client routing/DNS tests; pending work.
Hypervisor A — Proxmox VE / OPNsense Build Record
Build note: Objective: Build Hypervisor A once, document it, and migrate routing services safely. Proxmox management will move to the Intel i350 LACP connection; OPNsense will ultimately provide firewall/NAT, DHCP, and Unbound DNS. The TP-Link will become a Wi-Fi 7 access point after OPNsense is proven.
1. Known Hardware & Systems
| System | Role / Notes |
|---|---|
| Hypervisor A | Proxmox VE host; Linux 7.0.14-19-pve; hostname observed as hypervisor-a . |
| Intel i350 | 4 × 1 GbE NIC. Proxmox names: nic1 through nic4 . |
| Onboard Intel | nic0 ; currently provides safety/SSH path at 192.0.2.58/24 . |
| Aruba S3500 | L3 switch. Hypervisor A LACP ports: GE-0/0/22 + GE-0/0/23 . |
| RealHD 8-port managed L2 | Receives TP-Link uplink; ports 2 + 3 form Dexter WAN LACP. Former direct 3-port LACP to Aruba is disconnected. |
| TP-Link BE4500 / BE400 | Current NAT/DHCP router; target role is AP-only Wi-Fi 7. |
| fiber ISP / ISP ONT | 8-Gig fiber service. Future goal: remove eero/router layer from routing path and connect own firewall behind ONT if ISP provisioning permits. |
2. Current Physical Topology
ISP / ONT
|
eero
|
TP-Link router (upstream DHCP + NAT)
|
RealHD 8-port MANAGED L2 switch
| ports 2 + 3: ACTIVE 802.3ad LACP
+---- Dexter nic1 + nic2 / bond1 / vmbr0
| |
| Sentinel WAN (OPNsense)
| |
| Sentinel LAN / vmbr1 / Dexter bond0
| |
| Aruba S3500 ports 22 + 23 (ACTIVE LACP)
| |
| LAN clients, including Mikey
|
+---- Former direct 3-port LACP to Aruba:
PHYSICALLY DISCONNECTED (not an active path)
3. Hypervisor A — Configuration Observed Before Changes
| Interface | State / Address | Purpose |
|---|---|---|
| nic0 | UP — 192.0.2.58/24 | Onboard Intel; current SSH/safety connection. |
| nic1 | UP, enslaved to vmbr0 | i350 port; current temporary/private arrangement. |
| nic2 | DOWN | i350; available. |
| nic3 | DOWN | i350; target LACP member. |
| nic4 | DOWN | i350; target LACP member. |
| vmbr0 | 203.0.113.61/24 , gateway 203.0.113.1 | Existing bridge on nic1; to be reworked after new management path is verified. |
Original /etc/network/interfaces
auto lo
iface lo inet loopback
iface nic1 inet manual
auto vmbr0
iface vmbr0 inet static
address 203.0.113.61/24
gateway 203.0.113.1
bridge-ports nic1
bridge-stp off
bridge-fd 0
iface nic2 inet manual
iface nic3 inet manual
iface nic4 inet manual
iface nic0 inet manual
source /etc/network/interfaces.d/*
4. Aruba S3500 Link Aggregation
| LAG Name | Members | Link |
|---|---|---|
| RealHD-to-Aruba | GE-0/0/0, GE-0/0/1, GE-0/0/2 | 3 × 1G |
| Aruba2Workstation A | GE-0/0/12, GE-0/0/13 | 2 × 1G |
| Hypervisor A-LACP2port | GE-0/0/22, GE-0/0/23 | 2 × 1G |
The initial Hypervisor A LAG is intended to carry the existing LAN untagged. VLANs can be introduced deliberately later rather than complicating the first LACP migration.
5. Target LAN Address Plan
| Range / Address | Assignment |
|---|---|
| 192.0.2.1 | OPNsense LAN gateway; future DHCP and Unbound DNS endpoint. |
| 192.0.2.2–9 | Network infrastructure. |
| 192.0.2.5 | Hypervisor A Proxmox management target. |
| 192.0.2.10–109 | Servers / infrastructure / statics. |
| 192.0.2.110+ | Reserved static-address space in the historical plan; no deployment recorded. |
| 192.0.2.200–240 | Future OPNsense DHCP pool — 41 addresses. |
| 192.0.2.241–254 | Reserved for future infrastructure. |
6. Immediate LACP Migration
Build note: Safety rule Keep nic0 / 192.0.2.58 operational until the new LACP management path is verified from another machine. Do not remove the existing default gateway or TP-Link DHCP/NAT during this stage.
Hypervisor A i350
nic3 ─────┐
├── bond0 (802.3ad / LACP) ── management bridge ── 192.0.2.5/24
nic4 ─────┘
│
└── Aruba S3500 GE-0/0/22 + GE-0/0/23
nic0 ───────── 192.0.2.58/24 ← temporary safety/SSH path
Planned bond parameters
bond-mode 802.3ad
bond-miimon 100
bond-xmit-hash-policy layer2+3
bond-lacp-rate fast
Build note: LACP verification completed 2026-09-27. Hypervisor A bond0 was brought up independently before activating the bridge. Linux and the Aruba S3500 successfully negotiated IEEE 802.3ad with both members in the same active aggregator.
| Verified item | Observed value |
|---|---|
| Bond | bond0 |
| Mode | IEEE 802.3ad Dynamic Link Aggregation |
| Transmit hash | layer2+3 |
| MII status | UP |
| LACP mode / rate | Active / Fast |
| Active aggregator | Aggregator ID 1 — 2 ports |
| Hypervisor A members | nic3 + nic4 |
| nic3 | 1000 Mbps, full duplex, MII UP, Aggregator 1, 0 link failures |
| nic4 | 1000 Mbps, full duplex, MII UP, Aggregator 1, 0 link failures |
| Aruba partner MAC | [switch-LACP-MAC] |
| Aruba partner key | 2 |
| Aruba LACP group | Group 1 — GE-0/0/22 + GE-0/0/23 |
Current stopping point: bond0 is live and verified. vmbr1 has not yet been activated or connectivity-tested. The existing nic0 / 192.0.2.58 safety SSH path remains in use.
Verification commands
ip -br link
ip -br addr
ip route
cat /proc/net/bonding/bond0
bridge link
bridge vlan show
Success criteria: both nic3 and nic4 are link-up, both participate in the same 802.3ad aggregator, 192.0.2.5 is reachable from Workstation A, and the original 192.0.2.58 safety path remains available during testing.
7. Target OPNsense Architecture
Future ISP side
fiber ISP
│
ISP ONT
│
│ Ethernet handoff
▼
Dedicated Proxmox WAN bridge
(NO Proxmox host IP)
│
OPNsense VM WAN
│
OPNsense firewall / NAT
DHCP + Unbound DNS
│
OPNsense VM LAN
│
Proxmox LAN bridge / bond0
│
Aruba / RealHD
│
├── wired LAN
└── TP-Link in AP-only mode → Wi-Fi 7
Design principle: Proxmox itself should not have a management address on the true ISP/WAN bridge. The physical WAN interface/bridge should connect the ISP handoff to the OPNsense WAN vNIC while leaving the Proxmox host unnumbered on that segment.
8. OPNsense Service Targets
Build note: LAN 192.0.2.1/24 Default gateway for LAN clients.
Build note: DHCP 192.0.2.200–240 Enabled only after TP-Link DHCP is disabled.
Build note: DNS OPNsense Unbound LAN clients use OPNsense as DNS.
Build note: TP-Link AP-only mode after OPNsense routing/DHCP/DNS are tested.
9. Ripple / WAN Items to Obtain
- PPPoE username, if Ripple uses PPPoE on this circuit.
- PPPoE password.
- Required WAN VLAN ID, if any.
- Confirmation that a customer-owned router/firewall may connect directly behind the ISP ONT.
Do not dismantle the working eero/TP-Link path until OPNsense WAN connectivity has been configured and tested.
10. 8-Gig Capacity Note
The Intel i350 is a 1-Gigabit-per-port adapter. A 2-port LACP bond can provide roughly 2 Gb/s of aggregate capacity across multiple flows, but a single ordinary flow remains constrained by the selected physical member. It therefore cannot deliver the full Ripple 8-Gig service.
A future 10GbE NIC is appropriate if Hypervisor A is expected to route multi-gigabit WAN traffic through OPNsense. The ISP ONT can then feed a suitable high-speed WAN interface, while the LAN side also needs sufficient multi-gigabit capacity to realize the benefit.
11. Migration Sequence / Checklist
- Document existing Hypervisor A interfaces and Aruba LAG layout.
- Confirmed 192.0.2.5 unused: Workstation A neighbor discovery returned FAILED .
- Connected i350 nic3 + nic4 to Aruba Group 1, GE-0/0/22 + GE-0/0/23 .
- Created and activated Hypervisor A bond0 in IEEE 802.3ad mode.
- Verified both 1G/full-duplex members in the same active LACP aggregator with zero link failures.
- Create/activate management LAN bridge vmbr1 above bond0 at 192.0.2.5/24 ; no competing default gateway.
- Verify 192.0.2.5 reachability from Workstation A while retaining the 192.0.2.58 safety session.
- Retain nic0 / 192.0.2.58 until the new management path is proven.
- Clean up/recreate Proxmox bridges for final LAN/WAN roles.
- Upload OPNsense ISO and create OPNsense VM.
- Configure OPNsense LAN at 192.0.2.1/24 .
- Configure and test WAN using Ripple-provided requirements.
- Configure Unbound and DHCP pool 192.0.2.200–240 .
- Disable TP-Link DHCP/NAT and convert it to AP-only mode.
- Test wired/Wi-Fi clients, DNS, DHCP, NAT, firewall, Proxmox management, and fail/reboot behavior.
- Only after successful testing, retire temporary network paths and finalize documentation.
12. Change Log
| Date | Change | Status |
|---|---|---|
| 2026-09-27 | Initial Hypervisor A topology, current interfaces, Aruba LACP layout, addressing plan, OPNsense target, and migration sequence documented. | Baseline |
| 2026-09-27 | Verified nic3 and nic4 at 1000/full with carrier; backed up and validated network configuration; activated bond0 ; confirmed IEEE 802.3ad active/fast LACP with both members in Aggregator 1, Aruba partner MAC [switch-LACP-MAC] , partner key 2, and zero link failures. | Verified |
| 2026-09-27 | Next controlled step: activate vmbr1 at 192.0.2.5/24 , verify address, then test from Workstation A. Existing nic0 / 192.0.2.58 remains the safety path. | Pending |
Recordkeeping: Hypervisor A PMX Setup — working build record. Update this document as configuration is applied and verified. Avoid recording ISP passwords, private keys, or other credentials in this file.
Intel I350-T4 Identification and Linux / PCI Mapping
The four-port add-in NIC in Hypervisor A has now been positively identified from PCI data as an Intel I350 Gigabit Network Connection using the Linux igb driver. The motherboard NIC is a separate Realtek RTL8111/8168-family controller using r8169 .
Build note: Verified I350 mapping Proxmox/Linux name PCI function Controller Current role nic1 0000:04:00.0 Intel I350 Manual / currently unused nic2 0000:04:00.1 Intel I350 Manual / currently unused nic3 0000:04:00.2 Intel I350 LACP bond0 member nic4 0000:04:00.3 Intel I350 LACP bond0 member
Command: identify Ethernet controllers and drivers
lspci -nnk | grep -A3 -i ethernet
What it tells us: lspci enumerates PCI devices. -nnk includes vendor/device IDs and the kernel driver bound to each device. Filtering for Ethernet showed four Intel I350 functions at 04:00.0 through 04:00.3 , all using igb , plus the onboard Realtek controller at 06:00.0 using r8169 .
04:00.0 Intel I350 [8086:1521] driver: igb
04:00.1 Intel I350 [8086:1521] driver: igb
04:00.2 Intel I350 [8086:1521] driver: igb
04:00.3 Intel I350 [8086:1521] driver: igb
06:00.0 Realtek RTL8111/8168... driver: r8169
Command: map Proxmox interface names to PCI functions
for i in nic1 nic2 nic3 nic4; do
echo "$i -> $(basename "$(readlink -f /sys/class/net/$i/device)")"
done
What it tells us: Linux exposes each network interface’s backing PCI device under /sys/class/net//device . Following that symlink gives an authoritative mapping between the Proxmox interface name and PCI function.
nic1 -> 0000:04:00.0
nic2 -> 0000:04:00.1
nic3 -> 0000:04:00.2
nic4 -> 0000:04:00.3
Command: blink a NIC identification LED
ethtool -p nic3 10
What it is for: asks a supported NIC/driver to identify a port, normally by blinking an LED for the requested number of seconds. This can help trace cabling, but it should not override PCI/sysfs or bonding data when the visual result is ambiguous. During this setup the LED observations were ambiguous, so no cable was disconnected based on the LED test alone.
Command: verify the actual bond members
cat /sys/class/net/bond0/bonding/slaves
What it tells us: reads the kernel bonding driver’s current slave list directly. For Hypervisor A, the configured and previously verified LACP members are nic3 and nic4 .
Command: detailed LACP status
cat /proc/net/bonding/bond0
What it tells us: this is one of the most useful LACP troubleshooting views. It reports bond mode, MII state, LACP rate, transmit hash policy, active aggregator, partner information, individual slave speed/duplex, link failures, and churn state.
Build note: Learning note — physical jack numbering: Do not infer an I350 RJ45 jack’s Linux name solely from “closest to the motherboard” or “furthest from the motherboard.” Hypervisor A’s authoritative software mapping is the table above. Physical jack-to-PCI-function mapping should be verified rather than assumed.
Hypervisor A Network Verification Command Reference
These are the commands used during the migration. Keep this section as a quick diagnostic reference when changing Proxmox networking later.
Compact interface/address state
ip -br addr
Shows each interface, whether it is up/down, and its assigned IPv4/IPv6 addresses. This was used repeatedly to verify that vmbr1 retained 192.0.2.5/24 while old interfaces were removed.
Physical Ethernet link state
ethtool nic3 | grep -E "Speed|Duplex|Link detected"
ethtool nic4 | grep -E "Speed|Duplex|Link detected"
Confirms carrier, negotiated speed, and duplex. Both Hypervisor A LACP members were verified at 1000 Mb/s, full duplex, link detected .
Persistent ifupdown2 configuration validation
ifquery --check -a
Compares the configured interfaces with the running state and validates recognized interface attributes. After removing the obsolete 203.0.113.0/24 bridge, every bond0 and vmbr1 item returned [pass] .
Routing table
ip route
Shows the kernel’s active IPv4 routes. This exposed that the old live default route had already been removed while the persistent configuration still contained gateway 203.0.113.1 . That stale gateway was removed from /etc/network/interfaces before reboot testing.
ARP / neighbor table
ip neigh show 192.0.2.5
ip neigh show 192.0.2.51
Shows the Layer-2 neighbor/MAC learned for an IPv4 host. This helped reveal the temporary dual-homed condition while both onboard nic0 and vmbr1 were connected to the same 192.0.2.0/24 LAN.
Bring up only the new bond or bridge
ifup bond0
ifup vmbr1
Used deliberately instead of a full network reload. Bringing up one component at a time reduced the chance of losing the existing SSH safety path.
Retire the obsolete live bridge
ifdown vmbr0
After removing vmbr0 from the persistent configuration and validating the replacement path, this removed the old live 203.0.113.61/24 bridge without disturbing vmbr1 .
Connectivity test from Workstation A
ping -c 5 192.0.2.5
ssh [email protected]
With nic0 down, Workstation A still received 5/5 ICMP replies with 0% packet loss and the SSH connection to 192.0.2.5 reached Hypervisor A. This proved that management connectivity was independent of the old onboard-NIC path.
Build note: Current verified state as of 2026-09-27: vmbr0 and the obsolete 203.0.113.0/24 configuration have been removed from the persistent configuration and the live bridge has been taken down. nic0 is down; its old DHCP address 192.0.2.58/24 may remain attached in the current runtime until explicitly flushed or rebooted.
Hypervisor A management is working at 192.0.2.5/24 through vmbr1 → bond0 → nic3 + nic4 .
The reboot persistence test has PASSED . Hypervisor A returned at 192.0.2.5/24 over vmbr1 → bond0 → nic3 + nic4 , and post-boot LACP negotiation was independently verified.
Post-Reboot Persistence & LACP Verification — PASSED
Build note: Hypervisor A base management networking is now POST-REBOOT VERIFIED. After a real reboot, Workstation A successfully pinged 192.0.2.5 5/5 with 0% packet loss and SSH connected directly to Hypervisor A. The obsolete 203.0.113.0/24 network did not return.
Post-boot interface state
nic0 DOWN
nic1 DOWN
nic2 DOWN
nic3 UP SLAVE
nic4 UP SLAVE
bond0 UP MASTER
vmbr1 UP 192.0.2.5/24
Post-boot routing state
192.0.2.0/24 dev vmbr1 proto kernel scope link src 192.0.2.5
No obsolete 203.0.113.0/24 route and no stale 203.0.113.1 default gateway returned after reboot.
Authoritative post-reboot LACP check
cat /proc/net/bonding/bond0
| Item | Post-reboot result |
|---|---|
| Bonding mode | IEEE 802.3ad Dynamic Link Aggregation |
| Transmit hash | layer2+3 |
| MII status | UP |
| LACP | Active, fast rate |
| Active aggregator | Aggregator ID 1 |
| Number of ports | 2 |
| Partner key | 2 |
| Aruba partner MAC | [switch-LACP-MAC] |
| nic3 | 1000 Mb/s, full duplex, Aggregator 1, 0 link failures, no churn |
| nic4 | 1000 Mb/s, full duplex, Aggregator 1, 0 link failures, no churn |
| Aruba partner ports | 23 and 24 (corresponding to the configured Aruba LACP members) |
What this proves
The working management path is persistent across reboot and does not depend on the former onboard-NIC DHCP safety path. Hypervisor A management is now established as 192.0.2.5/24 → vmbr1 → bond0 → nic3 + nic4 → Aruba S3500 LACP Group 1 . This is the verified baseline before creating the OPNsense VM.
13. CURRENT AS-BUILT STATE — Firewall VM Online (2026-09-28)
Build note: CURRENT VERIFIED MILESTONE — supersedes the earlier target/pending states above. The preceding sections intentionally retain the original migration history, including former NIC addresses, temporary bridges, original upstream switch assumptions, and planned TP-Link AP conversion. Those passages describe what was known at the time , not the live network on September 28. In particular, vmbr0 was subsequently recreated as the IP-less WAN bridge; nic1 / nic2 are now active in WAN bond1 ; the TP-Link is still an upstream router with DHCP enabled. Do not use the old instructions to disable its DHCP in the current topology.
Current physical packet path
fiber ISP / ISP ONT
|
eero (192.0.2.1)
|
TP-Link consumer router (LAN 198.51.100.254/24)
DHCP on upstream: 198.51.100.170–199
|
RealHD 8-port MANAGED L2 switch (WAN-side)
ports 2 + 3
|
Hypervisor A nic1 + nic2 → bond1 (2 x 1G, 802.3ad)
|
vmbr0 (NO host IP)
|
Firewall VM VM100 WAN net0 / vtnet0
DHCP address observed: 198.51.100.191/24
|
OPNsense firewall / NAT / routing
|
Firewall VM VM100 LAN net1 / vtnet1
192.0.2.1/24; DHCP pool entered 192.0.2.200–240
|
vmbr1 (Hypervisor A management: 192.0.2.5/24)
|
bond0 → nic3 + nic4 (2 x 1G, 802.3ad)
|
Aruba S3500 (ports GE-0/0/22 + GE-0/0/23)
|
Workstation A and LAN clients
Critical isolation: the former direct 3 x 1G LACP link between the RealHD 8-port managed switch and Aruba has been physically unplugged . It must not be restored casually: doing so risks bypassing Firewall VM and/or introducing a layer-2 loop. Earlier references to a RealHD 5-port unmanaged switch are superseded by the verified RealHD 8-port managed L2 topology above.
Current Proxmox interface configuration
auto lo
iface lo inet loopback
iface nic1 inet manual
iface nic2 inet manual
auto bond1
iface bond1 inet manual
bond-slaves nic1 nic2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer2+3
iface nic3 inet manual
iface nic4 inet manual
auto bond0
iface bond0 inet manual
bond-slaves nic3 nic4
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer2+3
auto vmbr1
iface vmbr1 inet static
address 192.0.2.5/24
gateway 192.0.2.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
iface nic0 inet manual
source /etc/network/interfaces.d/*
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond1
bridge-stp off
bridge-fd 0
Host: hypervisor-a , Proxmox VE 9.2.20 . Management is only on the LAN bridge vmbr1 ; vmbr0 is the IP-less WAN bridge. The default route via Firewall VM was tested at runtime, then made persistent in /etc/network/interfaces . Backup of the previous file: /etc/network/interfaces.pre-opnsense-gateway . A post-cutover reboot retained management connectivity and active VM taps. Earlier Hypervisor A tests to Firewall VM, public IPv4, and external name resolution passed.
Why both DHCP servers stay enabled
| DHCP server | Scope / role | Current action |
|---|---|---|
| TP-Link consumer router | Upstream 198.51.100.0/24 ; supplies Firewall VM’s WAN DHCP lease. | KEEP ENABLED. |
| Firewall VM OPNsense | Downstream 192.0.2.0/24 ; LAN pool entered 192.0.2.200–240 . | KEEP ENABLED. |
| Aruba S3500 | LAN switching; not the LAN DHCP/DNS server. | Do not treat as the gateway/DHCP server. |
The two DHCP servers are on separate broadcast domains divided by Firewall VM. Turning off TP-Link DHCP now risks Firewall VM losing its WAN address on renewal. Earlier instructions to disable it were for a future, separate AP-only or direct-ONT cutover, not this working arrangement.
14. Firewall VM OPNsense VM100 — Installed Configuration
| Setting | As built |
|---|---|
| VM | ID 100 ; name firewall-vm100 ; Start at boot ON |
| Platform | q35, OVMF UEFI, EFI disk local-zfs , pre-enroll keys OFF, TPM OFF |
| Storage | VirtIO SCSI single; scsi0 32 GiB on local-zfs , discard ON, IO thread ON, SSD emulation ON, backup ON |
| CPU / RAM | 1 socket, 4 host-type cores, 8192 MiB RAM, balloon OFF, KSM ON |
| WAN | VirtIO net0 on vmbr0 , MAC [WAN-vNIC-MAC] → OPNsense vtnet0 ; DHCP lease observed 198.51.100.191/24 |
| LAN | VirtIO net1 on vmbr1 , MAC [LAN-vNIC-MAC] → OPNsense vtnet1 ; 192.0.2.1/24 |
| Proxmox NIC firewall | OFF on both VM NICs; OPNsense provides firewall policy |
| Guest agent | Configured in VM options; guest-side installation/operational status still pending verification |
| OPNsense identity | Hostname firewall-vm ; domain lab.example ; FQDN firewall-vm.lab.example |
| Management GUI | https://192.0.2.1 ; self-signed HTTPS certificate |
| Timezone | America/New_York |
Initial wizard decisions
- WAN: DHCP; MAC spoofing, MTU, MSS, and DHCP hostname blank; Block RFC1918 Private Networks UNCHECKED because the present WAN is a private upstream network; Block bogon networks CHECKED . Revisit RFC1918 rule if WAN becomes public.
- LAN: 192.0.2.1/24 ; DHCP enabled; pool entered 192.0.2.200–240 .
- General DNS: no manually specified DNS servers; Override DNS UNCHECKED; Unbound resolver enabled; DNSSEC and hardened DNSSEC data not enabled in the wizard.
- Deployment: Optimize for Multiwan OFF; Automatic DHCP/DNS registration ON; Optimize for IPsec OFF. The wizard described Dnsmasq DHCP/local registration integrated with Unbound, including port 53053 ; confirm live service details after the major upgrade.
- Root password was set earlier; wizard password fields left blank to retain the existing password. No credentials are stored in this document.
15. Firmware Upgrade & Post-Upgrade Verification — PASSED
Build note: Last checked 2026-09-28 ~03:02 EDT: OPNsense 26.7.4_1 (amd64) , 0 update candidates, 0 conflicts, packages up to date. GUI displayed no updates available on selected mirror.
The upgrade proceeded in stages: the earlier running version was 26.1.11_10 ; a major upgrade rebooted into 26.7.1_1 ; a subsequent batch reported 83 updates, 296.8 MiB, reboot required ; the final version was 26.7.4_1 . All stages completed without manually interrupting VM100.
Currently running OPNsense 26.7.4_1 (amd64)
Checking for upgrades (0 candidates): . done
Processing candidates (0 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Repeatable tests and actual results
| Test | Result | What it establishes |
|---|---|---|
| ping -c 3 1.1.1.1 on Workstation A | 3 sent, 3 received, 0% loss; average RTT 4.090 ms | Public-IP reachability via current routing path; DNS not involved. |
| nslookup openai.com 192.0.2.1 on Workstation A | Server 192.0.2.1#53 ; returned 104.18.33.45 and [private-IP] | Firewall VM DNS endpoint answers an explicit query; cache/forwarding mechanism not established. |
| resolvectl status on Workstation A | Active bond0 : Current DNS Server 192.0.2.1 , DNS Servers 192.0.2.1 , DNS Domain internal , Default Route yes | Workstation A is configured to use Firewall VM as its default DNS server; internal search suffix remains unexplained. |
| Firewall VM GUI login after reboot | Accessible at https://192.0.2.1 | LAN-side management GUI is reachable. |
Prior Workstation A observations included 192.0.2.215/24 with default gateway 192.0.2.1 . The currently displayed DHCP lease origin has not yet been independently confirmed. Public-domain DNS lookup success does not, by itself, identify which internal OPNsense DNS process or upstream cache was used.
Next investigations — not yet completed
- Trace Workstation A’s internal DNS search suffix (DHCP versus local network profile) before editing; intended domain is lab.example .
- Inspect actual Dnsmasq/Unbound status, DHCP pool, DNS registration, and DHCP lease after upgrading to 26.7.
- Consider split-horizon host overrides for firewall-vm.lab.example and hypervisor-a.lab.example .
- Verify the guest agent, Aruba management IP, and export/restore strategy for OPNsense configuration.
- For any future direct ISP ONT handoff or TP-Link AP-only conversion, first determine ISP VLAN/PPPoE/MAC/handoff requirements. The existing 2×1GbE LACP bonds cannot deliver 8Gbps to a single flow.
Change log additions
| Date | Verified milestone |
|---|---|
| 2026-09-27–28 | Created bond1 on nic1+nic2 for WAN and recreated vmbr0 as an IP-less bridge; retained bond0 / vmbr1 as LAN and Hypervisor A management. Physically removed the former direct RealHD8-to-Aruba trunk. |
| 2026-09-28 | Installed Firewall VM VM100; completed LAN/WAN mapping and initial wizard; tested host routing and LAN DNS. |
| 2026-09-28 | Persisted Hypervisor A gateway 192.0.2.1 ; completed staged OPNsense upgrades to 26.7.4_1 ; verified zero pending packages, Workstation A Internet ping, explicit Firewall VM DNS query, and default DNS configuration. |
This section is appended to preserve the original September 27 historical record while explicitly documenting the current September 28 state. All future changes should be verified and dated before inclusion.
September 28 night-session continuation — switch discovery, DHCP reservation and as-built inventory
Scope: Network build only. Earlier investigation lists are historical; Aruba discovery and reservation were completed, but password rotation was not.
16. September 28 Night Session — Aruba Discovery, DHCP Reservation, and Management Access
Verification standard: “Verified” means observed in the switch/OPNsense UI, logs, or a recorded test. “Planned” means discussed but not performed. This section updates the earlier chapter 15 investigation list without rewriting its historical statements.
16.1 Management-plane discovery
The Aruba S3500-24P was rediscovered after the LAN routing migration. Its management address is 192.0.2.220 and the observed management MAC is 02:00:00:00:06:C0. The previous TP-Link-era address was 192.0.2.55; that historical address is not the current management target. OPNsense Interfaces → Neighbors and a network scan were used to identify the device. Observed services included SSH on TCP/22, HTTP on TCP/80 redirecting to HTTPS on TCP/4343, and TLS identification associated with ArubaS3500-24P. A service scan also saw TCP/443; the working management browser URL was https://192.0.2.220:4343/.
The Aruba dashboard reported switch IP 192.0.2.220, default gateway 192.0.2.1, and model ArubaS3500-24P. Its ports 22 and 23 appeared up. Separately, Hypervisor A’s /proc/net/bonding/bond0 identified LACP partner system MAC 02:00:00:00:06:c0, corroborating the Aruba as the LAN-side LACP peer. This did not require changing switch VLAN, port, or LACP configuration.
16.2 DHCP reservation: observed before and after
| Field | Recorded value | Status |
|---|---|---|
| Service | OPNsense Services → Dnsmasq DNS & DHCP → Hosts | Configured |
| Host | core-switch |
Configured |
| Domain | lab.example |
Configured |
| Reserved IPv4 | 192.0.2.220 |
Verified |
| MAC | 02:00:00:00:06:c0 |
Verified |
| Client identifier | Blank | Recorded |
| Lease time / tag | Default / None | Recorded |
| Ignore / local DNS checkboxes | Unchecked | Recorded |
| Description | Aruba S3500 — LAN core switch | Recorded |
Initially the Dnsmasq lease for this MAC was marked dynamic. After saving the host mapping and applying configuration, the leases page explicitly showed the same MAC and 192.0.2.220 with hostname core-switch and lease type static. The UI briefly returned to an Apply state more than once; the later lease display and successful management login are stronger evidence of the active reservation than the transient button appearance. This verifies the DHCP host reservation; it does not prove the switch has a manually configured static interface address.
16.3 Dnsmasq DHCPv6 warning — not an IPv4 failure
Dnsmasq logs repeatedly displayed no address range available for DHCPv6 request via vtnet1, with some entries for vtnet0. These messages indicate DHCPv6 requests without a matching configured DHCPv6 range. They are not evidence that the Aruba’s IPv4 reservation failed. No global IPv6 disablement or DHCP service redesign was performed to suppress the warnings. If IPv6 service is later reviewed, scope the investigation by interface and desired IPv6 architecture.
16.4 Admin login succeeded; password rotation remains open
The Aruba web interface was reachable and the administrator authenticated successfully using the existing credential (the secret is deliberately not reproduced here). The web GUI’s Configuration → Basic Info page contains distinct fields for “Password for user Admin” (6–32 characters) and “Password for Enable Mode” (6–15 characters), each with a confirmation field. It also shows a page-level Apply button and a top-right Save Configuration control. Masked entries were visible, but the screenshot could not establish whether they were browser autofill, actual saved values, or placeholders. The password-change form was attempted repeatedly without a confirmed successful rotation. Current administrator password remains unchanged/unverified as rotated. No reset, switch reboot, LACP edit, or factory-default operation was undertaken. Future rotation should first back up the switch configuration, confirm field semantics using firmware-specific guidance or CLI, change only the intended credential, test a new session, and only then verify persistent save behavior.
The Aruba’s displayed manual clock/time zone differed from Firewall VM’s America/New_York setting; switch time synchronization remains a separate future task. Do not change it incidentally during password work.
17. Consolidated As-Built LAN/WAN Inventory — Confirmed at End of Session
| Element | Observed configuration / role |
|---|---|
| Hypervisor A hypervisor | Proxmox VE 9.2.20; management 192.0.2.5/24 on vmbr1; persistent default gateway 192.0.2.1. |
| WAN NICs / bridge | Intel I350 nic1+nic2 → bond1 802.3ad LACP → IP-less vmbr0 → Firewall VM vtnet0, MAC 02:00:00:00:19:B9. Physical LACP peer: RealHD8 managed L2 switch ports 2+3. |
| LAN NICs / bridge | nic3+nic4 → bond0 802.3ad LACP → vmbr1 → Aruba ports 22+23; Firewall VM vtnet1, MAC 02:00:00:00:17:87. |
| WAN chain | Ripple Fiber / Nokia ONT → eero (203.0.113.1) → TP-Link BE6500 (198.51.100.254/24) → RealHD8 → Hypervisor A WAN bond/bridge → Firewall VM WAN DHCP 198.51.100.191/24. |
| WAN-side DHCP | TP-Link pool 198.51.100.170–199 remains enabled for Firewall VM WAN; do not disable before a separately planned WAN cutover. |
| LAN routing | Firewall VM LAN 192.0.2.1/24; LAN DHCP pool 192.0.2.200–240; Unbound DNS responds on 192.0.2.1. |
| Firewall VM VM | VMID 100; OPNsense 26.7.4_1; q35, OVMF UEFI, 32 GiB local-zfs SCSI disk, 4 host vCPUs, 8 GiB RAM, two VirtIO NICs, autostart; sentinel.lab.example, America/New_York; GUI https://192.0.2.1. |
| Workstation A workstation | 192.0.2.215/24, default gateway 192.0.2.1, DNS 192.0.2.1. Previously verified ping to 1.1.1.1 3/3, 0% loss, about 4 ms average, and successful public DNS lookup via Firewall VM. |
| Other discovered host | 192.0.2.12, HP/Linux host with Pi-hole, Samba and web services observed by scan; no change made. |
| Aruba switch | S3500-24P management 192.0.2.220; Dnsmasq MAC reservation verified static; default gateway 192.0.2.1; ports 22+23 active; password rotation unresolved. |
| Physical bypass | Former three-port direct RealHD8 ↔ Aruba trunk physically unplugged to avoid bypass/loop. Do not reconnect casually. |
Topology note: Two distinct 802.3ad bonds carry WAN and LAN to separate L2 domains. A 2×1 GbE LACP bond does not provide a single 2 Gb/s flow and is not an 8 Gb/s WAN path. The existing working architecture is preserved pending a deliberate ISP/ONT cutover design.
Follow-up: correcting the active WAN wiring and auditing storage before the first backup
Evidence key: Interface and storage statements below are based on actual shell output. Physical cable assignments and the management-IP change are owner-reported. This is a pre-backup audit: no successful backup completion or restore test has yet been shown.
The managed switch is absolutely part of the active WAN
The earlier draft incorrectly depicted a five-port unmanaged switch and later risked implying that the eight-port switch was unused. Neither describes the current installation. The actual active path is:
Consumer router LAN port 1 — 2.5 GbE
|
+--> 8-port MANAGED L2 switch, port 1 (uplink)
|
+--> port 2 --+
| +--> two 1 GbE LACP members
+--> port 3 --+
|
Hypervisor Intel I350-T4 physical ports 0 + 1
|
nic1 + nic2 -> bond1 -> vmbr0
|
Firewall VM WAN
Firewall VM LAN
|
vmbr1 -> bond0 -> nic3 + nic4
|
Aruba S3500 ports 22 + 23
|
protected LAN clients
The RealHD-class managed switch is actively switching traffic from a 2.5 GbE uplink to a two-member 1 GbE LACP bond. LACP balances flows across members; it does not turn one flow into a 2 Gb/s connection. The former direct three-port trunk between the managed switch and the Aruba is a separate, obsolete connection: its cables were previously reported unplugged, while removal of its saved LAG configuration remains pending. The managed switch’s upstream port and active two-port LACP must not be disturbed.
The owner also reported moving the managed switch’s static management IP from the protected LAN subnet to the consumer-router-side subnet. The post-change address has not yet been independently verified from a new management session; that is not the same as a confirmed reachability test.
Exact Linux evidence: two independent bonds
The host reported nic1 and nic2 as UP,LOWER_UP and enslaved to bond1, which is attached to vmbr0. That WAN bridge has no host IPv4 address; the firewall VM’s first tap device is attached there. nic3 and nic4 were UP,LOWER_UP and enslaved to bond0, which is attached to vmbr1. The LAN bridge holds the host’s static management address and default gateway; the firewall VM’s second tap device is attached there. nic0 was down and outside both bonds.
Both bonds use the same recorded settings:
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer2+3
These commands establish interface membership and carrier state. They are not, alone, a substitute for checking negotiated LACP aggregator status via /proc/net/bonding/. The fe80:: addresses visible on the Linux bridges are ordinary IPv6 link-local addresses, not proof of a broken WAN.
Every physical drive accounted for—or explicitly marked unverified
| Device group | Observed status | Consequence |
|---|---|---|
| Two Samsung 870 EVO 1 TB SATA SSDs | rpool ZFS mirror-0, ONLINE, no known data errors |
Proxmox system and the firewall VM’s EFI/32 GiB virtual disk on local-zfs; firmware boot selection and EFI redundancy not separately checked. |
| Two Samsung 870 EVO 2 TB SATA SSDs | dexter-mirror ZFS mirror-0, ONLINE, no known data errors |
Independent pair of SSDs, approximately 1.76 TiB dataset-available for backup files. |
| Kingston KC3000 2 TB NVMe | Present, with a 1 GiB ext4 partition and a 1.9 TiB linux_raid_member partition; no active mdadm array shown |
Legacy RAID metadata, according to the owner, from an unsuccessful earlier setup. Candidate for future repurposing only after identity and contents are checked; not wiped. |
| Seagate Constellation 1 TB eSATA HDD | Owner-reported, absent from current lsblk |
Not yet available to Proxmox; physical/controller detection still to investigate. |
The two active mirrors are ZFS software RAID1 equivalents, not Linux mdadm arrays. /proc/mdstat therefore being empty is compatible with both ZFS mirrors operating normally. The Kingston’s stale RAID signature is not evidence of an active mirror.
A dedicated backup filesystem now exists—but the backups do not yet
The separate 2 TB SSD mirror was mounted at its existing ZFS mountpoint. A child dataset named backups was created and registered as backup-only directory storage in Proxmox. The verification showed the dataset mounted with approximately 1.76 TiB available, and the new backup storage was active in pvesm status.
This is an important milestone: the backup destination is ready on disks other than the system/VM pool. It is not yet a completed backup. At the last supplied output, the VM backup command had been proposed but no success log, archive file, restore test, host configuration archive, firewall configuration export, or off-host copy had been shown. The host inventory listed one running firewall VM and zero LXCs.
The remaining work is to execute and verify the VM backup, capture the Proxmox host and firewall configuration separately, and copy the result off the physical machine. A second mirror inside the same chassis protects against a different class of failure than an off-site or off-host backup.
The obsolete switch LAG cleanup, legacy Kingston investigation, and missing eSATA drive are separate pending maintenance items. No disks were reformatted and no working bonds were reconfigured during this audit.
As alwqays enjoy everybody. Mikey Likes_IT aka BCBC Technologies https://www.bchicbcow.com + ChatGPT
The rules your house runs on, in a file you own.
The Core-Community used AI to refine the post above.
!