Hi Everyone,
Been using pfsense for many years before switching to opnsense at its inception.
Used the forums for years silently, and am very happy with this fantastic piece of software.
Today I am reporting a successful deployment of Opnsense on an Oracle Cloud free instance.
Thought I'd share this short tutorial to give back to the community, hoping it'll be useful to someone out there:
Some Background (you can skip this section and go straight to the OCI tutorial):
I have been using Opnsense on several VPS providers successfully over the years.
First Digital Ocean, then Vultr, then Upcloud.
The switch was done each time to select a VPS location closer to me for better latency.
The three were good from a technical standpoint, although Vultr is to be avoided if you do not want to be outright scammed. ::)
Was most recently on Upcloud, they have a great platform, great customer service, but they jacked up their monthly subscription price by nearly 50% overnight due to rising energy costs, so I decided to check out alternatives.
A few months ago I started a free trial on OCI. Unfortunately the location I had selected did not have any available capacity during the initial free trial period.
Recently some "Always Free" AMD instances, 1CPU core / 1GB ram became available in my preferred location, so I decided to deploy Opnsense.
Freebsd & Opnsense are not offered as options on OCI for X64, so a bit of creativity was in order.
OCI free instance deployment steps:
-Prepare an xml config for the Oracle deployment on a local machine.
The main required changes from vanilla were:
- 1 NIC only for WAN, configured via DHCP
- Disable HTTP_REFERER enforcement check
- Create a VCN and Public subnet on OCI, setup the Oracle firewall to allow SSH/HTTP/HTTPS ingress
- Fire up an Oracle Linux instance on AMD (mine was VM.Standard.E2.1.Micro -1 core 2.55 GHz AMD EPYC\u2122 7J13).
An SSH key was uploaded during instance setup for remote access.
- Convert the Oracle Linux instance into a FreeBSD instance via SSH using:
# wget https://download.freebsd.org/ftp/releases/VM-IMAGES/13.1-RELEASE/amd64/Latest/FreeBSD-13.1-RELEASE-amd64.raw.xz
# xz -dc FreeBSD-13.1-RELEASE-amd64.raw.xz | sudo dd of=/dev/sda bs=1M conv=fdatasync
Where sda is the boot disk.
Reboot to Freebsd
- Upload the prepared config.xml to the Freebsd instance using ssh
- Convert FreeBSD into Opnsense via SSH using opnsense-bootstrap:
# pkg install ca_root_nss
# fetch https://raw.githubusercontent.com/opnsense/update/master/src/bootstrap/opnsense-bootstrap.sh.in
# sh ./opnsense-bootstrap.sh.in -r 23.1
- Using web based cloud console interface, load the previously prepared config.xml during opnsense first boot
Now the Opensense instance is accessible via the web interface for further configuration.
Since this setup was done using an Always Free instance, the good news is this cloud based edge router comes at no cost.
Enjoy :)
Great tutorial! Much easier than my old method (https://forum.opnsense.org/index.php?topic=22163.msg113748#msg113748) which involved uploading your own customised OPNsense image.
I would be a bit concerned that overwriting the volume while Linux is running from it could cause data corruption. Anyhow, you can avoid this by temporarily attaching the boot volume to a separate compute instance.
Uploading a config.xml is optional btw. You can also use the cloud console to perform the initial interface assignment and then use the Web GUI for everything else. If a machine only has a WAN interface, OPNsense by default allows Web GUI access from the WAN.
Next challenge: OPNsense on OCI Ampere instances (aarch64). More bang for your no buck. There even is an official FreeBSD 13.1 image for OCI Ampere, so you could skip the dd part. Unfortunately, opnsense-bootstrap doesn't support aarch64 yet.
Cheers
Maurice
can confirm that this works, and you don't need to prepare or download anything locally. Every step can be accomplished using the cloud console
thanks a lo! Wiping /dev/sda while running an OS mounted on it is wild!
In hindsight, I don't think these wild workarounds are actually worth it. I've since switched to the "normal" workflow (upload qcow2 image to storage bucket, then import as custom "Generic Linux" image). Much faster and easier. You need a "Pay As You Go" account, but get to keep all "Always Free" resources. My September bill was 8 Cents - because I needed a little more than the free 3000 Compute A1 OCPU hours. I don't think I've ever exceeded the free quota for custom images.
Oh, and I've also completed the switch from VM.Standard.E2.1.Micro to VM.Standard.A1.Flex. Much more powerful!
Cheers
Maurice
Hello
Many thanks for the heads up regarding aarch64, and for hosting the files.
I have managed to install opnsense on a free A1 instance (4cores,24GB).
In summary:
Using a linux instance, I copied the contents of your qcow2 file into a boot disk, that I then attached to a full blown A1 instance.
Everything works well so far, there was no need to upgrade to a paid account (like you said, I agree a "pay as you go upgrade" would have made it a tiny bit easier, but it can be done whilst staying on the Free Tier).
i have uploaded the xml file to freebsd.
how i can import the file within config import?
Press any key to start the configuration importer: .
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
<QEMU QEMU TARGET 2.5> at scbus2 target 0 lun 0 (pass0)
<ORACLE BlockVolume 1.0> at scbus2 target 0 lun 1 (pass1,da0)
Select device to import from (e.g. ada0) or leave blank to exit: da0
Starting import for partition '/dev/da0p3'.
Running fsck...done.
mount: /dev/da0p3: Invalid fstype: Invalid argument
The device could not be mounted.
The file /conf/config.xml could not be found.
where i have to put file in?
/conf/config.xml ?
thanks
daniel
Sorry I just read your post, and responded to your private msg.
As explained there, one way to do it is to use a separate block volume.
There is no need for a config import during first boot. Just perform the interface assignment (WAN = vtnet0), set a secure root password, log in on the Web GUI and go from there.
i got it running.
with cloud-console i didnt got a prompt.
i placed the config now in /conf/config.xml
after running the opnsense-script the config was imported automatically during boot. now all is working fine!
thanks a lot!
Hi guys, congratulations for the excellent work!
My dream would be to add a kubernetes cluster on oracle (OKE) to this setup that is protected by the OPNsense firewall, i.e. having the k8s cluster nodes behind the firewall.
This way I could make an on premise firewall scenario that has a site to site vpn with the OPNsense on oci, which would give me access to the micro services on the kubernetes cluster (like pi-hole).
I don't know which of the 2 solutions may be best:
1) use an oke cluster
2) create a VM and install a k8s cluster as if we were in an on premise scenario, but losing flexibility of oke
Any advice/suggestions?
Just to make sense of my post on this thread, you guys are going to use OPNsense for what/how on OCI?
Hey Alfa,
I'm actually not using OPNsense OCI instances as a firewall for other machines in OCI. Primary use cases currently are:
- DNS-over-TLS server with ad blocking, to be used with Android devices.
- WireGuard tunnel endpoint. When the cable WAN at home fails and my home firewall falls back to LTE, I can no longer access it remotely because the mobile ISP blocks all inbound connections. "OPNsense Home" then connects to "OPNsense OCI" via WireGuard and the home network remains accessible via OPNsense OCI.
I have no experience with Kubernetes, sorry.
Cheers
Maurice
Quote from: Maurice on October 19, 2023, 11:56:14 PM
In hindsight, I don't think these wild workarounds are actually worth it. I've since switched to the "normal" workflow (upload qcow2 image to storage bucket, then import as custom "Generic Linux" image). Much faster and easier. You need a "Pay As You Go" account, but get to keep all "Always Free" resources.
Hi Maurice, can you give a detailed explanation how to solve the "Disable HTTP_REFERER enforcement check" problem? I don't want to upload a config file if I don't have to.
A oneliner would be good because I can't save with nano in the Cloud Shell.
This problem is caused by Oracle's use of IPv4 SNAT, so make sure the instance has an IPv6 address. You can configure this when creating the instance or add one later in the instance's network settings. Then, access the Web GUI using its IPv6 literal, e. g. https://[2001:db8:1:2::a].
Next, go to System: Settings: General and enter the hostname and domain you want to use to access the Web GUI. You must create matching AAAA / A records in your DNS zone of course. You can then access the Web GUI using this hostname, e. g. https://opnsense.example.com. Adding a valid certificate is strongly recommended, you can use the ACME plugin for this.
Quote from: Maurice on April 12, 2024, 12:51:08 PM
This problem is caused by Oracle's use of IPv4 SNAT
Sry, I can't follow, IPv6 isn't supported on the free tier right now.
The only problem for me so far is that I get the "Disable HTTP_REFERER enforcement check" in the web-UI.
I need to disable it!
In pfSense this can be easily done in the console, not so much in OPNsense.
Also I can't connect via putty at this point, although I allowed this in OCI.
So I was trying changing the config with "sed" but no luck so far. Also I am only a novice user.
I already used IPv6 when I was still on the free tier and would be very surprised if they removed this. I'm happy to assist with issues specific to OPNsense on OCI, but I can't give a full introduction to OCI in general. There is a slight learning curve, but Oracle has pretty decent documentation.
Oracle assigns a private IPv4 address to the instance and then uses SNAT to translate it to the public IPv4 address. OPNsense is not aware of this and only sees the private IPv4 address. That's why it refuses connections when you enter the public IPv4 address in your browser.
IPv6 doesn't have this issue because the instance gets assigned a public address - there is no NAT.
You can't initially use putty because SSH is disabled by default. You have to enable it in the OPNsense Web GUI first.
There are certainly other ways like manually editing the config.xml, but I haven't tried that. Have you tried using ee on the cloud console? That should work.
Quote from: Maurice on April 12, 2024, 01:33:13 PM
Have you tried using ee on the cloud console? That should work.
Will give it a try but again, very novice user, so I might fail yet again. Thanks anyway. To bad there is no oneliner with "sed" or something.
Quote from: Maurice on April 12, 2024, 01:33:13 PM
I already used IPv6 when I was still on the free tier and would be very surprised if they removed this.
You where right, IPv6 is still possible. For whatever reason I can't login to the Cloud Console right now so I am unable to do anything.
Edit: it will not let my use the CC anymore, it had worked before...
Quote from: Maurice on April 12, 2024, 01:33:13 PM
I already used IPv6 when I was still on the free tier
So finally got it working with IPv6, thank you again.
The problem with the not working Cloud Console was that I downloaded the nano Image a second time and this time I had manually selected a mirror (cloudflare)... big mistake. The image that was given to me was not bootable/to be made bootable. Only took me some hours to find out and download from leaseweb again...
Embrace it as a great learning experience! 😊
Glad it works.
Quote from: Maurice on April 12, 2024, 07:18:49 PM
Embrace it as a great learning experience!
But what have I learned, that I can not trust the OPNsense Mirrors for downloads? It was a great waste of time.
Anyways, if you have an idea, so that my free tier doesn't get canceled by oracle, pls let me know. I heard some stuff, you need to utilize those to some degree.
https://docs.opnsense.org/manual/install.html#download-and-verification
When you're on the free tier, Oracle monitors your resource utilization and terminates instances it considers idle. That's to avoid orphaned non-revenue instances existing indefinitely. You can avoid this by upgrading to pay-as-you-go (which is still free as long as you don't exceed the free tier quotas).
Hey Maurice,
thanks for your effort bringing arm Images to OCI :)
Which qemu file do I need for the OCI Ampere Instance? ufs-serial oder ufs-efi?
When I do a "qemu-img resize filename.qemu 40G" will this create a swap partition? Or is it just 40G root and no swap?
I'm a bit lost with the next steps, I have a PAYG account. Which distribution do I need? Is it just adding a linux instance, upload the qemu to a bucket and insert it into the boot image and go?
Assistance is much appreciated, thanks everyone
Quote from: n0ir on May 21, 2024, 11:08:31 PM
Which qemu file do I need for the OCI Ampere Instance? ufs-serial oder ufs-efi?
Both work, but the serial image is recommended. It allows using the cloud console for initial configuration (interface assignment, root password).
Quote from: n0ir on May 21, 2024, 11:08:31 PM
When I do a "qemu-img resize filename.qemu 40G" will this create a swap partition? Or is it just 40G root and no swap?
qemu-img only resizes the disk image, not the partitions inside it. OPNsense then expands the root partition on first boot. It does not create a swap partition. I recommend building your own image if you need a swap partition.
None of this should be necessary for OCI though. They automatically expand the image to the configured disk size (default 47 GB), no manual resizing required. And I never felt the need for swap since the Ampere instances have plenty of RAM.
Quote from: n0ir on May 21, 2024, 11:08:31 PM
I'm a bit lost with the next steps, I have a PAYG account.
Upload the qcow2 image to a bucket, then import it as a custom image (generic Linux works) and configure it to support UEFI boot and Ampere instances. Then create an instance using this image.
Quote from: Maurice on May 22, 2024, 01:40:32 PM
Upload the qcow2 image to a bucket, then import it as a custom image (generic Linux works) and configure it to support UEFI boot and Ampere instances. Then create an instance using this image.
Thanks Maurice, did the steps till the Instance creation:
1. Downloaded the latest OPNsense-24.1.7-ufs-serial-vm-aarch64.qcow2.bz2
2. Unzipped and uploaded qcow2 to bucket
3. Imported custom image (type qcow2, launch mode native / paravirtualized -> tried both)
4. Create Instance from this custom image
I can't select the Ampere Instance. Below the Image there is a Lock-symbol with the letters "BS" inside. It says: This image supports confidential computing. Confidential computing is hardware technology in CPUs that encrypts data in-use while being processed and protects against these threats.
When I try to change the shape to Ampere it states: This shape is either not compatible with the selected image, or not available in the current availability domain.
Any ideas?
EDIT: Found that I need to Edit the Custom Image and allow VM.Standard.A1.Flex in Image Details. It was not ticket so I did and this solved the selection in Instances
Paravirtualized mode is correct, but you might have missed this step:
[...] and configure it to support UEFI boot and Ampere instances.
By default, custom images created by importing an image from a bucket are configured to only support legacy BIOS firmware and amd64 instances. You have to change this in the custom image's capabilities and details before creating an instance.
Thanks, got it working. The public IP is natted 1:1 for WAN interface, there is no way to expose the public IP directly the WAN interface i guess?
Quote from: n0ir on May 22, 2024, 08:32:08 PM
there is no way to expose the public IP directly the WAN interface i guess?
I don't think so but the IPv6-address is.
Hi!
I like to give this torial a try but it seems the version 13.1 of FreeBSD is no more available.
So i used 13.2 instead.
I was able to dd the image to the disk, also the reboot to FreeBSD works like a charm.
When i execute the opnsense-bootstrap.sh.in i get the error that ca_root_nss is already unlocket.
Then the script ends.
What can i do?
BR
Christian
Quote from: kohly on May 29, 2024, 04:23:03 PM
What can i do?
Use the latest nano image and don't select the mirror yourself.
Quote from: Bob.Dig on May 29, 2024, 07:55:54 PM
Use the latest nano image and don't select the mirror yourself.
Thank you for your response.
I found a 13.1 image at ftp-archive.freebsd.org and will try again with this one.
BR
Christian
solved the problem: just not install ca_root_nss
BR
Christian
Quote from: Bob.Dig on May 29, 2024, 07:55:54 PM
Quote from: kohly on May 29, 2024, 04:23:03 PM
What can i do?
Use the latest nano image and don't select the mirror yourself.
found the nano image and was able to deploy it on oci.
what kind of ease!
I did spend some time to document what I've done. So maybe useful for others: https://blog.dieschmitterlinge.de/cloud/install-opnsense-oracle-oci/