Originally published at: https://www.cyberciti.biz/faq/how-to-configure-a-freebsd-jail-with-vnet-and-zfs/
How do I install and configure a FreeBSD jail with vnet? How do I create FreeBSD jail with /etc/jail.conf without using iocage command or ezjail command line tool?
Edit: FreeBSD 12 user please follow updated guide - How to set up FreeBSD 12 VNET jail with ZFS
You show tun interface but no where do you explain anything about it. Where did it come from? How did it get ip addresses. Please explain.
tun0 is my openvpn interface running inside jail. I should update the page to avoid confusion.
Itâs time to redo this article. FreeBSD RELEASE-12.0 was published yesterday 12/10/2018. Vimage now comes built into the base kernel and ipfw and pf firewalls are now vnet aware. IPFW still has problem with its log intermingling vnet jail log records with the hosts log records in the security log file. Netgraph is a bad example as its so complicated to use. bridge/epair is the method most commonly used and is a far better example. The jib jng scripts are examples them selves and have their own problems. Should not base your example on them.
Would like to see your sample use ufs instead of zfs. IE: fetch the base.txz file and unpack it for a full directory tree vnet jail. Use a jail.conf that includes the ifconfig commands to create the bridge and epair instead of burying then in the vnet jails rc.conf. You can use ifconfig commands to rename the bridge to include the host interface name facing the public internet and rename the epair to include the jail name like jib does. Use the pf firewall in a vnet jail running on a gateway host so NAT is required. Explain how public traffic is driven to the vnet jail. pf can be loaded as a module now so donât have to run pf on the host as a requirement.
I have researched the internet and all the vnet jail how-to are based on vnet jails running on a host that is a node on a LAN. This is very different setup than for a vnet jail running on a gateway host.
Just to be clear, A gateway host is a host that is directly connected to the users ISP.
During my research I posted replies to the articles and you are the first one to ever reply. That means your site is maintained and has interest to keep up with changing times. I thank you for that.
I think this would really be helpful as itâs never been documented before.
Thank you for considering this request.
Yes, I am waiting for FreeBSD 12 host here. Mostly I will run on AWS and VM at AWS come default with UFS and directly connected to the Internet. I will write a new post covering that.
I have some errors here, seems my config file doesnât work?
Starting jails:ngctl: send msg: No such file or directory
jail: rsnapshot: jng bridge rsnapshot em0: failed
Make sure you run ALL command as root user. It seems command ngctl not found. Can you see ngctl?
type ngctl
It shows path:
ngctl is hashed (/usr/sbin/ngctl)
Yes, it run as root, ngctl was located at /usr/sbin.
Available commands:
config get or set configuration of node at
connect Connects hook of the node at to
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at âpathâ
msg Send a netgraph control message to the node at âpathâ
name Assign name to the node at
read Read and execute commands from a file
rmhook Disconnect hook âhookâ of the node at âpathâ
show Show information about the node at
shutdown Shutdown the node at
status Get human readable status information from the node at
types Show information about all installed node types
write Send a data packet down the hook named by âhookâ.
quit Exit program
/usr/sbin/ngctl
@yhq_34, Do you have em0 interface on FreeBSD server? Can you paste your /etc/jail.conf from the server host?
cat /etc/jail.conf
ifconfig -a
yes, same with your config.
cat /etc/jail.conf
Config:
freebsd1 {
host.hostname = "freebsd1"; # hostname
path = "/jails/freebsd1"; # root directory
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
# ##########################################################################
# netgraph/vnet config info
# ng0 is my vnet
# idb1 is my physical network interface connected to the LAN (use ifconfig)
# jng is located in /usr/sbin/
# rsnapshot is my jail name
# ##########################################################################
vnet;
vnet.interface = "ng0_freebsd1"; # vnet interface(s)
exec.prestart += "jng bridge freebsd1 em0"; # bridge interface(s)
exec.poststop += "jng shutdown freebsd1"; # destroy interface(s)
# Standard stuff
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_freebsd1_console.log";
mount.devfs; #mount devfs
allow.raw_sockets; #allow ping-pong
devfs_ruleset="5"; #devfs ruleset for this jail
mount.devfs;
}
NIC info:
ifconfig -a
Outputs:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
ether 00:23:24:5d:5d:e4
inet6 fe80::223:24ff:fe5d:5de4%em0 prefixlen 64 scopeid 0x1
inet 192.168.0.33 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Seems correct config. Are you using FreeBSD 11.x? What does /jails/freebsd1/etc/rc.conf says?
cat /jails/freebsd1/etc/rc.conf
Also look into /var/log/jail_freebsd1_console.log for more info:
cat /var/log/jail_freebsd1_console.log
Please format your command and output with Markdown. See https://commonmark.org/help/
I have upgrade to 12.0 version. /jails/freebsd1/etc/rc.conf was same with you written.
No .log file at /var/log/ folder.
This tutorial created for FreeBSD 11.x. So it wonât work on FreeBSD 12. There are some changes.
Do you plan to update the tutorial to suitable with FreeBSD 12?
I am planning to write a new one as there are many changes between 11 and 12 version. Stay tuned.
Hello Vivek Gite;
Been trying to get vnet jails to work on a gateway host. Having problems with vnet network access to the public internet. I need some help. From this article I can tell you have the networking knowledge that can help me figure out what is wrong with my vnet configuration. I wrote my problem as a draft for an article that you may use to post on this site. I can be reached at joeb_722 at yahoo dot com
This is an example of a vnet jail configuration that is run on a real hardware computer having a network interface connected to a public internet service provider. FreeBSD 12.0 is the operating system running on this hardware.
VIMAGE which is required for vnet jails is included in the FreeBSD base kernel in this release, and the PF firewall and its NAT (network address translation) function have also been adapted to be vnet jail aware. A vnet jail requires NAT to access the public internet.
The bridge/epair method is used to connect the vnet jails network to the hosts network. The setting up and the tearing down of the vnet jails bridge/epair is done in the vnet jails jail.conf file.
The hosts "service jail" command is used to start/stop the vnet jails. The setup configurations are further separated into all the vnet jails being defined in a single jail.conf file or each jail definition being in a separate jail.conf file.
Information on the sysrc command and the "service jail" command.
The sysrc command adds or changes statements in the hosts /etc/rc.conf file.
Additions are placed at the bottom of the /etc/rc.conf file.
$ sysrc jail_enable=YES
Used to enable the "service jail" command. Will auto start jails at boot time and auto stop jails at halt & shutdown time.
$ sysrc jail_list=""
Used by the "service jail" command. Empty list means all jails are in a single /etc/jail.conf file.
$ sysrc jail_list="jail-name jail-name âŚ"
Used by the "service jail" command. Means search /etc for jail.xxx.conf files where xxx is jail-name from the list.
This "service jail" command setup works for both vnet jails and non-vnets.
Format of service command
$ service jail start
$ service jail restart
$ service jail stop
or selectively;
$ service jail start jail-name
$ service jail restart jail-name
$ service jail stop jail-name
Creating vnet jails directory tree the easy way.
mkdir -p /usr/local/jails
cd /usr/local/jails
fetch -avrA "http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz"
Repeat the following 2 commands changing the jail-name to create as many vnet jails as you want.
mkdir jail-name
xzdec base.txz | tar --unlink -xpJf - -C "jail-name"
Copy host files needed by the jail.
cp /etc/resolv.conf jail-name/etc
cp /etc/localtime jail-name/etc
Preparing the hostâs /etc/rc.conf for jails.
Sample 1: Single jail.conf(5) method.
All jail definitions in single jail.conf file.
On the host system console issue following commands one time.
sysrc jail_enable=YES sysrc jail_list=""
Copy the jail.conf statements shown below and paste into /etc/jail.conf
Edit the /etc/jail.conf following the edit instructions below.
Sample 2: Per-jail jail.conf(5)
Each jail definition in individual jail.jail-name.conf file.
On the host system console issue following commands one time.
sysrc jail_enable=YES sysrc jail_list+="jail-name jail-name jail-name ..."
Copy the jail.conf statements shown below and paste
into /etc/jail.jail-name.conf
Edit the /etc/jail.jail-name.conf following the edit instructions below.
This is the example vnet jail jail.conf content.
jail-name {
host.hostname = "jail-name";
path = "/usr/local/jails/jail-name";
exec.consolelog = "/var/log/jail.jail-name.console.log";
mount.devfs;
devfs_ruleset = "70";
vnet = "new";
vnet.interface = "epair15b";
exec.prestart = "ifconfig bridge15 create up";
exec.prestart += "ifconfig epair15 create up";
exec.prestart += "ifconfig bridge15 addm epair15a";
exec.prestart += "ifconfig bridge15 addm re0";
exec.start = "ifconfig epair15b inet 10.0.110.25/24";
exec.start += "route add default 10.0.110.2";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig epair15a destroy";
exec.poststop += "sleep 2";
exec.poststop += "ifconfig bridge15 destroy";
}
Standard edit instructions for the jail.conf file.
Substitute all 4 places where the word jail-name is shown with the real jail name created above.
Vnet jails must have an ip address assigned to it. Change the ip address 10.0.110.25/24 to what ever ip address youâre using on your host system leaving the /24 suffix.
All vnet jails must have a route pointing to the hosts default gateway ip address assigned by the ISP. Issue "route -n get default" command. Take the gateway ip address shown in the output display and substitute it for the 1.2.3.4 ip address. ???
The re0 in the exec.prestart line represents the interface name of the interface facing the public internet. Replace re0 with the interface name facing your public internet. You can determine the interface name to use from the "route -n get default" command issued on the host.
Setup PF firewall to run inside of vnet jail.
Copy the following and paste it into the /etc/rc.conf file of the vnet jail youâre setting up.
gateway_enable="YES" pf_enable="YES" pf_rules="/etc/pf.conf" pflog_enable="YES" pflog_logfile="/var/log/pflog" ftpproxy_enable="NO"
Pre-allocate the pflog file
touch jail-name/var/log/pflog
Vnet jails need devfsrules to enable pf firewall to run.
On the host create /etc/devfs.rules file and populate it with the following content. Then issue the "service devfs restart" command to enable this new rule.
This only needs to be done one time.
[vnet_pf=70]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path âbpf*â unhide
add path pf unhide
add path pflog unhide
Copy the following and paste it into the /etc/pf.conf file of the vnet jail youâre setting up. This is very simple PF rules, pass every thing in and out and log, so traffic can be viewed.
set block-policy drop
set fail-policy drop
set state-policy if-bound
scrub in on epair15b all
set skip on lo0
nat on epair15b from 10.0.110.25 to any -> host-public-ip
pass out log (all) quick on epair15b from any to any
pass in log (all) quick on epair15b from any to any
PF information
PF is now a loadable module. If PF is used as the hosts firewall than nothing further needs to be done. If the host is running IPFW or IPFILTER firewall then PF has to be loaded into the kernel one time per boot using this command "kldload pf.ko pflog.ko" or added to boot/loader.conf. To see what modules have been loaded issue kldstat -v. To unload the PF modules
use "kldunload -f pf.ko pflog.ko" command.
Testing your vnet jail.
From the hosts console.
service jail start
service jail stop
It may take up to 30 seconds for the vnet jail to complete starting.
You can issue the "jls" command to verify the jail is running.
To log into the running jail issue "jexec jail-name login -f root"
Thanks for post, you got me running! Info for others: if you see errors like this:
# service jail start Starting jails:epair0a ifconfig: BRDGADD ix0: File exists jail: ns1: ifconfig bridge0 addm ix0: failed
I had a typo, tried to start the jail and if failed. I corrected the typo but the bridge0 was already there, so the jails didnât start. I manually ran âifconfig bridge 0 destoryâ and the jail started up find with âservice jail startâ.
Is there a way I can look into this on my own, your blog post is about the only thing Iâve found on the subject 
Hello all,
See update FreeBSD 12 specific guide