How to configure a FreeBSD Jail with vnet and ZFS

nixcraft · 2018-07-15 20:34:37 UTC

Originally published at: https://www.cyberciti.biz/faq/how-to-configure-a-freebsd-jail-with-vnet-and-zfs/

How do I install and configure a FreeBSD jail with vnet? How do I create FreeBSD jail with /etc/jail.conf without using iocage command or ezjail command line tool?

joe_luzar · 2018-12-10 22:26:46 UTC

You show tun interface but no where do you explain anything about it. Where did it come from? How did it get ip addresses. Please explain.

nixcraft · 2018-12-11 05:50:01 UTC

tun0 is my openvpn interface running inside jail. I should update the page to avoid confusion.

joe_luzar · 2018-12-11 16:22:06 UTC

It’s time to redo this article. FreeBSD RELEASE-12.0 was published yesterday 12/10/2018. Vimage now comes built into the base kernel and ipfw and pf firewalls are now vnet aware. IPFW still has problem with its log intermingling vnet jail log records with the hosts log records in the security log file. Netgraph is a bad example as its so complicated to use. bridge/epair is the method most commonly used and is a far better example. The jib jng scripts are examples them selves and have their own problems. Should not base your example on them.

Would like to see your sample use ufs instead of zfs. IE: fetch the base.txz file and unpack it for a full directory tree vnet jail. Use a jail.conf that includes the ifconfig commands to create the bridge and epair instead of burying then in the vnet jails rc.conf. You can use ifconfig commands to rename the bridge to include the host interface name facing the public internet and rename the epair to include the jail name like jib does. Use the pf firewall in a vnet jail running on a gateway host so NAT is required. Explain how public traffic is driven to the vnet jail. pf can be loaded as a module now so don’t have to run pf on the host as a requirement.

I have researched the internet and all the vnet jail how-to are based on vnet jails running on a host that is a node on a LAN. This is very different setup than for a vnet jail running on a gateway host.
Just to be clear, A gateway host is a host that is directly connected to the users ISP.

During my research I posted replies to the articles and you are the first one to ever reply. That means your site is maintained and has interest to keep up with changing times. I thank you for that.

I think this would really be helpful as it’s never been documented before.

Thank you for considering this request.

nixcraft · 2018-12-11 16:25:46 UTC

Yes, I am waiting for FreeBSD 12 host here. Mostly I will run on AWS and VM at AWS come default with UFS and directly connected to the Internet. I will write a new post covering that.

yhq_34 · 2018-12-16 07:58:10 UTC

I have some errors here, seems my config file doesn’t work?

service jail start

Starting jails:ngctl: send msg: No such file or directory
jail: rsnapshot: jng bridge rsnapshot em0: failed

nixcraft · 2018-12-16 15:17:57 UTC

Make sure you run ALL command as root user. It seems command ngctl not found. Can you see ngctl?

type ngctl

It shows path:

ngctl is hashed (/usr/sbin/ngctl)

yhq_34 · 2018-12-17 01:11:12 UTC

Yes, it run as root, ngctl was located at /usr/sbin.

ngctl

Available commands:
config get or set configuration of node at
connect Connects hook of the node at to
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at “path”
msg Send a netgraph control message to the node at “path”
name Assign name to the node at
read Read and execute commands from a file
rmhook Disconnect hook “hook” of the node at “path”
show Show information about the node at
shutdown Shutdown the node at
status Get human readable status information from the node at
types Show information about all installed node types
write Send a data packet down the hook named by “hook”.
quit Exit program

ls /usr/sbin/ngc*

/usr/sbin/ngctl

nixcraft · 2018-12-17 05:07:19 UTC

@yhq_34, Do you have em0 interface on FreeBSD server? Can you paste your /etc/jail.conf from the server host?

cat /etc/jail.conf
ifconfig -a

yhq_34 · 2018-12-17 05:26:57 UTC

yes, same with your config.

cat /etc/jail.conf

Config:

freebsd1 {
        host.hostname = "freebsd1";   # hostname
        path = "/jails/freebsd1";     # root directory
        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";
        # ##########################################################################
        # netgraph/vnet config info
        # ng0 is my vnet
        # idb1 is my physical network interface connected to the LAN (use ifconfig)
        # jng is located in /usr/sbin/
        # rsnapshot is my jail name
        # ##########################################################################
        vnet;
        vnet.interface = "ng0_freebsd1";               # vnet interface(s)
        exec.prestart += "jng bridge freebsd1 em0";   # bridge interface(s)
        exec.poststop += "jng shutdown freebsd1";      # destroy interface(s)

        # Standard stuff
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_freebsd1_console.log";
        mount.devfs;          #mount devfs
        allow.raw_sockets;    #allow ping-pong
        devfs_ruleset="5";    #devfs ruleset for this jail
        mount.devfs;
}

NIC info:

ifconfig -a

Outputs:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:23:24:5d:5d:e4
        inet6 fe80::223:24ff:fe5d:5de4%em0 prefixlen 64 scopeid 0x1
        inet 192.168.0.33 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

nixcraft · 2018-12-17 05:55:51 UTC

Seems correct config. Are you using FreeBSD 11.x? What does /jails/freebsd1/etc/rc.conf says?

cat  /jails/freebsd1/etc/rc.conf

Also look into /var/log/jail_freebsd1_console.log for more info:

cat /var/log/jail_freebsd1_console.log

Please format your command and output with Markdown. See https://commonmark.org/help/

yhq_34 · 2018-12-17 06:08:32 UTC

I have upgrade to 12.0 version. /jails/freebsd1/etc/rc.conf was same with you written.
No .log file at /var/log/ folder.

nixcraft · 2018-12-17 06:21:37 UTC

This tutorial created for FreeBSD 11.x. So it won’t work on FreeBSD 12. There are some changes.

yhq_34 · 2018-12-17 06:54:49 UTC

Do you plan to update the tutorial to suitable with FreeBSD 12?

nixcraft · 2018-12-17 06:56:20 UTC

I am planning to write a new one as there are many changes between 11 and 12 version. Stay tuned.

joe_luzar · 2019-01-09 19:11:40 UTC

Hello Vivek Gite;
Been trying to get vnet jails to work on a gateway host. Having problems with vnet network access to the public internet. I need some help. From this article I can tell you have the networking knowledge that can help me figure out what is wrong with my vnet configuration. I wrote my problem as a draft for an article that you may use to post on this site. I can be reached at joeb_722 at yahoo dot com

This is an example of a vnet jail configuration that is run on a real hardware computer having a network interface connected to a public internet service provider. FreeBSD 12.0 is the operating system running on this hardware.

VIMAGE which is required for vnet jails is included in the FreeBSD base kernel in this release, and the PF firewall and its NAT (network address translation) function have also been adapted to be vnet jail aware. A vnet jail requires NAT to access the public internet.

The bridge/epair method is used to connect the vnet jails network to the hosts network. The setting up and the tearing down of the vnet jails bridge/epair is done in the vnet jails jail.conf file.

The hosts "service jail" command is used to start/stop the vnet jails. The setup configurations are further separated into all the vnet jails being defined in a single jail.conf file or each jail definition being in a separate jail.conf file.

Information on the sysrc command and the "service jail" command.

The sysrc command adds or changes statements in the hosts /etc/rc.conf file.
Additions are placed at the bottom of the /etc/rc.conf file.

$ sysrc jail_enable=YES
Used to enable the "service jail" command. Will auto start jails at boot time and auto stop jails at halt & shutdown time.

$ sysrc jail_list=""
Used by the "service jail" command. Empty list means all jails are in a single /etc/jail.conf file.

$ sysrc jail_list="jail-name jail-name …"
Used by the "service jail" command. Means search /etc for jail.xxx.conf files where xxx is jail-name from the list.

This "service jail" command setup works for both vnet jails and non-vnets.

Format of service command

$ service jail start
$ service jail restart
$ service jail stop

or selectively;

$ service jail start jail-name
$ service jail restart jail-name
$ service jail stop jail-name

Creating vnet jails directory tree the easy way.
mkdir -p /usr/local/jails cd /usr/local/jails fetch -avrA "http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz"
Repeat the following 2 commands changing the jail-name to create as many vnet jails as you want.
mkdir jail-name xzdec base.txz | tar --unlink -xpJf - -C "jail-name"
Copy host files needed by the jail.

cp /etc/resolv.conf jail-name/etc
cp /etc/localtime jail-name/etc

Preparing the host’s /etc/rc.conf for jails.

Sample 1: Single jail.conf(5) method.
All jail definitions in single jail.conf file.
On the host system console issue following commands one time.

sysrc jail_enable=YES
sysrc jail_list=""

Copy the jail.conf statements shown below and paste into /etc/jail.conf
Edit the /etc/jail.conf following the edit instructions below.

Sample 2: Per-jail jail.conf(5)
Each jail definition in individual jail.jail-name.conf file.
On the host system console issue following commands one time.

sysrc jail_enable=YES
sysrc jail_list+="jail-name jail-name jail-name ..."

Copy the jail.conf statements shown below and paste
into /etc/jail.jail-name.conf
Edit the /etc/jail.jail-name.conf following the edit instructions below.

This is the example vnet jail jail.conf content.

jail-name {
host.hostname = "jail-name";
path = "/usr/local/jails/jail-name";
exec.consolelog = "/var/log/jail.jail-name.console.log";
mount.devfs;
devfs_ruleset = "70";
vnet = "new";
vnet.interface = "epair15b";
exec.prestart = "ifconfig bridge15 create up";
exec.prestart += "ifconfig epair15 create up";
exec.prestart += "ifconfig bridge15 addm epair15a";
exec.prestart += "ifconfig bridge15 addm re0";
exec.start = "ifconfig epair15b inet 10.0.110.25/24";
exec.start += "route add default 10.0.110.2";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig epair15a destroy";
exec.poststop += "sleep 2";
exec.poststop += "ifconfig bridge15 destroy";
}

Standard edit instructions for the jail.conf file.

Substitute all 4 places where the word jail-name is shown with the real jail name created above.
Vnet jails must have an ip address assigned to it. Change the ip address 10.0.110.25/24 to what ever ip address you’re using on your host system leaving the /24 suffix.

All vnet jails must have a route pointing to the hosts default gateway ip address assigned by the ISP. Issue "route -n get default" command. Take the gateway ip address shown in the output display and substitute it for the 1.2.3.4 ip address. ???

The re0 in the exec.prestart line represents the interface name of the interface facing the public internet. Replace re0 with the interface name facing your public internet. You can determine the interface name to use from the "route -n get default" command issued on the host.

Setup PF firewall to run inside of vnet jail.

Copy the following and paste it into the /etc/rc.conf file of the vnet jail you’re setting up.

gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
ftpproxy_enable="NO"

Pre-allocate the pflog file
touch jail-name/var/log/pflog

Vnet jails need devfsrules to enable pf firewall to run.

On the host create /etc/devfs.rules file and populate it with the following content. Then issue the "service devfs restart" command to enable this new rule.

This only needs to be done one time.

devfsrules for pf to function in a vnet jail.

[vnet_pf=70]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path ‘bpf*’ unhide
add path pf unhide
add path pflog unhide

Copy the following and paste it into the /etc/pf.conf file of the vnet jail you’re setting up. This is very simple PF rules, pass every thing in and out and log, so traffic can be viewed.

set block-policy drop
set fail-policy drop
set state-policy if-bound
scrub in on epair15b all
set skip on lo0
nat on epair15b from 10.0.110.25 to any -> host-public-ip
pass out log (all) quick on epair15b from any to any
pass in log (all) quick on epair15b from any to any

PF information

PF is now a loadable module. If PF is used as the hosts firewall than nothing further needs to be done. If the host is running IPFW or IPFILTER firewall then PF has to be loaded into the kernel one time per boot using this command "kldload pf.ko pflog.ko" or added to boot/loader.conf. To see what modules have been loaded issue kldstat -v. To unload the PF modules
use "kldunload -f pf.ko pflog.ko" command.

Testing your vnet jail.

From the hosts console.
service jail start
service jail stop

It may take up to 30 seconds for the vnet jail to complete starting.
You can issue the "jls" command to verify the jail is running.
To log into the running jail issue "jexec jail-name login -f root"

monkeybrains · 2019-01-17 03:49:22 UTC

Thanks for post, you got me running! Info for others: if you see errors like this:

 # service jail start
 Starting jails:epair0a
 ifconfig: BRDGADD ix0: File exists
 jail: ns1: ifconfig bridge0 addm ix0: failed

I had a typo, tried to start the jail and if failed. I corrected the typo but the bridge0 was already there, so the jails didn’t start. I manually ran ‘ifconfig bridge 0 destory’ and the jail started up find with ‘service jail start’.