Hello,
I’m unable to perform updates in the lxc container because it cannot connect to archive.ubuntu.com
DNS resolution is not working correctly, but I have temporarily resolved this by adding an entry to the hosts file.
Now, apt install can resolve archive.ubuntu.com but cannot connect to port 80
http is accessible on the archive.ubuntu.com host that I have added to hosts. This is an issue with the lxc host not being able to connect to http.
Honestly it shouldn’t be this hard
I wonder if the issue might be due to the lxc host intercepting the http replies?
If so, I might need some type of port forwarding, but ideally in a way that doesn’t interfere with http access on the lxc host (so that I don’t have to keep changing the port forwarding rule depending on which host is trying to access http).
Are you using firewall on Ubuntu host or router? Check those logs and see if it is blocking access.
I am running UFW on the host
Logging is set to low.
My understanding is that when logging is set to low, UFW is only supposed to log blocked connections.
So I run
sudo tail -f /var/log/ufw.log
And see this, while running a ping from the container to it’s default gateway (the ping is timing out)
Dec 25 16:50:27 DR3-4 kernel: [26437.122083] [UFW AUDIT] IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:15:5d:00:50:01:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=41561 PROTO=UDP SPT=68 DPT=67 LEN=308
Dec 25 16:50:34 DR3-4 kernel: [26444.380619] [UFW AUDIT] IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:15:5d:00:50:01:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=41562 PROTO=UDP SPT=68 DPT=67 LEN=308
Dec 25 16:50:50 DR3-4 kernel: [26459.873164] [UFW AUDIT] IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:15:5d:00:50:01:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=41563 PROTO=UDP SPT=68 DPT=67 LEN=308
Dec 25 16:51:22 DR3-4 kernel: [26491.858664] [UFW AUDIT] IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:15:5d:00:50:01:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=41564 PROTO=UDP SPT=68 DPT=67 LEN=308
Dec 25 16:51:25 DR3-4 kernel: [26494.983515] [UFW AUDIT] IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:15:5d:00:50:01:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=41565 PROTO=UDP SPT=68 DPT=67 LEN=308
Dec 25 16:51:32 DR3-4 kernel: [26502.232176] [UFW AUDIT] IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:15:5d:00:50:01:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=41566 PROTO=UDP SPT=68 DPT=67 LEN=308
I don’t understand what the log is showing me, but it is not blocked ICMP.
There appears to be an issue with the networking config of the container.
As the issue with accessing archive.ubuntu.com:80 is more than the fact that the container can’t access http. The container can’t ping it’s default gateway or ping the outside world generally.
However, the one thing that is working is incoming SSH to the container (which uses port 2222), which I setup ages ago and I can’t locate my notes for this. But incoming SSH is working.
The UFW firewall rules on the container host are
zen@DR3-4:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp ALLOW IN Anywhere
Anywhere on lxdbr1 ALLOW IN Anywhere
2222 ALLOW IN Anywhere
Anywhere (v6) on lxdbr1 ALLOW IN Anywhere (v6)
2222 (v6) ALLOW IN Anywhere (v6)
Anywhere ALLOW FWD Anywhere on lxdbr1
Anywhere (v6) ALLOW FWD Anywhere (v6) on lxdbr1
Thoughts?
Disable ufw. Does it work? If so you need to tweak those firewall rules.
That was a good idea.
Unfortunately, made no difference.
Do you have any VPN active in the host? Sometimes those VPN can block traffic too if not configured correctly.
No, I don’t.
I can’t find my notes when I first setup the bridge. I know I had some assistance from somewhere because the networking didn’t work out-of-the-box.
It surprises me that ping and traceroute don’t return results in the container even though SSH forwarding is working.
I’ve been looking on the net for some info on configuring an lxc bridge, so that I can try and work through my config without breaking the SSH.
Do you have any lxc articles that might assist?
Both the host, where networking is working as it should (i.e. I surf the Net, ping, traceroute etc)
And the container, where networking is largely not working, although somehow inbound SSH is.
In the /etc/ufw/before.rules on both the host and the container, the entries for allowing icmp on both input and forwarding are present.
I think as a starting point I need to resolve why the container can’t ping it’s default gateway, that just doesn’t make sense.
Any thoughts on what to try next in relation to troublshooting ping?
Did you set up a custom bridge? Typically lxd set up a bridge for you with a firewall rule. That is the most reliable way. With a custom bridge, you need to write the correct firewall rule. From the logs you posted, it seems that your host firewall UFW is blocking UDP 68/67 ports which are used by the DHCP client. So you have
LXD server with DHCPD <-> DHCP client running on your LXC instance.
My guess is you are not getting IP address hence you are not able to ping or trace. Use the ip command to see if you are getting any IP assigned
ip a s
The container has the IP address: 10.254.247.70/24 and that is “working” in the sense that I have a port forward on the site gateway that is forwarding SSH to the container, and this is working (i.e. I can SSH from the outside world to the container, that would not be possible if IP wasn’t working on the container).
That being said, outgoing networking from the container is totally blocked/cactus.
From the container, I can’t ping the bridge, or the container host. And of course if I can’t reach the container host, I can’t reach anything further (like an Internet host).
Here is the output
root@Ubuntu-DR6:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:7a:98:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.254.247.70/24 metric 100 brd 10.254.247.255 scope global dynamic eth0
valid_lft 3585sec preferred_lft 3585sec
inet6 fd42:ab5e:c791:7f94:216:3eff:fe7a:98e7/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3169sec preferred_lft 3169sec
inet6 fe80::216:3eff:fe7a:98e7/64 scope link
valid_lft forever preferred_lft forever
What is really strange, is that normally by default inbound access is not permitted, but outbound access is.
That is, normally, provided that you have your IP address and gateway configured correctly - you can surf the Net. But incoming access, like the incoming SSH doesn’t work.
In this case, the incoming SSH does work (thankfully, or I would have a bigger issue than what I have). But outbound access doesn’t work at all.
I tried changing the default gateway on the container, from the bridge IP address (10.254.247.10/24), to the IP address of the container host (10.254.247.85/24).
It made no difference, I still have all the same issues on the container in relation to trying to access external resources. And the incoming SSH still works.
Try removing all ssh forward rule from your /etc/ufw/before.rules. Save and close the file. Restart the ufw. Make sure those rules are gone including NAT rules you set in before.rules. And then add it as follows for ssh forwarding (adopt rule as per your set up):
lxc config device add ubuntu-nginx ssh-reditect proxy listen=tcp:1.2.3.4:2222 connect=tcp:10.83.200.253:22
Where,
I just want to confirm a couple of items to make sure that I understand your instructions correctly.
When I remove the ssh forward rules from /etc/ufw/before.rules on the container host; will that have the effect of deleting the previous lxc config device add rule that I have previously entered when I originally setup the ssh forwarding?
Or will I also need to enter some lxc config device remove command?
Did you add lxc config device rule for port forwarding (keep any other rules)? If so delete it. Also delete rules from /etc/ufw/before.rules. Restart
I finally found my earlier notes on the setup.
My memory of details, like whether it was possible to ping from the container in the past, is not great. However, it is clear from my notes, that in the past it was possible to ping from the container out. So, at some point, IP networking has partially broken.
Here are the commands that I used to assign the IP address to the container, do these look correct?
lxc stop Ubuntu-DR6
lxc network attach lxdbr1 Ubuntu eth0 eth0
lxc config device set Ubuntu-DR6 eth0 ipv4.address 10.254.247.70
lxc start Ubuntu-DR6
I’m wondering if this line:
lxc config device set Ubuntu-DR6 eth0 ipv4.address 10.254.247.70
Should have been this?
lxc config device set Ubuntu-DR6 eth0 ipv4.address 10.254.247.70/24
I can’t find the commands that I used to configure the bridge. I think the bridge must have been configured as part of
lxd init
And I assigned the following IP address to the bridge
10.254.247.10/24
I’m wondering, whether the networking configuration could be like the shared-disk access (where I initially used the root user, rather than a user other than root).
And I managed to deploy a bridge config that “worked” initially; but wasn’t best practice, and has subsequently broken.
I removed the ssh port rules from ufw
There were no ssh port rules in before.rules
I removed the original lxc config device ssh-redirect
I restarted the container host.
After the container host had restarted, I connected into the container to see if there had been any change in the networking behaviour of the container. Unfortunately, there was no change.
I then added the lxc config device add to reinstate the ssh re-direct.
I then tested external ssh access to the container, which unfortunately now times out. So, we may need to re-add the ssh rule to ufw on the container host (but I have not done that as yet).
Here is the contents of before.rules
#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
# ufw-before-input
# ufw-before-output
# ufw-before-forward
#
# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines
# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT
# quickly process packets for which we already have a connection
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT
# ok icmp code for FORWARD
-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT
# allow dhcp client to work
-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT
#
# ufw-not-local
#
-A ufw-before-input -j ufw-not-local
# if LOCAL, RETURN
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
# if MULTICAST, RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
# if BROADCAST, RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
# all other non-local packets are dropped
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
# allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT
# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT
# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT
I eventually got this working by trial-and-error.
My thinking was that somehow the networking had become corrupted. Don’t know how, but from my earlier notes, it was clear that the networking had worked (fully) in the past and I hadn’t made any changes to the networking.
So, I experimented with unsetting the IP address assigned to the container. Detaching the bridge. Changing the IP address assigned to the container etc.
Changing the IP address assigned to the container proved to be the answer, because after assigning a different IP address to the container. The default route configuration in the container was different to before.
Working config
root@Ubuntu-DR6:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway.lxd 0.0.0.0 UG 100 0 0 eth0
10.254.247.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
_gateway.lxd 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
Once I was able to get networking working, I performed the apt update/upgrade.
I figured that once networking was working, the underlying problem had been resolved and I could return the IP address of the container to it’s original IP address. But that changed the routing configuration (back to broken)
root@Ubuntu-DR6:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.254.247.10 0.0.0.0 UG 100 0 0 eth0
10.254.247.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
10.254.247.10 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
root@Ubuntu-DR6:~# ping 10.254.247.10
PING 10.254.247.10 (10.254.247.10) 56(84) bytes of data.
^C
--- 10.254.247.10 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3080ms
So, I changed the IP address back to 10.254.247.60, which updated (and restored) the routing.
Changed the ssh-redirect to 10.254.247.60
Long story, short, the answer was to change the IP address of the container. Which doesn’t make much sense to me as the previous IP address was not in use. Anyway, it is what it is.
I did need to re-allow 2222 through ufw on the container host.
This looks like a bug. Still the main thing is that it is now working.
Cheers
Thanks for your assistance. I find it really helpful to be able to chat to someone about these things, particularly when I’m not completely across the detail myself.
No problem. Just keep notes on your hard drive so that next time you can sort it out quickly