ClearFoundation Tracker - ClearOS
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0001108ClearOSapp-firewall - Firewallpublic2013-04-20 19:492019-04-24 09:38
Reporterdloper 
Assigned To 
PrioritylowSeverityminorReproducibilityalways
StatusclosedResolutionsuspended 
PlatformOSOS Version
Product Version5.2-SP1 
Target VersionFixed in Version 
Summary0001108: 1:1 NAT breaks ability for OpenVPN clients to be accessed by internal NAT target server
DescriptionHosts behind ClearOS are able to connect to clients within the OpenVPN network so long as those hosts are not part of a 1:1 nat rule. The offending rule is:

SNAT all -- 192.168.1.168 anywhere to:10.10.10.11

Where 10.10.10.11 is the 1:1 external address and 192.168.1.168 is the internal host. This line should NOT say 'anywhere' because this forces all 1:1 Hosts involved to NAT the traffic even for normal routing between networks. Networks that should be excluded from this 'anywhere' statement should be:

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
...and any network configured as a DMZ

Here is capture of ping packets from a working ping between the host and the OpenVPN client before the activation of the 1:1 rule:


[root@gen3 ~]# tcpdump -i tun0 icmp[icmptype]=icmp-echo
tcpdump: WARNING: arptype 65534 not supported by libpcap - falling back to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
21:11:38.703316 IP 192.168.1.168 > 10.8.0.6: ICMP echo request, id 3, seq 15099, length 40
21:11:39.701630 IP 192.168.1.168 > 10.8.0.6: ICMP echo request, id 3, seq 15100, length 40
21:11:40.699847 IP 192.168.1.168 > 10.8.0.6: ICMP echo request, id 3, seq 15101, length 40

Here is the same test after the activation of the rule:

[root@gen3 ~]# tcpdump -i tun0 icmp[icmptype]=icmp-echo
tcpdump: WARNING: arptype 65534 not supported by libpcap - falling back to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
21:15:08.467490 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 15234, length 40
21:15:13.105021 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 15235, length 40
21:15:17.742742 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 15236, length 40
21:15:22.380068 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 15237, length 40
Steps To ReproduceWhat makes this difficult to test is that the socket caching of the NAT translation can cause a misreading on the activation of the NAT rule. This can make it so that activating the NAT rule while there is active or TTL traffic between the HOST and the OpenVPN client will persist even after an introduction of the rule in the table. A reboot between states is often required.

Another strategy is to stop the active ping traffic, make a sandwich or go to the bathroom and try the state again. The TTL should have expired.

Steps:
1- Setup and configure ClearOS 5.x as a gateway.
2- Have a server or host behind the gateway with services
3- Setup OpenVPN
4- Connect to server using OpenVPN from the outside and validate that you can ping the OpenVPN client
5- Stop ping
6- Create a 1:1 nat rule involving the host server's services.
7- Attempt to ping the OpenVPN client (If it pings, stop ALL pings and connections; may need a reboot or a sandwich before this fails)
Additional InformationWORKAROUND

If this is your 'iptable -t nat -L' entry for the POSTROUTING chain:

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- 192.168.1.168 anywhere to:10.10.10.11
SNAT tcp -- 192.168.1.0/24 192.168.1.168 tcp dpts:ftp-data:ftp to:192.168.1.1
SNAT tcp -- 172.22.22.16/28 192.168.1.168 tcp dpts:ftp-data:ftp to:172.22.22.22
MASQUERADE all -- anywhere anywhere
MASQUERADE all -- anywhere anywhere

Pay special attention to this line:
SNAT all -- 192.168.1.168 anywhere to:10.10.10.11

...and make the following edit to /etc/rc.d/rc.firewall.local:

# Custom firewall rules.
# This file is executed by the firewall on stop/start/restart.
iptables -t nat -D POSTROUTING -s 192.168.1.168 -j SNAT --to-source 10.10.10.11
iptables -t nat -I POSTROUTING -s 192.168.1.168 -d ! 10.0.0.0/8 -j SNAT --to-source 10.10.10.11
TagsNo tags attached.
Relationships
related to 0001113closed  1:1 NAT and DMZ incompatibilities 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2013-04-20 19:49dloperNew Issue
2013-04-20 20:52dloperNote Added: 0000801
2013-04-20 21:02dloperNote Added: 0000802
2013-04-22 11:11user2ProjectClearCenter => ClearOS
2013-04-22 11:11user2Categoryclearos-release-professional => General
2013-04-22 11:12user2Statusnew => confirmed
2013-04-22 11:12user2CategoryGeneral => app-firewall - Firewall
2013-04-22 12:10user2Note Added: 0000807
2013-04-22 12:12user2Note Edited: 0000807bug_revision_view_page.php?bugnote_id=807#r63
2013-04-22 15:29dloperNote Added: 0000809
2013-04-22 15:35dloperNote Added: 0000810
2013-04-22 17:52user2Issue cloned: 0001113
2013-04-22 17:52user2Relationship addedrelated to 0001113
2013-04-22 17:54user2Note Added: 0000811
2013-06-13 19:01dloperNote Added: 0000891
2019-04-24 09:38NickHNote Added: 0011661
2019-04-24 09:38NickHStatusconfirmed => closed
2019-04-24 09:38NickHResolutionopen => suspended

Notes
(0000801)
dloper   
2013-04-20 20:52   
I can confirm that this happens on ClearOS 6.x as well. Again the offending line is:

SNAT all -- Admin-PC.clearos.com anywhere to:10.10.10.11
NOTE: Admin-PC.clearos.com is 192.168.1.168 the same server as the 5.x test.

Here is a transaction log that demonstrate it:

### Validation of working state ###
[root@gen3 ~]# tcpdump -i tun0 icmp[icmptype]=icmp-echo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
22:47:28.230850 IP Admin-PC.clearos.com > 10.8.0.6: ICMP echo request, id 3, seq 19123, length 40
22:47:29.228753 IP Admin-PC.clearos.com > 10.8.0.6: ICMP echo request, id 3, seq 19124, length 40
22:47:30.226663 IP Admin-PC.clearos.com > 10.8.0.6: ICMP echo request, id 3, seq 19125, length 40
22:47:31.224579 IP Admin-PC.clearos.com > 10.8.0.6: ICMP echo request, id 3, seq 19126, length 40
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

### Demonstrating internal IP to name resolution ###
[root@gen3 ~]# ping admin-pc.clearos.com
PING admin-pc.clearos.com (192.168.1.168) 56(84) bytes of data.
64 bytes from Admin-PC.clearos.com (192.168.1.168): icmp_seq=1 ttl=128 time=0.212 ms
^C
--- admin-pc.clearos.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 820ms
rtt min/avg/max/mdev = 0.212/0.212/0.212/0.000 ms

### Revised NAT table after 1:1 nat rule creation ###
[root@gen3 ~]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- anywhere 10.10.10.11 tcp dpts:ftp-data:ftp to:192.168.1.168

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- Admin-PC.clearos.com anywhere to:10.10.10.11
SNAT tcp -- 192.168.1.0/24 Admin-PC.clearos.com tcp dpts:ftp-data:ftp to:192.168.1.1
SNAT tcp -- 172.22.22.16/28 Admin-PC.clearos.com tcp dpts:ftp-data:ftp to:172.22.22.22
MASQUERADE all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

### Demonstration of NAT on VPN destined traffic ###
[root@gen3 ~]# tcpdump -i tun0 icmp[icmptype]=icmp-echo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
22:49:02.639454 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 19168, length 40
22:49:07.319848 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 19169, length 40
22:49:11.950739 IP 10.10.10.11 > 10.8.0.6: ICMP echo request, id 3, seq 19170, length 40
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
(0000802)
dloper   
2013-04-20 21:02   
WORKAROUND 6.x:
The workaround is the same for 6.x. You can also use the custom firewall tool to add the following lines:

/etc/clearos/firewall.d/custom

iptables -t nat -D POSTROUTING -s 192.168.1.168 -j SNAT --to-source 10.10.10.11 # Destroy Existing SNAT rule
iptables -t nat -I POSTROUTING -s 192.168.1.168 -d ! 10.0.0.0/8 -j SNAT --to-source 10.10.10.11 # Create replacement SNAT rule
(0000807)
user2   
2013-04-22 12:10   
(edited on: 2013-04-22 12:12)
When IPsec is enabled on the firewall, two SNAT rules are added to the POSTROUTING table. Following the example above, I'll be adding the "-v" flag to show network interface information. The output from "iptables -L POSTROUTING -t nat -v" for a 1.2.3.4 -> 192.168.4.10 1-to-1 NAT rule looks like:

 target prot opt in out source destination
 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 policy match dir out pol ipsec
 ACCEPT all -- * tun+ 0.0.0.0/0 0.0.0.0/0
 SNAT all -- * * 192.168.4.10 0.0.0.0/0 to:1.2.3.4
 SNAT all -- * * 192.168.4.0/24 192.168.4.10 to:192.168.4.1
 MASQUERADE all -- * eth0 0.0.0.0/0 0.0.0.0/0

The first rule is for IPsec, while the second rule is for all tunnel interfaces (OpenVPN). Any packet destined out a VPN will bypass the SNAT and MASQUERADE rules.

The fix is to make sure the tun/OpenVPN rule is enabled all the time (not just when IPsec is enabled).

Workaround 2: enable the IPsec protocol on the "Incoming Firewall" page, even for OpenVPN servers.
(0000809)
dloper   
2013-04-22 15:29   
This problem does affect DMZ but to a more minor extent:

192.168.1.168: LAN Server that has 10.10.10.11 1:1 NAT
192.168.1.1: ClearOS server LAN address
10.10.10.10: ClearOS server WAN address
10.10.10.11: ClearOS 1:1 nat rule alias assigned to 192.168.1.168
10.10.13.1 ClearOS server DMZ address
10.10.13.25 DMZ server with ClearOS as gateway

Pings from DMZ to any host on the LAN is free from SNAT issues. This is the default condition. But when you create a 1:1 nat rule, it breaks this state:

This demonstrates a ping from a LAN host (that is part of a 1:1 NAT definition) to a DMZ host.
17:03:27.488199 IP 10.10.10.11 > 10.10.13.25: ICMP echo request, id 3, seq 21219, length 40

As you can see the LAN host's address is now replaced with the SNAT address.

This demonstrates a ping from a DMZ host to a LAN host (that is part of a 1:1 NAT definition)
17:03:36.498403 IP 10.10.13.25 > 192.168.1.168: ICMP echo request, id 63789, seq 0, length 64

This is the normal path that exists between hosts where 1:1 NAT is wholly absent as well...both ways.

As you can see, the first ping takes on the NAT attribute which makes it impossible for the DMZ host to use any source based firewall rules with any confidence.

If you remove the one to one NAT rule, the first ping will look like this:
17:11:55.161892 IP 192.168.1.168 > 10.10.13.25: ICMP echo request, id 3, seq 21235, length 40

To repair this condition as well in hosts that are in a DMZ, the API for 1:1 nat rules should include all subnets defined as a DMZ to include the following template provision:

iptables -t nat -I POSTROUTING -o $DMZIF -j ACCEPT
(0000810)
dloper   
2013-04-22 15:35   
These rules are unnecessary when 1:1 nat is NOT in play but they are required when it is. Another thing that will happen is that any hosts across any routes defined in /etc/sysconfig/network-scripts/route-* should be read by the 1:1 nat module and exempted witht he following:

iptables -t nat -I POSTROUTING -d $routecidr -j ACCEPT

Without this rule, all hosts will work with routes except those defined as 1:1 NAT rules. This is because the SNAT rule will broadly apply to defined hosts.
(0000811)
user2   
2013-04-22 17:54   
This bug was split into to tasks:

- 0001113 is for the DMZ issue noted
- 0001108 (this bug) is for the OpenVPN issue

The OpenVPN fix is trivial and can be pushed out immediately. The DMZ fix is likely in a different section of firewall code.
(0000891)
dloper   
2013-06-13 19:01   
---Additional. This seems to affect HotLAN hosts that have custom pin-holes to the LAN network. Rather than fixing the issue with each separate technology, it seems better to address the single SNAT policy which is the source of all the trouble.
(0011661)
NickH   
2019-04-24 09:38   
Migrated to https://gitlab.com/clearos/clearfoundation/app-firewall/issues/12 [^]