ClusterXL - Do not Consume Public IPs for ClusterXL

Configuring Cluster Addresses on Different Subnets

Only one routable IP address is required in a ClusterXL cluster, for the virtual cluster interface that faces the Internet. All cluster member physical IP addresses can be non-routable.Configuring different subnets for the cluster IP addresses and the member addresses is useful in order to:

- Enable a multi-machine cluster to replace a single-machine gateway in a pre-configured network, without the need to allocate new addresses to the cluster members.
- Allow organizations to use only one routable address for the ClusterXL Gateway Cluster. This saves routable addresses.

ClusterXL virtual IPs and your members physical (or VLAN) interfaces do not need to be on the same subnet. So you can simply use whichever addresses you like for each of the cluster interfaces (apart from internal/management and external/VPN-routable interfaces obviously). And of course this applies to physical untagged interfaces unlike our case too.
I settled for using  tiny Class B private space /30 subnets for each VLAN, enough for just our 2 cluster members like this. The topology would then look like this.

Beware of the spoofing and routing

Now here’s just 2 catches with this configuration. First off, anti-spoofing will apply to the members local interface network and not the ClusterXL virtual one, so you can’t use the comfortable “Network defined by the interface IP and Net Mask” setting unless you want all your traffic dropped/detected as spoofed. Instead just define a specific subnet object representing the ClusterXL interface subnet.

The second thing which shortly caused some headache for me was that SPLAT/Gaia wouldn’t know where it needs to route the public subnet. Now that the physical interfaces to those subnets had different IPs, the OS naturally lacked the proper routing information and would forward traffic through the default route.
To solve this, I added static interface-based routes for each public subnet like this. To my confusion however, they didn’t help and seemed to have no effect.  Checking the firewall nodes routing table via SSH confirmed that there was no corresponding entry present.

Instead I had to issue the following in expert mode on the nodes to activate my routes:
# route add -net 47.88.145.40/29 eth8.356
The routing table would now look like this:
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.31.255.16   0.0.0.0         255.255.255.252 U     0      0        0 eth8.356
47.88.145.40    0.0.0.0         255.255.255.248 U     0      0        0 eth8.356

On the new Gaia CLI it looks like this:
> show route
Codes: C – Connected, S – Static, R – RIP, B – BGP,
       O – OSPF IntraArea (IA – InterArea, E – External, N – NSSA)
       A – Aggregate, K – Kernel Remnant, H – Hidden, P – Suppressed
C     47.88.145.40/29     is directly connected, eth8.356
C     172.31.255.16/30    is directly connected, eth8.356
The route command in expert mode alone doesn’t survive a reboot, you still need to set all routes in the Gaia/SPLAT CLI/Webinterface on all members. I confirmed that the routes are being applied properly after a reboot. Also, static routes using Gateway IPs do not need a reboot either, so this seems like a bug specific to using interface-based routes.

http://alpacapowered.wordpress.com/2012/08/23/running-check-point-clusterxl-member-interfaces-on-different-subnets-than-the-virtual-interfaces/