Well if you are in Pilot I highly recommend upgrading to the latest firmware versions. (currently 5.5.1). There are many new features and fixes. Also, if ever you approach Motorola support for assistance, their first step is always for you to upgrade to the latest firmware before they are willing to assist you.
Regarding 5.3.1 I had issues with the firewall, which was fixed when I upgraded. Also, the development stream was split after 5.1 - so v5.3.x focused on one feature set while v 5.2.x focused on other features. If you are running 5.3.x and you want a feature from 5.2.x then too bad (I think LT2pv3 is one of those!)
Also it caused a lot of confusion with the numbering because for example v5.2.13 is a much newer firmware than v5.3.1. The good news is that they joined the 2 streams together again from 5.4 onwards.
I am running the latest 5.5.1 at a large site and it is working well and I am making use of the L2TPv3 tunnels in this version.
Looking back at your original question I will see if I can answer more:
IP Sec cannot extend VLANs. IP Sec is Layer 3.
Guest data is encrypted by operating encrypted sessions between the client and the server, independently of the network.
Regarding your other question I can highly recommend the Captive Portal How-To Guide located here:
http://www.michaelfmcnamara.com/files/motorola/WiNG5_Captive_Portal_Design_Guide_June_2011.pdf