Hi,
1. For the Upgrade, you don't need to shutdown your vm's. maybes later for upgrading tools or hardware version.
2. As far your hardware is on the hcl and you have a valid support contract. There should be no extra cost. Only your workng time is the cost.
3. First, you have to convert your license to a valid esxi 5 license. That could be done on the myvmware we page licensing portal.
For the rest, there are serval websites and documents for upgrading. Here is one of them.
Frank