Don't press E until you're inside the boat or it will take off without you!
I panicked and didn't see a ladder after dropping out of the pipes, I frantically pressed spacebar around whatever surface I could find to get up and into the boat. I don't think there's enough time to swim! I also don't know if you have to physically be on the island or if you can still be in the boat when it docks, I jumped out onto land just in case.