I Saved All My Jewels For Hermits to get this one, but thats not totally necessary.
- Side Note, This is do-able in the "Shogun" Gamemode too.
--------------
Each Island has A Hermit Hut.
Each One Lets You upgrade a building in a special way.
Spend Jewels To Unlock Them and Then Coins to buy Each.
Afterwards, when you send your boat to the next island, the ones you've bought will board the boat with you.
Do This On 4 Islands And When You Leave, The Achievement Pops For Both You and Your Co-op/Multiplayer Friend.
Usually Your Hermits Will Stroll on over to the Fire at your base, but Occasionally...
---------
If The Greed On An Island Steals Your Hermit
-You Can Blow Up The Cave Portal (See Defeating The Greed On X Islands Walkthroughs), and Your Hermit Will Be Freed!
Alternatively, There Are 5 Different Hermits - So You Could Lose One Early and still clear it without Hitting the cave portals.
GL!