Each time you collect all the letter tiles for a level e.g. Goo Lagoon, you'll be sent to that level's tile puzzle. After you solve that, you can collect the Dutchman's Treasure for that level and you can then move to a new level/area.
I'm pretty sure the only thing you need the coins for is to collect the "Collect X Doubloons" letter tile for a given level.
As for the jellyfish, not only will you need to collect a certain amount in each level to get that level's "Catch X Jellyfish" letter tile, but there will be times when characters will ask you to have caught a certain number of jellyfish before you can progress (like Larry does before you can get the bus ticket to Goo Lagoon.)
Hopefully this helps, I was never really good at explanations :P
