If you stay on the inside, you'll be safe.
If you go around to the outside, you need to check the weather (NOAA:"http://www.nws.noaa.gov/om/marine/forecast.htm") on the water. Pay attention to wind and swell size.
If you go outside, you really, really need to pay attention to the rocks...especially if the wind is blowing tward the rocks. It sounds very basic but you really, really need to pay attention if you're working the rock wall, or anywhere near the rock wall. It's very easy to get caught by a rogue wave, and the wind and end up on the rocks...someone does it every year.
Basically, if you're asking if it's safe, then you shouldn't be on the outside...stay inside and make sure you know where your PFD's are. On the inside, kyaks are safe...I know a few crazy people that have paddled their float tubes out to the 2nd closest island (the closest island is in casting distance).
Anytime you're on the water, you really need to have a marine radio on you, even if it's a handheld.
"Helfin sie mir!"