Let's add high sea waves. For this tutorial, we will use a single voronoi fractal on the High Sea Material.

The surface ocean looks strange: bumps with flat areas. This is due to the default "Maximum" blend function between the two water materials. Change this to "Blend" and replace the "MonoFractal" with "BreakWaves Blend".

Now the ocean has uniform waves on its surface. But we want that the high sea waves are damped near the shore. We will do it by changing the "Ocean Depth" parameter to a function. In the function editor, add an "Ocean Depth" node (in category Utilities) and connect its output to the brown node. Drive its input with a "World Position" node. Finally, change again the "Ocean Depth" parameter, this time to texture.

The waves cover all the ocean, but near the shore. That's what we wanted.

Well, maybe we want a larger quiet area. The "Depth Factor" parameter has been designed for that purpose. As said before, everything is driven by the water depth. The "Depth Factor" let specify how many meters the water should be depth for the blend to be 100% high sea. So the result depends heavely of your beaches slope. On my world, a "Depth Factor" of 10.0 gives the following result:

Some may argue that the water should not be absolutely quiet at the beach limit. This is true and may be trimmed with the "Depth profile" blend curve. But the high sea waves have very unrealistic effects on the transition between the beach and the water. This is the main motivation for this damping feature.
OK! Now, we are ready to create the breaking waves on the Coastal Material leaf.