That's what I do. Although I frequently mix a little sand in, or top the pot with sand instead of gravel. The sand keeps the clay down. I have clay out in the yard, but don't trust it in the pond. I know the previous owners used lots of chemicals.
No matter what you decide, that Osmocote rocks!