You have to be ruthless, KGuy. The ugly ones (called Tobys) will never change color, so you'd best just cull them out.
What I do to help justify getting rid of a fish, is remind myself that koi are "dirty" fish, and each one puts a tremendous burdon on my pond's filter and oxygen level. Therefore, each fish must really be worth his weight in beauty, to be worth keeping.