The difference is going to be drainage. Potting soils are formulated for good drainage, the white stuff,
Perlite helps keep space open in the soil for air and for drainage.
Using garden soils in pots isn't recommended because they compact and don't drain well.
Putting garden soils in a pot changes their structure, even if they drain well in the garden.