If the home cannot be relocated without destroying it, then it has become an accession to that land, and I would think is legally attached to that mortgage. I'm sure if you read all the fine print on your mortgage, it will say that much.
Granted, I'm not a mortgage or real property expert. My comparison is a car loan ... if someone puts $10K rims on their car, stops making payments, and the car gets repo'd ... since those rims have become a permanant modification of the car, you can't have them back unless you redeem the repo.