Yours is a non-utilitarian view of ownership. What to you was stolen, to them was unused.The non-use of a good can be just as detrimental to the commons as over-use. Ownership regulates this imbalance by favoring and rewarding use for the common good.

Imagine you have a family of five, but only enough land to feed four. Beside you is land that doesn't grow anything at all. Some nomadic tribe claims it but doesn't farm it. Do you let kid 3 starve, or do you use the vacant land?

It's no different today with derelict structures. If an owner can't or refuses to maintain it, making it a safety hazard, the city reclaims the building to demolish it and sell to a developer.