Couldn't you only remove the tile if it has been completely or almost destroyed? Then it wouldn't disappear magically.
While damaging a tile at the top, you could also transfer a part of the damage to tiles below/around it. If you create a hole in the tile and continue to hammer, the hole will be expanded to the tile behind it. Like this, the next tile is already distorted when the first one is removed, so the removal doesn't lead to a squarish look.
Depending on how realistic you want to do it, you have to choose a trade-off between authentic physics and a simple, fast implementation.