If there is nothing in Banishment to swap back to then this object can never swap back.
§ 438.7.c Core Rules PDF (officieel)
§ 438 Replace
§ 438.7 Tokens that have been Created through a Replace action can be instructed to be “Swapped back.” This may also appear as “replace [the token] with the [Game Object] it replaced.”