Picking up any items can cause the invisible wall to pop up (particularly eggs), so I always tried to do the glitch first thing in the morning, or at least before my chickens. That and the festivals were the only things that affected the wall that I found.