I am having a hard time understanding the villagers when you are trying to trade with them. Sometimes when I want the fletcher to restock its sticks, he just wanders off to talk to other villagers. I heard it depends on the time of day when you are doing this. But is there any way to counteract with this issue?

it's just random and you have to hope it works

if it's like the first 2 minutes and last 2 minutes of the day then they won't restock or get/lose jobs yeah

I have found the best way to counteract this is to have multiple Fletchers. Pre-1.16 I could just use one Fletching Table to create multiple Fletchers. But it seems the way to create multiple Fletchers 1.16+ is to have multiple Fletching Tables, because it seems more difficult to create additional Fletchers with just one work station.

Also with extreme luck in 1.16 - 1.16.1 (Java Edition) you could potentially collect all the Ender Pearls you need in the Nether. First luck required would be to barter (requires gold bars) with a Piglin and get 8 Ender Pearls, then barter with a Piglin and get another 8 Ender Pearls. In 1.16.1 the amount of Ender Pearls the Piglin will give at approx 4.73% chance and 4 - 8 Ender Pearls, in 1.16.2 it is being reduced to approx 2.18% chance and 2 - 4 Ender Pearls. (Official Minecraft Wiki)

I have been able to collect Ender Pearls in the Warped Forest (1.16+), but not in large enough numbers to eliminate trading with Villagers.