These boxes have solenoids in them, one of which controls the 4th gear (lockup). For starters it wont go into lock up without the box reaching the correct temp, if your just giving it a blast round the block it might not go in so bare that in mind but one of the solenoids might be full of gunk etc this is worth checking, Secondly does it have over drive? if so and the box has a fault the overdrive light will flash on your dash multiple times giving you a fault code.
I have had all these symptoms and more, my box had overheated and wouldn't engage in drive or reverse sometimes, I gave up and put a manual in mine as the price of rebuild on the box was out of question, by the sounds of it i don't think you have anything serious on your hands, if you think its warm enough and should change then have a look at the solenoids.