I'm not familiar with Japanese imports Graham so i had wondered if they had use an Aisin-Warner transmission instead of the ZF usually fitted. Experience with an AW in a Volvo 760 taught me the overdrive switch could sometimes play up bringing up a warning light indicating overdrive was off.
From what you say, i'm guessing you have the more usual ZF box, some of the later 5-speed auto ZF boxes had their ECU mounted inside the gearbox, the seal could fail and this rendered the gearbox dead. This was more common on late 90s V8 models (X300 usually) but as you have a Sport mode, there is some electronic control. On the 4-speed if memory serves, the ECU is remote from the gearbox but i'm not sure where.
If you can locate the gearbox ECU, follow the wiring from it and there should be an earth point somewhere nearby, i'd remove the connector and clean the connector and earth point - everything you've said so far points to a bad connection of some sort, probably an earth.