In a simple call setup scenario INVITE / 100 (Trying) / 180 (Ringing) / 200 (OK) / ACK, t_check_trans() returns false when processing INVITE, and it returns true when processing ACK (it must return false since ACK does not belong to INVITE transaction). Moreover, when t_check_trans() is called second time, it always returns false when processing INVITE, but it now returns false also in case of ACK [E.Gabrielyan 2007-04-13].
Check the following web site for the openser.cfg file and the printout of the proxy in the debug mode http://switzernet.com/people/emin-gabrielyan/070413-openser-transactions/#t_check_trans