In UFC Undisputed 3 for the PS3 and Xbox 360, forcing someone to submit (aka tap out) is quite a bit different than in previous games.
To start a submission, you first need to be on the ground (obviously) in the right position. You use the directions on the right analog stick to move positions around your opponent.
Once you have your opponent in the position you want, you click down on the right stick to start the submission.
Once you do this, it starts the submission mini game. Basically, it’s an octagon shape and you chase your opponent around the octagon and try to cover up his color as long as possible until you fill the green bar. If you can fill the green bar when time runs out, your opponent taps out, if not then they escape or reverse you depending on how well you did.