In UFC Undisputed 3 for the PS3 and Xbox 360, submissions are quite a bit different than in previous games.
If your opponents starts a submission on you, a mini game pops up on the screen in the shape of an octagon. Your opponent will have a color, and you will have a color that you circle around the octagon. If your opponent manages to cover up your color for the majority mini game and fills the green bar, then they will force you to tap out.
To escape the submission and keep from tapping out, simply keep your color away from theirs as long as possible.
If you do really well at this, then your character should perform a reversal.