Mercenary mode in Resident Evil 5 is a fun mode where you’re trying to survive waves of mobs, and get as many points as you can before time runs out.
To unlock mercenary mode, all you have to do is beat all levels in story mode. This can be done by playing through in either of the co-op modes as well.
Once you unlock it, you’ll see that it has replaced the ?????? option at the bottom on the main screen when you’re starting your game.