In Red Dead Redemption for PS3 and Xbox 360, you can increase the capacity of your Dead Eye skill by ranking up your ambient challenges.
To increase your Dead Eye capacity, get the following rank in each of these ambient challenges:
Master Hunter rank 10
Sharpshooter rank 5
Rank 10 in all ambient challenges
Each of these will increase Dead Eye individually, so complete all of them for a fairly large increase.