The GNOME keyring daemon may interfere with gpg-agent, preventing gpg to communicate properly with the smart card.
Getting ssh to work on GNOME systems
Some GNOME users may find that the card works on their system but there is a problem with getting ssh subkeys on the card to work.
This is a problem caused by gnome-keyring. This application is taking control of the ssh socket and preventing gpg-agent from working properly.
You can fix this by issuing the following command in a terminal:
command> gconftool-2 --set -t bool /apps/gnome-keyring/daemon-components/ssh false
This fix for this problem was originally discussed here: http://www.gossamer-threads.com/lists/gnupg/users/43682
Getting gpg-agent and ssh to work on Ubuntu 12.04
If gpg fails to access the card on Ubuntu 12.04 (maybe after one or more successful commands), disable GNOME keyring autostart files under directory /etc/xdg/autostart: Rename or delete the files gnome-keyring-ssh.desktop and gnome-keyring-gpg.desktop. Then log off and on again.