Conntrack tools allows manipulating linux connection tracking table. Command conntrack is useful for deleting conntrack entries, but it is not included by default in RHEL/Centos.
After while of googling, I have found required library and conntrack tools in centos.alt.ru repository and installed them by running:
rpm -Uvh http://centos.alt.ru/repository/centos/5/x86_64/libnfnetlink-1.0.0-1.el5.x86_64.rpm
rpm -Uvh http://centos.alt.ru/repository/centos/5/x86_64/libnetfilter_conntrack-0.9.1-1.el5.x86_64.rpm
rpm -Uvh http://centos.alt.ru/repository/centos/5/x86_64/conntrack-tools-1.0.0-2.el5.x86_64.rpm