If you've ever needed to use a certain network interface on a linux (specifically RHEL6 here) you'll notice there are several different ways to do it. A search of the interwebs will give you numerous avenues but I'm going to outline the one here which worked for me. We'll use ifcfg-em1 and ifcfg-em2 for these examples and assume you have console access.
Symptoms: one of your nics has died and the software on the server is dependent on the interface name. i.e. Oracle RAC
First you will need to reboot afterwards, there was no way around it for myself. Give yourself that luxury instead of thinking you can just do a <code> service network restart </code> to finish it. Start by creating backups of ifcfg-em1 and ifcfg-em2. Rename ifcfg-em1 to ifcfg-em3 and then ifcfg-em2 to ifcfg-em1.
Next create a file if it does not exist already of "/etc/udev/rules.d/70-persistent-net.rules". If it does not exist you can attempt to create it using the automated tool of "/lib/udev/write_net_rules all_interfaces". This method did not work for me and I had to create the file manually which did work. *Note you are swapping the MAC addresses here and putting what you WANT em1 and em2 to be.
</etc/udev/rules.d/70-persistent-net.rules>
# PCI device 0x14e4:0x163a (bnx2) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:55", ATTR{type}=="1", KERNEL=="em*", NAME="em1"
# PCI device 0x14e4:0x163a (bnx2) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:44", ATTR{type}=="1", KERNEL=="em*", NAME="em2"
</etc/udev/rules.d/70-persistent-net.rules>
Once you've confirmed those two things are done reboot the server and it'll come back up proper. Took a lot of searching to get all this set up correctly. Hope this helps someone else.
