It turns out that the fuel cutoff solenoid switch on the firewall was stuck closed. I lubed it and now it works as it should.
On a side note, from what I understand, 3rd gear should have also had the problem but I guess there is another switch on the transmission that is probably not working correctly.