The vent does not have to be removed, although it is easier if you've already got them removed. But removing the vent is a royal PITA. You should be able to get a small screwdriver (I used a large jeweler's screwdriver) in where you see the plastic stick into the metal. Again, only going by recollection here, but there is a spring loaded catch that needs to be pried upward while the lever is pulled out. In the picture provided by 66andBlue, I believe the bright metal part is the catch. Seems like mine was either black or rusty, because I don't recall any bright metal in there.
If the levers don't work with ease, you'll probably want to lube the pivots with a light oil. I used a dry chain lube for bicycles.
