I would agree with Michael, if you only fill in the holes, it will stand out like a sore thumb.
Have you looked at what is available from new? You should be able to get a small piece from a vendor to check for a match of color. Note - there is one supplier of carpet for the US and the various vendors get it from them, Veterans I believe. I know there is some consolidation going on, so I don't know if anyone else has tried to get into the US market.
It sort of depends on how much the speakers bother you. Is it worth trying to find/wait/hunt for a piece of unicorn carpet? Only a question you can answer.
Most likely making a contact with a member (or them posting on this thread) of the group in Germany might be your best bet to see if they know of any source for the old carpet.