How about cases where "quite" is paired with an adjective with negative connotations, such as "quite annoying?"
Either 'not really', or, "really really really". See also (the other way around) "quite brilliant" meaning really really really brilliant, rather than a little over mediocre.
It all depends on tone, a really positive one, or a really negative one, would show that quite is not to mean 'a little', otherwise it doesn't really. Unless you are dealing with someone deadpan, which is something I don't tend to associate with En-US speakers.