It's pretty easy to tell a Brit from an American from an Australian by the way he or she speaks English. However, it's not so easy in other languages. When I hear accented Spanish, I can tell when it's that of a native English speaker, but I'd be hard-pressed to identify what kind of English. I once took an intermediate Spanish conversation course in which I was surprised to discover, weeks after the class had started, that one of my classmates was from England; I found this out only because I overheard her speaking English before class. And I fondly remember an Australian classmate I had in a Spanish immersion class I took in Mexico, whose "down-under" English was so heavily accented that I actually found it easier to converse with him in Spanish.
Similarly, I can't tell whether a person is Chinese or Taiwanese based on his English, but it becomes very obvious if he speaks Mandarin. And while Swiss German and "Germany German" (High German) are vastly different, I can't tell if I'm talking to a Swiss or a German if he is speaking English.
This makes perfect sense if one considers that a foreign accent is characterized by sounds that differ between two languages. The un-trilled rr in Spanish, for example, is a dead giveaway for an English speaker, because we don't have anything resembling a trill in English. And while a Swiss and a German might disagree on how to pronounce ch, they will both likely pronounce the English th as s, since neither dialect of German contains that English sound. The sound differences between dialects are what scientists might characterize as "second-order", whereas the differences between actual languages would be "first-order."
For the non-mathematically-trained readers, I offer a different analogy. It's like comparing two slightly different shades of red paint chips. If you look at either next to a blue chip, it just looks red. Only when you compare them with each other are you likely to notice that one is darker or has more orange or more purple. Perhaps a highly trained artist would notice the difference right off, or a highly trained linguist in the case of the accents, but I am neither.