Shy or defiant/rude?

OK, so this is a battle that I have fought since Andrew was really little.  He has always been super sensitive and shy, even with family.  We struggle to get him to speak to anyone or use manners (outside the house).  I've always just said he is super shy and we work on it.  I talk to him about it all the time but this week his teacher brought it to my attention.  She is concerned that he is being defiant to me, not shy.  She is also concerned about him being rude.  Last weekend, Andrew had a football game and his teacher came to watch and he refused to say hi to her or thank her for coming.  It's always been such a struggle and I'm not sure how to deal with it or what kind of consequences to use for this behavior.  Both of my kids are extremely backward around people and take a while to warm up.  It took Grace about 3 months (or more) to talk to her preschool teacher.  David has always been very shy and quiet (but not rude if people talk to him).  He doesn't typically start a conversation with people and I don't really think that's a big deal.  He isn't a talker.  I'm just trying to figure out if Andrew is being defiant and we are allowing him to get away with it or if he is truly that shy.  It's such a struggle because David always tells me to back off.  I can't force the words out of his mouth but he needs to understand that it is rude to not thank someone or acknowledge them.  Any ideas?  I'm very frustrated about this.  It has always bothered me but I don't know how to move forward to work on it.