Personally, I've decided that only s1 and 2 of Torchwood exist in the main Doctor Who universe and that the rest is some sort of alternate universe where the Doctor doesn't exist and everyone is stupid and/or evil. :)
Then it doesn't hurt my memories of the show. I really should rewatch the first two seasons - I haven't seen them since they first aired here - but recently got the dvds.