Why would it be inappropriate, are either of you married, for example? If not, I'm confused about why it would so wrong. Obviously, you should definitely not have sex *at* work (trust me, that's a terrible, terrible idea!) but lots of people have relationships with work colleagues, whilst still remaining professional at work. No-one except the two of you need to know that you are involved.