Probably less weird than being in a certain religion that a very large number of Americans belong to, because they have an imaginary friend and they don't think it's weird at all. But that's no reason to think it's safe to tell them about yours. They likely see their imaginary friend as an exception to the usual rule. To my mind, you have the advantage that you know your imaginary friend is imaginary, but to their minds, that might not help much.
But weirdness tends to be in the eye of the weirded-out, i.e. it's all rather subjective. I wouldn't start talking to an imaginary friend in the presence of others, unless they were extremely non-judgemental people. Nor would I mention it to others unless they were fairly non-judgemental. Unfortunately a lot of people are judgemental.
Objectively, weird means "strange or extraordinary." I have no data but I strongly suspect it's a common thing even for secular people to have imaginary friends here and there to prop them up psychologically. It's just that most people don't like admitting it, so it looks more extraordinary than it is.
I often have dialogues with myself when I don't think anybody can hear me. That probably counts as a kind of imaginary friend, though I know it's all me really. I might make an assertion for the "other me" to see if "he" can knock any holes in it. So it feels useful for keeping my thinking rigorous, and it gives me a sense that I'm not alone, which is nice because I often feel lonely when I'm on my own.
I don't think of lore about "him" though, because I'm not that much of a lore person, though I enjoy such stuff as Tolkien and ancient myths well enough. I don't suppose it does you any harm as long as you don't permanently forget that you've made it all up. Fantasy can be good fun if real life is boring.