More importantly, although "O" is a function, one doesn't usually think about it that way. If one were, I think it's important to be precise. Here's the best crack I can take at it, though.

Let ℜ be the set of all real numbers. Let F(ℜ) be the set of all functions f: ℜ → ℜ. Let X be a set and 𝒫(X) be the power set of X, i.e., the set of all subsets of X.

Then "O" is a function O: F(ℜ) → 𝒫(F(ℜ)) such that

  O(g) = { f ∈ F(ℜ) : limsup_{x→∞} f(x)/g(x) < ∞ }

That is, O takes as its input a function from the reals to the reals and returns a set of all functions which are in its "Big-O" class.

That seems really, really pedantic and not particularly illuminating.