Mathematics is a bit different than science, science doesn't claim truths, it says what the best description of things are based on the facts, the facts are evidence based. When you skip the facts and evidence part and just claim things, then you are in the land of woo. At the basis of science is essentially the laws of thought which, like good axioms should be, claim the smallest possible thing to reason from ( https://en.wikipedia.org/wiki/Law_of_thought ). Woo tends to claim large things partially based on existing knowledge and partially based on things we don't know but may be possible.
Axioms are only useful if they are widely accepted and they create an useful system. The modern axiomatic system of mathematics was created to support an already useful system, and they are (mostly) based on actual indisputable realities of our world.
The main axiom of panpsychism, if I understand correctly, is that all or most objects of reality have a mind. Now, as an axiom in itself it is pretty imprecise and not self-evident at all, as it depends on the definition of "mind". Also, it doesn't seem to create testable or useful theories out of that axiom.
The axioms of mathematics were _chosen_, because they have properties that are immensely useful. You can decide to use other axioms, and get different results, which may also be useful.
I'm genuinely asking. My understanding is that we also accept those axioms.