The apparent paradox stems from the implicit assumption that you are equally likely to win or lose, regardless of the amount in your wallet. However, the more money you have, the more likely you are to lose and as the same is true for your opponent, the game ends up being fair to both.
To solve the problem completely, we need to assume some probability distribution over the amount of cash you and your opponent have. For a simple example, suppose that each of you tosses a coin. If it shows heads, you put 10 dollars in your wallet and if it shows tails, you put 20, and then you play the game. Disregarding the two cases of ties (10-10 and 20-20), you win 10 dollars in one case and lose 10 dollars in the other, and the game is fair. The coin toss is a simple example but regardless of what the range of money is and what the corresponding probabilities are, the potential wins and losses for you and your opponent cancel each other and the game ends up being fair.