Combustion deposits and acids (and the result of their reactions and oxidations post-combustion) can create carcinogenic compounds that stay held in suspension or dissolved within the oil. Physical contact with these compounds could 'increase your chances' of getting cancer or other types of dermatitis.
The risks remain pretty low, but in these days of namby-pamby pussy-footing around even the slightest bit of risk for fear of criminal offences or litigation, you are advised to wear gloves of some sort. I don't bother. The tiny bits of used oil that get on my hands during oil changes etc are miniscule really (and only 'on' me for moments before I clean my hands with something (that'll turn out to be just as bad for us eventually anyway, like detergents).
Clean oil is clean, processed oil, and isn't really harmful at all (but don't try drinking it, or dripping molten lead in it etc).