Consider yourself lucky that you get calloused at all. I don’t. I like to joke my finger tips are made of mashed potatoes. My skin is hitting the fretboard before the string is contacting the fret. When I remove my finger the groove sticks around for a long time before blood flow (or whatever process) causes it to smooth out.
On a climbing trip my wife is filing down her callouses by the end of the first week and my tips are seeping blood. She should have been the one to try guitar.