A fun hypothesis.

Ah, Time.

Short answer, Yes and No.

The passage of time is relative and rather arbitrary measurement.

Most measurements are set against something for context, and human time is set against the rotation of 1 specific planet.

The magnetic poles or nature of the planet are not a factor.

If a species were present on the surface and "invented time", they would no doubt plant a stick in the ground and create a sun-dial.

Lets say the planet takes 24 hours to do a complete rotation, and the drift in the opposite direction makes it take 1 hour longer for the stick in the ground to reach the same point it was before.

From the local perspective yes a day would be now 25 hours, but as this is how they have always perceived time they do not need to re-calibrate.

If visitors (an observer outside the system) came to the Earth in this condition, they would have to make this adjustment when on the ground, as their observations from outside the system would probably relate to the spin of the whole mass.
Upon close observation they would see the visual difference between what is going on and their readings from a distance.

In the distant past the Earth spun a lot faster so 1 day, 1 minute, 1 second, were all a lot quicker than they are now.

From the point of view of those within the system of Earth, time is slowing down.
This means every few years we have to re-calibrate all the clocks around the world.
The electric companies change the frequency of the mains supply and the clocks running that are plugged in get adjusted (eg. the clock on your cooker).

These leads to the question, Which is a more accurate clock, an atomic clock or a sun-dial ?

For biological life within the system, the sun-dial is more accurate, but for observers outside the system atomic timing is more accurate.

Every day the planet spins slightly slower, so really there is a micro adjustment needed every day (depending on the type of clock you use).