There are no anchor fingers for the changes:
Am → Em
Em → D
You need to practice correct formation of the chords using chord perfect practice and you need to do one minute changes and you need to try air changes.
Finding difficulty forming and changing between certain chords is a very common issue. It can be different chords for different people. Why not try a different approach too?
To improve the formation of a chosen chord (let’s use E minor here):
Hold your fingers near to but not touching the strings.
Touch the fingers where the chord is but do not press.
Once you have all three touching at the correct place then press them down.
Do not strum - this is a fretting hand exercise only.
Release the pressure after a few seconds but keep touching the strings.
Then move your hand away from the strings by a small amount. All fingers away.
Repeat the process.
Then, to improve changes to and from the chosen chord and other commonly grouped chords, repeat the above process with one alteration. After the final step of lifting all fingers away, the next cycle would be to move fingers over the chord that you are changing to. Once that chord has been done and fingers are lifted away, go back to the first chord of the pair.
Wash, rinse, repeat. Make this exercise last about five minutes.
Example - E minor chord plus some associated chords
1 minute - Em alone
1 minute - Em & Am
1 minute - Em and D
1 minute - Em alone
Practice daily and within a week you will be smashing it.
Hope that helps.
Cheers 
| Richard_close2u | Community Moderator, Official Guide, JustinGuitar Approved Teacher