To make them feel more comfortable I place the feeders in or near trees. I have several feeders hanging from a dogwood and pear tree in the front of the house. They like to check out the view from the top of the trees and then filter down to the lower branches until they feel safe enough to approach the feeders.
And their seed has to be fresh and the feeder clean. To make sure your Nyjer seed is fresh scatter some seed on white paper and crush it with your fingernails. If you see if any oil stains on the paper the seed is good. The finches use their bills to twist the seed and sip the oil and then drop the shell. On these days where they need a lot of energy to change feather colors, if your seed has dried out your feeder will be skipped. (Wild Birds Unlimited receives a fresh load of seed each week.)
If they haven’t touched your feeder in a while make sure there is no mold in the bottom of your feeder. In Michigan where it can be wet in the spring the seed may not get a chance to air out and begin to mold. This can be dangerous to the finches and they will avoid your feeder again. To prevent mold in bad weather use Feeder Fresh (a silica grit that absorbs water and humidity, is safe for birds, and made from non-toxic absorbent sand). You can also shelter your feeder from the elements by using a WBU Weather Guard.
But overall I think you just have to be patient. Goldfinches do things in their own way, for their own reasons. Just keep trying. Change the seed or move locations to see if they will come back.