I've had my parakeet for 7 days now... what am I doing wrong?

He still is very jumpy and afraid of us. I have two small children, so I try to keep them from startling him. But I still have to live my life, like vacuuming! I've read articles and thought I was doing everything right. He will let me get my hand within an inch of him, but any closer and he flies around the cage or jumps from perch to perch away from me. He does seem interested in us, today he was near the front of the cage and I was talking to him, he just sat there and seemed entertained. So why is he afraid still? Hes a good eater AND pooper thats for sure! Though he wont eat anything other than seed or millet spray... what do I do next?