In theory, the builder/contractor you hire is supposed to supervise the subcontractors and examine their work to make sure that everything is done right. The builder is also responsible for assuring that the construction is done to agreed-upon specifications.
In practice, everyone makes mistakes, including me and you and builders and subcontractors. Not even NASA claims "zero defects" any more.
The builder might miss catching a mistake, or he might catch it, but his decision about what to do about it may not be what you would have chosen had you known about it.
Or everything may be built right, but then you realize that what you had planned is just not going to work for you and you need to change it. It is better to discover this as it is done rather than after the house if finished.
Aside from mistakes and changes, numerous decisions have to be made during the building process which affect the way the house will look.
If you are not around, the builder or his subs will make those decisions and you will have to live with them. If you visit the building site frequently, the subs will be asking you what you prefer. We would rather make these decisions ourselves and get things just the way we want them.
Some people have come in, hired a builder, left town, and not come back until the house is done. If you have a pretty relaxed attitude about how the house turns out, this may work for you and is certainly less stressful.
We took the opposite approach -- checking the house every day. We did not consider this much of a burden because for one thing, we went there to pick up our mail anyway (rather than getting mail at our rent house for 7 months and then having to change our address again).
For another thing, we enjoyed watching the house go up. We also took pictures every step of the way.
I am happy to say that our builder was exceedingly patient with my nosing around and questioning everything he did. How would you react to someone who doesn't really know anything about how to do your job looking over your shoulder asking questions and making suggestions all the time? Keep this picture in your mind if you do decide to frequent the building site.
The builder also has to act as a buffer between the client and the subs, who usually do not have the builder's public relation skills. For example, when Craft told the electrician that the satellite TV cabling would have to be redone, the look on the electrician's face was enough to make me glad that I don't have to deal with the subs myself.
Buss' contract for building Judy's house had terms about our not being around when work was going on. While it's a good idea not to be getting in the way, I did go by many times while work was going on and just looked to see what was happening. As I said before, people make mistakes and the best time to catch them is as they are happening. One time at Judy's, workers were putting vinyl siding on a wall that did not have the Tyvek wrap over the plywood. I called Buss' office and the workers were stopped, someone brought out some wrap, the siding in that area was removed and the sheathing put on. If I had not happened by, it is unlikely that anyone would have known about the missing wrap until the house was occupied and cold, damp air was coming through the walls.
As long as you stay out of the way and don't interfere with workers, there is no reason, contract terms not withstanding, why you should not go by and check up on things.