Unless the court has granted a modification of the visitation arrangement, then you cannot and should not deny visitation. You may believe you have a justifiable reason for denying the visitation rights, but by law you are not permitted to do so.
Often times late child support payments or the lack there of will cause visitation discrepancies. It is important to understand that the court looks and treats visitation and child support independently.
The court does have remedies for these situations, but the decisions must be made by the court. For example the court may decide that the support obligation should be suspended if the custodial parent is not cooperating with the visitation arrangements. On the other hand the court may suspend visitation rights until support payments are made. These remedies are not favored by the court, but obviously they will do what they have to do to act in the best interest of the child(ren). The court does not look favorably upon the parents who decide to take these actions without the permission of the court.