There is a direct route by car -- take Route 9 all the way down -- but I wouldn't recommend it unless you like 100 miles of stop-and-go traffic. This would be a long drive, but will probably feel even longer than that.
You could drive to the nearest Metro North station in Poughkeepsie and take the train down to Grand Central, or you could find an alternate route by car (maybe cut over eastward to the Taconic State Parkway and take highways from there). No matter how you slice it, though, it'll take over 2 hours.
