There are, in fact, lots of families with kids living in East Harlem. The best time to witness that is in the morning when mothers and fathers are walking them all to area schools, some of which are better than others. Within the past year, I am definitely seeing more Nafler families (that's my term for Non-African American or Latino types), if that's really what you're after.
I'm not feeling the crowded thing, certainly when you compare it to midtown, where we used to live near Lincoln Center, or the currently popular Lower East Side. And convenience really is a matter of where you need to go. If, for example, you want to go to the Upper East Side, it's 10 minutes away. So is LaGuardia, and JFK is less than 30 minutes in fairly light traffic.
Hey, we have a kid too, and I think she is all the better for seeing every day that not everyone is as fortunate as she is.