I use the Nikon 70-200mm F2.8 VR plus 1.4 converter with the D3 as a walkabout lens, and don't find it a problem even tramping through woods and fields all day.
If you think the Sigma 500mm F4.5 is heavy then you wouldn't want my AIS 400mm F2.8 which is 6kg.