FORT ALBANY was located on Arlington Heights, southwest of the Long Bridge that led into Washington, DC. The land to the rear of this earthwork structure was used as a parade ground for the frequent review of the thousands of soldiers encamped nearby (at Camp Seward and Camp Chase). 

Below is an 1861 drawing, printed in Frank Leslie's Illustrated Newspaper on 31 August 1861, while it was still being constructed. The Arlington House is in the background, on the left, while the Potomac River and the White House are in the distance on the right side of image. 

Below is another image of Fort Albany, this time facing south and looking at the main gate. This was also from 1861, with New York troops encamped outside. It was built to overlook the Columbia Turnpike, which is the road running horizontally through image (it turns south after passing the western view of the fort). 

To the east of the fort (left side of above picture from this view) is the Roach (or Roche) farm, which can be seen on the 1862 map seen below. As Seth Bryant mentioned in his journal (Nov 11th entry), the troop review was on the flat back of Fort Albany, which would have been below the Heights. On that day, he says that after the review they marched on to the road and, instead of taking a left back towards Camp (Seward), they turned right towards the Long Bridge, and marched down to the flat marshy ground by the river. The location of Gen. Casey's gathering would have been somewhere in the location marked with the red circle.  

Fort Albany no longer exists, as the earthworks and hillside was completely eliminated for the highway system, with Rt. 395 now runs through its former location. A historical marker stands in Prospect Hill Park (1025 S Arlington Ridge Rd, Arlington, VA 22202) to mark the nearby spot.

Links:
1862 Map of N. Eastern Virginia and vicinity of Washington -[LoC or Rumsey]
Defenses of Washington Marker Series [Fort Albany on Flickr]
Fort Albany at Arlington Heights, 1861 (E.F.Ruhl, Publisher - LoC site)
Frank Leslie's Illustrated Newspaper, 31 Aug 1861 [Vol. XII, pg. 253]
The Historical Marker Database: Fort Albany 

During his time with the 27th Maine, Private Charles Gooch, of Company I, wrote over fifty letters to his wife Julia, back home in Kennebunk, Maine. The following is a letter he wrote to her on the 15th of January 1863. I've transcribed it as written, with lack of punctuation and spelling errors included.

Notes:
(pg 1) "Theodore" is Theodore Gooch, a cousin of Charles W Gooch, who also lived in Kennebunk. His full name is mentioned in an earlier letter.
(pg 2) The Christian Watchman and Reflector was a Baptist newspaper style weekly periodical, printed in Boston, MA. The article mentioned by Charles was likely from the Dec 25th issue (viewable on Genealogy Bank w/ subscription), which included a letter written by a soldier at Fort Albany, describing the 27th Maine (and other units) marching onto the grounds.
   (fath) = perhaps he meant "faith"? As he wrote this word on both ends of the sentence where he declined the offer of milk, was he fibbing about already having coffee that morning? Charles was a religious man, so perhaps he felt bad about lying to the Irishman and wife, so called that out in his letter.
(pg 4) "Isaac" is Sgt Isaac Emery of Kennebunk, who was the brother of Charles' wife Julia.