When Joe joined the Army Air Forces in 1943 to become a Staff Sergeant in the Seventh Air Force, he did so because of societal pressures.  He was not stationed overseas and did not participate in combat.  Instead, he was stationed in California in 1943 and assigned to Special Services in Hawaii in 1944 to boost morale by playing exhibition games.  Joe was in California for the 1943 World Series, but in his autobiography he said:

"One of my greatest World Series thrills came in a game in which I wasn't a participant, indeed in which I wasn't even in the ball park.  I was thousands of miles away, in an Army camps in California, wearing a GI outfit, but I was able to listen in to the Series broadcasts of 1943 and I got a bang out of that final game when the Yankees beat the Cardinals four out of five to regain the title of World Champions."

A New York Times article, written December 31st, 1944, said that Joe commented thatthe Yankees should use the Honolulu ballpark for spring training.  He said the Chicago Cubs were interested in the spot, and if they would make room for a major league team, it would be a comfortable place to be.

Joe didn't set foot in Yankee Stadium until the end of the 1945 season; three years after he had left.  When he entered the Stadium, he did so as a civilian with his wife Dorothy and his son Little Joe.  Joe and his family were invited to the game by Larry MacPhail, Dan Topping, and Del Webb; who had bought the Yankees.