Before Joe signed with the Mission Red A's, he went to watch his older brother Vince practicing with the San Francisco Seals. Spike Hennessy, who was a scout for the Seals, saw Joe watching Vince. He invited Joe to go with him and speak to Charley Graham, the President of the Seals, who then invited Joe to begin working out with the team after Hennessy told him about Joe's performance in the sandlot games. 

A few days before the end of the 1942 season, Seals players Augie Galan and Henry Oana took a trip to Honolulu.  With the encouragement of Vince and the permission of Seals manager Ike Caveney, Joe played shortstop for the team.  After the game he was invited to report to training school for the 1933 season.   This training camp is where Joe said he began to love the game.  At camp, it was concluded that Joe didn't have a good enough sidearm to play in the infield, so
they moved him to the outfield. His hitting also improved.  At the end of the training he signed a contract with the Seals for $225 per month.

At the age of eighteen, he reported to the regular training camp for team members. He was young and shy and had not improved his fielding enough to play. He was benched with his brother Vince and "Prince" Henry Oana. However, when the season games were underway, he took over in right field for Ed Stewart. He did well and was not benched again unless he had an injury. The Seals released Vince in Joe's first season, but Vince went on to play for the Hollywood team. At the end of the 1933 season, Joe's father realized that money could be made by playing baseball and converted from Bocci to the American pastime.

During Joe's 1934 season, New York Yankee Pacific Coast scouts, Joe Devine and Bill Essick, were sent to watch Joe play.  However, Joe had incurred a knee injury after a double-header at Seal Park in June and was put in an aluminum splint for three weeks.  After one week back, Joe then slipped in the dugout and was out for the rest of the season.  Luckily for Joe, Essick contacted Graham and told him to send Joe to Dr. Spencer in LA. If the doctor said Joe would be able to play again then the Yankees wanted Joe.  The doctor gave Joe the OK and the Yankees offered him a contract for the 1936 season.  Joe then went through a series of eight treatments to restore his knee before the 1935 season.  Because o fthe treatments, Joe missed much of spring training and started the season off slow as a consequence; but he soon regained ground.  Lefty O'Doul also became the new manager for the Seals.