Seoul's Joint Chiefs of Staff said in a statement that bad weather and an engine problem likely caused the North Korean ship to cross its boundaries. No further violence was reported.
The incident was the second time that South Korea fired warning shots to drive back a North Korean ship since President Moon Jae-in took office in 2017. The first incident happened in September, according to South Korea's military.
Ties between the two Koreas are strained amid a stalemate in U.S.-led diplomacy on ending the North Korean nuclear crisis.
Earlier this month, the United States and South Korea said they would postpone a joint military exercise after the North deemed merely scaling back the exercises as insufficient.
North Korea on Monday said its troops conducted artillery firing drills near the sea boundary. South Korea on Tuesday sent a formal protest message to North Korea, accusing them of violating last year's inter-Korean agreements aimed at reducing front-line military animosities.
The sea boundary is a source of tension between the two Koreas, which have fought several deadly naval skirmishes in the area in past years. In 2010, two attacks blamed on North Korea near the sea boundary killed 50 South Koreans.
The Associated Press contributed to this report.