Williams, watched by her family in the stands, lost a close first set before leveling the match in the second on center court, the site of seven of her 23 Grand Slam singles titles.
She was a break in the third, but Tan showed her resilience as she fought back to take a thrilling final set in a biting tie-break.
When asked if this was the last Wimbledon match of her decorated career, Williams said it was a question she “cannot answer”.
“Who knows where I will appear?” she added.
On Tuesday, she made her long-awaited singles return, a year after retiring in her last opening round match at Wimbledon.
The momentum shifted between the two players over the three-hour, 11-minute encounter – first in favor of Tan as she edged out the first set, then towards Williams as she rallied in the second.
It looked like the 40-year-old Williams would prevail in the deciding set as she served for the win, but the momentous comeback was not to happen.
Tan replied at 5-5, and although she failed to convert a match point at 6-5, she made no mistake when she got a second chance in the tie-break.
Ranked No. 115 in the world, Tan was playing her first match in the main draw at Wimbledon and ninth in all Grand Slams.
“When I saw the draw, I was very scared,” she said after the game. “It’s Serena Williams, she’s a legend. I was like, ‘Oh my god, how can I play?’ And if I can win one or two games, that’s really good for me.”
Under the lights of center court, Tan comfortably did more than that – triumphing against one of the greatest athletes the sport has ever seen.
“For my first Wimbledon – it’s wow. Just wow,” she added, struggling to find the words to capture her emotions.
It remains unclear if this was Williams’ last outing at Wimbledon. When asked to sum up her legacy at the tournament, she was succinct, if understated.
“I think I’m pretty solid on the grass,” she said. “Maybe not today, but pretty solid there.”
Nadal and Swiatek advance
In Tuesday’s previous matches on center court, this year’s French Open champions Rafael Nadal and Iga Swiatek picked up wins.
Two-time Wimbledon winner Nadal survived a scare to beat Francisco Cerundolo 6-4 6-3 3-6 6-4 as he began his quest for a 23rd Grand Slam title, while Swiatek recorded his 36th consecutive victory by beating Jana Fett 6-0 6-3.
Nadal, who missed last year’s tournament through injury, received a warm welcome on center court, where he was severely tested by Argentina’s Cerundolo.
The Spaniard looked on course for a routine win as he trailed 2-0 and a break in the third set, but the free-swinging Cerundolo fought back with a double break to take the set.
The match seemed headed for a decider when the world No. 41 busted in the fourth, only for Nadal to keep his best until the end and seal the win with a double break.
Swiatek, meanwhile, dominated the first set against Fett in her first match since French Open but had to overcome a tough spell in the second when Fett nearly got a double break.
The top seed recovered from a shaky start to the set and played five games in a row, taking her winning streak to 36 games.