Tencent restricts users from sharing Douyin content on its instant messaging apps WeChat and QQ, which should be prohibited by anti-monopoly law, ByteDance said, adding that it has asked the court to order Tencent to stop such behaviour.
"We believe that competition is better for consumers and promote innovation," Douyin said in a statement. "We have filed a lawsuit to protect our rights and those of our users."
Tencent did not immediately respond to a request for comment.
Douyin's lawsuit comes as Chinese regulators step up regulations on tech giants. Beijing issued draft rules in November aimed at preventing monopolistic behaviour by internet firms, marking China's first serious regulatory move against the sector.
The State Administration of Market Regulation in December announced the launch of an antitrust investigation into e-commerce giant Alibaba.
ByteDance competes with Tencent in social media in China and has challenged Tencent previously. In 2018, it sued Tencent for anti-competitive behaviour, alleging Tencent's social platform QQ zone and its Guanjia software blocked ByteDance's news aggregator Toutiao's links.
Tencent filed a lawsuit accusing ByteDance of defamation around the same time.
(Reporting by Yingzhi Yang in Beijing and Brenda Goh in Shanghai, Additional reporting by Pei Li; editing by Ed Osmond and Jason Neely)