Much the way firms battled in the 1980s over VHS and Betamax video formats, tech rivals are fighting over the technology used to deliver and display Web video. Currently, video-streaming services like Netflix Inc. and Google’s YouTube pay patent royalties, as do makers of Blu-ray disc players and other hardware.

These firms pay royalties to an organization called MPEG LA, which is the target of the formal antitrust probe, the people familiar with the matter said. MPEG LA has amassed pools of patents covering widely used video formats and collects royalties for its members, which include Apple Inc. and Microsoft Corp.

Antitrust enforcers are investigating whether MPEG LA, or its members, are trying to cripple an alternative format called VP8 that Google released last year—by creating legal uncertainty over whether users might violate patents by employing that technology, these people added.

The probe, which pits Google and open-source software advocates against some technology giants like Apple, could help determine whether anyone will own rights over the creation and broadcast of online video in the next major Web programming language, called HTML 5.

At stake is “who is going to have competitive clout in the world after television,” said Eben Moglen, a Columbia University professor who supports free and open software.

The California State Attorney General’s office is also investigating the matter, according to people familiar with the matter.

MPEG LA didn’t confirm or deny it is under investigation. But the group says it isn’t acting to kill a competitor. It said it’s simply offering a service for patent holders and is agnostic about which video format prevails.

“We are effectively a convenience store” for licensing patents, said Larry Horn, MPEG LA’s chief executive. “We have no dog in that fight.”

Representatives of both law enforcement agencies as well as Apple and Google declined to comment. Microsoft didn’t respond to a request for comment.

MPEG LA, which was formed in the late 1990s,manages the licensing of more than 1,700 patents used in a high-definition video encoding standard known as H.264. The Justice Department is concerned the group’s actions may stifle competition to that dominant format, the people familiar with the matter said.

Google has been offering an alternative. The Silicon Valley giant last year paid $125 million to buy a company that developed the video-compression format called VP8. Google later released it as a royalty-free standard under an open license that enables software developers to use it any way they wish.

At present, no patent royalties are charged for using Google’s VP8 format. But MPEG LA has questioned that status, and last month issued a call for companies to submit patents they believe may be infringed by VP8. “I can tell you: VP8 is not patent-free,” Mr. Horn said. “It’s simply nonsense.”

For some people in the tech industry, the issue is less about cost and more about competition and control over technologies at the heart of the Internet. “How could it come to pass that it’s illegal to compete?” asked Monty Montgomery, who runs a free software foundation, XIPH.org, and supports VP8. “That’s when everybody’s antitrust bells should be going off.”

The threat of future lawsuits has helped persuade some companies to forsake VP8. Apple’s chief executive, Steve Jobs, explained in an email to the Free Software Foundation Europe last year that a patent pool was assembled to “go after” a previous open-source format.

“All video codecs are covered by patents,” Mr. Jobs wrote. “Unfortunately, just because something is open-source, it doesn’t mean or guarantee that it doesn’t infringe on others patents.”