How to modify the content of WebRTC MediaStream video track?

In WebRTC, MediaStream is an object representing media stream information, including video and audio. The Video Track is a component of MediaStream. Modifying video tracks enables various functionalities, such as adding filters, performing image recognition, or changing the background.

Modifying Video Tracks in WebRTC MediaStream

1. Acquire MediaStream: First, obtain a MediaStream object, which can be acquired from the user's camera and microphone or from other video streams.

javascript
navigator.mediaDevices.getUserMedia({ video: true })
  .then(stream => {
    // Use the stream
  })
  .catch(error => {
    console.error('Failed to get video stream', error);
  });

1. Extract Video Track: Extract the video track from the MediaStream.

javascript
const videoTrack = stream.getVideoTracks()[0];

1. Process with Canvas: Draw video frames onto the Canvas to process the video content during this step.

javascript
const video = document.createElement('video');
video.srcObject = stream;
video.play();

const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');

function processFrame() {
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
  // Add processing logic here, such as filters or image recognition

  requestAnimationFrame(processFrame);
}

requestAnimationFrame(processFrame);

1. Convert Processed Data to MediaStreamTrack: Create a new MediaStreamTrack from the Canvas output.

javascript
const streamFromCanvas = canvas.captureStream();
const processedVideoTrack = streamFromCanvas.getVideoTracks()[0];

1. Replace Video Track in Original Stream: Replace the video track in the original MediaStream with the processed video track.

javascript
const sender = peerConnection.getSenders().find(s => s.track.kind === 'video');
sender.replaceTrack(processedVideoTrack);

Application Example

Suppose we want to add a simple grayscale filter to a video call. We can integrate the following code into the Canvas processing steps:

javascript
function applyGrayScale(ctx, width, height) {
  const imageData = ctx.getImageData(0, 0, width, height);
  const data = imageData.data;

  for (let i = 0; i < data.length; i += 4) {
    const avg = (data[i] + data[i + 1] + data[i + 2]) / 3;
    data[i] = avg; // red
    data[i + 1] = avg; // green
    data[i + 2] = avg; // blue
  }

  ctx.putImageData(imageData, 0, 0);
}

function processFrame() {
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
  applyGrayScale(ctx, canvas.width, canvas.height);
  requestAnimationFrame(processFrame);
}

This code converts each frame of the video stream to grayscale and processes it further on the Canvas for retransmission.

Summary

Through the steps outlined above and a specific example, modifying video tracks in WebRTC is straightforward. It primarily involves acquiring the video stream, processing the video, and re-encapsulating and sending the processed video. This opens up numerous possibilities for developing creative and interactive real-time video applications.