Copilot Vision Peeks at Your Edge Browser Sessions to Answer Questions

Microsoft is testing a new Copilot AI feature in its Edge web browser that lets its AI assistant “see” and “hear” what’s on your screen, the company announced Tuesday.

Copilot Vision is a user-activated feature that will show Copilot your Edge browser’s screen to help answer questions or have a conversation. When viewing your Edge tabs, Copilot Vision won’t engage with the web for you; it’s not designed to post Reddit comments or add posts to your Instagram grid (though it could generate some ideas). Instead, Vision is in read-only mode but can still generate audio responses about what it’s seeing on your screen.

It will only be allowed to engage with a specific list of pre-approved websites to start, though Microsoft plans to add more in the future. Vision is not supposed to help you get around website paywalls or engage with sensitive content (like porn), however. If you ask Vision to respond to a paywalled or restricted site, it’s designed to ignore or reject the request.

Still, Vision is a big step toward having an AI assistant that can respond to what you’re seeing and effectively follow you around the web, while providing context and hopefully not too many AI hallucinations along the way.

Vision has been designed to answer questions for you using the internet. Sure, you could look up answers yourself pretty quickly. But Copilot Vision can view photos or read text and pull data from the web to help you quickly fill in the gaps about a cooking recipe or tell you about a location in an image, according to a Microsoft demo video viewed by PCMag. It can also tell you more about movies and TV shows listed on sites like Rotten Tomatoes to help you find something to watch this month without having to do the research yourself.

Activating Copilot Vision within Outlook via Microsoft Edge (Credit: Microsoft/PCMag)

While some users might show Copilot Vision some personal data, Microsoft promises “absolutely none” of the content shown to Copilot Vision will be stored by Microsoft or used for AI training.

Within Edge, Copilot will also generate “daily briefings,” or audio rundowns of news you might have missed that morning. Microsoft’s also testing a “Think Deeper” feature, which is intended to force Copilot to take its time and provide a more substantial, analytical response to questions.

Copilot Vision and Think Deeper are first being rolled out to those in Copilot Labs, a new Windows Insider-like program for Copilot Pro subscribers also announced this week. Microsoft says the Copilot Labs group will be a “small subset” of users who can give Microsoft feedback on its upcoming AI tools.

To access Copilot within Edge, type @copilot into your address bar. Then, if you have access to Vision, navigate to the lower pop-up Copilot menu and choose the square icon on the right of the Copilot chat box. From there, select “Continue.” You can then speak into your microphone to have conversations with Copilot Vision.

For more, check out 5 Intriguing Copilot Features That Promise to Make Microsoft’s AI More Personal.

3 tips to get the most out of Copilot in Windows 11

About Kate Irwin

Reporter

I’m a reporter covering early morning news. Prior to joining PCMag in 2024, I was a reporter and producer at Decrypt and launched its gaming vertical, GG. I have previous bylines with Input, Game Rant, and Dot Esports. I’ve been a PC gamer since The Sims (yes, the original). In 2020, I finally built my first PC with a 3090 graphics card, but also regularly use Mac and iOS devices as well. As a reporter, I’m passionate about uncovering scoops and documenting the wide world of tech and how it affects our daily lives.

Read Kate’s full bio

Read the latest from Kate Irwin