Skip to content

Build Your Own AI Browser Agent: A Step-by-Step Guide with Browser Use & Python

Build Your Own AI Browser Agent: A Step-by-Step Guide with Browser Use & Python

Ever wished you had an assistant who could just *handle* things online for you? I’m talking about booking flights, snagging concert tickets the moment they drop, comparing product prices, or even pulling specific information from websites and organizing it neatly. Well, what if I told you that you can build your own AI agent to do just that – control your web browser and automate those tasks? Sounds futuristic, right? But it’s surprisingly achievable, even if you’re not a hardcore coder!

Recently, I stumbled upon an incredible framework called Browser Use, and I was blown away by how easy it makes creating these AI browser agents. It requires very little coding (just a sprinkle of Python), it’s free to get started, and it works with various Large Language Models (LLMs) – the brains behind the operation. You can use cloud-based LLMs like Claude or GPT, or even run one locally using Ollama if you prefer keeping things private.

In this guide, I’ll walk you through exactly how to build your own AI browser agent using Browser Use, step-by-step. We’ll cover:

  • What Browser Use is and why it’s so cool.
  • Setting up your environment (it’s easier than you think!).
  • Connecting an LLM like Claude.
  • Running your first browser automation task.
  • Leveraging your *own* browser session (this is a game-changer!).
  • Extracting information into a structured format.
  • Some handy tips and tricks.

Ready to give your browser superpowers? Let’s dive in!

What is Browser Use (and Why Should You Care)?

So, what exactly is Browser Use? Think of it as a bridge between a powerful LLM and your web browser. The LLM provides the reasoning and decision-making (“Okay, I need to find the ‘Add to Cart’ button and click it”), while Browser Use provides the tools to actually *interact* with the website – clicking, typing, scrolling, navigating, etc. It uses another powerful tool called Playwright under the hood to handle the low-level browser control.

Here’s why I think it’s particularly exciting:

  • It Uses Your Existing Browser Context: This is huge! Many web scraping or automation tools run in a clean, isolated environment. Browser Use can control a browser *on your actual computer*. This means if you’re already logged into Amazon, Gmail, or your flight booking site, the AI agent can just pick up where you left off, bypassing tricky login processes.
  • It’s LLM Agnostic: You’re not locked into one specific AI provider. Use OpenAI’s GPT-4, Anthropic’s Claude, DeepSeek, or run a local model with Ollama – the choice is yours!
  • Free and Open Source: While they offer a cloud service, the core framework is free to use and run locally.
  • Impressive Capabilities: The framework allows the LLM to “see” the page (both visually and the underlying code structure, the DOM) and decide on the next best action. It can handle multiple tabs, go back and forth, and intelligently interact with web elements. Some benchmarks even show it outperforming other tools!
  • Beginner-Friendly: While some Python is needed, the setup and basic usage are quite straightforward, as you’ll see.

Getting Started: Setting Up Your Environment

Alright, let’s get our hands dirty. Setting this up involves just a couple of installation steps. You’ll need Python installed on your system first. If you haven’t got Python yet, head over to the official Python website and grab the latest version.

(Optional Tip: If you’re familiar with Python virtual environments, it’s good practice to create one for this project to keep dependencies separate. If not, don’t worry, you can proceed without one for now.)

Step 1: Install Browser Use & Playwright

Open your terminal or command prompt and run the following command:

pip install browser-use

(If you’re on macOS or Linux, you might need to use pip3 instead of pip).

Once Browser Use is installed, you need to install the necessary browser components for Playwright. Run this command:

playwright install

This command downloads the browser binaries (like Chromium, Firefox, WebKit) that Playwright needs to control the browsers. You should see some output indicating successful installation.

Step 2: Choose and Configure Your LLM

Browser Use needs an LLM to function. As mentioned, you have options. For this tutorial, I’ll follow the video’s lead and use Anthropic’s Claude, as it’s powerful and relatively easy to set up with an API key.

  1. Get an API Key: Go to the Anthropic Console. You might need to sign up if you don’t have an account. Navigate to the API Keys section and create a new key. Give it a descriptive name (e.g., “Browser Use Tutorial”) and copy the key immediately – you won’t be able to see it again! Important: Keep this key secret! Using cloud LLMs involves costs based on usage, so monitor your spending.
  2. Store Your API Key Securely: The best practice is to use environment variables. Create a new file in your project directory named exactly `.env` (yes, starting with a dot). Inside this file, add the following line, pasting your actual API key after the equals sign, with no quotes: 
    ANTHROPIC_API_KEY=YOUR_ACTUAL_API_KEY_HERE
  3. Install a Helper Library (Optional but Recommended): To easily load this `.env` file into our Python script, let’s install another package: 
    pip install python-dotenv

Alternative LLMs: If you prefer OpenAI (GPT), Google (Gemini), or a local model via Ollama, check the Browser Use documentation on Supported Models. It provides the specific environment variable names (e.g., `OPENAI_API_KEY`) and setup instructions for each.

Building Your First AI Browser Agent

Now for the fun part! Create a new Python file in your project directory (e.g., `main.py`). Let’s add the initial code based on the video’s example.


import asyncio
from browser_use import Agent, Browser, ChatAnthropic  # Use ChatOpenAI, ChatOllama etc. if using others
from dotenv import load_dotenv

# Load environment variables from .env file (for API key)
load_dotenv()

async def main():
    # Initialize the LLM (using Anthropic Claude in this case)
    llm = ChatAnthropic(model="claude-3-haiku-20240307") # You can try other Claude models too

    # Create the AI Agent
    agent = Agent(llm=llm, browser=None) # We'll add the browser later for local context

    # Define the task for the agent
    task = "Compare the price of GPT-4 Turbo with Claude 3 Sonnet."
    # task = "Go to amazon.com and search for 'ergonomic keyboard'" # Try other tasks!

    print(f"Running task: {task}")

    # Run the agent
    result = await agent.run(task)

    # Print the result (which includes thoughts, actions, and outcome)
    print("n--- Agent Result ---")
    print(result)
    print("--------------------")

# Run the asynchronous main function
if __name__ == "__main__":
    asyncio.run(main())

Before running: Make sure you close any currently open instances of Google Chrome. Browser Use (via Playwright) often works best when it can launch a fresh browser instance, and existing ones can sometimes interfere.

Now, run your Python script (e.g., `python main.py` in your terminal). You should see a new Chrome window pop up! In your terminal, you’ll see the agent’s “thoughts” and actions as it navigates the web to fulfill your request. It’s fascinating to watch it work!

Sometimes, the agent might not succeed, especially with vague tasks or complex websites. That’s normal! You might need to refine your instructions (the `task` variable) to be more specific.

Unlocking Real Power: Using Your Own Browser

The default behavior launches a clean, isolated browser instance (like Incognito). But the *real* magic happens when Browser Use controls your *actual* Chrome browser, complete with your logins, cookies, and extensions.

Here’s how to set that up:

  1. Find Your Chrome Executable Path: Browser Use needs to know where your Chrome installation lives. The path depends on your operating system:
    • Windows: Usually `C:Program FilesGoogleChromeApplicationchrome.exe` or `C:Program Files (x86)…`
    • macOS: Usually `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
    • Linux: Often `/usr/bin/google-chrome` or similar.

    Check the Browser Use “Connect to Your Browser” docs for specifics if needed.

  2. Modify Your Python Code: Update your `main.py` like this:

import asyncio
from browser_use import Agent, Browser, ChatAnthropic
from dotenv import load_dotenv
import os # Import os to handle paths potentially

load_dotenv()

async def main():
    # --- New Browser Setup ---
    # !! IMPORTANT: Close all Chrome instances before running !!
    chrome_path = "C:Program FilesGoogleChromeApplicationchrome.exe" # ADJUST THIS PATH FOR YOUR SYSTEM
    # Check if the path exists (optional but good practice)
    if not os.path.exists(chrome_path):
         print(f"Error: Chrome path not found at {chrome_path}")
         print("Please update the 'chrome_path' variable in the script.")
         return # Exit if path is wrong

    browser_instance = Browser(chrome_path=chrome_path)
    await browser_instance.init() # Initialize the connection
    # -------------------------

    llm = ChatAnthropic(model="claude-3-haiku-20240307")

    # Pass the browser_instance to the Agent
    agent = Agent(llm=llm, browser=browser_instance) # Use the connected browser

    # Define the task
    task = "Go to Tech With Tim's Instagram page." # Example using a specific page

    print(f"Running task: {task}")
    result = await agent.run(task)

    print("n--- Agent Result ---")
    print(result)
    print("--------------------")

    # --- Close the browser connection ---
    await browser_instance.close()
    # ----------------------------------

if __name__ == "__main__":
    asyncio.run(main())

Crucially:

  • Update the `chrome_path` variable to match your system.
  • Make sure all Chrome windows are closed before running the script.
  • We now create a `Browser` object, `init()` it, pass it to the `Agent`, and `close()` it at the end.

Run it again. This time, it should open your *personal* Chrome window (you might see your bookmarks, theme, logged-in accounts). Now, the agent can interact with sites you’re already signed into – incredibly powerful for automation!

Extracting Data Like a Pro: Structured Output

Often, you don’t just want the agent to *do* something; you want it to *retrieve* information in a clean, usable format. By default, the agent might return information in natural language, which can be messy to parse programmatically.

This is where structured output using Pydantic comes in. Pydantic helps define data structures (like classes) that Browser Use can populate, ensuring you get consistent, predictable JSON-like output.

Let’s modify the code to grab the captions and URLs of the latest Instagram posts and structure the output:


import asyncio
from browser_use import Agent, Browser, ChatAnthropic, Controller # Import Controller
from dotenv import load_dotenv
import os
from pydantic import BaseModel, Field # Import Pydantic
from typing import List # Import List for typing

load_dotenv()

# --- Define Pydantic Models for Structured Output ---
class Post(BaseModel):
    caption: str = Field(description="The text content of the Instagram post caption.")
    url: str = Field(description="The direct URL to the Instagram post.")

class Posts(BaseModel):
    posts: List[Post] = Field(description="A list of the most recent Instagram posts found.")
# ----------------------------------------------------

async def main():
    chrome_path = "C:Program FilesGoogleChromeApplicationchrome.exe" # ADJUST THIS PATH
    if not os.path.exists(chrome_path):
         print(f"Error: Chrome path not found at {chrome_path}")
         print("Please update the 'chrome_path' variable in the script.")
         return

    # Close all Chrome instances before running
    print("Please ensure all Chrome instances are closed before proceeding.")
    # input("Press Enter to continue after closing Chrome...") # Optional pause

    browser_instance = Browser(chrome_path=chrome_path)
    await browser_instance.init()

    llm = ChatAnthropic(model="claude-3-haiku-20240307")

    # --- Setup Controller for Structured Output ---
    controller = Controller(output_model=Posts) # Tell it to use our Posts model
    # --------------------------------------------

    # Pass both browser and controller to the Agent
    agent = Agent(llm=llm, browser=browser_instance, controller=controller)

    # Update the task to be more specific about extraction
    task = "Go to instagram.com/tech_with_tim and extract the caption and URL for the 5 most recent posts."

    print(f"Running task: {task}")
    result = await agent.run(task)

    print("n--- Agent Result ---")
    # Access the structured final result
    if result.final_result:
        print("Structured Output:")
        # Pydantic automatically validates and parses the data here!
        parsed_posts: Posts = result.final_result
        for i, post in enumerate(parsed_posts.posts):
             print(f"  Post {i+1}:")
             print(f"    Caption: {post.caption[:100]}...") # Print first 100 chars
             print(f"    URL: {post.url}")
    else:
        print("Could not extract structured data. Raw output:")
        print(result.extracted_content) # Fallback to raw extraction if needed
    print("--------------------")

    await browser_instance.close()

if __name__ == "__main__":
    asyncio.run(main())

Here’s what changed:

  • We imported `BaseModel`, `Field` from `pydantic` and `List` from `typing`.
  • We defined two classes, `Post` and `Posts`, inheriting from `BaseModel`. These define the *structure* we want. `Field(description=…)` helps the LLM understand what data to put where.
  • We imported and created a `Controller`, telling it to use our `Posts` model as the desired `output_model`.
  • We passed this `controller` to the `Agent`.
  • Our task is now more specific about *extracting* the data.
  • We access the structured data via `result.final_result`. Pydantic handles the validation!

Run this, and assuming the agent successfully finds the posts, you should get nicely formatted output with captions and URLs, ready to be used in other parts of your application!

Here’s a quick comparison:

Feature Default Output (Extracted Content) Structured Output (Final Result with Pydantic)
Format Often natural language, mixed content, potentially inconsistent JSON/list structure. Well-defined JSON-like structure matching your Pydantic models.
Consistency Can vary depending on the LLM’s interpretation and website changes. Highly consistent as long as the LLM can find the data matching the defined fields.
Ease of Use (Programmatic) Requires potentially complex parsing logic (string manipulation, regex). Directly usable as Python objects after Pydantic validation. Simple attribute access (e.g., `post.caption`).
Reliability Lower, as parsing unstructured text is brittle. Higher, thanks to schema enforcement and validation.

Tips and Tricks for Efficiency

As you build more complex agents, here are a couple of features from Browser Use that I found particularly helpful:

1. Speeding Up with Initial Actions

Sometimes, you know the first few steps the agent *always* needs to take (like navigating to a specific URL). You can pre-define these to save the LLM processing time (and potentially cost) and make the agent faster.


# Inside your async def main():
initial_actions = [
    {"action": "goto", "url": "https://www.instagram.com/tech_with_tim"}
    # You could add more actions like scroll, wait, etc.
]

# When creating the agent:
agent = Agent(
    llm=llm,
    browser=browser_instance,
    controller=controller,
    initial_actions=initial_actions # Pass the predefined actions
)

# Your task can now assume you're already on the page:
task = "Extract the caption and URL for the 5 most recent posts." # No need to say "Go to..."

The agent will execute these `initial_actions` *before* the LLM starts reasoning about the main `task`.

2. Handling Sensitive Data Securely

What if your agent needs to log into a site? You definitely don’t want to hardcode your password in the script or even send it directly to a cloud-based LLM.

Browser Use has a clever way to handle this. You can provide sensitive data (like usernames, passwords, API keys needed *on the website*) separately. The LLM will only see placeholder names (like `my_username` or `my_secret_password`) and instruct Browser Use to use the *actual* value associated with that placeholder locally, without the LLM ever knowing the real secret.


# Example sensitive data
sensitive_data = {
    "login_user": "my_actual_username",
    "login_pass": "MySup3rS3cr3tP@ssword!"
}

# When creating the agent:
agent = Agent(
    llm=llm,
    browser=browser_instance,
    # ... other parameters
    sensitive_data=sensitive_data
)

# Your task can refer to the placeholders:
task = "Go to example-login.com, enter 'login_user' in the username field and 'login_pass' in the password field, then click submit."

The LLM sees “use `login_user`”, but your actual username stays local. Check the Sensitive Data documentation for more details. If you’re still concerned, running the LLM locally with Ollama provides maximum privacy.

Responsible Automation: A Quick Note

While incredibly powerful, it’s important to use tools like Browser Use responsibly.

  • Respect Website Terms of Service: Excessive automation or scraping might violate a site’s ToS. Be mindful and avoid overwhelming websites with requests.
  • Security Implications: Giving an AI control over your browser, especially one with logged-in sessions, carries inherent risks. Be cautious about the tasks you assign, especially those involving purchases or sensitive actions. Ensure your API keys and sensitive data are handled securely.
  • Costs: Remember that using cloud-based LLMs incurs costs. Monitor your usage, especially during development and testing.

Wrapping Up: Your Browser’s New Superpowers

And there you have it! We’ve gone from zero to building a functional AI browser agent capable of navigating the web, interacting with pages using your own browser context, and even extracting data into a clean, structured format. I was genuinely impressed by how straightforward Browser Use makes this process.

The possibilities here are vast – automating repetitive online chores, monitoring information, interacting with web applications in sophisticated ways. Whether you use a cloud LLM like Claude or go the local route with Ollama, Browser Use provides a flexible and powerful framework to experiment with.

I encourage you to play around with different tasks, explore the Browser Use documentation further (it’s really quite good!), and see what you can build. The world of AI-powered browser automation is just getting started, and it’s exciting to be able to build these kinds of tools ourselves.

What do you think? What tasks would you automate with an AI browser agent? Let me know your ideas or any questions you have in the comments below!

 

Explore more:

Tags:

Leave a Reply