How to scrape Twitter (x.com) without authentication using Gaffa
How you can use Gaffa’s capture_dom action to scrape public tweets without an X account, and save the text, timestamps and engagement counts as JSON.
Oct 23, 2026
Scraping a few tweets sounds simple until you hit a login prompt or your script returns a page without the posts you need. Because X (formerly Twitter) loads its content with JavaScript, a basic request often isn't enough, and you end up setting up a browser just to read a single post. With Gaffa's capture_dom action, you can load a public X profile in a real browser through one API call, with no login and no local browser. In this guide, you'll use Python and Gaffa's capture_dom to extract tweet text and engagement counts from real posts and save the results as JSON.
Can you scrape Twitter without an account?
Yes, you can scrape a public X post without logging in, but the page usually has to be rendered in a real browser first. A plain request may return a login page or incomplete content, whereas a browser runs the JavaScript needed to load the post.
X also offers an official API, but it requires an X account, developer credentials, and pay-per-use pricing with request limits, so it isn't an account-free option. Gaffa loads the public page for you without an X username, password, or saved session.
You're still limited to what X shows while signed out. A public profile may show only a few posts before asking you to log in, so the results may not include its full history.
Prerequisites
Before building the scraper, you'll need the following.
- Python 3.10 or newer
- A Gaffa API key
- A terminal and code editor
- The requests and beautifulsoup4 packages
You won't need to install Chrome, Selenium, or Playwright, since Gaffa handles the browser work for you.
How to scrape Twitter with Gaffa and Python
We'll build a scraper that loads a public X profile, finds the available tweet links, and saves each post's data as JSON. We'll use OpenAI's X account, but you can swap in any other public profile. You can follow along with the finished project in our GitHub repository.
Set up your Python project
Start by creating a project folder and a virtual environment to keep its packages separate from your other projects. On macOS or Linux, run the following.
Creating the Project Folder and Virtual Environment
On Windows PowerShell, use python instead of python3 and activate the environment with
Next, install requests to make the API calls and beautifulsoup4 to read the HTML that Gaffa returns.
Installing the Packages
Create a file named
xscraper.py and add these imports and settings.Importing Packages and Defining Settings
PROFILE is the page we'll scrape, while LIMIT caps the number of tweets we process, and we're starting with three so the first run stays small. The script reads your key from GAFFA_API_KEY or asks you to enter it, so you never have to expose it in the code.Load a public X profile with Gaffa
We'll use capture_dom to get the page's rendered HTML and pass it to BeautifulSoup for extraction. Add the following function, which we'll reuse for the profile and for each tweet page.
Fetching a Page with capture_dom
The wait action holds the request for up to 15 seconds until an
article element appears, which gives the content a chance to load before capture_dom runs. Setting async to False makes the call wait for its result, and we also use a US proxy and cap media downloads at 1 MB, since we only need text.The final action returns a file URL for the captured HTML, so the function downloads it, saves a local copy, and returns a BeautifulSoup object. It also saves the raw API response so you can inspect any errors.
Now call the function to load the profile.
Loading the Profile
On a successful request, this creates
twitter_data/profile.json and twitter_data/profile.html.Find the available tweet URLs
With the profile loaded, we can look through its articles for tweet links that belong to the selected account.
Finding Tweet URLs
This works with the HTML we already downloaded, so it makes no additional API requests. Keep in mind that
LIMIT only sets a maximum, so raising it won't reveal posts that aren't in the captured page.Extract tweet text, timestamps, and engagement counts
Next, we'll define how to read an individual tweet page. We identify the requested post by its timestamp link, which stops us from accidentally collecting a reply beneath it. First, add a small helper for reading engagement counts.
Reading Engagement Counts
It returns the count as displayed on the page, or None when no count is available, so a value like "6.9K" stays as displayed instead of becoming an assumed exact number. Now add the function that extracts the tweet.
Extracting a Tweet
Extracting a Tweet
The function returns the tweet's ID, URL, author, publication time, text, links, and engagement counts. It also checks for missing content and a "Show more" button, so a shortened preview is never saved as a complete post.
Rather than parsing the date shown on the page, which changes with language and time zone, the function reads the publish time from the tweet ID itself. X builds every post ID from the moment it was created, so shifting the ID right by 22 bits and adding X's epoch (1288834974657 ms) gives the exact UTC timestamp, down to the second. The parser still depends on the HTML structure reflected in its selectors, so if extraction fails, compare those selectors with the saved HTML before changing the code.
Save the tweets as JSON
Now add the block that opens each URL with
fetch_page(), extracts its data with extract_post(), and collects the results.Processing Each Tweet
Each tweet is processed separately, so if a request or extraction fails, the script records the URL and error and carries on with the remaining posts. Finally, sort the successful records by date and write both lists to disk.
Saving the Results
posts.json contains the extracted tweets with the newest first, while failures.json lists any URLs that couldn't be processed along with their error messages. You can find the full code on our GitHub repository.Run the scraper and check the results
Run the script from your project folder.
Running the Scraper
If it prompts for your Gaffa API key, paste it and press Enter. The terminal will report how many tweet URLs it found and whether each post was extracted, and once it finishes, open
twitter_data/posts.json in your code editor to see the results.The extracted tweets in posts.json, sorted with the newest post first.
Why use Gaffa instead of the X API?
The official X API is the better fit when you need supported, structured access to X data, but it needs developer credentials, comes with its own authentication per endpoint, and enforces strict usage limits. For a workflow like this one, Gaffa is simpler because it works with the public web page itself, handling JavaScript rendering, browser infrastructure, and proxy routing behind one REST API.
Scraping public tweets doesn't have to mean fighting login walls or managing a browser yourself. With Gaffa's capture_dom action, you can load a real X page, extract the data you need, and save it as clean JSON in a few lines of Python. Sign up for free to grab your 500 free credits and run your first request in minutes, or see how to scrape a table with Python for another take on the same pattern.
Frequently asked questions
How do I scrape Twitter without an X account using Gaffa?
Use capture_dom to load a public profile or tweet in Gaffa's browser, then parse the returned HTML with BeautifulSoup. No X username, password, or saved session is needed for posts visible while signed out.
Do I need an X API key to scrape tweets with Gaffa?
No. This method uses Gaffa's browser API instead of the official X API, so you only need a Gaffa API key to make the requests.
How do I scrape a different public X profile?
Change the
PROFILE URL in the script to the account you want. Make sure its posts are visible while logged out, and check that the captured HTML still matches the selectors in extract_post().How do I scrape X without Selenium or Playwright?
Send your request to Gaffa's
/v1/browser/requests endpoint with wait and capture_dom actions. Gaffa runs the browser for you, so there's no Selenium, ChromeDriver, or local headless browser to install.How do I get more than a few tweets from a profile?
Raising LIMIT only helps if the captured page contains more links. Signed-out profiles often show just a few posts, so the results are limited to what X makes available without a login.
