Jacob Whitmore

What text do machines actually read on a page?

Checked 12 October 2026.

Body text, headings and tables survived in every tool I tested. aria-labels, data- attributes, comments, the meta description and JSON-LD were dropped by every tool.

Before a search engine or an AI tool uses a page, it turns the HTML into plain text. I wanted to know which parts of a page make it through that step, so I tested four free, widely used tools that do the conversion.

The test

I built one test page and put a made-up marker word in 14 different places: body text, a heading, a table, a caption, the footer, image alt text, a link’s title attribute, an aria-label, a data- attribute, an HTML comment, the meta description, JSON-LD structured data, a span hidden with display:none, and a link URL.

Then I ran the page through four Python libraries with their default settings: BeautifulSoup’s get_text(), html2text, markdownify and trafilatura. For each tool I searched the output for each marker. Kept or dropped, nothing in between.

What survived

Where the marker sat BeautifulSoup html2text markdownify trafilatura
Body paragraph Kept Kept Kept Kept
Heading Kept Kept Kept Kept
Table cell Kept Kept Kept Kept
Image caption Kept Kept Kept Dropped
Footer Kept Kept Kept Dropped
Image alt text Dropped Kept Kept Dropped
Link title attribute Dropped Kept Kept Dropped
Link URL Dropped Kept Kept Dropped
Text hidden with display:none Kept Kept Kept Dropped
aria-label Dropped Dropped Dropped Dropped
data- attribute Dropped Dropped Dropped Dropped
HTML comment Dropped Dropped Dropped Dropped
Meta description Dropped Dropped Dropped Dropped
JSON-LD Dropped Dropped Dropped Dropped

Four things stand out.

Keep it in the body text. Paragraphs, headings and table cells were kept by all four tools. Nothing else was.

Five places were dropped every time. aria-label, data- attributes, comments, the meta description and JSON-LD never appeared in the body text from any tool. trafilatura can return the meta description and the JSON-LD headline as separate metadata fields when you ask for them, but not as part of the page’s text.

Alt text depends on the tool. Two tools kept it, written as Markdown image syntax. Two dropped it.

“Hidden” isn’t hidden from a parser. Three of the four tools kept text hidden with display:none. They read the HTML and don’t apply the page’s styles.

What this means

This part is my reasoning, not something I measured.

If a fact matters (a price, a date, a name, a definition), put it in visible text on the page. Don’t rely on alt text, metadata or markup to carry it. Those may help with other jobs, but in this test they were the parts most likely to be lost.

What I will not claim

  • What ChatGPT, Google or any other engine actually uses. I don’t know their pipelines. I tested four open-source tools that are common building blocks.
  • Anything beyond default settings. Most of these tools have options that keep more.
  • Anything about pages that need JavaScript, or text inside images. My test page was static HTML.
  • Anything beyond one test page on one date.

Data

  • html-extraction-test-2026-10-12.csv: one row per marker and tool, with tool versions.
  • The test page and the script are below, so you can rerun the test in a few minutes. Save the page as page.html in the same folder as the script.

Tool versions: beautifulsoup4 4.15.0, html2text 2025.4.15, markdownify 1.2.3, trafilatura 2.3.1.

The test page

<!doctype html>
<html lang="en-GB"><head>
<meta charset="utf-8">
<title>Test page: where does the text go</title>
<meta name="description" content="Marker zulu-meta-description sits in the meta description.">
<script type="application/ld+json">{"@context":"https://schema.org","@type":"Article","headline":"Marker yankee-json-ld sits in structured data","author":{"@type":"Person","name":"Test Author"}}</script>
</head><body>
<header><nav><a href="/">Home</a> <a href="/about/" title="Marker xray-title-attribute">About</a> <a href="https://www.linkedin.com/" aria-label="Marker whiskey-aria-label"><svg width="16" height="16"></svg></a></nav></header>
<main><article>
<h1>Where does the text on a page go when a machine reads it</h1>
<p>This page is a test. Each marker below sits in a different part of the HTML. The visible paragraph marker is victor-visible-paragraph and it sits in plain body text that any reader can see on screen.</p>
<h2>Marker uniform-subheading in a heading</h2>
<p>Search engines and AI tools turn HTML into text before they use it. Different tools keep different parts of the page. This test checks which parts survive the conversion and which parts are dropped along the way.</p>
<figure><img src="chart.png" alt="Marker tango-alt-text in image alt text" width="600" height="300"><figcaption>Marker sierra-figcaption in a visible caption.</figcaption></figure>
<p>The next paragraph contains a span that is hidden with an inline style. <span style="display:none">Marker romeo-hidden-span is hidden with display none.</span> The sentence carries on after it so the paragraph still reads normally.</p>
<!-- Marker quebec-html-comment sits in an HTML comment. -->
<div data-note="Marker papa-data-attribute sits in a data attribute"><p>A data attribute is attached to the block that holds this paragraph. Readers never see it, but it is part of the HTML that a crawler downloads.</p></div>
<table><tr><th>Part of page</th><th>Marker</th></tr><tr><td>Table cell</td><td>oscar-table-cell</td></tr></table>
<p>The last paragraph links to <a href="https://example.com/november-link-url">a page whose address holds a marker</a>, so the test can see whether link addresses are kept or only the link text.</p>
<p>That is the end of the test content. Every marker is a made-up word pair, so a search of the output for each one gives a clean yes or no.</p>
</article></main>
<footer><p>Footer text marker mike-footer.</p></footer>
</body></html>

The script

import json, csv, sys
from bs4 import BeautifulSoup
import html2text, markdownify, trafilatura, bs4
html=open('page.html',encoding='utf-8').read()
markers=["victor-visible-paragraph","uniform-subheading","oscar-table-cell","sierra-figcaption","mike-footer","tango-alt-text","xray-title-attribute","whiskey-aria-label","papa-data-attribute","quebec-html-comment","zulu-meta-description","yankee-json-ld","romeo-hidden-span","november-link-url"]
outs={}
outs["BeautifulSoup get_text()"]=BeautifulSoup(html,"lxml").get_text(" ")
h=html2text.HTML2Text(); outs["html2text (defaults)"]=h.handle(html)
outs["markdownify (defaults)"]=markdownify.markdownify(html)
outs["trafilatura extract() (defaults)"]=trafilatura.extract(html) or ""
for k,v in outs.items(): open(k.split()[0]+'.txt','w').write(v)
rows=[]
w=csv.writer(sys.stdout)
w.writerow(["marker","where"]+list(outs))
for m in markers:
    w.writerow([m.split('-',1)[0], m.split('-',1)[1]]+[("kept" if m in v else "dropped") for v in outs.values()])