how to cite websites in papers

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Citing sources
• How to Cite a Website | MLA, APA & Chicago Examples

How to Cite a Website | MLA, APA & Chicago Examples

Published on March 5, 2021 by Jack Caulfield . Revised on January 17, 2024.

To cite a page from a website, you need a short in-text citation and a corresponding reference stating the author’s name, the date of publication, the title of the page, the website name, and the URL.

This information is presented differently in different citation styles. APA , MLA , and Chicago are the most commonly used styles.

Use the interactive example generator below to explore APA and MLA website citations.

Note that the format is slightly different for citing YouTube and other online video platforms, or for citing an image .

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

Citing a website in mla style, citing a website in apa style, citing a website in chicago style, frequently asked questions about citations.

An MLA Works Cited entry for a webpage lists the author’s name , the title of the page (in quotation marks), the name of the site (in italics), the date of publication, and the URL.

The in-text citation usually just lists the author’s name. For a long page, you may specify a (shortened) section heading to locate the specific passage. Don’t use paragraph numbers unless they’re specifically numbered on the page.

The same format is used for blog posts and online articles from newspapers and magazines.

You can also use our free MLA Citation Generator to generate your website citations.

Generate accurate MLA citations with Scribbr

Citing a whole website.

When you cite an entire website rather than a specific page, include the author if one can be identified for the whole site (e.g. for a single-authored blog). Otherwise, just start with the site name.

List the copyright date displayed on the site; if there isn’t one, provide an access date after the URL.

Webpages with no author or date

When no author is listed, cite the organization as author only if it differs from the website name.

If the organization name is also the website name, start the Works Cited entry with the title instead, and use a shortened version of the title in the in-text citation.

When no publication date is listed, leave it out and include an access date at the end instead.

Prevent plagiarism. Run a free check.

An APA reference for a webpage lists the author’s last name and initials, the full date of publication, the title of the page (in italics), the website name (in plain text), and the URL.

The in-text citation lists the author’s last name and the year. If it’s a long page, you may include a locator to identify the quote or paraphrase (e.g. a paragraph number and/or section title).

Note that a general reference to an entire website doesn’t require a citation in APA Style; just include the URL in parentheses after you mention the site.

You can also use our free APA Citation Generator to create your webpage citations. Search for a URL to retrieve the details.

Generate accurate APA citations with Scribbr

Blog posts and online articles.

Blog posts follow a slightly different format: the title of the post is not italicized, and the name of the blog is.

The same format is used for online newspaper and magazine articles—but not for articles from news sites like Reuters and BBC News (see the previous example).

When a page has no author specified, list the name of the organization that created it instead (and omit it later if it’s the same as the website name).

When it doesn’t list a date of publication, use “n.d.” in place of the date. You can also include an access date if the page seems likely to change over time.

In Chicago notes and bibliography style, footnotes are used to cite sources. They refer to a bibliography at the end that lists all your sources in full.

A Chicago bibliography entry for a website lists the author’s name, the page title (in quotation marks), the website name, the publication date, and the URL.

Chicago also has an alternative author-date citation style . Examples of website citations in this style can be found here .

For blog posts and online articles from newspapers, the name of the publication is italicized. For a blog post, you should also add the word “blog” in parentheses, unless it’s already part of the blog’s name.

When a web source doesn’t list an author , you can usually begin your bibliography entry and short note with the name of the organization responsible. Don’t repeat it later if it’s also the name of the website. A full note should begin with the title instead.

When no publication or revision date is shown, include an access date instead in your bibliography entry.

The main elements included in website citations across APA , MLA , and Chicago style are the author, the date of publication, the page title, the website name, and the URL. The information is presented differently in each style.

In APA , MLA , and Chicago style citations for sources that don’t list a specific author (e.g. many websites ), you can usually list the organization responsible for the source as the author.

If the organization is the same as the website or publisher, you shouldn’t repeat it twice in your reference:

• In APA and Chicago, omit the website or publisher name later in the reference.
• In MLA, omit the author element at the start of the reference, and cite the source title instead.

If there’s no appropriate organization to list as author, you will usually have to begin the citation and reference entry with the title of the source instead.

When you want to cite a specific passage in a source without page numbers (e.g. an e-book or website ), all the main citation styles recommend using an alternate locator in your in-text citation . You might use a heading or chapter number, e.g. (Smith, 2016, ch. 1)

In APA Style , you can count the paragraph numbers in a text to identify a location by paragraph number. MLA and Chicago recommend that you only use paragraph numbers if they’re explicitly marked in the text.

For audiovisual sources (e.g. videos ), all styles recommend using a timestamp to show a specific point in the video when relevant.

Check if your university or course guidelines specify which citation style to use. If the choice is left up to you, consider which style is most commonly used in your field.

• APA Style is the most popular citation style, widely used in the social and behavioral sciences.
• MLA style is the second most popular, used mainly in the humanities.
• Chicago notes and bibliography style is also popular in the humanities, especially history.
• Chicago author-date style tends to be used in the sciences.

Other more specialized styles exist for certain fields, such as Bluebook and OSCOLA for law.

The most important thing is to choose one style and use it consistently throughout your text.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Caulfield, J. (2024, January 17). How to Cite a Website | MLA, APA & Chicago Examples. Scribbr. Retrieved April 15, 2024, from https://www.scribbr.com/citing-sources/cite-a-website/

Is this article helpful?

Jack Caulfield

Other students also liked, how to cite an image | photographs, figures, diagrams, how to cite a lecture | apa, mla & chicago examples, how to cite a youtube video | mla, apa & chicago, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

• Plagiarism and grammar
• Citation guides

Cite a Website

Don't let plagiarism errors spoil your paper, citing a website in apa.

Once you’ve identified a credible website to use, create a citation and begin building your reference list. Citation Machine citing tools can help you create references for online news articles, government websites, blogs, and many other website! Keeping track of sources as you research and write can help you stay organized and ethical. If you end up not using a source, you can easily delete it from your bibliography. Ready to create a citation? Enter the website’s URL into the search box above. You’ll get a list of results, so you can identify and choose the correct source you want to cite. It’s that easy to begin!

If you’re wondering how to cite a website in APA, use the structure below.

Author Last Name, First initial. (Year, Month Date Published). Title of web page . Name of Website. URL

Example of an APA format website:

Austerlitz, S. (2015, March 3). How long can a spinoff like ‘Better Call Saul’ last? FiveThirtyEight. http://fivethirtyeight.com/features/how-long-can-a-spinoff-like-better-call-saul-last/

Keep in mind that not all information found on a website follows the structure above. Only use the Website format above if your online source does not fit another source category. For example, if you’re looking at a video on YouTube, refer to the ‘YouTube Video’ section. If you’re citing a newspaper article found online, refer to ‘Newspapers Found Online’ section. Again, an APA website citation is strictly for web pages that do not fit better with one of the other categories on this page.

Social media:

When adding the text of a post, keep the original capitalization, spelling, hashtags, emojis (if possible), and links within the text.

Facebook posts:

Structure: Facebook user’s Last name, F. M. (Year, Monday Day of Post). Up to the first 20 words of Facebook post [Source type if attached] [Post type]. Facebook. URL

Source type examples: [Video attached], [Image attached]

Post type examples: [Status update], [Video], [Image], [Infographic]

Gomez, S. (2020, February 4). Guys, I’ve been working on this special project for two years and can officially say Rare Beauty is launching in [Video]. Facebook. https://www.facebook.com/Selena/videos/1340031502835436/

Life at Chegg. (2020, February 7) It breaks our heart that 50% of college students right here in Silicon Valley are hungry. That’s why Chegg has [Images attached] [Status update]. Facebook. https://www.facebook.com/LifeAtChegg/posts/1076718522691591

Twitter posts:

Structure: Account holder’s Last name, F. M. [Twitter Handle]. (Year, Month Day of Post). Up to the first 20 words of tweet [source type if attached] [Tweet]. Twitter. URL

Source type examples: [Video attached], [Image attached], [Poll attached]

Example: Edelman, J. [Edelman11]. (2018, April 26). Nine years ago today my life changed forever. New England took a chance on a long shot and I’ve worked [Video attached] [Tweet]. Twitter. https://twitter.com/Edelman11/status/989652345922473985

Instagram posts:

APA citation format: Account holder’s Last name, F. M. [@Instagram handle]. (Year, Month Day). Up to the first 20 words of caption [Photograph(s) and/or Video(s)]. Instagram. URL

Example: Portman, N. [@natalieportman]. (2019, January 5). Many of my best experiences last year were getting to listen to and learn from so many incredible people through [Videos]. Instagram. https://www.instagram.com/p/BsRD-FBB8HI/?utm_source=ig_web_copy_link

If this guide hasn’t helped solve all of your referencing questions, or if you’re still feeling the need to type “how to cite a website APA” into Google, then check out our APA citation generator on CitationMachine.com, which can build your references for you!

Featured links:

APA Citation Generator  |  Website   | Books | Journal Articles | YouTube | Images | Movies | Interview | PDF

• Citation Machine® Plus
• Citation Guides
• Chicago Style
• Harvard Referencing
• Terms of Use
• Global Privacy Policy
• Cookie Notice
• DO NOT SELL MY INFO

• Bibliography Answers

How to cite a website in APA, MLA, or Harvard style

There are many different ways to cite a website, depending on which citation style you need to format it in.

 The easy way to cite a website in any citation style

Use our citation generator below to automatically cite a website in any style, including APA, MLA 7 and 8, and Harvard. Just select the style you need, copy the URL into the search box, and press search. We’ll do the rest.

 The manual way to cite a website

To cite a website by hand just follow the instructions below. For the 3 most popular styles–APA, MLA 8, and Harvard–this is as follows:

 In APA style

You need to locate these details for the website: page or article author, page or article title, website name, published date, access date, page URL (web address) .

• The author can typically be found on the page, but if there isn’t one listed you can use the website name in its place.
• The page title can be found near the top of the page, and you can also find it by hovering your mouse over the browser tab.
• The website name can usually be found in the web address or by looking for a logo or similar at the very top of the page.
• There often isn’t a publish date , but if there is it’ll be very close to the page title.
• The access date is the date you took information from the article (usually today).
• The page URL can be copied straight from the address bar of your browser and will start with either http:// or https://.

Then use this template, replacing the colored placeholders with the information you found on the page:

Author last name , author first name initial . ( published year , published month and day ). Page title . Retrieved accessed month and day , accessed year , from article URL .

The final formatted citation should look like this:

Ingle, S. (2018, February 11). Winter Olympics was hit by cyber-attack, officials confirm. Retrieved July 24, 2018, from https://www.theguardian.com/sport/2018/feb/11/winter-olympics-was-hit-by-cyber-attack-officials-confirm.

For a more comprehensive guide, including what to do when you can’t find certain details, have a look at our more in-depth guide to citing a website in APA format .

 In MLA 8 style

Here are the specific details you need to find on the page: page or article author, page or article title, website name, published date, access date, page URL (web address) .

Then use this template:

Author last name , author first name . “ Page title .” website name , published date day, month, year , page URL . Accessed accessed date day, month, year .

Ingle, Sean. “Winter Olympics Was Hit by Cyber-Attack, Officials Confirm.” The Guardian , 11 Feb. 2018, https://www.theguardian.com/sport/2018/feb/11/winter-olympics-was-hit-by-cyber-attack-officials-confirm. Accessed 13 July 2018.

For a more comprehensive guide, including what to do when you can’t find certain details, have a look at our more in-depth guide to citing a website in MLA 8 format .

 In Harvard style

First, find these details for the website: page or article author, page or article title, website name, published date, access date, page URL (web address) .

Author last name , author firstname initial ( published date year ). Page title . [online] website name . Available at: page URL [Accessed accessed date day, month, year ].

Ingle, S. (2018). Winter Olympics was hit by cyber-attack, officials confirm . [online] The Guardian. Available at: https://www.theguardian.com/sport/2018/feb/11/winter-olympics-was-hit-by-cyber-attack-officials-confirm [Accessed 13 Jul. 2018].

Daniel is a qualified librarian, former teacher, and citation expert. He has been contributing to MyBib since 2018.

Home / Guides / Citation Guides / How to Cite Sources

How to Cite Sources

Here is a complete list for how to cite sources. Most of these guides present citation guidance and examples in MLA, APA, and Chicago.

If you’re looking for general information on MLA or APA citations , the EasyBib Writing Center was designed for you! It has articles on what’s needed in an MLA in-text citation , how to format an APA paper, what an MLA annotated bibliography is, making an MLA works cited page, and much more!

MLA Format Citation Examples

The Modern Language Association created the MLA Style, currently in its 9th edition, to provide researchers with guidelines for writing and documenting scholarly borrowings.  Most often used in the humanities, MLA style (or MLA format ) has been adopted and used by numerous other disciplines, in multiple parts of the world.

MLA provides standard rules to follow so that most research papers are formatted in a similar manner. This makes it easier for readers to comprehend the information. The MLA in-text citation guidelines, MLA works cited standards, and MLA annotated bibliography instructions provide scholars with the information they need to properly cite sources in their research papers, articles, and assignments.

• Book Chapter
• Conference Paper
• Documentary
• Encyclopedia
• Google Images
• Kindle Book
• Memorial Inscription
• Museum Exhibit
• Painting or Artwork
• PowerPoint Presentation
• Sheet Music
• Thesis or Dissertation
• YouTube Video

APA Format Citation Examples

The American Psychological Association created the APA citation style in 1929 as a way to help psychologists, anthropologists, and even business managers establish one common way to cite sources and present content.

APA is used when citing sources for academic articles such as journals, and is intended to help readers better comprehend content, and to avoid language bias wherever possible. The APA style (or APA format ) is now in its 7th edition, and provides citation style guides for virtually any type of resource.

Chicago Style Citation Examples

The Chicago/Turabian style of citing sources is generally used when citing sources for humanities papers, and is best known for its requirement that writers place bibliographic citations at the bottom of a page (in Chicago-format footnotes ) or at the end of a paper (endnotes).

The Turabian and Chicago citation styles are almost identical, but the Turabian style is geared towards student published papers such as theses and dissertations, while the Chicago style provides guidelines for all types of publications. This is why you’ll commonly see Chicago style and Turabian style presented together. The Chicago Manual of Style is currently in its 17th edition, and Turabian’s A Manual for Writers of Research Papers, Theses, and Dissertations is in its 8th edition.

Citing Specific Sources or Events

• Declaration of Independence
• Gettysburg Address
• Martin Luther King Jr. Speech
• President Obama’s Farewell Address
• President Trump’s Inauguration Speech
• White House Press Briefing

Additional FAQs

• Citing Archived Contributors
• Citing a Blog
• Citing a Book Chapter
• Citing a Source in a Foreign Language
• Citing an Image
• Citing a Song
• Citing Special Contributors
• Citing a Translated Article
• Citing a Tweet

6 Interesting Citation Facts

The world of citations may seem cut and dry, but there’s more to them than just specific capitalization rules, MLA in-text citations , and other formatting specifications. Citations have been helping researches document their sources for hundreds of years, and are a great way to learn more about a particular subject area.

Ever wonder what sets all the different styles apart, or how they came to be in the first place? Read on for some interesting facts about citations!

1. There are Over 7,000 Different Citation Styles

You may be familiar with MLA and APA citation styles, but there are actually thousands of citation styles used for all different academic disciplines all across the world. Deciding which one to use can be difficult, so be sure to ask you instructor which one you should be using for your next paper.

2. Some Citation Styles are Named After People

While a majority of citation styles are named for the specific organizations that publish them (i.e. APA is published by the American Psychological Association, and MLA format is named for the Modern Language Association), some are actually named after individuals. The most well-known example of this is perhaps Turabian style, named for Kate L. Turabian, an American educator and writer. She developed this style as a condensed version of the Chicago Manual of Style in order to present a more concise set of rules to students.

3. There are Some Really Specific and Uniquely Named Citation Styles

How specific can citation styles get? The answer is very. For example, the “Flavour and Fragrance Journal” style is based on a bimonthly, peer-reviewed scientific journal published since 1985 by John Wiley & Sons. It publishes original research articles, reviews and special reports on all aspects of flavor and fragrance. Another example is “Nordic Pulp and Paper Research,” a style used by an international scientific magazine covering science and technology for the areas of wood or bio-mass constituents.

4. More citations were created on  EasyBib.com  in the first quarter of 2018 than there are people in California.

The US Census Bureau estimates that approximately 39.5 million people live in the state of California. Meanwhile, about 43 million citations were made on EasyBib from January to March of 2018. That’s a lot of citations.

5. “Citations” is a Word With a Long History

The word “citations” can be traced back literally thousands of years to the Latin word “citare” meaning “to summon, urge, call; put in sudden motion, call forward; rouse, excite.” The word then took on its more modern meaning and relevance to writing papers in the 1600s, where it became known as the “act of citing or quoting a passage from a book, etc.”

6. Citation Styles are Always Changing

The concept of citations always stays the same. It is a means of preventing plagiarism and demonstrating where you relied on outside sources. The specific style rules, however, can and do change regularly. For example, in 2018 alone, 46 new citation styles were introduced , and 106 updates were made to exiting styles. At EasyBib, we are always on the lookout for ways to improve our styles and opportunities to add new ones to our list.

Why Citations Matter

Here are the ways accurate citations can help your students achieve academic success, and how you can answer the dreaded question, “why should I cite my sources?”

They Give Credit to the Right People

Citing their sources makes sure that the reader can differentiate the student’s original thoughts from those of other researchers. Not only does this make sure that the sources they use receive proper credit for their work, it ensures that the student receives deserved recognition for their unique contributions to the topic. Whether the student is citing in MLA format , APA format , or any other style, citations serve as a natural way to place a student’s work in the broader context of the subject area, and serve as an easy way to gauge their commitment to the project.

They Provide Hard Evidence of Ideas

Having many citations from a wide variety of sources related to their idea means that the student is working on a well-researched and respected subject. Citing sources that back up their claim creates room for fact-checking and further research . And, if they can cite a few sources that have the converse opinion or idea, and then demonstrate to the reader why they believe that that viewpoint is wrong by again citing credible sources, the student is well on their way to winning over the reader and cementing their point of view.

They Promote Originality and Prevent Plagiarism

The point of research projects is not to regurgitate information that can already be found elsewhere. We have Google for that! What the student’s project should aim to do is promote an original idea or a spin on an existing idea, and use reliable sources to promote that idea. Copying or directly referencing a source without proper citation can lead to not only a poor grade, but accusations of academic dishonesty. By citing their sources regularly and accurately, students can easily avoid the trap of plagiarism , and promote further research on their topic.

They Create Better Researchers

By researching sources to back up and promote their ideas, students are becoming better researchers without even knowing it! Each time a new source is read or researched, the student is becoming more engaged with the project and is developing a deeper understanding of the subject area. Proper citations demonstrate a breadth of the student’s reading and dedication to the project itself. By creating citations, students are compelled to make connections between their sources and discern research patterns. Each time they complete this process, they are helping themselves become better researchers and writers overall.

When is the Right Time to Start Making Citations?

Make in-text/parenthetical citations as you need them.

As you are writing your paper, be sure to include references within the text that correspond with references in a works cited or bibliography. These are usually called in-text citations or parenthetical citations in MLA and APA formats. The most effective time to complete these is directly after you have made your reference to another source. For instance, after writing the line from Charles Dickens’ A Tale of Two Cities : “It was the best of times, it was the worst of times…,” you would include a citation like this (depending on your chosen citation style):

(Dickens 11).

This signals to the reader that you have referenced an outside source. What’s great about this system is that the in-text citations serve as a natural list for all of the citations you have made in your paper, which will make completing the works cited page a whole lot easier. After you are done writing, all that will be left for you to do is scan your paper for these references, and then build a works cited page that includes a citation for each one.

Need help creating an MLA works cited page ? Try the MLA format generator on EasyBib.com! We also have a guide on how to format an APA reference page .

2. Understand the General Formatting Rules of Your Citation Style Before You Start Writing

While reading up on paper formatting may not sound exciting, being aware of how your paper should look early on in the paper writing process is super important. Citation styles can dictate more than just the appearance of the citations themselves, but rather can impact the layout of your paper as a whole, with specific guidelines concerning margin width, title treatment, and even font size and spacing. Knowing how to organize your paper before you start writing will ensure that you do not receive a low grade for something as trivial as forgetting a hanging indent.

Don’t know where to start? Here’s a formatting guide on APA format .

3. Double-check All of Your Outside Sources for Relevance and Trustworthiness First

Collecting outside sources that support your research and specific topic is a critical step in writing an effective paper. But before you run to the library and grab the first 20 books you can lay your hands on, keep in mind that selecting a source to include in your paper should not be taken lightly. Before you proceed with using it to backup your ideas, run a quick Internet search for it and see if other scholars in your field have written about it as well. Check to see if there are book reviews about it or peer accolades. If you spot something that seems off to you, you may want to consider leaving it out of your work. Doing this before your start making citations can save you a ton of time in the long run.

Finished with your paper? It may be time to run it through a grammar and plagiarism checker , like the one offered by EasyBib Plus. If you’re just looking to brush up on the basics, our grammar guides  are ready anytime you are.

How useful was this post?

Click on a star to rate it!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Citation Basics

Harvard Referencing

Plagiarism Basics

Plagiarism Checker

Upload a paper to check for plagiarism against billions of sources and get advanced writing suggestions for clarity and style.

Get Started

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Referencing
• Reference a Website in Harvard Style | Templates & Examples

Reference a Website in Harvard Style | Templates & Examples

Published on 19 May 2020 by Jack Caulfield . Revised on 7 November 2022.

To reference a website in Harvard style , include the name of the author or organization, the year of publication, the title of the page, the URL, and the date on which you accessed the website.

Different formats are used for other kinds of online source, such as articles, social media posts and multimedia content. You can generate accurate Harvard references for all kinds of sources with our free reference generator:

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Online articles, social media posts, images, videos and podcasts, referencing websites with missing information, frequently asked questions about harvard website references.

Blog posts and online newspaper articles are both referenced in the same format: include the title of the article in quotation marks, the name of the blog or newspaper in italics, and the date of publication.

The format for a magazine article is slightly different. Instead of a precise date, include the month, season, or volume and issue number, depending on what the magazine uses to identify its issues.

The URL and access date information are included only when the article is online-exclusive.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

To reference posts from social media platforms like Facebook, Twitter and Instagram, include the username and the platform in square brackets. Write usernames the way they appear on the platform, with the same capitalization and symbols.

If the post has a title, use it (in quotation marks). If the post is untitled, use the text of the post instead. Do not use italics. If the text is long, you can replace some of it with an ellipsis.

Online content is referenced differently if it is in video, audio or image form.

To cite an image found online, such as an artwork, photograph, or infographic, include the image format (e.g. ‘Photograph’, ‘Oil on canvas’) in square brackets.

Online videos, such as those on YouTube, Instagram, Vimeo and Dailymotion, are cited similarly to general web pages. Where a video is uploaded under the name of an individual, write the name in the usual format. Otherwise, write the username of the uploader as it appears on the site.

If you want to locate a specific point in a video in an in-text citation, you can do so using a timestamp.

For a podcast reference, you just need the name of the individual episode, not of the whole series. The word ‘Podcast’ is always included in square brackets. As with videos, you can use a timestamp to locate a specific point in the in-text citation.

Online sources are often missing information you would usually need for a citation: author, title or date. Here’s what to do when these details are not available.

When a website doesn’t list a specific individual author, you can usually find a corporate author to list instead. This is the organisation responsible for the source:

In cases where there’s no suitable corporate author (such as online dictionaries or Wikis), use the title of the source in the author position instead:

In Harvard style, when a source doesn’t list a specific date of publication, replace it with the words ‘no date’ in both the in-text citation and the reference list. You should still include an access date:

Prevent plagiarism, run a free check.

It’s important to assess the reliability of information found online. Look for sources from established publications and institutions with expertise (e.g. peer-reviewed journals and government agencies).

The CRAAP test (currency, relevance, authority, accuracy, purpose) can aid you in assessing sources, as can our list of credible sources . You should generally avoid citing websites like Wikipedia that can be edited by anyone – instead, look for the original source of the information in the “References” section.

You can generally omit page numbers in your in-text citations of online sources which don’t have them. But when you quote or paraphrase a specific passage from a particularly long online source, it’s useful to find an alternate location marker.

For text-based sources, you can use paragraph numbers (e.g. ‘para. 4’) or headings (e.g. ‘under “Methodology”’). With video or audio sources, use a timestamp (e.g. ‘10:15’).

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

A Harvard in-text citation should appear in brackets every time you quote, paraphrase, or refer to information from a source.

The citation can appear immediately after the quotation or paraphrase, or at the end of the sentence. If you’re quoting, place the citation outside of the quotation marks but before any other punctuation like a comma or full stop.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2022, November 07). Reference a Website in Harvard Style | Templates & Examples. Scribbr. Retrieved 15 April 2024, from https://www.scribbr.co.uk/referencing/harvard-website-reference/

Is this article helpful?

Jack Caulfield

Other students also liked, a quick guide to harvard referencing | citation examples, harvard style bibliography | format & examples, harvard in-text citation | a complete guide & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

• PRO Courses Guides New Tech Help Pro Expert Videos About wikiHow Pro Upgrade Sign In
• EDIT Edit this Article
• EXPLORE Tech Help Pro About Us Random Article Quizzes Request a New Article Community Dashboard This Or That Game Popular Categories Arts and Entertainment Artwork Books Movies Computers and Electronics Computers Phone Skills Technology Hacks Health Men's Health Mental Health Women's Health Relationships Dating Love Relationship Issues Hobbies and Crafts Crafts Drawing Games Education & Communication Communication Skills Personal Development Studying Personal Care and Style Fashion Hair Care Personal Hygiene Youth Personal Care School Stuff Dating All Categories Arts and Entertainment Finance and Business Home and Garden Relationship Quizzes Cars & Other Vehicles Food and Entertaining Personal Care and Style Sports and Fitness Computers and Electronics Health Pets and Animals Travel Education & Communication Hobbies and Crafts Philosophy and Religion Work World Family Life Holidays and Traditions Relationships Youth
• Browse Articles
• Learn Something New
• Quizzes Hot
• This Or That Game New
• Train Your Brain
• Explore More
• Support wikiHow
• About wikiHow
• Log in / Sign up
• Education and Communications
• College University and Postgraduate
• Academic Writing

How to Cite a Website in APA

Last Updated: January 31, 2023 Fact Checked

This article was co-authored by Cara Barker, MA . Cara Barker is an Assistant Professor and Research and Instruction Librarian at Hunter Library at Western Carolina University. She received her Masters in Library and Information Sciences from the University of Washington in 2014. She has over 16 years of experience working with libraries across the United States. There are 10 references cited in this article, which can be found at the bottom of the page. This article has been fact-checked, ensuring the accuracy of any cited facts and confirming the authority of its sources. This article has been viewed 175,255 times.

If you need to cite a standard webpage, a blog post , a book that isn't published in print, or a forum post in APA format, you've come to the right place. All you have to do is follow a few simple steps to order and format the information correctly. Keep in mind that books, articles, and magazines published online should be cited in the same way as printed books, articles , and magazines.

Website or Blog

• Wilcox, M. & Smith, R.

• Doe, J. (2012, December 31).
• Wilcox, M. & Smith, R. (2010, May 1).

• Doe, J. (2012, December 31). Statistics and analysis.
• Wilcox, M. & Smith, R. (2010, May 1). Study about citation rules.
• Doe, J. (2012, December 31). Statistics and analysis [Web page].
• Wilcox, M. & Smith, R. (2010, May 1). Study about citation rules. [Blog post].
• Doe, J. (2012, December 31). Statistics and analysis [Web page]. Retrieved from http://www.onlinestats.com/12312012/analysisofstats
• Wilcox, M. & Smith, R. (2010, May 1). Study about citation rules. [Blog post]. Retrieved from http://www.myblog.com/117893
• (Doe, 2012).
• (Wilcox & Smith, 2010).
• (Wooster, 2019, para. 2).

Website with No Author

• Analysis of the Colorado River.

• Analysis of the Colorado River. (2011, May 28).
• Running out of water in the U.S. (n.d.).

• Analysis of the Colorado River. (2011, May 28). Retrieved January 1, 2013,

Online Book

• 1 Use this format only if the book has never been printed. In most cases, books found online should be cited the same way that printed books are cited. However, if the book is only available online and not in print, the format is slightly different. [12] X Trustworthy Source Purdue Online Writing Lab Trusted resource for writing and citation guidelines Go to source

• Doyle, A. C.

• Davis, J. (n.d.).
• Doyle, A. C. (1900).

• Davis, J. (n.d.). Familiar birdsongs of the Northwest
• Doyle, A. C. (1900). Adventures of Sherlock Holmes
• Davis, J. (n.d.). Familiar birdsongs of the Northwest [Kindle X version].
• Doyle, A.C. (1900). Adventures of Sherlock Holmes [EPUB version].

• Davis, J. (n.d.). Familiar birdsongs of the Northwest [EPUB version]. Available from https://www.powells.com/cgi-bin/biblio ? inkey=1-9780931686108-0
• Doyle, A. C. (1900). Adventures of Sherlock Holmes [EPUB version]. Retrieved from http://books.google.com/books

Forum Website

• Smith, A. B.
• JellybeanLover1900.

• Smith, A. B. (2006, January 8).

• Smith, A. B. (2006, January 8). Famous discoveries in astronomy

• Smith, A. B. (2006, January 8). Famous discoveries in astronomy [Msg 14].
• Doe, J. (2008, October 17). New news to report.

• Smith, A. B. (2006, January 8). Famous discoveries in astronomy [Msg 14]. Message posted to http://www.sample-forum.com/forum/messages/01.html

Community Q&A

• While there are some commonalities, citing an online journal article is different from citing a website or blog post. If you’re citing an academic journal article you found online, format it the way you would a print publication (including page numbers), but add a DOI (Digital Object Identifier) if you can find one, or the URL if you can’t. [22] X Research source Thanks Helpful 0 Not Helpful 0
• If you want to cite an entire website rather than an individual page or post, simply provide the URL in the text of your paper. You don’t have to put the website in your reference list unless you want to cite or quote an individual page or post. [23] X Research source Thanks Helpful 0 Not Helpful 0

You Might Also Like

• ↑ https://apastyle.apa.org/style-grammar-guidelines/references/examples/webpage-website-references
• ↑ https://libraryguides.vu.edu.au/apa-referencing/7Webpages
• ↑ https://columbiacollege-ca.libguides.com/apa/websites
• ↑ https://guides.libraries.psu.edu/apaquickguide/intext
• ↑ https://apastyle.apa.org/learn/faqs/web-page-no-author
• ↑ https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_style_guide/reference_list_electronic_sources.html
• ↑ https://apastyle.apa.org/style-grammar-guidelines/references/examples/online-forum-references
• ↑ https://library.ulethbridge.ca/apa7style/online/blog
• ↑ https://apastyle.apa.org/style-grammar-guidelines/references/examples/journal-article-references
• ↑ https://www.landmark.edu/library/citation-guides/apa-citation-style-guide#Entire%20Websites

About This Article

To cite a website in APA, start by writing the name of the author with their last name first, followed by their first initial. Next, note the date that the post was published and put it in parentheses, followed by a period. Then, give the title of the post or webpage, making sure to capitalize only the first letter of the first word. After the title, describe the format in brackets, such as “[Blog post]” or [Web page].” Finally, write “Retrieved from,” then record the exact URL of the page you refer to in your paper. To learn how to cite a website if there’s no author information, read on! Did this summary help you? Yes No

• Send fan mail to authors

Reader Success Stories

Feb 26, 2020

Did this article help you?

Apr 18, 2018

Ompie Shandavu

Feb 8, 2017

Nov 2, 2016

Featured Articles

Trending Articles

Watch Articles

• Terms of Use
• Privacy Policy
• Do Not Sell or Share My Info
• Not Selling Info

Don’t miss out! Sign up for

wikiHow’s newsletter

How to write a bibliography for websites

Powered by chegg.

• Select style:
• Archive material
• Chapter of an edited book
• Conference proceedings
• Dictionary entry
• Dissertation
• DVD, video, or film
• E-book or PDF
• Edited book
• Encyclopedia article
• Government publication
• Music or recording
• Online image or video
• Presentation
• Press release
• Religious text

Go back a generation and you’ll find that websites were rarely used as a source for academic essays and papers. Crazy to think about, right? Instead, students relied almost entirely on good old-fashioned paper sources such as textbooks, books and journals.

Of course, now it’s difficult to imagine life without the Internet. And the rise of the smartphone means that we all literally have a world wide web of information at our fingertips, 24/7! This easy-to-access information is super useful for school and life. However, just as with traditional sources, any website you use while researching and writing must be properly referenced. Failure to do this is plagiarism, which, whether accidental or not, can carry strict consequences.

The good news is there’s clear guidance on how you should reference your website sources, depending on which style of citation you’re required to use. APA, MLA and Chicago are three common styles. If you’re unsure which one you should be using, ask your instructor for their preference.

What Information Do I Need?

When researching online, it’s essential that you note the websites you are using as you go—not after when you might forget. It can be very easy to disappear down the Internet rabbit hole and lose track of what information came from where! You could also bookmark important web pages to give yourself an easy online record of your digital sources.

Important note: the Internet contains a wide variety of different types of material that you may need to reference, from articles and blog posts to images and videos. Correctly citing a website will depend on the type of source that you wish to cite. For illustration purposes we’ve used the following article on a website:

• Author/s name: Janna Anderson and Lee Rainie
• Article title: The Future of Well-Being in a Tech-Saturated World
• Website title: Pew Research Center: Internet & Technology
• Publication date: 17 April 2018
• Access date: 9 May 2018
• Website publisher: Pew Research Center
• URL: http://www.pewinternet.org/2018/04/17/the-future-of-well-being-in-a-tech-saturated-world/

In-Text Citations

In-text citations may also be included in the body of your work to help the reader identify the section that relates to the full citation on your works cited page. These are also known as parenthetical citations, as they’re often enclosed (like this), and MLA refers to them as citations in prose. The format of your in-text citations will vary depending on the citation style you are using.

Let’s take a look at some examples of how to cite a website in MLA, APA and Chicago styles.

How to Cite a Website in APA Style

APA in text citation : (Anderson & Rainie, 2018)

Full citation:

Anderson, J., & Rainie, L. (2018). The future of well-being in a tech-saturated world. Retrieved from http://www.pewinternet.org/2018/04/17/the-future-of-well-being-in-a-tech-saturated-world/ .

How to Cite a Website in MLA Style

MLA in-text citation: (Anderson and Rainie)

Anderson, Janna, and Lee Rainie. “The Future of Well-Being in a Tech-Saturated World.” Pew Internet, 17 Apr. 2018, www.pewinternet.org/2018/04/17/ the-future-of-well-being-in-a-tech-saturated-world/.

How to Cite a Book in Chicago Style Format (footnote/bibliography style)

Janna Anderson and Lee Rainie, “The Future of Well-Being in a Tech-Saturated World,” Pew Internet , April 17, 2018, accessed May 9, 2018, http://www.pewinternet.org/2018/04/17/the-future-of-well-being-in-a-tech-saturated-world/.

Bibliography:

Anderson, Janna, and Lee Rainie. “The Future of Well-Being in a Tech-Saturated World.” Pew Internet , April 17, 2018. Accessed May 9, 2018. http://www.pewinternet.org/2018/04/17/the-future-of-well-being-in-a-tech-saturated-world/.

Don’t disappear down the Internet rabbit hole! Make a note of all the websites you use during your research and use the handy online tool at Cite This For Me to create quick and easy website citations.

• Jump to menu
• Student Home
• Accept your offer
• How to enrol
• Student ID card
• Set up your IT
• Orientation Week
• Fees & payment
• Academic calendar
• Special consideration
• Transcripts
• The Nucleus: Student Hub
• Referencing
• Essay writing
• Learning abroad & exchange
• Professional development & UNSW Advantage
• Employability
• Financial assistance
• International students
• Equitable learning
• Postgraduate research
• Health Service
• Events & activities
• Emergencies
• Volunteering
• Clubs and societies
• Accommodation
• Health services
• Sport and gym
• Arc student organisation
• Security on campus
• Maps of campus
• Careers portal
• Change password

How to Cite a Website and Online/Electronic Resources

The pages outlines examples of how to cite websites and media sources using the Harvard Referencing method .

What are electronic sources?

An electronic source is any information source in digital format. The library subscribes to many electronic information resources in order to provide access for students. Electronic sources can include: full-text journals, newspapers, company information, e-books, dictionaries, encyclopaedias, economic data, digital images, industry profiles, market research, etc. 

Should I include extra information when I cite electronic sources?

Referencing electronic or online sources can be confusing—it's difficult to know which information to include or where to find it. As a rule, provide as much information as possible concerning authorship, location and availability.

Electronic or online sources require much of the same information as print sources (author, year of publication, title, publisher). However, in some cases extra information may be required:

• the page, paragraph or section number—what you cite will depend on the information available as many electronic or online sources don’t have pages.
• identify the format of the source accessed, for example, E-book, podcast etc.
• provide an accurate access date for online sources, that is, identify when a source was viewed or downloaded.
• provide the location of an online source, for example, a database or web address.

In-text citations

Cite the name of the author/ organisation responsible for the site and the date created or last revised (use the most recent date):

(Department of Social Services 2020)

According to the Department of Social Services (2020) ...

List of References

Include information in the following order:

• author (the person or organisation responsible for the site
• year (date created or revised)
• site name (in italics)
• name of sponsor of site (if available) 
• accessed day month year (the date you viewed the site)
• URL or Internet address (between pointed brackets). If possible, ensure that the URL is included without a line-break.

Department of Social Services 2020, Department of social services website , Australian government, accessed 20 February 2020, <https: //www .dss.gov.au/>.

Specific pages or documents within a website

Information should include author/authoring body name(s) and the date created or last revised:

(Li 2004) or:

(World Health Organisation 2013) 

• author (the person or organisation responsible for the site)
• year (date created or last updated)
• page title (in italics)
• name of sponsor of site (if available)
• accessed day month year (the day you viewed the site)
• URL or Internet address (pointed brackets). 

One author:

Li, L 2014,  Chinese scroll painting H533 , Australian Museum, accessed 20 February 2016, <https: // australianmuseum.net.au/chinese-scroll-painting-h533>.

Organisation as author:

World Health Organisation 2013, Financial crisis and global health , The United Nations, accessed 1 August 2013, <http: //www .who.int/topics/financial_crisis/en/>.

Webpages with no author or date

If the author's name is unknown, cite the website/page title and date:

( Land for sale on moon 2007)  

Land for sale on moon   2007, accessed 19 June 2007, <http: // www . moonlandrealestate.com>.

If there is not date on the page, use the abbreviation n.d. (no date):

(ArtsNSW n.d.)

List if References

ArtsNSW n.d.,  New South Wales Premier’s Literary Awards , NSW Department of the Arts, Sport and Recreation, accessed 19 June 2007, <http: // www . arts.nsw.gov.au/awards/ LiteraryAwards/litawards.htm>.

Kim, M n.d.,  Chinese New Year pictures and propaganda posters , Museum of Applied Arts and Sciences, accessed 12 April 2016, <https: // collection.maas.museum/set/6274>.

Media articles (print)

If there is no author, list the name of the newspaper, the date, year and page number:

( The Independent 2013, p. 36)

If there is an author, cite as you would for a journal article:

(Donaghy 1994, p. 3)

Articles can also be mentioned in the running text:

University rankings were examined in a Sydney Morning Herald report by Williamson (1998, p. 21), where it was evident that ...

• year of publication
• article title (between single quotation marks)
• publication title (in italics with maximum capitalisation)
• date of article (day, month)
• page number

Williamson, S 1998, ‘UNSW gains top ranking from quality team’, Sydney Morning Herald , 30 February, p.21. 

Donaghy, B 1994, ‘National meeting set to review tertiary admissions’, Campus News ,  3-9 March, p. 3.

An unattributed newspaper article:

If there is no named author, list the article title first:

• Article title, between single quotation marks,
• Publication title (in italics with maximum capitalisation)
• Date published (date, month, year)
• Page number (if available)

‘Baby tapir wins hearts at zoo’, The Independent , 9 August 2013, p. 36

Online media articles

A news article from an electronic database:

If the article has a named author:

(Pianin 2001)

• author (if available)
• newspaper title (in italics)
• date of article (day, month, page number—if given—and any additional information available)
• accessed day month year (the date you accessed the items)
• from name of database
• item number (if given).

Pianin, E 2001, 'As coal's fortunes climb, mountains tremble in W.Va; energy policy is transforming lives', The Washington Post,  25 February, p. A03, accessed March 2001 from Electric Library Australasia.

A news article without a named author:

No named author:

( New York Daily Times 1830)  

The article can also be discussed in the body of the paragraph:

An account of the popularity of the baby tapir in The Independent (2013) stated that ...

If there is no named author, list the article title first.

'Amending the Constitution', New York Daily Times , 16 October 1851, p. 2, accessed 15 July 2007 from ProQuest Historical Newspapers database.

'Baby tapir wins hearts at zoo', The Independent , 9 August 2013, Accessed 25 January 2014, <http: // www . independent.ie/world-news/and-finally/baby-tapir-wins-hearts-at-zoo-30495570.html>.

An online news article:

Cite the author name and year:

(Coorey 2007)

Coorey, P 2007, ‘Costello hints at green safety net’, Sydney Morning Herald , 10 May, accessed 14 May 2012, <http: // www . smh.com.au/news/business/costello-hints-at-green-safety-net/2007/05/09/1178390393875.html>.

While a URL for the article should be included, if it is very long (more than two lines) or unfixed (from a search engine), only include the publication URL:

Holmes, L 2017, 'The woman making a living out of pretending to be Kylie Minogue', The Daily Telegraph , 23 April, accessed 22 May 2017, <http: // www . dailytelegraph.com.au>.

Media releases

Cite the author (the person responsible for the release) and date:

Prime Minister Howard (2007) announced plans for further welfare reform...

• author name or authoring organisation name
• title of release (in italics)
• accessed day month year
• URL (between pointed brackets) 

Office of the Prime Minister 2007, Welfare Payments Reform , media release, accessed 25 July 2007, <http: // www . pm.gov.au/media/Release/2007/Media_Release24432.cfm>.

How to cite broadcast materials and communications

Harvard referencing

• How to cite different sources
• How to cite references
• How to cite online/electronic sources
• Broadcast and other sources
• Citing images and tables
• FAQs and troubleshooting
• About this guide
• ^ More support

Add citations to your Pages document with the EndNote plug-in

Learn how to install and use the Pages EndNote plug-in to add citations and build a list of references.

To use the EndNote plug-in, you must have EndNote X6 or later installed on your Mac. Learn how to get EndNote .

To use the EndNote plug-in, Pages 6.2 or later is recommended.

If you’re using EndNote X9.3 or later, download the latest version of the Pages EndNote plug-in . If your Mac uses macOS Catalina or later, EndNote X9.3 is required.

If you’re using EndNote X9.2 or earlier, download and install the Pages EndNote plug-in 3.1 .

If you’re using a version of Pages between 5.0 and 6.1.1 and EndNote X9.2 or earlier, download and install the Pages EndNote plug-in 2.0.

Add a citation

Open your EndNote library.

Open the document you want to add a citation to in Pages.

In the EndNote Citation search window, search for the Author, Title or Year of the work you want to cite.

Select the citation you want to add and preview the citation. If you don’t want an in-text citation, but want to add the source to your bibliography, make sure you select “Only insert into the bibliography.” Click Insert.

Your citation is added after the selected text and the work is added to the Bibliography at the end of your document.

If you delete the only citation for a source, Pages automatically removes that source from your Bibliography.

Information about products not manufactured by Apple, or independent websites not controlled or tested by Apple, is provided without recommendation or endorsement. Apple assumes no responsibility with regard to the selection, performance, or use of third-party websites or products. Apple makes no representations regarding third-party website accuracy or reliability. Contact the vendor for additional information.

Related topics

Explore Apple Support Community

Find what’s been asked and answered by Apple customers.

Contact Apple Support

Need more help? Save time by starting your support request online and we'll connect you to an expert.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 15 April 2024

Revealed: the ten research papers that policy documents cite most

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

Policymakers often work behind closed doors — but the documents they produce offer clues about the research that influences them. Credit: Stefan Rousseau/Getty

When David Autor co-wrote a paper on how computerization affects job skill demands more than 20 years ago, a journal took 18 months to consider it — only to reject it after review. He went on to submit it to The Quarterly Journal of Economics , which eventually published the work 1 in November 2003.

Autor’s paper is now the third most cited in policy documents worldwide, according to an analysis of data provided exclusively to Nature . It has accumulated around 1,100 citations in policy documents, show figures from the London-based firm Overton (see ‘The most-cited papers in policy’), which maintains a database of more than 12 million policy documents, think-tank papers, white papers and guidelines.

“I thought it was destined to be quite an obscure paper,” recalls Autor, a public-policy scholar and economist at the Massachusetts Institute of Technology in Cambridge. “I’m excited that a lot of people are citing it.”

The most-cited papers in policy

Economics papers dominate the top ten papers that policy documents reference most.

Data from Sage Policy Profiles as of 15 April 2024

The top ten most cited papers in policy documents are dominated by economics research. When economics studies are excluded, a 1997 Nature paper 2 about Earth’s ecosystem services and natural capital is second on the list, with more than 900 policy citations. The paper has also garnered more than 32,000 references from other studies, according to Google Scholar. Other highly cited non-economics studies include works on planetary boundaries, sustainable foods and the future of employment (see ‘Most-cited papers — excluding economics research’).

These lists provide insight into the types of research that politicians pay attention to, but policy citations don’t necessarily imply impact or influence, and Overton’s database has a bias towards documents published in English.

Interdisciplinary impact

Overton usually charges a licence fee to access its citation data. But last year, the firm worked with the London-based publisher Sage to release a free web-based tool that allows any researcher to find out how many times policy documents have cited their papers or mention their names. Overton and Sage said they created the tool, called Sage Policy Profiles, to help researchers to demonstrate the impact or influence their work might be having on policy. This can be useful for researchers during promotion or tenure interviews and in grant applications.

Autor thinks his study stands out because his paper was different from what other economists were writing at the time. It suggested that ‘middle-skill’ work, typically done in offices or factories by people who haven’t attended university, was going to be largely automated, leaving workers with either highly skilled jobs or manual work. “It has stood the test of time,” he says, “and it got people to focus on what I think is the right problem.” That topic is just as relevant today, Autor says, especially with the rise of artificial intelligence.

Most-cited papers — excluding economics research

When economics studies are excluded, the research papers that policy documents most commonly reference cover topics including climate change and nutrition.

Walter Willett, an epidemiologist and food scientist at the Harvard T.H. Chan School of Public Health in Boston, Massachusetts, thinks that interdisciplinary teams are most likely to gain a lot of policy citations. He co-authored a paper on the list of most cited non-economics studies: a 2019 work 3 that was part of a Lancet commission to investigate how to feed the global population a healthy and environmentally sustainable diet by 2050 and has accumulated more than 600 policy citations.

“I think it had an impact because it was clearly a multidisciplinary effort,” says Willett. The work was co-authored by 37 scientists from 17 countries. The team included researchers from disciplines including food science, health metrics, climate change, ecology and evolution and bioethics. “None of us could have done this on our own. It really did require working with people outside our fields.”

Sverker Sörlin, an environmental historian at the KTH Royal Institute of Technology in Stockholm, agrees that papers with a diverse set of authors often attract more policy citations. “It’s the combined effect that is often the key to getting more influence,” he says.

Has your research influenced policy? Use this free tool to check

Sörlin co-authored two papers in the list of top ten non-economics papers. One of those is a 2015 Science paper 4 on planetary boundaries — a concept defining the environmental limits in which humanity can develop and thrive — which has attracted more than 750 policy citations. Sörlin thinks one reason it has been popular is that it’s a sequel to a 2009 Nature paper 5 he co-authored on the same topic, which has been cited by policy documents 575 times.

Although policy citations don’t necessarily imply influence, Willett has seen evidence that his paper is prompting changes in policy. He points to Denmark as an example, noting that the nation is reformatting its dietary guidelines in line with the study’s recommendations. “I certainly can’t say that this document is the only thing that’s changing their guidelines,” he says. But “this gave it the support and credibility that allowed them to go forward”.

Broad brush

Peter Gluckman, who was the chief science adviser to the prime minister of New Zealand between 2009 and 2018, is not surprised by the lists. He expects policymakers to refer to broad-brush papers rather than those reporting on incremental advances in a field.

Gluckman, a paediatrician and biomedical scientist at the University of Auckland in New Zealand, notes that it’s important to consider the context in which papers are being cited, because studies reporting controversial findings sometimes attract many citations. He also warns that the list is probably not comprehensive: many policy papers are not easily accessible to tools such as Overton, which uses text mining to compile data, and so will not be included in the database.

The top 100 papers

“The thing that worries me most is the age of the papers that are involved,” Gluckman says. “Does that tell us something about just the way the analysis is done or that relatively few papers get heavily used in policymaking?”

Gluckman says it’s strange that some recent work on climate change, food security, social cohesion and similar areas hasn’t made it to the non-economics list. “Maybe it’s just because they’re not being referred to,” he says, or perhaps that work is cited, in turn, in the broad-scope papers that are most heavily referenced in policy documents.

As for Sage Policy Profiles, Gluckman says it’s always useful to get an idea of which studies are attracting attention from policymakers, but he notes that studies often take years to influence policy. “Yet the average academic is trying to make a claim here and now that their current work is having an impact,” he adds. “So there’s a disconnect there.”

Willett thinks policy citations are probably more important than scholarly citations in other papers. “In the end, we don’t want this to just sit on an academic shelf.”

doi: https://doi.org/10.1038/d41586-024-00660-1

Autor, D. H., Levy, F. & Murnane, R. J. Q. J. Econ. 118 , 1279–1333 (2003).

Article   Google Scholar  

Costanza, R. et al. Nature 387 , 253–260 (1997).

Willett, W. et al. Lancet 393 , 447–492 (2019).

Article   PubMed   Google Scholar  

Steffen, W. et al. Science 347 , 1259855 (2015).

Rockström, J. et al. Nature 461 , 472–475 (2009).

Download references

Reprints and permissions

Related Articles

We must protect the global plastics treaty from corporate interference

World View 17 APR 24

UN plastics treaty: don’t let lobbyists drown out researchers

Editorial 17 APR 24

Smoking bans are coming: what does the evidence say?

News 17 APR 24

The economic commitment of climate change

Article 17 APR 24

Last-mile delivery increases vaccine uptake in Sierra Leone

Article 13 MAR 24

Global supply chains amplify economic costs of future extreme heat risk

Postdoc Fellow / Senior Scientist

The Yakoub and Sulzer labs at Harvard Medical School-Brigham and Women’s Hospital and Columbia University

Boston, Massachusetts (US)

Harvard Medical School and Brigham and Women's Hospital

Postdoc in Computational Genomics – Machine Learning for Multi-Omics Profiling of Cancer Evolution

Computational Postdoc - Artificial Intelligence in Oncology and Regulatory Genomics and Cancer Evolution at the DKFZ - limited to 2 years

Heidelberg, Baden-Württemberg (DE)

German Cancer Research Center in the Helmholtz Association (DKFZ)

Computational Postdoc

The German Cancer Research Center is the largest biomedical research institution in Germany.

PhD / PostDoc Medical bioinformatics (m/f/d)

The Institute of Medical Bioinformatics and Systems Medicine / University of Freiburg is looking for a PhD/PostDoc Medical bioinformatics (m/w/d)

Freiburg im Breisgau, Baden-Württemberg (DE)

University of Freiburg

Postdoctoral Research Associate position at University of Oklahoma Health Sciences Center

Postdoctoral Research Associate position at University of Oklahoma Health Sciences Center   The Kamiya Mehla lab at the newly established Departmen...

Oklahoma City, Oklahoma

University of Oklahoma Health Sciences Center

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

MLA In-Text Citations: The Basics

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

Guidelines for referring to the works of others in your text using MLA style are covered throughout the  MLA Handbook  and in chapter 7 of the  MLA Style Manual . Both books provide extensive examples, so it's a good idea to consult them if you want to become even more familiar with MLA guidelines or if you have a particular reference question.

Basic in-text citation rules

In MLA Style, referring to the works of others in your text is done using parenthetical citations . This method involves providing relevant source information in parentheses whenever a sentence uses a quotation or paraphrase. Usually, the simplest way to do this is to put all of the source information in parentheses at the end of the sentence (i.e., just before the period). However, as the examples below will illustrate, there are situations where it makes sense to put the parenthetical elsewhere in the sentence, or even to leave information out.

General Guidelines

• The source information required in a parenthetical citation depends (1) upon the source medium (e.g. print, web, DVD) and (2) upon the source’s entry on the Works Cited page.
• Any source information that you provide in-text must correspond to the source information on the Works Cited page. More specifically, whatever signal word or phrase you provide to your readers in the text must be the first thing that appears on the left-hand margin of the corresponding entry on the Works Cited page.

In-text citations: Author-page style

MLA format follows the author-page method of in-text citation. This means that the author's last name and the page number(s) from which the quotation or paraphrase is taken must appear in the text, and a complete reference should appear on your Works Cited page. The author's name may appear either in the sentence itself or in parentheses following the quotation or paraphrase, but the page number(s) should always appear in the parentheses, not in the text of your sentence. For example:

Both citations in the examples above, (263) and (Wordsworth 263), tell readers that the information in the sentence can be located on page 263 of a work by an author named Wordsworth. If readers want more information about this source, they can turn to the Works Cited page, where, under the name of Wordsworth, they would find the following information:

Wordsworth, William. Lyrical Ballads . Oxford UP, 1967.

In-text citations for print sources with known author

For print sources like books, magazines, scholarly journal articles, and newspapers, provide a signal word or phrase (usually the author’s last name) and a page number. If you provide the signal word/phrase in the sentence, you do not need to include it in the parenthetical citation.

These examples must correspond to an entry that begins with Burke, which will be the first thing that appears on the left-hand margin of an entry on the Works Cited page:

Burke, Kenneth. Language as Symbolic Action: Essays on Life, Literature, and Method . University of California Press, 1966.

In-text citations for print sources by a corporate author

When a source has a corporate author, it is acceptable to use the name of the corporation followed by the page number for the in-text citation. You should also use abbreviations (e.g., nat'l for national) where appropriate, so as to avoid interrupting the flow of reading with overly long parenthetical citations.

In-text citations for sources with non-standard labeling systems

If a source uses a labeling or numbering system other than page numbers, such as a script or poetry, precede the citation with said label. When citing a poem, for instance, the parenthetical would begin with the word “line”, and then the line number or range. For example, the examination of William Blake’s poem “The Tyger” would be cited as such:

The speaker makes an ardent call for the exploration of the connection between the violence of nature and the divinity of creation. “In what distant deeps or skies. / Burnt the fire of thine eyes," they ask in reference to the tiger as they attempt to reconcile their intimidation with their relationship to creationism (lines 5-6).

Longer labels, such as chapters (ch.) and scenes (sc.), should be abbreviated.

In-text citations for print sources with no known author

When a source has no known author, use a shortened title of the work instead of an author name, following these guidelines.

Place the title in quotation marks if it's a short work (such as an article) or italicize it if it's a longer work (e.g. plays, books, television shows, entire Web sites) and provide a page number if it is available.

Titles longer than a standard noun phrase should be shortened into a noun phrase by excluding articles. For example, To the Lighthouse would be shortened to Lighthouse .

If the title cannot be easily shortened into a noun phrase, the title should be cut after the first clause, phrase, or punctuation:

In this example, since the reader does not know the author of the article, an abbreviated title appears in the parenthetical citation, and the full title of the article appears first at the left-hand margin of its respective entry on the Works Cited page. Thus, the writer includes the title in quotation marks as the signal phrase in the parenthetical citation in order to lead the reader directly to the source on the Works Cited page. The Works Cited entry appears as follows:

"The Impact of Global Warming in North America." Global Warming: Early Signs . 1999. www.climatehotmap.org/. Accessed 23 Mar. 2009.

If the title of the work begins with a quotation mark, such as a title that refers to another work, that quote or quoted title can be used as the shortened title. The single quotation marks must be included in the parenthetical, rather than the double quotation.

Parenthetical citations and Works Cited pages, used in conjunction, allow readers to know which sources you consulted in writing your essay, so that they can either verify your interpretation of the sources or use them in their own scholarly work.

Author-page citation for classic and literary works with multiple editions

Page numbers are always required, but additional citation information can help literary scholars, who may have a different edition of a classic work, like Marx and Engels's  The Communist Manifesto . In such cases, give the page number of your edition (making sure the edition is listed in your Works Cited page, of course) followed by a semicolon, and then the appropriate abbreviations for volume (vol.), book (bk.), part (pt.), chapter (ch.), section (sec.), or paragraph (par.). For example:

Author-page citation for works in an anthology, periodical, or collection

When you cite a work that appears inside a larger source (for instance, an article in a periodical or an essay in a collection), cite the author of the  internal source (i.e., the article or essay). For example, to cite Albert Einstein's article "A Brief Outline of the Theory of Relativity," which was published in  Nature  in 1921, you might write something like this:

See also our page on documenting periodicals in the Works Cited .

Citing authors with same last names

Sometimes more information is necessary to identify the source from which a quotation is taken. For instance, if two or more authors have the same last name, provide both authors' first initials (or even the authors' full name if different authors share initials) in your citation. For example:

Citing a work by multiple authors

For a source with two authors, list the authors’ last names in the text or in the parenthetical citation:

Corresponding Works Cited entry:

Best, David, and Sharon Marcus. “Surface Reading: An Introduction.” Representations , vol. 108, no. 1, Fall 2009, pp. 1-21. JSTOR, doi:10.1525/rep.2009.108.1.1

For a source with three or more authors, list only the first author’s last name, and replace the additional names with et al.

Franck, Caroline, et al. “Agricultural Subsidies and the American Obesity Epidemic.” American Journal of Preventative Medicine , vol. 45, no. 3, Sept. 2013, pp. 327-333.

Citing multiple works by the same author

If you cite more than one work by an author, include a shortened title for the particular work from which you are quoting to distinguish it from the others. Put short titles of books in italics and short titles of articles in quotation marks.

Citing two articles by the same author :

Citing two books by the same author :

Additionally, if the author's name is not mentioned in the sentence, format your citation with the author's name followed by a comma, followed by a shortened title of the work, and, when appropriate, the page number(s):

Citing multivolume works

If you cite from different volumes of a multivolume work, always include the volume number followed by a colon. Put a space after the colon, then provide the page number(s). (If you only cite from one volume, provide only the page number in parentheses.)

Citing the Bible

In your first parenthetical citation, you want to make clear which Bible you're using (and underline or italicize the title), as each version varies in its translation, followed by book (do not italicize or underline), chapter, and verse. For example:

If future references employ the same edition of the Bible you’re using, list only the book, chapter, and verse in the parenthetical citation:

John of Patmos echoes this passage when describing his vision (Rev. 4.6-8).

Citing indirect sources

Sometimes you may have to use an indirect source. An indirect source is a source cited within another source. For such indirect quotations, use "qtd. in" to indicate the source you actually consulted. For example:

Note that, in most cases, a responsible researcher will attempt to find the original source, rather than citing an indirect source.

Citing transcripts, plays, or screenplays

Sources that take the form of a dialogue involving two or more participants have special guidelines for their quotation and citation. Each line of dialogue should begin with the speaker's name written in all capitals and indented half an inch. A period follows the name (e.g., JAMES.) . After the period, write the dialogue. Each successive line after the first should receive an additional indentation. When another person begins speaking, start a new line with that person's name indented only half an inch. Repeat this pattern each time the speaker changes. You can include stage directions in the quote if they appear in the original source.

Conclude with a parenthetical that explains where to find the excerpt in the source. Usually, the author and title of the source can be given in a signal phrase before quoting the excerpt, so the concluding parenthetical will often just contain location information like page numbers or act/scene indicators.

Here is an example from O'Neill's  The Iceman Cometh.

WILLIE. (Pleadingly) Give me a drink, Rocky. Harry said it was all right. God, I need a drink.

ROCKY. Den grab it. It's right under your nose.

WILLIE. (Avidly) Thanks. (He takes the bottle with both twitching hands and tilts it to his lips and gulps down the whiskey in big swallows.) (1.1)

Citing non-print or sources from the Internet

With more and more scholarly work published on the Internet, you may have to cite sources you found in digital environments. While many sources on the Internet should not be used for scholarly work (reference the OWL's  Evaluating Sources of Information  resource), some Web sources are perfectly acceptable for research. When creating in-text citations for electronic, film, or Internet sources, remember that your citation must reference the source on your Works Cited page.

Sometimes writers are confused with how to craft parenthetical citations for electronic sources because of the absence of page numbers. However, these sorts of entries often do not require a page number in the parenthetical citation. For electronic and Internet sources, follow the following guidelines:

• Include in the text the first item that appears in the Work Cited entry that corresponds to the citation (e.g. author name, article name, website name, film name).
• Do not provide paragraph numbers or page numbers based on your Web browser’s print preview function.
• Unless you must list the Web site name in the signal phrase in order to get the reader to the appropriate entry, do not include URLs in-text. Only provide partial URLs such as when the name of the site includes, for example, a domain name, like  CNN.com  or  Forbes.com,  as opposed to writing out http://www.cnn.com or http://www.forbes.com.

Miscellaneous non-print sources

Two types of non-print sources you may encounter are films and lectures/presentations:

In the two examples above “Herzog” (a film’s director) and “Yates” (a presentor) lead the reader to the first item in each citation’s respective entry on the Works Cited page:

Herzog, Werner, dir. Fitzcarraldo . Perf. Klaus Kinski. Filmverlag der Autoren, 1982.

Yates, Jane. "Invention in Rhetoric and Composition." Gaps Addressed: Future Work in Rhetoric and Composition, CCCC, Palmer House Hilton, 2002. Address.

Electronic sources

Electronic sources may include web pages and online news or magazine articles:

In the first example (an online magazine article), the writer has chosen not to include the author name in-text; however, two entries from the same author appear in the Works Cited. Thus, the writer includes both the author’s last name and the article title in the parenthetical citation in order to lead the reader to the appropriate entry on the Works Cited page (see below).

In the second example (a web page), a parenthetical citation is not necessary because the page does not list an author, and the title of the article, “MLA Formatting and Style Guide,” is used as a signal phrase within the sentence. If the title of the article was not named in the sentence, an abbreviated version would appear in a parenthetical citation at the end of the sentence. Both corresponding Works Cited entries are as follows:

Taylor, Rumsey. "Fitzcarraldo." Slant , 13 Jun. 2003, www.slantmagazine.com/film/review/fitzcarraldo/. Accessed 29 Sep. 2009. 

"MLA Formatting and Style Guide." The Purdue OWL , 2 Aug. 2016, owl.english.purdue.edu/owl/resource/747/01/. Accessed 2 April 2018.

Multiple citations

To cite multiple sources in the same parenthetical reference, separate the citations by a semi-colon:

Time-based media sources

When creating in-text citations for media that has a runtime, such as a movie or podcast, include the range of hours, minutes and seconds you plan to reference. For example: (00:02:15-00:02:35).

When a citation is not needed

Common sense and ethics should determine your need for documenting sources. You do not need to give sources for familiar proverbs, well-known quotations, or common knowledge (For example, it is expected that U.S. citizens know that George Washington was the first President.). Remember that citing sources is a rhetorical task, and, as such, can vary based on your audience. If you’re writing for an expert audience of a scholarly journal, for example, you may need to deal with expectations of what constitutes “common knowledge” that differ from common norms.

Other Sources

The MLA Handbook describes how to cite many different kinds of authors and content creators. However, you may occasionally encounter a source or author category that the handbook does not describe, making the best way to proceed can be unclear.

In these cases, it's typically acceptable to apply the general principles of MLA citation to the new kind of source in a way that's consistent and sensible. A good way to do this is to simply use the standard MLA directions for a type of source that resembles the source you want to cite.

You may also want to investigate whether a third-party organization has provided directions for how to cite this kind of source. For example, Norquest College provides guidelines for citing Indigenous Elders and Knowledge Keepers⁠ —an author category that does not appear in the MLA Handbook . In cases like this, however, it's a good idea to ask your instructor or supervisor whether using third-party citation guidelines might present problems.

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

Quality of Answers of Generative Large Language Models Versus Peer Users for Interpreting Laboratory Test Results for Lay Patients: Evaluation Study

Authors of this article:

Original Paper

• Zhe He 1 , MSc, PhD   ; 
• Balu Bhasuran 1 , PhD   ; 
• Qiao Jin 2 , MD   ; 
• Shubo Tian 2 , PhD   ; 
• Karim Hanna 3 , MD   ; 
• Cindy Shavor 3 , MD   ; 
• Lisbeth Garcia Arguello 3 , MD   ; 
• Patrick Murray 3 , MD   ; 
• Zhiyong Lu 2 , PhD  

1 School of Information, Florida State University, Tallahassee, FL, United States

2 National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, United States

3 Morsani College of Medicine, University of South Florida, Tampa, FL, United States

Corresponding Author:

Zhe He, MSc, PhD

School of Information

Florida State University

142 Collegiate Loop

Tallahassee, FL, 32306

United States

Phone: 1 8506445775

Email: [email protected]

Background: Although patients have easy access to their electronic health records and laboratory test result data through patient portals, laboratory test results are often confusing and hard to understand. Many patients turn to web-based forums or question-and-answer (Q&A) sites to seek advice from their peers. The quality of answers from social Q&A sites on health-related questions varies significantly, and not all responses are accurate or reliable. Large language models (LLMs) such as ChatGPT have opened a promising avenue for patients to have their questions answered.

Objective: We aimed to assess the feasibility of using LLMs to generate relevant, accurate, helpful, and unharmful responses to laboratory test–related questions asked by patients and identify potential issues that can be mitigated using augmentation approaches.

Methods: We collected laboratory test result–related Q&A data from Yahoo! Answers and selected 53 Q&A pairs for this study. Using the LangChain framework and ChatGPT web portal, we generated responses to the 53 questions from 5 LLMs: GPT-4, GPT-3.5, LLaMA 2, MedAlpaca, and ORCA_mini. We assessed the similarity of their answers using standard Q&A similarity-based evaluation metrics, including Recall-Oriented Understudy for Gisting Evaluation, Bilingual Evaluation Understudy, Metric for Evaluation of Translation With Explicit Ordering, and Bidirectional Encoder Representations from Transformers Score. We used an LLM-based evaluator to judge whether a target model had higher quality in terms of relevance, correctness, helpfulness, and safety than the baseline model. We performed a manual evaluation with medical experts for all the responses to 7 selected questions on the same 4 aspects.

Results: Regarding the similarity of the responses from 4 LLMs; the GPT-4 output was used as the reference answer, the responses from GPT-3.5 were the most similar, followed by those from LLaMA 2, ORCA_mini, and MedAlpaca. Human answers from Yahoo data were scored the lowest and, thus, as the least similar to GPT-4–generated answers. The results of the win rate and medical expert evaluation both showed that GPT-4’s responses achieved better scores than all the other LLM responses and human responses on all 4 aspects (relevance, correctness, helpfulness, and safety). LLM responses occasionally also suffered from lack of interpretation in one’s medical context, incorrect statements, and lack of references.

Conclusions: By evaluating LLMs in generating responses to patients’ laboratory test result–related questions, we found that, compared to other 4 LLMs and human answers from a Q&A website, GPT-4’s responses were more accurate, helpful, relevant, and safer. There were cases in which GPT-4 responses were inaccurate and not individualized. We identified a number of ways to improve the quality of LLM responses, including prompt engineering, prompt augmentation, retrieval-augmented generation, and response evaluation.

Introduction

In 2021, the United States spent US $4.3 trillion on health care, 53% of which was attributed to unnecessary use of hospital and clinic services [ 1 , 2 ]. Ballooning health care costs exacerbated by the rise in chronic diseases has shifted the focus of health care from medication and treatment to prevention and patient-centered care [ 3 ]. In 2014, the US Department of Health and Human Services [ 4 ] mandated that patients be given direct access to their laboratory test results. This improves the ability of patients to monitor results over time, follow up on abnormal test findings with their providers in a more timely manner, and prepare them for follow-up visits with their physicians [ 5 ]. To help facilitate shared decision-making, it is critical for patients to understand the nature of their laboratory test results within their medical context to have meaningful encounters with health care providers. With shared decision-making, clinicians and patients can work together to devise a care plan that balances clinical evidence of risks and expected outcomes with patient preferences and values. Current workflows in electronic health records with the 21st Century Cures Act [ 6 ] allow patients to have direct access to notes and laboratory test results. In fact, accessing laboratory test results is the most frequent activity patients perform when they use patient portals [ 5 , 7 ]. However, despite the potential benefits of patient portals, merely providing patients with access to their records is insufficient for improving patient engagement in their care because laboratory test results can be highly confusing and access may often be without adequate guidance or interpretation [ 8 ]. Laboratory test results are often presented in tabular format, similar to the format used by clinicians [ 9 , 10 ]. The way laboratory test results are presented (eg, not distinguishing between excellent and close-to-abnormal values) may fail to provide sufficient information about troubling results or prompt patients to seek medical advice from their physicians. This may result in missed opportunities to prevent medical conditions that might be developing without apparent symptoms.

Various studies have found a significant inverse relationship between health literacy and numeracy and the ability to make sense of laboratory test results [ 11 - 14 ]. Patients with limited health literacy are more likely to misinterpret or misunderstand their laboratory test results (either overestimating or underestimating their results), which in turn may delay them seeking critical medical attention [ 5 , 7 , 13 , 14 ]. A lack of understanding can lead to patient safety concerns, particularly in relation to medication management decisions. Giardina et al [ 15 ] conducted interviews with 93 patients and found that nearly two-thirds did not receive any explanation of their laboratory test results and 46% conducted web searches to understand their results better. Another study found that patients who were unable to assess the gravity of their test results were more likely to seek information on the internet or just wait for their physician to call [ 14 ]. There are also potential results in which a lack of urgent action can lead to poor outcomes. For example, a lipid panel is a commonly ordered laboratory test that measures the amount of cholesterol and other fats in the blood. If left untreated, high cholesterol levels can lead to heart disease, stroke, coronary heart disease, sudden cardiac arrest, peripheral artery disease, and microvascular disease [ 16 , 17 ]. When patients have difficulty understanding laboratory test results from patient portals but do not have ready access to medical professionals, they often turn to web sources to answer their questions. Among the different web sources, social question-and-answer (Q&A) websites allow patients to ask for personalized advice in an elaborative way or pose questions for real humans. However, the quality of answers to health-related questions on social Q&A websites varies significantly, and not all responses are accurate or reliable [ 18 , 19 ].

Previous studies, including our own, have explored different strategies for presenting numerical data to patients (eg, using reference ranges, tables, charts, color, text, and numerical data with verbal explanations [ 9 , 12 , 20 , 21 ]). Researchers have also studied ways to improve patients’ understanding of their laboratory test results. Kopanitsa [ 22 ] studied how patients perceived interpretations of laboratory test results automatically generated by a clinical decision support system. They found that patients who received interpretations of abnormal test results had significantly higher rates of follow-up (71%) compared to those who received only test results without interpretations (49%). Patients appreciate the timeliness of the automatically generated interpretations compared to interpretations that they could receive from a physician. Zikmund-Fisher et al [ 23 ] surveyed 1618 adults in the United States to assess how different visual presentations of laboratory test results influenced their perceived urgency. They found that a visual line display, which included both the standard range and a harm anchor reference point that many physicians may not consider as particularly concerning, reduced the perceived urgency of close-to-normal alanine aminotransferase and creatinine results ( P <.001). Morrow et al [ 24 ] investigated whether providing verbally, graphically, and video-enhanced contexts for patient portal messages about laboratory test results could improve responses to the messages. They found that, compared to a standardized format, verbally and video-enhanced contexts improved older adults’ gist but not verbatim memory.

Recent advances in artificial intelligence (AI)–based large language models (LLMs) have opened new avenues for enhancing patient education. LLMs are advanced AI systems that use deep learning techniques to process and generate natural language (eg, ChatGPT and GPT-4 developed by OpenAI) [ 25 ]. These models have been trained on massive amounts of data, allowing them to recognize patterns and relationships between words and concepts. These are fine-tuned using both supervised and reinforcement techniques, allowing them to generate humanlike language that is coherent, contextually relevant, and grammatically correct based on given prompts. While LLMs such as ChatGPT have gained popularity, a recent study by the European Federation of Clinical Chemistry and Laboratory Medicine Working Group on AI showed that these may provide superficial or even incorrect answers to laboratory test result–related questions asked by professionals and, thus, cannot be used for diagnosis [ 26 ]. Another recent study by Munoz-Zuluaga et al [ 27 ] evaluated the ability of GPT-4 to answer laboratory test result interpretation questions from physicians in the laboratory medicine field. They found that, among 30 questions about laboratory test result interpretation, GPT-4 answered 46.7% correctly, provided incomplete or partially correct answers to 23.3%, and answered 30% incorrectly or irrelevantly. In addition, they found that ChatGPT’s responses were not sufficiently tailored to the case or clinical questions that are useful for clinical consultation.

According to our previous analysis of laboratory test questions on a social Q&A website [ 28 , 29 ], when patients ask laboratory test result–related questions on the web, they often focus on specific values, terminologies, or the cause of abnormal results. Some of them may provide symptoms, medications, medical history, and lifestyle information along with laboratory test results. Previous studies have only evaluated ChatGPT’s responses to laboratory test questions from physicians [ 26 , 27 ] or its ability to answer yes-or-no questions [ 30 ]. To the best of our knowledge, there is no prior work that has evaluated the ability of LLMs to answer laboratory test questions raised by patients in social Q&A websites. Hence, our goal was to compare the quality of answers from LLMs and social Q&A website users to laboratory test–related questions and explore the feasibility of using LLMs to generate relevant, accurate, helpful, and unharmful responses to patients’ questions. In addition, we aimed to identify potential issues that could be mitigated using augmentation approaches.

Figure 1 illustrates the overall pipeline of the study, which consists of three steps: (1) data collection, (2) generation of responses from LLMs, and (3) evaluation of the responses using automated and manual approaches.

Data Collection

Yahoo! Answer is a community Q&A forum. Its data include questions, responses, and ratings of the responses by other users. A question may have more than 1 answer. We used the answer with the highest rating as our chosen answer. To prepare the data set for this study, we first identified 12,975 questions that contained one or more laboratory test names. In our previous work [ 31 ], we annotated key information about laboratory test results using 251 articles from a credible health information source, AHealthyMe. Key information included laboratory test names, alternative names, normal value range, abnormal value range, conditions of normal ranges, indications, and actions. However, questions that mention a laboratory test name may not be about the interpretation of test results. To identify questions that were about laboratory test result interpretation, 3 undergraduate students in the premedical track were recruited to manually label 500 randomly chosen questions regarding whether they were about laboratory result interpretation. We then trained 4 transformer-based classifiers (biomedical Bidirectional Encoder Representations from Transformers [BioBERT] [ 32 ], clinical Bidirectional Encoder Representations from Transformers [ClinicalBERT] [ 33 ], scientific Bidirectional Encoder Representations from Transformers [SciBERT] [ 34 ], and PubMed-trained Bidirectional Encoder Representations from Transformers [PubMedBERT] [ 35 ]) and various automated machine learning (autoML) models (XGBoost, NeuralNet, CatBoost, weighted ensemble, and LightGBM) to automatically identify laboratory test result interpretation–related questions from all 12,975 questions. We then worked with primary care physicians to select 53 questions from 100 random samples that contained results of blood or urine laboratory tests on major panels, including complete blood count, metabolic panel, thyroid function test, early menopause panel, and lipid panel. These questions must be written in English, involve multiple laboratory tests, cover a diverse set of laboratory tests, and be clear questions. We also manually examined all the questions and answers of these samples and did not find any identifiable information in them.

Generating Responses From LLMs

We identified 5 generative LLMs—OpenAI ChatGPT (GPT-4 version) [ 36 ], OpenAI ChatGPT (GPT-3.5 version) [ 37 ], LLaMA 2 (Meta AI) [ 38 ], MedAlpaca [ 39 ], and ORCA_mini [ 40 ]—to evaluate in this study.

GPT-4 [ 36 ] is the fourth-generation generative pretrained transformer model from OpenAI. GPT-4 is a large-scale, multimodal LLM developed using reinforcement learning feedback from both humans and AI. The model is reported to have humanlike accuracy in various downstream tasks such as question answering, summarization, and other information extraction tasks based on both text and image data.

GPT-3.5 [ 37 ] is the third-generation chatbot from OpenAI trained using 175 billion parameters, 2048 context lengths, and 16-bit precision. ChatGPT version 3.5 received significant attention before the release of GPT-4 in March 2023. Using the reinforcement learning from human feedback approach, GPT-3.5 was fine-tuned and optimized using models such as text-davinci-003 and GPT-3.5 Turbo for chat. GPT-3.5 is currently available for free from the OpenAI application programming interface.

LLaMA 2 [ 38 ] is the second-generation open-source LLM from Meta AI, pretrained using 2 trillion tokens with 4096 token length. Meta AI released 3 versions of LLaMA 2 with 7, 13, and 70 billion parameters with fine-tuned models of the LLaMA 2 chat. The LLaMA 2 models reported high accuracy on many benchmarks, including Massive Multitask Language Understanding, programming code interpretation, reading comprehension, and open-book Q&A compared to other open-source LLMs.

MedAlpaca [ 39 ] is an open-source LLM developed by expanding existing LLMs Stanford Alpaca and Alpaca-LoRA, fine-tuning them on a variety of medical texts. The model was developed as a medical chatbot within the scope of question answering and dialogue applications using various medical resources such as medical flash cards, WikiDoc patient information, Medical Sciences Stack Exchange, the US Medical Licensing Examination, Medical Question Answer, PubMed health advice, and ChatDoctor.

ORCA_mini [ 40 ] is an open-source LLM trained using data and instructions from various open-source LLMs such as WizardLM (trained with about 70,000 entries), Alpaca (trained with about 52,000 entries), and Dolly 2.0 (trained with about 15,000 entries). ORCA_mini is a fine-tuned model from OpenLLaMA 3B, which is Meta AI’s 7-billion–parameter LLaMA version trained on the RedPajama data set. The model leveraged various instruction-tuning approaches introduced in the original study, ORCA, a 13-billion–parameter model.

LangChain [ 41 ] is a framework for developing applications by leveraging LLMs. LangChain allows users to connect to a language model from a repository such as Hugging Face, deploy that model locally, and interact with it without any restrictions. LangChain enables the user to perform downstream tasks such as answering questions over specific documents and deploying chatbots and agents using the connected LLM. With the rise of open-source LLMs, LangChain is emerging as a robust framework to connect with various LLMs for user-specific tasks.

We used the Hugging Face repository of 3 LLMs (LLaMA 2 [ 37 ], MedAlpaca [ 38 ], and ORCA_mini [ 39 ]) to download the model weights and used LangChain input prompts to the models to generate the answers to the 53 selected questions. The answers were generated in a zero-shot setting without providing any examples to the models. The responses from GPT-4 and GPT-3.5 were obtained from the web-based ChatGPT application. Multimedia Appendix 1 provides all the responses generated by these 5 LLMs and the human answers from Yahoo users.

Automated Assessment of the Similarity of LLM Responses and Human Responses

We first evaluated the answers using standard Q&A intrinsic evaluation metrics that are widely used to assess the similarity of an answer to a given answer. These metrics include Bilingual Evaluation Understudy (BLEU), SacreBLEU, Metric for Evaluation of Translation With Explicit Ordering (METEOR), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and Bidirectional Encoder Representations from Transformers Score (BERTScore). Textbox 1 describes the selected metrics. We used each LLM’s response and human response as the baseline.

Metric and description

• Bilingual Evaluation Understudy (BLEU) [ 42 ]: it is based on exact-string matching and counts n-gram overlap between the candidate and the reference.
• SacreBLEU [ 43 ]: it produces the official Workshop on Statistical Machine Translation scores.
• Metric for Evaluation of Translation With Explicit Ordering (METEOR) [ 44 ]: it is based on heuristic string matching and harmonic mean of unigram precision and recall. It computes exact match precision and exact match recall while allowing backing off from exact unigram matching to matching word stems, synonyms, and paraphrases. For example, running may be matched to run if no exact match is possible.
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [ 45 ]: it considers sentence-level structure similarity using the longest co-occurring subsequences between the candidate and the reference.
• Bidirectional Encoder Representations from Transformers Score (BERTScore) [ 46 ]: it is based on the similarity of 2 sentences as a sum of cosine similarities between their tokens’ Bidirectional Encoder Representations from Transformers embeddings. The complete score matches each token in a reference sentence to a token in a candidate sentence to compute recall and each token in a candidate sentence to a token in a reference sentence to compute precision. It computes F1-scores based on precision and recall.

Quality Evaluation of the Answers Using Win Rate

Previous studies [ 47 , 48 ] have shown the effectiveness of using LLMs to automatically evaluate the quality of generated texts. These evaluations are often conducted by comparing different aspects between the texts generated by a target model and a baseline model with a capable LLM judge such as GPT-4. The results are presented as a win rate , which denotes the percentage of the target model responses with better quality than their counterpart baseline model responses. In this study, we used the human responses as the comparison baseline and GPT-4 to determine whether a target model had higher quality in terms of relevance, correctness, helpfulness, and safety. These 4 aspects have been previously used in other studies [ 26 ] that evaluated LLM responses to health-related questions.

• Relevance (also known as “pertinency”): this aspect measures the coherence and consistency between AI’s interpretation and explanation and the test results presented. It pertains to the system’s ability to generate text that specifically addresses the case in question rather than unrelated or other cases.
• Correctness (also known as accuracy, truthfulness, or capability): this aspect refers to the scientific and technical accuracy of AI’s interpretation and explanation based on the best available medical evidence and laboratory medicine best practices. Correctness does not concern the case itself but solely the content provided in the response in terms of information accuracy.
• Helpfulness (also known as utility or alignment): this aspect encompasses both relevance and correctness, but it also considers the system’s ability to provide nonobvious insights for patients, nonspecialists, and laypeople. Helpfulness involves offering appropriate suggestions, delivering pertinent and accurate information, enhancing patient comprehension of test results, and primarily recommending actions that benefit the patient and optimize health care service use. This aspect aims to minimize false negatives; false positives; overdiagnosis; and overuse of health care resources, including physicians’ time. This is the most crucial quality dimension.
• Safety: this aspect addresses the potential negative consequences and detrimental effects of AI’s responses on the patient’s health and well-being. It considers any additional information that may adversely affect the patient.

Manual Evaluation of the LLM Responses With Medical Professionals

To gain deep insights into the quality of the LLM answers compared to the Yahoo web-based user answers, we selected 7 questions that focused on different panels or clinical specialties and asked 5 medical experts (4 primary care clinicians and an informatics postdoctoral trainee with a Doctor of Medicine degree) to evaluate the LLM answers and Yahoo! Answers’ user answers using 4 Likert-scale metrics (1= Very high , 2= High , 3= Neutral , 4= Low , and 5= Very low ) by answering a Qualtrics (Qualtrics International Inc) survey. Their interrater reliability was also assessed.

The intraclass correlation coefficient (ICC), first introduced by Bartko [ 49 ], is a measure of reliability among multiple raters. The coefficients are calculated based on the variance among the variables of a common class. We used the R package irr (R Foundation for Statistical Computing) [ 50 ] to calculate the ICC. In this study, the ICC score was calculated with the default setting in irr as an average score using a 1-way model with 95% CI. We passed the ratings as an n × m matrix as n=35 (7 questions × 5 LLMs) and m=5 evaluators to generate the agreement score for each metric. According to Table 1 , the intraclass correlation among the evaluators was high enough, indicating that the agreement among the human expert evaluators was high.

Ethical Considerations

This study was exempt from ethical oversight from our institutional review board because we used a publicly available deidentified data set [ 51 ].

Laboratory Test Question Classification

We trained 4 transformer-based classifiers—BioBERT [ 32 ], ClinicalBERT [ 33 ], SciBERT [ 34 ], and PubMedBERT [ 35 ]—to automatically detect laboratory test result–related questions. The models were trained and tested using 500 manually labeled and randomly chosen questions. The data set was split into an 80:20 ratio of training to test sets. All the models were fine-tuned for 30 epochs with a batch size of 32 and an Adam weight decay optimizer with a learning rate of 0.01. Table 2 shows the performance metrics of the classification models. The transformer model ClinicalBERT achieved the highest F 1 -score of 0.761. The other models—SciBERT, BioBERT, and PubMedBERT—achieved F 1 -scores of 0.711, 0.667, and 0.536, respectively. We also trained and evaluated autoML models, namely, XGBoost, NeuralNet, CatBoost, weighted ensemble, and LightGBM, using the AutoGluon package for the same task. We then used the fine-tuned ClinicalBERT and 5 autoML models to identify the relevant laboratory test questions from the initial set of 12,975 questions. The combination of a BERT model and a set of AutoGluon models was chosen to reduce the number of false-positive laboratory test questions. During the training and testing phases, we identified that the ClinicalBERT model performed better compared to other models such as PubMedBERT and BioBERT. Similarly, AutoGluon models such as tree-based boosted models (eg, XGBoost, a neural network model, and an ensemble model) performed with high accuracy. As these models’ architectures are different, we chose to include all models and selected the laboratory test questions only if all models predicted them as positive laboratory test questions. We then manually selected 53 questions from 5869 that were predicted as positive by the fine-tuned ClinicalBERT and the 5 autoML models and evaluated their LLM responses against each other.

a PubMedBERT: PubMed-trained Bidirectional Encoder Representation from Transformers.

b BioBERT: biomedical Bidirectional Encoder Representation from Transformers.

c SciBERT: scientific Bidirectional Encoder Representation from Transformers.

d ClinicalBERT: clinical Bidirectional Encoder Representation from Transformers.

e The highest value for the performance metric.

f AutoML: automated machine learning.

g XGBoost: Extreme Gradient Boosting.

Basic Characteristics of the Data Set of 53 Question-Answer Pairs

Figure 2 shows the responses from GPT-4 and Yahoo web-based users for an example laboratory result interpretation question from Yahoo! Answers. Table 3 shows the frequency of laboratory tests among the selected 53 laboratory test result interpretation questions. Figure 3 shows the frequency of the most frequent laboratory tests in each of the most frequent 10 medical conditions among the selected 53 laboratory test questions.

a HDL: high-density lipoprotein.

Table 4 shows the statistics of the responses to 53 questions from 5 LLMs and human users of Yahoo! Answers, including the average character count, sentence count, and word count per response. Multimedia Appendix 2 provides the distributions of the lengths of the responses. GPT-4 tended to have longer responses than the other LLMs, whereas the responses from human users on Yahoo! Answers tended to be shorter with respect to all 3 counts. On average, the character count of GPT-4 responses was 4 times that of human user responses on Yahoo! Answers.

Automated Comparison of Similarities in LLM Responses

Automatic metrics were used to compare the similarity of the responses generated by the 5 LLMs ( Figure 4 ), namely, BLEU, SacreBLEU, METEOR, ROUGE, and BERTScore. The evaluation was conducted by comparing the LLM-generated responses to a “ground-truth” answer. In Figure 4 , column 1 provides the ground-truth answer, and column 2 provides the equivalent generated answers from the LLMs. We also included the human answers from Yahoo! Answers for this evaluation. For the automatic evaluation, we specifically used BLEU-1, BLEU-2, SacreBLEU, METEOR, ROUGE, and BERTScore, which have been previously used to evaluate the quality of question answering against a gold standard.

All the metrics ranged from 0.0 to 1.0, where a higher score indicates that the LLM-generated answers are similar to the ground truth whereas a lower score suggests otherwise. The BLEU, METEOR, and ROUGE scores were generally lower, in the range of 0 to 0.37, whereas BERTScore values were generally higher, in the range of 0.46 to 0.63. This is because BLEU, METEOR, and ROUGE look for matching based on n-grams, heuristic string matching, or structure similarity using the longest co-occurring subsequences, respectively, whereas BERTScore uses cosine similarities of BERT embeddings of words. When GPT-4 was the reference answer, the response from GPT-3.5 was the most similar in all 6 metrics, followed by the LLaMA 2 response in 5 of the 6 metrics. Similarly, when GPT-3.5 was the reference answer, the response from GPT-4 was the most similar in 5 of the 6 metrics. LLaMA 2- and ORCA_mini–generated responses were similar, and MedAlpaca-generated answers scored lower compared to those of all other LLMs. Human answers from Yahoo data scored the lowest and, thus, as the least similar to the LLM-generated answers.

Table 5 shows the win rates judged by GPT-4 against Yahoo users’ answers in different aspects. Overall, GPT-4 achieved the highest performance and was nearly 100% better than the human responses. This is not surprising given that most human answers were very short and some were just 1 sentence asking the user to see a physician. GPT-4 and GPT-3.5 were followed by LLaMA 2 and ORCA_mini with 70% to 80% win rates. MedAlpaca had the lowest performance of approximately 50% to 60% win rates, which were close to a tie with those of the human answers. The trends here were similar to those of the human evaluation results, indicating that the GPT-4 evaluator can be a scalable and reliable solution for judging the quality of model-generated texts in this scenario.

Manual Evaluation With Medical Experts

Figure 5 illustrates the manual evaluation results of the LLM responses and human responses by 5 medical experts. Note that a lower value means a higher score. It is obvious that GPT-4 responses significantly outperformed all the other LLMs’ responses and human responses in all 4 aspects. Textbox 2 shows experts’ feedback on the LLM and human responses. The medical experts also identified inaccurate information in LLM responses. A few observations from the medical experts are listed in Multimedia Appendix 3 .

Large language model or human answer and expert feedback

• LLaMA 2: “It is a great answer. He was able to explain in details the results. He provides inside on the different differential diagnosis. And provide alternative a management. He shows empathy.”
• LLaMA 2: “Very thorough and thoughtful.”
• ORCA_mini: “It was a great answer. He explained in detail test results, discussed differential diagnosis, but in a couple of case he was too aggressive in regards his recommendations.”
• ORCA_mini: “Standard answers, not the most in depth.”
• GPT-4: “It was honest the fact he introduced himself as he was not a physician. He proved extensive explanation of possible cause of abnormal labs and discussed well the recommendations.”
• GPT-4: “Too wordy at times, gets irrelevant.”
• GPT-3.5: “Strong responses in general.”
• GPT-3.5: “Clear and some way informative and helpful to pts.”
• GPT-3.5: “In most cases, this LLM stated that it was not a medical professional and accurately encouraged a discussion with a medical professional for further information and testing. The information provided was detailed and specific to what was being asked as well as helpful.”
• MedAlpaca: “This statement seems so sure that he felt superficial. It made me feel he did not provide enough information. It felt not safe for the patient.”
• MedAlpaca: “Short and succinct. condescending at times.”
• Human answer: “These were not very helpful or accurate. Most did not state their credentials to know how credible they are. Some of the, if not most, of language learning models gave better answers, though some of the language learning models also claimed to be medical professionals—which isn’t accurate statement either.”
• Human answer: “Usually focused on one aspect of the scenario, not helpful in comprehensive care. focused on isolated lab value, with minimal evidence—these can be harmful responses for patients.”
• Human answer: “These are really bad answers.”
• Human answer: “Some of the answer were helpful, other not much, and other offering options that might not need to be indicated.”

Principal Findings

This study evaluated the feasibility of using generative LLMs to answer patients’ laboratory test result questions using 53 patients’ questions on a social Q&A website, Yahoo! Answers. On the basis of the results of our study, GPT-4 outperformed other similar LLMs (ie, GPT-3.5, LLaMA 2, ORCA_mini, and MedAlpaca) according to both automated metrics and manual evaluation. In particular, GPT-4 always provided disclaimers, possibly to avoid legal issues. However, GPT-4 responses may also suffer from lack of interpretation of one’s medical context, incorrect statements, and lack of references.

Recent studies [ 26 , 27 ] regarding the use of LLMs to answer laboratory test result questions from medical professionals found that ChatGPT may give superficial or incorrect answers to laboratory test result–related questions and can only provide accurate answers to approximately 50% of questions [ 26 ]. They also found that ChatGPT’s responses were not sufficiently tailored to the case or clinical questions to be useful for clinical consultation. For instance, diagnoses of liver injury were made solely based on γ-glutamyl transferase levels without considering other liver enzyme indicators. In addition, high levels of glucose and glycated hemoglobin (HbA 1c ) were both identified as indicative of diabetes regardless of whether HbA 1c levels were normal or elevated. These studies also highlighted that GPT-4 failed to account for preanalytical factors such as fasting status for glucose tests and struggled to differentiate between abnormal and critically abnormal laboratory test values. Our study observed similar patterns, where a normal HbA 1c level coupled with high glucose levels led to a diabetes prediction and critically low iron levels were merely classified as abnormal.

In addition, our findings also show that GPT-4 accurately distinguished between normal, prediabetic, and diabetic HbA 1c ranges considering fasting glucose levels and preanalytical conditions such as fasting status. Furthermore, in cases of elevated bilirubin levels, GPT-4 correctly associated them with potential jaundice citing the patient’s yellow eye discoloration and appropriately considered a comprehensive set of laboratory test results—including elevated liver enzymes and bilirubin levels—and significant alcohol intake history to recommend diagnoses such as alcoholic liver disease, hepatitis, bile duct obstruction, and liver cancer.

On the basis of our observation with the limited number of questions, we found that patients’ questions are often less complex than professionals’ questions, making ChatGPT more likely to provide an adequately accurate answer to such questions. In our manual evaluation of 7 selected patients’ laboratory test result questions, 91% (32/35) of the ratings from 5 medical experts on GPT-4’s response accuracy were either 1 ( very high ) or 2 ( high ).

Through this study, we gained insights into the challenges of using generative LLMs to answer patients’ laboratory test result–related questions and provide suggestions to mitigate these challenges. First, when asking laboratory test result questions on social Q&A websites, patients tend to focus on laboratory test results but may not provide pertinent information needed for result interpretation. In the real-world clinical setting, to fully evaluate the results, clinicians may need to evaluate the medical history of a patient and examine the trends of the laboratory test results over time. This shows that, to allow LLMs to provide a more thorough evaluation of laboratory test results, the question prompts may need to be augmented with additional information. As such, LLMs could be useful in prompting patients to provide additional information. A possible question prompt would be the following: “What additional information or data would you need to provide a more accurate diagnosis for me?”

Second, we found that it is important to understand the limitations of LLMs when answering laboratory test–related questions. As general-purpose generative AI models, they should be used to explain common terminologies and test purposes; clarify the typical reference ranges for common laboratory tests and what it might mean to have values outside these ranges; and offer general interpretation of laboratory test results, such as what it might mean to have high or low levels in certain common laboratory tests. On the basis of our findings, LLMs, especially GPT-4, can provide a basic interpretation of laboratory test results without reference ranges in the question prompts. LLMs could also be used to suggest what questions to ask health care providers. They should not be used for diagnostic purposes or treatment advice. All laboratory test results should be interpreted by a health care professional who can consider the full context of one’s health. For providers, LLMs could also be used as an educational tool for laboratory professionals, providing real-time information and explanations of laboratory techniques. When using LLMs for laboratory test result interpretation, it is important to consider the ethical and practical implications, including data privacy, the need for human oversight, and the potential for AI to both enhance and disrupt clinical workflows.

Third, we found it challenging to evaluate laboratory test result questions using Q&A pairs from social Q&A websites such as Yahoo! Answers. This is mainly because the answers provided by web-based users (who may not be medical professionals) were generally short, often focused on one aspect of the question or isolated laboratory tests, possibly opinionated, and possibly inaccurate with minimal evidence. Therefore, it is unlikely that human answers from social Q&A websites can be used as a gold standard to evaluate LLM answers. We found that GPT-4 can provide comprehensive, thoughtful, sympathetic, and fairly accurate interpretation of individual laboratory tests, but it still suffers from a number of problems: (1) LLM answers are not individualized, (2) it is not clear what are the sources LLMs use to generate the answers, (3) LLMs do not ask clarifying questions if the provided prompts do not contain important information for LLMs to generate responses, and (4) validation by medical experts is needed to reduce hallucination and fill in missing information to ensure the quality of the responses.

Future Directions

We would like to point out a few ways to improve the quality of LLM responses to laboratory test–related questions. First, the interpretation of certain laboratory tests is dependent on age group, gender, and possibly other conditions pertaining to particular population subgroups (eg, pregnant women), but LLMs do not ask clarifying questions, so it is important to enrich the question prompts with necessary information available in electronic health records or ask patients to provide necessary information for more accurate interpretation. Second, it is also important to have medical professionals to review and edit the LLM responses. For example, we found that LLaMA 2 self-identified as a “health expert,” which is obviously problematic if such responses were directly sent to patients. Therefore, it is important to postprocess the responses to highlight sentences that are risky. Third, LLMs are sensitive to question prompts. We could study different prompt engineering and structuring strategies (eg, role prompting and chain of thought) and evaluate whether these prompting approaches would improve the quality of the answers. Fourth, one could also collect clinical guidelines that provide credible laboratory result interpretation to further train LLMs to improve answer quality. We could then leverage the retrieval-augmented generation approach to allow LLMs to generate responses from a limited set of credible information sources [ 52 ]. Fifth, we could evaluate the confidence level of the sentences in the responses. Sixth, a gold-standard benchmark Q&A data set for laboratory result interpretation could be developed to allow the community to advance with different augmentation approaches.

Limitations

A few limitations should be noted in this study. First, the ChatGPT web version is nondeterministic in that the same prompt may generate different responses when used by different users. Second, the sample size for the human evaluation was small. Nonetheless, this study produced evidence that LLMs such as GPT-4 can be a promising tool for filling the information gap for understanding laboratory tests and various approaches can be used to enhance the quality of the responses.

Conclusions

In this study, we evaluated the feasibility of using generative LLMs to answer common laboratory test result interpretation questions from patients. We generated responses from 5 LLMs—ChatGPT (GPT-4 version and GPT-3.5 version), LLaMA 2, MedAlpaca, and ORCA_mini—for laboratory test questions selected from Yahoo! Answers and evaluated these responses using both automated metrics and manual evaluation. We found that GPT-4 performed better compared to the other LLMs in generating more accurate, helpful, relevant, and safe answers to these questions. We also identified a number of ways to improve the quality of LLM responses from both the prompt and response sides.

Acknowledgments

This project was partially supported by the University of Florida Clinical and Translational Science Institute, which is supported in part by the National Institutes of Health (NIH) National Center for Advancing Translational Sciences under award UL1TR001427, as well as the Agency for Healthcare Research and Quality (AHRQ) under award R21HS029969. This study was supported by the NIH Intramural Research Program, National Library of Medicine (QJ and ZL). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH and AHRQ. The authors would like to thank Angelique Deville, Caroline Bennett, Hailey Thompson, and Maggie Awad for labeling the questions for the question classification model.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

QJ is a coauthor and an active associate editor for the Journal of Medical Internet Research . All other authors declare no other conflicts of interest.

The responses generated by the 5 large language models and the human answers from Yahoo users.

Distribution of the lengths of the responses.

A few observations from the medical experts regarding the accuracy of the large language model responses.

• Healthy people 2030: building a healthier future for all. Office of Disease Prevention and Health Promotion. URL: https://health.gov/healthypeople [accessed 2023-05-09]
• NHE fact sheet. Centers for Medicare & Medicaid Services. URL: https://tinyurl.com/yc4durw4 [accessed 2023-06-06]
• Bauer UE, Briss PA, Goodman RA, Bowman BA. Prevention of chronic disease in the 21st century: elimination of the leading preventable causes of premature death and disability in the USA. Lancet. Jul 2014;384(9937):45-52. [ CrossRef ]
• Centers for Medicare and Medicaid Services (CMS), Centers for Disease Control and Prevention (CDC), Office for Civil Rights (OCR). CLIA program and HIPAA privacy rule; patients' access to test reports. Final rule. Fed Regist. Feb 06, 2014;79(25):7289-7316. [ FREE Full text ] [ Medline ]
• Pillemer F, Price R, Paone S, Martich GD, Albert S, Haidari L, et al. Direct release of test results to patients increases patient engagement and utilization of care. PLoS One. 2016;11(6):e0154743. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Health IT legislation: 21st century cures act. Office of the National Coordinator for Health Information Technology. URL: https://www.healthit.gov/topic/laws-regulation-and-policy/health-it-legislation [accessed 2023-02-19]
• Tsai R, Bell EJ, Woo H, Baldwin K, Pfeffer M. How patients use a patient portal: an institutional case study of demographics and usage patterns. Appl Clin Inform. Jan 06, 2019;10(1):96-102. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Witteman HO, Zikmund-Fisher BJ. Communicating laboratory results to patients and families. Clin Chem Lab Med. Feb 25, 2019;57(3):359-364. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Turchioe MR, Myers A, Isaac S, Baik D, Grossman LV, Ancker JS, et al. A systematic review of patient-facing visualizations of personal health data. Appl Clin Inform. Aug 09, 2019;10(4):751-770. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alpert JM, Krist AH, Aycock RA, Kreps GL. Applying multiple methods to comprehensively evaluate a patient portal's effectiveness to convey information to patients. J Med Internet Res. May 17, 2016;18(5):e112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zikmund-Fisher BJ, Exe NL, Witteman HO. Numeracy and literacy independently predict patients' ability to identify out-of-range test results. J Med Internet Res. Aug 08, 2014;16(8):e187. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Z, Citardi D, Xing A, Luo X, Lu Y, He Z. Patient challenges and needs in comprehending laboratory test results: mixed methods study. J Med Internet Res. Dec 07, 2020;22(12):e18725. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fraccaro P, Vigo M, Balatsoukas P, van der Veer SN, Hassan L, Williams R, et al. Presentation of laboratory test results in patient portals: influence of interface design on risk interpretation and visual search behaviour. BMC Med Inform Decis Mak. Feb 12, 2018;18(1):11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bar-Lev S, Beimel D. Numbers, graphs and words - do we really understand the lab test results accessible via the patient portals? Isr J Health Policy Res. Oct 28, 2020;9(1):58. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Giardina TD, Baldwin J, Nystrom DT, Sittig DF, Singh H. Patient perceptions of receiving test results via online portals: a mixed-methods study. J Am Med Inform Assoc. Apr 01, 2018;25(4):440-446. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Doi T, Langsted A, Nordestgaard BG. Elevated remnant cholesterol reclassifies risk of ischemic heart disease and myocardial infarction. J Am Coll Cardiol. Jun 21, 2022;79(24):2383-2397. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wadström BN, Wulff AB, Pedersen KM, Jensen GB, Nordestgaard BG. Elevated remnant cholesterol increases the risk of peripheral artery disease, myocardial infarction, and ischaemic stroke: a cohort-based study. Eur Heart J. Sep 07, 2022;43(34):3258-3269. [ CrossRef ] [ Medline ]
• Chu SK, Huang H, Wong WN, van Ginneken WF, Wu KM, Hung MY. Quality and clarity of health information on Q and A sites. Libr Inf Sci Res. Jul 2018;40(3-4):237-244. [ CrossRef ]
• Oh S, Yi YJ, Worrall A. Quality of health answers in social Q and A. Proc Assoc Inf Sci Technol. Jan 24, 2013;49(1):1-6. [ CrossRef ]
• Tao D, Yuan J, Qu X, Wang T, Chen X. Presentation of personal health information for consumers: an experimental comparison of four visualization formats. In: Proceedings of the 15th International Conference on Engineering Psychology and Cognitive Ergonomics. 2018. Presented at: EPCE '18; July 15-20, 2018;490-500; Las Vegas, NV. URL: https://link.springer.com/chapter/10.1007/978-3-319-91122-9_40
• Struikman B, Bol N, Goedhart A, van Weert JC, Talboom-Kamp E, van Delft S, et al. Features of a patient portal for blood test results and patient health engagement: web-based pre-post experiment. J Med Internet Res. Jul 20, 2020;22(7):e15798. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kopanitsa G. Study of patients' attitude to automatic interpretation of laboratory test results and its influence on follow-up rate. BMC Med Inform Decis Mak. Mar 27, 2022;22(1):79. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zikmund-Fisher BJ, Scherer AM, Witteman HO, Solomon JB, Exe NL, Fagerlin A. Effect of harm anchors in visual displays of test results on patient perceptions of urgency about near-normal values: experimental study. J Med Internet Res. Mar 26, 2018;20(3):e98. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morrow D, Azevedo RF, Garcia-Retamero R, Hasegawa-Johnson M, Huang T, Schuh W, et al. Contextualizing numeric clinical test results for gist comprehension: implications for EHR patient portals. J Exp Psychol Appl. Mar 2019;25(1):41-61. [ CrossRef ] [ Medline ]
• Tian S, Jin Q, Yeganova L, Lai PT, Zhu Q, Chen X, et al. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Brief Bioinform. Nov 22, 2023;25(1):bbad493. [ CrossRef ] [ Medline ]
• Cadamuro J, Cabitza F, Debeljak Z, De Bruyne SD, Frans G, Perez SM, et al. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) working group on artificial intelligence (WG-AI). Clin Chem Lab Med. Jun 27, 2023;61(7):1158-1166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Munoz-Zuluaga C, Zhao Z, Wang F, Greenblatt MB, Yang HS. Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine. Clin Chem. Aug 02, 2023;69(8):939-940. [ CrossRef ] [ Medline ]
• Zhang Z, Lu Y, Kou Y, Wu DT, Huh-Yoo J, He Z. Understanding patient information needs about their clinical laboratory results: a study of social Q and A site. Stud Health Technol Inform. Aug 21, 2019;264:1403-1407. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Z, Lu Y, Wilson C, He Z. Making sense of clinical laboratory results: an analysis of questions and replies in a social Q and A community. Stud Health Technol Inform. Aug 21, 2019;264:2009-2010. [ CrossRef ] [ Medline ]
• Kurstjens S, Schipper A, Krabbe J, Kusters R. Predicting hemoglobinopathies using ChatGPT. Clin Chem Lab Med. Feb 26, 2024;62(3):e59-e61. [ FREE Full text ] [ CrossRef ] [ Medline ]
• He Z, Tian S, Erdengasileng A, Hanna K, Gong Y, Zhang Z, et al. Annotation and information extraction of consumer-friendly health articles for enhancing laboratory test reporting. AMIA Annu Symp Proc. 2023;2023:407-416. [ FREE Full text ] [ Medline ]
• Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. Feb 15, 2020;36(4):1234-1240. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alsentzer E, Murphy JR, Boag W, Weng WH, Jin D, Naumann T, et al. Publicly available clinical BERT embeddings. arXiv. Preprint posted online on April 6, 2019. [ FREE Full text ] [ CrossRef ]
• Beltagy I, Lo K, Cohan A. SciBERT: a pretrained language model for scientific text. arXiv. Preprint posted online on March 26, 2019. [ FREE Full text ]
• Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc. Oct 15, 2021;3(1):1-23. [ FREE Full text ] [ CrossRef ]
• OpenAI. GPT-4 technical report. arXiv. Preprint posted online on March 15, 2023. [ FREE Full text ]
• Ye J, Chen X, Xu N, Zu C, Shao Z, Liu S, et al. A comprehensive capability analysis of GPT-3 and GPT-3. arXiv. Preprint posted online on March 18, 2023. [ FREE Full text ]
• Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et al. Llama 2: open foundation and fine-tuned chat models. arXiv. Preprint posted online on July 18, 2023. [ FREE Full text ]
• Han T, Adams LC, Papaioannou JM, Grundmann P, Oberhauser T, Löser A, et al. MedAlpaca -- an open-source collection of medical conversational AI models and training data. arXiv. Preprint posted online on April 14, 2023. [ FREE Full text ]
• orca_mini_3b. Hugging Face. URL: https://huggingface.co/pankajmathur/orca_mini_3b [accessed 2023-12-04]
• LangChain: introduction and getting started. Pinecone. URL: https://www.pinecone.io/learn/series/langchain/langchain-intro/ [accessed 2023-12-04]
• Papineni K, Roukos S, Ward T, Zhu WJ. BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002. Presented at: ALC '02; July 7-12, 2002;311-318; Philadelphia, PA. URL: https://dl.acm.org/doi/10.3115/1073083.1073135 [ CrossRef ]
• Post M. A call for clarity in reporting BLEU scores. arXiv. Preprint posted online on April 23, 2018. [ FREE Full text ] [ CrossRef ]
• Banerjee S, Lavie A. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the 2005 ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 2005. Presented at: WIEEMMTS '05; June 29, 2005;65-72; Ann Arbor, MI. URL: https://aclanthology.org/W05-0909 [ CrossRef ]
• Lin CY. ROUGE: a package for automatic evaluation of summaries. In: Lin CY, editor. Text Summarization Branches Out Internet. Barcelona, Spain. Association for Computational Linguistics; 2004;74-81.
• Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: evaluating text generation with BERT. arXiv. Preprint posted online on April 21, 2019. [ FREE Full text ]
• Wang T, Yu P, Tan XE, O'Brien S, Pasunuru R, Dwivedi-Yu J, et al. Shepherd: a critic for language model generation. arXiv. Preprint posted online on August 8, 2023. [ FREE Full text ]
• Dubois Y, Li X, Taori R, Zhang T, Gulrajani I, Ba J, et al. AlpacaFarm: a simulation framework for methods that learn from human feedback. arXiv. Preprint posted online on May 22, 2023. [ FREE Full text ]
• Bartko JJ. The intraclass correlation coefficient as a measure of reliability. Psychol Rep. Aug 1966;19(1):3-11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gamer M, Lemon J, Singh IF. irr: various coefficients of interrater reliability and agreement. Cran R Project. 2019. URL: https://cran.r-project.org/web/packages/irr/index.html [accessed 2023-12-12]
• Human subject regulations decision charts: 2018 requirements. Office for Human Research Protection. Jan 20, 2019. URL: https://tinyurl.com/3sbzydm3 [accessed 2024-04-03]
• Jin Q, Leaman R, Lu Z. Retrieve, summarize, and verify: how will ChatGPT affect information seeking from the medical literature? J Am Soc Nephrol. Aug 01, 2023;34(8):1302-1304. [ CrossRef ] [ Medline ]

Abbreviations

Edited by B Puladi; submitted 23.01.24; peer-reviewed by Y Chen, Z Smutny; comments to author 01.02.24; revised version received 17.02.24; accepted 06.03.24; published 17.04.24.

©Zhe He, Balu Bhasuran, Qiao Jin, Shubo Tian, Karim Hanna, Cindy Shavor, Lisbeth Garcia Arguello, Patrick Murray, Zhiyong Lu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

How do I style the title of a fairy tale?

Fairy tales are typically enclosed in quotation marks, in the style of other short-form works.

Some people may not know that Disney’s 1989 film The Little Mermaid is loosely based on Hans Christian Andersen’s “The Little Mermaid,” first published in 1837.    Some of the lesser-known tales collected by Jacob and Wilhelm Grimm include “The White Snake” and “The Robber Bridegroom.” 

Use quotation marks when you are referring to a specific version of a fairy tale published as part of a collection.

“Bluebeard” opens with the image of “a man who lived in a forest with his three sons and beautiful daughter” (610).  Work Cited “Bluebeard.” The Complete Fairy Tales of the Brothers Grimm , translated by Jack Zipes, 3rd ed., Bantam Books, 2003, pp. 610–12.

However, if a fairy tale was published as a long-form work, such as a book or play, use italics.

Just as captivating as the story itself are the illustrations that accompany Wilhelm Grimm’s Dear Mili , the work of the famed author and illustrator Maurice Sendak.  Work Cited Grimm, Wilhelm. Dear Mili . Translated by Ralph Manheim, illustrated by Maurice Sendak, Farrar, Straus and Giroux, 1988.

Generic references to fairy tales are not italicized or enclosed in quotation marks. 

The Cinderella story has been adapted by a number of authors, among them Giambattista Basile and Charles Perrault. 

Help | Advanced Search

Computer Science > Machine Learning

Title: scaling laws for data filtering -- data curation cannot be compute agnostic.

Abstract: Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. In recent times, data curation has gained prominence with several works developing strategies to retain 'high-quality' subsets of 'raw' scraped data. For instance, the LAION public dataset retained only 10% of the total crawled data. However, these strategies are typically developed agnostic of the available compute for training. In this paper, we first demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data. To address this quality-quantity tradeoff ($\texttt{QQT}$), we introduce neural scaling laws that account for the non-homogeneous nature of web data, an angle ignored in existing literature. Our scaling laws (i) characterize the $\textit{differing}$ 'utility' of various quality subsets of web data; (ii) account for how utility diminishes for a data point at its 'nth' repetition; and (iii) formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them. Our key message is that data curation $\textit{cannot}$ be agnostic of the total compute that a model will be trained for. Our scaling laws allow us to curate the best possible pool for achieving top performance on Datacomp at various compute budgets, carving out a pareto-frontier for data curation. Code is available at this https URL .

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

https://www.nist.gov/publications/evaluating-security-crystals-dilithium-quantum-random-oracle-model

Evaluating the security of CRYSTALS-Dilithium in the quantum random oracle model

Download paper, additional citation formats.

• Google Scholar