andrew ng reading research papers

How to Read Research Papers (Andrew Ng)

During FastAI part 2 we have to read many research papers. I’ve personally found this a bit of a chore, and I have lacked motivation. I googled around to see if there were ways to do this better. I found several blog posts and youtube videos. The best I found was a great lecture from Andrew Ng (Stanford, deeplearning.ai): Career Advice / Reading Research Papers [YouTube]

I made a summary of it, which I will share here:

How to Read Research Papers

• Worst strategy: reading from the first word until the last word!
• Especially in deep learning, there are a lot of papers where the entire paper is summarized in one or two figures.
• You can often get a good understanding about what the whole paper is about without reading much of the text.
• Part of the process of writing papers is convincing the reviewers that your paper is worthy of acceptance, so you find that the abstract, intro, and conclusion are where the authors summarize their work really carefully. These are therefore the most useful parts to read.
• Neural network architectures are often written up in a table.
• Maybe also skim Related work section, for context or see if there is something you have read before.
• Read the Paper, but skip the maths
• In cutting edge papers we don’t always know what is really important and what isn’t important.
• Some great, highly cited papers have some parts which are groundbreaking and other parts which later turn out to be unimportant, but at the time the paper was written the authors could not know.
• Maybe what was the key part of the algorithm wasn’t what the authors thought.

Questions to Keep in Mind

• What did the authors try to accomplish?
• What were the key elements of the approach?
• What can you use yourself?
• What other references do you want to follow?

Deeper Understanding

• If you want to make sure you understand the maths of a paper, read through it and make some notes then try to re-derive from scratch on a blank piece of paper.
• As you get good at doing this you will gain the ability to derive novel algorithms yourself.
• Learn from the masters, not from their students.
• Lightweight: download and run their open-source code (assuming they have it).
• Deeper: reimplement their code from scratch.

General Advice: Steady reading, not short bursts. Better off reading 2-3 papers a week for the next year, than cramming everything in over Christmas.

Where to Find Papers

To keep up with the state of the art:

• ML Subreddit: r/MachineLearning
• Top ML Conferences: NeurIPS , ICML , ICLR
• ArXiv Sanity
• Friends / Online Community (e.g. fastai forums).

Other Stuff

• This older thread from fastai has great info too: Reading deep learning papers
• I’ve been using Mendeley for organizing and reading papers on my computer and iPad (it syncs across devices).

This is awesome summary. Thank you.

Thanks for sharing, very helpful summary!

Stanford Machine Learning Group

Our mission is to significantly improve people's lives through our work in artificial intelligence.

Professor Andrew Ng started the Stanford ML Group in 2003, which has since expanded to the broader Stanford ML Group .

This page describes the research directed by Professor Ng.

We work on developing AI solutions for a variety of high-impact problems

Earth Observation Dataset for Methane Source Identification

Deforestation driver classification using satellite imagery.

Solar Forecasting

Calibrated probabilistic solar irradiance forecasting.

Oil and gas infrastructure mapping in aerial imagery.

Chest X-Ray Transformation Dataset And Competition

CheXpedition

Generalizability of top chest X-ray models on real world challenges.

Probabilistic Prediction with Gradient Boosting

A Large Chest X-Ray Dataset And Competition

ECG Arrhythmia

Cardiologist-level arrythmia detection from ECG signals.

Diagnosis of abnormalities from Knee MRs

PPG Arrhythmia

Arrythmia detection from ambulatory free-living PPG signals.

Chest radiograph diagnosis of multiple pathologies

Introducing a large dataset for abnormality detection from bone x-rays.

Countdown Regression

New approach to probabilistic time to event predictions.

Radiologist-level pneumonia detection from chest X-rays.

Palliative Care

Using Electronic Health Record Data to direct palliative care resources.

Designing natural language models to detect writing errors and provide feedback.

Swati Dube Batra

Program Manager

Jeya Maria Jose

Jeremy Irvin

PhD Candidate

Yixing Jiang

PhD Student

Adam Coates

Alan asbeck.

Anand Avati

Andrew maas, andrew saxe, ashutosh saxena.

Awni Hannun

Brody huval.

Honglak Lee

Ilya sutskever, jiquan ngiam, morgan quigley, pieter abbeel.

Pranav Rajpurkar

Rajat raina, richard socher, sameep tandon.

Sharon Zhou

Zico Kolter

Bootcamp current + alumni.

Nomin-Erdene Bayarsaikhan

Abdulaziz Abdulrahman S Alharbi

Aakriti Lakshmanan

Eish Maheshwari

Melanie Zhang

Riya Dulepet

Santino Ramos

Ines Dormoy

Jinyoung Kim

Apoorva Dixit

Mabel Jiang

Ishan Sabane

Benjamin Yan

Ethan Hellman

Pratyush Muthukumar

Pura Peetathawatchai

Joanne Zhou

Haijing Zhang

Manuka Stratta

Ji Hun Wang

Spencer Paul

Quentin Hsu

Maya Srikanth

Eric Frankel

James Zheng

Daniella Hacco Grimberg

Felipe Godoy

Ayush Singla

Beri Kohen Behar

Muhammad Ahmed Chaudhry

Sahil Tadwalkar

Aditya Gulati

Alex Donovan

Finsam Samson

Sameer Khanna

Vivek Shankar

Vrishab Krishna

Xiaoli Yang

Nicholas Lui

Timothy Dai

Suhas Chundi

Langston Nashold

Jake Silberg

Matt Kolodner

Sarthak Kanodia

David Dadey

Gil Kornberg

Raghav Samavedam

Sergio Charles

Collin Kwon

Benjamin Liu

Cecile Loge

Daniel Michael

Ellie Talius

Niveditha Iyer

Pujan Patel

Raj Palleti

Rehaan Ahmad

Lyron Co Ting Keh

Jake Taylor

Mauricio Wulfovich

Chris Rilling

Andrew Yang

Bryan Gopal

Gautham Raghupathi

Nhi Truong Vu

Pranav Sriram

Soham Gadgil

Sam Masling

Erfan Rostami

Andrew Hwang

Michelle Bao

Eric Matsumoto

Niranjan Balachandar

Pratham Soni

Richard Wang

Stephanie Zhang

Jared Isobe

Adriel Saporta

Ishaan Malhi

Rayan Krishnan

William Ellsworth

Andrew Ying

Heejung Chung

Shawn Zhang

Sasankh Munukutla

Christopher Cross

Akshay Smit

Dahlia Radif

Damir Vrabac

Jiangshan Li

Oishi Banerjee

Saahil Jain

Viswesh Krishna

Anirudh Joshi

Cheuk To Tsui

Hari Sowrirajan

John Peruzzi

Keyur Mithawala

Michael Zhang

Nick Phillips

Sonja Johnson-Yu

Eric Zelikman

Cooper Raterink

Neel Ramachandran

Neethu Renjith

Ashwin Agrawal

Christian Rose

Kaushik Ram Sadagopan

Yancheng Li

Andrew Kondrich

Vincent Liu

Jabs Aljubran

Will Deaderick

Anuj Pareek

Jingbo Yang

Mark Sabini

Nathan Dass

Vinjai Vale

Amirhossein Kiani

Amit Schechter

Bora Uyumazturk

Chloe O'Connell

Nishit Asnani

Rebecca Gao

Soumya Patro

Bryan Casey

James Rathmell

Zach Harned

Behzad Haghgoo

Ben Cohen-Wang

Chris Chute

Nidhi Manoj

Philip Hwang

Robin Cheong

Silviana Ciurea Ilcus

Allison Park

Andrew Huang

Atli Kosson

Henrik Marklund

Jessica Wetstone

Matthew Sun

Michael Bereket

Nicholas Bien

Norah Borus

Shubhang Desai

Suvadip Paul

Thao Nguyen

Tanay Kothari

Aarti Bagul

Brandon Yang

Hershel Mehta

Collaborating Faculty

Curt Langlotz

Matt Lungren

Jeanne Shen

Sanjay Basu

Bhavik Patel

Kristen Yeom

Leanne Williams

Utkan Demirci

Gozde Durmus

Sidhartha Sinha

Catherine Hogan

Sebastian Fernandez-Pol

Yaso Natkunam

Mitchell Rosen

Geoff Tison

Andrew Beam

Rob Jackson

Ram Rajagopal

Daniel Rodriguez

Gavin McNicol

Chris Field

Jackelyn Hwang

Peter Kitanidis

Etienne Fluet-Chouinard

Duncan Watson-Parris

Work with us, by working with our group, you will:.

• Work on important problems in areas such as healthcare and climate change, using AI.
• Build and deploy machine learning / deep learning algorithms and applications.

Here are some values that we would like to see in you:

• Hard work : We expect you to have a strong work ethic. Many of us work evenings and weekends because we love our work and are passionate about the AI mission. We also value velocity, and like people that get things done quickly.
• Flexibility : You should be willing to dive into different facets of a project. For example, besides developing machine learning algorithms, you may also need to work on data acquisition, conduct user interviews, or do frontend engineering. This may also require going outside your comfort zone, and learning to do new tasks in which you’re not an expert.
• Learning : You should have a strong growth mindset, and want to learn continuously. This can involve reading books, taking coursework, talking to experts, or re-implementing research papers. We will also prioritize your learning and help point you in the right direction; but you need to put in the work to take advantage of this.
• Teamwork : We work together in small teams. You are expected to support and collaborate with others; in turn you will also receive support from your teammates.

Prerequisites

You should have a strong ML background, or a strong software engineering background.

• ML/AI background : You have a solid background in probability and linear algebra, and have done well in AI/ML coursework. For example, Stanford students should have taken CS229 before applying. Previous ML/AI research experience would be a plus but is not required.
• Software engineering background : We also encourage engineers without much AI background who are interested in developing ML applications to apply. Applicants should have made significant contributions to software projects in the past, for example through developing software systems at a company or through significant open source contributions.

Please see below for how to apply to work with our group. Due to a high number of applicants we may be unable to respond to individual emails. We can only work with Stanford students at this time.

Stanford PhD Students

• Stanford PhD students interested in rotating with Professor Ng should email us at [email protected] using their Stanford email with the subject line “FirstName LastName PhD Rotation”.

Other Stanford Students

• We encourage other Stanford students who want to work with us to apply to either the AIHC , AICC , or Medical AI bootcamp.
• Outside of coursework, we expect this to be your primary academic activity.
• As it takes time to familiarize oneself with a research project and to make significant contributions, we expect that students will be involved for at least two quarters, with a strong preference for those who can potentially stay involved for the full school year.

If you're looking to partner or work with us, contact us at

[email protected].

Dr. Andrew Ng is a globally recognized leader in AI (Artificial Intelligence). He is Founder of DeepLearning.AI , Founder & CEO of Landing AI , General Partner at AI Fund , Chairman and Co-Founder of Coursera and an Adjunct Professor at Stanford University’s Computer Science Department.

As a pioneer in machine learning and online education, Dr. Ng has changed countless lives through his work in AI, and has authored or co-authored over 200 research papers in machine learning, robotics and related fields. In 2023, he was named to the Time100 AI list of the most influential AI persons in the world.

Get Andrew’s letters delivered to your inbox every week.

Publications

Origins of the Modern MOOC (xMOOC)

Online education has been around for decades,with many universities offering online courses to a small, limited audience.What changed in 2011 was scale and availability, when Stanford University offered three courses free to the public, each garnering signups of about 100,000 learners or more.The launch of these three courses, taught by Andrew Ng, Peter Norvig, Sebastian […]

Mechatronic design of an integrated robotic hand

Historically, robotic hand research has tended to focus on two areas: severely underactuated hands, and high-degree-offreedom fully actuated hands. Comparatively little research has been done in between those spaces. Furthermore, despite the large number of robotic hand designs that have been proposed in the past few decades, very few robot hands are available for purchase […]

Deep Learning with COTS HPC Systems

Scaling up deep learning algorithms has been shown to lead to increased performance in benchmark tasks and to enable discovery of complex high-level features. Recent efforts to train extremely large networks (with over 1 billion parameters) have relied on cloud- like computing infrastructure and thousands of CPU cores. In this paper, we present technical details […]

Deep Learning and Unsupervised Feature Learning 

Machine learning and AI through large scale brain simulations (artificial neural networks).

• Machine Learning Specialization
• Machine Learning Engineering for Production (MLOps) Specialization
• Deep Learning Specialization
• AI For Everyone

Stack Exchange Network

Stack Exchange network consists of 183 Q&A communities including Stack Overflow , the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

As a intermediate in Deep Learning, from which research paper should I start reading from? [closed]

I have done Andrew Ng's ML and DL courses, and some projects and implemented some important ML algorithms from scratch. Now reading the deep learning book. <=(Edited)

I want to start from the beginning (in terms of reading research papers), i.e, deep feedforward networks, regularization techniques,{then maybe conv nets and others}etc, etc and some tips on how to tackle the difficulty in understanding it. Thank You.

• machine-learning
• deep-learning

• 3 $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$ –  Community Bot Commented Oct 21, 2022 at 18:21
• $\begingroup$ For your question, I strongly suggest you start from this book: deeplearningbook.org instead of papers. $\endgroup$ –  Allonsy Jia Commented Oct 21, 2022 at 20:00
• 1 $\begingroup$ The best research paper for you to read will depend on your specific interests and background knowledge in deep learning. However, a good place to start might be the CS230 Deep Learning, which covers a wide range of topics in deep learning and provides links to further reading on each topic.[ cs230.stanford.edu/] $\endgroup$ –  Faizy Commented Oct 21, 2022 at 22:11
• 1 $\begingroup$ @Faizy cs230.stanford.edu $\endgroup$ –  Jaume Oliver Lafont Commented Oct 22, 2022 at 20:31

4 Answers 4

At your stage, I don't think jumping straight into reading research papers would be efficient. Generally, reading textbooks/review-articles, or simply watch a couple introductory youtube courses would do a better job at getting you up to speed with the background knowledge. Of course, you can always find a project that interests you and try to incorporate some elements of ML into it, which allows you to naturally learn ML at the same time.

Some standard introductory textbooks/courses are:

• The deep learning textbook , more theoretical driven
• Andrew Ng's courses on youtube, more application driven

which should cover the topics you mentioned.

If you want to focus on a specific topic (e.g. ConvNets, transformers, recurrent networks, etc.), it's generally helpful to find a recent review article on this topic and read through it. This is just to understand the current state of the field, and you can then read specific papers that interests you with this contextual knowledge in mind. Note these fields are moving so fast that certain seminal papers are no longer hugely relevant (e.g. many network architectures and training methods proposed in the classic AlexNet paper are outdated.)

• $\begingroup$ I have done Andrew Ng's ML and DL courses, and some projects and implemented some important ML algorithms from scratch. Now actually reading the deep learning book. $\endgroup$ –  vivian.ai Commented Oct 23, 2022 at 3:17
• $\begingroup$ Also, wiki is free XD. $\endgroup$ –  MathematicsBeginner Commented Apr 19 at 11:09

There is no "beginning" with research papers. Papers are published as they are ready, in no particular order with respect to complexity or topic. I think you just have to jump in.

Pick papers that match your interests. Look things up as you read to understand. You might need to brush up on Math.

Here is a list of resources where you can find research papers to start. I am sure you can find others as you learn what interests you.

The good news is that there are many freely available educational resources online and at your local library. Here are some that I used to get me started:

• Kaggle: Intro to Deep Learning
• Kaggle: Computer Vision
• Pattern Recognition & Machine Learning (Chapter 5)
• Machine Learning (Chapter 4)

Good luck and happy learning!

P.S. I know these are not research papers, but I would encourage you to start with these anyway.

• $\begingroup$ Thanks, some of the links were helpful, but I want an answer specific to my question. $\endgroup$ –  vivian.ai Commented Oct 21, 2022 at 15:28

Not sure what to recommend, since you say "from the beginning" in the text but "intermediate" in the title...

Anyway, for the "then maybe conv nets" part, there is a tutorial from 2021 that relates convolutional networks with the matched filter, a well grounded technique in signal processing. I find this a great idea; depending on your background, it may be interesting for you as well.

https://arxiv.org/abs/2108.11663

Is this the kind of papers you are looking for?

• $\begingroup$ In terms of reading research papers, I meant the beginning. $\endgroup$ –  vivian.ai Commented Oct 23, 2022 at 3:11

Not the answer you're looking for? Browse other questions tagged machine-learning deep-learning research .

• Featured on Meta
• Site maintenance - Mon, Sept 16 2024, 21:00 UTC to Tue, Sept 17 2024, 2:00...
• User activation: Learnings and opportunities
• Join Stack Overflow’s CEO and me for the first Stack IRL Community Event in...

Hot Network Questions

• How would platypus evolve some sort of digestive acid?
• Is it a correct rendering of Acts 1,24 when the New World Translation puts in „Jehovah“ instead of Lord?
• BASH - Find file with regex - Non-recursively delete number-only filenames in directory
• Concerns with newly installed floor tile
• How can I analyze the anatomy of a humanoid species to create sounds for their language?
• MSSQL - query runs for too long when filtering after a certain date
• How can a microcontroller (such as an Arduino Uno) that requires 7-21V input voltage be powered via USB-B which can only run 5V?
• Father and Son treasure hunters that ends with finding one last shipwreck (childrens/young adult)
• How to prove that the Greek cross tiles the plane?
• Is this a misstatement of Euclid in Halmos' Naive Set Theory book?
• What about the other 35 children who were born in the same manner in The Umbrella Academy. Do we hear what happened to them in the comic or TV Show?
• Should I write an email to a Latino teacher working in the US in English or Spanish?
• What is the rationale behind 32333 "Technic Pin Connector Block 1 x 5 x 3"?
• Should I change advisors because mine doesn't object to publishing at MDPI?
• NSolve uses all CPU resources
• Is Produce Flame a spell that the caster casts upon themself?
• Why would the GPL be viral, while EUPL isn't, according to the EUPL authors?
• Philosophical dogma hindering scientific progress?
• Second cohomology of nonabelian finite simple group vanishes.
• Why was Panama Railroad in poor condition when US decided to build Panama Canal in 1904?
• Problem to find 32768 Hz crystal with 0.1 pF of Cshunt
• Why is resonance such a widespread phenomenon?
• Is this map real?
• What film is it where the antagonist uses an expandable triple knife?

How to read Machine Learning and Deep Learning Research papers

Tips on preparing Literature survey of a field and how to read a ML / DL research papers. The 3 pass method to read ML or DL research papers is discussed.

Jul 31, 2021 • Sai Amrit Patnaik • 23 min read

research   reading_papers

Introduction

Dynamically expanding field of deep learning, why to read research papers, step 1: assembling all available resources, step2 - filtering out relevant and irrelevant resources, step3: taking systematic notes, organization of a paper, how to read a research paper, second pass, important questions to answer.

How to read a research paper, is probably the most important skill which any one who is into research or even anyone who wishes to be updated in the field with latest advancements has to master. When someone thinks of starting out in a domain, the first advice that comes is to look for relevant literature in the domain and read papers to develop an understanding of the domain. Papers are the most reliable and updated source of information about a particular domain. A research paper is a result of days of brainstorming of ideas, and structured and systematic experimentation to express an approach.

But why is reading papers considered such an important skill to be learnt ? Why is even reading papers necessary ? Let’s take on some motivation as to why is reading papers important to keep-up with the latest advances.

This article is the summary of a talk that I delivered for the Introductory Paper Reading Session generously supported by Weights and Biases whose recorded version can be found here and slides can be found here .

The field of deep learning has grown very rapidly in the recent years. We can quantify growth in a field by theh number of papers that come up everyday. Here is an illustration from one of the studies by ArXiv which is one of the platform where almost all of the papers, whether published or unpublished are putup.

From the figure we can see that the average no of papers has grown to 5X averaging from 300 papers per month in 2017 to around 1500 papers per month in 2019. The figure would probably be close to or above 2k papers per month in 2021. This is a huge number of papers coming up everyday. This shows how dynamic the field is at the current time and it is just growing exponentially in terms of number of papers and amount of new ideas and experiments coming up everyday.

Let’s look at another figure from another study by arXiv

From the figure, the number of papers in the field of Computer Science has grown like a step exponential curve and we see that around 36k papers come out each year out of which around 24k of them as we saw in the previous section are in the field of ML and DL. We can also see in both the figures that the DL field in Green and CV in yellow are among the dominant areas in terms of percentages of papers coming out every year since the early 2000s while the field of CV has grown and opened up a lot after 2012 probably when the prominent work on Image classification by deep networks showed significant performance. These studies definitely speak how fast the field of computer Science is growing and amongst it, how the sub areas related to Machine Learning and Deep Learning are evolving too.

I hope these give a good idea of how fast the field has been evolving and would continue to evolve even faster in the future. But in this fast evolving field, How can we keep up with the pace and develop a expertise in the field ?

Quoting Dr. Jennifer Raff , To form a truly educated opinion on a scientific subject, you need to become familiar with current research in that field. And to be able to distinguish between good and bad interpretations of research, you have to be willing and able to read the primary research literature for yourself.

• To have a better grasp and understanding of the field: For a particular field, there may be a lot of video lectures and books but with the rate at which the field has been growing, no book or video lecture can accomodate the latest information as soon as they get published. So research papers provide the most updated and reliable information in the field.
• To be able to contribute to the field in terms of novel ideas: When we start working in a field, the first thing that we are advised to do is to do an extensive literature survey, going through all of the latest papers that have come up in the field till date. That is advised because we can have a very good understanding of the directions of works in the field and how the people actively working in the field are thinking by reading papers. Only then we can start coming up with our own ideas to experiment upon.
• To develop confidence in the field: Once we start learning about the latest works in the field and we start to develop a good understanding by performing a extensive literature survey, we start developing more confidence to perform more experiments and exploring deeper in the field.
• Most condensed and authentic source of latest knowledge in the field: A reseach paper comes out of days and months, or some times even years of brainstorming of ideas, performing extensive experiments and validating the expected outcomes. The condensed experiments and thoughts is what is best expressed in a research paper that the authors write. Any new content that comes in the field in terms of state-of-the-art works is through research papers. Research papers are the source through which works that push the limits of knowledge in a field come up.

Motivated enough ?

Now that we have attained enough motivations as to why we should read research papers, lets look at how to do literature survey in a domain.

Let’s do it !

Literature survey of a domain

The basic steps to perform literature survey in a field are the following:

• Assemble collections of resources in the form of research papers, Medium articles, blog posts, videos, GitHub repository etc.
• Conduct a deep dive to classify the relevant and irrelevant material.
• Take structured notes that summarises the key discoveries, findings and techniques within a paper.

We shall take Pose Estimation as a example domain and understand each step.

First of all we collect all the resources in the form of blog posts, github repositories, medium articles and research papers available in the field, for our case it’s pose estimation. The important question here is, where can we find relevant resources in the field ?

Following are sources where we can find the latest papers and resources:

• Twitter : We can follow top researchers, groups and labs actively working and publishing in our field of domain and be updated with what they are currently working on.
• ML subreddit
• arXiv : Platform where almost all of the papers be it accepted to a conference or not, are uploaded.
• Arxiv Sanity Preserver : Created by Anderj Karpathy which used ML techniques to suggest relevant papers based on previous searches and interests.
• Papers With Code : Redirects to the paper’s abstract page on arXiv, open source implementation of the papers along with links to datasets used and a lot of other analysis and meta information like the current state-of-the art method, comparision of performance of all previous methods in the field e.t.c.
• Top ML, DL Conferences ( CVPR , ICCV , NeurIPS , ICML , ICLR etc): Proceedings of the following conferences are a great place to look for latest accepted works in the domains accepted by the conference.
• Google Search

Once listed down all the papers that we wish to look at and all resources we could find be it relevant or irrelevant, a table of this format shown in figure 3 can be prepared and in the first column, all the resources collected can be listed down.

Once listed down all the resources and prepared a table like the one shown in figure 3, the next step is to keep the relevant resources and reject the un-necessary ones which may not be directly related to what we want to work on our our research objectives. Follow the following steps to do that:

For all the resources listed down, finish 10% of reading of each resource or research paper(first pass reading, we will discuss about it later). If we find it not related to our research objective, we can reject it.

If that resurce is related to our objective and is relevant and important to us, do a complete full pass reading over the paper. From the references, if we find any other relevant reference then mark those in the original paper and add them to the list and repeat the same over this new paper or resource now.

So after this, this is what the final table might look like this,

Notice that the 2nd, 4th and 6th resources were important and relevant so we read it in detail but the other oned were not very important or the entire thing was not relevant so we read through some portion of each, whatever was necessary and left the rest.

Such a table can be really useful when we return back to it after some months or years to look for or recall what we have read or the papers we have already looked at and rejected. It helps us to save a lot of time iterating over unnecessary resources and helps us effectively dedicate time to the useful resources.

Once decided on which papers to read, this step depends on the individial about they want to go about taking notes. I personally follow a annotation tool to annotate different sections of the paper according to my comfort. I prepare some flow charts for the entire flow of the paper, write some explaining notes on the paper and summarise each paper to the best of my understanding to a github repository. Here I would Like to give a shoutout to Akshay Uppal who had generously shared his blogpost with his annotated version of the MLP Mixer paper for the Weights and Biases paper reading group . I also wish to share one of my repositories of literature survey when I started working on the field of face spoofing.

Tip: You can use your own ways of making yourself comfortable with the content and taking notes either on github, notion or google docs e.t.c to organise notes.

The majority of papers follow, more or less, the same convention of organization:

• Title: Hopefully catchy ! Includes additional info about the authors and their institutions.
• Abstract: High level summary of the entire work of the paper.
• Introduction: Background info on the field and related research leading up to this paper.
• Related works: Describe the already existing literature on the particular domain.
• Methods: Highly detailed section on the study that was conducted, how it was set up, any instruments used, and finally, the process and workflow.
• Results: Authors talk about the data that was created or collected, it should read as an unbiased account of what occurred.
• Discussions: Here is where authors interpret the results, and convince the readers of their findings and hypothesis.
• References: Any other work that was cited in the body of the text will show up here.
• Appendix: More figures, additional treatments on related math, or extra items of interest can find their way in an appendix.

Finally coming to the most awaited section of the blogpost !

Now that we know about the different sections of a paper, to understand how to read a paper, we need to understand how a author writes a paper. The intension of an author writing a paper is to get it accepted at a conference. In conferences, reviewers read all the submissions and take a decision based on the work and the scope and expectations of the conference. Let’s have a quick understanding of how the review process works at a very high level.

Warning: Reading a paper sequentially one section after another is not a good option.

In most of the top conferences, there are two submission deadlines: one, the abstract submission deadline. Second, the actual paper submission deadline. So why exactly are there 2 deadlines ? A separate deadline for abstract even before the actual paper deadline definitely implies that abstract is an important part of the paper. But Why is abstract important ?

Note: While Considering to submit for a conference, always note they have 2 deadlines: One, for abstract submission. Second, for the full paper submission.

Every year, a lot of papers get submitted to each conference. The number of submissions are in tens of thousands and it is not feasible to read through all the papers irrespective of how many reviewers the conference can have. So to make the review process easier and quicker, there is a guideline how different sections of a paper must be written and the reviewer also reads in that same pattern.

The first level of review is always the abstract filtering . The abstract is supposed to summarise the entire work briefly and it should clearly state the problem statement and the solution very briefly. If the abstract doesnot satisfy these criterias, the paper gets rejected in this filtering. So the abstract should clearly expain the gist of the work. Hence while reading paper too, the abstract is the place where we can find the gist of the paper clearly and briefly. Hence the abstract is read first to get an overall idea of the entire work. The authors also spend a lot of efforts in getting one figure which gives a visual illustration of the entire approach or a complete flow chart of the entire work. Even this figure contains a gist of the entire method of the paper. The authors try to condense and pack of information about thier work in a single figure.

Note: The abstract is one of the most important sections in a paper and it explains the entire gist of the paper in brief and the most important figure summarises the method adopted.

The reviewers then read the introduction section as it should explain the problem statement in a detailed way and the main proposal of the paper and the contributions. Immediately after this section, once you know what the paper is assuming, the conclusion section tells about the conclusion of the work and whether the assumptions and expectations presented in the introduction are satisfied or not.

Note: The introduction section is supposed to explain the problem statement in detail and the major contributions of the paper. We get to know the intent of the author from this section. The Conclusion section validates the assumptions and propositions given in the introduction through experiments and proofs.

After validating that the assumed propositions have been validated successfully, the method section is seen in detail to see what approach was taken to acheive the goal. In the discussion section, the experiments are explained as to why exactly the proposed method works. This is basically how a reviewer reads a paper and it is the same approach that is to be taken by a reader like us to read a paper.

3 pass approach to read a research paper

A 3 pass approach is taken to read research papers. The content covered in each passes is in sync with the discussion on the review procedure from last section. Following are the 3 passes:

• Should be able to answer the five C’s (Category, Context, Correctness, Contribution, Clarity)
• Second Pass: Read the Introduction, Conclusion and rest figures and skim rest of the sections(ignoring the details such as mathematical derivations proofs e.t.c.).
• Third Pass: Reading the entire paper with an intention to reimplement it.

Lets go into detail of each section.

The main intension in the first pass is to understand the overall gist of the paper and have a bird’s eye view of the paper. The intension is to get into the authors intent about the problem statement and his thought process to develop a solution to it. The major sections which should be focused in this pass are the Abstract and the summarising figure and extract the beat possible information of the problem statemant the paper is addressing, solution and the method. The following points are what we cover in the first pass:

• Read through the Title, abstract and the summarising figure.
• Skip all other details of the paper.
• Glance at the paper and understand its overall structure.
• Category : Which category of paper is it, whether its an architecture paper, or a new training strategy, or a new loss function ar is it a review paper e.t.c.
• Context : What previous works and area does it relate to. E.g - while Reading the DenseNet paper, it falls in the context of architectural papers and it falls into the resnet kind of networks architecture context.
• Correctness : How correct and valid is the problem statement that the problem is addressing and how correct does the proposed solution sound. Honesty this can’t be totally jugded from just the first pass completely as a complete answer and unserstanding of correctness would need looking at the conclusion section, but try to judge as best to your knowledge about the correctness.
• Contribution : What exactly is the contribution of the paper to the community. Eg - the resnet paper contributed the resisual block and skip connection architecture.
• Clarity : How clearly does the abstract explain the problem statement and their approach towards it.
• Based on our understanding of first pass, we decide weather to go forward or stop with the paper for a detailed study into further passes.

While discussing about literature survey, I mentioned about the 10% study on each resource to figure out if that resource is relevant to us. The 10% basically meant doing a first pass over all the resources.

Note: After the first pass, we understand the gist of the paper and get into the intent and thinking of the author.

After getting an overall gist of the paper after the first pass, we headon to the 2nd pass of the paper. The main intention of this section is to understand the paper in a litle more detail in terms of understanding the problemstatement in detail, validating if the paper validates the propositions it made to solve, understand the method in detail and understand the experiments well through the discussion section. The following is what we do in a 2nd pass:

• Reading more in depth through the Introduction, conclusion and other figures.
• Literature survey, Mathematical derivations, proofs etc and any thing that seems complicated and needs extended study from the references or other resources are skipped.
• Understand the other figures in the paper properly, develop intuition about the tables, charts and analysis presented. These figures contain a lot of latent information and explain a lot more things. so it is important to extract the maximum understanding from the figures
• Discuss the gist of the paper and main contents with a friend or colleague.
• Mark relevant references that may be required to be revisited later.
• Decide weather to go forward or stop based on this pass.

After the 2nd pass, we have a good understanding of the paper in terms of the method of the paper, experiments and conclusions out of them. Depending on understanding from it, we go on to the next pass.

Tip: A second pass is suitable for papers that you are interested but not from your field or is not directly related to your research goal.

After getting a more indepth understanding of the paper after a second pass, we go on to the final pass of reading which is the most detailed pass over the paper. This pass is only for papers which are most important for the research objective and are directly related to the objective we are working on. Following are the key points for a third pass:

• Reading with an intention to reimplement the paper.
• Consider every minor assumption and details and make note of it.
• Recreate the exact work as in the paper and compare it with original work
• Identify, question and Challenge every assumption in the paper.
• Make a flow chart of the entire process considering each step.
• Try deriving the mathematical derivations from scratch.
• Start looking at the code implementation of various components if an open source implementation is available else try to implement it.

After a third pass, we should be knowing the paper inside out including every minor assumption and detail in it along with a clear understanding of the implementation and good understanding of the hyperparameters of each experiment perform and presented in the paper. After all the passes we can claim to have a clear understanding of the research paper.

To validate our understanding of the paper, there are a few generic question we can try to answer about the paper and if we are able to answer these questions, we have more or less understood the paper to a level where we can use it for our own research as per our requrement and our objective.

• Answer to this can be found in brief in the abstract section and in detail in the Introduction section.
• self assessment of the problem statement.
• Answer to this can be found from the Introduction section.
• Answer to this can be found from the Introduction section section in the contributions section and also the methoda section.
• Answer to this is the entire method sections and discussions section.
• Answer to this is the entire conclusion section.
• Many a times a paper has many key elements which they put together to solve their problem statement. At times your problem statement maybe just a subset of the papers problem set or viceversa or a particular element of the paper may be solving some problem youa re interested into and not the others. So it is important to figure out what part of the paper is useful to you.
• Some sections of the paper may seem complicated or you may need to look at some previous references to understand this work completely. Also you might find some papers from the citations which are also useful to your research. So figure out the necessary references and refer to them.

Being able to answer all these question to the ebst of our understanding and abilities validates our level of understanding of the paper. These questions can also be attempted after the 2nd pass itself and we can check our understanding after the 2nd pass itself. Then again try to answer them after a 3rd pass and judge if our understanding has improved over the 2nd pass or another pass with deeper exploration is again needed.

Tip: Nothing teaches better than implementing the entire thing from scratch and experimenting and comparing the results with original results. Even if a open source implementation is available, experimentation with the opensource code and coming up with own tweeks to the code, running different hyperparameters can improve our understanding a lot.

Finishing with a important note that reading papers is a skill that can be learnt with consistency over a long period of time. It is not a sprint but a marathon and demands lot of patience and consistency.

I hope I have been able to justify the title of the blog post and explain everything in detail about how to do literature survey of a domain and how to read an ML / DL research paper. Incase I missed out on anything or you have any other comments, reach me out @SaiAmritPatnaik

Thank you !

• Andrew Ng’s lecture in CS230 on how to read research papers
• S. Keshav’s paper on how to read research papers
• Slides of the talk
• Blog Post 1 on reading Papers
• Blog Post 2 on reading Papers

Andrew Y. Ng

Selected Papers:

Some links:, java things:.