Updated: Jul 11, 2019
Dr. Le Lu is the Executive Director of PAII Inc. Bethesda Research Lab and former staff scientist at the National Institute of Health (NIH) Clinical Center. His research focuses on improving the modern medical image understanding and semantic parsing to fit into revolutionary clinical workflow practices. Dr. Lu served as the Technical Area Chair for the CVPR 2019 Conference.
This episode is a live recording of our interview with Dr. Lu at the conference. He discussed how deep learning is applied in medical research and some of the challenges the industry is facing.
Highlight 1: Applying Deep Learning to Clinical Trials
Highlight 2: Boundaries and Limits of Deep Learning in Medical Research
Robin.ly is a content platform dedicated to helping engineers and researchers develop leadership, entrepreneurship, and AI insights to scale their impacts in the new tech era.
Subscribe to our newsletter to stay updated for more CVPR interviews and inspiring talks:
Host: Le, thank you for joining us today and welcome, of course to CVPR.
Thank you, I’m glad to be here.
Host: Yeah, I'm looking forward to our discussion. So you received your PhD in computer vision and machine learning from John Hopkins University in 2006. Can you tell us more about your PhD research?
My PhD research is about general computer vision. Before that I did about 10 years of research in computer vision in [the] 1990s, from basically 1990 to 2000; that's the years of geometry. So people basically were doing projective geometry, 3D reconstruction for 10 years. And then from 2001 to 2006, I spent five years at John Hopkins. Also, I was kind of co-educated by Microsoft Research at that time and got a flavor of deep learning. Deep learning in general can be considered as a special case of probabilistic method or statistical method. It's just deeper and shallower in that sense.
Host: We're at CVPR, and we're three days into a very intense program and a lot of information that's being discussed here. What excited you about the research that you're seeing that the folks are discussing?
We had a couple of medical imaging related workshops. I ran workshops and also I was invited to give a talk at another workshop. So I was mostly focusing on my own workshop, which is fully medical imaging and deep learning related. That’s the 8th year of running this workshop called Medical Computer Vision. From 2015, actually, each time we saw full house, from very early morning to very late night, from 8:30 to 5:30. Today, I saw a couple of things I think [were] very interesting, kind of aligned with our research called Big Data Weak Labeling. This morning was Low-shot Learning, and in another room was Weakly Supervised Learning. I am happy with the progress. I'm also running area chair for CVPR this year, so I did see the progress.
Host: And I see we have the most attendance ever at CVPR, the highest attendance is this year. So it's great to see that the program and the conference is developing so fast and maintaining such relevancy as well.
NIH is the primary government Medical Research Agency in the US, and you worked there for over six years and ever received the 2017 NIH clinic CEO award on research excellence and patient care impact. And it seems you’ve primarily focused on applying machine learning and deep learning in image processing, especially for radiology image and image segmentation. Could you summarize your major achievements and learnings during your work at the NIH?
I would not just contribute everything to the work I did at NIH. But before NIH, I spent 6 years in the industry. I realized the limitation by just working in the industry, being a little bit distant from the physicians and the hospital. So I decided to pursue a career as a government agency in the hospital, basically do some research that nobody else can do. That [place is] very unique. We are the largest research-focused translational hospital on the planet - not just in the US - it’s really the biggest on the planet. I was lucky enough, I worked with a list of doctors, very famous doctors, very engaged in the research. The research in clinical means like if you want to have a new cure, new guidelines about treating certain disease for better patient outcomes.
So I engaged there for 5, 6 years - 5 years as an employee, one extra year as a special volunteer. We are one of the early groups to adopt deep learning for medical imaging. In 2013, a lot of people didn't believe you can use deep learning for medical imaging because you have ImageNet, and ImageNet has one million-plus images, 1.2 million images that you have labeled precisely, but medical images don’t have that many labels. So we made huge progress actually, for example, enlarged lymph node detection. Enlarged lymph node is a very important biomarker for cancer vision, and we improved the performance relatively from 55% to 83%. So that's one thing which shows how deep learning can really be applied to a limited dataset, like 180 patients for example, and it can work amazingly well compared with previous work.
And you know, another thing I want to mention is using deep learning doesn't mean to re-do everything you’ve done well using traditional method[s], but it more means you solved [a] certain, very important problem not so well previously; now you can solve it better. So that’s one major strategy that we're making pioneer work in computer-related detections. And last year, our work got awarded as the most influential paper in the last five years by MICCAI. So that's basically cancer provisional diagnosis and detection.
But starting from there, I slowly realized this is an important problem. This is a problem mostly studied. Many of the hospitals, many of the startup companies are actually doing this. They're doing the computer-related detection. But then I realized a better problem to solve, or a more practical problem with higher urgency to solve is actually precision medicine.
Host: Precision medicine.
In my opinion, precision medicine is more feasible. It has higher impact for our practice as of now. In fact, it’s very straight forward. So basically if you go to the clinical practice and look at the cardinal protocol, doctors all know their conclusions are not necessarily the best treatment guidelines for the patient or best ways of diagnosing certain disease because doctors only have 20 to 25 minutes for one patient and is the only most feasible way you can get that as a human. So they do not have much help from the computer or from computing. For example, one technical thing you do is you measure the size of a tumor. In theory, it's better to measure the volume. But this is not possible because human doctors don't have time to pick out the voxel of the tumor in 3D and compute the volume. And also, you need to calculate this over time, because often a patient is under certain treatment, and his/her physician is working on that. So right now the protocol is not ideal. It's very manual. So you observe the slides back and forth, you find a slide you think that has the highest extension, spacial extension, then you measure the longest diameter, that's called long axis. Then you measure the short axis, which is orthogonal to long axis. So these two axes basically in clinical informatics were recorded into the patient record. That presents how big the tumor it is. But suppose we should measure it in the volume matrix. So over time, when next time even with the same physician, your measurement may not be precise.
So we are very good at doing reasoning. Doctors are very good at doing abstract reasoning, which a machine can not do. Even today, we don't have algorithm for doing that, that’s how brilliantly our brain works. But for this kind of simple measurement, this is something a machine can do very well, but humans can make mistakes. Has this tumor grown by 10% or shrunk by 10%? That means completely different things in the clinical indication. So this is one application in precision medicine for tumor measurement. That's critical, that’s one of the simplest things, but you can make a big difference. And this is quantitative imaging or precision medicine. You want to measure things more accurately, more consistently over time.
This is one of the things we're working on. For example, we detect lymph node, we segment lymph node, we measure the volume of lymph node. We measure tumors, we do a lot of tumor management, and also we do certain kind of algorithm volume matrix measurement(9:50). So for example, for pancreatic cancer, we have like 40,000 deaths every year in the US alone. It’s very deadly, it has almost no cure because it's too late; it’s very difficult to find. So we can compute some imaging biomarkers, model the volume and shape and derive that from the raw voxel to do something like a meta feature that physicians can understand. The machine can tell the physician, this is the volume of the pancreas, it’s not normal. This is precision medicine.
Host: Yeah. Thank you for those explanations. Certainly, we know that this is in our future and this is going to be more informing of how medical research is being done as we move forward. So we think about precision medicine and other types of research as well, we know that there are a lot of restrictions to apply black box methodologies like deep learning to clinical trials. How do these restrictions reflect your research?
I would say it's hard to argue deep learning just being a black box. It’s not totally explainable, or architecture-wise, it is too complex with too many parameters. But I do not feel this is really a big issue because statistical models were also kind of a black box before, they are not very readable for humans. They use thousands of features from millions of features. That's pretty much how boosting and random forest works. I think that deep learning in a sense, is a better way. It's easier to be explainable.
I agree we need to do a lot of research along this line to make it more explainable. I want to take another way. Don't expect the models themselves to be explainable. They won’t. They have billions of parameters. How are we going to understand what's going on? The thing is, we can make it more manageable in terms of what we learn. So we're not just working on X, we're working on Y for example, something human verifiable. So if you were learning something, you tell human physicians, this is a benign tumor or malignant tumor -- how are we going to verify? I think you have no way to verify. This is where we tend to call it black box. And the issue is you can not use it, in the clinical practice because this is not the reason it's not explainable. It's the reason it's not verifiable. Actually when learning something, you have to tell the physician why you think this is benign or malignant. So you have to go through millions of oncologists and tell them how benign or malignant tumors can be defined. You have to work with the physicians. Sometimes oncologist’s conclusion is incomplete. And instead of making the final decision of Y, you can make a decision on Y prime or Y1, Y2, Y3 to Y sub-task. So you learn maybe the texture features, the certain shape features.
Host: The explainability is there.
Yeah. Human can verify these kind of tasks [and] know if the machines are making mistakes. So after you have all these explainable, verifiable meta-features, you need a guideline saying, you can make a decision, for example, to integrate all these meta-features into a final decision whether it is malignant or benign. And from there, you can contribute the final decision into these meta-features. So this is a sub-task. So you first have to learn a lot sub-tasks. And these type of sub-tasks are verifiable. Then you put another layer of decision-making mechanism, depending on these verifiable features and that part can also be explainable.
Our goal is very simple. Right now, humans use human physicians, using 30 minutes to make a decision which is sub-optimal. Can we provide them much more information than now, so that they can make a better decision within 15 minutes? You lower the cost. I assume it consumes just some electricity, won't cost too much, but you lower the time of all the physicians, and you can work on something else. And combining them together, we can improve by using less physician time and make a better decision, with much better patient outcome. You need to have critical value, a positive value for this society in order to be sustainable.
Host: I mean those are always the commercialization aspect of course.
Yeah, definitely. Someone needs to pay I guess.
Host: Yeah, at some point. So moving on from that and let’s look at your current role. You joined PAII as executive director about a year ago. What are you and your team currently working on?
OK, first of all, I’ll explain a little bit about what’s PAII. PAII is a research company founded by one of the largest insurance companies. This insurance company’s businesses include financial investment, full-license bank, and a lot of healthcare related businesses. They have many people insured. So, for one thing that I actually believe is, if you are working in the medicine industry, you know medicine can be divided into three parts: patients, providers, means the healthcare providers, and payers. Payer means the insurance company. It's a big issue to have providers and payers if they are separate because they have different interests.
Host: And when you think from PAII's perspective then, looking at these problems, what work are you and your team doing to address that?
I think the big priority is actually, I know what I'm trying to do. I'm trying precision medicine. How can we offer it to more patients? And another thing is called Stanford AI100 Report. It was published in 2016. My advisor Gregory D. Hager (AAAS/ACM/IEEE/MICCAI/AIMBE fellow) wrote part of it. From there, they were saying, in the US, the innovation part will come from Kaiser Permanente or Geisinger. Why? Because they are both the insurance company and health care providers. So the hospital and the insurance company are the same organization. So they have a motivation about providing better healthcare and reduce the cost. Because there's no complication about two parties, they fight each other. The only consumer is the patient. I hope there could be many of this kind of hospital, or many this kind of insurance companies plus hospitals, so they'll be fighting for better efficiency and lower costs to guarantee better patient outcome, better health outcome and manage your health. Basically, if someone gets a disease, they want to cure it as soon as possible with better outcome. Because that would make sense economically. I hope someday that's the future. So I think the PAII is helping, it’s part of the formula, part of the component to make that happen. The parent company of PAII is one of the biggest insurance companies, and they want to do the same thing. They have the vision and decided that this is also the way to go. So instead of being a passive payer, we're more like a more active patient management, health management. As an insurance company, we get the premium to guarantee your health. That's our function.
So we’re doing pretty much doing the similar things as I was at NIH, but it's also very different. Because right now I work with a lot of primary physicians, I get the full spectrum about the patients. So in a hospital, I used to work in the radiology department of Siemens. But radiology department doesn't really control the patient, they don’t directly manage the patient. They are more like a consultant department in the hospital. So whenever the primary physicians have an issue they can not solve, they have a puzzle, they don't know what's going on. So they tell this patient to do an imaging exam, and bring the imaging report back in. That can resolve the puzzle. Right now we are mostly working with the primary physicians to know what's the puzzle. We have more and more interest on integrative medicine, which means imaging is an important part of it is. As you know, I'm professionally trained, but we should look into the whole picture of the patient health, and the whole spectrum of patient management.
Host: Just leading on from that, because we've been talking a lot about research, we’ve been talking about the application on hospitals, the relationships with the doctors. Certainly the future's very bright in terms of those improvements in collaboration and partnerships.
So I want to end with this final question to you. Many AI researchers at the beginning, believe deep learning can disrupt everything. However, deep learning is data hungry and performance is strongly correlated with the amount of available training data. In your perspective, what are the boundaries and limits of deep learning? And I know we've covered some of this before as well. But if you were to summarize it where do you see today? And then where do you see where we can lead to? So where are we going to move forward to?
Yeah, this is actually a very good question. So first of all, we humans don't learn well, we do not learn in a data hungry way. In the data science sense, my proposal is [that] we need to learn a lot of things among X, make the data itself regular, [so] they can talk to each other so we can model the relationships between different data samples. Then maybe you only need a few mappings from X to Y, and you can get the problem solved. This is the mathematical answer.
Another answer is in the clinical side. I also give talks to the hospital, to the physicians. One problem is, physicians will never be employed to create such thing called ImageNet in the same way using Amazon Mechanical Turk or something. Physicians are not trained to work on labeling images to train computers. One primary example is that, a primary physician get a puzzle and says, has this answered me? And he actually tries to read the patient profile, what happened before? Is this the first imaging, or is this the second time imaging, or is this a follow-up imaging? Now he looks at the images, probably even look previous images. So we have images, questions, and also what he learned from the medical school. A lot of things from his fellowship. So this is kind of knowledge-based thing. He or she will integrate three pieces of information together to make a final decision. That's what they do. Physicians already viewed a lot of information by doing the daily work. You think about a physician describing what's going on in this image, it's text report. But that contains one human’s presentiment about this image.
40% of what we do is we are trying to build a novel method. Last few years, our six CVPR papers and one WACV papers were all about that; we tried to leverage clinical annotations. This is something we produce in our routine clinical work, so we have the information. We're trying to track down, decoding what they are saying and using that as a label to learn images, instead of reading everything, it's impossible. I often give an analogy saying, we have a small hospital, we have 100 million patient records. If you factorize it's about approximately 100 doctors a year. Will you be able to do that, which is by creating a new data set, and label in the same way? One hundred doctors a year? And then think about the cost. It costs a lot, and you don't have time to wait. So the best way is already there. We need a new method to verify what human is doing. I got the words from a Clinical Center CEO, he said now we are doing this very big data set on chest X-ray and that was two CVPRs ago. And at the last CVPR, we released a big data set of more than 10,000 patient studies with all different kinds of tumors and all different kinds of things doctor measured. That somehow also can serve as ImageNet. None of the tags and the labels were produced by Amazon Mechanical Turk. They could not do it. It was mined from doctors reports, doctor information, and even with a visual grounding. So you have to train your method, your algorithm to annotate itself. And it's also a learning progress, human do that, and that’s why we can learn from a few examples. So this is the future, in my opinion. We have to do this kind of normal method in order to progress.
Host: Right. And maintain always the human centered approach as well.
Host: Le, thank you so much for your time today. This was a great conversation. Thank you.
Thank you very much.