BONUS / Running Out of Time / Michael Donnelly
I'm Dr Selene bounder and this is at polemic in this episode. I speak with Michael Donnelly. Whose background is an economic research financial policy and data science. He's worked in the US. Federal Government in various research positions for six years in the private sector for almost seven years last. Wednesday Michael published an analysis to provide a broad sense of the scale of the crisis. The country is facing. He found that reported cases of Kovic. Nineteen in the United States. Dramatically understate the current number of infected Americans the same week Michael published and other analysis. Looking at the New York City Metro area has results were startling. The outlook for New York City and Kovic. Nineteen is bleak. The policy response has been far too slow and too weak to meet the needs of the moment. His analyses came just days before an appeal college. Report predicting high fatalities in the US. If the country doesn't take extreme measures the place where I start with all these conversations that I'm not an epidemiologist and I'm not a public health researcher My background more in economic research and financial policy and data science so prior to my current job as a data scientist from the tech sector I worked in the federal government for about six years and in the private sector for about three years in financial. Policy research specialties. In all of this quantitative communication and data science and forecasting. How did you decide that you wanted to delve into the semi? This is not your regular job right. No it really isn't it was really. It was a fascination I think is to be totally honest. It's kind of topic that I've always been interested in. I did some training and health economics and Grad school and I started digging into this really in January and was pretty convinced by February. That this was you know like pretty major even for the US. found over those weeks that I was getting pretty exhausted trying to convince friends that this was not the seasonal flu And so because of my background my skill set I turn towards Quantitative Analysis and Empirical Research to answer these questions. I and that's you know that's really where I started going To begin so really in March after becoming pretty tired of trying to convince friends. I decided it'd be a lot easier to compile all of my thoughts in written form and so also providing that really as a resource for people to feel empowered to explore the existing scientific research on the topic Rather than just believing some friend of theirs who thinks he's got a good opinion So I started writing up About a week and a half ago All of my notes and it quickly turned into a twenty page medium post with Research and my own quantitative analysis and modeling and before I put it out there because They had some disturbing conclusions to it. I ran this by a bunch of friends. I think probably about six to eight and my friends who I had reviewed. Phd's JD's or some sort of doctorate in a variety of field between Public Health Economics and political science and law so it was really making an effort not to put out that information but my main goal was to put on information at a level that you didn't have to have a degree in science or in statistics. Understand so that was my goal my first medium article it was. It was really kind of the national overview. Shortly after that. I think I released it a week ago. Today I started thinking through the consequences. My own life and part of that meant thinking about My exports we which was about to restart it season here and so I ran the realize this the the executive board meeting with starting about a half an hour and so I ran down to the to the board meeting to ask them to cancel the upcoming season. And as I was waiting for my turn to speak I Replicated National Analysis For The New York City region and Got Pretty disturbing results about what that meant for. New York's healthcare system and its capacity to absorb You know the forecasted to during critical cases into hospital that was the realization that I needed to do more formal analysis for New York and by the next day around lunch I had been the analysis and was ready to host on twitter after you know getting some other friends to to review that specific analysis as well. That's when I started reaching out to all the government officials that I could possibly reach out to because the conclusions were so jarring. It really told me that'd be had very little time left to take much much stronger. Governmental action to increase social distancing before we had essentially locked him too many infections. That would overwhelm your hospital system. Some follow up questions about that so having looked at sort of the national scenario as well as the New York City scenario how divergent are different. Are Those will. I'll start off by saying that. There are a lot of models out there that do a really good job dealing with Different geographies and densities and the way that people move around in different ways they some different cities those models also unfortunately very complex and difficult to understand models that I used. Are you know essentially simple? Almost one hundred year old models called. Si our models and they just work at the top line summed up the whole country or summed up the whole city because they have the same structural form if you are saying that the virus progresses in the same way. They kind of look very similar in terms of how quickly they progress under these simple models. So that's Kinda short. Answer the the wrinkle in that is that it matters a great deal about where you estimate your current position to be Versus the relative total susceptible population so in the New York City region. It really seems that I haven't done it for more analysis of all the cities in the US it seems like it's the most the highest priority city be addressing the problem right now You could also think maybe Seattle would be close on heels to some people that would be surprising because Seattle was hit. I why would that be a triumph point out throughout this conversation where I've done what I think is firsthand quantitative analysis? And where I I feel like a greater uncertainty in this place I haven't done as much. Firsthand quantitative comparisons but I think It's important to point out that it seems that most of Seattle's cases detected cases upfront. Were in a nursing home. Among much more susceptible population are much more vulnerable population. My older people whereas in New York you had its first cases detected on fifty. Somethings forty somethings who are very mobile throughout the city and didn't have a clear lineage to trace back how they had coming back in the first place and so since it could really have been you know. New York City anywhere along the Metro North quarter or in the subways of Manhattan. It stands to reason that there were substantial numbers of undetected cases that were circulating around America's most density and so for that reason. I think it's reasonable to assume that the overall reproductive rates have been detected in a variety of studies based either on the experience of China or the experience on Diamond Princess cruise which I said he's leverage heavily my papers because they really could. Do you know full population. Study In in newspapers I think they give you a really good picture of what it's at in New York without intervention. How are you able to guesstimate estimate you know how far back? Corona virus may have been introduced into New York City when when the first cases may have started circulating here. Couldn't do that analysis But the last time I updated the analysis was the end of March thirteenth which was last Friday. As at that time we had about four hundred cases in the New York City region that were reported and we were estimating anywhere from thirteen hundred to twenty three hundred cases overall the regional population that most of which hadn't been detected. That's just that it had been in New York City for for many weeks. But any greater specificity is not it? Couldn't just be guessing. So maybe if you can just talk us through first of all you know what you found you. Sort of gave us A very a broad overview of what that was but maybe a little bit more specifically sure the numbers that I found a New York City which are now five days old about a third of the analysis is just focused on trying to understand the actual current number in the population of infected and potentially infectious cases. As I mentioned before we had reported at the time Four hundred cases. And then there's a real question of okay. How do you kind of Ballpark? The right number of cases in the population you know the place to start is to just acknowledge that directionally. The reported number of cases is an underestimate assuming also a fairly low false positive rate in the testing results over that that may be somewhat of a week assumption given some reporting that. I've seen on that but regardless even with a relatively high false positive rate and testing. It's almost certainly an underestimate. Because of the limited number of tests out there and the strong restrictions that even still hold to this day about who can be tested but it was even stronger restrictions on who could be tested at the time last Friday so the approach for coming up with the estimate which I detail a lot more in my first paper was to attempt to estimate four different parameters. And those tremors were the ace Amana great mild case rate and then to detection rates. So I'll start first on. WanNa look at the dramatic and the mild case rate mild. Innocent medications are just a lot harder to catch because they present as you know as a as a cold and people don't get tested for colds regularly And so we kind of need to know. First how many people who get cove in nineteen are going to experience these mildmay symptomatic version of it so we went to a few different papers. There's a couple of papers on the clinical characteristics on the virus coming out of China one in the Lancet that used them one in the New England Journal of Medicine and they both play the mild rate at around eighty percent. I think one had eighty one percent and both papers Had about a one percent asymmetric rate if I'm remembering correctly whereas there were two papers coming out of the Diamond Princess cruise which was unfortunately for the passengers. Pretty good natural experiment or research purposes. Because you can test everyone On the cruise they came up with a couple of different estimates for rate Pay Some medications. One pick replace that at the lower end around seventeen percent and another paper place. The higher end around thirty five percent. So why does that matter matters if we believe those papers that the easiest way to reconcile them is to say that the clinical papers coming out of China just weren't getting into their sample the dramatic cases because those labs confirmed cases and make sense that they would be missing quite a few cases based on what I know that the Chinese testing protocol so how to adjust them this mild case rate. It's actually not that hard. You just assume that if the overall population actually has They thirty five percent Symptomatic cases than other remaining sixty five percent. Eighty percent are mild. Then you can just add up. Those percentages and you can get the overall mild as some domestic rate since both mild and Cincinnati. -CATIONS tend to be harder to detect. We've been separate out the detection rates for these non severe cases and for severe in critical cases by the way the calculation of the severe critical case rate is much easier. Once you've figured out the mild and Eysenck matic rate you can just subtract that from one and you. The remainder is going to be severe and critical cases which ends up being somewhere just under ten percent depending on depending on which estimates us the next step ones. We've kind of got. That is to just understand how quickly people transition from one ovulation to the other. There's also been a lot of discussion as you know about School closures mass gatherings was there to slightly different questions based on your modeling. And you're looking at the evidence and I think you've been involved in the discussions at least in New York City around school. Closures what do you think the evidence shows and what's been your involvement in those discussions. It does seem to show in various studies that I've been reading that you know. Approximately a third Transmissions happen At schools and at workplaces so closing down schools. Well trigger and don't seem to have as many severe side effects do seem to be a vector and one interesting analysis that I saw on the out of South Korea where they've done really extensive testing their population and a lot of people pointed to that. Extensive testing is being one of the reasons they've been able to get the the spread of the virus under control their thought. The signing was that there is a quote unquote over indexing of cases among twenty to twenty nine year olds and so even though they don't tend to show as many symptoms many severe symptoms. It seems that because it's a more social group In the age pool that they're one of the biggest factors that the spreading and so while that's not the age group for public schools. It is part of the age group for universities. I think between the Imperial College study that found a significant part of transmission happening in schools and finding out of South Korea that a significant part of transmissions are coming from Twenty somethings it does make sense that we're gonNA continue doing Restrictions on movements and and restrictions on schools universities. How different are your methodologies? I mean it sounds like some of it may be that you're just trying to approach it as simply as possible Especially for lay people to be able to understand what you're doing. But how different are your methodologies from say Ferguson Neil Ferguson or mark lips itch the Ferguson paper this is the Imperial College paper uses what I believe is an agent based model and this effectively mean that they're attempting to model the individuals in society and how they move about and how much they contact each other This is a pretty new approach to modeling. At least in my background and Financial Stability Research and and it can yield some interesting results. That are different than what Topline statistical analyses yield but on average. It turns out that the very similar results and so that I can point to. Is the conclusions that both Neil Ferguson. I reach. And what parameters us to reach them. The Neil Ferguson paper uses an are not of two point four which is almost the same as my two point three. They use essentially a serial interval. So this is how frequently that compounds of just over five days which is slightly more than my four point seven they find for the US that without intervention about eighty percent of the US population become infected. I find higher than that but It would make sense if there's slightly different results particularly on that factor and finally the total number of deaths which is super important metric. We all agree they find About two point two million deaths peaking sometime this summer Which is slightly later than my timing estimates for that but it's also just under twice as highs my estimates for that from forecasting in statistics standpoint. All of these results are what we've Kinda referred to as within an order of magnitude so while they approach the modeling from a pretty different mathematical standpoint. What's interesting is that the results come out very similarly and when you're using models to forecast it's actually a good thing to have people pursuing a lot of different model forms because if they're all coming out with roughly similar conclusions And they're not collaborating then you can have a much higher degree of confidence in the findings so in other words you're approaching the problem in different ways but you're all sort of your results are all pointing in the same direction. Yeah that's exactly right and part of the reason. I chose the simpler model. Aside from the fact that I'm not epidemiologist was also I think it's important for people to feel like they have some agency over understanding why political leaders are taking such strong actions when it seems like the death rate or the critical case rate is still pretty long? The country and so. I hope that this math this explanation damalf gives people a better sense of why it's important to act now. You know another thing that's been up for. Debate is duration of the social distancing measures when we can lift them. What is your model show? And how does that align with the Ferguson model? I haven't done what. The Ferguson model did mine. Alice's went up through the first part of the Ferguson Model And it aligns very well with their hypothetical no government intervention and no individual response to the to the virus. Our results are within essentially or magnitude at each other and had very similar peaks in terms of the interventions. I think that there's limited information right now to estimate exactly how big an effect each of these interventions have on the virus. And so the Ferguson. Approach using agent based model is some of the best ways that we can deal with that right now. I think that there's going to be more evidence coming out of the change in the reproductive rate in each of the countries that we've seen China South Korea Japan France Spain Italy to see how much we can attribute chain reproductive rates of these policy implementation. There's just very limited data on that right now so I Attempted that work yet. One of the things that were beginning to discuss among some other. Us based data scientists is trying to come up with a way of tracking at least directionally in real time changes in the reproductive factor in different regions of the United States. But we're also going to have to estimate changes to detection rates and that's all related to the number of tests as a proportion of the number of infections so in order to to do that we're going to have to continue to forecast so there's a lot of unknowns right now and it's a big mathematical problem essentially and a data problem that a lot of people have just begun to work on so. I'm hoping to have a better answer for that coming weeks. But right now. I think the Ferguson papers one of the Best I've seen on it. They seem to suggest that the first period of lockdowns is going to be the longest because we need to get the the disease spread under control before we can think about lifting the lockdowns to that. We don't overwhelm hospital immediately. After it left them and I think they played some around two months to the initial period. Then you lift the measures but then you have these subsequent waves of disease. So how do you decide when to reimpose measures and can you do so in a more local fashion as opposed to doing so across the country? I think there's a couple of points there. One key point to get. I understand the intuition behind. The subsequent point is that I see you. Cases or severe hospitalized cases occur virus are lagging indicator. So if you stop people from catching the virus by imposing lockdown when you're at a hundred percent of your hospital capacity well you've got all the transmission that have been happening a week before then but haven't been showing symptoms and so you're going to have all of those cases that haven't should of the hospital yet arriving at your hospital after your hostels already full over the next week. So what the paper does it says. We're going to need to estimate capacity triggers for this and. I think that there's a real data problem here. So let's say the capacity trigger is like a third of the of the available capacity in the hospital and that seems kind of reasonable to me but in order for that trigger to be placed. We're going to need to know what the capacity of the hospitals are in the region and what be number of hospitalized cases are on the region and as far as I can tell you I've been speaking to epidemiologists and data scientists Across the country. There doesn't seem to be a good centralized source for that information and so it's GonNa be a lot harder for even sophisticated state and local governments to do that and I think as unsexy problem as it is to say. We need to invest in data. Right now I think that's gotta be a high priority. How well do both models predict how things have played out in Italy and how things have already started to progress here in the US? You haven't seen many back tested the Italy case I can only speak to my own on that one and when I ran it on about two week old data coming out of Italy my model was actually somewhat under predicting the two week out number in Italy. Now that is essentially within what WE EXPECT TO BE ERA FOR MODEL. And so it's suggested that these models and dust the parameters that we use the model that the are not that reproductive rate how frequently that compounds those sorts of things it really suggests that these models are right and left. Broadly unchecked like they were in Italy. The future extrapolations even out a month or two are right and I think that's really why people might governments are beginning to take this seriously right now because People have really convinced national leaders. That if they don't do something now he's really horrific scenarios could play out. How far out can you accurately forecast? And how you know how. How much can we trust these predictions? Some people are saying three months five months of these kinds of interventions you know. How long are we looking at this? You know we're going to have to wait eighteen months for a vaccine you know. What can you say about the You know how far out you can really project with almost any forecast. The underlying factor is that the further out. You try and forecast the less accurate. You're going to be so a year from now is always going to be harder than a month from now and a month from now that we're going to be harder than a week from now in case in particular The biggest source of uncertainty. At least from my perspective is the policy response the models that I've done in the model the compared my work to the Imperial College model done by Ill Ferguson and and his team. They've been criticized in a in what I think is a productive way for one instance. They haven't considered the case. Tracing that could happen alongside some of these other policy implications and that in fact could reduce future infection rates similarly in in their paper. They seem just doing essentially on and off and his lockdown our suppression tactics In order to deal with both the at the economic and the health policy consequences of these policies of keeping people indoors and and social distancing and that would essentially flattened occur by having curb kind of go into waves but those waves over time may not be evenly spaced and those waves may not even space for a variety of reasons. One is increasing technology increasing tests loads but also increasing Acquired immunity because more people have been exposed so those sort of dynamics are very challenging to forecast and I think what a lot of people who are involved in trying to forecast this right now are doing is. They're trying to do two steps. Forward one step back. And I think we're just GONNA keep leveling up the quality of these forecasts overtime as we decide which forecasting approach Is yielding results. Do you sink New York City has acted quickly enough and for that matter the US The short answer is no. I think we knew enough last week to have taken the action that the mayor took on Sunday. Which is the close restaurants bars public venues movie theaters and schools now school? That doesn't matter much over the weekend but bars clubs I think reduced to quote unquote fifty percent capacity. I believe by the governor but I don't believe that most of those venues were observing got order at some point I think Some clever researchers are going to figure out Just how many illnesses are to blame for that. Glow Movement on. I know more about the New York situation than the than the national one but I think New York has some of the best Health surveillance systems in the country. And I think that's probably why we know more about New York. We also probably have more because it's a more density but I could see a similar pattern playing out across the country and it certainly hasn't helped anything when you have national leaders telling people that it's oaks and it's not even as bad as ever seasonal flu people will take action to protect themselves. But they won't do it if they don't think it's real even even just those words Have been a detriment to the spread of the virus hundred and this was the things are going off. People is yeah I mean I can't agree with you more. How soon do you think we might see hospitals in New York overwhelmed by this question? It's hard to know how much the decision to close bars and restaurants and other public venues over the last couple of days and slow down the virus. I the model that I published on March fourteenth using data through march thirteen suggested that we were going to hit enough infection by March. Twenty third or twenty fifth that by the beginning of April. We would be at hospital capacity recent analysis that. I've done using data from yesterday and today suggests that I was under counting the starting place in the New York City region The trend was accurate. But it's generally bad news. I'm very nervous for the hospitals in New York City. I think the news that the Defense Department sent hospital ship to New York City which is frankly unimaginable. A month ago is actually very good news and will hopefully with this surge capacity but from last year the outer boroughs of New York Have only seen a fifty percent reduction in subway. Travel when you look at the turnstile bank MTA. I don't think fifty percent reduction is enough. If we were already out capacity in a number actions we have today which we may be at so we may be adding beyond capacity every day. That goes by that. New York doesn't adopt stronger lockdown procedures. Michael before we close. Is there anything else you wanted to make sure to get across or wanted to add? I'm impressed by people's desire to get involved and from wide variety walks of life. You know I think. There's a lot of volunteer work. That's coming out. I think we're resilient I've seen people who work in gyms where personal trainers we've lost her income source for a long time immediately hop online and find ways of making income for themselves by doing video fitness classes. I think we're just GONNA have to be creative going forward to get through this and honestly I think there's some opportunities for really good things to come out of this going to be painful and meet in the meantime but I'm not fully pessimistic about the next few months epidemic is brought to you by human productions. Today's episode was produced by Jordan Gospel. Ray and me are music by the DOTS sessions. If you enjoy the show please tell a friend about it today. And if you haven't already done so leave us a review on apple podcasts. It helps more people find out about the show. You can learn more about this. Podcast how to engage with us on social media and how to support the podcast at epidemic dot. Fm that's epidemic dot f. m. just human productions is a five zero one. C Three nonprofit organization so your donations to support our podcasts are tax deductible good epidemic DOT FM to make a donation. Starting next week we'll be releasing epidemic twice a week on Tuesdays and Fridays producing a podcast costs money. And we've got a pair sound engineers so please make donation. Help US keep this going. Also check out our sister. Podcast American diagnosis. You can find it wherever you listen to podcasts or American diagnosis dot F. M. on American diagnosis. We cover some of the biggest public health challenges affecting the nation today in season one we covered youth and mental health in season to the OPIOID overdose crisis and in season three gun violence in America. I'm Dr Selene grounder. This is epidemic.