EP.1 How do you know the crime lab did it right?

Ep 1
===


[00:00:00] Intro
---

Angela: Hello and welcome to secrets from the crime lab. An educational podcast, where we discuss forensic science and related scientific fields and topics. I'm your host, Angela Swarts. And I've been a scientist working in laboratories for the last 22 years. 19 of which have been in forensic laboratories. I'm excited to share with you the knowledge and expertise I've gained over that time and bring the reality of science to you, to demystify what happens behind the closed doors of the laboratory and bring you not only some practical information, but also the truth of how the science works and sometimes how it doesn't work.


[00:00:43] Episode 1
---

Angela: Greetings all and welcome to this episode of secrets from the crime lab. I'm your host, Angela Swarts. And as this is the first episode that I've recorded in 2022, I wanted to go ahead and wish everyone happy new year. You made it as a quick reminder, or for those of you who are joining us for the first time, please note that this podcast is for educational purposes only. It is not intended to be providing you with any type of legal advice. If you find yourself in a situation where you need something in a little bit more of an official capacity, you can always visit my website at practicalforensics.science, that's practicalforensics.science. And I can see if I'm in a position to assist you in any.

Unless you've been living under a rock in the last few years. you'll have noticed that there's been a lot of times where you will hear the phrases trust the science and that the science changes. And you've been hearing all about and different types of testing that maybe you haven't heard about before: PCR testing and antigens and antibodies and spike proteins, right.

We pretty much all have been inundated with this information. And I can tell you from my side of things that not all of this information that you've been inundated with is strictly speaking, correct. So as a scientist, that's been conducting PCR testing for pretty much my entire career. I have some issues with how it's being portrayed by those who may be not, don't have that knowledge.

So. Probably or if you would like to, we can certainly discuss that in the future, but that's not the purpose of today's talk today. I want to kind of circle back a little bit to how we're hearing this trust to the science, believe in the science. Right? Well, science doesn't require your belief. It's not something that you believe in. Science is a process.

It is a systematic way that we can explain what we see in the world around us. And as we gather more information and as we have new ways of measuring and, recording what we see around us in the world, our understanding of that information is what changes, but the scientific process itself has not changed.

Right? So when we start talking about things like, well, how do I know that this lab test result that I've gotten is correct. And, and is it even valid, right? And how do you ensure that you can be confident in that test? I mean, what kind of assurances do you have that it's correct. And how, how you get those? Now, the primary way that laboratories assure the quality of their testing is through a rigorous quality assurance program.

And the way that you can determine if a laboratory has that generally is whether or not they are accredited to perform that type of testing. And so here we are with today's topic, laboratory accreditation. Please do not run for the Hills now that I've said, we're going to talk about laboratory accreditation and quality assurance.

I know from experience that even forensic scientists, when we are daily steeped in quality assurance processes in our day-to-day work. If we're given the opportunity at a seminar of going to a breakout session on interesting cases versus a breakout session on laboratory quality assurance and the new standards that have been issued, there's not going to be an empty seat in the interesting case presentation, and you can have front row access and the quality assurance, talk.

So, which is unfortunate in a way, because I think in general, all of us as forensic scientists need to have a better understanding of the why behind a lot of the little administrative seeming type steps that we have to perform in our day to day work. And it does tend to fall more on the laboratory managers, the quality managers, the technical leaders, section supervisors, to kind of ensure that all of those things are happening and they might have a deeper understanding of the why behind them, because they're more directly responsible for the quality assurance program.

Whereas the scientists that are doing the work at the bench in the laboratory, they don't have as much contact with the accrediting body necessarily. They're not as familiar with the audit checklists and all those processes and standard 4.1 .2. And what, what does it mean and how, how do you implement policies and procedures?

They just know that I have this Bible, which is basically, you know, typically it's a quality assurance manual. They just know that they have to follow everything in there and they're not allowed to deviate. And if something happens, that means that they weren't able to follow it. They have processes that they have to invoke in order to investigate and determine whether or not something went wrong, why it went wrong if any kind of corrective action needs to be implemented and how do they prevent that in the future? Right. So it's, it's a very important aspect of laboratory work. And when you're talking about forensics laboratories, it's even more important because it impacts how things are going to ultimately end up in a court of law.

And if the forensic science isn't done correctly, then you end up, you can end up and you have, they have ended up in many instances with miscarriages of justice because things weren't done correctly. And this happens at both accredited laboratories and non-accredited laboratories. So it's a laboratory accreditation is not a rubber stamp by any means, but it doesn't necessarily mean that a laboratory gets it right all the time.

And some of it, we shouldn't ascribe malice to a laboratory that is accredited and gets something wrong from time to time. Because as I said before, the science didn't change, but our understanding of the data that we've collected and how we should interpret and report that information does evolve over time.

And in a large part, I think that one of the reasons for this evolution that happens, these improvements that happen are precisely because we recognize that something went wrong. Whether it's because the customer complained or if the laboratory recognized it on their own, or if it went all the way to the end and someone was wrongfully convicted of a crime and they were later exonerated by, by newer types of methods like DNA. Cause we know DNA has exonerated a significant number of people who are wrongfully incarcerated. So lab quality assurance, laboratory accreditation is an important tool in our toolbox to do better. So what exactly are these tools? How do they work? This is what we're going to focus on today, is giving you a foundational understanding of what these requirements are because in subsequent episodes, I am, we're going to have some more in-depth conversations about what happens when it breaks, when it doesn't quite work the way we want it to. Because we know, as I mentioned before, that all accredited laboratories have issues sometimes, and those issues lead to miscarriages of justice.

And we want to ensure that we're preventing those where possible. And if we are truly committed and truly have a quality mindset, I believe that the laboratory accreditation can be an effective tool to ensure the quality of the test results and also how those results are presented in the courtroom.

Because sometimes it's not so much that the laboratory did anything incorrect. And maybe when they gave and provided evidence in the courtroom, when they gave their testimony, it was misconstrued. And sometimes that's because the scientists misstate something or the way that it is expressed, it's given more weight than it should have by the court. That happens quite a lot. So the University of Lausanne actually has a course on Coursera, where they talk about how science should speak to the court. So if that is something that you're interested in learning more about, you can visit their website and take that course. It's, it's free unless you want a certificate from it, but I'll put a link to it on my website, at, practicalforensics.science and in the episode show notes. So you'll be able to go and check that out, if that's something that you want to do. 

Let's talk a little bit more about ISO, because I mentioned it quite a few times and despite what a lot of us think, including me, cause, I'm guilty of this, myself that I thought ISO stood for the international standards organization for the longest time. But ISO is not an acronym. It is short, well not, it's not short for, but it's stands for the International Organization for Standardization. It's a Federation, a worldwide Federation of national standards bodies, ISO member bodies, and everybody, every country that participates in it has a representative that they send to the member meetings and such, and they're broken down into different committees based on industry.

Many of you may have seen companies that are certified ISO 9000 or 9001. This is a foundational document for ISO quality management systems and the, all the other standards that we have especially for laboratories are built upon this. The ISO 17000 series refers back to it. Now there are different things =under the 17000 series as well.

We've got the forensic standards under there, but we also have standards for companies that produce products that are used in forensics, or that we use to assess the competency of the people doing the work in the forensic laboratory. So those are different 17000 series standards. Now the scientific ones that we use in the forensic laboratory are from the Committee on Conformity Assessment or CASCO. They're the ones that issue it for us. One of the really important things to understand about any ISO standard is that they are voluntary. And if you would like to learn more about ISO and what they do, and a little bit more about the voluntary nature of the standards, I would say go to iso.org. They have a wealth of information on their website. 

So we're going to go through just in general terms what's included in the 17025 standards. 17020 is very similar. There's just a few more requirements in the 17025. Then you have in the 17020 for the inspection bodies. Once we go through those, you should have a better idea of what it is that laboratories have to do in their day-to-day work. One of the things that I find with new scientists that come into the laboratory, a lot of them were super gung ho to get started, very eager beavers, and they just want to go and do it, and it's going to be just like CSI on television. And it really isn't like that at all. Pretty much everyone underestimates the amount of paperwork and documentation that is required in this type of testing. It's little, it seems like it's a little thing, but it's actually an enormous undertaking to ensure, for example, an unbroken documented chain of the evidence from its inception to its final disposition. And that's just one of the things that you have to keep track of. Cause then you also have to keep track of the work product, right? And then you have to keep track of all the people that touched everything through the entire process. Laboratory information management systems or LIMS have significantly assisted in this realm. But of course, then there's the software aspect of things. You know, you've got to validate and verify your software and make sure it's tracking things and things aren't getting lost. The accreditation program that a laboratory develops controls every aspect from start to finish that's going to happen in the laboratory. And it's not just the test that the lab is doing. It's also the people who are conducting that test. The way it's portrayed in the media is not really representative of what it's like as what I was saying, you know, the eager kids come in and they're fresh out of school and they're very excited to get started. The cases are there, the stories are there, the interesting things that maybe grabbed their imagination and their desire to join the profession, you know, those are still present, but the reality of the practical day to day work, isn't something that's captured in those television shows.

And I'm not sure that, you know, university programs, when they're teaching young scientists, the foundational theories, we don't really touch too much on that. So if they're lucky enough to get an internship, that's where you would probably first get exposed to what it actually is like in a forensic laboratory.

I'm not going to go specifically and delve into every little aspect of these standards. I'm just going to give you an overview. Part of that is because I don't want to step on any copyright toes and give out information that I'm not supposed to be giving out, you know, copyright infringement type things. So we're just going to look at it. 

So the scope of, of the standards is specific to testing and calibration laboratories. And it's really just talking about the general requirements for the competency of testing and calibration laboratories and again, I'm specifically talking about the 17025 standards.

So they really focus on, the competence of the people that are doing the work and the laboratory's overall impartiality, which I think everyone can agree is an important aspect to forensic science especially since it's science and it intersects with the criminal justice system. And consistency, so we want consistent operations of the forensic laboratory in country A and country B if they're adhering to the same set of standards, right? So if I've got a laboratory in Canada, that's doing forensic testing and they're ISO 17025 accredited. I would expect that a laboratory in Washington state or in Southern Florida that's also accredited to the same set of standards would give me consistent results if I gave them the exact same sample. And that's an aspect that gets touched on, in one of the standards further down, of course. Right? So it's like any kind of standards document. First, it talks about the scope, what it applies to and specifically it's those three items that I just mentioned to you: competence, impartiality and consistency, right? So we're wanting to make sure that all of these organizations that are performing this type of testing activity, regardless of how big or small it is that we're going to get consistent results. Now the standards also refer to other ISO guides, and so in order to understand these standards, or if you're going to implement them in your laboratory, you do need to have some understanding of those other guides that were used to develop these standards. It also talks about some terms and definitions that are specific to forensic science that, you know, need to have that special, that specific jargon defined. Because, we all know that when you go and do your day-to-day work and you use a word, it's not necessarily the same meaning as when you're talking to your spouse or your kids, right. It's a little bit different. So it's just making sure everyone has the same understanding of the jargon that we're using. And again, the document refers to some external definitions and things. So it doesn't go through and redefine everything. Right. There's no need to reinvent the wheel if you've already got it stored away somewhere. 


[00:17:13] Impartiality
---

Angela: One of the big definitions in the document and you can tell it's important because not only did they mention it up in the scope, but they also talk about it pretty much immediately when they get into the definitions section. And that is, you know, it's talking about what is impartiality? What, what does that mean? And simply put, it's just the presence of objectivity, right? We want to make sure that the laboratory's testing that's being done is free from conflict of interest, right? The laboratory should be able to operate without fear or favor, right? There's no ex-, shouldn't be any external pressures on the laboratory at all. Especially if they could affect the test results. And this is one of the big challenges that forensic laboratories face, because safeguarding the work from these external pressures is, is very difficult, right? Politics kind of intersects, right? Because a lot of these laboratories are publiclyfunded. You know, if you're talking about government labs, obviously, and when you're talking about private laboratories, are they still impartial if they're being, more driven by revenue than by public service, right? So that's a criticism they get. And on the other hand, a government laboratories that, oh, well, you're biased towards the government because you are inherently a government laboratory. So these are different aspects that are very, challenging to ensuring impartiality, right? If we want to ensure that we have, you know, a lack of prejudice, we have freedom from bias, we were able to be neutral and fair and open-minded and detached, right? These forensic cases are traumatic quite often. You know, crimes can be very violent and it's difficult when you've got the, you know, you're investigating the murder of a child, especially if someone is a parent, to detach themselves from that and just work the evidence as it's presented, without any thought about, you know, the actual crime itself beyond using it to understand what type of tests you should perform. And that is a difficult thing to do sometimes. So it's something that you have to be aware of and protect against in your processes. Impartiality is the first section under the general requirements in the standards. So it's the first meaty section of the document. And again, that tells you how important it is to the successful operation of a forensic science laboratory. And one of the things that the laboratory has to do is... besides not ignoring the possibility that there are things that could threaten impartiality. And these things are both internal and external to the laboratory, because internally, there's a lot of pressure to get these cases out the door now, now, right? So you can't be sitting on your laurels. They, you know, you've got, you've got quotas to meet and does that kind of pressure affect the scientist's ability to be impartial in how they're doing their work? We're not saying that these threats to impartiality or these risks to impartiality don't exist, but we have to have processes in place to recognize when they're occurring. Analysts are required to speak up if they have a conflict of interest where they see something that looks like a conflict of interest, right? Because the risk assessment is going to have to be done. Right. So see what can we do to mitigate this risk to impartiality? And if they do identify it, what have you done to safeguard against it, to ensure that we've eliminated or minimized that risk? And we're going to speak a little bit more on this topic in the next episode, when we kind of delve into some of the scandals and issues that have been in the media about different laboratories. We're going to talk about some things that... some reports from a forensic science commission and, and again, some of the scandals that are quite easy to find in the media should you just take a few moments to look. 


[00:21:21] Confidentiality
---

Angela: One of the other really important aspects of testing is confidentiality, right? So what is the laboratory doing to safeguard the confidentiality of the work? Now, this kind of seems to fly in the face of transparency right? Remember I said that the name of this podcast is secrets from the crime lab. And I was trying to be a little funny with that because there shouldn't be any secrets in the crime lab. Right. We should be open. We should be transparent. But that does not mean that we do not safeguard confidential information. People's genetic information is confidential, right? We don't want to be sharing that out in public forums. Right. But so the people who are allowed access to the information that is specific to individuals in the case is limited to those who have a right to have access. So typically this is going to be the customer that has put in the request for service with the laboratory or other entities or persons that are legally entitled to have access to it. Right. So somebody who works for, you know, a newspaper, can't get all of that information. Now, if that information has been given to the court under the correct disclosure processes for the jurisdiction, and then the court makes that available in the public, then of course, journalists could have access to that portion of it. Right? So, but when it comes to the actual personal information, the case information, you have to be legally entitled to have that information in order to be given access to it. A lot of times you can get access to it, but the laboratory has to have written authorization from the customer before they can release it. The document does talk about ensuring that you have processes in place to protect the confidentiality. I'm going to qualify this the same way that I qualified impartiality. Confidential does not mean that you can't still be transparent. A lot of laboratories have actually gone down the road of sharing their policies and procedures on their organization's website. Now, private laboratories may not be doing this as much because a lot of their processes might be proprietary, but public laboratories by and large have started to put things on the forensic science laboratory's website. So I know for example, the Virginia Department of Forensic [Science] has been doing this for a number of years. There's the, I think it's the Houston, the Institute of Forensic Sciences, they... they've been doing it too. One of the other things that's been great about the transparency is they've been putting any time they have things that their quality assurance program recognizes as having been a problem, -that sometimes these call these quality action improvement process, or, you know, they call them QAP or corrective actions. Anytime they have an issue that could affect the testing at the laboratory, they are also posting those on their websites. So they're publicly available. So if you have a case that you're involved in in Houston, and you want to see if that laboratory has had any issues with any of the equipment or things that may have been used for your case, you can actually go to their website and see if they've had anything that had a date that coincided with when the work was done on your case. So... releasing your information about the processes that you do, the laboratory testing or the laboratory policies is not a breach of confidentiality. And it is in a very laudable way to be transparent to your customers and the public at large. 


[00:25:13] Structural Requirements
---

Angela: The next aspect covered in the standards has to do with whether or not the laboratory has the correct structural requirements to be in conformance with the document. So, first of all, the laboratory has to actually be a legal entity of some type. So if it's a government laboratory by virtue of being government it is a legal entity, but if you're talking about, you know, a private laboratory, you know, for example, there's very large groups like LabCore and Bode Technology. These are actual companies that perform forensic testing, right? So they've got their incorporation paperwork, right? They, they exist as a legal entity. So that's the first aspect of it. But the part that I wanted to touch on in this section of the standards is that the scope of the activities, right? The type of laboratory testing that the lab is going to do that is in conformance with the standard has to be defined. A laboratory can do all sorts of different types of tests, that does not mean that every test that they can do conforms with the standards, which means they cannot claim accreditation for those tests. For example, one of the laboratories that I was with when they first became accredited, it was only the DNA testing services that were under their scope. That was the range of activities that they had stated were in conformance with the standards, but they also conducted toxicology. And at the, at that time, toxicology did not meet the standards. It was not on the laboratory's scope of accreditation and the laboratory could not claim accreditation for anything under that forensic discipline. Now, later on that was brought into compliance with the document and then you can apply for an expansion of scope and have it added on later. But that's something to keep in mind when you're looking to see, okay so, my work was sent to laboratory A and they did some DNA and some firearms and some drug chemistry, and some toxicology. You cannot assume that all of those services fall under their scope of their document. There's two things that you need to go and do look at the report for one, because any services that do not fall under their scope of accreditation cannot claim to have it, right? So if the laboratory letterhead has, you know, the logos and things that say, this is who I'm accredited by, and this is my certificate number on it, but they're not accredited in drug chemistry. There should be a statement on that report somewhere that clarifies that any testing done in drug chemistry is not covered under the laboratory scope of accreditation. It is a big no-no if they do otherwise. The next thing that you want to do is look and see who it is they're saying they're accredited by, and you can go to the accrediting body's website and look them up. So the accrediting bodies are required to do that. So anyone that they're accrediting, they have to have the scope document and details on what testing is under that scope. So it's publicly available and anyone can go and look it up. So if you get a laboratory test report, even if it's for paternity testing from a laboratory that says they're accredited by ANAB, you can go to ANAB's website and look them up and double check for yourself. You do not have to take the laboratory's word for it. 

The next point I want to talk about in this section of the standards has to do with the structure, the organizational structure of the laboratory. Who fills what roles and how were they all related to each other within that structure, right? And who's got the authority to do what, who's responsible for these other aspects of testing. And this becomes really important because it does actually kind of go back a little bit to the previous standards with impartiality. For example, if the quality assurance manager notices that someone at a senior level in an organization may have a risk to impartiality, right. They need to be able to have the authority to come up and say something about it and actually be able to do something about it. You know, it needs to be within their remit that they can, they can take action when needed. Similarly, if you're talking about somebody who's responsible for technical operations, right? Maybe one of the analysts that works for them comes and says, look, this instrument that we're using, it's having a lot of problems. I keep having all of these, these faults and these errors. And I'm really concerned that it's, you know, affecting our ability to get good test results from, from the, the samples. So the technical managers should be able to go to higher level managers and say, we, I'm sorry, we have to stop testing because there is a risk to the validity of our test results. Right. They shouldn't. At that point, they should be the final arbiter to decide that right. We have to halt testing because if we do otherwise, we're going to be reporting out incorrect results. Having the authority to carry out your responsibilities, understanding how all of the different personnel that fill the roles in the laboratories, how all of those relationships work it's very important to have those clearly defined so that everyone in the organization understands how it works and how those processes relate to each other. So it's important to understand that if someone is, you know, based on an organizational chart, lower down in seniority or authority within the organization, that people who are senior to them in the organization, understand when they bring a policy or technical issues to their attention, they're doing that as part of their job and telling a senior person in the organization. I'm sorry. We have to hold testing. doesn't mean, and then, you know, the, the manager says, no, we can't stop testing because we've got customer demands that have to be met. The manager has to recognize that the, the person that's bringing them, that information isn't being insubordinate. They're not trying to undermine their authority. They're identifying a risk to testing. And that's something that everybody throughout the organization, when you have a properly functioning management system, you have to be able to do right. Everybody's invested in this quality mindset. If there's a risk to testing, you have to take action and people have to have the authority to step up and make sure that happens. 


[00:32:04] Resource Requirements
---

Angela: The next major portion of the standards is entitled resource requirements. And I have to say when I first learned how the standard had been reorganized and that resource requirements equated personnel and equipment, you know, all as resource requirements, I kind of had a little bit of a chuckle about it, but then if you think about it, yeah. Your, your personnel are part of the resources that the laboratory needs in order to do the work... looking at it that way and you know, from the aspect of, well, it, it is, has always been called human resources. So it does... it does make sense. I don't know why my brain just didn't... didn't ever kind of connect it that way. But the important part for this section is ensuring that the personnel that you have in the laboratory doing the work, whatever portion of the work it is that they do, even if it's just a small little part of it, that they are competent to do so. The laboratory has to have processes in place to ensure that the personnel are competent. And of course the standard doesn't tell you how to determine whether or not a person is competent. They just say you have to have processes in place to ensure that it happens. Now, typically this is because this is done by selecting your personnel based upon the type of education they have. Do they have the relevant university degrees, for example, do they have an advanced degree? Is that necessary for the type of testing that you're going to have them conduct in the laboratory? Right? So their education is important. What kind of training do they need to do? Going back to our eager beaver people that have entered into forensic science, you know, they just went and they finished their four year university and they've got their bachelor of science and then moved on. Maybe they, maybe they've gone and they've also gotten their master's of science as well. And then they come into the laboratory -you mean, I have to go through to your training program? Yeah, because it takes time. So it's not just being trained in the methods that we're using in the laboratory. It's training them and understanding the quality assurance program and the management system and making sure that they understand that they're part of a very large process and all of the pieces of that process need to work together to ensure a quality product on the end. That's one part of it. But then of course, there's mentorship time, right? Because you need time to work with experienced scientists that have done it, because every case that comes into the forensic laboratory is different. We might be all working burglaries, but they're not always exactly the same. There might be sexual assaults and there might be murders and all of them have different circumstances surrounding the case. So when you're being trained in forensics, again, it's one of those things where we're providing you with a toolbox and guidelines on the types of tests that you need to perform to find the objective evidence that would be relevant and probative in that particular investigation. And while there are similarities, there are unique circumstances in pretty much every case that you're going to encounter. So your training program and the subsequent time of mentorship is essential to ensuring that your laboratory staff are competent to perform the testing. And then of course you need to make sure that you've maintained records that they've gone through training. So you've got records that they have the correct education, you have records that show the training program that they went through, how they performed on it, and then final authorizations to show that they are competent and have been approved to perform that type of testing in the laboratory. Now, if they come in and at first, all they're doing is urine drug screening, if then they go in there, say, okay, well now we're going to learn how to do GC/ MS confirmations of the drugs- well, that's a different method. So you have to be trained in that method and you have to be competent ...deemed competent in that method, usually from through a competency test or a qualifying test. And then of course you have to be approved and authorized to conduct that testing. And the laboratory has to maintain records of all of those happening. And so this happens every time, you know, an examiner or an analyst learns a new method. They have to demonstrate competency in that method. If the laboratory says, oh, look, there's this brand new instrument, and it's going to be able to do this different type of test. And this is going to be very helpful to our customers. You bring that method in- we're gonna talk about how that's done later on- but you bring that method in and all the people who are going to perform that test have to go through a training and competency determination before they can actually perform that test in the laboratory. Competency is absolutely essential and ensuring that it's properly documented is equally important because if you didn't document it, it didn't happen. 

The next aspect of resources that we should look at are the facilities and the environmental conditions does the laboratory ensure that its facility and the environmental conditions in it are such that we're going to prevent any kind of deleterious change from happening to the samples that are there for testing. Simply put ... you're monitoring temperatures, you're monitoring humidity, if humidity is going to affect the results, right. We want to make sure that wherever we're storing the evidence, whether it's at ambient temperature or in a freezer that we're monitoring that to make sure that it doesn't get too hot for example. It's very important on biological samples is that. Heat and humidity. These are things that can be destructive to biological samples. And of course, you know, anybody who's lived in a high humidity environment knows that, you know, it doesn't take much for mold to start growing and mold, it likes to eat up your DNA samples. So it's very important that you're controlling environmental conditions. Also, if you're working with volatile substances and things like that, you certainly don't want to be storing something that's going to ignite above 85 degrees, you certainly don't want to have that in hot temperatures either. And monitoring fridges and freezers it's a little bit of a time consuming process sometimes, but now there are some remote systems that kind of notify you immediately if something goes outside of its temperature range. And it's not just the samples, it's the other items that you're using for testing as well. So the, the chemicals that you're using a lot of them have specified temperature ranges under which they need to be stored. It's the same thing when you think about, you know, that that jar of mayonaise that you use on your sandwiches, right. So you need to keep that in the refrigerator after it's been opened, because otherwise it's going to spoil, and it's the same thing when we're talking about laboratory chemicals and reagents that we use to do the tests, right? So you've got to make sure things don't go off.

The temperature monitoring and notification of the customer, if that environmental condition is such, that it could have caused an issue with the samples you are required to notify your customer- if that happens. And an example of where laboratory got into trouble for this was the Austin Police Department Crime Laboratory, which doesn't exist anymore as it, as it once was it's, it's been absorbed into the Texas Department of Public Safety. In addition to some previous issues, then it came out that they had had some issues with their freezers that were storing their samples. And so there were a lot more cases that were potentially affected because of the storage issue and the issue was compounded because they hadn't notified their customer. So it's a very important aspect is to ensure the environmental conditions are correct for the type of work that's being done. And if something happens to, to knock them outside of the range that they're supposed to be in, you need to assess the risk to the samples and the testing and if it's something that is a risk, you need to make sure that you've notified your customer that it happened.

The other important part about your facility requirements is ensuring that incompatible activities are separated from each other. For example, you know, if you've got a drug chemistry section, you'll have some evidence that's like bulk drugs from bulk seizures and things like no bricks of cocaine, bushels of cannabisand you would not want those items to be examined or worked with in the same space where you've got someone else who's sampling scales, for example, checking to see if there's trace amounts of a particular type of controlled substance, on ... a weighing scale, or if you're talking about a very small quantities of drugs that were found on a person, you certainly don't want those types of things to cross-contaminate each other. So you'd want to make sure that those are handled in separate spaces, entirely. Now, similarly, we have that issue when we're talking about DNA testing as well. So you're looking at contamination prevention measures. Essentially the laboratory needs to document what those are. So for, for DNA, it means that we've also separated out the different steps. The supplies that we use to do DNA testing are kept separately from any kind of DNA containing sample: our controls, our positive control samples shouldn't be kept in the same place where we keep the nascent reagents that we would use, right. They need to be in different places. So we don't accidentally cross contaminate things. The area where you examine your evidence is going to be different than the area where you're extracting your DNA. These things have to be separated by time or space. So you can use the same area for your evidence examination, you know, happens sometimes in really small laboratories. You have to ensure that you have a process in between that ensures that you've decontaminated that space and you haven't introduced something into the samples. The next step after that is once you start going from your clean, nice, and pristine DNA sample that you've, you've extracted from that bloodstain- you also have to make sure that anything that happens at the next step, where you're doing PCR, where you're making lots and lots of copies of that DNA sample, you don't want that to go backwards and get into your DNA purification area. All right, so you're going to try to make sure that happens at a different place and they're completely separate. And this includes too, you know, like they have one way in molecular biology or DNA testing laboratories. It's a one way workflow. Right. You go in one door and you go out the other, you never go backwards because there's such a potential for carrying contaminants from the later steps back to the earlier steps.

Now, similarly, then you start talking about the equipment that you're using. A lot of the stuff for drug chemistry and toxicology, you've got a little bit of overlap in the equipment that are used. So you have to ensure that the equipment, even though you can use it for both of it, you would want to make sure that the one you're using for trace drugs and toxicology is separate from the one that you're going to be using for where your bulk drugs. Right? So you don't want to cross-contaminate those things. 

And of course, that brings us into talking about equipment requirements. What kind of equipment is the laboratory going to have? How are they ensuring that... the equipment is fit for purpose. That, if it needs to have traceability to an international standard, some kind of a reference material that's defined somewhere and, conformance with it has been documented. Right? So for example, if you're weighing out, you know, somebody's in a drug possession case, for example, and when, when the laboratory is weighing out that drug, they have to ensure that the scale that they're using to do that one is traceable to an international standard. And second that they've performed intermediate checks to ensure that it's properly calibrated and still in conformance with that international standard. And if they have to apply correction factors in there that has to be in there. Right? So you're ensuring that the equipment is fit for purpose and capable of making the measurement that you want. And that applies to any type of analytical instrument that's in use in the laboratory. You have to plan the maintenance on it, too. It has to have a maintenance and calibration plan. You have to ensure that those are done on a schedule. If it's in every six months, thing... some things have to be checked every day before you use it. You have to run calibration standards on it every single time. Then you have to run controls on it every single time, right? As soon as you turn that machine off, none of those calibrations and controls are valid. Again, you turn it back on, you have to do it again. Others, they can be calibrated once every six months or once every year. It just depends on the type of test that's being performed on it. But the laboratory must ensure that they maintain those things, because if the equipment is not capable of performing the measurement then they're not capable of giving you a valid test result at the end. So that's something else that if you're getting laboratory test results back and you're like, I don't know, this doesn't seem right, or I have some questions- you can actually ask the laboratory for documentation of the maintenance and calibration records for that machine to show whether or not it is meeting the requirements for the type of tests that's being performed.


[00:45:29] Process Requirements
---

Angela: Okay. So I'm going to skip over a few points in the standards, and we're going to jump on up to some things that are under standard seven that are very important. And this is the selection, verification and validation of methods. Forensic laboratories can't just read something in a scientific paper and go, oh, that's cool and then just go in the lab and do it based on what was in the scientific paper. You actually have to go through a process to ensure that you've thoroughly tested that method. ...One, you get it working the way it's supposed to work, then you have to determine, is it actually fit for the purpose I intend to use it for. Right? And then you have to test it to make sure before you implement it. And you do that because you need to understand what are the limitations of that test. Under what circumstances does it stop working? Right. And it's different, depending on the complexity of the test. Validations [are] one of the places where a lot of laboratories fall down because they haven't thoroughly validated it enough. One of the big examples of this is the probabilistic genotyping softwares that have been started to ... be used in forensic DNA. And these are software programs that are performing some pretty complex calculations. And part of it is because determining the number of contributors that are in a DNA mixture, you have to actually study two person, three person, four person, five person mixtures, and put them into the statistical software. Right. And test it to see for all of those different types of scenarios. Does it work? What are the limitations? When does it stop working? Right. A lot of times laboratories will only look at two person mixtures and three person mixtures, but they won't have specific information for four person mixtures. My opinion, there's very limited value in looking at anything that's a four or 5 person mixture, but that is a very involved discussion for another day. But if a laboratory is going to look at mixtures that are that complicated, they have to have a validation study to support the fact that they're doing it. Validation studies can be quick and simple. And sometimes they can take you months because they're not simple at all. But they are absolutely essential to ensuring the validity of the testing that's done at the laboratory. And remember, once you've validated a new method, you have to draft the policies and procedures for how you're going to carry it out in the laboratory. You have to develop a training program so that you can ensure that the laboratory staff that are going to be doing that test are competent to do so. 

Now the next interesting aspect has to do with sampling. This is, sampling is something that can be difficult to understand if you're not in, in testing, but, you know, for example, if they have a very large drug seizure. And there's, you know, a hundred bricks of, cocaine. Well, you're not going to sample every single one of those bricks to determine whether or not they're all cocaine. You have to have a sampling plan. And there are statistical methods that you should employ to determine how many of those items do you need to sample to ensure that you've gotten a representative sample and then do your testing and your reporting accordingly. So there's some additional training that laboratory personnel can go through to understand sampling more, but just for you guys to understand that you don't have to sample every brick to have gotten a representative sample, right? You can just test some of it, but you have to have a plan and it has to be documented and it has to be available for the scientist when they're onsite doing testing.

So I'm trying to get through these standards as much as possible and wrap up. So we're almost done with it, but even though I've skipped over a bit, if you're really learning about the standards, you know, it's at least a week long class, probably all day long with not enough bio breaks for you. If you are interested in learning more about the standards, usually the accrediting bodies do provide some basic training and for you, some of those are free, some of them aren't. They may provide some free webinars to just kind of introduce you to the topic as well. But if you really interested in learning more I would suggest that maybe it's worthwhile for you to pay a small fee, to learn a little bit more about the standards that apply to the laboratory that you're interested in learning more about. 

We also have to look at, excuse me, while I turn my page and make page turning noises on the recording. Handling of test of calibration items. So some of this goes back to things that we've already mentioned, but you have to make sure that you've tracked those items from start to finish through the laboratory processes, that you've ensured that they've been handled correctly through the entire process, and that you've done everything you can to minimize things that could compromise that sample. And if you're transporting it anywhere, you need to have processes to ensure that no kind of deleterious change happens to it, that the item isn't lost, it's not compromised in any way. And then you also have to talk about, are you going to retain those items? Are you going to dispose of that item? How are you going to determine what its final disposition is? So all of that has to be defined within the management system. Basically the policies and procedures that the laboratory has, and you have to maintain technical records of all of this. So from the point that that the sample arrived in the laboratory for testing, all the way through the testing process, through the technical review process or verification process- which is essentially the peer review where second person looks at it to ensure that you followed the procedures, that the procedures that were used were the correct ones, that they met the customer's request, that they were valid procedures and that you're, they agree with your opinions interpretations because the other scientist is not required to agree with your conclusions. If there's some kind of discrepancy between the two scientists, laboratories have to have a way to arbitrate that and determine, you know, is one person correct, and the other person incorrect, can we not make a decision? And you can actually seek external advice for this- to have somebody who's completely external to the organization, kind of be a final arbiter of ... these technical disputes, if that's necessary.

Part of the other thing to look at with technical records is ensuring that if you've got a result where a measurement uncertainty is important to understanding the result... a lot of times this comes up when you're talking about, the, the amount of drugs in a person's system and whether or not they were impaired. So you'll see the toxicologists will report a level plus or minus a certain amount. Right? So if, for example, somebody right next to the cutoff on the amount of cannabis in their system or THC that they have in their, in their urine sample, and you look at the measurement uncertainty and, you know, maybe it's plus or minus 10, well, you know, the cutoff is 50 and they were at a level of 60 and then you apply the measurement uncertainty, well that puts them at the cutoff. So then, you know, that is pertinent to the customer on whether or not they would move forward with prosecuting someone for being under the influence of that substance. Right. So measurement uncertainty is important and there's ...laboratories have to make sure that part of their method validation includes measurement uncertainty, and they have to monitor that over time as well, because it can change. And it does change depending on the person that's doing the test. So those, all those things have to be taken into account when you're doing the test and you're reporting the result. The customer needs to know what the measurement uncertainty was. 

All right, so you've made it this far in this conversation. Thank you for sticking around. We are almost finished. Couple of big bullet points to talk about is standard 7.7, for example, is how do laboratories go about ensuring the validity of the results? Well, one it's all of those things that we've already discussed, but they need to make sure that they have a procedure where that they use for monitoring the validity of the results, right. And making sure that things have been recorded properly and that they're reviewing these things on a regular basis. So if they notice that when they used this certified reference material, if there's one instrument that suddenly starts going... it's not passing its quality control tests, right. That's something that they would need to investigate and, and look into further part of this overarching ensuring the validity of test results is making sure that not only is the equipment still functioning and working as it's appropriate as it's supposed to do, and that we're doing checks on it to make sure that that's happening. We're reviewing the results that have been reported. The technical manager goes and samples, you know, a certain percentage of cases that everyone in their department has put out over the last six months and double checks that, you know, we've included everything in there that we're supposed to- the technical records are complete, that the reports that have been issued are true and correct, and represent the work that was done. And the conclusions in there are valid and properly expressed to the customer. And then of course, there's these other little thing called proficiency testing. We've got inter-laboratory comparison and intra-laboratory comparison. So proficiency testing is, you know, 17043, ISO 17043, approved proficiency testing [provider] provides blind tests to laboratories that have enrolled in their programs. And so periodically they send out samples to laboratories and everybody in the laboratory has to do the test. And then they submit the results to the proficiency testing, company who then compiles the results from all the other laboratories and all the people at those laboratories that participated in the test and they issue reports and they show whether or not everybody got the same answer on that proficiency test. Now everyone should be passing their proficiency tests. So if you're an unsure about whether or not the analyst had signed the report you received as the customer, you can actually go and check with the laboratory and find out who their proficiency test provider is, and you can actually ask to see, or even just know whether or not the examiners that performed the work, if they've passed all of their proficiency tests, very simple thing to request. And it shouldn't be something that the laboratory has any issue with responding to. I don't think that you would even necessarily have to go through an FOI request for it, especially if you are the customer.. ... often times in court, you will be asked as an expert witness: are you proficiency tested? Do you participate in a proficiency testing program? And have you ever failed a proficiency test? So analysts are actually used to answering that type of question. And sometimes people do fail the tests, but it's not because they made a technical mistake. A lot of times it ends up being some kind of administrative thing. Like, you know, they, they fat fingered something on the keyboard when they were entering the results on the provider's website. For example, I would say the majority of failed proficiency tests I've seen wasn't an actual failed proficiency test. It wasn't administrative error that you catch on further review and so the laboratory needs to implement corrective actions to prevent the administrative errors from happening. I'm not saying that technical errors don't occur. They do. but I haven't seen that those are the majority of them, at least not in, forensic DNA and toxicology that I've seen. 

The other one is intra-laboratory comparisons. And this is the one you might use. If you want to check and see that all of your, for example, all your DNA analysts are getting the same result from the same sample in a DNA case. You know, even if it's a mock case, you want them all to get the same answer and you want them to all be interpreting the results the same. So you might conduct an intra-laboratory comparison. Or if you're bringing online a new machine that does the same thing as a machine that you already had, you might take the same set of samples, run them in parallel on both machines and do an intra-laboratory comparison as part of validating that machine. So inter-laboratory comparisons, typically participation in proficiency testing programs or intra-laboratory comparisons are a couple of tools that are available to laboratories to use to ensure the validity of the test results. 

And then of course, once you've finished all of your laboratory tests. You need to report those to the customer. And it's very detailed on what information you have to provide to the customer. And I am not going to go into it, but I am going to tell you that for example, one of the standards has sub-bullet points, a) through p). Now some of those reporting requirements are different between issuing a laboratory test report or issuing a certificate of analysis and that's going to be, you know, a lot of the chemistries will do more of a certificate of analysis, whereas, you know, when you talk about DNA, it's typically a laboratory test report. Anything that's in the test report, that is an opinion or an interpretation has to specifically be flagged in there as such so that the customer knows. Furthermore, the reports have to tell the customer what methods were used to perform that test and when those methods are, when those, when that testing was performed. Sometimes it's a bit annoying for those of us in the laboratory, because it's not like I did this test on this day. Some of these things, these tests take multiple days in a row to complete. So it's not just one testing date. Now this is important for the customer, because if you want to know during the period that the work was done on this case did the laboratory have any quality issues or any kind of corrective action that needed to be implemented. And then, you know, if it's a good lab, they might have that reflected on their website and you can go and look and see, okay, did they have any incidents that occurred during the testing time period for my case and were they relevant to the case that was done? Even if the laboratory isn't publicizing those on their website, you can actually ask the lab to disclose that information as well. Some of them might actually require you to go through a freedom of information request to get it. So it was going to take a lot longer, but if you are the customer, you should be able to get that information pretty easily. If you are you know, the defense and you're looking to get that information from a laboratory that was hired by the prosecution. It might be a little bit more difficult. You might actually need to go and get a court order for it. But that information, I would freely provide that information to anybody that ever requested it from... me when I was in the laboratory.

Okay. Now, as I said, that there's a lot of details in these standards and I'm trying not to go into them too much because there is a lot. But you should know that if there's a mistake on a report and a new report has to be issued, that the laboratory has to indicate that on a report, they have to say, it's amended. They have to give the reason for the amendment, and say why they had to change it. That's something else to look out for. So you might get, have multiple reports. Cause sometimes you'll have ... report A and then you'll have report B and then you might have a, an amended report A, ... then you shouldn't be referring to the original report A, you should only be referring to the amended report, but you might have questions as to why they changed that report. And that's, it's a valid curiosity. You, you want to understand why that was. You can always contact the laboratory and ask to talk to the scientist and they can, they can help you. They should want to help you and walk you through what happened. 

And of course, if something goes wrong and you're not happy as the customer, or in some way, you've been affected by the work at the laboratory, you are able to make complaints. Laboratories have to be, have a way to handle complaints and they have to respond to all complaints. Even if all it is to say is we recognize that you filed a complaint and we've referred it to another organization. Cause a lot of times, depending on the type of complaint, they wouldn't be able to investigate themselves. Right. So that's an important aspect of being impartial. Right? .... 

Anytime you have work that is not conforming to any of these standards that we've been talking about. The laboratory has to document that and they have to do something about it. Right. That's what I was talking about when, you know, a lot of them will put information about non-conforming work in the form of quality incidents and corrective actions on their website. So you can actually go in and look at that information for yourself. 

So the last aspect that I wanted to talk to you about just very briefly is the final standard. And it has to do with the quality assurance program as a whole. And this is called the management system. This is the whole kit and caboodle. The whole thing that controls the laboratory processes. When we're talking about everything from what's input into the laboratory, to the outputs that come out at the end and everything in between it's how, how is that system supposed to work? Are we monitoring that system to make sure that it's working? Right. So this goes back to, if any of you guys are familiar with his name Deming, right, back in the seventies with manufacturing. And some of you may be familiar with the process control of Six Sigma. It's very similar concept. It's this idea of plan, do, check, and act. That's your management system, right there. Plan what you're going to do, do what you said you were going to do, check to make sure that you're doing it, and then if something isn't fitting into that, or it's not effective, take action to fix it or prevent it. And then start the whole process over again. So that is the whole process in a nutshell, that's your management system.


[01:03:48] Wrap-up
---

Angela: Okay. So here we are. We're past the hour mark on this recording, and I told you all I was doing was giving you an overview. And that's a lot of, that's a lot of me talking into the microphone about the standards for it to have only been an ... overview, but that should give you some kind of indication of what it takes to ensure the quality of testing at a forensic science laboratory, to ensure that the results that you're receiving are valid. So I hope that, after this, you understand that it's not a simple thing for a laboratory to get accredited and to stay accredited. There's a lot of moving parts in there. It's a very complex thing, regardless of the size of the laboratory, they all have to conform with these standards. Even if, even if conformance with them is voluntary. It still is. It's, it's a lot, it's a lot for them to have to do. And everybody who's involved in that process is human and they're going to make mistakes. So we can't assume that if something doesn't go as planned, that they did it on purpose or that they were incompetent. Sure. That that could be the reason why that it failed, but it could be that it's just, the process needs to be improved and we need to do... figure out a way to be better. Ultimately it's this recovery from errors, it's this actions, it's those actions that we take to correct course and prevent bad things from happening that has really driven change and improvements over time. In the 20 years that I've been involved in this field, that there've been so many changes, and so many things that have been implemented to improve the systems, to make sure that the customers that are on the receiving end of the scientific information, have a better understanding of what those scientific results mean in the context of their case so that they're not misconstrued in some way. And this is something that we're still struggling with all the time, but it is a lot better than it used to be. 

The next episode that we're going to talk about. It's going to be a little bit more about what happens when it's not working. Right. What were the influences within and without the laboratory that kind of maybe compromised the management system and compromised the laboratory's ability to function properly, right. Was this external influence or was it something that was rotten on the inside of the laboratory? We've had several instances that have come out in the media where, you know, analysts were doing what was called dry labbing it, and you won't have to wonder what kind of safeguards were in place to prevent that type of thing from happening? Now, dry labbing means that basically somebody didn't actually do the test on the sample, but they reported out the result anyways. Now whether they were reporting it out after having not done the test as a positive for a substance or a negative for a substance obviously impacts how awful the end result would be, but it does actually bring into question all of the testing that happens at a lab when incidents like that happen. And we know that the national registry of exonerations said that, you know, the, the data that they have showed that false or misleading forensic evidence, contributed to 24% of the miscarriages of justice in the United States. So that's a significant amount there. Now that's not the only thing that contributed to it. Usually forensic evidence was only part of that piece. It's rarely that that was the only piece that led to a miscarriage of justice. If you're interested in learning more about some other, scandals related to crime laboratories, you can also go and look at the marshallproject.org. They track a lot of that information, as well.

Okay. In the next few episodes, we're going to explore a little bit more in what happens when the quality assurance program either fails or isn't capable of picking up on issues and challenges that the crime laboratories have. And as a result, the science suffers and the laboratories suffer as well. And a lot of them lose their accreditation or they have other actions taken against them or the scientists do. And for this, for these discussions, I'm going to bring on a couple of external, independent defense experts. One's going to be an expert in the United States, and one's going to be an expert that works in the UK. So if you found this slightly interesting ... or even if you didn't, I expect that the next few episodes would probably be more interesting for you as they build on the information that I provided in this episode and kind of lets you apply that information to some real-world issues that we have in forensic science. Both of them, like I said, are on the defense side of things, so they've got a different perspective than, than I do. I've primarily worked for government laboratories and not as much on the defense side of things. In my mind, it doesn't matter who hires you to do the work because the science doesn't change, but they've been on the receiving end of it and seen that sometimes it does, does make a difference. And it's not necessarily the prosecution lab deliberately did something wrong. And sometimes they're just not aware of their own bias. Bias, again, is something else we're going to talk about in a future episode. And I'm working on getting us a guest for that one as well.

So if you want to make sure that you don't miss it, please be sure to subscribe wherever you get your podcasts. If you liked this episode, I'd appreciate you leaving a review on iTunes. And always, if you have any comments, suggestions, questions, or any kind of feedback or ideas for future episodes that you would like to listen to, I would love to hear from you. Please email me at hello@secretsfromthecrimelab.com. That's hello@secretsfromthecrimelab.com and I hope that you will tune in next time when we're talking to our defense expert. Thank you. Have a great day.