0:07
All right, next up, we have let me introduce her head’s AI and data science practice and MCL technologies for the Asia Pacific and the Middle East region based in Sydney. She has spent the last 21 plus years delivering decision support systems using artificial intelligence and machine learning. She holds a Master of data science and then the focus is on how to use AI and data science at scale to solve business challenges. Apart from this other area, she’s highly passionate about our women in ai, ai, and responsible AI and how AI is helping organizations. Expectations. Please welcome. Breaking production thanks again for inviting me to this conference. I’m really honored to be part of this great lineup of speakers that we’ve got

1:18
Everyone now definitely shares my spots. I see as well. So is that yeah, yeah, people can see if you want to go into presentation mode.

1:56
In my slide, in your slide, you can go to the present and most one question if you’re not able to see the

2:09
Screen coming up, but I’ll just keep going up and only see the presentation. Providence aware sort of enabling. Alright, so again, thanks up, and hello, everyone. So the topic of justification today. ML ops in AI. Now we are facing an interesting era of artificial intelligence and machine learning and AI helping us to make better decisions to enable us to work smarter. However. There is a lot of questions that need to be answered. Such as how do we scale we how do we build a better ethical and trusted AI system. So that is the topic that I’m going to be covering today. You will see a little bit of alignment to what he was saying and hopefully will be more on the lens of how ml Ops is actually going to help in addressing some of the requirements and challenges because he just went through in detail so so hopefully there is a bit of alignment there. So in order to explain this concept of how we scale gain that trust today, I just want to use a business use case. calculator right the concept so let’s look at trust management in tracking this case I’ve got many years of experience having developed and implemented models for financial services. So what is credit risk is the probability of someone defaulting on a loan or does not comply with the obligations that he had. Now, redness is an important function for packing. Packing. cannot completely eliminate reference but of course, we can put some mitigation in place to address some of the challenges. Now the scope of the presentation now it’s going to I’m going to be covering some of the use cases. What are some of the challenges we are facing in the machine learning models MSP deployment and how Belotti scaping mitigate those challenges? Quickly about regular management stages. So when a customer applies for banks has to make it efficient. So banks usually use a great credit scoring model to help them. Credit scoring is an important patient support system that uses credit scoring models to determine the rate. The taste of scoring models updates depending upon what state it is during the assessment phase. Great scoring is used in this data using the data basic minimum data so that we can quickly assess if the applicant meets the minimum requirements approved or rejected. Then during the loan approval phase, we build a credit scoring model using all the available information about our vision so that we can make a precise decision whether we can approve the loan or not. And subsequently, during the monitoring phase, we use great scoring models to understand the health of the great red portfolio. How was it performing? So one of the things is this indirect process where how greatly scoring models. Now, have a look at some of the AI use cases and machine learning use cases in great risk management and banking. We are using machine learning models to forecast the expected loss that the bank might incur. We are using it for early warning signals. If you want to know if there are any delinquencies in the accounts or we wanted to know about those very early on before the actual before

7:06
Having machine learning models to provide those early warning signals. Anomaly detection is another area where we want to identify any optimal behaviors of the customer, customer, or even the payments that they’re making. And of course, in the grid scorecard, we wanted to assess the initial application screen them whether they comply with all the requirements, and also identify the portfolio management. So these are some of the typical use cases where machine learning has been used. And of course, there are going to be some challenges the AI and machine learning models have challenges in building those models as well as implementing those models in production. So some high-level challenges. You know, the first is the bias. There are many types of biases, biases and want to highlight in the database. And so data bias happens when you’re using the historical data to build a model, predict an outcome and that historical data which have a bias for example, traditionally, the loans credit rotation data has gone bias towards the minority for minorities, women, if we’ve seen the loans are being rejected, those minority groups, then those buyers will get propagated into the AI system and then we get the similar outcomes. So that’s the data bias we need to make sure we have diverse data when predicting the defaulting of a loan or not algorithmic bias this is a bias that can happen when individuals or the developers who are actually leading the system, can introduce bias consciously unconsciously. We don’t have diverse skill sets in the team. So that’s why I emphasize the diversity of women in those jobs as well as other minority groups as well. So you need to have diverse people to build AI systems. And then the next important challenge is Alibaba and the black box. You know, these days more and more sophisticated algorithms are being developed, which the services industries are exploring, exploring. And of course, they are giving us a better, better prediction with more accuracy. However, these models are quite complex. In nature, it’s very hard to understand why a certain outcome was meant. Because it’s a black box is complex to understand. So that’s another challenge. How do you trust an outcome if you don’t know how we know how it was made? The other one of the other major challenge is the operationalization of the models. So it’s often seen that more than 80% of the models which are explained in the experimental phase do not go into production and deploy because of the various challenges we face when we deploy models into production. I wanted to go a little bit into the operational challenges that we have which are quite important. For machine learning. As more and more models are being developed. We need to have a robust system to deploy and maintain. So some of the challenges we see are around deployment. So, data scientists, models tested validated, it’s all good and heavy-handed over to the deployment team. And due to the lack of proper handover and knowledge transfer, you might have might introduce errors or delays, in general, another challenge is data changes. So the machine learning models change with the change in

11:16
The data changes and the parameter changes. While the algorithmic code and hyperparameter changes are within the control of the developers. The data changes. So we need to make sure we have similar versioning as we have. The other aspect that we need to take into account is the iterative nature. So machine learning models have what are experimental and iterative in nature. They do a lot of you know parameter tuning and feature engineering pipeline, pipelines that are used. They actually use the record and the data and hyperparameters. If any of these changes, then you have to redo the experiment and then recalculate the metrics. Testing, machine learning these days undergo a lot of testing we need to test to ensure data quality is there, you know, pre-processing, validation, data validation, algorithmic validation, algorithm, fairness, etc. We need to make sure all of those different testings are being carried out in securities and other things challenge so often we’ve seen that machine learning models become a part of a bigger system, and outputs are actually being fed into other applications to make decisions, and which we may or may not know that might lead to security challenges. Monitoring is another key aspect where we need to continuously monitor the models deployed, to make sure that the model is behaving, how we demand the outcomes are the same, even with a new set of data that is coming through today. As well as make sure that models have not deteriorated over time. So continuous monitoring is very important. Infrastructure again plays a very key role with the more sophisticated algorithms are being built. We need to have the ability to scale and always have the compute power, which takes us through complex infrastructure requirements. For example, when you’re experimenting, what would we have the GPU, and then when you’re going to deployment in production, you want to be able to scale dynamically. Collaboration so often we’ve seen that the machine learning model is part of the last stages of the project lifecycle. Often the developer’s silo building those models and the other team members. For them. It’s a black box and with no proper feedback and collaboration. These are some of the operational challenges that machine learning models face. With the more and more machine learning models we are putting into the system we need to be able to manage these challenges. This is where you know we see ml ops playing a very important role. ML ops have become a key field in AI. And then it’s becoming more and more popular these days many people are talking about. Now, before getting into the ML ops I just want to give a very high-level view of what various ops terminologies are and how they differ. So there is DevOps, which will be focused on the baby’s memory during the infrastructure. You have ideas about the system and network administration. Security Ops is about the security of the IT systems and data centers and the cloud infrastructures that we have. Ai Ops is about using AI to automate IT operations, incident management resolution, so on and so forth. As its name suggests, it’s about managing end-to-end data management and ml ops. This is the vision of machine learning, data engineering, and operations, but that is required for this deployment and monitoring.

15:29
Just quickly looking at what is my loss? ML ops, to me, is a framework. It is the integration of people, processes, practices, and technologies. So that we are able to deploy and monitor the production and ensure that actually delivering outcomes and also make sure they’re scalable, fully governed, provide whites of business. So it’s been to one framework, which makes the anilox very important. It’s, it covers all the different aspects, frameworks, practices, and procedures we need to have in place. As I said, it’s a fusion of machine learning. One is like one case data engineering and operationalization of projects in production. So why ml Ops is important to definitely address some of the challenges that I’ve previously addressed. The three things that I want to highlight are mitigating risk. You know, when you deploy models into production, there is some risk associated and we want to make sure mitigate them and the price scaling, you know, financial firms and most of all businesses are moving from 10s and hundreds of models 1000s of models, how do we scale these models and still ensure that it is going to produce in the right and responsibly this aspect was really covered it very well. And Emma lots actually helps by providing the frameworks policies and procedures. The responsibility is just when you’re looking at three areas where an officer is helping one is the mitigating risk. So this is about performance. Monitoring and adjusting as and when needed to be learning models that are in production so that they don’t have any adverse business. You know, some of the risks that might be you know, in the models when it’s in production is you know, it may not be available for a certain period of time system, the old day’s downtime, whatever may be issue or model, due to the type of data that has been coming into the models, it can provide back prediction, we need to make sure that’s not the case. As time goes by with the PVC and flatness of the models, there is potential to decrease as well. And also, we have the rights cases necessary to be able to monitor these models, refine the models, and address any risk associated with the models. So, these are some of the risk areas that we need to assess and make sure we address them and also your very need to assess the risk based on the usage of the models, whether you’re using it just as a playpen sample environment, experimentation, or you’re actually deploying the models into production. So they actually give me this making some dedication like, you know, uploading the loads and stuff, based on the usage you might want to have controllers and governance implies that they are operational processes in place to understand risk-based approach. So what is the probability of an event occurring and how does it impact? So based on that, you might want to tighten your governance processes and operational losses and regulations are other important things. So there’s a lot of especially in the banking sector, it is highly governed and regulated by law. So you need to make sure the compliance that we have those are whatsoever addressed when we are on operationalizing. The models enterprise scaling, as I mentioned, so these days organizations are moving from a handful of models in production to central houses of Congress, and it all has a positive impact. In order to for us to have a positive impact on the business, we need to make sure we have mL ops discipline in place. You know, we need to be able to have the environment the scale, the ability to use the model in production with high scale data, you know, large volumes of data as well as be able to train more and more a number of models. 1000s of models we still need to be able to train them and ensure they’re actually predicting the right outcome. So how ml ops help with auto-scaling we have the same

20:29
Process in place that teams are in place, which is auto-scaling capabilities. Keep track of the worsening when the number of models increases and more and modern models are moving from the experimentation phase to design, implementation as well as doing experiments in the design phase. To understand whether the return models of the previous version so you might want to continuously refine your models and compare your models in production. To that what you have to find that you know, make sure you actually deploy the new models they are performing better and ensure that the model performance is not degraded in production which we have seen before. So if the price scaling is another key factor for machine learning on a large scale, ml ops have got frameworks and structures in place to do that. The other important aspect with ml ops really helps with your responsibility. I wouldn’t go into the details of it. I guess. He has really covered it very well. But two things I wanted to highlight. It is about building an ethical and trustworthy AI system. So not only brings the business outcome but also it is ethical and trustworthy can be trusted. The two key or key pillars I see for the sign of trust, the trust side of the eye, again, was really, really, really well. I’m just going to skip those to just tell me the kindness that we have. So ethical AI is around accountability who is accountable, the people who are building the model should be accountable for the models. We will have an ethical committee do we have a centralized team that ensures that you know who is building the model who is making the changes if any adverse effects who is accountable for this inclusiveness, our diversity in the data diversity, the people who are building those algorithms. inclusiveness is very important. reliability and safety, you know, have we done enough testing, the CDC agenda models running in production? Have we got the security measures in place? All of this stuff needs to be taken into account and again, ml ops really have got the process to help you trust AI. That’s it on sadness. Abi excluding a particular community, gender-based gender community raise swats for transparency. You know, everybody should know how the model has been built. What data has been used, how the outcome has been derived. We are able to explain the outcome to the people. The issue is being used and privacy and security. We need to make sure we comply with the company’s security standards. The regulations put forth by the regulatory authorities and also individual personally identifying information so be protected. While we are using just on a final couple of slides. So how is ml Ops is enabling trusted and ethical AI? So as you know, as it says here, teams must have mL ops principles to practice responsible and also responsible AI. Actually, it’s a proper ml of strategies to be implemented. So they are going hand in hand. They complement each other responsibilities I say this is what we need to take into account the right framework, policies and standards into a model lifecycle starting from establishing the business objectives to deploying in production and so you’ve got into a framework of skills enable. Final note, as you know, yes, we have all the processes in place we have technologies in place, but the people that need to be assistance organizations need to train the people on the knowledge of AI on responsible AI, what is

25:19
Helping them training technologies and so pretty much wrapping up on sorry Thank you. I was just saying that when passionate people start talking about their area of passion, then it’s hard to stop so I could see that you’re so passionate about this topic. Great coverage of my labs. I wish we had a little more time to go over things in detail. But you know, maybe we can have you again for like another workshop or something where you can give us more details on this and I know you’re very knowledgeable and you have to share so maybe we can have that but thank you so much for your time. I know it’s like really early morning for you in Australia. Really appreciate you not calling Thank you for having me. Actually. Afternoon so it’s not too bad. So it’s okay and I love to share and learn so take care. Bye