hello welcome everybody for today's broadcast in English from the studio here in Copenhagen I'm panila the AI program lead of fellow mind and today we're going to hear from Danish Railways and with me in the studio I have uh ingred and a little bit later on once I've done a few introduction slides on fow mind I'll give the word over to you and I'm very pleased to have you here today and to listen to all the great um experiences that you've had with the co-pilot for Microsoft 365 so far so uh let's just uh dive into a little bit of slide presentation and run through that uh rather quickly so yeah this is us obviously uh working out from fellow mind a large European Microsoft partner uh 2,000 employees across the northern Europe and uh what is what is it that we do well we uh make people enjoy working with technology and make technology work with them which is uh for today's topic very relevant because it's all about having a co-pilot by your side not an autopilot but getting the help you need from the technology uh that is relevant to you so this setup is part of a larger Series so we're having like six different angles on AI currently running and uh we are number four uh running today but as you can see we also have uh something on the 25th of May and uh the 2nd of May so uh so tune in for those episodes or watch uh the old episodes uh if if interested and with those words I would like to hear from you ingri get a little introduction uh on yourself what is it that you're doing at the Danish Railways currently thank you and thank you for having me it's a it's a pleasure to be here today and to uh have you guys out there listening um my name is ingred hegberg and I work as a senior prodject manager in in the Danish main train operator dsb so Danish Railways or or dsb as we're called uh and uh as said I work in uh in our strategy department and that means that I have the privilege of diving deep into different strategic priorities that we have in dsb and since last summer uh that really for me has been AI um and the background of that is that we uh during last summer we we formed a strategic initiative around AI um acknowledging you know that this really is a a gamechanging technology and uh we have we have a really good like an excellent technical team working in the space and so on so we had done quite a lot in the area but it just with all new things coming and with generative AI being such a huge part of of uh the discussion in society uh we formed a strategic initiative around it and I myself have a a huge interest in in exponential technology development and have been working in that space for a couple of years uh so I couldn't kind of keep my my hands off so I also took part of uh of leading this initiative that we formed uh there and since August my main focus has been on gen um exploring what that can do for our employees for our colleagues um exploring new ways of working and how this really can uh create some opportunities for our co-workers um and of course co-pilot then is a natural part of that I think there's a lot of listeners or viewers out there today very keen on on this whole topic and also looking into implementing these tools or some of these tools themselves uh so if you're out there and if you have a good question for us if you're listening or watching this through the what we call the Liv storm that means that you've signed up uh for this through our website or another link that's been shared with you you can just uh pop qu questions uh in into that platform if you're watching from LinkedIn you're you you don't have that uh possibility uh obviously but then just lean back and enjoy because ing you you actually have really good experiences from what you have done so so not only are you like very into technology and and a spokesperson and also very energetic about it so you do like I've seen you on stage you're like out there you you really want people to engage in this and that is uh that's awesome uh to see because it's also quite important uh don't you feel it when doing stuff like this I I absolutely agree I think this is it's such an interesting time that we're in and there's so much happening uh not only for us in DSP I mean absolutely not isolated for us but this is a global agenda that is constantly moving and I feel that being extrovert and talking about uh what our learnings are H is a is a great way to inspire others but also to learn from others because you the more extrovert you are the more uh other people want to knowledge share with you and the more learnings you can get from the outside world as well yeah so for I think this is a such an interesting agenda to be kind of um collaborating around because it's not unique for us it's not unique for any any other company or agency or organization it's really a an in uh development for society and I think it we need to kind of figure that out together yeah I agree uh very much indeed so so what we're going to dive into in the beginning is just a presentation of what is it or what was it or where are you kind of what was the whole project and how did you do it and and um and then later on we're getting into some of the the findings that you made that's very probably like a cliffhanger for somebody who who might be thinking of jumping off right now we're coming to the results so stay with us but uh initially we're going uh to just maybe present to you how it was all built and the whole scope of it and maybe other stuff that dsb is actually currently uh diving into also in the realm of AI so I'll just uh give you uh the stage and uh and also the slide presentation so uh just let us know if you want to if I want to show anything but I can maybe just start a bit with the background because as I said we formed this strategic initiative in uh Last Summer and we uh there's different parts of it looking at more how do we transform our business and core parts of our business using not only generative AI but more traditional Ai and so on but then there's this whole Space of modern ways of working um enabled by geni uh and that has been my focus area primarily and last summer when we uh when we started we knew that Microsoft were coming with products and we knew that co-pilots were coming but the timeline for that was somewhere out in like 2024 I mean where we are right now right so we started building our own tools and we started to put together gen based tools that we could offer to our colleagues so that we could start learning and testing these capabilities together uh and we launched our own dsb chat upt in uh in September and then came October and Microsoft surprised us they took us by surprise I think it was in October by saying you know we're launching co-pilot for Microsoft 365 1 of November we're going General availability and we were uh a bit I said taken by surprise and we also needed to then figure out you know okay we have these tools that the people started in our business really started to like and we got a lot of traction and a lot of attention and we were out like talking about the potentials and pitfalls with our colleagues in order to mobilize the organization around what these tools can do uh but these were the tools that we built ourself on top of Asher and all of that uh and now came a a off-the-shelf product which is of course a Ben like a beneficial solution if that works with being connected to everything uh I mean we have as many companies do we have uh great use of of all the Microsoft uh 365 Suite with I mean I don't know how many PowerPoint presentations I've created in my life um so so we absolutely saw the potential and we said to ourselves that we need to explore this as well uh and this needs to be part of our portfolio to kind of understand the potential uh of this one also in relation to the things that we already had um so we started uh to reinitiated a test and learn process we called it because for us we our first step needed to be okay what can this do also knowing that we're a very Danish organization we have a lot of international colleagues in the organization but a lot of the communication and the work that we do is in Danish um and we knew that this was only available in English but we knew that there were still some things working with Danish and we started to form like we tried to form a test group that we actually could uh leverage for for testing the English version uh and just to figure out how can this create value for our employees and um I'm going to share a bit of the slides that um kind of explains how we did it because we um we um what we decided to do quite early was that we wanted to test it with some relevance users and therefore we formed a super user group that was part of our test uh like part of our more detailed test where we uh found in the more englishe speaking departments in our organization which is it and S&T it's called it's the division that buys new trains and so on yeah um there we have uh English as our as the language of corporate language the corporate language of those uh those divisions yeah so we decided to quite early to see say you know let's find a group of people um Within These divisions that can uh can test it out for us uh so we formed this like like more focused test group with one or two users from each team inside of that division to uh in during an 8-week period really try it out for us MH and and that group were given more love and attention so together with Microsoft and also together with with fellow mind we put this process together where we had uh we had a kickoff we had a Midway sharing we had a finale that was like the three Center points where we gathered people uh in order to uh really kind of show the potentials of the tool get people to share how they used it h and really like get them engaged around test and trying it out together with us yeah uh and we that was that was the Super us that we see yeah exactly and we had open Office hours and we did uh we did check-ins with them to see how they were doing and how their process was and so on um but at the time that we bought licenses you had to buy 300 licenses in order to to be part of the game right yes um that has changed since but uh for us that meant that this super user group was like little less than 100 so we had licenses that we also wanted to test with other groups um so we also decided to involve a group of managers um that also became one of the test groups and that's number two that's number two and then we still had some licenses to spare so to say and we also had an an idea that people that have a natural interest in that like like from other parts of the organization that has a natural interest in this that we also wanted to provide them some licenses and that also gave us an opportunity to have kind of three different user groups that had three different uh levels of um support So to say uh in order to see kind of what is needed for us to actually succeed and to get a good experience out of this that was at least our hypothesis but I'll get back to that uh when we look at the results yeah um yeah so did it was it like now you're running uh uh a test with 300 licenses you you would like to utilize all the licenses at once obviously to be able to to kind of measure this the value Creation in the best possible way and you focus on 100 users that you call the super users and they get your full love and attention and then the other groups they they don't get that much so did they know that that that that could have been in another group or that you were actually handling it that way uh I think there was information uh like we shared information about it but then how much people actually understood the different levels like the different groups I'm not I'm not totally sure but what everyone knew that was that this was a test process yeah so even those those people who got like just an early license they knew that this is part of a test um we're trying it out together uh and we love your feedback and we want your uh like your learnings yourself but it's not a roll out so it's it's also about managing expectations but of course because of course like if they're given a license and they try it and it doesn't work have we have we missed an opportunity to roll it out properly but this was never a roll out this has always been an ambition about testing and learning uh and doing that together with um the people from the business because I really do think that the collective effort of this test group has been such a uh like insightful uh Journey because if we were 10 people just testing it out we would never have had those versatile learnings that we've had from this test group so at the beginning at the very beginning of the pro process did you have like a plan for how you want to measure it measure it like was it all planned out we did I mean we of course it was difficult because we didn't when we designed this process we hadn't even tried it out ourselves so it's always difficult to kind of design a process for a tool that you haven't used yeah uh but we had some expectations and some some hypothesis based also on the other tools that we had and and in general kind of the tools out there on the market with everything from open AI to uh the whole Suite of of opportunities that lie out there uh so we had three hypothesis and that was that a co-pilot could um we wanted to check if it could help us to um make more with less time so on productivity to kind of you know do things faster uh free up time for more value adding activities and so on uh and then we we had a second one which was about quality uh cuz we one of the learnings from our like when we had worked with our own tools during the fall was that uh it can really help us to create sometimes things with better higher quality because we can test and we can iterate and we can get feedback from gen like give them a project plan and and ask you know what are the uh deficits or what what what improvements can we do here they never closed they they always open feable like you can really use it as a sparing partner I think that's one of the one of the surprises for from my end was really how much people used gen tools for increasing the quality of the work yeah um and not only for producing more like faster uh so that was our second hypothesis that we wanted to check and then we had a third one which um focused on Modern ways of working because we also I mean we're we're a uh a train operator and we're a huge organization and we're a we're an employer and we of course want our employees to have a a nice working environment with good good tools that help them to uh perform the best work that they can um so we also wanted to check is this something that we believe supports a new and Modern Way of working yeah so uh three hypothesis uh quality yeah productivity and employee engagement modern work style modern work SL employee branding I think for us it's important in terms of talent and in terms of uh yeah in terms of uh attracting talent and keeping Talent of course we want to be a nice place to work and and that's part of part of that um agenda so uh so you think like just from a personal perspective would you like if you were to to change to a different uh place to work would you consider that would you consider the Tool Part like the package of tools would you ask I don't know I just from my imagination I what I would maybe sometimes also expect like I when you if you if you started like maybe I wouldn't ask in a job interview what tools do you use but I do have some expectations and if those are not met when I enter a position I would get frustrated as an as an employee to not be supported by the tools that I know are out there that could help me in my everyday um and yeah and I think it's more of of that and I think offering uh offering great tool tools and being open about also when you try new tools and Innovative tools I think that can attract people uh and I think that's also um why it's important to also be be talking openly about it because I think it's it should be uh something that more like having a good Suite of tools supporting your uh employees should be a priority yeah I agree so I would be very frustrated also myself if that was not a possibility to use some of these tools because I I've already developed some kind of an addiction to to towards working with them and I think that's right now we're not probably there yet because it's still so new to the market but I could foresee that in I don't know a few years uh depending on development of course and depending of how successful it becomes it could be a a way to differentiate yourself if it doesn't become household for everyone I mean we'll we'll uh live to see it uh so uh true we will yeah that's that's very exciting so uh should we dive into some of the results that you that you found during this process let's do that and and again I should emphasize that our test process was only eight weeks and uh it ran between January and like early January to February so it was eight weeks uh where there since even have been improvements and so on and it is the version that was available or is still available in English and not in Danish so it of course does have effect because I think uh we should also re or acknowledge that even though we tried to find a Target group that fitted and were suitable for testing the English version the Danish language is very prominent in in dspb and some of the um uh applications in co-pilot in for example co-pilot chat and co-pilot in word and PowerPoint can to some extent understand Danish and handle the Danish language even though you need to prompt in English uh but in Outlook for example uh if I click the summarize button it doesn't recognize the language of the meals that I have so I think also that needs to be taken into account when we look at the results just so we have that in mind yeah um but if we do look at the results we can see that uh overall um what our users report back and it should be said that I showed the different user groups before and this the result here are actually based on uh the results from all the respondents that we had because we did do a final survey with everyone uh that we did together with you guys in fellow mind yeah uh so we uh these results are based on the feedback from from all the users that have tried co-pilot so we do have I mean I think everyone who has rolled out licenses have a lot of licenses that are not being maybe not have been used so but we've looked at the people that actually have used it and what their uh feedback has been yeah and if we look at the first uh parameter which was productivity uh we see that uh we do actually have almost 80% reporting back that they do expect that they have some time savings coming in from co-pilot and should be said that this is self-reported so we haven't been out measuring you know how long does it exactly we didn't do that we did a this is survey based and this is uh based on on their own experiences and expectations uh but 80% of the users and 84% of the super users indicate that they do save time uh using co-pilot and we have an average saving of approximately 34 minutes per week and this is looking at everyone that has responded both the satisfied user users to the ones that think that maybe this isn't uh for me this isn't for me or or it doesn't yeah they have different experience we'll get back to that so uh we see that we get kind of a a a quite POS positive result there I would say h but it's more if you look at it on an average it's still half an hour here and there it's not the um we um when we come back to looking at the numbers of productivity it's more in that space and not the you know I've saved very many hours every week I know but some people save more hours and others save less yeah and we'll get back to that yeah uh but then also for Quality that was actually um there we have lower results I would say so we have uh uh 48% of all users believe that co- pallet can help them improve the quality of the work and I would say I would expect that this has a lot to do with uh co-pilots maturity in the productivity tools like in the um if you look at PowerPoint and Excel that if we were to map them out on a maturity uh line scale scale I would map Excel the lowest and then comes PowerPoint and then up on the a bit more mature is word co- pellet chat Outlook and so on and teams yeah so I think that kind of indicates also that the maturity is it still leaves kind of room for improvement in Excel and PowerPoint that I think many had expectations around also considering um the really cool marketing videos that gave some promises that yeah I don't think maybe we were fully there yet when they uh when they decided to launch early true so so um that's I think that's actually quite positive especially also on the maturity scale so so quality of work okay so let's say we get the Excel let's say we get the PowerPoint a little bit more mature uh we get the language like suddenly we are going to hopefully get to where we actually standing with a tool that is uh quite uh valuable to the employees I I think that's an assumption yeah I know and I think we'll get back to that assumption uh in a bit but I think I agree and but I think also that considering that it is also an early version of the tool I think one experience has been that sometimes there's some issues with uh you know the tool is constantly developing so a prompt that you tried yesterday might not work as well uh today so I think that has also been kind of a um I think this results actually also encompasses that a bit uh but we'll get back to to that um but then also I think from Modern ways of working I think the message is very clear uh I think that U it's safe to say that our colleagues and and the people that had have reported back on this uh believe that this is supporting a modern way of working yeah and you can see that the result is 84% of uh all the uh respondents and as high as 97% of the super users do believe that this is part of modern way of working that's that that should be just reflected upon just for a little while because that actually refers back to what we just talked about changing jobs with with or without this tool in the other end uh so so if if the employees if we Ser ourselves as colleagues and as workers every day uh actually feel this tool is the way forward in in working yeah this tool or other generative AI tools I think this this movement of and I think the the expectations and and we will get used to it as you as you referred to yourself I I think that uh development will go quite fast uh because it's not like what we did did during the fall as well we were out talking to uh almost all of our employees that have administrative tasks or at least very many of them uh and it's not only the ones that are interested in the technology that can see the potential H just as long as they try it out when they try it out people really really get the understanding of how can it help me and it helps them to to kind of redesign the workday H and that has been actually one of the coolest experiences in my career is to see how these kind of tools when they come into hand of to the hands of uh my talented and qualified colleagues how their imagination starts to uh create use cases for how they can use it and I think that's uh that's really cool in in in general with the generative AI M I agree MH do we grab a few more slides yes yeah because I think I mean now we've seen the results on the on the um on the overall kind of criteria that we were looking at but I think one interesting thing is and and this talks into the maturity is that based on our respondents I think the message is that very many believe that it holds potential uh but that it still doesn't meet the users full expectations and I referred earlier to these kind of uh really great marketing videos where you felt like as long as you had a co-pilot in your hands you could do anything and I think that the uh results on this slide show that that is not totally the case no we do have a quite a a broad range of of uh satisfaction scores in our test cohort um and I think that kind of HS the story that people were going in with expectations of this can solve anything for me this is Magic and this it isn't an autopilot but it's still a co-pilot that can help me uh with everything almost yeah and I think that was not really the case it was also launched uh early and I think that might also be be a result of that uh but still people do believe that it does hold potential and I think that's uh um that's interesting to see and we do also see that uh among those that are satisfied we do get uh better results and this is Maybe a bit of a hen and egg situation but if we we see that some of our testers really have figured it more out or have use cases where it really does work uh so uh and they also these users they report that the user more often they uh they um save more time I mean in Aver on average they save 52 minutes uh compared to the 34 minutes I just said before on average in the full cohort so we do um see that we have some users that are figuring it out you would assume that all of these users were super users that we had been out teaching and and telling them how to use it but that's actually not the case so we do have an over represent a slight over representation from the super users but we can see also that it's it's probably more about that cellet should support the use cases that are relevant for you and therefore it's also about kind of for us it's it's important now in the next steps to figure out or we have hypothesis and and learnings about who those people are and really try to match the tool with and what it can right now with the roles and the users that that have those needs yeah so it's really about kind of figuring that out as a first uh step in our uh yeah in our journey onwards really cool okay so users that are actually not really that well taken care of in terms of adoption or or part of that uh first uh super user track we saw with all these activities so so they actually a bit neglected still get I'm that's a strong word but but they still get value out of uh some of them do and some of them don't I mean we also do have users that have tested it and and given up I would say um but I think the the learning from this is also that uh that we need when for us going forward we need to to Really design training materials that are very specific that are focusing on the use cases that matters uh so that we can teach a few things that matter to the people that we have that get have have the licenses rather than opening up the opportunity of everything that you can do but really try to find Focus yeah um and at least that's that's going to be our our main main focus for the next uh couple of months while we await kind of how the tool matures develops becomes available and stable in Danish also an important factor for us but until then it's about the applications that work and the use cases where where you really can succeed with co-pilot yeah that's uh awesome to hear actually uh going more Ro focused of a function business line kind of focus on it so you give the tools to the people that need it the most and have the relevant use cases yeah that's just a textbook so really cool really cool to hear so we're uh almost at the end of uh of it all uh I hope that everybody out there gained a lot of uh cool information just quickly showing that there are more of these uh broadcasts to be joined if you felt that this provided you with some value or some cool insights from a real life customer in the studio uh then you can just uh reach out and uh and let us know if you if you also want our help and support to move on in your uh journey within uh AI co-pilot could be anything but I want to thank you very much uh ingrit for making your time taking your time to join us today in the studio and uh wish everybody out there a really good day so should we say goodbye goodbye by much bye e e