hi and welcome to fabric freny your monthly source of news spotlights deep Dives and more around Microsoft fabric my name is Matias hellare and I will now guide you through the the news of um of what we have from fabric for November and today we also of course have a special Spotlight spot which is the co-pilot features so today we'll look at some of of the ways that co-pilot can help our work with the within fabric uh within data engineering data science powerbi and uh and data Factory so for anyone new tuning in who are not completely certain what exactly fabric is or where this is completely new a very very very very quick recap here is that fabric is the new endtoend platform that would help us with everything around our data analytical workloads so we can fetch our data we can prepare that data inside the fabric uh platform we can finally use that data make it worth something and of course along the way store the data manage it in the in the unified uh one L so it's a one platform to manage all our data loads all our data task and and everything we want to do around our data to do proper analytics AI machine learning and so on November has been quite an eventful month in the topic of fabric we have had some major releases there are some very big news and there's probably much more than we can cover in um in this session because the list is very long but there are some key features I would like to point out co-pilot was released in public preview in fabric as of as of November so in um a range of the the experiences we can now use cooby pilot features to accelerate to improve and to help us with our work that we do with our data inside Fabric and then probably the biggest news of all is that fabric went into General availability meaning that we can now go live in fabric with production scenarios it's now a supported feature by Microsoft we don't have to fear that it's going away over the weekend or something like that it's a it's an established offering an established service and uh it's much more immature than than what we saw at the release of the public preview we also are looking into some enhanced visualization um options in powerbi which will really help us make more sleeker neat looking designs that can hopefully help by um making reports easier to understand and better to use then we also have data Wrangler I mean data Wrangler is not new in fabric but its usability just got a lot better because data Wrangler now supports data uh sorry Spike data frames instead of only Panda's data frame which means that the main kind of uh data processing data transformation that we may might be doing in uh in pisar in a notebook we can now use data Wrangler to help uh build build our code with the with visual aid now as well and finally we also saw the release or at least the hint of a new fabric certification uh a certification that is going live live in the beginning of next year uh with I believe initial previews and January or the full release something some We There is a release date in in January so we're looking to certify ourselves and and demonstrate that we know ARR around the different workloads in fabric this is a a neat way to do so so let's Jump Right In the first step is the new visualization enhancements for powerbi because if we were to build a report today we would have to rely on the classic um visualization options of course we could do third party uh integration third party visualizations but with a standard with the standard set of visualizations we're kind of bound to the the the the core visuals now now the the product team at Microsoft has been working very hard uh on enhancing the capabilities of these core powerbi visuals and two of the first visuals that they've uh or two of the visuals that they gave an overhaul this uh around this time has been the tile slicer and the multi row card so the tile slicer whenever we want to make buttons that we can click and uh and use for a selecting slices or filters of our data so what's new here well it used to be a a symol button with the not that many formatting options but with the new slices we can do much more we can add uh images we can add subheaders we can add formatting options based on some sort of criteria and we can make really Sleek looking selection buttons like the example that you can see here on the page um so that's definitely going to be improving the uh the look and feel uh and usage of our reports secondly there is the multirow card which is really nice visualization if we want to show multiple values of our data in sort of a collected overview but it just hasn't been able to compete with some of the third third party visualizations and third party tools and uh at times have felt um lacking but with the new update we again have a whole new range of capabilities first of all is looking so much better we have options for formatting the background differently between the main value and the sub values as you see here um we can add icons to the individual uh measures the individual uh metrics we can have a a header proper distancing between the numbers and we can have these subn numbers with something like a percentage increase like you see here conditionally formatted with need colors so again uh with this new Option uh for our reference labels in the multi card we can yeah we can can uh we can make a much better looking much easier to understand and use overview of our key matrics in our reports so all in all very nice enhancements to uh to what we can do with our visualizations today that we couldn't do um last month so that was the visualization enhancements and I should emphasize that this is just the beginning the the PBI core visuals team is working really hard on on revamping all the visualization that has been lacking or could use the new functionality generalities and they're quite open on on social media Twitter LinkedIn and so forth with sharing the progress along the way so if you're interested in in visualizations I can only recommend to follow the hasht PBI core visuals just a personal recommendation so next news is the data Wrangler for spark so data Wrangler was here already since the launch of the public preview of uh of fabric data Wrangler is the tool that allowed us to go in and configure a pandas data frame do some modifications using the UI and then based on those mod modifications some python code would be generated that could then manipulate this Panda data frame so it's pretty neat for someone who's a beginner in in uh in writing python or even for an established uh experienced programmer who just want to avoid writing out a few lines of code and just click some buttons instead it tracks all the steps of our processing and it it reverts into code so the new thing is that it now works with pypar it works with the uh notebooks that we can use in the data engineering and data science it works with pypar and it can generate not python code for pandas but now now also ppar code for our pbar data frames meaning that we can now use this data Wrangler tool for our Notebook based data Transformations workflows because honestly pandas was more oriented towards machine learning Ai and those kind of use cases and for actual data Transformations we would often just rely on uh on ppar and now we can also use this tool to generate our P [ __ ] so we now have the option between data Wrangler to help us write the code ourself and of course uh also being assisted by co-pilot which we we will look into just in just a bit aside from the visualizations the data Wrangler um we of course also before I goes to the next next one we of course also had so many more new that uh I would love to bring here but the list because of the release of fabric is just very extensive uh so these are some of the the core news but one of them is of course also the new fabric certification because we can now be a certified fabric analytics engineer associate why this need to be four long words I don't know but this is this will be the title um so I guess if you are sitting out there and wanting to call yourself a fabric analytics engineer associate you can get this certifications but joger side I think I think it's a nice step for Microsoft to allow us to demonstrate our skills uh within Fabric and um and they actually cover many of the the needed skill sets pretty well in the um certification so the certification page is already there you can go check it out today if you'd like it has whatever skills we need to learn and it has the overall list as you can see here on the slide skills at a glance um you have we have to be able to plan Implement uh and manage a solution for data analytics we have to be able to prepare surf data Implement and manage a semantic model and explore and analyze data basically we need to be able to do endtoend analytics within the fabric platform but if you go into the page and you're interested in this certification you can dig into each of those topics and find out what exact uh pieces of knowledge topics that you need to familiarize yourselves with to prepare yourself for the certification but also to prepare yourself for using fabric as a whole and Luckily everything around this certification is um covered by the Microsoft learning um freely available mini courses as well so you could of course sign up for a a paid course somewhere but you could also get started today with the tutorials and training materials available um directly in the same Learning Center where you would find the certification so that's really nice um and uh yeah it's a it's a beta beta uh certification and it's it will be available from January 2024 yep and for those who are curious about it the new exam will be called imp implementing analytic Solutions using Microsoft Fabric and yeah this certification itself is fabric analytics engineer associate so hopefully someone out there is an aspiring new certification holder of this certification next news is the spotlight of today co-pilot because co-pilot is everywhere to be seen in this month news so if I go to the page of the fabric news and just search for co-pilot it immediately gave me 44 44 places that copile is mentioned throughout the news um and looking through the major areas that that are now utilized in copor Pilot i' I located at least four of them so we have co-pilot features inside the data Factory experience we have co-pilot features inside inside the data engineering experience we have co-pilot features in the data science experience and finally we have co-pilot features in the powerbi experience and probably before we end this session I would have said co-pilot more than there those 44 times that's uh that's on the news page but that's fine so one thing is that now we have some co-pilot everyone has some co-pilot now there's co-pilot everywhere it was one of the main themes if not the main theme of of Microsoft ignite uh which was two years sorry two weeks ago uh from from today so what exactly do we have uh capability Wise from co-pilot in Fabric and what can we use it for let's explore a little bit so first of all some basic information here co-pilot in Microsoft fabric is not supported on trial s skus meaning if you signed up for a PR trial and you want to try out co-pilot that is simply not possible sorry to disappoint but that is the case you need to have a paid Microsoft fabric subscription and furthermore it needs to be at least the size of an x60 f64 or higher or a P1 and higher so either a powerbi premium capacity or a fabric capacity of an uh capacity unit size larger than 64 so that's a requirement and it's not supported uh if you don't have that furthermore even if you have that Microsoft is enabling co-pilot in stages so not everyone will have access even though they have this capacity is being rolled out continuously and the only thing we can find uh on the documentation pages is that Microsoft is claiming that everyone will have access by March 2024 so if you're sitting out there and you have the right license you have the right capacity and you really want to uh test out these features or try them out or even use them um there's only two things two ways for you to go about this either wait it out and hope that you'll be rolled out soon or reach out to someone some some some contact from Microsoft your Microsoft representative and ask them if there are possibilities of being fast-tracked for this roll out or something because there's no button to activate it there's no setting in your admin center it's uh it's rolled out continuously so once it's rolled out of course you need to go and enable it as well in the admin Center but it's uh you you could be uh you could be stopped by simply not having the feature in your specific tenant yet something to be very aware about so the first feature from co-pilot that I would like to point out today is the narrative visuals so narrative visuals is not a completely new thing um we already had the opportunities to make some visuals uh be like more like written text where you could do a summary of the page or you could U make a comment on some specific data so the new thing is that the new narrative visuals is uh is built on co-pilot it uses the open AI engine behind the scenes to really superpower what we can uh what we can do with a natural language questions and um and of course the text and the answers has been got has been getting much much better so what we can do is we can set up standard questions standard prompt standard summaries we want to make and once we've set this up in our visualization well then every time we um we refresh our data or page this this summary is going to be updated as well meaning meaning that we could set this up once and then still every time someone is using the report it will have a fresh new summary or conclusion or just uh comments on the data and all of this is chosen by how we write and design design those prompts inside the the the narrative visual so we can as you can see on the page here we can also make some specific settings here where we can choose which data to include and which not and we can make sure everything is selected or none is selected and only a few measures or something like that so we can specify what kind of data it should use for its contextual awareness before writing this this a really need addition um and then you see here in the the demo here we can we have an example here go here to the visual write our description it could be like here summarize five key insights and then then it's there so it's a it's a simple point and click create the visual write your prompt and then the information information uh comes from the from the AI language engine then we also have of course co-pilot for powerbi report authoring so today we can go into fabric we can Target some data set or some data and we can have co-pilot create a whole starting point for a report for us so we write a prompt we describe what we want to uh report what we want to visualize on this page and it it it creates a page for us uh with the kind of getting started point of course it's not going to be perfect of course it's not going to hit whatever our stakehold holders need or ourself what we need at the first goal but it will give us a neat starting point and we can go and change uh change some visualizations and and yeah use this as our starting draft so again a very nice way to get out of this um so-called fear of the blank paper or or how we want to describe it so in this case we uh can do things like suggest content for the report yeah and it will basically WR make the report for us then we also have co-pilot for data Factory um but only for the data flows artifact in data Factory so if you looking for help with your data factory data pipelines in Microsoft fabric then I'm sorry to disappoint you there's no co-pilot features for that yet but if you're using data flows generation 2 inside data fabric to do your data ingestion or your data Transformations you can now use copilot to help help you create the right steps uh to transform to merge to join to uh yeah to um to manipulate your data and that can now be be done with um simple language queries like the example here where we asking it to only keep European countries and it it recognized itself what kind of column we could mean when we're talking about only key European company so it it automatically detects what column should I do a filter on it recognize what value should I filter on and then it implements that into our um into our steps and since it's data flow we can always just remove that step again and start over try over and then then then redo it so it's a it's pretty nice how it's contextualizing it understands the data that we're working with so whenever we ask it something it will then use whatever our columns are called it will infer uh which it should do the Transformations on and it will then make its best guess because guessing is what it what it does so it's uh again it's not perfect I don't think we any of us expects that we can rely on just saying get me data from this database and just clean everything up and transform it the way that it becomes a need sty schema and then give me the output of that data and everything should be good and expect that it then gives us the perfect result that we need um that's not what it's for at least it's not in that stage at all yet but it can help us do that next simple step of transformation and um maybe just teach us a thing or two about how we could have done it otherwise than we would have done it ourself then we finally also have co-pilot forign notebooks co-pilot for notebooks is really interesting um because in notebooks we are writing actual code and co-pilot as well as chat gbt open AI all these large language models are surprisingly good at writing programming code so being able to just ask questions to a code have it summarize what does this workbook do what does this specific command do can you please change the this code into something else or just from a start write please ingest data from this data sort and do XY C with to the data and give me that code back this it can U help us uh work with one of the examples here is load data from my lak housee into a data frame well just use we can just use language and co-pilot will then help us build it um finally we also have the chat Magics the so-called chat Magics so Magics is a way to change the programming language in our notebooks for each of the individual cells and with the chat Magics we can now ask questions or prompts to our co-pilot directly inside the notebooks and we can have an output and that output could either be some raw text that can help us move forward or it could be some actual code that it that we could then Implement so uh again very neat addition and now it's time for the probably biggest announce announcement of the month it's the release of this guy the release of Microsoft fabric because fabric is now General availability meaning we can actually start taking the platform very serious and look at if it should be U um if it's something that should be housing our next data platform so what exactly are some of the considerations we need to make if we need to evaluate if fabric is something for us first of all we need to acknowledge that fabric is a simplified architecture it takes the complexity of what we could build in Cloud Solutions and then it makes a lot of choices for us so it makes sure that this security is somewhat ex acceptable it make sure there are a proper architecture that the tools can communicate with each other that data can flow between the different AR artifacts and the different work spaces and so on and all of this is a managed solution so all of this is just handled we can then just focus on developing kickass Solutions we can use the proper tools to do our data Transformations and we can uh rely on Microsoft making the choices about the overarching underlying architecture uh for us of course for someone who has very high production grade or Enterprise uh specific needs um those may not be part of the um chosen settings chosen from configurations in fabric so in fabric some of these uh dials that we can switch on and and buttons we can click those choices have been taken for us and um if we want to do something very specific or heavily optimize a specific workload well then then then maybe something like Asha synapse is more ideal for our use case but if we have a more General need and we just need a PL platform to get starting making some value adding cool uh Data Solutions well then fabric is definitely to be considered it's also a great platform if we want to increase the collaboration around our data so fabric really unifies this collaboration while by by standardizing uh how we store our data and what format we store it in so with one leg in fabric we are standardizing on storing our data in a data Lake Manner meaning even even if we do a SQL Warehouse or we do a data lake or whatever the main storage where data is actually lying is on a data Lake and then we still get the vast different experiences but this means that people can easily talk to each other because when the data analyst needs some output from from a data scientist they can easily just point them towards the data inside one leg say well the data is there can you please make a machine learning model data scientist can pick it up crunch the data generate some insight and save that data directly back into the same platform so the datal analyst can get started using this for reporting uh or something else immediately so no more sending csvs back and forth or emailed no more productionizing things by having some shared folders in teams or or something like that a proper way to unify and store our data in know could almost call it a data data Hub finally it's also a platform that allows us to do vast different array of different architectures so it's it's flexible we can do realtime workloads we can do machine learning we can do some chat with your data kind of workloads combining co-pilot and and and fabric and powerbi we could do classical data Warehouse using SQL we could do a Lakehouse or we could do pisar Transformations and we can design the Solutions in the the best way that that we see fit so let's finish here by talking a little bit about the prices because we also cut got some new insights to the prices of fabric after the the the latest release so fabric comes with a bunch of different tiers with f64 being a a significant one because any capacity larger than f64 comes with the traditional free powerbi licenses for developers that we also have from powerb premium meaning that we could save some licensing cost by going up to this capacity S size but if we need smaller than that we can rely on the smaller capacity so we already had some prices here as payu go prices so the new thing is that if we sign up for reservations uh we can now save up to 41% on our fabric capacity by reserving for a given period in time so that's a pretty pretty pretty remarkable saving uh right there it should really um really be considered finally there's also some expenses for storage it's not very much and for most organizations this will be a nonissue the the main main price will be the compute and Microsoft is also warning us that there may come some networking billing at some point in this future but for now uh it is uh not applicable so that was it for today the release of fabric General availability it's gone live it's can be used for production scenario it's a supported feature for Microsoft and that was the big news for November in the world of fabrics so enjoy the rest of November and have a nice Christmas when you get there and till next time see you all