hey everyone and welcome to fabric freny for April it's been a few months since the last time we did this show yeah there's been so many new news conferences things like that it's uh almost impossible to find the right balance to what to show you today and what not to yeah and and and uh as you said it has been a while since you did this last and it has been quite a while since I've been on this stage with you yeah on this in this uh in this studio I think it was so almost a year ago yeah at the launch of fabric at the launch of fabric yeah exactly so speaking of which yes in a little less than a month we have build at Microsoft which will Microsoft build yes which was event where fabric was announced last year so there is a history of cool things being announced we expect the same this year and we are going to do a similar day with the with a session on recapping all all the great news from build especially about build um a month from now yes the week after build the week after build on Wednesday we'll be live during the midday so uh reserve your calendars it's going to be a blast it will great but we're here to talk about the news from April and as usual we're going to have a spotlight feature mhm you know what it is today people who checked my LinkedIn also know what it is today so uh spoilers there we'll go through the the overall news mhm you have I actually attended Las Vegas conference AG to come here and share your top three features favorite features or and and and you made me select only three I couldn't so but I still I still did it I I think that's great yeah yeah you could probably have picked 20 yeah and then we are going to do a quick feature demo coincidentally it's going to be the same one that we're doing at Spotlight M and finally a little tip of the month yes to take home and think about so we glad to have you all here and you're in for an interesting half hour here yes so our Spotlight feature for this month is of course mirroring in fabric because it was announced almost a half year ago but now finally we have it in public preview so anyone who's out there tuning in watching this can go and sign up for this preview themselves and you don't have actually don't even have to sign up you just need to go to your fabric um tenant and check in administration settings and you're able to test it did you look into it yet I did look into it and and I am I am still on my heels about how easy and fast and just approachable this new feature is in fabric I don't need to do anything techn not in in in writing code or technology I can just do this by by with with a mouse and then I'm good to go exactly and later today we will actually go through and show a live demo of how to set up a mirror and Fab break we'll talk about some of the prerequisits but once those are done we'll show how absolutely easy it is to set up how easy it is to keep it running and how quick it will run interesting to see if it will be fast enough for our preference yeah and also actually how how fast we will get changes and how fast the mirror actually works exactly so yeah great a cliffhanger there Cliffhanger there so we are talking about Fabric and we've been talking about that every month for the past year uh for anyone who's listening in who don't know yet about fabric fabric is an endtoend platform serving all the um needs of an analytics data platform so you have all the tools that you need to retrieve ingest data you have all the tools and more you need to transform that data prepare that data make it ready for what you need to use it for be that machine learning or data analytics or reporting and then you of course also have have the tools to actually use that data powerbi for reporting data activator for alerts and more and finally one leg in the middle of the whole thing stitching it all to together in a nice coherent um unified platform the one drive for your they one for your data so the news today we brought here a few slides of the news and it is it is cily a lot I mean there's no way we're going to be able to go through all of these so as usual we've have to pick and and this is only for April this is for April and it is so it is so overwhelming a little bit of March but but yeah but still overwhelming yeah even though you're trying to catch up on what is happening inside fabric then this or last month's releases were just out out of this world in in both amounts and features and usability so I'm sure if you wanted to check up on every new feature try it out work with it it would be a full-time job or more just to keep up with those features um so we picked some top news and uh I picked some and you picked three for later but but the these are some of our common favorites so there is a new visual in powerbi yes that's always interesting um there are several notebook improvements and a new uh percentage run command for notebooks which is uh doesn't seem like a lot but it really is a a neat feature that gives a lot of flexibility and usability we have an Expression Builder for pipelines I love it yeah yeah it's uh back to the old ssis St going be much more intutive to to build something and honestly for me the expression experience in pipelines has always been the little awkward thing that tried to be a no code thing but then you had suddenly a programming language yeah that you need to learn that need to learn but there's no there were no intelligence and there were no clear documentation now there is an actual UI to help you make those expressions and they suddenly feel a lot more intuitive than they did before this then we also have an event house which I know that you are excited about I'm I'm somewhat excited about the vent house because now we have the possibility to actually logically group our kql databases our realtime analytic services in an event house uh look at it as a server for your databases and and then with a Sprinkle of of functionality and features but we will go back to that later on exactly we have some Auto ml features which is very neat if you want to get started with machine learning AI not the not the generative kind of AI not the the let's talk to a chatbot kind of AI but the the actual number crunching deep math statistics the algorith the algorithm part of AI if you want to predict your uh turn of your customers or something like that well then there is a need tool to get started and get some quick results and get some um some Inspirations of what models could perform perform the best yeah and also a a a fastpac paced uh Journey on on I have some some data set I would like to what what kind of parameters do I need to to use in my machine model and then I have that then I I I'll get help from autl exactly and finally more support for git cicd and deployment pipelines uh for various artifacts and fabric very always important I think we've had this on the news for the last four times because there's we all we get always more but it's also been so much um a hurdle waiting for some of these yeah and in my experience we have been talking about is fabric ready yet for Enterprise use cases uh we if if we're not there yet I would say we're getting awfully close with some of these aw close some of these new features and if if if if the streak for releases uh in Fabric and will be the same going forward then we will be there pretty fast oh yes oh yes so let's start with that with the git with the cic and the deployment so we have now we we have support for notebooks we have support for pipelines and we probably have support for other artifacts that I may have missed spark job definitions yeah things like that we also have the capability now to go and add automatic rules during the deployment of these so for example if we have a bunch of notebooks in our Dev environment we want to push that to our test or production environment of course we don't want it to Still Point back to our development Lakehouse we want to of course switch that to point to our test Lakehouse or production Lakehouse we can use this um deployment rule to now switch what default Lakehouse or notebook book is using yes as sort of a sort of a parameterization of connection strings and workspace names and so forth and I'm glad you say that because that is something that I would have hoped we also had the ability to make our own parameters and change them them here in the deployment rule that is not a thing yet we only we only able to switch the default Lakehouse yes and in a lot of notebooks if you use it in a data platform you need to take data from one place and you need to move it to another so only being able to switch that one default lake house um it causes some trouble but it's definitely solvable because what I learned recently when you switch the default lake house you actually also switch the entire catalog of lake houses that you're connected to so it actually works after all um you just need to know what you're doing otherwise you will have notebooks connecting back and forth and the small comment there is actually I attended the Las Vegas conference a few we a few weeks ago where we saw a sneak peek of what will what will be the future of ccdg devops uh things in Fabric and your infot treat um it will be so easy for the business user to deploy to new environments and it will be not as easy but still easier than it is today for technicians and diff people to to work with cicd with both G and devops and so forth and they don't need to leave fabric at all because that's what we really want to get at because today or at least at least before we had this weird Clash where yes we could do deployment pipelines with a UI that helps the the the business users or the Light technical users um still do some deployment but it didn't necessarily speak too well with the with the with the big complex asure devop setup and if we can make the business users happy and not having Turles of of going through as devops and we can keep the um technical people yeah happy would be very nice and things will still be in devops but you don't have to leave the fabric portal toly that is that is what we want we saw it as a sneak peek a so-called sneak peek in Las Vegas so we don't know when that will be available uh but we just saw it and as we only saw it on a slide take I can't even show you screenshots or something because I don't have access of course then we also have the new visual new visual which is the 100% stack tet yeah I personally not not the one I was waiting waiting out the most for but I really do like that the visuals now one by one is getting this overhaul it gets all the new sets of formatting features and uh and if you follow any people on Twitter or LinkedIn or X um you'll see that the amount of high quality visuals and pretty reports that are being created these days are it's it's really it's really better than it used to be yeah and also Microsoft is is is from the product group now publicly announcing that they are focusing they have done this for quite a while now but that they are focusing on the core visuals in powerbi to actually reamp them rewrite them completely exactly to fit the future of of reporting I don't know what that means but yeah would be interesting yeah then we have automl aut ml automl is machine learning that kind of just does the thinking for you so I I I have an example here I can show you very quickly because time but I took a notebook here this mode notebook is loading in some test data it looks like Churn it is churn data okay got it do some data data cleaning prepare it make it ready feature engineering prepare the right features and then finally we have an example of approximately how the data looks here you know what I love the fact that you have changed all the parameters to zeros and ones yeah that is that is best pref for ML it is right now indeed machine learning um then we'll do all the the hurdles of turning this into a data frame prepare it for the right format for the for the for the machine learning and then finally we can see here now we get to the important notebooks because again here we importing ml flow we do some light testing but this part is the interesting part because here I just import this Flamel as autl mhm I can then use this to create an automl instance and I set up some settings down here that's all I need to do on my data and then I can the important setting here is uh for example this 60 UH 60 60 sorry that was actually correct 61 which Define how much time I want to set rice by it trying different things and optimizing and finding the best model and we're in a we're in a hurry here so we don't we don't have the small note to to the attendees we have a minor glitch in the technology today so if you see a blackout in the screen sharing then we're sorry for that we are trying hard in a live stream to handle that right now so sorry yeah can run this and we can now import that training data oh well that's just one one all of them anyway we don't need so we can run run all of them so while this is running I think for me the um the really interesting part about automl is that you don't need to be a machine learning expert to try to approximate a model that may be a little bit useful no it'll help you create that first draft it's almost like for powerbi we have the copilot that could help us create the first report it may not be a perfect report it may not be the perfect machine learning model that it ches but it gives us a starting point and it gives an idea of what could be the potentially great models to work with here for for for the machine learning use cases um so I really like like how we can with a pretty simple code set up a test run and actually run it yeah and and so the autom ml feature output a machine learning algorithm that I can then afterwards tune to fit my business needs yes great or you can pick up some metadata from what it found and and check what kind of model did it run what kind of parameters did it prioritize and then you can use that as your own insight into giving it a new go maybe from scratch or fine tuning or or or so so so actually workflow wise similar to having that Power report help you create a draft the autom ml helps you create a draft and I do want to call it a draft because um it's an AI so the automl is kind of an AI that makes AI so it's an AI on AI it's a machine learning that creates other machine learning models and it doesn't understand your data so you may if you don't know your data and you just blindly trust it it may infer things things that are impossible it may in first things that are already yeah and it may may look at reverse causality and suddenly it predicts your your salary based on how much taxes you're paying which in it's the other way around in reality so yeah so this is not a silver bullet a help and a support to get inputs and guidance on where your machine learning models could go and it's yeah and it's a tool that can make you uh able to do a proof of concept or an experiment in a in a much shorter time than before okay so the important part while we wait for the last results of this one because it we we did tell it it could run for a minute so that's what it's doing um the last message I want to say about alter met is that it is simple and easy but as you saw before we went through a ton of notebooks to prepare the data to feature engineer the data to initialize it to make it the right format that's efficient for for being trained here and I think it goes without saying that if you ever want to do some machine learning the first H hurdle you'll ever stumble upon is that to have any good machine learning model you need to have good data and it needs to be transformed in a way that fits machine learning the machine learning Paradigm yes so data preparation is the first step to any machine learning or AI for that matter and uh and shouldn't be seen as a shortcut to skip through that part and just throw data at at some algorithm that won't work it needs to be structured it needs to be need and it needs to be good data so AI is is yes it is the future because that will drive us to new possibilities but data fuels the AI exactly we need data exactly great got it and you can see here it it came back with the settings so the best hyper parameter config here it found was doing six isation with four leaves and 24 something something and ending up with a learning rate here so we get some uh two nonata scientists trying to demo we we get some metadata back that helps us explain how good did it perform and what are the important parameters okay hopefully we will see this in the demo later on on how how it actually works with a description on the details behind it because yeah yeah yeah yeah we have we have data scientists here running around and and perhaps we can tackle them to be here should be fun next one event house do you want to say something about event house yeah event house imagine that you have a a if if we take this back to the SQL Server days uh then in aure you have a an an as SQL which is as you just have a database um and you can't manage the server on top of it now with the ventouse we have a server on top of your kql databases look at it as a server um where you can put different uh kql databases underneath this event house or put it into the event house and then you will have a a a contained view of all your kql databases in that event house and with the event house you will also get a high level details on on the metrics in the event house how many data are you storing in each database how is the compute running do you have any outliers do you have anything that you need to take care of uh in the event house and there is a limit to the amount of kql databases in the event house but I've heard I haven't found a documentation yet but I heard in Las Vegas that we we are talking hundreds and thousands hundreds of thousands of databases inside the event house so I would love to see that business case where I need 100,000 one database inside and and I don't think that will happen this also gives us uh capabilities to build um Medallion architectures in realtime analytics where we can now stream data real time into a kql database and actually have that process to the gold layer within milliseconds using the vent house um so that will be extremely well placed uh uh um in the fabric Universe uh so we can now begin to handle uh iot devices messaging logs Telemetry whatever live data you have in your data estate and have that processed within milliseconds to a through a medallion architecture and out to powerbi or whatever application you have afterward so that that that's that's awesome so there's a reason it's called the house here is that it is really a lake house for your real time data in almost all senses yes yes very nice very nice and then also the the next news is the expression build for pipelines where we have this intuitive UI which now has Intelligence and it gives us recommendations of what kind of functions what kind of uh references can we make in our in our expression code um and we also have these neat little uh new objects inside the code which will actually point to the object instead of just being static text if if if if if I came from the Power Platform would I then recognize this Builder that was a leading question yeah okay why you explain me okay uh this Expression Builder actually looks a lot like the Expression Builder in in power apps and power pages and power whatever power we have right now yep in um so if you are fond of no code L code approach to build building Expressions then you get a lot of help here and if if if you already know your way around power apps and power pages and expressions there then you are already good to go here great great notebooks notebook upgrades I have been waiting for this and complaining about the the problems with notebooks paralyzation since at least half a year ago we so if we turn back time half year ago if we wanted to run any notebooks I could start working in a notebooks you could start Notting working in the notebooks but if we had a third colleague who wanted to also do this his notebook would fail because already with two notebooks running in parallel yeah we were at the capacity what was that capacity that capacity was 128 CS as you see on the slide here so even though I was on a trial I could not run three notebooks in par nope wow even though you're on f64 you could not run three notebooks in parallel that was change that has changed since so we had a Max course 128 but we had each of the notebooks reserving 64 cores because they had this default setup with being mediumsized and it could scale up to eight Co per um per note uhhuh so it took up 64 reserved capacity for your notebook to make sure that it always had enough for your notebook to run very problematic um it says total par notebooks 3 here but actually not right it was only two in parallel two and a half yeah so that that has changed out three features help change this the first feature is bursting so now we can actually use three times as many cors as our capacity is sized for in a short amount of time so in a in short term we can actually use a lot more than than that limit that's nice as a beginning then we also have job Crews so if you and I worked on a notebook our colleague also worked on one and he ran over that limit his job wouldn't fail anymore it now will queue up patiently when wait until it gets its time in the queue and it will then run like we needed to do so I will not no no longer see a a an arbitrary error measure to when I try to run the notbook that's no guarantee but uh not for this not for this reason any anyways and finally optimistic job admissions because now our notebooks got a little bit more optimistic little bit more smiley smiley maybe they can feel the the sun mhm coming coming forward here uh which means that now instead of reserving the max needed coise that it could anticipate it needs to to have it reserves the minimum amount great so notebooks works much better finally we have the percentage run which can actually let us create one notebook and then reference that notebook in other notebooks so I can have a library or a connection of need functions that I want to use and then I can use that for all my other notebooks in my my entire data platform or in that workspace at least it's very very cool very flexible very neat and fabric conference yeah we got we got a lot of news in fabric conference tell us your three favorite features and tell tell us only three only three and very short because we also need to show mirroring okay uh the first one is Task flows we saw it and at as as a sneak peek uh in Las Vegas it is also now public on uh on on the Microsoft blocks where we get an an a a logical uh abstraction layer on top of our fabric uh uh with workloads where we can uh see the architectural approach to all the services and how they are connected great the next thing was folders uh and folders and folders best news ever now we can put our things in folders and we can actually create our items directly in the folders you have you need a a small uh hack to do that but perhaps we can get back to that in a later L's structure things much better yeah and one of the big things here is that we also get data flows gen two with Delta loads so we can now with a very very few clicks with a mouse uh enable Delta load on our data flow 102 we actually only have to mark our date time column and then Microsoft fabric will do the rest for us that is amazing so for someone out there who maybe didn't hear yet what exactly is a Delta load yeah but but the the changes since last load based on it on on on on on a daytime column so similar to an incremental load where we get only the changes yes and then we get that dumped there's a big question here do we get the leads I don't know yet don't know but we'll see mirroring mirroring the fastest demo ever we have two and a half minutes left let's get to it so mirroring is a way to connect snowflake Azure SQL Cosmos DB directly inside fabric so you just have access to that live copy of whatever tables you have in your other databases so that is nice and what it really helps help with is you have fabric but now you and you may use the shortcuts to have something like a data breaks data platform directly inside fabric now you can also have snowflake or Asia SQL inside fabric so you can help make fabric your unified platform despite having investments in other platforms you can also add add Amazon as well there um and then it really becomes that core of all your data yeah the borderless data platform borderless data platform uh um so let's talk about what we need to do for mirroring we need to be on the same tenant for this to work yes we need to have manage identities we need to have public networking and available we need to access this access from Asher Services enables and you need a certain size of the aser SQL it needs to be 100 dtus or more got it yeah then we can configure Fabric and uh and change this Administration setting and then we can create a mirror database so with one and a half minutes let's see if we can create a mirror in one and a half minute should be interesting I'll go to my workspace I'll create a new object here I'll click mirrored as your SQL database hang tight we're clicking FAS here AB is the name because I don't have time to type more one moment it opens and asks us for a connection to this database if we already loaded data and powerbi from this database we can just click that already existing connection got it it already says mirror all data yes I don't need to do anything else and click and that's it yes pleas oh so you're already done we're done now we're just waiting for fabric to we just waiting to fabric to actually execute on all the things prepare this make it ready for us and in a few minutes maybe up to 5 or 10 depending on data size it will just be ready we don't have time for wait to wait for those few minutes so let's just skip this and go to the already pre-created mirror I have here this database so I have a query here which will check one of these tables of all these mirrored database tables here you see it run it here on the mirror and it give us gives us some data so this is a live replication of our data we can go to our database now try to make a change and let's see how oh shoot that's fine live demos demos we just change this value to 123 yeah and we can also go at the same time and try to delete customer number 11 to see if it catches that one as well and then we just need to click run here and see how fast it's actually going to catch this change um which is a little nerve-wracking when we are so close to finishing but I pretested this I'm comfortable that that this will be ready it will happen eventually yeah yeah yeah yeah we'll uh leave people in suspense here here so we are looking at the we looking at the one value that should change it says one yep and it will the demo ghost might show up today but it will eventually change to what was the value one two three one one two three 123 okay it worked like I saw this an hour ago um I think we got the demo ghost do you think so I think so okay so let's let's age here we give it 10 seconds and if not we say this is uh the fault of the demog guards yeah and we uh perhaps we can round it up and then just as as as a last screenshot we can see if if if if this works yeah so thank you for joining today it's been an amazing month of so many fantastic news we have had a hard time condensing this to just half an hour and we probably should do a an extra session which we luckily will do next month so there should be plenty to uh to to catch up on there um it's been a pleasure and if we look back into the the demo here we can see that it actually just finished so this was within one and you have you have deleted row number 11 and you have updated the value to 1 through three updated the row is deleted our data is live we are talking minutes at the maximum seconds in some of the attempts that I've been been trying so it's really really cool feature half an hour extremely condensed version of what has happened in April thank you for attending and hope to see you soon again see you again bye e for