hey everyone and welcome to fabric fency June edition which will also be our last Edition before the summer holiday so uh we look very much forward to showing you the news from um May and a little bit from June this time yeah because we are in June now but uh in June now but uh but we also haven't received received the June news from from Microsoft yet so we are talking about what was released in in May and the few features that we have seen as blog post in the last few few weeks yes so we are going to uh to show the monthly Spotlight on a feature which this time will be drum roll semantic link semantic link yes we also show the news show um talk about some of the fabric conferences that are coming up here in the next uh next half year of course also a little bit of a feature demo I heard uh someone tell me that they had brought a a real time reporting yes demo to today so I look very much forward to seeing that and the tip of the month which will be be the the final icing on the cake can you give me a cliffhanger on tip of the month tip of the month is just a cliffhanger without telling it just um different not commonly known way to uh dig into your fabric consumption so it's not the typical fabric consumption app okay a canger there so hanger there in 27 minutes or so you will get that one exactly so today's Spotlight is semantic link which we will go more into depth with in just a little bit as well mhm so for anyone tuning in who still haven't gotten a glimpse of exactly what fabric is and I think at this time it may not be many of you but fabric is of course this end to-end solution that let us do anything around our full data State everything from getting data to Preparing that data and finally use that data all while this data is being stored in a common shared data format and a common data storage in one Lake and uh without without tossing you under the bus I know I know this is there an icon too much on that slide could be one too many I don't know let's uh let's see what the news tell us got it um so the first news is of course Microsoft build news so we just had a Microsoft build coming up here um a few weeks ago three weeks four weeks ago where there was a lot of news that was announced and um does that mean that we have a bunch of all these big Grand news for you today actually not but that the reason is that we already did a news from Bill video uh video stream here two weeks ago so if you are interested in some of the more Grand features especially around real time analytics and things like that do check out this uh this uh link from uh from that that that last stream which show uh we will paste into the the chat in just a moment yeah and if you don't get the chat it's aka. FM news from build news from build thank you yeah news from build in one word without spaces or so forth yes and we will also talk about real time dashboard that you're showcasing us we have some new pipeline connectors uh event house one leg availability what Spike autotune I love this feature especially the the word of it and and finally some new security features sounds like a modern pop singer who can't sing yes yes got it so this is the tool if you can cannot do your data engineering work now you have the autotune to do it for you if you can't sing fabric you can't sing sing with data you can have your data sing for you I know we're getting too far so security features first and foremost so we um have seen three new main features in in security that are private links for fabric tenant trusted workspace access and manage private endpoint and the private private link secures inbound access to fabric from selected virtual networks it will will allow us to block access from the internet enhancing our security and compliant by preventing unauthorized or malicious traffic from RE reaching our fabric tenant which is really nice that that that that that sounded that sounded an extremely long sentence so more or less it is is more security and I can now block internet access more less basically okay got it and TOS the workspace access we can now allow seamlessly to secure access to fire firewall enabled Asia storage account so if you're asure storage account is be behind the firewall we can access that now using trusted workspace access and finally endpoints well we can now use private endpoints in fabric so really cool if these were bloggers for you looking into fabric well know now that these are possible we can do this m it doesn't mean that this is for everyone that everyone needs these features to to succeed with their fabric but they can be a really good idea and um and you could have utilized some of these feature and need them if you want to go to fabric so less of a blogger now and if you want to learn more Microsoft also released the Microsoft fabric End to End security white paper where all of this and more is detailed there are um many many pages of documentation yeah more than two okay got it more more than two more than 10 got it got it okay so um a question before we move on all of these new security fees do they work on F2 or F4 that's a good question nope they don't so the thing is that they only work first of all they only work on on fabric capacity so fqq capacity mhm and they only work for f64 and and above okay great so so the old P1 and you can no longer buy a P1 from January so you won't be able to use them in the own op1 and um the technical reason here is that P1 is technically not a cloud resource so therefore it cannot utilize the cloud resource um ecosystem got it but the fsq is got it yeah and then news from build we had a Fantan fantastic bunch of new uh new releases from build and there's a bunch of interesting things here AI skills among others but all of this and more you can see if you go to this uh this link posted in the chat aka. FM newws from build not and not Ms but FM FM as AKA FM fellow mind as in fellow got it yes next feature next I've look look forward to this one real time dashboards real time dashboards real time dashboards that so we can finally do real time reporting yeah um I think we we need to clarify a few things because there is there is somewhat a a discussion or a dialogue in in the community uh as we speak around why did Microsoft release this real-time dashboarding because we have powerbi yeah how does that work how do I now choose what visualization tools should I use well the short answer is Real Time dashboards is is put into this world for um for analytics on millions and millions or even billions of rows of time Ser data yeah and it and it sits right next to the kql database underneath the realtime intelligence service inside fabric so if you want to do analytics on millions and millions of rows in your dashboards or in your um whatever analytics workspace you are in then real time Sports is the way to go if you want analytics on aggregated some PS look alike way to go then power is is is is the way to go but I would also say and this is for my own account uh that I believe that Microsoft is on a path to perhaps join the two I don't know but it it it could be a possible scenario without further guessing on the future uh I've actually bought a demo nice yeah so um the first one here is uh on on my screen there you go uh I've put on up a dashboard and actually for the people uh attending today I built this dashboard in perhaps 10 minutes it is not the most beautiful dashboard in the world but it works and it is a matter of of actually writing some kql and and then putting those kql statements into a dashboard it it is quite easy so if if I could actually um do a big a a a quick run through of what is happening on on this dashboard and then we can build a new tile together okay great yes absolutely first of all in the top here I can select my time range this is a built-in feature in realtime intelligence uh where I can select my own time range if I only want to see the last 15 minutes then my charts will change live uh uh accordingly uh but let me go back to the last 30 minutes and if the attendees on this live session sees that the charts are changing over time that is actually correct because we are looking at a real-time intelligence dashboard so when data is updated in in the in the database the dashboards will also update all its visuals in real time in perhaps not in real real time because right now we have a a minimum uh refresh rate of of 30 seconds okay uh so I have set this dashboard to refresh every 30 seconds and I know that the data that we get from the source is is is a new record every 1 minute mhm and just to demystify this data this data is is coming from a a JavaScript framework called fer so it is just faking data in an arbitrary way uh so you might not see continuous lines that's why we have trains in the middle of the ocean exactly yeah gotcha I've also built in as as when when you talked about trains I've also built in a filter on train Land Ocean and and air and we can see different devices uh going along if I filter on train um then we will see that uh thing and if I also filter on Ocean um and on air so I can now filter all my data um um in my dashboard as as as I would any other things and if if if I want to to see the underlying data set I can always click on this small icon here and then I'm I'm presed with the chart of course and the um the underlying data set that supports this this this chart and I can see the um the colums that I can that that that I can that I can use and also those columns give you a small a small dat profile of things on what is happening inside this data set cool what if I wanted to make a changes or see the the the code behind this theod behind um right now via I'm in I'm I'm in I'm in viewing mode if I go to editing mode I get a small pencil and if I click that I can actually see the the code the cust code behind this um um and if you don't know how to write this cost code uh when when we go back to the kql statements uh work Paine we can actually use co-pilot so we also got co-pilot to help us write Cal queries now nice yeah if I want to change things here I can just change it if I don't want a time chart I perhaps I want a bar chart I can just write bar chart here uh and then I can run it and you will see immediately that the uh that doesn't work for some reason should work it doesn't that's fine the demo coost is here uh but normally the visual would also update M but let me apply those changes and so let us add a new tile here I have cheated a bit and I have already done one here so uh this is this is so there you go um this is a a kql statement that is is based on these Faker events that I mentioned earlier so these are just arbitrary events happening every minute and uh when I developed this I was hard filtering the the time for the last three hours just just just for the uh just for the demo the demo purposes and I'm working with adjacent data set so I I have to pass this as strings and and and all that stuff and then I do a a summarization by device ID by time and transportation mode if I run this yeah then I'm presented with the area chart here yeah when I'm done and I would like to add this to my kql uh sorry to my real time dashboard I just click this pin to dashboard it's quite easy to an existing dashboard or I can create a new dashboard nice and I have an an existing dashboard I can select that dashboard right here I can only select one and the dashboard name is there and the ti name could be I don't know area chart demo and I'll open the dashboard after creation if I don't open this then it will just uh uh create it for me but the problem is that right now this chart is not filtering by the default filters up here so I need to go back in in into editing and hit the pencil and then in my time filter in the top here I can see what what parameters I have to work with M uh I can put in between uh and then it should be start time and over here in time now I've implemented a filter on time if I also want to impl implement Transportation type I uncommon this one and the transportation mode should be equal to it's called Transportation there you go okay so you're telling me that for each tile I need to have a custom query and I also need to specifically add the logic to handle filtering based on my parameters yes do you know what this reminds me of nope reports ooh pent reports we all we all love this Rel experience who whoes that so so now I have changed my tile to fit my needs on the filtering and I can actually just now move it to where I want it perhaps I want it right there and I want it to fit inside my my window there and I can go back and save it save and hit viewing so now I have my that my new tile here and when I fil filter on land it changes accordingly nice and ocean and so both um and and if the audience were quick enough you just saw a new line of data coming in yeah this second uh and I also tried to do some some some some rules on on how to color code stuff here so there is a lot and a lot and a lot of possibilities here I also try to do a map visualization but it is randomized Data so we have trains on the water but it it's it's it it might still yeah pretty nice and when you said it looked like P need reports it actually reminds me that I'm not fixed to a canvas here I can put tiles in wherever I want in whatever width and height I want it's just a matter of putting the them in and and then it can fit your need of course you will get scroll bars and you can zoom and zoom in and out but things will still work for you very neat so if this is this works for use cases it's a little bit more work than dragging and dropping in power is this my only option if I want to do real time data or other other Alternatives if I would like to do something more similar to powerbi uh you can actually directly from a KQ query set mhm when I'm done with my k query set um um this my this is the same query that we just built the new a by there's also a button called build power bear report great so when I click this uh uh fabric link this specific query to a powerbi report and there you go I can now begin to uh to build my my report based on based on on the events coming in here so it just works out of the box great so multiple options for Real Time data reporting we just have to choose what's the best for our use case sure yes nice very good yep so let's move on to the to the ne next point on the agenda that is of course the spark autotune so fabric can sing now fabric can sing great and the cool thing about spark autotune and don't get me wrong I really do like the feature because what it does is that you've written some sort of spark job request that fetch a DAT to do something with it and you can go and manually adjust the settings of your Spar cluster and try increasing the the noes or try to change some settings and there is a few settings that you can you can um leverage and change MH and depending on what you do you can actually optimize your query quite a bit as in this example from from running in 5 to 600 seconds to getting down to only a little bit more than 300 wow the thing is with autotune you don't even have to do this you can just let the engine itself try out different settings over time until it lands on the best possible setting and the graph I'm showing here is actually not an optimization that we've done this is an example of what autotune has done so so this is autot tune's work going from 550 seconds to 350 without anyone having to do this manually this is a really cool use case of actual good AI it is it is but but what you just told me is so it it doesn't change the code it only Chang the settings behind the code execution change the engine and optimize and and the the query will be the same the code will be the same the results will be absolutely identical it will just be faster we also have options together with this setting for doing more anal analysis into what this Optimizer found and what is going on behind the scenes of here so we have depths to understand what made it run so so well and finally which is the best part the only thing we have to do to enable this is to go to our spark properties write this thing spark. ms. auto. enable set it to true and that's it it will just work from there and help us make our query run faster so this is actually the this is mind-blowing to me because in SQL in the SQL world in the SQL Server we always wanted and we always joked with can we get an option go fast please exactly exactly this is your this option go fast in spark it is your option go fast in spark okay got it yes um absolutely and I think I think Microsoft mentioned that sometimes um most jobs running up to four times faster in in some examples so yeah so yes we can do queries much faster and then and the the other thing we would like to show here is the event house one L availability o do you want to tell what this is yes please thank you first of all we need to to discuss what what what an event house actually is yeah and the vent house is is look at it as as your server for your kql databases um it enables uh a new UI uh portal to look at what is happening inside your KQ data bases in one View and it also enables each kql database to talk to each other cross database groups and all that stuff um and the each kql database you can now enable through the vent house to copy or mirror the data out to one L it is still stored in the proprietary kql format in in in realtime intelligence but it is also in an internal process built by Microsoft and in fabric uh copied out in k files in one L so and and I don't remember the the the time from ingestion to kql to availability in one leg but it is a matter of minutes um and even seconds if we're lucky uh so this enables us to do a hybrid Solution on on real time because now we can do real time analytics on perhaps a Time window of two weeks or something and then the the grand Enterprise data warehousing B build on fabric we can do based on on on one leg but on the same data from from the real time in intelligence I was so if I was working in a production facility monitoring a lot of my machines yes and I want alerts right near as close as possible pretty pretty near real time near real time um I could use the event stream to stream that data in and have it in an event house set up alerts but at the same time offshore this data to a lake house so that I could once a week or once a month do a more thorough analysis on everything that happened on the data yeah that is really powerful that is powerful yeah and and uh the thing is also uh as I understand it Microsoft please correct me but but you will only pay for the storage once you only pay for the storage in the kql database and the storage and the copage one leg is is is free of charge so very cool yeah very cool so after that we also have new connectors for pipelines in data Factory Oracle Oracle is one now Google big query some Asia MySQL Snowflake and Dynamics so for Oracle we have Oracle database and Oracle cloud storage Google we have Google big query mhm so uh so if you already have a big data estate in Google p big big query you can definitely now uh link that into Fabric and start analyzing that data in powerbi or in some of the other tools aser AI search aser files as well MySQL database is easier to connect to now also with fabric snowflake adbc which will make it um easier and quicker to fetch data from snowflake snowflake if you are not in a use case where mirroring for snowflake makes sense where you if you are in a case where you want to copy your full data shortcuts announced on Las Vegas or that well that will come ever I don't know and finally we have something elusive called here Dynamics AX which last time I was an old product name for Dynamics 365 Finance but I assume it's connecting to the same thing we're pending trying to find find out but it seems Seems promising if we have a new way to connect to that data source but but the name is Dynamics a the name is a connector for Dynamics AX I'm I'm getting thoughts back to 10 12 years ago yes of a product so it's a time travel feature we have here got great that was the the main news I have of course still the the spotlight waiting to to show you all the power of the semantic link but before we go to that I would like to talk about some of the great conferences that we that that are coming up in Europe in the in the in the near future so we have a data platform next St conference coming up already tomorrow tomorrow and Friday yeah tomorrow and Friday um I think it's sold out so not much chance to to get a ticket but uh but great conference nonetheless uh where you'll be able to find both myself and you Brian standing there greeting people so if you you're watching here and you are at this conference don't hold back reaching out and say hello to us and we will bring candy we will bring candy and you will have a be able to have a game of Mario Kart racing with up on a on a console so you can beat us in that if you want to do that you can't beat us but try great the other one the next one is the European Microsoft fabri Community Conference the absolute longest name for a conference in the history of your conferences in Europe but nevertheless it's a semiofficial it's a very much Microsoft supported conference as and will be the absolute biggest fabric conference in Europe ever yeah we are talking 120 sessions and and thousands of attendees and hundreds of speakers and all the stuff yes so it will be pretty pretty epic stock Y and finally we have data mines which is always uh recommendable as well data mines connect which will be held in Belgium by the October 7 to 9th yes and with that said let's let move on to the spotlight of the day the semantic link because I think semantic link is a pretty overseen very very powerful feature in fabric so fabric link is a literally a link between the capabilities of notebooks and semantic models so it lets you do things with your semantic models from your notebooks and what some of those things could be I'll demo right now I'm in for a treat here because I haven't seen this no so again totally unscripted I don't know what I'm going to see here it's going to be interesting so let's dig into the code we have here a notebook oh we have here a presentation we have a sharing issue so hang tightas is coding [Music] with and there you go there you go yeah notebook so the Mantic link is a library for python so in Notebook you can go it's actually it's a it's a library for python in fabric notebooks actually not able to use this outside of fabric got it but it's a it's so it's a fabric feature but you can install it simply by wiping pip install semantic link that's it got standardized standardized package you install with a with a single line after that you this is some form settings you of course need to go and import the packages that you need so there there three main packages here there's Senpai relationships Senpai Fabric and Senpai dependencies with each of their own tool boxes and tools once we've done that we can do simple things like give me a list of the data sets in my workspace wow and in hopefully a few seconds we'll just run through the first ones here just to be sure run until this is always the fun part the last thing I did before we went live was run this entire notebook make sure the engine was ready but of course it's going to be uh and the engine turns itself off it's a clever clever engine cost-saving engine but when this runs uh and the result will come I will see a list of all the data sets that yes is it also adhering to the security so if I don't have access to to the data set can I still see it no okay you will you will use your own user as your principle for what you can see and what you can see so um so it's important to security building yes so in the meantime we'll wait for this to work it's ready now we can see a list of data sets it's fetching through that workspace generating a list and boom you see all the all the data sets so we can actually go through a specific data set like this example here I have a c cust profitability sample Auto which I can explore by running this relationship uh code here and then I see just the relationship Connections in my data set oh so I can oh okay so I can begin to perhaps if I'm thinking this through I could perhaps begin to autod document my my data model you could absolutely but you could even improve the relationship of your model because if you look at this this looks kind of wrong there are missing pieces you have many to many relationships I can even dig into and see the specifications of these like this is a many to one this is a one: one and somewhere it should show a many to many it's probably better down the list yeah yeah but it it it it should be there yeah yeah yeah okay got it so I could go and say okay I know these two are wrong mhm so I just drop them here this is just dropping them from my data frame so I'm not actually changing the model right now I'm just changing the way it looks got it so I can visualize it again already now it looks cleaner yes so what I would like to say well this is a good starting point M let me find out what are more good candidates for actual relationships good relationships we're like a um what is it called like a Matchmaker here we want to find good relationships um so we can read all the tables here with this code read all the list of the tables I get all list of all the tables and then I can use this find relationships code that I will give the input of the different tables I can exclude the relationships I already have oh because great want to find the same on again I can set a similarity threshold to see how little of a connection it needs because we want to match make them and I can even when I run it run into the deep analysis of all the steps that it's actually doing here if I'm interested in that so it tries to to find the relationships inside the data model without you doing anything yes okay great and finally it will give me the actual output which is the suggested relation new relationships here so I can take these five new relationship relationships and go add them to my model I actually did this here added them and suddenly we have a much much better model but there's more I can do more of course there is once I have my full data model like I want it to be I can go and run relationship violations which will scan through all these relationships and find any small things that are potentially wrong with them like in this case it found out that in my fact product key so in my fact table um one out of seven of my product keys for my products were not present in my fact table oh which means I have products that are not used in facts or it could be the other other way around I have products in my fact table which are not in my Advent Min table this script will will find all these mistakes and help us correct them okay so that is one of a whole range of features that we can use semantic link for um so definitely recommendation from here go try it out it will be a super power once you you get to to learn it got it so let's finish off today with the small quick tip of the month yeah so tip of the month for this month is not a lot of people know it it's a new it's another way to look at your fabric consumption if you don't have access to the to the the report the consumption breakdown report you can actually if you are owner of the Asher subscription with the fabric capacity so if you're an Asher admin or similar you can actually go there and use the new preview of the new cost analysis and budget in in AER you can dig into the resource of your fabric capacity and you can see your consumption all the way down to the different engine so you can see is it spark consumption is it Warehouse consumption and you can even see how much consumption are unused which could be utilized and so forth so great tip to uh to end the stream on thank you for listening in and have an excellent summer holiday and uh hope to talk to you see you bye see you bye e e e e e