hey everybody and welcome back to R freny this time 2024 today I have a guest here with me in the studio maybe you want to introduce yourself jaob hello everyone it's great to be back um my name is Jacob I am a solution architect and uh I'm very happy to be back next to you very happy to have you it's been quite a year of a lot of new things and we are going to talk about a lot of those things during the next 30 minutes we have an action-packed schedule um of course we have the most recent news so what's what's new since the last time we did a fabric freny and this has been little bit more more than a month because we did a winter holiday break so what's news in in in December and in January there are good few good things don't you think I think so and then we also have the monthly Spotlight so we have a specific feature we want to dive even more into and emphasize and I picked a good one this this time we added something else this time we also picked each of our own top five features for 20 2025 and we'll talk about why we believe that those are really important then you got a demo of a specific feature yeah a mystery feature y we'll be making uh sweet music with our data like literally music oh yeah literal music with our data and finally that's the ti of the mon month isn't it making music without data yeah and uh and we will demo the the the spotlight feature as well so today's Spotlight feature is the Lakehouse connected to aser data Factory ooh drum roll so something completely outside fabric something that sounds very boring on paper I picked the the most gray titled feature um but I really think that this one is very important and I'll tell you why a little bit later looking forward to it yes so for anyone tuning in who may not already know exactly what fabric it is let's just very shortly recap that fabric is the endtoend platform where you can host all your data needs so you can do everything from ingesting your data preparing that data and finally using the data um everything together inside Fabric and inside the one L the one common storage so if you were ever looking for a platform that hosted data legs machine learning workflows data warehouses data lake house and all these fancy design patterns within data well all of that can be done in Fabric and possibly more then that's fabric for you and in fabric new things are being released almost every month except for January so we got it's been so for many years been it's it I think no I don't think the people in Microsoft also needs to have some time off oh so disappointing it's a very depressing month begin with that's fine so we have features from December which are still new features which are still worthy of discussion which I think we should talk about y the list is long the list is also far longer than what we are able to go through and to be honest not all of the all of the items on this list is equally interesting that's right so we picked a few yeah well I think that's my CU because first on the list we have a few items that I think are pretty cool um there they are the unob um um interaction update and the uh column styling for and Bard styling and the customization of labels all really good visual properties that we've been looking for for quite some time and frankly maybe even uh and I'm talking about uh the data label labels MH and maybe even something we should have had a long time ago in my humble opinion but uh it's uh really good that we have it now and uh and uh uh I'm just looking forward to uh use it so I can have put much more context now into my data labels without doing a lot of efforts yes it's really just a a as if you would have uh created a tool tip ni and uh but it's on display but going back to this one so the the body chart this really needed I mean now I can add air in between my bars I mean how do I even interpretate what is this ER supposed to I'm sure some of the uh more uh backend oriented developers would would uh suggest that visualizations aren't needed at all give me a select statement and and we're good to go but I think it's wonderful and I think it's uh uh visually appealing it is it is and we already seen it used on Microsoft some of Microsoft websites themselves where it doesn't make it easier to to to interpretate the the chart but it is it but it feels nicer so so yeah maybe that yeah that that that surely has merits in itself but yeah it's not going to change your life it's not going to change lives yeah then we also have data alerting something that I'm interested in um so we had the announcement and the the release of data activator a few few months ago yeah and now we also have a tight integration with powerbi so if we're inside powerbi and we're looking at some actual data we can now actually get a dialogue box to pop up and just enter a few information and suddenly we have an alert that can tell us if our data is going in the wrong direction or the right direction or the right direction if we above or below some sort of threshold or even more advanced rules so so we now have more easier way to add these rules to data activator right from inside powerbi and at First Data activator seemed a bit underwhelming but certainly it's it's getting there yeah surely not the functionality more maybe more that the UI was um a little bit cumbersome to set up I mean now that we can just go to a report and and click on our data and say I want to alert I want to make alerts on this data mhm that that should make it a lot more powerful and here's a curve ball yeah um why do we need data activator if we have a metric or kpis so I think the the it's it's it's a curveball um so the the the quick answer is you can do much more advanced ruling with the with data activator so if you are a company with refrigerators and you want to check that they're not that your food are not spoiling inside these refrigerators with data activator you can set up rules for like check the last five points if three of them are above a certain temperature make an alert yeah so if you had to do this in metric you would get three or four alerts before the system even calibrated and found out if this was an actual problem or not and those would be false uh false positives with data activator you can actually look at the data even the the data going backwards you can find out where is that sweet spot where you don't get all these full positives but you actually catch those refrigerators where the problem was there in in real life so so more advanced rules is the the the short answer cool sorry about the curve no no no that's fine and then we have something that is completely unsexy but but important we have automatic lock checkpointing and restore points inside the data Warehouse of fabric it's getting hot in here maybe you want to tell me what this is oh I have no idea yeah I know so log checkpointing is um and we're getting technical here so when we store data in a Delta Lake uh in a Delta table which everything in fabric is stored in mhm that is just dumb data it's not an actual database so we need something on top that makes it behave and act like if it was an actual database that is the Delta that is the Delta layer the Delta layer provides a bunch of metadata files saying well this file that was added needs to be included this file that was added needs to be deleted from the whole thing and as such every time we add data we add a new metadata as well so if we have data that is um updating very frequently maybe every five minutes then a checkpoint no sorry then a a metadata loog is added every 5 minutes wonderful so for a year this could be hundreds of thousands of of metadata and before we can even read our data the engine has to read all of those hundreds of thousands of metadata files to just start consuming the data even if it was for one single data point so it is a tool for generating more metadata no that's very inefficient so it's a tool that takes all that history and say well we have like a whole year of history of what has happened to our data and to read the data we need to understand that full history history it takes it all it clashes it into one and then makes a checkpoint basically like if you're playing a game we add a safe point at checkpoint and that means whenever we do a new query going forward we never have to go back before the checkpoint we can just use that checkpoint and say I want the data as it was there and then any changes after that okay and with the new feature here this becomes completely automatic It's Just Happening behind the scenes it will make an automatic checkpoint every five times new data has been updated so we don't have to do anything we get more efficient queries and everything is just running better side effect is it's also adding these restore points which also means we get better options of restoring data if a problem arises if we have corrupt data or something like that okay data data data integrity and data security is just out of the box better with these features well sign me up yes sure thing and then we have a one leg integration for semantic models so we can now go and click on our semantic models which we should add is the new word for the powerbi data set yes well actually actually our friends in Italy have been saying this word for many years and uh our friends that Microsoft has been as well or at least some some of them this had this has been widely seen as the term for the data set for a long time so the data set in paria is the same same thing as the semantic model and now it's officially also called semantic model the new feature here is that I can go to the semantic model and I can click a button and by clicking this button whenever data is imported into this model mhm it's also making a copy into one leg as a Delta file aha mm and this in itself it's nice if you ever wanted to move this data somewhere else or if you wanted to expose OS this data as SQL or something like that yeah it would it's nice to have it there I think I would but it's also nice especially nice in conun conjunction with our next feature our next news which is the updates for the semantic links oh yeah because the semantic link which is a python package that lets us use our powerbi semantic models outside of powerbi yeah got a bunch of new features so we can now do actually things like refreshing our powerbi data set from code outside of powerbi we can we can execute um um updates to our table object model via uh via also python code now we can do tracing and we can optimize our Dex queries we can do data Discovery make rest rest API request and and much more but one of the new things is also that if our data is connected to one leg if we query this data with the with the semantic link it's actually not going to be adding consumption to the capacity for the semantic model you can now just read the data right outside of of the one leg copy of that data no yeah saving away consumption saving away complexity making it um pretty easy to just point at our semantic models and be that one point of contact and we can get data both into Ro format or in the model format in the in the model and it's just just cool SS cool yeah and what it means is that we can do much much more with powerbi from code which we have been needing a lot for a long time yes and the final news is we got the release of the fabric career Hub yeah so can I have a career please yes you can There is five steps and uh then at least at least you have a certification which should be a good good uh Step at least so now there's a joker side there is a pretty good list of materials inside this so there's guidance of how to approach the the certifications with fabric there are places for group learning for mentorship for finding communities around uh fabric there's inspirational materials from from other people who have um focused a lot of their career on fabric and there are role guidance for specific roles so even if you're a data engineer data analyst data scientist or an an analytical analy analytical engineer there should be plenty of material for you to to browse and learn from inside the new fabri career data Hub so definitely uh worthwhile to check out I will and then we have now the final drum one now then we have the one I've been looking forward to that Lakehouse connector for Asher data Factory wow we got a feature for fabric that is even not inside fabric how cool is that wow yeah mind buckling actually so there's been um there's been a limitation in Fabric or ever since it was released um and limitation that a lot of companies out there it actually has been a showstopper for so if we wanted to pull data from an un premises data source that has been cumbersome if not outright impossible for some data sources in Fabric and in those cases we've had to do workarounds building mini Solutions in Asher or even not moving to fabric because I mean if we cannot get our source data it's impossible to do data analyst work in fabric so that's a challenge we've been able to get this data inside data Factory but but that didn't help us either because we couldn't get it from factory data Factory to fabric and inside fabric we couldn't get access to to the data in um in y Prem sources so we kind of had this mismatch or disconnect we we couldn't solve it and the the the easy solve would be to get unpr support inside fabric so just go to fabric connect to some unpr source and pull in the data since we don't have that we have now the next best best thing which the whole no no no build it from scratch which will enable us to pull data through a aure data Factory directly to fabric so we don't need to store the data outside of fabric anymore oh nice so even though the the compute even though the data Factory is not part of fabric it actually just becomes a tool for ingesting that data inside Fabric and we never have to store it in aure we don't have to set up some uh complicated aure Landing Zone where we have very tight control of our data and our data security because the data security is actually never uh going anywhere else than directly inside fabric so that's neat and the implementation is fairly simple I'll but you can't do it I i' I've prepared of course so I have here an example created a workspace in powerbi sorry in fabric called sandbox which has a Lakehouse and I want to have data in this this Lakehouse that was not the right Lake housee that was the right so let's just say that I don't have this data right now I have some data but not the data that I want I want some data from Venture works I can go now to this uh Factory uhhuh and in data Factory I can make a pipeline uhuh copy data pipeline is right here and this one we just drag in a simple copy activity like this copy we add it here and we need to set up some so and sync and for the so I can just pick my adventure works this was already set up this is an technically this is not but this could have been an onp permises uh data source loading the data and I have the data ready in in data Factory I can even preview and check that I have actual data and in this case yes I have a single line of data that's nice so now I can also add for these data sets I can add these new Lakehouse connectors so I can either connect to fabric Lakehouse files or fabric L Lakehouse tables and here I've set up a data set that is connecting to the table I can add here the table name we'll we'll call it fabric freny demo I think it's called fa fa probably not and we'll can we can save that and let's run it so now the days is set will run the pipeline will run and we can now get data from oh no that's an [Music] ill oh no oh no are the demo guards tickling your belly probably we'll give the demo guards a few minutes okay to get their things in order sweet demo guard please help us out in our our there go go data connected so when I refresh this we should see data and there you go oo data preview loading slowly Shing the one one row okay but it's still a working functional row it's still a working function world this is this is on purpose so let's run it again because I've had this question what what about then rerunning this daily or every hour what actually happens here then with my data set does it because in FA break we have the choice do we pick override mode or aent mode do we just stag it on top of each other a new copy and get a lot of duplicates that would be wonderful it would wonderful if we did incremental load or we did Delta loads but if we do a full load we really do not want to Stack it on top of each other we just want so then we delete everything so question is what is actually happening here what what what is it is it doing if we if we just use it out of the box and the question is if this a little bit quick enough here could have succeeded again can we force it to refresh yes we can good one one and we have two identical lines meaning wonderful it was appended it's not so wonderful now we have duplicate data Double Happiness right not Double Happiness so the thing is that this may be very much desirable if we can do Delta loads if we can do incremental loads because we can reduce the load time a lot but um you're saying all these fancy words incremental load what what is a Delta load it's Delta load is the same thing as an incremental load we Target only the new updated rows so only rows with changes rows that had been deleted or rows that are new and then those are the only rows that we insert into our data ah I see and then instead of pulling our for example sales history from the last three years we only have to pull the new sales from yesterday so it becomes much much less data every time we move data it's much more efficient but sometimes we have a system where we do want to make the classical import full load full copy of the whole data and just put it somewhere and right now this is not possible in the UR but we have a trick the trick that I actually learned from the also very nice Microsoft MVP anti Cutler who told me that I should just go inside the code which of course it's it's obvious we should just visit the raw code and inside the raw code we have this little little tiny setting called table action option if I simply just change this to overwrite oh oh I will actually make it override and this is of course well documented within Microsoft documentation documentation about this feature oh I see but we can figure it out because it's called override in fabric so basically the functionality is there the UI just isn't there yet okay so but that's big news for me especially that there are jsons that are not documented it happens once in a while once in a blue moon you will have undocumented things from okay so that means now with the override mode I can have the the behavior I intend I can now pull in data from my unprint source unpr Mrs sources and we can move that to fabric so if anyone out there is working with on premises data and that has been the Blogger for you not moving to fabric well now is the time the solution is as simple as ever it can be set up in in a relatively short time yeah it doesn't require a lot of elevated um permissions in your Asher it does need a service principle but but all that needs to have for permission is to be invited inside the workspace it needs to work in so it's it's fairly simple setup um that's really great and we have in our Shel A playbook for setting it up this up rather quickly so uh if you have this challenge feel free to reach out we'll happily share share the guidance of of how how you do this so if you have a table with one line no and it doesn't no of course it works for all the tables all the lines this is this is especially good for large amounts of data um but you if you ever have an unises basically if you ever have an unises system and you wanted to move into fabric now is your time yeah great so that was the news and you also had sweet demo that you wanted to show us today of a cool funny maybe not as useful but often overseen feature in h is that right well you're absolutely wrong it's very useful and um and and before I dive into this I should of course um give a huge shout out to David Elders the most inspiring um man who walked the uh the Northern Hemisphere um he he thought of this thing and I'm just mimicking um but really what we want is to make sweet music with our data that that's what we try to do when we're working in powerbi and he figured out how to do that um so basically what he did was um create a table with um this uh type of um HTML yeah HTM HTML um binary sound and it has a frequency and uh and uh we connected that to our data and uh great so when you say make music with data you're talking about actual sounds yes so um before I turn this on I should maybe um give a quick warning to to those of you that are wearing headsets because um you are in for treat uh and here we go drum roll let's hear the data let's hear the amount of data month by day can can can I sing alone okay so basically you're going to go tell your bus well for the eth day of January yeah we had Boop amount of dat and then it was Boop no so that's one well frankly um we could connect this sound to to a streaming data set and it could give a a beep every every time we had a any sort of alert or oh hold wait for it and good one boom so yeah that was the um that was the gag um if any of you are curious uh uh on the details of how to set this up um well I do have I do have a few minutes can I show it yeah yeah wonderful so basically what we have is the HTML visual um and that is by far the most useful visual that you could ever have um if for instance you want to to show a YouTube video as your background maybe rolling waves on a beach or if you'd like um a bit of music with your with your data um this is something that the uh HTML visual can do for you and many many other things you can even embed other powerbi uh embed Solutions inside your desktop you can do all sorts of wonderful things nice I have still one minute right yeah yeah and if we also connect the play Axis then two wonderful um custom visuals will be um ever so useful together ever so it's uh it's Synergy um okay okay so let's for the final minute let me ask you the the serious question here do the do you see any real life applications of playing sound with your data ah sure sure maybe um maybe if someone is not as geeky as you and they find data a bit boring and they maybe fall asleep and then sometimes once once in a while M we we give them a nasty squeak but on a serious note I mean there' probably be better ways you could have an AI narrate your data for you you could have as something that's too easy yes but but there could be cases where someone cannot see yeah yeah I still not not convinced that this is the best option but it surely is an option for them to listen to the data rather than than looking at the data okay okay it is a silly goofy thing but um but it just goes to show that uh you can actually have sound in you combine third party tools with what we have in powerbi it's it's actually surprising astounding what we can actually do inside this this tool yeah and we were playing the uh playing the bar chart um for some time yep but let's wrap it up it's uh the half hour is gone it's been an absolute pleasure we talked about some very neat features and also a very cool one although maybe not as useful but very cool indeed so so good seeing you today thank you for tuning in and see you next one bye Cheers Cheers that