hey and welcome to the fabrick insights where we bring the news from Microsoft build yeah together here with me we have Brian in the studio and myself Matias and we look very much forward to uh to talk about in the next little hour about all of the amazing new feature that we've uh we've seen from Microsoft build around Microsoft fabric a lot of Microsoft yeah um and um actually build fabric was was released what a year ago at build a year ago so it is also the Oney year birthday or anniversary what you would like to call it of of fabric yep it for sure is so we brought you an agenda for today we exactly as you're mentioning we want to talk about this year and what year it has been MH um so we're talking about the year fabric we also are going to talk about one of the very big news this time which is the realtime intelligence inside a fabric yeah then we also have a bunch of different features uh we have flows we have multitude of AI features and uh and a lot of new things for the whole ecosystem of fabric so um so let's let's get started so we've had now a year of Microsoft fa fabric yeah and what a year if it's been it's the birthday of fabric we CBR whatever yeah and and it has been a a an exciting ride and and th thrilling right because fabric was was adopted quite quickly from our clients saying hey this fabric looks interesting can can you can you help us understand what what what it is utilize the platform and how do we actually get started yeah so yeah been a lot of interest even in the first half year of doing Pilots improve a concept and really seeing that that this this last half year so many organizations are adopting this in their production workflows and as their newk unified platform MH so the the the state of art here for Microsoft fabric is yeah where where do we see ourselves are we is it a mature platform yet uh I think it is becoming more mature from my point of view um um I still lack but that is perhaps due to my nerdy approach to the SQL language I still lack a few options in in in in the SQL language world uh but I I do believe that the platform from this from these releases got at built is is way more mature than we just saw like three months ago yeah absolutely um and we still maybe have a little bit of a catchup especially around the the Enterprise features of cicd and deployment yeah the API support and so forth but everything around just creating your actual solution seems to be uh getting yeah very mature and very very usable yeah and if you follow along on the Microsoft blogs you will also see that I think it was yesterday or the day before that we got even more features like API support for real time and and all that stuff so things are coming after builds um um and in in the upcoming few months I hope we will see even more news yeah and what I've also really liked is that while we are playing or Microsoft is playing this catchup of of getting those Enterprise feature we are still getting a bucklo of interesting new features that would just help us uh create awesome Solutions and I I think we have brought a lot of these kind of features also for the the session today so if you tuned in you're in for a treat there's going to be a lot of interesting stuff that we've seen from build from U from this year yeah so before we talk about everything new let's look a little bit about what we see as the game changers of this time um i b a few of of what I believe are the game changers here and I would also like if you have some someone some in your mind so let's start with my with my my Pixie here MH I am truly a fan of uh one Lake I think it's been talked about a lot but I don't I don't believe it's hadn't even like it still feels like it's it's underwhelming how how how people are talking about it because well the worst I hear is someone saying it's a marketing gimmick or something like that but I really truly believes in the power of us being able now to collect all of these other platform in inside in one place in Fabric and and unify them all and I've seen it firsthand how cumbersome complex migration projects if we were to do them two years ago now can get this intermediate step where we just Mount whatever previous platform people are using into Fabric and can let fabric be that unified platform and nice and easily start migrating the the the different workflows the different sub Solutions without the rush of having to to move them all as a big bang very quickly so we can do it much more incrementally and much more in the the right Tempo in the project it is also I I I don't remember the correct wordings here but I think I saw Communications before built that one leg would become the the the the one place for all your data Without Borders and boundaries and and and brick walls so no matter what what data provider you have Google Amazon and and and all and all the other stuff um um you will be able to use that data in one day exactly so if you show here on this on on the slide exactly what you're describing we are able to inside the fabric platform we're able to have data from data verse snowflake custos DP Amazon S3 a your data breaks um and in the future even more asure SQL uh on Prem SQL um and and many more and we can have them in Fabric and they will just appear as one unified data platform with all your data we can do reports on it with queries and things and all those queries are just going to be distributed and and push back to their original sories and you suddenly have this live overview of your data across so many systems it's a really neat vision and I really think that that one Lake did a good job here yeah then I also uh have the capacity building which seems like a mundane thing and some would even you that it's uh it's a little bit um annoying but but I really I really um experienced a very positive push back from from clients organizations who are wanting to use fabric that they really like this cost model MH um the thing with with the capacity building in fabric is that beforehand we had to pay for all the individual tools whatever they were using so uh so if we used a little bit of data Factory we paid for that if we used a little bit of notebooks we paid exactly what we were using which also meant that the as you can see at the graph that the cost is fluctuating and it's it's unpredictable for the for the cloud version whereas in fabric we have this hard limit of the capacity we know what we pay we have the same stable bill every month um and yes we may pay a little bit for that extra excess capacity that we are not using but um but in my experience that's a pretty good tradeoff for knowing what your bill will be month over month yes so it's worked out pretty neatly I totally agree that was my two two game changers do you have any we should uh we also should talk about if anyone out there just follows briefly along my my LinkedIn profile then you you will see that I'm I'm I'm a lot in into the real time world yeah um and this is also a main part of this stream that uh the what I saw from build was that the main the main area of news at build was in real time uh absolutely we got a new name realtime intelligence instead of realtime analytics because now realtime intelligence services are now in in the next era of uh delivering insights on your data on your streaming data delivering Custom workloads Custom event streams and custom things at the next level after analytics we can also now go into the intelligence part and and react to data react to events that are happening um um and this also means that uh the data activator is is now no longer there uh which actually reminds me on this t-shirt because this is now a collectible item because I still have the de activated local right there that's right but that will in future no longer be so this might be a collectible item in in in a few months I don't know so we're saying goodbye to data activator but not really saying goodbye to data activator what we are may be saying goodbye to is this we have a powerbi problem here wow we're saying goodbye to the term here real time analytics even the even the PowerPoint wanted to get rid of it for us exactly wow uh so now we are saying hi to realtime intelligence and with realtime intelligence we also got a bunch of new services and features great well lud let's move on to that that and let's talk about what is new in the intelligent data platform the first sorry no because as you said you've been really looking into this for the past past year and uh and I don't know if if the features were game changing features yet but but we surely have received a bunch of news around this topic now so if it wasn't before at least it is now I assume yeah let us let us here first of all we got the uh the new main menu item at the left in fabric called realtime hop MH and realtime hop is your centralized access for everything around streaming data it can show you how how many where your streams are where the streaming data is and and uh it can also help you with ingestion of of your aure resources and also help you start event driven actions based on what is happening inside your P tenant yeah I have a we have a short demo late later on um um and should we just quickly show people where they find this real time yeah I can do that oh yeah we have it here I have it right there so in in a in a workspace it would be just as you said the left left hand side yeah but for some reason it is no longer here it's fine it's there you got it oh Matas has it okay there you go it's on mat's screen uh it is real time Hub at the left um and um and the icon here actually tells a story around the the the the name I was I was at a live stream on on Friday with uh with dang from the Microsoft team and he told me that this wave look alike icon actually comes from the working title of of of real time Hub called called um called um data River oh cool sorry I I just I just lost that the the the one River for your data streams um so the waves is is trying to imitate a river so yeah uh and the event Hub is is uh the real time Hub is let me if I can click there um the real time Hub has three elements it has if if we look at the top just zoom a bit here it has the data streams it has something called Microsoft sources and and fabric events I'll get back to that to the to the two other ones in in in in a few minutes um if we start at the data streams this is the first point of entry we get into the real time Hub this now shows us everything we have around event streams I'll just scroll down uh to find my items here and if I zoom in right there this is an event stream I have built uh that has been running for quite a while now just collecting data to to use for for demos it is loading in the New York City Taxi data stream from Microsoft it's a demo data set yeah and uh it also shows me that uh just underneath this this is not in a given uh um um um order so I I'm just lucky that the New York City Taxi data set is just underneath this um so so the entry for data streams actually gives me an overview of what streams do I have and what syns SL destinations do I have around my real time data cool and not only is it giving me an overview it also allows me to click on any event any item to get an instant overview of what is happening inside my in this case event stream if I assume a bit here then at the first thing the main part I can see that this is the parent item for my things I clicked on the event stream and I have I'm now being presented with both the Upstream so the ingestion things where data coming from and where is data saved so the downstream directly here uh and to the right I will also see what are the incoming and outgoing messages as this is only ingesting data into my uh Cal database it is only showing me incoming messages but uh when you begin to read data out of it it will also show you the outgoing messages on the top I can begin to create I can begin to endorse it I can begin to create an alert based on this event stream directly and I can also begin to preview the the the the data um if I open any other like a a a a data set I'll see more or less uh the same thing but without the Upstream Downstream things because this is the last thing this is the parent but what I will get when I get to a specific um table in the right hand side I will get uh I'll get the metad data from that table what what is what are we talking about here so so we have the real time Hub giving us an overview of everything we have yeah for Real Time data streams and dig into each of them yeah and uh and see what they do and and metadata on them yeah and I remember on the last uh fabric fry streams you made you and I made yeah you showed us how to ingest uh live data from an azure database right how do we do that how how do we tap into the change data capture yeah with the with the CDC connector and the mirroring this is actually where Microsoft and the RM Hub is now also helping us if I if I go to the Microsoft sources uh you will see here all the resources from the aser tenant that I have access to that are real time capable ah yeah um so because the mirroring are actually a realtime connection to database yes I get a bit of glare here so I need to there you go if I just may I use your metadata database yes great I hope this is this is an example of of Matas has a metadata database in aure that I have access to for some reason um um and if I click this icon directly from the event Hub the realtime Hub I can now configure the CDC uh collection of that data into my real time uh database if I need that this is a help to actually generate an event stream yeah yeah yeah uh quite easily with a few with a few Mouse clicks I can generate um I can generate the uh the event stream to begin to gather the data from your CDC on this a SQL DB so quite quite quite awesome nice yes let me cancel this one if I can there you go yes cancel so the last thing I would like to show you is the fabric events when we had when we were introduced to synup analytics uh we got the possibility to start a pipeline based on a file being stored in a in a BL storage sure uh and we all asked Microsoft hey how can we do how can we how can we do this in fabric Y and they were just saying hm just wait but now we don't have to wait anymore now we can start any process in fabric based on what is happening in a blob storage uhhuh so if I click the blob storage event to the right you will see and this is only the initial load you will see all the things that I can now react to I can see if a blob is created if a blob deleted blob renamed and so forth and so forth and so forth so everything that is that is happening inside an Azure blob storage yeah I can now listen to as an event trigger and start another I react to it I can set an alert and I can create some action based on this happening uh and if I want to make if I want to see what kind of information that I can get of these events I can just unfold them and I can see all all the M all the Met data that has so many use cases yes but I think for the average user it can be hard to even grasp what we could do with with this because it's it's so so flexible do you have any ideas of yeah it could be uh if I may just uh also introduce they've also released things for pebri workspace items so when an item is created deleted failed and so forth so so forth I can also react to this so this is a completely new thing MH that I can now begin to start processes based on what is happening inside my fabric tenant or fabric workspaces uh use cases for this could be governance yeah how can I govern my fabric um workspaces uh what is happening there if someone by by mistake deletes a an an item I can get a an alert saying hey someone deleted this um um this is what is happening and who did it and if you have a cicd workflow or deployment pipelines you can go back and you can catch that error before you are too late to to detect it oh that's really nice and as we just saw in in both the uh the um the streaming uh Hub and and the Microsoft uh data sources it is just a matter of Select creative instream and then I'm being guided with the UI on on what do I need what would I'd like to happen when things are happening what if I wanted to get rid of in something like a data Factory pipeline I'm I'm pulling some new data and I put up a schedule maybe I do this every hour or every 15 minutes could I skip that whole schedule and just listen and wait if new data arrives start processing this data more or less yes nice nice nice of course it it depends on on the origin of data and all that stuff but I'm a consultant so I said it depends but but the first answer yes um cool yeah so that was the real time H and and if you have access you everyone has access to this it is right now in in public preview uh so you might see errors here and there um um but begin to play with it and uh around costing here it is the cost here whatever you are doing in this real time Hub and of creating elements is actually just creating an event stream by the UI so uh the cost is the same as the event stream is uh right now cool what what is what if I wasn't using real time Hub yet sitting out there maybe I have a a symbol data warehouse data platform something and I don't necessarily have iot data or Telemetry data or something is this then not for me is it for those use cases or would there be uh reasons for me to look into this there would be definitely reasons to look into this because as we saw from the real time so it from the Microsoft sources uh when I have a database somewhere in ashure that I would like to get data from you could do it from the real Sam Hub with a few clicks of of a mouse um um and the fabric events uh actually can now trigger pipelines notebooks things to execute based on things that are happening inside your tenant or in a BL storage somewhere in the world so I truly believe even though you don't have iot Telemetry on all that type of data look into the realtime Hub that is also why it is a main part of the left menu item because it is for everyone to use even though you are not a candidate of telary and RT devices so great so if I want to do something quicker than every day or every hour something that is reacting reacting in in real time this is this is for me yes this is and who wouldn't want that exactly so um this is this is I see this as a huge addition to the fabric Universe after this this um absolutely this release so uh uh what else do we have on the slides uh yes this is the one um we also got a a lot of of new updates around existing things in uh in in in real time we got real time dashboards um real time dashboards is is a new option to visualize your data based on on the realtime data that you have so if you have if you have ever worked with real time you know that you need to write the kql the cust query language statements in order to select your dat and work with it every kql statement can now be the resource set from that can now be translated into a visual and and placed on a standardized dashboard inside fabric nice and they will update whenever new data comes in um we get updated event streams I'll show you that in just a few seconds uh the Rhythm Hub we went through that and we got a new co-pilot of course we got co-pilot co-pilot everywhere yeah co-pilot everywhere uh if you have trouble writing your C World quer sets your select statements and all that stuff you can now get help directly from co-pilot if you have an f64 capacitor or above uh we also got the event house the event house is when I tell this to people that haven't worked with with kql before or are great at SQL then I tell them look at event house as the as the server for your databases in fabric we have kql databases and kql database and the nql databases but now we can consolidate those databases in an event house and when when databases are part of an event house they can talk to each other great I was just about to ask what what do they actually benefit from being part of this virtual server they can actually now talk to each other um uh quite easily so you can do cross database selections um and initions and um I don't think we can reach that part today but imagine that you have three databases and you would like to do medalin architect in real time mhm then with the event house and if those databases are in the same event house you can now within a few milliseconds modify translate transcribe and all that stuff with your data that you used to between your bronze silver and gold layer within a few milliseconds through the event house that is awesome cool yeah that's awesome so if I may I would like to show you some of things some of the things around the updated EV streams yes you may absolutely great I have a work space right here and let me just create a new inent stream uh um event stream right there when I create a new event stream I'm actually prompted to do you want the enhanced capabilities we want the enhanced capabilities we always I always want the new things so I would always always say yes enhan capabilities and enhanced capabilities is currently in preview and it is enabling you to scale up scale down uh manipulate and govern your your event streams in a whole new way cool um so I would let me just do that help me follow along here so what are we trying to achieve with what you're showing us now what's the what's the end goal end goal here uh I am trying to uh to show you what event streams can and and also how easy it actually is to collect data from a data source and to manipulate that to whatever need you have great so you will show us collect data and transform it but not in the batch way not in the the the typical way streaming way in the streaming way so when when an when when a record arrives within a few milliseconds I'll have that in my syn somewhere in the world nice yeah looking forward to it yeah so this usually takes a few seconds to spin up a new event stream um um so right now you'll just see a wide screen uh right now it's loading uh what's happening behind the scene is everything in fabric is based on a standardized aure service mhm I think it is probably no known butse you know that now um and on these event streams we have event Hub we have uh um um streaming uh analytics streaming analytics thank you and and other services to support whatever things you're doing in Fabric and in event streams we have uh event Hub and and streaming analytics um this is the new look and feel for event streams and I would like to create a new new resource this is also brand new so I can now get data from halfa from from confluent Kafka nice I can get data from let me see if I can find it there is more um the UI is changing uh every day so uh there you go yeah we have Kinesis as well yeah we have I can't scroll there you go we have Amazon Kinesis we have uh we have confluent we have Google cloud pops up and so forth and so forth so but for for this easy PC I will just uh connect to the demo databases if I can it is no longer there Dem bases are gone it is fine so let me just we're talking too much it seems like your end stream EV stream just idled yeah yeah because we're not doing stuff uh I just selected as and as a SQL and I will now create a new connection nice uh okay this will take too long time in in order for this demo to work so bear with me and I will uh just show you put oven and it's already ready yeah come on this is already ready great yeah that was easy yeah that was easy peasy sorry for that uh here I have selected uh the bicycle demo data from Microsoft yeah and uh and in the new in the new uh event stream experience I have either a look at it as give me the live in the the live things or I can select to edit it if I select to edit it I'm translated directly into a new UI where for every item here I can begin to edit the elements and notice this one down there uhhuh the new capabilities of the event stream is that if you have a stream that you would like to split into more substreams you can actually stream data to another stream inside the event streams oh so in in in in this part of this of the event stream I'm filtering the database on the neighborhood that equals the London Bridge yeah and then whatever filter whatever data is coming in I I'm I'm I'm I'm saving that into the sync called diagnos sync and the London Bridge stream or sorry the London Bridge data is now being streamed to another stream so so if I want to collect every data from this event stream I can connect directly to the event stream called diagnosing data this is from Friday's demo or if I only want the London Bridge data I can actually connect to the London Bridge data as an individual event stream without having to do filters on the full stream or something you you just have this substream exactly and let me show you because I can actually find it here uh find it words London it is no longer there I'm sorry it should have been here um but the stream should have been called London London Bridge stream yeah uh but we are still in in in in preview and then I could connect collect sorry connect directly to that stream and and and do my do my reporting based on that cool so and we got a whole lot of new things around dashboarding co-pilot and and and and and things so without bypassing the entire session of this I would uh hand it over to before we yeah no I'm curious about the the realtime dashboard that's that that that's that's that sounds interesting it is it is quite interesting uh if we think back on powerbi yeah then we need to wait is it 30 seconds for power to refresh automatically uh when new data arrives um and that might not be sufficient for for some organizations needing instant insights to things that are happening um imagine you have uh perhaps a a huge Park of windmills mhm and before all the wings break off we would like to have an alert that hey this bearing is is is beginning to sound strange please change that yep and the sooner we can change that bearing the less cost it will have in the maintenance going forward right Y and we might not have enough to wait 30 seconds to dispatch someone to do stuff mhm um the real time dashboards is now giving us instant insights on what is happening in our right now only in our Cal databases or in our kql databases based on the kql events that are the kql statements that are happening um um and we can begin to build elements more or less in a it is not the powerbi look and feeli but you can build elements as with charging and with with cards and information cards and all that stuff with instant updates whenever dat arrives yeah I mean for The powerbi Savvy uh viewer out there they would probably say I I actually believe we can get it down to even 5 seconds so we can we can get it down to 30 seconds could also be a short duration um I don't think time is the problem and and I think this is actually quite powerful because if we were to go for like 5 Seconds what would happen every 5 Seconds is that all those visuals would have to call whatever database the data is on even if it's custo or SQL and would have to fetch the entire data set do the aggregation and then forward this to the visual um now this can be quite fast if you have enough compute and kql should be able to do this quite fast but if you're calling that those tables every 5 Seconds getting a large sum of data that's going to be very very costly yes um very very quickly very very costly yes so if you instead could turn it around and say we don't want the visual to refresh itself every 5 Seconds we want it to just pick a single piece of data get that data and then update only the change based on that little data blip well that would be ideal and I believe this is what we are having here with the real time dashboard it it's actually a a real time architecture behind it exactly exactly it is it is it is it is it is based on the K Grill engine and the event streams and what is happening there um um to update question is why do we need a new reporting tool in fabrick and not just use the reporting tool we already have in powerbi do you have a good answer I'm not even sure Microsoft has one I'm I'm I'm not sure because this Friday I I I may have said something that was incorrect uh because I said that I looked at real time dashboarding as the second visualization to from Microsoft uhuh and on that call um uh on the live stream one from Microsoft actually says hm we need to drink a beer about that because that is not the intent from Microsoft uh and I still think that Microsoft is building on the story around real time dashboarding um um um why it is there I know a bit of why it is there it is to cover as you just said the help for only picking the single entry of the record coming in and updating the according visual going to run faster and better yeah um um but the I'm still waiting and if Microsoft is following along please email me because I would love to hear what Microsoft has take on where the Real Time Dash barding pit okay yeah okay and it if you are going to Germany this summer I will also do a Recon on that so that's we will keep our eyes in your and here watch and see how this develops because it yeah it feels weird to have two different places to do visualizations exactly okay but it is a nice addition I mean if I had a dashboard that I wanted to put up on my production wall or something like that and watch if my machines were nearly close to breaking down or my performance if if we doing work at the tempo we need to do it would be really neat to use this this real time dashboard for that so for for an engineering facility that that has huge amounts of of machines running and things are going haywire the real time dashboard could be an option there great yeah excellent that's cool oh this one is actually interesting as I just mentioned quickly is there should be more if there's more um um this is my own buddings and I stand to it to to my death that the event stream sorry the event house now utilizes us to build the world's fastest Medallion architecture Lo but I it's not it's not more than I think half year ago when I remember you saying on I think it was during one of the streams that we cannot build a medallion architecture with the real time tools yeah we couldn't at that point but now we can now we have as I told you that the event house now consists of one or more databases and those databases can now talk to each other so based on update policies and materialized views we can actually now load from bronze to silver to gold uh the bronze color is is off here but but but um but we can now load through through The Medallion architecture for every record coming in within a few milliseconds and those records can then be exposed through the real time dashboarding it could be that the events are coming in from your sources and you might add reference data to it you might add a join you might clean it up you might aggregate it or something right now we can't do joints in event streams I hope that will come uh I know that there's an idea on Pak ideas on Microsoft page to actually join data on the event streams when it comes in but right now if we need to join reference data and and add more data to the incoming data we need to do that in the K database we can do that by update policies that listens to what is coming in and doing the data manipulation as the records arrive in the database and then hands it over to the next table and the materialized view is just an always ready aggregated res set of a query that is based on top of one or more tables it sounds like a really interesting concept and idea the only thing concern I have is that the last time we talked about this we we talked about why the fact that we could not do the leader update was such a big big big blogger because we had to redesign everything to be incremental and and lots of filtering logic behind that so what has changed around that we also got deletes and updates in so yes um we also got uh the possibility to delete records that are coming into our K database and we also got possibilities to do updates well that makes sense that Chang is everything yeah but but please please don't use the realtime analytics to build your corporate data warehouse because then you won't have enough money in the world uh okay but but build it to whatever needs you have to get realtime insights into your data State great so we're talking about smaller Niche uh like um it it it it could be bigger yeah it it it could be specific use cases specific use cases and and the data size could be huge it could be an F1 formul one car that that streams Telemetry into into a server and then you have realtime dashboards to to make sure that the O temperature and all that stuff is is is okay that is a huge amount of of of of data there or it could be an airport trying to manipulate where people are are walking and and and and and going that could also be used for re again huge amounts of of data volumes there but but the corporate business-wise powerbi reporting look and feel and so forth it is not the real time way to go so interesting well thank you for a quick rundown into all the new things about real time and it does really look like it's becoming a a mature offering that is very very capable and and we have even we have not not not even touched upon everything that has happened in in real time after build um so more things are coming and has ComEd uh yesterday or the day before as I mentioned we just got apis to create databases and tables uh in in in fabric in in in real time intelligence so yeah and we've um had the chance to also try out this tool with with some of our clients we have examples of uh of clients who are using it actually like you're describing for their core Medallion architecture to do realtime reporting yes we're doing a big big EU funded project on using it to for sustainability and to actually save companies monies on their production while also doing something good from the environment so this is not just I would call not just a hype it actually has very valid use cases around there in the m in the wall so very very nice let's talk about task flows the next big thing and task Clow is interesting because when I first saw this I my initial opinion do you remember what I said at the time why it is there and and and why not just use folds exactly what what do we even have to use this for and I was a little bit hesitant on trying it out I actually think you need to sell this to me because I'm I'm I'm not what what is this I I I can try I can try thank it's hard because I didn't I didn't myself recognize the the value before I started using it okay so the thing is I was um I was building a little solution uh to do monitoring of a powerbi MH itself and fabric checking workspaces and ownership of workspaces and what workspaces were hosted hosted by which capacity simple simple monitoring stuff um put together a little solution and then I figured let me just try to put a task flow around it to help demonstrate for the client what exactly does the solution do how does it work whatever and see how long time that took so to my surprise as you see here I I set up a few boxes of getting data um a monitor Hub reporting I can't sorry and added items to each of them and what happened is that within honestly 5 to 10 minutes there was an overview of what happens in this workspace what the solution is doing there was descriptions for each substep oh so I can do commenting on commenting on all the boxes so you you have um kind of an a self-documenting solution we don't need to go and write a word document to explain what this is doing because the solution itself now tells me what it's doing and more than that as you see on the on the artifact list in the bottom each of the actual tools that we use the pipelines the lak house the data set and the report they are also tacked with what task they belong to so when when I go this one there yeah so when I go into this solution I can very quickly get an overview of what everything is doing in what relation is doing it and I can I can focus in on a specific Sub sub so it's a it's a great helper to get an overview of solution even though it technically doesn't do anything yeah it's it's it's it's just an an an abstraction level on on top of all your items it's uh and and this is this is this is uh somewhat simple so you have a get data you have a mon Hub and then you have some reporting can I add more complex task flows I'm glad you're asking we didn't even this so I'm just let's jump into a demo here where I I prepared a more complex workflow than this very simple solution so as you can see here I'm going to zoom a little bit in for you guys wait we have something here where we're collecting high volume data M together with low volume dat data so high volume would be pipelines or streaming low volume would be maybe data flows or uh or data dumps in CSV or something Bronx data processing of the data silver further Transformations golden and then finally serving to machine learning and data visual a big complex set of things and I can actually even say I want to start from scratch so if I if this is not what I wanted to do I could go and say well we're going to delete this this is not what we wanted to do on oh so I create a new new new um new thing where's my button there and I can get some inspiration here so the UI is not to create a new one in the top left but to click the green one yeah okay okay so I can get some starting I could do my general solution where I just go from scratch and add activities but I can also be inspired by something that already exists so let's say we want to do some complex Lambda architecture with real time data m um I can click this Lambda and I can select it and it will also already help me schedule and and show what a good good setup could be then I can go to each of the activities say well for this is helping us to understand the best practices or it is and it's helping us to build the solution so I'm I'm looking at you now the expert of realtime data so we want to do realtime data ingestion I click new and it's already suggesting me here well well maybe you should use an Advan stream I would also yeah great I'm totally there we are agreeing with the tool and I can click it if it works there you go and I can give it a name and we can use the hand capabilities yay and it starts creating the thing it can go back and it already here is it then what about the mapping then so when I click the green box to real time oh it is already mapped it's already mapped so I can go to the next one and say well now we want to do some streaming data storage sorry it's a it's a and and here it suggest just event house or a c cruet I go to event house we will go for the event house okay so we get the best practice architecture or blueprints or call it what you want and and then based on on on the boxes we can create elements that fits that architecture yes wow and we can help it can help us build it out the right way and we can have support from expert fleshing out the overall workflow and then a business user or or um newcomers in the industry would have a easier time creating the right solution but maybe even more importantly we can start having also templates for organization if we have a new solution for onboarding uh a new um acquired company or something we can give it this is the template you should you should design your your your data platform like this please follow follow this structure and uh you have this as guid to okay expert question perhaps I'm I'm not trying to toss you onto the bus here but can I do a task flow based on a task so can I have a task flow that that that has the Enterprise architecture level and then inside every box I have another task flow no okay not a possibility I'm not speculating if it will be but it's not part of the features set we have now but what you can do is that once you've added activities to all these boxes you can actually also use it for organization so if I wanted to dig into what com components are inside my real time adjust I can click it and it will filter my my components down here to be only those nice and all of these steps has a description for name it has a description for text so we can explain what's going on in all of them and finally the whole solution as a whole the the workspace itself has also a description which means it will be very easy to help whoever has to take over this solution explain what does the solution do what are the components inside of it um what are the individual artifacts we're using within each component and why are they there and everything is um as we talked about documented so if I look at real time inest it will show the description the header but it will also show the actual um elements elements got got it yeah sure yeah um so and if I did if this if this doesn't fit with what I want to do I can always go and add my own thing so I have to the standard things like get data store data but I also add a general one that I want to make my own thing which could be its own name it could be like uh I don't know documentation or something catch data this doesn't mean anything I'm just writing something and um and uh and I can add item to this as well and I can even drag and drop and let me zoom in add connectors and we can also add it as part of the flow here there suddenly we have map change to the flow we can also start from scratch and not follow a template just create our own flow so it's a very very neat addition helps create more coherent Solutions uh is there a co-pilot help to get the descriptions in in in the tasks not yet but it's a really good suggestion I myself in my when when I tried it out I immediately went and took a screenshot of the page and say this is my solution these are the artifact this is the task flow this is the activity I'm talking about and I sent that that screenshot to uh to co-pilot and said co-pilot help me make a description for this step of the solution and I got something back so I know for a fact that that it would help it helps to create this description we don't have the capability we need to manually copy and paste back Microsoft if you listening perh add co- would be nice would be nice speaking of which co-pilot let's talk about the new AI features Inc fabric yeah because because it's it's is is it is it becoming easier or harder to work with AI and fabric easier leading question okay got it we further than we've ever been before AI everywhere nice so AI we said it earlier AI everywhere you added AI to the to the realtime intelligence M we've added or not we've Microsoft has added real copil to uh to um to the warehouse yes and actually finished adding it to everything but before we talk about co-pilot I would talk about something I find even more interesting which is the new AI skills AI skills AI skills because with AI skills we are now able to chat with our data warehouse but we are able to to do even more than that so as you can see here on the slide um AI skills is a feature that allows you to can we no we can to to I can try yeah to look at the tables of our warehouse and we can click and enable we want to create a new skill for our AI that understands these two table so in this case it's customer loyalty and customer support table these two yeah yes we need to click these two do nothing else suddenly the AI under understands these two we can add relations if we want it to understand the the relation The Joint between the two and we can start asking questions so for example here how many calls have we had from loyalty member by the S they joined from and it will give us the answer but even more impressively it will also give us the query that gave us the answer so it's not just guessing or hallucinating it's actually using its skills to create a query that we can validate and check which will give us the answer okay so so this must be the query then this is the query I can now base so when people ask where do you get those numbers from I can actually just know that this is the select statement that copilot helped me with and then there go okay so and it get and it Returns the query itself as well as as the output of the query so as you see on the slide it had the query but on top of the query there is the actual data yeah so that's a pretty good way to get an and answer this is the resource yeah got it okay and there's even more than that oh yes because this is an AI skill so we can take the output from this part we can publish it we get a URL which now we have a URL for this skill so an API so we can take this skill and uh and add it to co-pilot what yes this is a feature that is announced but not available yet but we can take this skill once we we created it we can go to co-pilot Studio we can add it to whatever AI engine we agent we are creating which means that this AI agent will Now understand our loyalty program and we'll understand it in a way where if we ask it questions it can go run this AI skill it can give a result from our for data warehouse and it can get the results as well as the query and can use both of those to give us a fact-based answer so so okay okay mind-blowing yep absolutely mind-blowing so actually if I have normally in a corporation you have powerware reps like a lot yeah and and even though you have hundreds and hundreds of of reports there's always this these questions that are not answered y So based on this if I have enough AI skills in my fabric Universe I can actually now begin to chat with my data and get the answers to those questions that I have Bas based on your data yes and even more than that if it's if it feels like complex for you to go create an agent in co-pilot which I think everyone should explore doing because it's so powerful but anyhow that can be complex just wait for them for Microsoft to make the feature here where you can add the AI skill directly to co Office 365 co-pilot which means that you don't need to go further than teams find that co-pilot tab ask it questions so based on our data and of our lawy programs which kind of members are dropping out mostly based on um our last communication with them and you will get an answer and the co-pilot will not guess or hallucinate it will go and fetch your actual data from your data warehouse using the AI skill and give you a fact B based ANS based on your own data and fact-based answers it's such a mind-blowing thing because a lot of hype has been around uh AI agents large language models and so far but their biggest weakness is that they are not fact-based they're guessing if you want to use it to predict some numbers it will take a guess um not much more much much more more valid guess than if you asked your six-year-old or seven-year-old to take a guess based on these number I mean maybe a little bit but it's not fact-based it's it's guess based it's uh statistics based yeah with skills it becomes fact based and you can track it down to what actual data it used to give you the answer based on queries to a SQL data and and okay so this also demands and this this demand has has always been there but it also demands for the people asking the questions to actually know how to ask the right questions and this this has this has been a problem like since we got data because then we got requests from the business I want this number and then they got a number say no that's not the one I want this number yes so they need need to know something about it but this is this is a great question and this is a NE extra feature if you can zoom in on the Model Behavior part you can see to the right there's actually a notes for the model this one yeah yes so if you for example want to tell the model that in our organizations we usually don't use the term Revenue instead we call it uh cash performance or whatever weird scenario you can actually tell the model if someone is asking about this this is related to to to this kind of data so you can actually help the model understand the question it may be asked okay all part of this symbol setup and all you need to do here is two clicks perhaps write a a note for the model yeah deploy your AI skill and then you have your skill and then get your your your M M365 admin copilot something to add that skill to go that's a feature we don't have yet but it will it is on the road map it's been demonstrated it it will come so right now we can start creating the AI skills we can use them here in the UI we can use it with with copilot studio and we will in the future be able to just embed it into our Office 365 co-pilot so everyone in the organization can ask it skills and utilize our our data much much better so honestly for me this is the this is the pill moment I've I've had a lukewarm feeling about the use of uh of of llms in in data I mean large language models yes um AI models yes had to do a lot of work preparing the data feeding it to apis getting results fine tune prompt engineering chain of thoughts there's so many Concepts to fix to make this work um and sure those are still applicable if you have complex cases but here we have such an easy way to let the AIS the the generative AIS speak on behalf of our data and actually use our data for the answers I think this is a awesome this is a groundbreaking it's game changing yeah and on top of that so that was the AI skills on top of that like we talked about copilot is now part of every fabric experience so in the data Factory um in the warehouse uh in data science realtime intelligence and powerbi we have co-pilot everywhere helping us do our work better and fast faster and later on in the time period from the releases in Microsoft the connection between these five co-pilots and the co-pilot in in teams we get that connection later on great so um so definitely ai ai everywhere and and if that wasn't enough we also have a huge Focus from Microsoft on their whole ecosystem so they partnered up with the with data breaks with Snowflake and with Adobe um and they um they really made fabric that one platform to host and collaborate with those others but they did a lot more than that they also realized that if fabric really needs to shine operation system for your data as that one one unified platform they microsofts is a buttl neck they cannot keep up keep up with all the ones that people have around data Quality Systems about process mapping about so many other things um but what they can do and what Microsoft do best is open up the platform for partners to start developing tools so they now opened up fabrick for all sorts of Partners to make their own tools inside a fabric wow so this is not about opening the data exposing it through API this is about actually giving developers the opportunity to create their own Solutions artifacts things in Fabric and it communicates with fabric but it also lifts inside Fabric and you can use it so just like you have third party visualizations in powerbi that lifts inside pure powerbi and you have applications for powerbi Monitoring Solutions and so forth you now are able able to add your your whole custom third party solution inside Fabric and manage it inside Fabric and manage it and right now this doesn't mean much because we don't have much many solutions but it means in the future we will see this whole range of as you can see on this slide here whole range of uh of workloads that will be entering Fabric and and I think this is just the beginning prediction here is a year from now we're probably going to see 100 or a thousand different sub Solutions being being um to the fabric platform great so we can start do even more than we can with fabric itself so so that was the the the developer workload we also are seeing user data functions which is a way for for any user or or or developer on fabric to actually create udfs user user defined functions here inside fabric uh which can be used by the different workloads you can use these functions in your Warehouse or in your pipelines or Etc so it's a neat way to to modularize and and improve your your Fabric and finally we have a graph I missed this news I didn't see how big of a news it was because I thought this was a new quick way to query our metadata in fabric but it's not it's a way for us to just one click and then our data warehouse or lake house is exposed through a graph graph C externally which means that by just going here and clicking and adding a lake house the next step for me is to add this code to my web solution yeah and the next step here is I suddenly have a web solution that when I interact with it as a user it fetches data from so you're actually interacting with fabric so actually interacting with fabric through web applications wow so now we have definitely bridged the gap or Microsoft has bridged the gap between the data of fabric and application development this is going to be huge yeah and finally in the last minute I know for this one this one is a brief little thing to end on we have something as boringly called as native execution engine for fabric runtime 1.2 is Apache Spar 3.4 it sounds just like a I updated the P Spar engine we're not going to go in technical details but what it is you see on the slide is it's like the road runner for for for for spark it makes an in in a lot of The Benchmark Cas it it makes it run up to four times faster or actually I think the the lowest of the workload was four times faster so four times or more faster without having to change anything in your platform beside clicking a little setting on and that's it so if you haven't already explored this and you missed it because it sound so boring check it out it's an amazing addition to spark that was our news that was all the stuff from buil I mean not all there's probably a lot left we didn't get a chance to there a lot that we I think we could speak a whole day about this but but this was the most essential things in one hour thank you so much for joining we look forward to seeing you next time and uh the next Fabric freny in two weeks yep and uh enjoy your work explore the features they're amazing have a great day yes bye e e