Hey everyone and welcome to Fabric Frenzy December edition. I was supposed to be here today with my excellent colleague Kevin who also joined me last time. He was supposed to be standing right here next to me. But unfortunately, even though Kevin was planning on being here, he was sick today so he couldn't join. So yeah, so we will have to manage without Kevin today. Sorry Kevin, hope you get better. But that also means that I could use your guys help so everyone tuning in on LinkedIn or on the broadcast, the 23 broadcast, please feel free to ask question right in the chat or everything and I'll try to monitor this and answer your question instead of answering. Kevin's questions this time. But as usual, we are going to go through the typical agenda. So it's a Christmas edition. This time, I'm wearing the nice, nice Christmas sweater too. But yeah, so we'll go through the news. We'll do feature spotlights on today. So let's get started. So the news that I want to talk about today is three news that I find quite interesting. Also some of the bigger announcements from the recent Microsoft Ignite. First feature is open mirroring. A cool feature which will allow us to write data directly from applications inside fabric. Other feature is the fabric databases, which are, well, it is what it says it is. It is databases, actual databases for Microsoft fabric. So that also opens up a whole world of possibilities. And finally, there is a, if you ask me, a little overseen feature and fabric. It's not exactly new. It's just released as GA, which means ready for production. Which is a kind of a plug and play API that we can put on top of our data and instantly have it available through an API interface. And I think that that is really powerful. So there's kind of a theme today. All of these three new features all center around the notion of moving fabric from being just an analytical platform to being an actual data platform for all data use cases. So we can suddenly do more with data than just making reports and just consuming these analytical KPIs and ad hoc analysis. Now we can actually do something that are crossing into the realm of application development, application usage, data products served with API and et cetera, et cetera, et cetera. So three cool features that kind of opens up this new feature of fabric being more than just an analytical platform turning into a full fledged data platform. So let's see if they hold up. The first feature is open mirroring and open mirroring is a flexible technology that will allow us to egress, to push data from any source that we have some sort of ownership over. So let's say we already developed an application and we want to make it easy for our colleagues to consume this application's data inside of fabric. What we typically would have to do to make this happen is we would have to set up some sort of ETL pipeline, set it on a schedule, perhaps even use the real-time tools in fabric to get the data on a second over second, minute over minute interval. But with the new open mirroring, we can simplify this a little bit. And if we are able to from our application to write a file with the new data or the data changes, well, then then fabric suddenly will, will keep that data updated and represented as a, as a Delta table. All happening automatically, as long as we can send that file to the right landing zone. So let's check out how it works in practice. So the idea is that we go to fabric and we create this new item as usual, new item to create an item. And in this case, we're searching for mirror, mirror database. It's this, this feature here, click that, give it some sort of a name. So let's call it mirror frenzy. And it is being provisioned. So what is now happening behind the scenes is that it's creating a few, a few, few technologies or items or services. So it's creating a data lake storage inside one lake. It's also creating a, what seems to be similar to either the fabric database or the warehouse. So as a SQL based, a storage representation. And it also creates a SQL endpoint and a semantic, default semantic model layer. So as soon as this data is loaded into fabric, it's instantly exposed and served through T -SQL and through the semantic model of fabric. That really simplifies pushing this data even further. So how it works is that this is, this is the location. So we need to send files here and they will automatically be caught and updated by this, by this mirroring UI. So for the sake of the demo, I don't actually have an application where we can send the data from, but we can do some, something similar. We can, we can cheat a little bit here and use a Parquet generator. So there are some formatting here that we need to know when we get started. So to get started here, we actually need this specific metadata file. We need to add this where we define the key columns and we need to do something else. We need to send the data to this landing zone in a specific format. So we need to use a Parquet file. It has to have a certain name and it has to have a certain metadata column being this row marker that then signals to the underlying engine. If it's an insert, an update, or if it's a delete or if it's an upset. So let's, let's see if we can make this work. So we needed a Parquet to begin with, and we'll just generate some random data here. Since we're getting started, look at the definition again, zero for insert. We'll just turn them all into zero. So this could be, oh, oh no. Let's, let's just add one more column for the sake of it because I accidentally clicked the button. Let's generate it. Um, and I can download this. And here is a Parquet file. So I can now go to this Parquet file. Go here. Rename it. There you go. And since I don't have an application to, uh, to, to push this from, I'm just going to go manually to my file explorer here and find the, find the, um, and the, we were. So mirror fancy here, it will have a file tap and a table tap with this landing zone here. And inside the landing zone, we need to create a folder for each new table. So we'll just call this. Frenzy demo. And inside this one, I'm just sending a, a Parquet file. So already here, data is there, but we don't have this metadata file definition file that needs to be there. So if all goes well here, the application, uh, or the service here will actually complain that it doesn't have the required data. But while it does that, let's just go and create this required metadata file. Um, so here we just create a very simple definition where we say the name is the key column. I mean, it's not, it's not actually a good key column in this case, but for the sake of the demo, it should work. So we store this as underscore metadata, which is, which is the name it needs to have. And we can then use this file to give the open mirroring engine, the metadata that it needs. As you can see here, it actually tried to ingest it and gave us a warning that it couldn't find this metadata, metadata, JSON file, and therefore gives an error in it. It cannot actually do the mirroring. But if we added metadata, JSON, we should be able to see data going in. And so far we created three, uh, three rows. So let's add just a few more, uh, samples here. So for this second file, we're going to try out some of the, the features. So, so, um, if we look at this one, we can say two is delete one is update and several was inside. So let's insert. So let's see what happens. If I try to delete this one, I try to update this one and this one will change this. So let's add this one. So one will be deleted, two will be changed and two will be added. And if we've done this the right way, we should hopefully end up with four columns. Sorry, four rows. Generating it. And while this is being generated and downloaded, let's go and see if the first one is working. And it is, um, at least it's running. Rose has not been replicated yet. That's interesting. Uh, there you go. So three columns from that first file. So we need to go and put the second file in there and see if we can make this update. So it needs to have a different name work. It needs to have the next number in this order here. And that's it. Hopefully here it should find the new file and it should automatically pick it up and then merge this new data into the replicated data. While we wait for this, let's talk a little bit about what are the use cases we could use this for. So the first one is if we are an application owner, or if we have an internal application and we just want an easy way to share this data inside of fabric. Um, well then, then you could possibly set up a quick way of exporting the data changes in near real time by using this, uh, this, this feature. Because as you saw before updating data takes less than a minute. And if we go back and look again, um, some here must have been, been made on my part because it actually ups inserted five new rows. Um, I guess something is wrong with the row marker, but either way tested it before going live here. It worked fine in the initial day demo. So I'm sure there's just a, a, a mistype somewhere and the demo guards are not, not with me today. But anyway, it, it, it, it should have, would have updated and returned with four rows. One being deleted to being, being updated and who being inserted. Um, so one use case is getting data from our own application inside of fabric. Now that another user usage is if we have an application and we don't necessarily need the data inside of fabric, but the fact that we can set up open mirroring and then quickly put on top something like GraphQL API, uh, that gives us a really, really, really easy way to create a prototype for, for serving this data outside of our application, uh, with an API. Um, equally so we get the SQL endpoints. So, so, so some user can query this data with SQL, um, outside of our application. Also gives us a possibility when we are designing a data platform or a data warehouse, we use this as an ingestion technique. So if we have some sort of a, a tricky source system, a tricky source system that doesn't have a native connector or, or it doesn't have a, um, an API that plays well with the rest of the tools and fabric, or some exotic, uh, data source that's just hard to get the data from. Well, if you can find a way to, uh, to, to load this data with a, a, a function, an Azure function, or a piece of code, turn it into Parquet and just send it to, uh, to an open mirroring, well, then you have a pretty quick real-time ingestion, uh, opportunity at your hand, uh, using this, this open mirroring. So I'm personally thinking about that this could be, uh, potentially combined with, uh, the, the, the BC2 ADLS for getting data outside of Business Central could be a possibility, uh, something experimenting with. Um, but, but, well, any, any system where, where we want a, uh, easy real-time way of getting data out, this isn't necessarily the solution, but it could be something to consider. Great. That was open, open mirroring. So let's jump to the next one, the fabric databases. And before we do so, let's just quickly check the chat. If anyone is asking questions so far, I'm not seeing it. And I do apologize if someone on LinkedIn is asking questions because I don't have them available or with me, but I'll make sure to go in and answer all those on LinkedIn after the fact, if there are questions there. Um, so let's move on to the fabric databases. So fabric databases is the next new thing in fabric. It is truly the, the, the new engine that changes fabric from being, um, just analytics. And suddenly now we actually have actual transactional databases. Databases with fabric databases. We've also been introduced to a new word. I have personally never heard before. It is the word of translitical workflows. So the, the, the hybrid or the bestest is if you will, between analytical workflows and transactional workflows. So we're talking about use cases where we're not just analyzing, querying big chunks of data, but, but that we're doing something that looks a little bit more like an application, but the purpose is still analytical workflows. So think something like, uh, stream lit, think something like, uh, power apps on top of a power BI report or power apps inside a power BI report. Well, those are the kind of use cases where the lines between, uh, operational workflows and analytical workflows, they tend to blur a little bit. And with the fabric databases, that's also now an option inside fabric to help with the, the proper storage engine and compute engine for workloads that are operational transactional in nature and not so much column store and analytical in nature. So I think it's really neat how Microsoft now is, is bridging the gap between, uh, people working with data only from an analytical point of view and people who are working with data from an operational point of view. And now are acknowledging that there are a ton of use cases in this intersection between the two. So let's, let's try out the fabric database and see if it works the way we would like it to. Okay. Great. So again, create a fabric database, like any other thing by going to the new button and finding the item. In this case, it is the SQL database. So this thing has been called fabric database, fabric SQL database and SQL database, um, in, in different blog posts, different areas. So, uh, which one will end up B. That's interesting to see. But for now, the official term of the UI is equal database in fabric. So it needs a name. We'll just do the same thing. SQL DB Frenzy. So that's it. That's all it took to create a SQL database that can host your application that can host some metadata config store for your analytical workloads, or it can host your, be power apps. It could be something else. Um, was a few click of a button, like click, click, give it a name, click, and that's it. We have a, a, a database in our hands. Now, if you come from a, from an application development background, I think the first concern we'll hear will be, I don't want to work with some web UI to, to develop my, my data. I mean, this is, this is kind of neat. There is an interface. I can go and write an ad hoc query, like write my select statement. Um, if I want to get started quickly, I could even with a click of a button, get a little bit of sample data imported inside my solution. It's nice to, to get started with. Um, yeah. And we can do things like, like using pre-written code templates, like for creating new schemas, table tables, view things that, that helps us. Get started easier. But the thing is, UI will always be kind of a limitation. And what we really want is of course, to connect to this database from wherever we're doing our app development. So we can, of course, connect to this new database from something like a SQL Server Management Studio. All we have to do is go fetch the connection string. See it here. We can use this in whatever application we are working with. Um, in this case, I'm just gonna want to use the open in SQL Server Management Studio button. And it just gives me the, the, the, the data I need. So I can go, add my server name here, extend the options. Go to database. Copy the database as well. Go and connect. Now, Fabric has a thing with naming databases based on semi-random GUIDs. So it's a little hard to keep track of and you will have to rely on, on your UI to help you make sense of what this object actually is. But, uh, but luckily that's something most other, uh, yeah, most IDs help with in some sense or another. But yeah, if we open up here, we can see the tables if they're already loaded. So our sample data we just loaded. If we had any views, they're also readily available here. We can start adding, uh, uh, stored, stored procedures. Um, well, and, and most of the things you would expect from a, from a, a cloud SQL database. So if you think like something like an Azure SQL database easily provisioned, now you have something similar, but inside Fabric and it's provisioned within just a few clicks. And it's all goes, uh, automatically inside the same billing engine of the capacity, which means that you actually don't have an extra cost just for spinning up this, uh, this, this database. So we can connect from, from SQL server management studio. We can connect from Azure data studio. We can connect from, well, the, the, all the other places we would expect to connect from, um, um, when we are working with, with databases. And finally, we also have the ability to create this new, uh, or sorry, create this, uh, uh, GraphQL API on top of our data, uh, on top of our tables to, to easily expose our data with API. Great. So one of the questions I've had, uh, had so far is how much does this service cost? The first question I would like to have answered is if there is a baseline cost. So if there is a constant cost of just having this created and, and open and, and even though it's not running any queries, just having it there, are there any overhead? And my investigations has shown that, and this is of course subject to change because it's still preview, but so far everything is built as, um, um, as just viable, uh, uh, consumption. Meaning that there's no overhead. If I don't send any queries to my database, it actually don't have any, uh, any consumption. And if I send queries its way, it's, it has a consumption. So it's, it's very much built in a, in a kind of a serverless manner, uh, so far. And so far it's also built as interactive, non-billable. And that will mean that, that if it keeps being interactive, we really should monitor that we don't hit our capacity unit percentage limit. Because very quickly we will see poor performance and we will see the system starting lacking. Um, this is different from how the warehouse works, where all queries in the warehouse is converted to background consumption, which means it's flattened and it's spread out over 24 hours. Um, it could be that the actual billing when it's, when it is billable and not built, non, not, uh, non-billable that, that it changes and becomes background. So I have no knowledge about, uh, about this, but as it is of now, it's counted as interactive, although it is non-billable because it's in preview. So, uh, today we can test it out, uh, for free. Another thing that you can see on this chart that I tried to experiment with a long running query. It was running for, for half an hour to an hour. I tried to scale up the capacity before running it again to check whether, uh, it would actually run faster with a higher tier of capacity. And from my experience, it seems that the performance is quite, uh, uh, quite fixed. So scaling up my capacity didn't actually give me any extra, uh, extra performance. My queries didn't run faster, which is, could be ideal because that means that no matter how small the capacity we have, we get the maximum amount of performance that we need. Um, but the downside is that if we need more performance that we actually get, well, then we may have to migrate outside of fabric and we may not have any levers to, uh, to turn on to, to solve this. But Hey, it's a preview feature. So that may very well come in the future. Great. Third and last feature. Uh, the API for GraphQL. So the API for GraphQL is probably the easiest way I've seen, uh, in data analytics on how we can set up an API on top of our analytical data. So all I have to do. Go inside my workspace, find an object that supports this, uh, GraphQL, uh, sorry, API for GraphQL. And so far I know it works for the, uh, um, the SQL database. It works for the mirrored, mirrored, uh, the open mirroring. I think it also works for the semantic models and there are possibly more, I would assume also the lake house, but I didn't test that yet. So if we just try to create one on top of the new database we just created, we can actually click here and create an API. Let's just call it API Frenzy. And I can choose the tables I want to use in this Frenzy API. So in this case, I'm going to go with the product description. Oh, this one product details. Yeah. Yeah. That's fine. Oh, wait, wait. Let's take the product description. Yeah. That's fine. And that's it. I have an API. I can start using it. So I can write a query here, like, like so, and I can query my product models and I can add a filter if I want to here. Where I say, what are some of the names? I don't remember this. Let's just skip to the pre-cooked one here. So you can do something like here, create a query, use products here, filter for the color, set some variable here. So it filters to color blue and you can actually run this query. And get data back. Now we don't just need data back in a UI in Fabric. What we actually need is to use this outside of Fabric. So with a single click here, I can click copy endpoint. And I actually have here my API endpoint that I can start sharing. So others can use this endpoint to fetch data from my backend using this endpoint. And if I'm not that code savvy or if I just I'm looking for a shortcut to get started even easier, I have this other button that says generate code. It just generates the code that I need to get started either in Python or in Node.js. And I can just use this as a starting point. It already sets up some token fetching for authorization. It sets up the endpoint. It has the query. And that's it. So easy way to get started with API. So with almost no time to go. Final thing of the day. World's quickest tip of the day. I want to tell you guys, the capacity metric app in Fabric is actually a flexible solution. A lot of you people out there actually assume that it's a fixed report and we cannot change it. But the truth is that if we just go and duplicate the report on top of the already existing datasets that's being created when we enable this monitor report, we can actually go and edit it. And quite quickly, we can do stuff like this was for me a two minute proof of concept prototype. Here, I added two new columns to the report because I wanted the percentage of the capacity that each individual job were taking. And I wanted the cost attribution of how much does this mean in actual money. And I could even duplicate the report page and simplify it by then now showing the metrics on workspace level instead of on item level. So this is actually quite flexible and is something we can move around with, play around with and adjust to our needs. That was what I had for you guys today. I just hope you all will enjoy the rest of December and have an amazing Christmas holidays with everyone around you. And then I hope to see you again in the next year in 2025. Yes.