Many of our D365 F&O customers reach the limits of the internal system reporting capability in short order, and then look for a mechanism to extract data from D365 to support richer, more intuitive reporting in a tool like Power BI, often underpinned by a data warehouse combining data from multiple heterogenous data sources.
For many years the standard approach to extracting data from D365F&O to support external reporting and analytics used the Data Management Framework and an external Azure SQL Database in a framework called “BYOD” (Bring your own database). During most of our D365F&O implementations, this framework would be either be used to support integration of data into a data warehouse, or as the basis for reporting in a BI tool like Power BI.
At the end of 2021, Microsoft released into General Availability a new extract mechanism that can be used by organisations to markedly increase their capabilities to deliver business insights to key stakeholders; Azure Data Lake Export.
The release of this functionality has prompted a number of questions from our customers about the value that the Data Lake could bring in supporting organisational data analytics. This blog is intended to provide a summary of the benefits we see our customers realising by incorporating Azure Data Lake into their data landscape, either as a transition from BYOD or as the core of the data management capability on new implementations.
Data Availability
A key constraint to the previous mechanism for data extract (BYOD) was that data was curated into ‘Data Entities’ within D365 which abstracted the base tables in D365 to ‘views’ for export. This approach restricted the data that was available in the system with not all base tables or columns being included in the available entities. While the data entities could be customised through development, this becomes a hinderance from a time and cost perspective.
With Azure Data Lake Export, we can export the base tables to the data lake, meaning significantly more data is available for users to discover. This puts a data discovery into the hands of the BI team and/or users that was previously not available.
Data Latency
With a trickle feed out to the Azure Data Lake, data is now more near real time than ever. Where previously a batch overnight refresh was the norm, with potentially a small number of incremental loads, the Data Lake provides the capability for intra day report updates based on data from D365.
Ease of Setup and Reliability
BYOD has been notoriously difficult to setup and manage. The requisite data entity publishing, change data capture settings, data project setup, batch and schedule setups were all configurable by users and could potentially clash. Significant load failures put a strain on the adoption of BI based on D365 data. A single click to activate a table feed out to the data lake and the removal of any batch or schedule maintenance means the risk of failure is massively reduced, a key benefit in gaining trust in the data & analytics platform.
Development Cost Reduction
The associated benefit of Data Availability is that it removes the need to customise the D365 data entities. Customisation of data entities has historically been a significant cost to our customers when new data is required within the data & analytics platform. The advent of Data Lake export has removed the development cost away from D365 across to the BI team with significant cost savings already being realised by our customers.
Support for Advanced Use Cases
The Data Availability and Data Latency benefits detailed above, along with the ability for Azure Data Lake to serve data to users in both raw and curated form, has unlocked several advanced analytics architectures for D365 data analytics. With more data available quicker, Data Scientists can leverage the capability of the data lake, and architectures like the Data Lakehouse, to use raw, cleaned, or curated data for advanced analytics and AI/ML.
