Using Pebble Templates in WhereScape RED to Deal with Hard Deletes in an ODS Table.
In a recent YouTube video, we discussed how to use Pebble Templates in WhereScape RED to Deal with hard Deletes in an ODS Table
Giving an overview of WhereScape RED, and the benefits it has for you and your organisation.
Then delving into Data Stores and how we expect them to work, especially around Historic Data Stores.
Enabling you to store data and capture changes to your data in a historic Data Store, WhereScape RED is a great piece of software to do this.
Also, we discussed how we have created our FREE Pebble Template which can be run as a custom procedure after loading the Data into the Data Store.
Our Pebble Template has been designed to identify and end or update the DSS_CURRENT_FLAG and consequently update the DSS_END_DATE in line with the setting within the Data Store.
Data automation is fast replacing conventional methods for building and maintaining various centralised data repositories, which organisations use to deliver data-driven value and insights.
Businesses can generate vast amounts of data taking on many forms. Structured and unstructured data from CRMs, HR systems, social media and websites – these varying forms and sources of data need to be stored somewhere – this is where Data Automation can help you!
Start accelerating business success automatically with Data Automation.
What is Data Automation?
Traditionally, data pipelines to move or transform data have been manually scripted by developers. not having a development standard, these processes allowed for developers to use their own coding style to apply different methodologies, either to an individual pipeline or a set of pipelines. This made amending code or supporting issues difficult to read through, interpret or alter. This problem was then scaled significantly by the number of developers in a team.
In addition, this can lead to poor process performance where the technology is not used to its fullest extent – or cannot process everything demanded.
Yet, Data Automation eliminates these problems – making them a distant memory after the implementation of Data Automation. Data Automation is collecting, processing and presenting data using automated tools instead of performing these tasks manually. Manually updating data presents a risk of being delayed and creates an additional workload for your team to complete alongside their day-to-day responsibilities.
Replacing manually coded scripts with a low-code or no-code tool helps your team focus on processing and modelling the data without spending hours writing code. With minimal human intervention, Data Automation will collect, transform, store and analyse data using well-designed methods, software and, where applicable, Artificial Intelligence. Eliminating the reliance on manual labour with bots that will do the job for you.
In addition, code standards can be applied to the data processing scripts, creating a consistent and easy-to-support data platform. Gone are the days of flawed, manually coded scripts.
Data Automation tools automate three key elements of Data Automation. This is known as ETL or ELT, depending on what tool you are using:
Extract – Data is exported from source systems like social media, emails, SQL Servers, etc. Into a staging area. This data could consist of various types and be pulled from most structured or unstructured sources.
Transform – Changing the data to fit the purpose. For example, transforming the data into a Star Schema ready for further analysis by Visualisation or Business Intelligence tools.
Load – The data is moved from the staging area into storage. For example, a Data Warehouse or a Data Lake.
What Tools are Available for Data Automation?
With a plethora of Data Automation tools in the market, there is a significant amount to choose from; some are more efficient than others. Some do a good job, and some do an excellent job.
At Engaging Data, to be relevant to the needs of our customers, we only partner with specifically selected Data and Information partners. This approach helps our clients bring true value in realising and delivering their data-led business change journey. Our Partners include:
Microsoft – Take care of what’s important. Automate the rest. Do more with less by streamlining repetitive tasks and business processes – increasing efficiency and reducing cost.
WhereScape – Accelerate with Data Automation. Rapidly deliver a meaningful and future-proofed data platform.
Matillion – Unlock new levels of productivity and get data business-ready faster.
Snowflake – Join the cloud. Execute your most critical workloads in a fully managed platform that capitalises on the near-infinite resources of the cloud.
VaultSpeed – Delivering unparalleled automation of upfront data warehouse design and development. Fully compliant with Data Vault 2.0.
Why You Need Data Automation.
As previously mentioned, Data Automation helps improve productivity around the use of data within your organisation. Some other benefits influence improved Data Quality, simpler Data Cleaning and optimized Data Transformation.
Improved Data Quality –
As a result of datasets being standardized in accordance with formats and schemas, you can use Data Automation not detect and fix any records that don’t adhere to the format. Data Automation helps you improve data integrity and quickly detect errors.
Simpler Data Cleaning –
Data Cleaning transforms raw data, usually unformatted and unstandardised, into a significantly more suitable format for data analysis. Generally, the steps taken in Data Cleaning are repeatable, thus frameworks and standards for future datasets.
Automating these steps, using scripts and schedules to make new data match existing schemas and running steps on a recurring basis whenever new data is needed or imported.
Optimized Data Transformation –
The process of changing datasets from one format to another, Data transformation involves checking the datasets to ensure its in the correct format in the destination repository. Ensure that the data quality is good and checking the dataset follows the set standards for your organisation’s requirements or the data’s destination.
By automating Data transformation, it will help accelerate processes as you will need to manipulate, transform or analyse the data, and it is already in a standardized format.
Not only does Data Automation grant the large benefits discussed above, but Data Automation also allows for these general business benefits:
Enables Better Decision-Making – Businesses can react rapidly to the data’s narrative. Allowing for decisions to be made quickly with more data-led insights
Reduced Risk of Human Error – Manual data entry is prone to human errors that can cause Data Quality issues and affect reporting accuracy.
Improved Operation Efficiency – By setting up the rules and processes with software, you can easily audit and redefine these processes. You can quickly identify any broken or unnecessary workflows.
More Efficient Workforce – Teams no longer need to spend their precious time and advanced skillsets on mundane admin. Instead, they can add value elsewhere as their skill sets are appropriately optimized.
Reduces Operational Costs – You can get substantially more done with the same resources: minimal investment whilst maximising return.
Scalability – Automation software is ready to scale with your business, so you can utilise its benefits regardless of where your business is now and how much it grows in the future.
How Engaging Data can help You with Data Automation.
At Engaging Data, we understand that everyone needs data built efficiently to gain value quickly.
Using innovative Data Automation tools, we help you seamlessly integrate your data into accessible and secure platforms. Building data for a purpose, we only process relevant information to achieve your goals.
Do more with less effort.
Let us help you enable your teams to do more, eliminate human error and produce a higher standard of work with minimum effort.
Automate your data now.
Fill out the form below, and work with the experts behind Data Automation.
Have you ever needed to create high-level documents of your data automation that explains a project/sprint within your WhereScape Red repository? Maybe so it looks a little like the above?
We recently worked with a client who wanted to represent the amount of change undertaken within a single project. They required something simple yet demonstrated the amount of change within each layer of the data automation.
Instead of creating something new, we re-used the WhereScape RED Overview design, WhereScape used to illustrate the design of the architecture.
A sample data solution shaped into the high-level design.
Engaging Data consultants worked with the client to create a solid naming convention and development standards. With this foundation and the metadata repository, we developed a script that produced an HTML document with details the client was looking for.
The idea continued to develop and now has options to include the following details:
Number of Jobs, Procedures and Host Scripts that support each layer.
Data Volume per object and summarised per layer
Processing time for each layer, with avg completion time & avg time to run
WhereScape RED and 3D speeds up development & documentation of the technical design. This solution utilises the metadata to create support or narrative documents for other business areas.
Build re-usable scripts, dashboards or reports for non-technical business teams & provide clarity around the technical function of your automation.
If you are interested in receiving a copy of the script that produced this report, please email simon.meacher@engagingdata.co.uk
Modern commerce is an environment in which companies are increasingly being required to make complex, data-backed decisions. Dealing with vast amounts of information has become an essential feature of a business, which can often lead to siloed data. This is difficult enough to store, let alone analyse or understand.
In many cases, business use demands require a more sophisticated system, improving data management and providing a holistic overview of essential aspects of the company. One of the best ways to achieve this is to invest in a data warehouse. Yet, many companies are still unaware of what this entails or how it can help their business.
What is Data Warehousing?
In simple terms, a data warehouse is a system that helps an organisation aggregate data from multiple sources. Instead of experiencing the sort of separation and siloing discussed previously, data warehousing makes it possible to draw together information from disparate sources. It’s almost akin to a universal translator of languages. Typically, data warehouses store vast amounts of historical data, and this can then be utilised by engineers and business analysts as required.
Data warehousing is particularly valuable as it essentially provides joined-up information to a company or organisation. This was quite impossible until relatively recently, as data has always been based on separate sources of information. Transactional systems, relational databases, and operational databases are often held entirely separately, and it was almost unthinkable until recently that the data from the systems could be effectively combined.
But in this Information Age, companies are seeking a competitive advantage via the leveraging of information. By combining the vast amount of data generated together into one source, businesses can better understand and analyse key customer indicators, giving them a real insight into the determining factors of the company. Data warehousing can build more robust information systems from which businesses can make superior predictions and better business decisions.
In recent years, the escalation and popularisation of the cloud has changed the potential of data warehousing. Historically, it was more usual to have an on-premise solution, which would be designed and maintained by a company at its own physical location. But this is no longer necessary. Cloud data architecture makes it possible to data warehouse without hardware, while the cloud structure also makes implementation and scaling more feasible.
Data Lakes
However, those who are uninitiated in deep data topics may encounter terminology that can be somewhat baffling! The concept of a data lake seems rather surreal and tends to conjure up imagery that is, ultimately, completely useless! Inevitably, people who have never encountered the concept of data lakes before find themselves imagining an expanse of azure water glittering in the sunlight. Well, data lakes aren’t quite like that.
A data lake is used for storing any raw data that does not currently have an intended use case. It really can be seen as similar to the wine lakes that used to be in the news quite regularly, but it doesn’t seem to be a talking point any longer! You can equally view a data lake as a surplus of information; it is data that may become useful in the future but does not have an immediate usage at this point in time. Thus, it is stored away in a lake until it can be consumed adequately.
This differs from data warehousing, which is used to deal with information that is known to be useful more efficiently. Data warehousing may deal with data stored in an impenetrable format. Still, there is a clear use case for understanding this information, or it needs to be stored for a particular reason.
When to use a Data Warehouse
There are a variety of reasons that a company or organisation would choose to utilise a data warehouse. The most obvious would be as follows:
If you need to start a large amount of historical data in a central location.
If you require to analyse your web, mobile, CRM, and other applications together in a single place.
If you need more profound business analysis than it has been possible to deliver with traditional analytic tools, by querying and analysing data directly with SQL, for example.
In order to allow simultaneous access to a dataset for multiple people or groups.
Data warehousing makes it possible to implement a set of analytical questions that would be impossible to address with traditional data analytics tools. Collecting all of your data into one location and source makes it possible to run queries that would otherwise be completely unfeasible. Instead of asking an analytical program to continually run back and forth, back and forth between several locations, the software can get to grips with one data source and deliver efficient and more holistic results.
Data Warehouse Factors
Many businesses now require data warehousing services to deal with the vast amount of data that is now generated. And that ‘many businesses’ will rapidly become ‘most businesses’, and then ‘virtually all businesses in the near future. But those that are inexperienced in this field are often confused about what factors to take into consideration.
Thus, we would recommend looking at these six key elements when considering warehousing:
The sheer scale of data that you wish to store.
The type of information that you need to store in the warehouse.
The dynamic nature of your scaling requirements.
How fast you require any queries to be carried out.
Whether manual or automatic maintenance is required.
Any compatibility issues with your existing system.
Concerning the first of these factors, data can be somewhat different in terms of its basic structure. Some data may be highly complex, but it can still be quantifiable, easily organised. However, in the era of Big Data there is a vast amount of unstructured data, which cannot be easily managed and analysed. Companies that generate a vast amount of unstructured data and need to collate and understand it are certainly excellent candidates for a data warehousing solution.
There is a lot to learn when it comes to the subject of data. And it can frankly be a little daunting at times. But what is certain is that this topic isn’t going anywhere. Big Data is here to stay. That’s why we have created our Data Vault 2.0 solution. Data Vault can ideally serve your organisations’ data needs when this is becoming an issue of paramount importance.
Many companies are looking to make Code changes/deployment easier. Often the ability to deploy code to production is surrounded by red tape & audited control. If you don’t have this, count yourself lucky!
Jenkins & Octopus Deploy are two, to name a few (see here), that are helping to automate the deployment of code to production. Allowing companies to adopt a continuous deployment/delivery approach.
For a long time, WhereScape RED has had its own method of automating deployment, using the command line functions.
Why Automate?
Using tools such as WhereScape RED allow elements of automating deployments; however, we know that companies like to use a common toolset for their code deployments; like having a single picture of all the deployments and, in most cases, realise that they want to release multiple code deployments on different platforms because RED doesn’t do everything.
Git?
No problem! There are several ways to do this. Our perfered option is to push the deployment application to the code store respository. Afterall, it is more practical to store the changes you want to push to Production and not every change to any objects, including those that are not meant for Production!
Can I do This Now?
WhereScape RED uses a command prompt file to send commands to the admin EXE. All changes will be applied to the destination database (via ODBC). Installation settings/config is set using XML & a log file is created as part of the process. The XML file contains the DSN of the destination database. Let’s come back to this point later. The XML contains all of the settings that are applied when deploying the application. Settings like Alter or Recreate Job. Please make sure you have this set correctly. You do not want to re-create a Data Store table to lose the history!
Permissions are important. The key to running the command line to publish changes to production is that the service account executing the commands has permissions to change the underlying database.
Integration with Octopus
Octopus deploy uses PowerShell as it’s common execution code. So we have adapted all of our WhereScape BAT files to PowerShell in order to get everything working.
Building a list of repeatable tasks within Octopus is easy & provides an opportunities to create a standard release process that meets with your companies standards/processes. Tasks like database backup, metadata backup and much much more!
It can even run test scripts!
We used a PowerShell script to create a full backup of the database, to be used should the deployment fail. With a larger database, this may not always be the best solution. Depending on your environment set up you may have options to use OS snapshots or other methods to roll back the changes. The good news is Octopus Deploy works with most technology, so you should find something that works for your technology stack.
Recently, we been playing with creating rollback WhereScape applications on the target data warehouse. This is great for restoring the structure of the objects quickly and easily. Reducing risk is a great value add!
Go, Go, Go!
Triggering the deployment was easy, we could set this up in many ways, but used “does the application files exists” trigger to get things started – until the humans learned to trust the Octopus process.
However, linking the release to Jira is just as simple. Imagine, you’ve completed development and want to sent the code to UAT. You click the button to update the ticket…….wait a few seconds…..and the code is deployed! It’s complicated to set up, but you get the idea.
Final Thoughts
Octopus is a great tool and the automation really helps to control the process of deployments. Coupled with WhereScape automation, this provides and excellent end to end solution for data warehousing.
If you are interested in CI/CD and WhereScape RED/3D, book a call us and find out how it could help your team.
When you’re operating a modern-day data warehouse, documentation is simply part of the job. But it’s not necessarily the easiest or most logistically straightforward part of the process, while also being important. Documentation is, in fact, invaluable to the continued development, expansion, and enhancement of a data warehouse. It’s therefore important to understand everything that is entailed in adequately documenting, in order to ensure that your data warehouse processes run smoothly.
Understanding your Audience
One of the first things to understand is who you are compiling the documentation for. Support, developers, data visualisation experts, and business users could all be possible recipients. Before you answer this question, you really need to fully understand the way that your organisation operates, and open the lines of communication with the appropriate departments.
A two-way dialogue will be productive in this ongoing process. This process of communication will then help ensure that you keep the documents in line with the design. This is vitally important, as any conflicts here can render the whole process less than constructive than is ideal.
And it’s especially vital considering how fast documentation moves nowadays. Everything has gone online, and is based on Wiki. Whether it’s Confluence, SharePoint, or Teams, all sorts of Wiki documents are being produced by businesses with the intention of sharing important information. These shareable documents are updated with increasing regularity, meaning it is important to get your strategy in place before beginning.
Different approaches to data warehouse design can also impact the amount of time that a document is live before being updated. If you are lucky enough to make weekly changes to your data warehouse, you will be making incremental changes to the documentation itself. Development teams spend hours on updating the documentation rather than doing what they are good at….developing data solutions! Naturally, minimising this where possible is always preferable.
Self-Service Business Intelligence
Documentation is also crucial in self-service business intelligence. The integration of private and local data in this area, into existing reports, analyses or data models, requires accurate documentation. Data can be drawn in this area from Excel documents, flat files, or a variety of external sources.
By creating self-service functionality, business users can quickly integrate data into what can often be vital reports. Local data can even be used to extend the information delivered by data warehousing, which will limit the workload that is inevitably incumbent on data management. The pressure on business intelligence can be quite intense, so anything that lessens the load is certainly to be welcomed.
Another important aspect of documentation is that it reduces the number of questions that are typically directed at the IT and data warehousing teams. One thing anyone that works in IT knows only too intimately is the vast amount of pressure that can be heaped upon them by both internal and external enquiries. Again, anything that reduces this will certainly be favourable.
The data warehouse team also has huge responsibility within any organisation. They are required to produce a vast amount of information for front-end business users, and getting documentation right can certainly assist with this process.
Importance of Transparency
One important aspect of documentation that can sometimes be overlooked is the importance of transparency. This works on every level of an organisation, with the importance of sharing everything related to documents absolutely vital. Once this level of transparency is implemented, people who understand the data deeply can improve the documentation, or suggest changes to the Extract, Transform, and Load (ETL) and Extract, Load, and Transform (ELT), if this is indeed deemed necessary.
Conversely, it’s also important to understand that not all technology is suitable for documentation. As much as businesses and organisations would love this process to be completely holistic, this is not always possible.
Thus, packages such as Power BI, QlikView and QlikSense, and even Microsoft’s trusty Excel, are not necessarily ready to be documented. These software packages can use data, but often do not have the ability to provide a document set that explain how the data is being used, and for what purpose. Recently, Power BI has taken steps to ensure that the app can help with data lineage, but this remains better suited to IT teams, as opposed to Business Users.
Attempting to document data across multiple technologies is tricky, but Wikis can provide IT teams with the ability to collate all of this information into a central hub of knowledge, making access much more logistically convenient.
Conclusion
Ultimately, IT departments, data warehousing teams, and report developers should all be encouraged to produce documentation that contributes to the overall aims of their organisations. Anything excessively technical is not good enough for modern business requirements, especially considering the importance of communication, and of ensuring that everyone within an organisation is acquainted with as much vital data as possible.
Modern-day technology makes this goal a reality, and this means that it is increasingly an expectation of end-users. Failing to prepare properly in this area could indeed mean preparing to fail, as organisations will simply have failed to meet the compelling desires of the market. It is therefore vital for documentation to be dealt with diligently.
Getting this piece right, will go a long way to help with data governance!
If you would like to know more about how Engaging Data help companies to automate documentation, please contact us on the below.
