How is CI/CD (Continuous Integration / Continuous Delivery) Used to Modernise a Data Warehouse? 

How is CI/CD (Continuous Integration / Continuous Delivery) Used to Modernise a Data Warehouse? 



.

How is CI/CD (Continuous Integration / Continuous Delivery) Used to Modernise a Data Warehouse? 


A specially designed type of data management, a data warehouse is a system that has been built to enable and support business intelligence and analytics. For example, a data warehouse will read large amounts of data to understand relationships and trends within an organisation.  

Data warehouses contain large amounts of historical data and are intended to perform queries and carry out analysis. The data within a data warehouse is often derived from various sources, such as application log files and transaction details. You’re centralising and consolidating large amounts of data into one location by utilising a data warehouse. Over time, this data can be invaluable to data scientists and business analysts, allowing informed business decisions to be made using valuable business data insights. As this data record builds, a data warehouse is often considered an organisation’s single source of truth.  

Analytics and data have become indispensable to allow businesses to stay competitive, and companies rely on reports, dashboards, and analytic tools to extract data, monitor business performance, and support future business decisions. The power behind these processes are data warehouses, which store data efficiently to minimise the input and output of data and deliver query results quickly.  


What’s the Architecture of a Data Warehouse?  

A data warehouse architecture is created in tiers. The top tier is the front-end, where results are presented through reporting, analysis, and data mining. The middle tier is where the analytics engine used to access and analyse data sits. The bottom tier of the architecture is the database server, where data is loaded and stored.  

Data is stored in two types of ways:  

  1. Data accessed regularly is stored in fast storage, like an SSD drive.  
  2. Whilst data that is less frequently accessed is stored in an object store.  

The data warehouse is competent at differentiating between the two types of data. It will automatically ensure that frequently accessed data is available in the fast storage option, thereby optimising query speed.  

So, how does a Data Warehouse work?

A data warehouse might contain multiple databases, and within each database, the data will be organised into tables and columns. Within each column, you can then define a description of the data, for example, integer, string, or data field. Tables can be organised within schemes, using a similar structure to files and folders. So, data is added to a data warehouse, it’s then collected and stored into various tables defined by the schema, and query tools utilise the schema to identify which data tables to access and analyse.  

What are the benefits of using a Data Warehouse?

As we mentioned above, there are multiple benefits to using a data warehouse, for example, making more informed business decisions based on data and insight. Here are some of the additional benefits of using a data warehouse:  

  • Consolidated data from many sources 
  • Historical data analysis 
  • Data quality, consistency, and accuracy 
  • Separation of analytics processing from transactional databases and improves the performance of both systems 

What’s a Modern Data Warehouse?

Across your organisation, multiple teams and users will have different needs for a data warehouse. Traditional data warehousing can’t always keep up with the demands of rapidly growing volumes of data, processing workloads and analysing data. In contrast, a modern data warehouse architecture addresses the different needs of your organisation by providing a way to manage everything you need with various integrated components that all work together. 

A modern data warehouse would typically include:  

  • A streamlined database that simplifies the management of all data types and provides different ways to use data across your organisation 
  • Self-service data ingestion and transformation services  
  • Support for SQL, machine learning, graph, and spatial processing 
  • Multiple analytics options that make it easy to use data without moving it 
  • Automated management for simple provisioning, scaling, and administration 

Ultimately, a modern data warehouse can efficiently streamline data workflows in a way that traditional data warehouses can’t. This allows everyone to perform their jobs more effectively and efficiently, from analysts and data engineers to IT teams and data scientists.  

If you have a lot of data, and multiple teams that all need to access it, then modernising your data warehouse should be a key component of your data strategy. But how can you update your data warehouse? This is where CI/CD, which stands for Continuous Integration (CI) and Continuous Delivery (CD), comes into play. CI/CD creates a faster, more precise, and overall efficient way of combining the work of multiple teams into one streamlined product. For example, in app development and operations (DevOps), CI/CD would streamline coding, testing, and deployment by creating a single space for storing work and tools, thereby consistently combining and testing code to ensure it works.  


What is the CI/CD pipeline, and how can it support modernising a Data Warehouse?  

By utilising a CI/CD pipeline, any software development or engineering processes that combine automated code building with testing and deployment, you can deploy new and updated software safely and precisely.  

In other words, a CI/CD pipeline is the behind-the-scenes plumbing of your data and analytics, making your life easier and your work more consistent. Here at Engaging Data, we have developed a solution that integrates WhereScape 3D & RED with CI/CD and DevOps pipelines in one ecosystem to modernise the world data warehousing. If you’d like to know more about our CI/CD pipeline, take a look here.  

Because a CI/CD pipeline isn’t just a linear process, it allows DevOps teams to write code, integrate it, test, deliver updates & releases, and make changes to the software in real time. In addition, the ability to automate critical parts of the CI/CD pipeline allows development teams to work more efficiently and more effectively and improve other DevOps metrics.  

What are the benefits of the CI/CD pipeline in modernising a Data Warehouse? 

The most significant benefit of a CI/CD pipeline is the automation of releases from the initial testing to deployment. Additional benefits of the CI/CD pipeline for DevOps include:  

  • Automated testing makes the development time more efficient; CD and automation mean that a developer’s changes to a cloud application could go live within minutes.  
  • Thanks to faster, more efficient testing and development, less time is needed to be spent in the development phase, therefore reducing cost.  
  • The CI/CD pipeline is a continuous code, test, and deploy cycle. Every time code is tested, developers can react to feedback and improve the code.  
  • A CI/CD pipeline allows a more collaborative and integrated process with everyone across the organisation who needs to access the data warehouse.   

If you have any questions about a CI/CD pipeline or the deployment process, we have diagrams of the flow here.  


Conclusion

Overall, a modern data warehouse utilising CI/CD should be an essential part of your data strategy if your organisation has multiple touchpoints requiring access to data, insight, and analytics.  

Do you have any questions about modernising your data warehouse? Or about creating and implementing a CI/CD pipeline? Our expert team of data specialists can implement a modern data warehouse for your organisation.

Fill out our form below, and one of our team will be in touch.  


The Gold Standard – A Comprehensive Guide

The Gold Standard – A Comprehensive Guide

Engaging Data Explains :

The Gold Standards in Data –

A Comprehensive Guide


Achieving an excellent level of data architecture is far from easy. But it is certainly possible if you implement certain guiding principles. Central to this is the implementation of a gold standard benchmark, which can then underpin any effective data architecture operation.

However, a gold standard is not something that comes naturally. In our experience, it requires diligent thought, effort and openness to change.

Within this blog post we’re going to discuss some of the considerations related to this important goal for many organisations.


Resource Considerations

When we think about creating engaging analytics or data platforms to shape the growth of an organisation, the focus is often on the finding tool capable of developing the solution and not the surrounding aspects. But all of the ingredients that go into the mix are critically important.

Imagine you’re the owner of a cake shop providing bespoke cakes for your customers. Your products have to be good enough to keep customers coming back, but they also have to retail at an attractive price point. This means that there are immediately resource considerations.

You may choose to focus on providing a premium product, creating high-quality goods for a premium price.  Alternatively, you may deliver a higher volume product, baking lots of different cakes on a larger scale, which are still of a good standard, but only suitable for a lower price point.

In order to make this decision, you need to understand the following:

  • Product – what we are providing, and the value that we create.
  • Place – the environment that makes it possible to create the necessary standard of products at a sustainable rate.
  • People and Process – the team, the processes and the delivery environment (the bakery, the storage, the front of house, stock control, delivery, billing, etc.) that produce and maintain the consistent quality of product and experience.

If you can put all this together then you have the beginning of a gold standard in cake production. By the same token, in our field Engaging Data helps companies to review all of the data elements supporting such a setup, combining this with their aspirations to form bespoke gold standards. This enables our clients to achieve profitability and success.

Controlling The Input

Controlling input is a critical component of processing gold standard products. These vary, depending on what you are trying to produce, but examples include:

  • Requirement gathering.
  • Sources of data.
  • Quality of data.

Controlling inputs and creating quality is critically important, as if you put terrible into a system then the ultimate outcome will be a terrible product! Thus, you need to understand the requirements of your customers. In the cake shop example, this would mean knowing what type of cakes your customers desire, the toppings needed, the date and time of delivery, any dietary requirements, and so on.

The good news is that controls can be quite straightforward. They can be something as simple as checking data. So in a cake shop, it’s vital to confirm the direct requirements of your customers, noting down all relevant information. This can make the vital difference between providing the ideal products for your customers, or producing something that seems excellent, but is rendered useless or sub-par by one important constituent. For example, you might produce a cake for someone with allergy needs that is simply inedible from their perspective.

Quality control can be achieved by creating a simple order form. For example, a cake shop might include:

  • All vital information being distilled into yes/no questions – eg. “should cake contain nuts?”.
  • Ensuring that all product types are selected, and that nothing out of the unusual is ordered.
  • Product limitations being noted expressly on the form – acting as a reminder and preventing incorrect ordering.

Such a review process ensures that information is gathered correctly, and creates a collective responsibility for discerning the appropriate information. Important questions that you can ask yourself in a data environment to acquire such critical information include the following:

  • How will the requirements come into the team? 
  • How do we need to record them?
  • Do we have the right tools to collect the data?
  • How will we handle data quality?

Output Consistency

The output is the result of your efforts, so you have an innate interest in ensuring that it’s the best possible product. In common with the input, it is important to understand what you can control to reduce risk, as this can have a big impact on your output.

Central to this process is building systems and controls that enable you to monitor outputs. This in turn makes it possible to assess if they need to be altered in any way. This means that in a cake shop, you may consider the impact that each of the following areas has on the supply chain of cakes:

  • Production Team (bakers, shop front, etc.).
  • Ensuring similar standards and experience.
  • Providing the same customer experience.
  • Ensuring knowledgeability about the production processes, industry and competitors.

Each aspect of the order and production process also needs to be assessed and standardised:

  • Enjoyable and consistent ordering experience.
  • Stock control to manage high-quality ingredients. 
  • Quality control of all products.
  • Meeting all food hygiene regulations with a 5-star rating.

And then the tools of the trade should also be taken into consideration, as part of an ongoing auditing process. Central to this is ensuring that any equipment being used is within acceptable operational parameters, particularly not being overloaded or overstretched in any way.

So when you’re working in a data environment, or any working context, if you want to create gold standards then it’s important to continually monitor and challenge your processes. Ask yourself questions continually, such as:

  • Do we have the right team in place?
  • Do we understand what standard of products that we need to create?
  • Do we have processes in place that enable us to produce quality products?

Reviewing is never the most exciting nor relished procedure! It often feels like an unnecessary grind that slows down your whole operation. But it’s actually a critical part of any good business, enabling you to identify major faults before they become ingrained.

Reviewing Processes

We’ve already seen in the first part of our gold standards blog post that the cake shop reviewed their existing processes before making a decision to change their order form. Often it’s only from self-assessing in this way that you can uncover new ways of working that enhance your productivity and efficiency.

It’s also important to emphasise that this needs to happen across the business. It’s normal to have a review as part of project governance for new projects, but it’s always worthwhile to reassess existing projects as well. They may be ‘good enough’, but they also might not be reaching the gold standard.

So in our cake shop scenario, how will the business assess the standard of the cakes being produced? Well, when the business reviewed the kitchen they found that each cook bakes one order, while also being responsible for checking their own cakes before they move to the decoration stage. This all seemed fine, but then the retailer received a few complaints about burnt edges and sub-par ingredients. Consequently, the cake shop reflected that its existing process of ‘marking your own homework’ was not sufficiently robust to identify problems.

In order to address this issue, several ways of reviewing the production of cakes were decided upon:

  • A simple visual inspection – does the cake look uncooked or overcooked?
  • A thorough test, such as breaking the centre of the cake to see if it is cooked. 
  • Pressing the centre of the cake to see if it springs back.
  • Hiring Paul Hollywood as a tester!

All of these checks are designed to see if the cake has been cooked satisfactorily. The shop decided to opt for all options, with the exception of hiring Paul Hollywood! Instead, each baker will review one another’s baking, with the hope that the business will grow to support a head baker who will review all cakes.

Just as the cake shop reviewed its processes to put a more stringent review structure in place, the same can also be implemented in a data-driven environment. The following questions are examples of some that you can ask yourself as part of this process:

  • Does the team need training?
  • Do you need to recruit new people with different skill sets?
  • Do the products need to change?
  • Is the supply line quick enough? 
  • Would more people or different processes help with efficiency?
  • Is the product still worth the effort that is invested in producing it?

Producing Gold

Once you’ve baked some quality cakes, you then need to take steps to market the product. It’s not enough to just produce the cakes and leave your customers to eat them if you want to maximise your marketing efforts. Building advocacy and influence via your customers is a great way of marketing your cakes to the right people. Word-of-mouth feedback from advocates is trusted by other potential customers, and is therefore far more effective than other forms of marketing.

However, there are two sides to the coin here. Negative feedback can be dangerous if it’s not managed effectively. Negative comments about the burnt edges of cakes will spread like wildfire to existing and prospective customers. But there are ways of recovering from this. Making courtesy calls to customers can provide you valuable insight into the process of ordering and consumption. 

Getting the right team in place, tailoring products for your target market, and taking feedback onboard in an active process are all important facets of contemporary marketing. 

What are the Considerations?

Gold Standards always begin with what you are looking to deliver and who this will benefit. Gold Standards should be designed to support outcomes, having considered both the internal and external factors that will influence design. Creating steps in the process to continually challenge the functionality of the end product and ensure that standards are still relevant to the end user should therefore be considered essential.

Sponsors and influencers can also play an extremely important role. Both can become prime advocates of your product, with the added benefit with sponsors that they pay you to advertise your goods or service!

Internal Considerations

Data

Data can be compared to ingredients within a cake. Naturally, good quality ingredients are critical to producing the best cake possible. The same applies to data. As we mentioned in previously, if you put rubbish data into your systems, you can expect rubbish outcomes!

At some point, the cake shop company realised that the bakers are not periodically reviewing the ingredients within their cupboards. To address this, the manager inspects everything that they have on hand, ensuring that any poor quality ingredients are replaced, and that anything out of data is thrown away. Labelling is updated, while processes are put in place to ensure that there is no repetition of these mistakes.

The key point here is that while complaints were registered about burnt edges, it may have been the ingredients that contributed to the final product that were the problem. Going forward, the team at the bakery put in place a series of key questions that would inform their processes in future baking:

  • Do we have enough data to make the size of cake required?
  • Are we getting our ingredients from the right suppliers? 
  • Does this product contain nuts?
  • Have I mixed sugar up with salt?
  • The milk smells as if it’s on the turn, should I use it?
  • Do we store the ingredients in the right place, in the correct containers? 

Resources and Teams

When baking your cake, you can select from many different types of bakers or specialist chefs to assist with the process. Or you may decide that you wish to train yourself, or an existing employee, so that they can handle the most challenging baking tasks.

In some cases, if the cake shop utilises industrial equipment, people who have been trained to use this equipment can be deployed, as opposed to bakers or specialist cake makers. Having the right team with the right skills, and/or the aptitude to learn them, can be critical to successfully achieving gold standards. Instilling this in your team culturally is critically important in providing direction to your whole operation.

Achieving this can be as simple as asking yourself the following questions:

  • Is the team right to build the end product? If not, what needs to change?
  • Is the team open to changing or evolving in order to improve the product or efficiency?
  • Do we have the right skills? If not, do we need to second or buy them in?

Company Culture

Finally, failing to understand the company culture will lead to failure. Gold standards must fit into the existing culture, or the direction the company is moving towards.

Understanding how your customers think, managing their expectations and developing a standard to consistently perform to those expectations is simple to conceive. However, the human element of this could result in you developing hundreds of different gold standards for multiple different customers. 

Important questions to ask yourself here:

  • Are the customers knowledgeable about your products? If not, can you educate them?
  • Do you share your practice? Would it help your customers to know what you do and how you do it?
  • Are there any expectations that you can manage? 
  • Are they any difficult expectations you have to work towards?

Implementing a gold standard for data may seem like an all-encompassing and intimidating goal. But it instead should be seen as a granular process. Breaking down the ingredients and individual components that collectively create gold standards is the best way to achieve this aim.


We have juxtaposed the process of dealing with data with the running of a cake shop. And when you’re developing the ideal processes for either type of business, simplicity is critically important.

But ‘process’ also refers to several aspects of the business. It can refer to the discussion of new orders with customers. It can be related to the process involved; the time it may take to bake and distribute cakes. It can also be related to the management of customer expectations and orders.

The process involved with the business can help your team understand what they do and how they fit in, as well as playing a vital role in deciphering and organising team responsibilities, interactions and achieving gold standards. It means that even if team members aren’t specialists in a particular area, they can still refer to established processes and be aware that they will result in a strong final product. For example, if a baker doesn’t know how to decorate a cake, they can understand that it can be decorated, and potentially the time that this will take. 

Understanding The Process

A good rule of thumb is to aim for the production team to understand enough about the process that they can make accurate estimates on production schedules. This demonstrates how much you can produce in one week if everyone works at 100% efficiency. And it also offers details of the effort needed to create the cake, and therefore the return or profit derived from the product. 

Once you have a good understanding of customer demand and production time, you can better manage cost and resourcing, potentially enabling you to offer alternatives to customers who have particular requirements. For example, customer demand may require the cake shop to produce 1,000 cakes in a week instead of just 100. By developing a process around the type of cake shop the establishment wishes to be in the local market, the orders can be successfully fulfilled.

In order to address this, the cake shop conducted some research on the market, reviewing the orders received over the last 12 months. Based on this data, they decided to optimise their operation so that they produced a consistent standard of high quality, mid-price range cakes using the best quality local produce. It is anticipated that this will constitute 80% of the company’s orders. Alongside this, some premium cakes will also be produced, with the intention of satisfying exclusive clientele, after the manager of the store noticed that this sort of cake tends to produce positive social media feedback.

With this in mind, the business decided to create a unified process, enabling her staff to process both types of orders using the same equipment. Different ingredients and alterations in the production process were required for the bakers to produce premium cakes, consequently more effort and attention to detail was needed. Furthermore, separate ingredient storage and labelling – to avoid premium ingredients being used within the standard cake – was also deemed essential.

Whether making cakes en masse to sell to local shops, or an artisan cake shop producing a range of bespoke, high quality and expensive cakes, each manufacturer will develop its own process to suit the desired output.

Questions you can ask yourself:

  • Do we understand what cake or range of cakes that we are trying to produce?
  • Can you easily explain your process to a customer, in terms they understand?
  • Is the manufacturing process suitable for the product/product range?
  • Do you understand the running costs involved with making more cakes?

Tools and Delivery

Producing gold standard cakes is not as simply as merely building a process or hiring the best people. In some cases, you can be limited by your tools. Realistically, there may be budgetary restraints that prevent you from purchasing the appropriate equipment for the job. So it can be best to utilise existing tools, but highlight them within the process as a potential risk to production. This will help with any customer discussions, particularly understanding any impact on production costs.

The new process at the cake shop enabled existing equipment, including the all-important ovens, to be fully utilised in the production of the desired range of cakes. As a result, business is soon booming and there is a large backlog of orders.

After assessing the production process and resources, the manager of the bakery decides that the most cost-effective way of increasing production is simply to add another oven within the production line. The additional oven enables the same staff and process to double the number of cakes that can be produced, while retaining the same processing time. The initial purchase price and ongoing maintenance costs are easily met by the increased revenue.

Questions you can ask yourself:

  • Do my tools enable me to produce our products effectively and efficiently?
  • What limitations do we have?
  • Do we utilise our tools correctly?
  • Do our production process utilise our tools in the best way possible?
  • Are there any other tools that can help me, either by saving resources or production time?

Time

Production time is often hidden from customers. All of your fantastic looking cakes in the shop window. or publicised on social media, take time to produce. Yet customers rarely appreciate the intricacy of the production process. Nor are they usually willing to wait! Therefore, managing expectations around deadlines from the very minute that any customer orders a cake will either address any concerns, which is why it’s always better to talk about this issue first.

Sometimes the thinking or desires of customers can change at the eleventh hour. On one occasion, the cake shop received a call from a customer late on Friday evening. At the last minute, Mr Jones had remembered that he needed to order a cake, so that it was ready for tomorrow’s big party. As one of the company’s bakers completed the order form, he realised that this order may not be possible, based on the existing orders that are already due to be baked tomorrow.  After a quick check with the kitchen staff, he confirms that the order cannot be satisfied, and subsequently informs Mr. Jones. The customer was disappointed, but understood after the kitchen capacity and pre-existing orders were explained to him.

Being transparent with customers and influencers about existing orders and production processes will help any business, but these processes will never stop unexpected orders from appearing at 8pm on a Friday night. However, it does create a consistent customer experience and help companies to mitigate any associated disappointment. Communication can be critical when all other approaches fall short.

Creating gold standards in data, or any business, is always dependent on efficient processes being put in place. These processes encompass every aspect of dealing with internal and external factors, and consequently it is critical for companies to continually refine every aspect of their operation.


Examining the importance of external considerations, these are factors inherent to the industry and location of your business that impact directly on your company and its day-to-day functioning.

Firstly, every industry has its own regulatory body and best practices that influence the way you work. In recent years, laws around the use of data (GDPR) and how it can be stored have evolved, with the aim of better protecting personal data.

To return to our cake shop example, the Food Safety Act 1990 provides the framework for all food legislation in England, Wales and Scotland. The legislation is concerned with, among other things, ensuring that food businesses take responsibility for how food is labelled, advertised and presented, so that it is neither false or misleading.

Laws and regulations therefore directly impact the processes, people and technology used to create cakes, while also potentially increasing production costs.

Questions you can ask yourself:

  • What regulatory bodies impact my cake shop, and how does this impact on the gold standard?
  • Are there any laws that may impact my cake production?
  • What if a new law or regulation arises, am I prepared for this? 

Economic (Demand drivers)

All customers have different demands; they require a range of products at varying price points. As we have discussed previously, the product price is dependent on the cost of production, however this is also influenced by the economic strength of the country in which the product is created.

Production Costs

The economic status of a country, whether recession or prosperity, has a direct impact on the cost of production and the purchasing habits of customers. Furthermore, various countries can produce the same products at different prices, while the strength of currencies can also impact the global market.

If a country is in recession, customers may shift from premium to mid-range products, accepting that the quality may diminish. Consumers may also purchase less frequently, or even stop buying entirely.

Here are some simple examples of economic impacts on the cake shop:

  • Recession may lead to using low-quality ingredients, or reducing the bakers that manufacture the products.
  • When the market is growing, this may lead to higher demand and / or production, or to an increased quality of ingredients.

The economy of a native country will drive the demand for cakes. This change in behaviour is something that cannot be controlled, but planning can be put in place. Building flexible and adaptable processes can enable your cake production to shift with demand, even though achieving this can be challenging. But identifying trends in demand is key to adapting efficiently and keeping your customers happy.

Questions you can ask yourself:

  • How can you detect changes in customer demand?
  • Are your gold standards set up for change? Can they adapt?
  • Are you and your team prepared for change?
  • Can your infrastructure cope with change?

Technology (Levers for improvement)

Technologies used in cake production are varied, but they must always support the processes and people involved in manufacturing.

Within the cake shop, the oven could be considered a bottleneck. If you have the space for more ovens then you can scale up your operation. And if there is an oven that can cook the same amount of cakes in half the time, this may be a better solution to the bottleneck.

Each item of technology within your process has its own limitations and potential. Understanding this will help you adapt to any situation that you encounter.

Questions you can ask yourself:

  • Is there a warranty on the ovens that we use?
  • Do I licence the equipment that we use, or do I own the equipment?
  • Do I understand the lifespan of our equipment?

Ongoing Delivery

The job can never be considered done, especially for data warehouses, particularly if data is used to make business decisions about customers, or to support business change.

Once the initial gold standard has been created, both periodic and failure reviews are recommended to ensure that standards continue to produce the value expected. These reviews can start at any point of the production process, but should be thorough and constructive, assessing all key elements of the people, processes and technology.

These reviews should be consistent in assessing elements against the following criteria:

  • Does the Process still fit – did anything about the process slow down the delivery?
  • What was the customer experience – did you create a good news story? Did the customer get what they wanted? Did this add value above what was expected?
  • Best Use of Technology – did the technology let you down?
  • Best use of People – were there any gaps in knowledge or skills, or impacts on customer experience?

Ask the difficult questions rather than reviewing the successes and you’ll have a much better chance of improving your operation.

Internal Considerations

As well as considering internal outcomes, there are also internal factors that will inevitably impact on the production process. Once production is up and running, there are usually support functions that are also part of the process, which are separate from the production team. These functions will help keep everything running smoothly. 

When the cake shop initially began trading, bakers were responsible for ensuring that the company’s machinery was in good working order and fit for purpose. This was a collective responsibility, but the bakers decided amongst themselves that one particular employee, Jerry, would carry out the maintenance, as he restored classic cars in his spare time and therefore already had an advanced skillset.

This was fine for some time. But when Jerry was out of the country on holiday, the food mixer broke down at the least opportune moment; right at the start of a large order. Jerry had the most experience and knowledge of this machine, with the other bakers having relied on him in these scenarios. In order to ensure that the order was fulfilled, the manager of the shop took the decision to purchase smaller food mixers, and pay the bakers overtime. The order was completed, and in the aftermath the management team reviewed the root cause of the delay in processing.

Subsequently, the decision was made to use an external maintenance company to monitor and maintain kitchen equipment. The shop’s equipment would be checked every month, to ensure that everything is in good working condition. In addition, the maintenance team will also be on call should something break during normal production hours.

The maintenance company ensures that all engineers have the relevant training and certifications to work on the machinery. Although the bakers may know enough to fix the machinery, it is not their job to deal with this, enabling them to focus solely on production.

But the shop continues to consult with the bakers when purchasing new equipment, as they keep abreast with the technology trends in the industry. This helps best practices be maintained.

Questions you can ask yourself:

  • Are roles and responsibilities suitable for your process?
  • Do you have any key risks?
  • Does your support team have the right experience?
  • Are there any service level agreements with the support team?
  • Does the support team add to the overall process?

Testing

Developing a testing method is another important internal consideration, yet it involves a simple process. Agile development is key here – revealing important data insight as it’s being developed. This constant checking builds certainty and comfort with business users, as they approve the state of data during development. This approach breaks down testing time and, in some cases, can reduce the overall development time. 

However, not all businesses can adopt this approach. Company policies, laws and regulatory requirements may impact the options available, and will be reflected within the internal process. This may lead to a more formal approach to testing.

Failure to control how testing is conducted can have a negative impact on the perception of the production team. Care is required to ensure that the team takes the right approach to obtain the correct output.

Questions you can ask yourself:

  • Does my testing methodology fit with the resource availability?
  • Is my testing thorough enough to meet business expectations?
  • Will the testing form part of my production process?

Creating gold standards requires internal and external considerations to be factored into the equation. Accounting for these diligently, and developing processes that reflect this, are critical for success in data-driven operations.


At Engaging Data, the implementation of the Gold Standard is something in which we always strive for and perpetuate in our work.

Deeply entwined in our work Engaging Data and The Gold Standard are synonymous.

Therefore, this being said, wanting to have a Data Consultancy in which you can have confidence in will be assured their work will be up to The Gold Standard, feel free to contact us for our services and products!


Gold Standards – Part 4

Gold Standards – Part 4

Engaging Data Explains :

Creating The Gold Standards in Data –

Part IV –

Balancing Internal and External Considerations


In this final part of our four-part blog on gold standards, we’re going to examine the importance of external considerations. These are factors inherent to the industry and location of your business that impact directly on your company and its day-to-day functioning.


Firstly, every industry has its own regulatory body and best practices that influence the way you work. In recent years, laws around the use of data (GDPR) and how it can be stored have evolved, with the aim of better protecting personal data.

To return to our cake shop example, the Food Safety Act 1990 provides the framework for all food legislation in England, Wales and Scotland. The legislation is concerned with, among other things, ensuring that food businesses take responsibility for how food is labelled, advertised and presented, so that it is neither false or misleading.

Laws and regulations therefore directly impact the processes, people and technology used to create cakes, while also potentially increasing production costs.

Questions you can ask yourself:

  • What regulatory bodies impact my cake shop, and how does this impact on the gold standard?
  • Are there any laws that may impact my cake production?
  • What if a new law or regulation arises, am I prepared for this? 

Economic (Demand drivers)

All customers have different demands; they require a range of products at varying price points. As we have discussed previously, the product price is dependent on the cost of production, however this is also influenced by the economic strength of the country in which the product is created.

Production Costs

The economic status of a country, whether recession or prosperity, has a direct impact on the cost of production and the purchasing habits of customers. Furthermore, various countries can produce the same products at different prices, while the strength of currencies can also impact the global market.

If a country is in recession, customers may shift from premium to mid-range products, accepting that the quality may diminish. Consumers may also purchase less frequently, or even stop buying entirely.

Here are some simple examples of economic impacts on the cake shop:

  • Recession may lead to using low-quality ingredients, or reducing the bakers that manufacture the products.
  • When the market is growing, this may lead to higher demand and / or production, or to an increased quality of ingredients.

The economy of a native country will drive the demand for cakes. This change in behaviour is something that cannot be controlled, but planning can be put in place. Building flexible and adaptable processes can enable your cake production to shift with demand, even though achieving this can be challenging. But identifying trends in demand is key to adapting efficiently and keeping your customers happy.

Questions you can ask yourself:

  • How can you detect changes in customer demand?
  • Are your gold standards set up for change? Can they adapt?
  • Are you and your team prepared for change?
  • Can your infrastructure cope with change?

Technology (Levers for improvement)

Technologies used in cake production are varied, but they must always support the processes and people involved in manufacturing.

Within the cake shop, the oven could be considered a bottleneck. If you have the space for more ovens then you can scale up your operation. And if there is an oven that can cook the same amount of cakes in half the time, this may be a better solution to the bottleneck.

Each item of technology within your process has its own limitations and potential. Understanding this will help you adapt to any situation that you encounter.

Questions you can ask yourself:

  • Is there a warranty on the ovens that we use?
  • Do I licence the equipment that we use, or do I own the equipment?
  • Do I understand the lifespan of our equipment?

Ongoing Delivery

The job can never be considered done, especially for data warehouses, particularly if data is used to make business decisions about customers, or to support business change.

Once the initial gold standard has been created, both periodic and failure reviews are recommended to ensure that standards continue to produce the value expected. These reviews can start at any point of the production process, but should be thorough and constructive, assessing all key elements of the people, processes and technology.

These reviews should be consistent in assessing elements against the following criteria:

  • Does the Process still fit – did anything about the process slow down the delivery?
  • What was the customer experience – did you create a good news story? Did the customer get what they wanted? Did this add value above what was expected?
  • Best Use of Technology – did the technology let you down?
  • Best use of People – were there any gaps in knowledge or skills, or impacts on customer experience?

Ask the difficult questions rather than reviewing the successes and you’ll have a much better chance of improving your operation.

Internal Considerations

As well as considering internal outcomes, there are also internal factors that will inevitably impact on the production process. Once production is up and running, there are usually support functions that are also part of the process, which are separate from the production team. These functions will help keep everything running smoothly. 

When the cake shop initially began trading, bakers were responsible for ensuring that the company’s machinery was in good working order and fit for purpose. This was a collective responsibility, but the bakers decided amongst themselves that one particular employee, Jerry, would carry out the maintenance, as he restored classic cars in his spare time and therefore already had an advanced skillset.

This was fine for some time. But when Jerry was out of the country on holiday, the food mixer broke down at the least opportune moment; right at the start of a large order. Jerry had the most experience and knowledge of this machine, with the other bakers having relied on him in these scenarios. In order to ensure that the order was fulfilled, the manager of the shop took the decision to purchase smaller food mixers, and pay the bakers overtime. The order was completed, and in the aftermath the management team reviewed the root cause of the delay in processing.

Subsequently, the decision was made to use an external maintenance company to monitor and maintain kitchen equipment. The shop’s equipment would be checked every month, to ensure that everything is in good working condition. In addition, the maintenance team will also be on call should something break during normal production hours.

The maintenance company ensures that all engineers have the relevant training and certifications to work on the machinery. Although the bakers may know enough to fix the machinery, it is not their job to deal with this, enabling them to focus solely on production.

But the shop continues to consult with the bakers when purchasing new equipment, as they keep abreast with the technology trends in the industry. This helps best practices be maintained.

Questions you can ask yourself:

  • Are roles and responsibilities suitable for your process?
  • Do you have any key risks?
  • Does your support team have the right experience?
  • Are there any service level agreements with the support team?
  • Does the support team add to the overall process?

Testing

Developing a testing method is another important internal consideration, yet it involves a simple process. Agile development is key here – revealing important data insight as it’s being developed. This constant checking builds certainty and comfort with business users, as they approve the state of data during development. This approach breaks down testing time and, in some cases, can reduce the overall development time. 

However, not all businesses can adopt this approach. Company policies, laws and regulatory requirements may impact the options available, and will be reflected within the internal process. This may lead to a more formal approach to testing.

Failure to control how testing is conducted can have a negative impact on the perception of the production team. Care is required to ensure that the team takes the right approach to obtain the correct output.

Questions you can ask yourself:

  • Does my testing methodology fit with the resource availability?
  • Is my testing thorough enough to meet business expectations?
  • Will the testing form part of my production process?

Creating gold standards requires internal and external considerations to be factored into the equation. Accounting for these diligently, and developing processes that reflect this, are critical for success in data-driven operations.



Gold Standards – Part 3

Gold Standards – Part 3

Engaging Data Explains :

Creating The Gold Standards in Data –

Part III


In this third part of our blog on gold standards, we’re going to assess the importance of processes in creating an ideal data environment. If you recall from previous parts of this blog, we have juxtaposed the process of dealing with data with the running of a cake shop. And when you’re developing the ideal processes for either type of business, simplicity is critically important.

But ‘process’ also refers to several aspects of the business. It can refer to the discussion of new orders with customers. It can be related to the process involved; the time it may take to bake and distribute cakes. It can also be related to the management of customer expectations and orders.

The process involved with the business can help your team understand what they do and how they fit in, as well as playing a vital role in deciphering and organising team responsibilities, interactions and achieving gold standards. It means that even if team members aren’t specialists in a particular area, they can still refer to established processes and be aware that they will result in a strong final product. For example, if a baker doesn’t know how to decorate a cake, they can understand that it can be decorated, and potentially the time that this will take. 


Understanding The Process

A good rule of thumb is to aim for the production team to understand enough about the process that they can make accurate estimates on production schedules. This demonstrates how much you can produce in one week if everyone works at 100% efficiency. And it also offers details of the effort needed to create the cake, and therefore the return or profit derived from the product. 

Once you have a good understanding of customer demand and production time, you can better manage cost and resourcing, potentially enabling you to offer alternatives to customers who have particular requirements. For example, customer demand may require the cake shop to produce 1,000 cakes in a week instead of just 100. By developing a process around the type of cake shop the establishment wishes to be in the local market, the orders can be successfully fulfilled.

In order to address this, the cake shop conducted some research on the market, reviewing the orders received over the last 12 months. Based on this data, they decided to optimise their operation so that they produced a consistent standard of high quality, mid-price range cakes using the best quality local produce. It is anticipated that this will constitute 80% of the company’s orders. Alongside this, some premium cakes will also be produced, with the intention of satisfying exclusive clientele, after the manager of the store noticed that this sort of cake tends to produce positive social media feedback.

With this in mind, the business decided to create a unified process, enabling her staff to process both types of orders using the same equipment. Different ingredients and alterations in the production process were required for the bakers to produce premium cakes, consequently more effort and attention to detail was needed. Furthermore, separate ingredient storage and labelling – to avoid premium ingredients being used within the standard cake – was also deemed essential.

Whether making cakes en masse to sell to local shops, or an artisan cake shop producing a range of bespoke, high quality and expensive cakes, each manufacturer will develop its own process to suit the desired output.

Questions you can ask yourself:

  • Do we understand what cake or range of cakes that we are trying to produce?
  • Can you easily explain your process to a customer, in terms they understand?
  • Is the manufacturing process suitable for the product/product range?
  • Do you understand the running costs involved with making more cakes?

Tools and Delivery

Producing gold standard cakes is not as simply as merely building a process or hiring the best people. In some cases, you can be limited by your tools. Realistically, there may be budgetary restraints that prevent you from purchasing the appropriate equipment for the job. So it can be best to utilise existing tools, but highlight them within the process as a potential risk to production. This will help with any customer discussions, particularly understanding any impact on production costs.

The new process at the cake shop enabled existing equipment, including the all-important ovens, to be fully utilised in the production of the desired range of cakes. As a result, business is soon booming and there is a large backlog of orders.

After assessing the production process and resources, the manager of the bakery decides that the most cost-effective way of increasing production is simply to add another oven within the production line. The additional oven enables the same staff and process to double the number of cakes that can be produced, while retaining the same processing time. The initial purchase price and ongoing maintenance costs are easily met by the increased revenue.

Questions you can ask yourself:

  • Do my tools enable me to produce our products effectively and efficiently?
  • What limitations do we have?
  • Do we utilise our tools correctly?
  • Do our production process utilise our tools in the best way possible?
  • Are there any other tools that can help me, either by saving resources or production time?

Time

Production time is often hidden from customers. All of your fantastic looking cakes in the shop window. or publicised on social media, take time to produce. Yet customers rarely appreciate the intricacy of the production process. Nor are they usually willing to wait! Therefore, managing expectations around deadlines from the very minute that any customer orders a cake will either address any concerns, which is why it’s always better to talk about this issue first.

Sometimes the thinking or desires of customers can change at the eleventh hour. On one occasion, the cake shop received a call from a customer late on Friday evening. At the last minute, Mr Jones had remembered that he needed to order a cake, so that it was ready for tomorrow’s big party. As one of the company’s bakers completed the order form, he realised that this order may not be possible, based on the existing orders that are already due to be baked tomorrow.  After a quick check with the kitchen staff, he confirms that the order cannot be satisfied, and subsequently informs Mr. Jones. The customer was disappointed, but understood after the kitchen capacity and pre-existing orders were explained to him.

Being transparent with customers and influencers about existing orders and production processes will help any business, but these processes will never stop unexpected orders from appearing at 8pm on a Friday night. However, it does create a consistent customer experience and help companies to mitigate any associated disappointment. Communication can be critical when all other approaches fall short.


Creating gold standards in data, or any business, is always dependent on efficient processes being put in place. These processes encompass every aspect of dealing with internal and external factors, and consequently it is critical for companies to continually refine every aspect of their operation.



The Gold Standard – Part 2

The Gold Standard – Part 2

Engaging Data Explains :

Creating The Gold Standards in Data –

Part II : Assessing the Gold


In this second part of our four-part series on gold standards in data, we’re going to examine the importance of instilling a review process.

Reviewing is never the most exciting nor relished procedure! It often feels like an unnecessary grind that slows down your whole operation. But it’s actually a critical part of any good business, enabling you to identify major faults before they become ingrained.


Reviewing Processes

We’ve already seen in the first part of our gold standards blog that the cake shop reviewed their existing processes before making a decision to change their order form. Often it’s only from self-assessing in this way that you can uncover new ways of working that enhance your productivity and efficiency.

It’s also important to emphasise that this needs to happen across the business. It’s normal to have a review as part of project governance for new projects, but it’s always worthwhile to reassess existing projects as well. They may be ‘good enough’, but they also might not be reaching the gold standard.

So in our cake shop scenario, how will the business assess the standard of the cakes being produced? Well, when the business reviewed the kitchen they found that each cook bakes one order, while also being responsible for checking their own cakes before they move to the decoration stage. This all seemed fine, but then the retailer received a few complaints about burnt edges and sub-par ingredients. Consequently, the cake shop reflected that its existing process of ‘marking your own homework’ was not sufficiently robust to identify problems.

In order to address this issue, several ways of reviewing the production of cakes were decided upon:

  • A simple visual inspection – does the cake look uncooked or overcooked?
  • A thorough test, such as breaking the centre of the cake to see if it is cooked. 
  • Pressing the centre of the cake to see if it springs back.
  • Hiring Paul Hollywood as a tester!

All of these checks are designed to see if the cake has been cooked satisfactorily. The shop decided to opt for all options, with the exception of hiring Paul Hollywood! Instead, each baker will review one another’s baking, with the hope that the business will grow to support a head baker who will review all cakes.

Just as the cake shop reviewed its processes to put a more stringent review structure in place, the same can also be implemented in a data-driven environment. The following questions are examples of some that you can ask yourself as part of this process:

  • Does the team need training?
  • Do you need to recruit new people with different skill sets?
  • Do the products need to change?
  • Is the supply line quick enough? 
  • Would more people or different processes help with efficiency?
  • Is the product still worth the effort that is invested in producing it?

Producing Gold

Once you’ve baked some quality cakes, you then need to take steps to market the product. It’s not enough to just produce the cakes and leave your customers to eat them if you want to maximise your marketing efforts. Building advocacy and influence via your customers is a great way of marketing your cakes to the right people. Word-of-mouth feedback from advocates is trusted by other potential customers, and is therefore far more effective than other forms of marketing.

However, there are two sides to the coin here. Negative feedback can be dangerous if it’s not managed effectively. Negative comments about the burnt edges of cakes will spread like wildfire to existing and prospective customers. But there are ways of recovering from this. Making courtesy calls to customers can provide you valuable insight into the process of ordering and consumption. 

Getting the right team in place, tailoring products for your target market, and taking feedback onboard in an active process are all important facets of contemporary marketing. 

What are the Considerations?

Gold standards always begin with what you are looking to deliver and who this will benefit. Gold standards should be designed to support outcomes, having considered both the internal and external factors that will influence design. Creating steps in the process to continually challenge the functionality of the end product and ensure that standards are still relevant to the end user should therefore be considered essential.

Sponsors and influencers can also play an extremely important role. Both can become prime advocates of your product, with the added benefit with sponsors that they pay you to advertise your goods or service!

Internal Considerations

Data

Data can be compared to ingredients within a cake. Naturally, good quality ingredients are critical to producing the best cake possible. The same applies to data. As we mentioned in part one, if you put rubbish data into your systems, you can expect rubbish outcomes!

At some point, the cake shop company realised that the bakers are not periodically reviewing the ingredients within their cupboards. To address this, the manager inspects everything that they have on hand, ensuring that any poor quality ingredients are replaced, and that anything out of data is thrown away. Labelling is updated, while processes are put in place to ensure that there is no repetition of these mistakes.

The key point here is that while complaints were registered about burnt edges, it may have been the ingredients that contributed to the final product that were the problem. Going forward, the team at the bakery put in place a series of key questions that would inform their processes in future baking:

  • Do we have enough data to make the size of cake required?
  • Are we getting our ingredients from the right suppliers? 
  • Does this product contain nuts?
  • Have I mixed sugar up with salt?
  • The milk smells as if it’s on the turn, should I use it?
  • Do we store the ingredients in the right place, in the correct containers? 

Resources and Teams

When baking your cake, you can select from many different types of bakers or specialist chefs to assist with the process. Or you may decide that you wish to train yourself, or an existing employee, so that they can handle the most challenging baking tasks.

In some cases, if the cake shop utilises industrial equipment, people who have been trained to use this equipment can be deployed, as opposed to bakers or specialist cake makers. Having the right team with the right skills, and/or the aptitude to learn them, can be critical to successfully achieving gold standards. Instilling this in your team culturally is critically important in providing direction to your whole operation.

Achieving this can be as simple as asking yourself the following questions:

  • Is the team right to build the end product? If not, what needs to change?
  • Is the team open to changing or evolving in order to improve the product or efficiency?
  • Do we have the right skills? If not, do we need to second or buy them in?

Company Culture

Finally, failing to understand the company culture will lead to failure. Gold standards must fit into the existing culture, or the direction the company is moving towards.

Understanding how your customers think, managing their expectations and developing a standard to consistently perform to those expectations is simple to conceive. However, the human element of this could result in you developing hundreds of different gold standards for multiple different customers. 

Important questions to ask yourself here:

  • Are the customers knowledgeable about your products? If not, can you educate them?
  • Do you share your practice? Would it help your customers to know what you do and how you do it?
  • Are there any expectations that you can manage? 
  • Are they any difficult expectations you have to work towards?

Implementing a gold standard for data may seem like an all-encompassing and intimidating goal. But it instead should be seen as a granular process. Breaking down the ingredients and individual components that collectively create gold standards is the best way to achieve this aim.