By Ben Atchison, Delivery Lead, Altis Melbourne
Data projects present unique challenges for agile delivery, especially when scaling your team for a large program delivery.
These 3 tips will help your agile data program delivery be a success.
Agile Concepts
Since the publication of the Agile Manifesto in 2001, the adoption of agile delivery techniques has steadily grown. The Project Management Institute’s Pulse of the Profession 2021 reported 70% of organisations are wholly or partly using agile delivery techniques for a wide range of project needs.
There are many approaches to agile delivery, with popular frameworks including Scrum, Kanban, Extreme Programming, Lean Development and Disciplined Agile, each with pros and cons that are worthwhile considering for a given situation or organisation. While the details may vary, these frameworks all share features including
- Iterative development in short cycles (or sprints)
- Encourage close collaboration between those with the product vision and those tasked with implementing it
- Regular re-assessment of priority, based on stakeholder feedback or new requirements
- Empower the team to work creatively and effectively in the manner that best suits them
The Altis Agile framework shown below is one such approach.
- The Product Owner consults with and represents business stakeholders to product a Product Backlog which is a prioritised list of functionality to be added to the solution. Change to project outcomes is supported by a regular re-assessment of priority as part of this backlog grooming process.
- The Scrum Master facilitates the Sprint Planning meeting in which the Team reviews and refines the requirements and commits to the work tasks to be committed to for the Sprint.
- During the course of the Sprint, typically of 2 – 3 week duration, the Team completes the designated tasks according to an agreed Definition of Done (this may vary depending on the nature of the task e.g. requirements gathering, design, coding, testing, planning).
- Each day of the Sprint a brief Daily Standup is held, with each team member stating what they recently worked on, their plan for the day and any blockers in their way.
- Collaboration is key, with the Scrum Master focusing on task definition and removal of blockers, and Team members putting on cross-functional hats to assist other team members as required to complete the committed work.
- At the conclusion of the Sprint, completed tasks are captured as Finished Work. A Sprint Retrospective supports a continuous improvement mindset by looking for improved ways of working and lessons learned.
Scaling Agile
With its origins in software development, agile initially focused on small, self-organising teams. As agile has matured and been adopted by more and larger organisations the need to scale the approach has become more important.
The introduction of agile often requires a fundamental revisit of ways of working for organisations, and the scaling of agile is another level of challenge. With the expansion of agile practices comes an increased risk of misalignment of product development; inadequate coordination, collaboration and management of dependencies between teams; and unrealistic expectations of delivery speed.
Approaches to support the scaling of agile teams have been developed, with popular options including the Scaled Agile Framework (SAFe) and Large Scale Scrum (LeSS).
A common approach is to introduce a portfolio level view of initiatives, with an agile story board dedicated to a high-level view of the delivery stage (e.g. Concept, Scoping, In Development, Business Acceptance, Release, Done) each initiative is at. This portfolio view is very useful to communicate to senior stakeholders how the overall program and individual initiatives of interest are tracking.
Each initiative will then have its own agile story board, product owner, scrum master and team working through the detail as described earlier.
In order to keep the individual initiatives aligned, a weekly Scrum of Scrums (SoS) with the various product owners and scrum masters will review the portfolio level board to communicate progress and discuss dependencies and issues.
Scaling agile delivery successfully is not simply a matter of organising the project and portfolio level team teams in this manner. It is important that senior leadership of the organisation buys in to the approach, and that the objectives of the portfolio remain aligned with strategic objectives of the organisation.
How Data Projects are Different
Data projects have some characteristics that need consideration from a project methodology perspective.
- Organisations typically have a wide and varied list of candidate data sets to acquire into a data platform. The prioritisation of where to start should result from a wide consultation with stakeholders and aligned with the organisation’s strategic objectives. Data owners of each operational area will often see their data as the most important, meaning an enterprise view of data and its relative value is needed, along with endorsement from the executive level. This exercise should be done as part of an up-front data strategy that produces a roadmap of data initiatives and bus matrix of business processes that reflect the organisation’s prioritisation.
- It is important that technical and data architecture is aligned and co-ordinated across the portfolio. Technical components and design patterns used should be determined at the enterprise level to ensure consistency and an enterprise grade solution. Key data domains (e.g. Customer, Product, Organisation) will often be reused across the enterprise solution and a consistent approach to their data modelling is important. Regular communication between teams and showcasing of work performed on key data domains is required to keep the solution aligned.
- Both backend (data modelling and engineering) and frontend (visualisation and downstream outputs) can be delivered iteratively provided sufficient work has been done up front to produce a high-level architecture, design and bus matrix summarising the target business process areas for the solution. The backend work can commence with landing data and building dimension tables in the initial sprints, followed by a series of sprints to build fact tables and additional dimensions as required. The frontend work can start in parallel by gathering detailed data output requirements and prototyping solutions with stakeholders, feeding refined requirements to the backend team for subsequent sprints, before moving on to building the data outputs themselves as the underlying data structures become available. As such, the backend and frontend teams remain tightly coupled and dependent upon one another, and well suited to a structure of iterative build / showcase / refine.
In Altis’ experience, these considerations mean a scaled agile approach can work well for the delivery of a data project, provided sufficient work has been done up front on a high-level architecture and data design.
3 Tips for Success
By focusing on these 3 principles, you can maximise your chances of success for scaling your agile data project.
Alignment
- Establish a Portfolio level view of data initiatives which:
- Shows the stage each initiative is at (e.g. Concept, Scoping, In Development, Business Acceptance, Release, Done)
- Aligns with strategic organisational objectives, owned and driven by one or more Product Managers
- Provides a clear overview of data initiatives to senior stakeholders
- Facilitates regular conversations between leaders of the individual data projects to discuss progress, dependencies and related issues
- Establish individual team backlogs and story boards
- Product Owner and Scrum Master for each team
- Epics of work map back to the Portfolio view
- Used to run the team day-to-day and produce the required data outputs
Transparency
- Communicate and share the organisational strategic aims that the portfolio is working to address
- Ensure the portfolio and team-level backlogs and storyboards are all visible to each other
- Encourage discussion and collaboration across the individual sprint teams working in the program. Rotate team members across the teams.
- Use standardised tools for planning (e.g. Jira, Rally, Azure DevOps), code management and testing.
Built-in Quality
- Variable quality standards across teams are a common hindrance to program success. This can be mitigated by proactive quality assurance provided at the program level and agreed standards for modelling, development and testing
- Undertake regular reviews of architecture and design to ensure individual teams remain aligned
- Standardise the creation of regression tests with each data set and automate the execution of this regression testing suite.
- Centralise deployments of code to ensure a consistency of approach and to minimise any conflicts
Summary
Running your data program using a scaled agile methodology is a powerful way to achieve quick value for the organisation and to allow for flexibility in targeted business outcomes. By using a well-proven approach such as the Altis Agile framework and focusing on the 3 principles of Alignment, Transparency and Quality, your program is well set up for success.
