1. Waterfall
This classic approach was first described in 1956 and is still in use by many Multinational Corporations (MNCs) today as it fits nicely with their budgeting and Purchase Order (PO) workflow. With the Waterfall method, all requirements are scoped upfront in complete detail and a fixed budget is assigned and accepted based on the estimate. Then, the project is implemented following the schedule to the letter, and finally tested. This works well with off-the-shelf (OTS) solutions that have fixed requirements and limited customization, but is less suitable for custom projects as it has very little flexibility and limited tolerance for unforeseen risks. This leads to a significant chance of budgets and timelines not being met due to the unpredictable nature of custom projects.
2. Agile
There is ongoing debate on what Agile actually is, but a particular variant called Scrum has become especially popular in web and mobile project management. In general, you scope the main requirements upfront, and start implementing, iterating and testing in weekly or bi-weekly cycles of deliverables. Agile rightly states that not all risks can be predicted, and its processes are better suited to custom development projects as it allows for early prediction of missed deadlines and budgets. However, the process seems more focused on predicting these issues than mitigating them. Agile is also difficult for MNCs to implement as it requires significant changes to their traditional workflows, flexible budget and timelines, frequent touch points and fast decisions.
3. Iterative
While Agile is also considered iterative, you can push that to an extreme. In Iterative development, there is almost no scoping upfront, and development starts as soon as possible with frequent iteration and experimentation. This is a preferred method of tech entrepreneurs who initially do development themselves, but it does not scale well. In a team context, it is great for producing prototypes quickly where quality is not a concern. Mistakes are expected and part of the discovery process. This makes the Iterative method highly unpredictable but quick to deliver something tangible.
The shortcomings of the three methodologies, along with factors like poor planning and communication, are the leading cause of many pain points in custom development projects.
Changing Tack
The approach I take is mostly Agile but also draws on Waterfall for additional up-front research and risk assessment. I believe that research and planning beforehand can identify and mitigate risks which will help better manage expectations with regards to budget and deadlines. It is equally important not to get stuck in a drawn-out research phase, as tends to happen with the Waterfall approach. Focus on those topics that have the highest chance of disrupting your timeline and budget. Research enough to gain an understanding of the risk and be able to estimate the effort to mitigate or resolve this risk. Keep in mind that you do not need to reinvent the wheel. Consider who might have already researched or worked on something similar and see if the knowledge can be gained through other means - partnerships, or simply purchasing it, for example.
As with Agile, it is essential to recognize that you cannot predict all risks: you must always expect, and budget for, unforeseen events, and be flexible enough to accept changes during development. You can achieve this by quantifying and calculating the risks and likelihood of changes, which will give you margins that you can add to the timeline and estimate. Having margins will help with managing expectations and changes, and unforeseen events can be implemented and managed within these margins.
For MNCs, following the Waterfall method means that the development team is not involved until after much of the scoping has been done. This leads to technical decisions either being made by non-technical people or being pushed far too late into the process, resulting in significant changes to the plan after the technical team has reviewed the requirements. On the other hand, startups working with an Agile methodology involve technical team members early on, but with overwhelming expectations of independent feedback and ideation placed on them. This level of involvement also requires the technical team to have a deep understanding of the business, industry and long-term roadmap of the project and business. When working with third party development agencies, in the current market of commoditization, these are unrealistic and unfair expectations.
I have had the best results from projects where the development team was identified early on in the process, and heavily involved in research and technical scoping of the project. However, their involvement was structured and focused on the immediate technical aspects of the project, with the business and long term goals thought out and briefed to them prior to their involvement.
The first part of the project scope (the brief) should give the development team sufficient information to work with and reduce the level of ideation expected from them outside of the immediate technical requirements, while still letting them know the longer term plans for the solution so they can be factored into the architecture and scalability.
This approach is more compatible with MNC workflows and is easier for them to implement than pure Agile. Planning and scoping well will ensure that sufficient margins are added to estimates, as well as managing the development team’s expectations early on to ensure they fully understand what is expected of them and are able to estimate appropriately - including sufficient time for consultancy, research and testing.
One more thing of note is that the three methodologies above almost exclusively talk about project requirements, goals and technical features. Rarely do you see any mention of Return On Investment (ROI), business requirements, risk mitigation or other commercial aspects of a project which play an equally important role in my approach.