Virtual health – evaluating in a shifting landscape

Earlier this year, Rachael Butler and I completed a series of rapid reviews for Waikato District Health Board, looking at current thinking in virtual health – where advances in technology enable remote forms of healthcare delivery. Some anticipate that it will comprise a substantial proportion, if not the majority, of patient interactions in the future.

What is virtual health?

Discussion about telehealth and telemedicine approaches are often used interchangeably with virtual health. In the work we developed with our partners, virtual health was proposed as an evolutionary step beyond telehealth and digital health approaches, which is patient-centred and engages people as partners with clinicians in maintaining and improving their health.

Reported benefits of virtual health include improved access to healthcare and health and wellness education, increased patient engagement and satisfaction, a reduction in direct and indirect costs, and improved health outcomes. Virtual health programmes have also been linked to positive outcomes for health providers, including improved operational efficiencies, a reduction in costs, alleviation of staff/provider shortages, and access to a pool of clinical resources.

There’s always a 'but'

We found that virtual health initiatives have the potential to deliver patient-centric care, create more collaborative ways of working and make health systems more efficient. But when you’re talking about potential, there’s always a but. And with virtual health, clinical uptake is often a key challenge, along with the readiness of organisations to implement technology, and the suitability of technology for the purposes intended. Furthermore, users of virtual health systems – both clinicians and consumer/patients – should be involved in developing new initiatives, be adequately trained, and feel confident that the technology is safe, offers clear benefits, and is easy to use.

A copy of the synthesised reviews can be accessed here, where the above discussion is dealt with in substantially more detail.

Evaluating virtual health

When we presented on these findings at the 2018 ANZEA Conference, I offered some reflections about evaluating these types of approaches, drawing on the writing of Trisha Greenhalgh, one of the leading thinkers in this space. Trisha Greenhalgh’s 2018 book, How to implement evidence-based healthcare, is an excellent starting point.

plans.png

The landscape of virtual health technologies is one that is constantly being iterated and refined. Maintaining a stable, unchanging intervention is really challenging in the context of apps that are being updated on our phones on a near-daily basis. So evaluations that are based on a linear assumption of stability and predictability may well be flawed from the outset. Evaluating technological interventions need to be seen in the light of something that is iterative, recursive and a long-term process – a developmental process.

Source: http://thedoghousediaries.com/5468

The six worlds

Greenhalgh writes of six ‘worlds’ with significantly different stakeholder perspectives and expectations of technology interventions; each with their own evaluation needs:

  • Political, where technology is a vehicle for delivering policy, improving efficiency and providing measurable benefits to patients
  • Clinical, where technology is a tool to support professional practice and improve quality of care
  • Personal (patient/carer), where technology supports the individual’s personal health and wellbeing
  • Technical, where the focus is on design and as a software development project
  • Commercial, as a way of delivering return on investment
  • Legal/regulatory, in terms of information governance.

Getting comfortable with the complex

Greenhalgh and colleagues are particularly critical of systematic reviews of randomised control trials that overly simplify both the technological interventions and the outcomes they expect, and which often fall back on needing more research to gain a greater effect size. Instead, she argues for greater use of mixed methods and ethnographic research that embrace both context and complexity, to better understand the pathways and processes through which technological interventions fail or succeed, and use real-world experience to make sense of multiple and imperfect data sources.

In short, virtual health approaches are here to stay, and they will continually be refined and adapted to better meet the needs of people – many of which will fail to do so. Understanding technological value and impact needs to embrace the complexity of the different worlds of technology, and work similarly developmentally and iteratively as their real-life experience unfolds.

 

The review that this post is based on was commissioned by Waikato District Health Board over late 2017 and early 2018. The Smarthealth and Healthtap initiatives were outside the scope of our review.

 

Dovetail in 2017

Another year goes by and another opportunity for reflection. Thanks again to all who have been part of Dovetail's success - as clients, collaborators, and conversations.

For Dovetail, it was another productive and satisfying year. One of the biggest highlights was being part of the Te Ara Mua Future Streets team which scooped up the Innovation Hub and Supreme Awards at the Bike to the Future Awards.

I was also very pleased to be part of publishing a few journal articles, my first for a while, on evaluation, bikelash and urban form.

Clients kept things busy as always, which is great because the variety of work is constantly engaging, and it gives an opportunity to help people working at the coalface. Some new research and evaluation directions emerged through this, including social procurement, financial capability, social enterprise support, and just at the tail end of the year, a new foray into ocean conservation.

The election blogs provided time for me to geek out to my heart's content, and to keep things on an even keel, I set myself an audacious goal to get to 52 gigs in 52 weeks.

And for me personally, 2017 had some pretty tough moments. I'm incredibly grateful to the friends, family, colleagues and clients who were there to get us through. I can't thank you enough.

Dovetail Annual Report 2017.jpg

For a clearer and printable version, you can also download a pdf here.

Getting more out of business as usual

Every day we see public works taking place – street improvements, parks development, public facilities, and transport improvements. Fundamentally these are about making places better for people to live, work and play.

But what if councils and government agencies could make this happen in a way that delivers additional benefits, or social value, to communities? What if the value that ratepayers receive from public services could deliver some extra social value that can help strengthen the fabric of communities?

What if the way that councils and government agencies purchase services could be used to get these additional benefits?

One area of work that I’ve really enjoyed this year has been social procurement. It’s an emerging practice in New Zealand, particularly in Auckland, and is well-established overseas, especially in the UK and Australia.

It’s about using business as usual purchasing and contracting to get extra social value. The key ingredients are:

  • Policy – to set the conditions for social procurement to occur in practice
  • Contracting – to specify the social value that an organisation is seeking providers to deliver, and to shape the market’s response
  • Market development – to build innovative approaches and new supplier markets
  • Supplier development – to support organisations, such as social enterprises or community organisations, to become fit for purpose players in the market and able to deliver social value

When organisations like Auckland Council spend over $3 billion per annum on procurement, there's a real opportunity to use the procurement level to get more benefits for communities.

Earlier this year, Rachael Butler and I completed an evaluation of a social procurement initiative in Mt Roskill, Auckland. The project, called Te Auaunga, was at one level simply a flood mitigation initiative for a local stream (also called Oakley Creek).

At another level, the initiative offered a range of placemaking opportunities, including park and environmental restoration, and new community facilities.

And at yet another level, Auckland Council and the Puketāpapa Local Board saw opportunities to bring about further social benefits to train and employ local young people, and set up a social enterprise plant nursery.

The employment initiative recruited 17 young people to complete a construction skills certificate course at a local training centre; many of them were then channelled into positions with the works contractor and to the social enterprise.

The social enterprise, Te Whangai Trust, established a native nursery on the grounds of a local school, and in the process created a new community hub, and hands-on learning for school children. The trust’s work focuses on bringing long-term unemployed and people with mental health and/or offending histories into sustained employment through building work and life skills. The nursery brings in 60 people each year to deliver the plants, planting and plant maintenance for Te Auaunga.

As our evaluation showed, the process was not without its challenges, and we understand that retention in the youth employment initiative was a particular challenge. But the project highlighted some real potential for social procurement, and lessons for future activity.

I’m currently working with The Southern Initiative at Auckland Council to document a series of social procurement case studies. The report on this will be coming out in the next few months. In the meantime, our report on Te Auaunga can be accessed here, at Auckland Council's RIMU website.

With a new government committed to regional development and sustainable employment, there may well be a role for social procurement to do more than just deliver capital works over the next few years.

With thanks to Auckland Council for commissioning the evaluation of Te Auaunga.

Relentless dynamism - the 2017 election

After a tumultuous campaign, one of the most dramatic in living memory, the election results rolled in on Saturday night, and I thought a few reflections would be useful from the initial voting data. Special votes, some 384,000 of them (15% of overall voting), are currently being counted, and the final result won’t be known for a few weeks. But a few things are worth noting…

Turnout increased... and fell

The number of people voting increased by 110,000 from 2014, up from 2.45 million to 2.56 million. This means that turnout has increased for two elections in a row. Furthermore, the growth in the number of votes cast, at 4.8% between 2014 and 2017 is higher than the growth in the electoral population (3.6%). While this does suggest a more engaged voting population, it wasn’t enough to deliver Labour the breakthrough they were looking for.

And, as Andrew Geddis points out in this piece on coalition prospects, in 2017 only 91.1% of those eligible enrolled to vote, compared to over 93%. He estimates that the turnout of those eligible to vote actually declined from 72.1% to 71.7%.

At least two of the main polls got it right at the end of the campaign

There was a lot of talk in the campaign about the reliability of the polls, fuelled by the contrasting findings coming out between different polling companies. Many queried if landlines were a suitable way to poll people.

In the penultimate week of the campaign, the Newshub-Reid poll had National substantially ahead. At the time, I wrote that it looked like it was an outlier, and the perceived poll volatility led some to go so far as to suggest poll failure. Yet in the final week, the final Colmar-Brunton poll came into line with the Newshub poll, with almost identical results. The final Roy Morgan poll was less accurate, and which in likelihood reflected that it was completed 10 days earlier, and over a longer timeframe.

The table below shows the eve of election poll results published before the election, and the election night initial results. The green cells show which were closest for each party. The closeness of Newshub and TVNZ polls to the election night results are striking. Across all parties, Newshub varied by only 0.8 percentage points on average, and TVNZ by 1 percentage point.

 Comparing the main publicly-available eve of election polls with election night results

Comparing the main publicly-available eve of election polls with election night results

Momentum shifted

The shift in the polls in the final two weeks of the campaign was equally striking. National’s scaremongering campaign clearly had a substantial effect on voting intentions, as shown in the next two graphs, updating earlier posts (this time using a non-linear trendline, one of my own take-homes for future blogging). For National, their vote returned to the mid-40s by the end of the campaign.

nat trend 170925.png

Labour’s polling appeared to have peaked in the first week of September, then fell away afterwards. The overall growth in the Labour vote will be pleasing for them (increasing from 24% in July to 36% on election night, and a huge gain from the 2014 election), but the decline in the final two weeks will still be deeply felt. 

lab trend 170925.png

Prospects for the final count

The election night result had National with a projected 58 seats, Labour with 45, New Zealand First with 9, the Greens with 7 and Act with 1. With 61 needed for a majority in the House, New Zealand First have the choice of aligning with National for a comfortable majority (67 seats), or Labour and the Greens for a bare one-seat majority (61 seats).

Over the next two weeks, special votes will be counted; this time around, there are more special votes than ever before, totalling some 384,000. In past elections, special votes tend to favour left-aligned parties. Two excellent pieces of analysis by Graeme Edgeler on Public Address, and Michael Appleton on Twitter, look at the possible impact on the final vote tally if the pattern of special voting in 2014 is repeated.

Their analyses suggest that the final result could increase the number of seats held by Labour and the Greens by two; and a corresponding fall for National. Time will tell.

If this is borne out, it would give Winston two quite feasible coalition options on paper – 65 with National and 63 with Labour and the Greens, both with some level of stability to pass legislation and budgets, and hold a coalition together. At this stage, it is anyone’s guess which way he will go. Because at that point, the coalition becomes less about numbers and more about policy, spending and portfolio concessions – and weighing up if the overall vote is a call for stability or a call for change.

With the largest number of seats, National are claiming victory. However, despite the triumphalist tone of the party on election night, there’s no constitutional obligation in New Zealand to align with the largest party; the Governor-General simply needs to be satisfied that the coalition that has formed has sufficient members to command a majority in the House.

The next two to three weeks will be full of speculation and intrigue, until the next house of cards is eventually built.

Everything to play for

Last week I looked at the variation in the polls, with the Newshub-Reid poll appearing to be an outlier amidst a trend towards Labour. This week however, the final polls are in, and the Reid and Colmar Brunton polls are almost identical, showing National leading Labour by 8%, at 46% to 37%. This was apparently helped along by a bruising but effective campaign against Labour’s budget figures and tax policies (and widely criticised as misinformation). The Roy Morgan poll, completed 9-10 days earlier, had National and Labour level-pegging at around 40%.

And so to the poll of polls, which in this final version, averages out the eve of election polls for each of these three publicly available polls. National, at 44% have a lead over Labour at 38%. The Greens appear to have recovered from their slump and are at 8%. New Zealand First are at 6%, still losing support week by week. 

Among the minor parties, TOP are on 1.3%, Māori Party are on 1.1% and Act are on 0.5%

When the campaign began, I wrote that Labour needed to strengthen their position to be the leading party of the left; energise and mobilise the many disaffected voters who simply didn’t vote in previous elections; and chip away at National’s vote. As the graph below shows, they very nearly did all three, but National have just pulled away in the last week.

parties 170920.png

And this is where things get interesting. If this scenario holds – and acknowledging the Roy Morgan is more than a week shy of the Colmar Brunton and Reid polls – it points to a hung parliament. The table below shows what happens when these figures are put into the Electoral Commission’s MMP seat allocation calculator.

I’ve assumed that Act will retain their Epsom seat, and that the Māori Party will pick up two electorates as well. I’ve also put my finger in the air and suggested National will pick up 35 electorate seats and Labour 30.

With MPP it’s the party vote that determines overall proportionality. So, under this scenario, pictured below, National would have the most seats with 54, but Labour and the Greens combined would have 57. Both left and right would have to try to negotiate a deal with New Zealand First – and in National’s case, the Māori and/or ACT parties as well, to get a majority.

parliamentary seats 170921.JPG

At this point, it’s about counting to 62. And that will come down to Winston Peters. In this scenario, would he align with National, but also Māori Party and ACT, two parties that he sees as an anathema to his own policy agenda? Or would he go with Labour and the Greens, many of whose policies he has also criticised (and in 2005 he refused to go into government with the Greens)?

Of course, this poll of polls scenario may not play out that way. Those of you who have read my posts can fairly easily read between the lines to see which side I personally favour. But whichever side of the fence you prefer, this scenario suggests that all parties still have everything to play for.

So, get out and vote, it really counts.

 

The data in this blog is taken from a poll of polls of the last three publicly available polls in New Zealand: One News-Colmar Brunton, Newshub-Reid Research, and Roy Morgan. The figures are simply an average of the most recent poll from each organisation.