Much is murky in open access, but this at least is clear: academic libraries have committed different amounts of money and staff toward an open-access future, from a flat zero up to hundreds of thousands of dollars' worth.
It's the zeroes and near-zeroes that concern me (why, hello there, Yale, and hello again, Yale), though I also believe quite strongly that libraries that have made significant investments of money, staff, and/or political capital should be recognized and praised for it.
The difficulty here is that it isn't just the scale of open-access investment that varies. The nature of investment varies as well. Some libraries pour resources into their institutional repositories, while others have one but don't support it well (or indeed even adequately), and still others don't have one at all. Some libraries have flourishing open-access publishing programs. Some libraries have a plethora of memberships with open-access publishers, while others have a few strategic ones, and still others none whatever. Some libraries have dedicated staff to questions of scholarly communication, at various heights in the hierarchy; others have not. A few libraries have open-access author-fee funds.
At this early date, I don't think it wise to constrain experimentation and local decision-making by assessing only certain sorts of open-access investment. (I am aware I differ in this from other open-access advocates, and that bothers me not in the slightest.) Institutional repositories make more sense at some institutions than others, as do author-fee funds and almost any other intervention one could name. Likewise, we want to encourage new kinds of open-access advocacy, such as collaborations between libraries and university presses to make more work open access, or work toward campus-wide mandates; nailing down a laundry list of interventions too soon might damage library incentive to innovate.
I do think it's time and past that academic libraries should be evaluated on the scale of their open-access investment, however. Because all academic libraries benefit when library resources are usefully reallocated to open access, it behooves academic libraries as a group to discourage freeloading. Naming freeloaders in a nationwide assessment context should do beautifully, because like it or not, libraries pay significant attention to their position in such rankings. It would also be useful for academic-library decisionmakers to have a general sense of how libraries are allocating open-access resources, so that they can gauge their own commitments and shift them as seems appropriate.
In the United States, two organizations dominate nationwide academic-library assessment: the Association of Research Libraries and the Association of College and and Research Libraries division of ALA. It has sometimes been noted, and not favorably, that the criteria these organizations use to assess libraries are mired in last-century practices. I suggest that assessing commitment of resources to open access is an important way to shift that perception usefully, while helping academic libraries to make smart choices about their own open-access investments, and employing judicious peer pressure to increase the size of everyone's open-access pie.