v32#1 One Academic Library’s Approach to the Learning Analytics Backlash

by | Apr 1, 2020 | 2 comments

by Kirsten Kinsley  (Assessment Librarian, Florida State University) 

Editor’s Note: A correction was issued for this article after publication for an omitted citation. A note from the author is included as a comment below, and the citation has been added as requested. A correction will also be published in the v32 #3 June issue of ATG.

Initially, the editors of Against the Grain asked us to share our experience developing our library data dashboard or library cube prototype.  In the meantime, we had to wrestle with the backlash to library learning analytics (LA) activities which included the development library data warehouses.  This has prompted us to take immediate action within our library to begin to address some of the issues presented by the backlash. We wanted to share our response in the hopes that telling our story might inspire others to share theirs with us as we move forward. 

Like many academic libraries, when ACRL’s Value of Academic Libraries: A Comprehensive Research Review and Report (Oakleaf, 2010) (VAL Report) was published, we took the initiative seriously.  We planned our assessment activities to align with university-wide goals and metrics following the Great Recession in 2008.  The financial crisis further drove the need for institutional accountability which demanded more data to prove value and impact while it heightened competition for more limited resources within the university and among colleges and universities in the state.  Those realities in the last 12 years have steered higher education’s goals and strategies to align with the emergence of statewide higher education performance metrics. These “political and economic drivers” are one of many reasons for the development of learning analytics, which emerged as a “separate field” in 2010 (Ferguson, 2012, p.309,311).  As defined by the Society for Learning Analytics Research (SoLAR), “Learning Analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs …” (SoLAR, 2011/2020; Ferguson, 2012, p. 305). LA is also known as educational data mining (Jones & Salo, 2018).  In the last few years a backlash to educational data mining or LA, has emerged or been portrayed as in direct conflict with library professional ethics of intellectual freedom, privacy, and confidentiality, and the balance between the intellectual property rights of vendors with the interests of library users (Jones & Salo, 2018; ALA Code of Ethics II, III, & IV, 1939/2008).1 

At our institution, the effects of the Great Recession were no different from other institutions.  A library assessment department at our university library formed in 2010, which coincided with the publication and influence of the VAL report (2010).  Our departmental research moved from an ethnographic focus to a more quantitative one. The focus shifted to measuring possible relationships between library services, spaces, and collections to student success by merging library data with demographic and student performance data collected by the campus office of institutional research.  Working in partnership with other campus collaborators, we intentionally sought linkages between library usage with aggregated and de-identified student performance metrics valued at the university and statewide levels. Moreover, as a culmination of our local research and data collection efforts, we sought to build a library cube or data analytics warehouse prototype (Klein, Kinsley, & Brooks, 2018) in part to de-silo our data and “gather and connect these studies and datasets in one unified database for easy querying and reporting” (Klein, Kinsley, & Brooks, 2018, p. 359).

However, this past year, just as we made a local breakthrough in the development of our cube/warehouse proof-of-concept, the learning analytics library backlash was in full swing.  We were going to write a piece about the warehouse for this publication, but the concerns about learning analytics as practice began a robust, ongoing, and challenging conversation.  From around 2015 till now there have been a number of articles written, along with the IMLS-funded Data Doubles Project (Asher et al, 2018), and discussions at conferences that reflect the backlash to the Learning Analytics movement (Jones & Salo, 2018). 

Before we discuss our response to this, it is essential to attempt to summarize some of the issues brought up by the backlash literature:

Consent & Transparency:  There is a lack of consent from and transparency to our users about the data we collect and why we collect it.  For example, we collect card swipe data of those who come in and leave the library, but we do not tell our students what we do with that data.  Visitors to the library can not opt-out from having their data collected because it is data owned by a third party (campus police). We want to find out how they feel about it once they know what we do with it (See the recent Pew Research Center Report by Auxier et al., 2019). 

Data management practices & library-wide data management standards or a lack of standards of practice library-wide could open the risk for a confidentiality breach (Briney, 2019).  For example: by not having a checklist for off-boarding former employees, they could retain access to user data after they depart from their position.

Validity: Do we measure learning by performing learning analytics?

°  Learning Analytics is often a misnomer as research under that description as its analytics often do not measure actual learning.

°  There is an over-reliance on institutional metrics, such as GPA, as a proxy measure of learning.

Qualitative Approach

°  Librarians need to be more equipped to conduct quantitative research.  For example, learning the importance of using a measure of effect size, “rather than being satisfied simply with statistically significant findings…” or p-values (Robertshaw & Asher, 2019, p. 90).

°  By searching large data sets, it might be tempting to look for patterns in the data that are statistically significant (go on “fishing expeditions”) rather than starting with a research question, collecting data, and following it with analysis (Jones & Salo, 2018).

Social Justice Concerns

°  Using LA for predictive analytics seems to benefit the institution more than the agency or intellectual freedom of the at-risk student (Nicholson, Pagowsky, & Seale, 2019).

°  Assuming LA measures what users’ value: One of the challenges in the development of LA posited by Ferguson (2012), was to “focus on the perspectives of learners” (p.313). 

°  The use of big data could “expand racial inequality” because its algorithms contain the explicit and implicit bias of the creator to predict behavior that targets particular groups (Le, 2017).2

At our library, one of the ways we responded to the local conversations about this resulted in healthy dialogue, prompted in-house library training information sessions (FERPA, IRB, and ethics talks), and has started to improve assessment awareness and practices.  But, before we pat ourselves on the proverbial back, we do this knowing that we in libraries can strive to be a bastion of privacy and advocates of intellectual freedom, all the while we, together with our patrons carry our wifi devices, or “spies in our pockets,” everywhere we go (Thompson & Warzel, December 19, 2019, para. 9).  Even access and usage data of our services and resources are not entirely under our purview or control.

Another way we tried to address the backlash was to create a privacy policy.  We crowd-sourced a draft of our most comprehensive library privacy policy with the knowledge that it might need to be a flexible and adaptable document that can promote our ethics in a data-cooperative environment within the university.  There is a need to balance our values with the goals and metrics that align with our institution. We have to accept that though we can employ practices that eliminate risk to data breaches while operating within the realities of the academy, we cannot eliminate all risk.  

Some authors of the draft privacy policy wanted to provide powerful language and strict guidelines for data sharing and consent by taking an advocacy stance for student privacy.  That stance needed balance with the necessity of being a team-player with our institution. A section was added to the privacy policy that stated that we would not participate in LA practices.  It was modified to attempt to find a middle ground and allow for LA activities, but it ended up reading with two differing voices. Since those differing points of view could not quite reconcile, that section has been set aside for now.  The privacy policy was recently submitted to the library administration for review and approval without that section. Those that drafted the policy would like to know what the library administration thinks about learning analytics practices.

In our efforts to address these concerns we may also be tempted to adopt the politically charged/inflammatory rhetoric sometimes used by proponents of the backlash to point out the issues and problems with using LA in libraries.  We need to understand the problems and concerns the backlash enumerates. However, we also need to be realistic about how we communicate and advocate for these concerns within the socio-political institutions within which we co-exist and operate.  Framing the problem with exaggerated, provocative, unrealistic language and scenarios may not allow us to be effective advocates of user privacy. For example, words likening library research that tracks usage as student “surveillance” implies continuous, round-the-clock monitoring of every student’s movement as if a library or university intends to run itself like a police state. Inflammatory rhetoric ends Kyle Jones’ (2019) piece, Just Because You Can Doesn’t Mean You Should: Practitioner Perceptions of Learning Analytics Ethics where he quotes Hathcock (2018, para.4):

In stark terms, April Hathcock argues that learning analytics ‘is a colonialist, slave-owning, corporatizing, capitalist practice that enacts violence, yes violence, against the sanctity of a learner’s privacy, body and mind.’ (18) 

The shock value of this quote gets us to pause, think, ask more questions, and listen to some more.  However, it also has the effect of intimidating and shaming those who are trying to be true to the profession’s code of ethics while they seek to understand library users without malicious intent in order to make connections with how libraries benefit our users as collaborators with our institution and its endeavors.

At the same time, the radical response of the backlash to LA has resulted in some very productive improvements and refinements in our practices locally.  It revealed an unspoken potential disconnect between our professional values, ethics, and practice. It may spur libraries that consider conducting LA activities to create a comprehensive privacy policy.

It provides an opportunity for libraries to: 

1. Model proper data management and research practices on campus.  Implement library-wide standards of practice template to address personally identified information protection across the library.  We have developed a standards of practice template and are testing it with assessment datasets before sharing it across the libraries.

2. Keep the LA dialogue going without denying either side of the argument.  Draft a policy which includes divergent voices.

3. Tell users about what we are doing with their data and find out how they feel about it.  We are developing a poster to display by our card swipe turnstiles to inform and educate library visitors about the data we collect when they swipe in and how we use it.  We simultaneously want to survey them about how they feel about this.

4. Be champions of student and faculty privacy and intellectual freedom while balancing that with being an active partner in aligning with campus initiatives, goals, and strategies.  We partnered with our campus compliance and ethics officer to draft the privacy policy. As a result, the student affairs office wants to see our data privacy policy once it gets approved by library leadership. 

5. Advocate and educate our users about their privacy and intellectual freedom by opening up this conversation on campus and influence campus policies on data governance (Jones & Salo, 2018, p. 305).  We have two librarians involved with campus-wide data governance initiatives on campus. 

6. Improve the quality of our research.  Within the library literature and research, we can intentionally describe how we handle our privacy concerns.  We can “invest in training staff and providing infrastructure in security, up-to-date anonymization procedures, and data privacy practices …”  (Briney, 2019, p.29). 

Even with all these positive opportunities, we will still need to reconcile the ALA Code of Ethics (1939/2008) as it relates to patron privacy and the perceived threat of LA to data privacy on college campuses.  Though we have been wrestling with the issues brought up by the backlash efforts, we have not even begun to employ LA on our campuses in a pervasive way.  Therefore, now is the time for us to be engaged in these conversations locally. There is a lot we can do to prepare for our response to LA on our campus. We can both engage with the campus and operate within the boundaries of our profession.

References

Aguilar, S. J.  (2018).  Learning analytics: At the nexus of big data, digital innovation, and social justice in education.  TechTrends, 62(1), 37-45.  https://doi.org/10.1007/s11528-017-0226-9

Asher, A., Briney, K., Goben, A., Perry, M., Regalado, M., Salo, D., Smale, M. A., … Jones, K. M. L.  (2018).  Data doubles project.  Institute of Museum and Library Services (LG-96-18-0044-18).  Retrieved January 14, 2020, from https://datadoubles.org/.

Auxier, B., Rainie, L., Anderson, M., Perrin, A., Kumar, M. and Turner, E.  (2019, November 15).  Americans and privacy:  Concerned, confused and feeling lack of control over their personal information.  Pew Research Center: Internet, Science & Techhttps://www.pewresearch.org/internet/2019/11/15/americans-and-privacy-concerned-confused-and-feeling-lack-of-control-over-their-personal-information/

Code of Ethics of the American Library Association. (1939/2008).  Retrieved January 14, 2020, from http://www.ala.org/advocacy/sites/ala.org.advocacy/files/content/proethics/codeofethics/Code%20of%20Ethics%20of%20the%20American%20Library%20Association.pdf.

Ferguson, R.  (2012).  Learning analytics: Drivers, developments and challenges.  International Journal of Technology Enhanced Learning, 4(5/6), 304.  https://doi.org/10.1504/IJTEL.2012.051816

Hathcock, A. (2018, January 24). Learning agency, not analytics. At The Intersection. https://aprilhathcock.wordpress.com/2018/01/24/learning-agency-not-analytics/

Jones, K. M. L.  (2019).  “Just because you can doesn’t mean you should”:  Practitioner perceptions of learning analytics ethics. Portal: Libraries and the Academy, 19(3), 407-428.  https://doi.org/10.1353/pla.2019.0025

Jones, K. M. L., and Salo, D.  (2018).  Learning analytics and the academic library:  Professional ethics commitments at a crossroads.  College & Research Libraries, 79(3), 304-323.  https://doi.org/10.5860/crl.79.3.304

Klein, J., Kinsley, K., and Brooks, L.  (2019).  Building a “library cube” from scratch.  Proceedings of the 2018 Library Assessment Conference: Building Effective, Sustainable, Practical Assessment: December 5–7, 2018, Houston, TX, 359-370.  https://doi.org/10.29242/lac.2018.33

Le, V.  (2017, October 13).  Big data and social justice are on a collision course.  The Greenlining Institutehttps://greenlining.org/blog-category/2017/big-data-and-social-justice-are-on-a-collision-course/

Nicholson, K. P., Pagowsky, N., and Seale, M.  (2019).  Just-in-time or just-in-case?:  Time, learning analytics, and the academic library. Library Trends, 68(1), 54-75.  https://doi.org/10.1353/lib.2019.0030

Oakleaf, M. J.  (2010).  The value of academic libraries: A comprehensive research review and report.  Association of College and Research Libraries, American Library Association.

Robertshaw, M. B., and Asher, A.  (2019).  Unethical numbers?  A meta-analysis of library learning analytics studies.  Library Trends, 68(1), 76-101.  https://doi.org/10.1353/lib.2019.0031

Society for Learning Analytics Research (SoLAR).  What is learning analytics? Retrieved January 14, 2020, from https://www.solaresearch.org/about/what-is-learning-analytics/.

Thompson, S. A., and Warzel, C.  (2019, December 19).  Opinion | Twelve Million Phones, One Dataset, Zero Privacy.  The New York Times.  Retrieved from https://www.nytimes.com/interactive/2019/12/19/opinion/location-tracking-cell-phone.html.  

Endnotes

1.  Many concepts of the backlash are encapsulated in the Library Trends special issue v.68 issue entitled:  Learning Analytics and the Academic Library Critical Questions about Real and Possible Futures — Kyle M. L. Jones features as editor).  

2.  There are benefits to performing library LA, and we should be able to discuss these as well.  For example, the benefits of LA and social justice in education include getting a more individualized education and the ability to self-monitor progress to one’s educational goals (Aguilar, 2017).

Sign-up Today!

Join our mailing list to receive free daily updates.

You have Successfully Subscribed!